That's normal to a point. Unless you're bench rested it's probably impossible to hold a gun perfectly still.
If you're just laying your gun on something, and don't have an attached bi-pod, you could try moving the rest point forward as far up the stock as you can. Then can change the fulcrum point of your rest so that your movement is less exaggerated.
For obvious reasons it's good to avoid too much coffee or energy drinks before you go shooting. You need to be very relaxed.
When I'm grouping I will even try to take it real easy. Stay relaxed, take my time walking to the target or setting up. Anything to keep my heart rate down.
You could also try finding ways to loosen up on your gun a bit so that you're not holding it so tight with your hands. One thing I like to do when using a bi-pod is position myself so that I can lean shoulder weight into the buttstock in such a way that the rubber feet on the pod kind of grab the surface I'm on, then I can relax my hold while still maintaining control of the gun.
On the bench, with just the stock rested on a block or something, grab the bottom of the buttstock with you left hand to pull it into your shoulder, a sling helps with this one, then relax the grip on your right hand. Try shooting with only the very tip of your finger on the trigger, and only your thumb on the back of the stock to counter it, turning your squeeze into more of a pinch. Mind you that technique works better with light triggers.
Also, if it's your scope, and problems with parallax, one habit I've gotten into, is once I've gotten my shot line up, I scan my eye around the the end of the scope and make note of how even the "fuzzy circle" appears to be. If I notice one side is thinner then the other, I know my eye is not perfectly lined up and correct so that the fuzzy circle appears even all around. If you don't know already, you primarily focus on your reticle.