dry fire is your friend here. You are pulling into the recoil (pre-compensating as it where) Have a friend load a snap cap at random in your mag if you don't think that's what your doing. Odds are at the shot you'll see the gun pull down.
Start dry firing against a target taped to the wall, or a easily sighted on point like a light switch. For heaven's sake, make sure you don't have any ammo out and around, and clear the gun multiple times before you do this at home. The wife WILL be pissed if you shoot the TV!
Once you can hold the gun thru the trigger break and not have the sights move from your aiming point (no dip, weave or bob as the hammer drops) your ready to try a few rounds live. Odds are that in a mag or two you will see yourself doing it again, so go back to dry firing and concentrate on breaking the shot smooth again. If you have a 22 auto, this is an excellent tool...you will usually learn to break it smooth before the centerfire. Once it's breaking smooth, go back to the 9 again. Shoot deliberately until you see the low and left, then stop and dry fire or 22 until your shooting center again....repeat until your consistent with the 9. Might come quick, might take some work.....
99.99% of the time, accuracy issues are trigger manipulation, with a side order of grip'/stance. Don't try to mash the trigger when the sight picture is perfect...you will pull it farther off that way then if you center the sights and slowly squeeze thru. The sights will float a bit, that's ok to an extent. your error will be smaller if you don't try to jerk the trigger, even though the sight picture is slightly off. If you see the gun start to really weave, shake, circle, finger off the trigger and lower the pistol and take a break. Remember to breathe, but pausing while you break the shot it ok.
The more you practice this, the quicker you will be able to do it accurately. Please note, it's far easier to learn the be accurate and then get fast, then it is to teach yourself to shoot fast, then try to get accurate. TAKE YOUR TIME, make EACH shot count. Don't be afraid to only put 3-5 rounds in a mag and try to shoot each one perfectly then take a short break. Again, the more you practice this, the easier the concentration becomes, and the better you will shoot.
Practice does not make perfect, perfect practice makes perfect. You can bang off 10 000 rounds and learn nothing if you just toss it all down the range as fast as the gun can cycle....try to analyse each shot as you take it. Sometimes it helps to put up multiple targets so you just take one shot at each target then pull them in and look. Mark/patch the strike then do it again, lets you know what each shot has done.
Most service type pistols will shoot 3-4" at 25 yards if you practice some much better, at 12-15, they should be 9-10 ring solid (1.5")...that gives you something to work towards
This WILL make you a much better shot if you work at it. It is work/practice. Once i got good at it, i found it relaxing, but initially it can be less fun then just dumping fast in the general direction of the target. Good luck