Welcome to shooting single actions. I have heard stories similar to yours for decades, which is a common scenario after developing skills with double-action revolvers, then changing to single actions. In short, single actions with traditional grip frames do indeed require more skill to master, but if you can master them, it will generally improve your DA revolver shooting as well.
In spite of the Ruger XR3 grip frame being rather small, many shooters with XL/XXL hands can grip and re them just fine. Shooters with large hands such as yours often drop the little finger below the grip frame. This may feel odd at first, but this method can help in exact duplication of your grip position from shot to shot. Your hand should be positioned on the grip frame so the palm supports the backstrap, or is somewhat behind it, and the barrel is in direct alignment with your forearm. It is imperative to position the gun in your hand exactly the same for each shot. Likewise, the amount of grip pressure executed must be identical from shot to shot. Lots of dry- fire practice (with snap caps) is suggested. Watch your sights after the hammer falls to be certain you are not “pulling” shots.
After much practice, if you are still unhappy with the results, you might consider purchasing oversized target stocks from Herrett’s Stocks that are much more forgiving as to hold and hand positioning.
The Ruger Bisley grip frame is very forgiving (as to hand position, etc.) while offering outstanding comfort, especially for shooters with large hands. It is also the best grip frame I have used for handling heavy recoil from cartridges such as the .475 and .500 Linebaughs – not that it is a problem with the .44 Special. Unfortunately, these benefits are not free, as it adds significant weight, is slower for fast-draw work and, if the revolver is carried in a hip holster, increases the height of the balance point.
Another important factor to improve your SA shooting skills is to obtain a good trigger pull. Most Ruger New Model Blackhawk revolvers are too heavy out of the box. A competent revolver smith can tune that pull for a light, crisp let-off, which will help with offhand shooting. Be sure to check and tighten screws as needed, as a loose grip frame will contribute to poor accuracy.
I hope these tips help, and don’t give up on that single action too quickly. Once you have mastered it, they are marvelous field guns.