Nothing wrong with being different. Experimentation is how we learn, as reloaders. But I would like to offer some advice. I knew a fella years ago, who used very light...125-130 grain bullets in his .300 Winmag. Your Short Mag is pretty close ballistically, so a relatively fair comparison. He also used 4064 powder. Don't remember the load he was using, but he got some very high velocities with them. His accuracy was erratic though, as yours appears to be. If you limit your shots to under 200 yards.... these bullets should prove effective on deer sized game. But, if you don't place that light bullet just so...you're likely going to end up with a big splat, an ugly, but shallow wound and a lot of tracking of a wounded game animal. That light bullet is going to start out very fast, but, will shed energy pretty quickly, too. The TSX is a good bullet. Still, I wouldn't consider it the best choice for that cartridge. My opinion only...based on hunting with a .300 Winmag for 30-plus years, I wouldn't trade good penetration and expansion, for a few fps of velocity. I believe if you start with a good 150 grain bullet, then work up until you find the bullet your rifle likes....your groups should improve dramatically. Food for thought if nothing else.
Also, when developing loads for this cartridge, there is no need to go up tenths of grains in load development. Increments of 1/2 grain will work quite well. Once you've found a powder charge that delivers good accuracy, a bit of tweaking of bullet seating depth should get the results you're after. One thing to remember is that, one hole groups are fine in a target rifle. A hunting rifle that is consistently capable of delivering MOA groups under real hunting conditions, is all a guy needs. I don't know what sort of hunting experience you have....only that you are a newbie reloader. But hopefully you can take something from this.