OP, I've had that issue with more than one rifle.
Some can be very finicky, especially those that have been "weight reduced" by using light stocks and ultra thin barrels.
I've been using a Remington 700 Mountain Rifle for hunting this fall. When I first started shooting it, groups were abysmal at best.
I pillar-bedded the laminated stock and glass-bedded it afterward. This rifle is a dream to carry and comes up perfectly to my shooting eye, without having to search for the scope.
Even after all of this, I had the same issue you have.
I removed the scope and bases, cleaned up the threaded holes, then put it all back together, being sure to make sure the screws and bases were not moving around.
The scope is a H5 Swarovski and parallax is excellent.
None of it worked and this is usually where I stop fooling around and get out a different rifle.
Then it dawned on me, I had this issue before with similar rifles, such as Remington Model 7 types with pencil thin barrels.
That meant I had to start looking at my handloads.
The first thing I did was take 20 rounds to the range, shoot them from a cold barrel, and set the cases that shot fliers aside.
I weighed those cases, as well as the others, they were close but a bit heavier than those which didn't shoot fliers. They were also "longer"
That meant the necks had to be trimmed, which brought the weights close enough.
Then, I reduced the powder charge, of H100V, by two grains. I left seating depth alone because the rifle was only throwing odd flyers and the groups were very good.
I ended up reducing the powder charge another grain, before the fliers, which wasn't my fault, stopped.
We all chase velocity, it's normal. High velocities make us feel good and when the cartridges shoot well with high velocities, even better.
However, when we try to push the envelope, and pressures increase, they do so erratically, once they've reached a certain level.
When this happens, you get different harmonics and unexplainable fliers, regularly.
This can be very hard to track down. Especially when there aren't any visible pressure signs. Not only that, the load may shoot extremely well in another rifle, but give issues in the one you're shooting.
Check the torque on the scope mounts and rings, then check the receiver screws to make sure they aren't loose or to tight, causing stock crush, then run a piece of paper under your barrel, between it and the stock for free float and that it isn't touching anywhere.
Some older stocks have a "bedding" block at the front of the fore-end, to create upward pressure on the barrel. This is to settle down harmonics and can create "fliers" as the barrel warms up. Sometimes it's prudent to remove that block and sometimes it isn't.
If your rifle's bedding is sloppy, then no matter what you do, until its glass bedded, you're going to be chasing a rainbow.
