About the grouping problem...
Most older (and maybe newer ones, too) Remingtons with wood stocks have a pressure point built in at the end of the forearm, about an inch or so long. You can see it when you remove the stock. The idea is to tame, at least to some degree, barrel vibrations and thus provide better grouping, which sometimes works, and sometimes doesn't. (IMHO, most of the time it doesn't.)
As your barrel heats up it will start to deflect off this pressure point and throw shots as you describe. My Vanguard would shoot 3" groups no matter what. I replaced the stock with a good synthetic that free floated the barrel. The first group I shot during sight in was 5/8", and it consistently group under 1" now.
My belief is, when in doubt, float the barrel.
This can be done with a bit of judicious work with sandpaper...or have a smith do it for you if you're not comfortable with that kind of work.
Can almost guarantee you this will solve your problem.
Or...
It should take no more than about 10 lbs of pressure, which isn't much, to pull the forearm away from the barrel so that it's not contacting. If it takes more than this, too much pressure. Again, sandpaper is the solution, taking it down a tad bit at a time until you get a reasonable level of pressure.
But, again, floating it completely is usually the best solution.
It's highly doubtful that the scope is the problem.
IMHO.