Rifles changing point of impact from one shooting session to another, is most likely caused by the scope mounts.
I have found this with every rifle, including 22 rimfire, that I had side mounts on.
Good bases and rings, and that includes Weaver, bolted solidly on the top of the action, as far apart as possible, holds from year to year, for me.
In my case though, they are always on a rifle which has been well bedded, so there is little change in the rifle over reasonable time.