As bizarre as it sounds, bullets do tend to settle down in flight.
It can be compared to a spinning top. As soon as you pull the string and get it spinning, it wanders around a bit, but soon stabilizes.
I recall reading where the US Army did penetration tests on steel plate with AP ammo. They sometimes found that the bullet went in slightly off angle at 25 yards, but hit square on at 100 yards, and penetrated further.
It is important to keep in mind the difference between line of departure, which of course would expand the group as you go farther away, and the small but measureable "orbits" that the bullet does in flight.
If you were sighting in a laser gun (dead straight light beam, no wobble, etc.), then your theory would hold true.
In reality, you probably will have a reasonable clue as to how your load will do at longer ranges, but not an exact micrometer reading.