I am assuming that you are shooting at or near 1000, in which case 2.5" of horizontal spread is only a quarter minute, so wind, mirage, and gyroscopic drift aren't getting you, it's just the vertical stringing. 25 fps variation will give you about a one inch spread at 1000, so actually while your ES could be tighter, there are other things going on here. Perhaps there is a tight spot in your barrel channel which puts upwards pressure on the barrel as it heats up.
If your rifle range is on the side of a mountain, air currents could lift the bullet sharply - just like a strong gust of wind does horizontally - just prior to striking the target. Do you always shoot at the same time of day? The air might be more stable early in the morning.