I agree. This is probably what happens the vast majority of times. I know this from my own experience. It's either that or I get those "cold bore fliers" on a variety of rifles, not all the time, but often enough that it's my shooting that is at issue. That first shot is a bugger to get down cold, the very first time.
You know yourself and you know your rifles, that's probably correct for your experience. What though, if someone consistently has the first cold shot go out a certain direction and distance from the rest of the group? I would think if shooter error were to blame the flier would be random direction and dispersion distance and like you say, not occur all the time. Enough people have documented having a rifle or two that do this and they know they are not to blame as they have other rifles that do not suffer this malady. OP really isn't giving enough information though, some feedback from them would be appreciated.
It is unlikely to be anything relating to a barrel "heating" up as this just doesn't happen between the first and second (or third, fourth, and fifth) shots. These are not CF rifles, after all.
Yeah, the barrel gets just a little "warm" after shooting a box of 50 nowhere near enough heat to warp the barrel. Are we talking about cold bore as in freezing, stick to your tongue if you lick it cold? Or is it just took it out of the case in the middle of summer "cold"? Again not enough info from OP. In a truly "cold" barrel the lube will take on a different viscosity than it has when warm, friction from a shot or two will warm it up and get it slippery again. I can see this having an effect on the first cold shot.
Myself I've never experienced a rifle with cold bore issues. I always start my range day with a clean barrel and I know it takes some shooting to get things settled in the bore, I can then rack the rifle shoot a different one and go back to it later with no flier issues. Cold weather turns ammo performance to crap so I haven't done testing about temperature related fliers as I just don't shoot in winter.