Like iceman said.
Some rifles more than you would like,others,next to nothing.
It seems to be more to do with oil in the bore than fouling , when you fire a fouled bore group from a cold bore every shot you often get your best grouping, and, to original point of aim. Some of mine will stay right on 1st shot if "lockeaze" is left in a clean bore, all are closer with than without it.My "problem" rifles get (labouriously) sighted in with a clean and lube for each shot as I hate rusty holes.
A bedding fault will keep the p.o.i. shifting about. check the barrel channel first.
My rifles are more consistent year round with magmum primers, minimizes the way low shots at 25 below. Best develop loads in summer to avoid nasty pressures from heat.