If you shoot a fast 5 shot group, how far does the first shot strike out of the group? This is more likely a problem with a light barrelled hunting rifle with a pressure point in the barrel channel of the stock than it would be for a free floated heavy barrelled target/tactical rifle. If the barrel is light enough to heat up quickly, then it is light enough to cool off quickly, but as stated above, much depends on the case capacity of the cartridge you fire, not to mention the ambient temperature, sun or overcast conditions and wind conditions on the day of shooting.
You can bring down the barrel temperature quickly by pouring water down the barrel. The purists will scream, but it does no harm. When I conduct rapid fire drills with my .375 I limit myself to 5 rounds then dunk the rifle in a slew to cool, then run the drill again. Some of us are not adverse to shooting in the rain either.