POI shift can be caused by a lot of variables. While it is true you may see some shift with a smaller contour barrel, like your average hunting rifle, but there can be other factors as well. Lack of a proper mate between the barrel/action and the stock, improper torque of the action screws, improper fitting or worn recoil lug mating surface or gaps/flexing, improper ring mount torque, ring torque, and we havent even talked about ammunition variables.
If you sit down to shoot a string, even a light countour hunting barrel of decent qualitity (in a suitable platform) will pump out 5 back to back shots without much of a POI change, in under a minute. Beyond that, it will be difficult as the heat will slowly start to make changes. You may get lucky and get 7-10 shot string, but that will depend on the how hot the load you are shooting. Having a heavier contour will allow you more shooting with less down time, so shooting a 10 or 15 shot string now becomes easily feasible, as the larger barrel will act as a larger heat sink. All that being said, a heavy contour barrel installed improperly or on a flimsy stock, you'll see the same issues youve already experienced.
Hope that helps a little.