My universal load with Varget in a .30-06 is to use 50 grains with a 165 or 168-grain bullet.
This load seems to work in everything, including my Garand. In fact, it is pretty well the only load I use in my Garand, but it is also very accurate and consistent in bolt guns.
With your 24-inch barrel, this combo should give you between 2,750 to 2,800 f.p.s. and I'd be surprised if it grouped larger than 1.5 inches (though it may well group much better).
I've never met a .30-06 that didn't like 50 grains of Varget with a 165 or 168-grain bullet.