Hmmm.....This is the precision rifle forum, so I'm not sure how tough comes into the equation, but lets see. I'll say that a Mauser style CRF makes a superior hunting rifle action, and the Remington solid ring of steel which keeps the case centered in the chamber is advantageous in the accuracy game. Trouble is that I've had M-700's which have been dependable in the field, and I've had target rifles with M-98 actions which could hit a dime on demand at 100, so perhaps there is more too it.
If you were to choose the .300 magnum I would advise you get the longest barrel available to best take advantage of that cartridge's long range capability. The .308 will do fine with a shorter barrel - and in the accuracy world shorter barrels are stiffer, and stiffer barrels are considered more accurate - all things being equal. There is no reason why if you are a good shot that you cannot make the claim, "If I can see it....I can hit it!" when armed with a .308.
Stock material comes into question when we consider that wood stocked rifles can shift their POI as temperature and humidity changes throughout the year, and sometimes from location to location. In your case it seems that a synthetic stock would be advantageous.
I guess when toughness is a consideration, the concept of using the rifle in the field might come into play. Most of us these days have come to use high powered, dimensionally large, variable powered scopes. This might not be the best arrangement on a field rifle as these scopes can be fragile, but the new products available from US optics seem to take toughness to a new level. I have a fixed 6X Khales sniper with a steel tube which is much tougher than my 6-20X56 variable, and comes in at less than half the price of the US Optics stuff - and it has a bullet drop compensator for the 168 Gr. .308 load. Some scope mounts made for bench rest are made very precisely, but fail in the toughness equation. So despite the additional cost and weight one should consider tactical mounts on a rifle which is to be used in the field.
A tough rifle for use in the field, should be equipped with a bi-pod, and the PH is the toughest of the breed. I would also recommend an 1.25" leather shooting sling from Leslie Tam in Hawaii.
By now your rifle probably weighs close to 20 pounds, which brings up the question...how tough are you?