The care and handling ( or neglect and abuse) that hunters subject their guns to varies widely. Just be mindful and careful of your nice rifle and you'll be OK. Here's my experience.
I've never done serious damage to any of my fine rifles while hunting. I have a couple dozen hunting rifles that I use regularly. All are stocked in wood. Some of the wood has stunning figure and is rare and expensive.
I've used my rifles on long canoe trips for moose that involved many portages in miserable weather, and while backpacking in the mountains of BC and the Yukon, or pounding along while stored in a case on an ATV in Saskatchewan, and while in hand and on a Land Cruiser in the dust and mud of Namibia. In and out of vehicles, up and down tree stands, boats and blinds. Strapped to a pack or in a scabbard on a horse. None were harmed more than slight scuffs that can be touched up with a little Tru-oil. Never gouged, or cracked and certainly never broke a stock.
I did get lucky a few times. Like when my rifle was snatched from the saddle scabbard by a branch while descending a mountain in the Yukon. I was leading my horse down a creek bed through a hellhole of willow tangles. Rifle spent the night in the creek, mostly submerged until rescued the next day. Retraced our steps and found it, no visible harm done. I dried it off, did a test shot, and went on to shoot a big mountain caribou that afternoon.
A properly sealed wood stock is tough! Hunting with an artistic firearm is part of the pleasure that I get from the whole activity. So I'd also say hunt with yours!