Odds are a varmint load won't get to the vitals, leading to a horribly wounded animal. 150 -165 gr. are more than capable of taking any Canadian game animal at humane/ethical distances.
If you shoot them in the arse with a cannon, like as not they will run away and likely die elsewhere.
True story. A guy I worked with bought ENTIRELY in to the 7mm STW myth, and shot a moose in the ass with it, then watched slack-jawed, as it walked off into the swamp, no doubt to die painfully.
Bullet placement is key!
FWIW, I shot a doe in the ribs with a .223 and a Winchester 50 grain bulk bag bullet. She took exactly one step and cratered in. I recovered 39 of 50 grains of that bullet, against the inside of the off-side front leg. Lungs were mush, heart not salvageable. Meat loss was pretty close to nil. One of several deer that I have been involved with, taken with too small, cartridges, and varmint bullets.
What 'everybody knows', isn't the same as what happens.
If you push a bullet really fast, and don't choose your spot to hit well, you can reasonably expect to dump a LOT of your deer meat onto the floor while trimming. If you keep the speeds reasonable, or even slow, you can pretty much, as they say, eat right up to the hole.