Buy the hunting rifle and learn to shoot it with precision.
This.
And yes, a pound or two makes a difference. Every extra ounce tires you out - especially if it's something you're keeping "in hand" for extended period. Try and get a steady shot off after 8-10 hours of wandering around in dense bush (hot, humid, or cold and sh!tty and raining or snowing - always seems to be one or the other), with a rifle in hand, and muscle fatigue starting to set in. A pound, one way or the other, can make a big difference.
The upside to a heavier rifle is comes down to prone target shooting (tactical-ing). The weight is inertia, and will dampen your muscle movements to an extent. The weight and corresponding inertia helps to dampen felt recoil. From a stand/sit/kneel position, that isn't as much of a big deal, you can roll with it. From prone, it matters - you're in a locked in position and you're body doesn't move with the recoil, and it also hits you on the top of the collarbone where you have less natural padding.
The heavier barrel of a "tactical" barrel will be no more accurate on a cold bore shot. The advantage comes during continuous, repeated firing. A heavier barrel will take longer to heat up, and be less affected by it, than will be a lighter barrel. But that's not even in play for the first three or four shots.
So the question is, what do you really want to do with it? Which activity is more important?
Get out in the bush and look for that trophy buck?
Or sit at a bench or lie on the ground and put tiny holes in paper as close together as possible?
I like doing both, but I don't expect one rifle to excel at both.