As everyone seems to agree, there's a point of diminishing returns going light. Maybe not when you're dragging yourself up a mountain, but definitely when it comes time to take the shot that got you going up that mountain in the first place.
That's the demising returns part of the argument. Then there's the sanity part.
Many years ago on a hunt with Alaskan guide Jake Jefferson, we had our grizzly down early and were waiting in camp for the float plane to come fetch us. While passing the time, Jake offered to let me try a shot with his guide gun. It was one of the very early Ultralight Arms bolt actions. No scope, just peep sights -- and as I recall, it weighed in at something like 4-1/4 pounds. It hardly felt like a rifle at all -- more like a wand.
We had camped on the shore of a small body of water just big enough to get a float plane to land and take off. It was maybe 200 yards to the other side, and on the far shore was a boulder around 24" across. I shouldered the rifle offhand, squeezed carefully and ...
Jake was impressed, he told me afterwards -- the shot was a good one and I had hit the rock dead centre. I neither knew nor cared. All I was really aware of is that I had never, ever been hit that hard. And with certainty, I knew that I never wanted to be hit that hard again.
And thus it was that I got to experience what a full house load shooting a 400 grain bullet out of a .416 Remington Magnum feels like from a 4 pound rifle.
Never again.