My first gun as a teen at not much more than a hundred pounds was a Mod70 featherweight carbine in 30.06. My Dad loaded some light loads for me and I still use it today, only with 180s. I've had it for about 35 years and its never had a problem. Filled every deer tag with it this year.
A light gun will kick a bit harder given the same round, but not much harder and its lighter to carry.
I think you're making a good choice for a first hunting rifle, even in .308
You had to add that "I shot mine as a little hundred pound boy" After seeing my post eh?

Just kidding man, but in all honesty, I have a 7mm rem mag and a 12 ga , neither hit as hard as the model 70, I didnt say it hurts or that its to much , just that it kicks. Totally worth it though. Fun fact, it still has a x4 made in japan 40 year old bushnell scope on it lol. I wanna replace it, but its how it came to me 100% zero'd and has not budged, and I'm finding it hard to replace it.
I should mention to the OP, I inherited mine from a friends father and since the day I brought it home, my brother has wanted to buy it ,every time he sees it or handles it , he makes an offer

It's just a beauty of a rifle. Took it out shooting the other day, we had all kinds of guns, levers, pumps, semi's and of the 4 people there , every one of them fell in love with the rifle.