OP, the biggest issue when you get older is trying to keep up with a 20 year old.
I'm 75 and I did a back pack hunt for Black Bear last spring.
You know your capabilities, hunt accordingly.
If you hunt with someone, make sure they're willing and able to do their share of the work on such a hunt, or it can go sour pretty quickly.
I went out with a fellow last fall, when the animal was down, he folded on me. Didn't help with the gutting, skinning deboning and didn't want to touch "raw meat."
Luckily, the trek out was only a klik, but that was his last hunt with me.
55 isn't a stretch for some folks, depending on their mindset and fitness. Other folks convince themselves they're too old by this time, defeating the hunt before it starts.
Now, what do you describe as a "backpack hunt?"
Such hunts can be morning to dark, walking the sidehill trails, overnight with a small tarp for a tent, and hopefully a lightweight sleeping bag capable of keeping you comfortable enough to sleep. Then some folks go all out, such as "toebako," but from his pics, someone "carried a stove into camp?" on foot? on a backpack?
I haven't done it this year, but maybe during the last couple of days of the season, in November, I will put my two man insulated ice tent in the box on my SxS quad, to get back to an old camp I know, to see if I can't come close enough to a 4x4 Mule Deer, that doesn't come down until the snow drives him out. Maybe I can talk my 55 year old neighbor into coming along?