As mentioned, recoil is a subjective thing...and no two people are going to perceive the felt recoil from a rifle in the same manner.
Body size and weight can affect felt recoil, as well as your body configuration.
Those with short necks and arms are going to feel recoil differently than those with long necks and arms, as their hold on the rifle will be different. This is where stock design is going to be a factor and whether a straight stock vs a monte carlo stock is going to be more comfortable to get a good cheek weld on and lessen the felt recoil.
I have witnessed smaller people handling more recoil from larger calibers better than larger people...but much of this has to do with shooting form too. Again, as mentioned above, knowing how to stand and shoulder the rifle properly has some to do with it, as well as allowing your shoulder/torso to roll with the recoil, instead of trying to horse that rifle into a stiff shoulder, so that your overly tensed shoulder cannot roll with the recoil, and your torso doesn't flex so that your hips and legs can help absorb the energy and assist you in maintaining your balance. This is also why we feel recoil differently in different shooting positions; shooting prone or from the bench does not allow your body to roll with the recoil as we do when shooting from the standing position.
Rifle weight is another factor that helps with felt recoil. A lighter rifle or shotgun is going to move more with the recoil of larger cartridges or shells than a heavier firearm. While an ultra lightweight rifle or shotgun is much nicer to toe around in the field or up a mountain for hours, they do not hold as steady, and behave vastly different during recoil than a heavier forearm. The trick is to find that balance in firearm weight compared to the power of the cartridge that the firearm is chambered to, in order to not be beat up by the firearm while shooting it. There is a reason why so many bolt action firearms are built in the 5-8 lb range, without rings, bases and scopes.
Another factor is the buttstock.
Drop in the comb and heel for traditionally open sighted rifles was more to allow a good cheek weld for using those open sights. Using scopes on these firearms raises the cheek higher from the stock to allow use of the scope, but causes the side of the chin to take the recoil. This definitely changes how you feel that recoil.
The type and size of the butt pad that it has. Many light weight cartridges saw crescent butt pads of metal or plastic, as the cartridges didn't produce a lot of recoil. You normally do not see larger, harder hitting cartridges with this configuration.
Better recoil pads are available for use on harder hitting rifles and shotguns, in order to reduce felt recoil and allow the shooter to be more comfortable and shoot better.
Use of a muzzle break will help reduce felt recoil, but will increase the noise dramatically. Other shooters on the shooting line will feel the impact of the air coming out of your brake, so you may not be overly popular on a busy range shooting line. It will also kick up snow, dust, grass, leaves, etc. when shooting prone off a bag or bipod, and this can affect your ability to call your shot when hunting. Personally, I do not care for them, and do not have one on any of my rifles, but they are popular and do work.
Personally, I can handle the recoil of most rifle/cartridge combinations; but like most people, I shoot the milder ones better.
My old Rem 700 BDL in 338 Win Mag with the factory butt pad would beat my shoulder black and blue off the bench after 20 rounds of factory 210gr PT ammo. After installing a LimbSaver pad, I could go 60 rounds off the bench without being bruised and sore. I could tell that I had done some shooting, but it wasn't uncomfortable. While I could produce 1" groups consistently with the original pad, the groups did get smaller (3/4") after replacing the pad.
I have the same pads on my 376 Steyr and 416 Taylor as well, and they are manageable off the bench, the latter shooting 350gr bullets @ 2400 fps. Both rifles will consistently produce 1/2"groups when I am doing my part.
I am a big guy at 6'2" and 300 lbs, I spent years shooting various rifles and calibers as a child and teenager, from various shooting positions under the guidance of my grandfather who had been taught in the US Navy. Shooting form was drilled into us as kids. I didn't shoot a rifle with a scope until I was in my early 20's.
Rifles that have hurt to shoot have been a friend's single shot rifles chambered in 450-500 3 1/4" with full powerhouse loads with 500gr bullets, and a 510 Wells with 700 gr bullets. These make the No.1 in 458 Win with 350gr bullets feel a lot more manageable.
I have only ever had two rifles recoil so that the scopes brushed my brow (no cuts); one a friends BRNO in 416 Rigby with hot loads with 400 gr bullets and another friend's Winchester Model 70 XTR in 300 Wby with factory 180gr ammo. These two rifles produced wicked felt recoil! I have shot rifles in 300 and 338 Ultra Mag and 340 Wby that did not recoil as sharply as the two aforementioned rifles did.
The worst recoiling rifle I have ever owned was a Sako Finnbear Deluxe in 300 Win Mag. This rifle was Magnaported, and lightened by the original owner and produced a lot of noise and felt recoil. I developed a flinch shooting this rifle when I was in my early 20's. Since then, I have plugged and muffed in order to help work on the flinch. It usually takes me 100-200 rounds every year to get back to where I can shoot a rifle and not close my eyes (blink) during recoil, as my flinch is more from the noise than the recoil. The extra practice also helps re-establish muscle memory and shooting form, which greatly enhances my ability to shoot consistent groups with my rifles.
I almost got that proverbial black scope eye once lol. I was never able to finish a full pack of 20 rounds without feeling that my shoulder was dislocated 
