If it was me I'd stick with a .308 or a .30/06 and handload my ammo to a level that wouldn't be disturbing due to blast and recoil. Light bullets kick less than heavy bullets, and shooting light bullets at lower velocities lessen both recoil and blast. Consider the loads that I taught my grandson to shoot with; 125 and 130 gr bullets loaded to 2600, or 195 gr cast bullets loaded to 1800 fps.
So much for recoil, now lets look at accuracy. Typically you would use a .308 on big game out to 300 yards. Even a smallish big game animal presents a target size of about 12" on a broadside shot. If your rifle shoots consistent 4" groups at 100 yards, chances are it will print all its shots in 12" at 300. Now here's the thing about a 4" group, particularly one that is made with 5 shots; 80% of those bullets will be within an inch of your aiming point and the remaining 20% will be within 2".
If I was you I'd stop shooting groups. The only purpose it seems to have in your case is to destroy the confidence you have in your rifle and your ability to use it. Rather than shooting groups at bullseye targets, put up a large piece of cardboard with no aiming mark on it, and shoot 2 shots at it as quickly as you can from a supported field position. Then after checking the placement, wander off and fire another two shots at it from a different range, a different angle, and from a different position.
Do this for a couple of weeks with ammo you can tolerate, and I bet you become a better than average game shot, and it won’t take long before you can tolerate full powered ammo. If on the other hand you're looking for an excuse to purchase a new rifle, something between a 6.5X55 and a .270 sounds about right.
Again back to the recoil boogeyman . . . blast tends to be more upsetting to most people than the bump on the shoulder. But the bump on the shoulder is best mitigated through proper rifle fit to the shooter, a quality recoil pad. Be sure the scope is not too close to your eyebrow when you shoot. Although the eye relief might be adjusted optimally when you holding the rifle off hand, shooting prone or shooting at an uphill angle brings the ocular closer to you. My rule of thumb is not to have the ocular behind the cocking piece of a bolt gun, or at the very worst, not behind the rear bow of the trigger guard.