Eriky, have you slugged your bore??
Most of the Swede 6.5x55 rifles have very close to mean bore measurements. I have a take off barrel that looks excellent but measures out at .269 thou groove diameter.
I had a similar issue with that rifle, which was being put together, from parts for a 12 year old, whose father had a similar rifle that shot very well, no matter what it was fed. He wanted a rifle for his daughter that he was already set up for when reloading.
That barrel wouldn't shoot better than 3 moa, unless I used the old cupro/nickel jacketed 160 grain FMJ, with exposed lead core, flat bases which took up the extra diameter in the bore, when the pressure forced the lead cores into the jackets and into the grooves.
After replacing that barrel, with a new in white M94 bbl that came out of tradex, a banner supporter of CGN, that little rifle shot everything well, from factory to handloads of different weights.
Those old Swedes are lovely rifles and make dandy sporters, especially when you can find one at a decent price that hasn't been bubbaed to badly to rescue.
NOT ALL OF THEM SHOOT WELL.
Most of them shoot better than the people operating them are capable of holding.
The pristine rifles are usually excellent. The later rifles that came into Canada were well used after their last FTR and may not shoot as well, for all sorts of reasons, especially worn throat leades.
Cutting down the original stock can have some negative effects on the accuracy of these rifles as well. You might want to try a ship under the barrel, at the tip of the fore end, putting some upward pressure on the barrel. That sometimes works very well.
Use something like folded paper to check it out and if it works, then either install a screw to apply pressure or set up a fiberglass pad.
Another thing, is your action glass bedded into the stock?? A lot of those old stocks, if it's a cut down original, had sloppy fits.