The Swede is not at all fussy. Most of them will group MOA or better with minimal load work up. Partial full length sizing (2/3 down neck), and seating bullets out, will likely shrink your groups.
My go to recipies:
-120 grain. Nos. Ballistic tip or Sierra hpbt match, H4350, CCI br2, start @44.5gr, max. up to 47.5gr in SOME rifles.
-140 grain. Horn. Amax or Sierra hpbt match or Lapua scenar, Reloader 22, CCI br2(Fed 215M in the winter), start @45gr, max. up to 48 in SOME rifles.
All my Swedes shoot tighter as you warm up the loads. The M96 is not weak, but prudence IS required, when primers just start to flatten in fireformed cases I call a halt to charge increases, drop back a few tenths, and play with seating depth for best groups. 2650 fps/140gr. is where most of them seem to want to be.
You may want to check your bore for cupro nickle jacket buildup just in front of the chamber. Had 1 that was so badly constricted that start loads would flatten primers, hard bolt lift too, FWIW.
Concentricity of the loaded round is important to lose those occasional 'flyers' and achieve consistent rounded clusters.
We expect to hit a cigarette pack at 250 with the handloaded, as-issued M96's .... the target scoped/tuned/bipod sporters?, well, they ring steel just fine out to 1000, so far.