Ok, so I was at the range today and redid some of the groupings. I took the advice beltfed gave and cleaned the bore before shooting a different kind of ammo. The result was a noticeable reduction in group size for all brands of ammo except for the MFS (which actually got worse...

).
60gr Hornady TAP. This 5 shot group measured 1.7", much more along the lines of what should be expected of the Swiss Arms rifles. I still find that mine is picky on ammo, but most brands are now giving solid 2MOA groups for 10 shots.
Remington 55gr PSP. From cleaning the bore before shooting this ammo, the group shrunk by about an inch. It's still not the best shooting ammo for this gun, but it's doing much better than before.
Remington Premier. I was quite impressed by the improvement in performance in the Remington Premier 55gr accutip. 7 of the 10 shots are under an inch, and I suspect that the 3 that are way outside of the main group have something to do with me. Either way, much better than before. I'll have to pick up another few boxes of this and do some more testing as the accuracy potential with this ammo seems to be high.
Remington UMC 55gr FMJ. I did a really big test for the UMC. In addition to the original 10 round group re-shoot, I decided to try a 20 rnd group. At just around 2.5" for the 20 round group shot at 100 yards, you do get quite consistent performance out of the UMC. Like with the other brands, about 1" was shrunk off of the grouping by cleaning the bore before switching ammo brands.
I found this to be a great learning experience for myself. I learned something that I did not know before (clean the bore when switching ammo makes), and I learned that my hand loads were not quite as consistent as I thought they were, and I may have found a factory ammo that can give my hand loads a challenge for accuracy. I'll keep doing more ammo tests as time goes on, and I'll update this thread if I find anything interesting.