M305s and other M1A-type rifles are very sensitive to ammo. High quality match ammo, like the Barnes bullets that you bought, might not be what shoots well in your rifle. Try Hornady 168 match ammo and a variety of other brands. The best solution is to work up a load for the rifle but that would mean a jump into reloading.
The 3-round group at 50 yards is likely not indicative of how the rifle is performing, I'd recommend 5 round groups minimum (but I recognize that commercial match ammo is expensive). You may have "fluked" a nice 3-round group at 50 yards while the 100 yard group is more indicative of how the rifle performs with that ammo. Your initial tests seem to suggest this.
One thing you might want to keep in mind is that many M305 rifles seem to be sensitive to barrel temperature. The first 3 or so rounds from a cold/cool barrel will string vertically then the group will stabilize. I find that a warm-up is needed even after a target change. Going from the 50 yard to 100 yard ranges may have allowed your barrel to cool enough to throw the first few shots. My rifle always shoots 2 MOA high and 1 MOA left from a cold bore. The shots then drift down and then the groups stabilize to under 1.5 MOA. This is with my hand loads.
Accurizing an M305, even in a Blackfeather, is needed to get good performance but it will never shoot with even a sub-standard bolt action rifle. What optics are you using?
See the thread below to calibrate your expectations of the platform.
http://www.canadiangunnutz.com/forum/showthread.php/1049913-M1A-Precision-Challenge-1-5-MOA