those sound like fairly hot loads for 50 cal mini or maxi balls. Not unsafe but rather they could be distorting the skirt if you are shooting mini balls. As mentioned above, with black powder at least, you probably have to go to 5 gr increments to make any significant difference in accuracy or velocity.
Two things come to my mind; try dropping down to 50 grains are start working up from there, preferrably at 50 yards at a square black bullseye so that you have a sharp definition of where the bullseye is. Try shooting a 6 O clock hold and don't worry where they hit as long as they are on the paper. The point being to get a sharp sight picture and see what groups you get with each load.
Second suggestion would be to use a square post forward and a square notch with a flat top at the back and line the top of the front sight with the top of the rear sight; again to get a sharp repeatable sight picture.
I am also assuming that you do not have a round ball barrel with a slow twist; try shooting patched roundball and see how it groups; around 50 grains of 3F powder
On the mini balls the lube should be fairly soft like baby poo; make some up using 50% lard and 50% bees wax and see if that helps. Finally if you are casting your own slugs, the alloy is important and would suggest something close to pure lead.
cheers mooncoon