Shatter when dropped from the mold, or when they are cold?
Lead is hot short. Means it does not handle being beaten on or otherwise impacted when it is really hot. If the ingots are breaking up when you dump them from the mold, let them cool longer.
You process these with a gas fired burner or electric?
Less control over the heat on gas fired. Good for fast production, but not so much for allowing the zinc weights to float to the surface where they can be skimmed off. Object lesson in why one should not just load up the pot and go for coffee.
Try casting with the alloy. See what happens. Do some reading on CastBoolits before you give up on the stuff completely. If it casts decent bullets, then, yer in!
But, if all else fails, Cannonball sinkers for downriggers are retailing for $70 for 10 pounds at crappy tire. You should be able to beat that, eh?
Never give up! Chances are that you can find a use for the stuff if you have contaminated it.