Won't hurt a thing in the gun but as Jet has said your seating & crimp dies will need cleaning periodically. Watch for your bullets to start to seat a little deeper than you originally set them at, a clear indication that you have a build-up of residue on the seating plug and probably a hard packed ring glued to the die barrel. I just consider it just an inconvenience that is far overcome by the affordability of cast bullets.
If it is a real problem for you, Bullet Barn hard cast products are usually very well lubed without the residue everywhere but they are somewhat more expensive than the "garden variety" I use.