Gun barrels are like my arse: no matter hard you wipe and scrub, they will never come clean! While the liberals and wanks shamelessly lobby to ban guns, nobody gives a thought to banning white underwear for men!
*Ahem*
Don't worry about getting your patches out lily white. You are primarily concerned about rust and fouling associated with lead. Some of the higher intensity handguns will be subject to copper fouling as well with jacketed bullets. Try to keep your cast lead reloads under 1000 FPS and leading shouldn't be a problem. If you are going faster than that at least use gas checks - I personally say to hell with them and go with FMJ's myself.
I clean my pistols once a month. They get soaked in solvent, wiped dry and lubed. Barrels get about 20 passes with the wire brush and then wiped down. I still get grey patches after cleaning but I don't care, my guns have never rusted and they shoot as well as any on the firing line. The only time to go to town on cleaning is if you plan on mothballing a gun for a long time in an aggressive environment. Out here in Alberta I have stored guns for decades with just a smidge of oil. Would that work for somebody in a humid climate like BC? I dunno, if I lived there I might put a little more on.
Routine cleaning shouldn't take you more than 10-15 minutes and shouldn't be a pain. Don't sweat grey patches.