I own and occasionally shoot .34, .38, and .40 plus larger ones. The .34 has deep rifling and I use fairly damp patches and a brass ramrod so that loading is not a problem. On the other muzzle loading lists I am on, however, people regularly complain about needing to clean between shots with their .32's and .36's probably because they are using spit patches and have shallower rifling.
For flexibility of cleaning I prefer the larger calibers. I clean with a bronze brush and hot water and worry about having a brush stuck in the bore or bending/breaking a cleaning rod. On the smaller calibers, using a muzzle protector on the rod is often not practical.
When shooting gongs and trails, it is a bit important to have relatively light targets for the small calibers, because the small balls do not move the gongs nearly as much.
In terms of ease of buying bulk balls, the .45 is the smallest locally for which I can buy them by a box of 500.
cheers mooncoon