I've got a Thompson Center .36 caliber percussion "Seneca", and a couple of friends have .32's. They are actually a lot of fun to shoot - and very economical too - but as you say are pretty much limited to plinking and small varmints.
Mind you, that little pill is moving at pretty high velocity if you use a decent powder charge - I can say from personal experience that it will quite literally turn a gopher-sized animal inside out! (That's our prairie version of squirrels y'know - not enough trees, so they live in the ground.

) Without looking it up, I seem to recall that the .350 round ball for my .36 weighs about 50% more than a .22 Long Rifle bullet, and can be made to travel at significantly higher velocity.
You may have read of the practice of "barking squirrels", back in the days when they were regulalry hunted for food - even a small caliber like .32 produces too much meat damage with a body shot ... and you gotta be mighty good to consistently hit squirrels in the head, as you can imagine. So hunters would fire into the top of the tree branch just below the squirrel (or the trunk bark next to him if he was vertical on the trunk.) Done correctly, the resulting "explosion" of bark and wood chunks would usually knock the animal to the ground, and stun it if it didn't kill it. Another squirrel for the bag, without meat damage!
Accuracy depends on the gun, and load, of course - but they can be deadly. One of my friends with a .32 (a custom flinter) swears that it even does better in a strong crosswind than his larger caliber round ball rifles - attributes that to the higher velocity, and the smaller surface area of the projectile on which the sidewind can act ... which certainly
sounds plausible.
Cleaning is no more difficult than with a larger caliber. Not sure about the .32, but both Hornady and Speer offer swaged .350 round ball for .36 caliber rifles. Other supplies are much the same as any other muzzle loader. You might have to use 3F powder for consistent ignition in a .32, but I've never experienced any noticeable problems using 2F in my .36 ......