I had a remington 1858 in .44 and it was fun to shoot. About same recoil as a .38 special if you load with 24 gr. Of course, if you put as much gunpowder as you can, it's another story.
I heard that .36 have almost no recoil. Again, the load you will use will make a difference.
If recoil and powder cost are issues, i'd go for a .36.