Ed Smurf: Barrel fouling is the most common accuracy robber when dealing with muzzle loading rifles. Barrel heat, not so much. You mention that you run a patch after every shot. Is this a dry patch only? If so, that alone won't get rid of stubborn fouling build up. Which is at the root of your accuracy problems.
When sighting in a muzzle loading rifle, I like to duplicate hunting conditions as closely, as possible. In other words: First shot, out of a clean, cold barrel. Every time.
So, my goal is to try to achieve that kind of consistency... throughout the shot string.
Here's a method I've been using for quite a few years:
After firing, wet a patch with isopropyl alcohol (rubbing alcohol),and swab the bore with it. Followed, by one or two(or as needed) clean, dry patches. The alcohol does a great job of cleaning out the fouling. Evaporates quickly, leaving little if any, residue behind. The dry patches take care of any that might remain. The bore ends up clean, shiny and dry. Ready for the next reload.
This method works for me. Possibly for you, too.
Al
ps: Old flannel shirts, etc., make great cleaning patches. Simply cut to the size you need.