Gentlemen:
While I do not have any experience with in-lines, however I have spent a good bit of time with traditional cap-lock muzzle loaders, and the application of ideas should be similar, so I will offer these thoughts.
First off, we need to figure out what your rifling twist is, and that to some extent will give you a general rule of thumb as to what type of projectile one should us and in what weight range.
The general rule of thumb that I use in 50 cal is
1 in 60 for round balls
1 in 48 combo twist - can shoot balls or short conicals
1 in 30 for mid range length conicals (as dictated buy caliber)
1 in 20 for long heavy for caliber conicals
Then we must examine what propellant you are using. Personally while pellets are convenient, I find them overly expensive, and understand the can suffer from inconsistent ignition. I prefer to use powder, and myself have had good success with Pyrodex.
Next, one should examine the smoothness of your barrel, and the sharpness of your rifling. This is really done by feel and a DAMP tight fitting patch. You should be able to feel the rough spots. Further, look at your damp patch when you take it out, and have a look if the rifling is cutting your patches.
If you suffer from the above, then you may need to smooth it some with a mild abrasive paste and tight fitting patches. Make sure your rifle is clean to begin with, and do this buy hand. Top cap and ball shooters do this to their revolvers so there should be no issues doing this with a rifle. I have the procedure some where and if interested I can look it up for you.
If you are shooting sabots, depending on the smoothness of your barrel and how sharp your rifling is, you could be suffering from plastic fouling, just like shotgun shooters. The only way to get this out is with a good stiff bronze brush soaked in a solvent like shooters choice (this one is designed to cut plastic fouling), followed buy patches.
Depending on bore condition, it may not take many shots to foul the rifle.
The above is also true for lead fouling, if you are shooting a steady diet of conical bullets.
Further, another accuracy ailment is shooting conicals that are too hard in alloy. You want to make sure that you are shooting the softest lead possible so that the bullet will obdurate properly and “seal” the rifling bullet fit. If the alloy is to hard, you will get blow by and accuracy will be erratic at best.
The other thing to do, is keep detailed notes on what an how you experiment, so that you can see what you have tried and where there might be some promise.
The other rule to remember, is that there are no hard and fast rules. For example, my main cap-lock rifle has a 1 in 60 twist, so it’s a dedicated round ball shooter. However I have had great success using sabots and get about 2" 3 shot groups at 50 yards. This flies in the face of the “rules of thumb” for rifling twist sated above, however, you should always try something for real in your rifle before you dismiss it.
Well I have rambled on long enough. If you have any further questions please feel free to ask and I will do my best to answer them.
Happy new years
Regards and all the best
R