Sorry for the necro, but I figured this advice might be of some use.The last take down rifle I owned was a Browning semi SA-22. It was a dream to carry and shoot with a scope, but the stock comb is too high for the iron sights that come with the rifle.
I bought it for pack in hunts, where space was limited.
It was very accurate with standard velocity ammo and fed, ejected flawlessly. However, if it was taken apart, and reassembled, if the torque on the retaining nut wasn't exactly where it liked to be for best accuracy, it wasn't an ideal rifle for taking small game and Grouse for camp meat, because poi kept shifting.
When that retaining nut was right where it needed to be, head shots on Grouse out to 20 yds were doable, but that depended on the shooter.
Getting the torque setting was just a matter of shooting at a target and tightening the nut until the gun shot to POA.
The big problem with it was the torque required on the nut was greater than the strength of my hands could generate.
Takedown guns are meant to be shot with a sighting system that is attached to the barrel in some way. Barrel mounted iron sights will 'follow' the barrel in any directional variation it has after remount (this will still work for just the front sight if you have a receiver mounted rear peep sight). If you're using a scope or red dot it should be mounted to the barrel.
Also from the description you gave it sounded like you were mounting the barrel and adjusting the tensioning nut on the gun after every remount. This is not how it's supposed to be done. The tensioning nut is meant to be adjusted with the barrel removed and once it is set it is not meant to be touched; just unlock the bottom lever and twist. With the barrel removed you'll be able to get it much tighter than you ever would have been able to on the gun. Basically you set the ratchet to the point where it just barely allows you to remount the barrel.