I took some sand paper to the outside of the barrel where it goes into the receiver. It did not take much at all, pretty much just taking it so the blueing was gone, then re-blueing was enough.
The barrel is super tight still, however, no longer has to be tapped out with a small hammer...