In polishing the barrel you make the outside diameter of the barrel smaller, not much, but some.
The barrel bushing holds onto the outside of the barrel, and is the part that aligns the barrel with the slide, which is where your sights are. So in making the outside diameter of the barrel smaller, you are loosening the alignment between barrel and sights, and therefore decreasing accuracy.
How much? That depends on how much material gets removed, the more you remove the bigger the groups get. Reducing the OD two thousandths of an inch, introduces four thousandths of an inch of play. On a 5.25 inch barrel that will open your twenty-five yard groups about three quarters of an inch.
Would it take some real effort to polish that much off a steel barrel? Yep.
Have enthusiastic folks with power tools managed to do so? Also; Yep.