There is absolutely no reason to polish it out. A little old scratch, or a nice new patch of polished metal? Which is more noticeable?
Yes, that's true. The gouge is not in the chamber and so likely won't hurt the brass or function of the arm. In my case the brass (after firing) would be stuck in the chamber and needed to be poked out with the ram-rod after each firing.
I do think however, if the OP does want to reduce the raised portion of the gouge, a dremel tool may not be the best method of removal. IMHO


















































