Clean mine each and every time. Seems weird to not clean all the aspects of the brass. Clean mine with a dremel and a brass end brush, takes all of one second.
Aren't you worried about the lead being dusted up by the dremel? One of the reasons I went with SS wet tumbling.
Don't see where the lead would come from as there is no lead in the primer pocket, unless I'm wrong ??? I do clean them out on the deck though