I don't think you're wasting your time at all, depending of course on what your definition of "cleaning" entails.
I'm probably the exception in this respect, but I don't tumble or ultrasonically clean my rifle brass. It gets a light polish on the outside with 000 steel wool, then I brush the inside of the neck with a nylon brush before lubing both the outside of the case and the inside of the neck, then resizing.
I've had good accuracy with this method, I also don't shoot enough volume to feel justified in buying a tumbler. Maybe I'm missing something?
What did people do to clean their brass before tumblers and ultrasonic cleaners were available?