IMHO it does depend on how long the brass is under tension. If you have loaded rounds that have been fired multiple times even if annealed, you may get neck splits. It all depends on the metallurgy of your brass. Brass available at different times has different percentages of copper and tin, which will make them more or less ductile. There are other components as well that will effect this but those seem to be the main ones.
It isn't unusual to open a box of Western Cartridge Company, or milsurp 30-40 Krag/30-06 and find factory loads with split necks. I read an in depth article on this in "Guns" magazine many years ago explaining this phenomenon. To boil it down, if you find factory cases with split necks it is usually either because of an extended period of tension from being stretched around the bullet and or metallurgy.
This will cause problems for hand loaders. Recently, the brass available for extruding is being made to a different standard. This is done for several reasons, usually financial but sometimes just a lack of components. The brass casing is the most expensive part of the cartridge. Governments are the main procurers of ammunition and even though they will reload training ammunition, anything that makes it to the field on exercises or battle, is new. When it gets fired, it usually stays there. I used to love going to the areas the militia had exercises in the field went. 6.5x55 cases were almost impossible to find and the 7.62x51 blanks worked just fine, after cutting them down and fire forming them. I still have some of those cases with at least 50 reloads on them, IVI brass.
IMHO, it makes perfect sense to re anneal after every 5-6 re loadings, sooner if the cases are hard to form. I mostly use a 40 year old RCBS JR press for reloading serious stuff as it is absolutely true to center if the dies are OK. In fact I use it to check out new die sets for true. This press doesn't have nearly as much force as a Rock Chucker, which I like because I can "feel" what's happening. Work hardened brass makes for inaccurate ammunition.
When I was in Brazil about 35 years ago, I went to a metal manufacturing complex in Belo Horizonte', they made everything you can imagine, mostly steel. They also made brass and other metals. One of the buildings that was off to one side, out of the smoke and dust, was a cartridge refinishing factory. I was amazed at what I saw and learned. Some here would roll and scream foul in the aisles, so I won't go into detail. Suffice to say, the raw material was mostly fired brass casings purchased as scrap from their military and police. After cleaning, polishing, inspecting and sorting, all of the brass was annealed. No attempt was made to check dates or amount of firings etc. It was all dumped into huge bins and taken to another factory for depriming, sizing and loading. It was all very impressive and so fast I could hardly follow what was happening. At the reloading plant, the necks were re annealed again, before being loaded, this wasn't done to soften the necks but to harden them enough to create the desired neck tension on the bullets. Big wake up call there. Civilians don't usually worry about that but in a full automatic firearm, it is very important.
IMHO, if you anneal regularly, mark your cases with an indelible marker or keep each batch separated, keep your loads below maximum and trim when needed, your brass will last almost indefinitely. Taking care of casings is far more involved than just trimming, cleaning, uniforming the primer cups and flash holes. It's a very good start, just not quite enough.