I generally use a guideline I heard years ago: 3 sets. One set new in the wrap, one set tested and set aside for serious use, either in competition or defense or whatever, and one for training.
So if you use, say, 3 mags to train with, you probably want to keep 9-10 mags for that gun on hand.
If you compete and require 6 mags, say, for competition, I'd keep adding to the collection until I had 18 or 20.
But I start at 6 and work up from there. IMO 6 is the minimum to feel comfortable, for me at least.
Unless you're talking 1911 mags, in which I measure it the way I measure brass...in pounds.