Do you have the Cartridges of the World Book? This is a great place to start. As a young fellow in the mid 1970's my dad got a copy of this book which started me on a life long passion of collecting, which encompassed not only cartridges but firearms at a later age. Early on I collected everything, which became a rather confusing and daunting endeavor. I started to focus on specific calibers and head stamp variations.
After nearly 20 years I sold most of the cartridge collection in one lot of 800 pieces, then strictly focused on military related ammo. I have not kept count though now it covers Civil War Musket cartridges through to Vietnam era ammo, with a huge number of 45-70 Frankford Arsenal items.
Some of the interesting modern rounds to find are the Newton variations (the 257 is very scarce), the Weatherby's ( the 224, 257 and 378 are tough one to find)., plus any of the British rounds. Even rim-fire cartridges are an interesting area to specialize in. Shotshells are virtually limitless in their variations, with, not only manufacturer variants, but load differences as well.
The best place to find stuff...gun shows!!! There are usually dealers set-up with an awesome selection of stuff. Don't be afraid to occasionally buy a high end item...these are good solid investments in the long run.
Cheers
