rugged303, sounds like you live south of the 49th and I will assume your varmint are the small furry kind ie PD or marmot. Since you say LR is not needed, then I will assume you mean inside 500yds.
If all the above is correct, the 223 with a 9 twist barrel is my recommendation. You can launch 40gr Vmax for some close to mid range splat. You can launch 75gr Amax or 77gr Nosler/Sierra to reach out past 600yds if the need arises. No issue if the varmint is from the dog family too.
Big bonus is the ease of getting ammo and brass. Plus it is quieter, low recoil and most importantly, long barrel life. I sense that rd count can get up there in most PD fields.
My next choice would be the slowly fading 222Rem. I just love this cartridge. The 17's, 19's, and 20's are all fun and work but limit the size of varmint at distance. The 204 is certainly getting lots of press but at the end of the day (or when the wind blows), it is a short to mid range cartridge only. The 22cal is simply more versatile.
I do alot of LR shooting where bullet performance is more important then seeing a few summers with the same barrel. I use some pretty big cases and they work wonderfully. Way overkill for anything inside 500yds. Thanks for the reference shortround. That 6.5-06 is awesome but a bit much in this instance.
If deer or other large game in on the menu, then sure go big. Just about every cal has varmint weights as well so pick your favorite cartridge and have at it.
A bit more info would be helpful.
Jerry