I love 204, and it's been the only varmint gun I've been running this year so far. (groundhogs) I started with a 223 for the job years ago, and it's every bit as good a caliber for that, at the ranges I shoot them. (200-300 yards) Whether you reload or not, .223 will make your life allot simpler as I find it easy to work a load up for AND find good factory-made options. 204 Ruger-I only got back into the caliber because I know I can load for it if I need to, but I also searched what rifles are likely to shoot the 32gr bullet well...thinking about Hornady's Superperformance ammo which is around.
Before working up any loads, I zeroed the scope with that factory ammo and learned it shoots under 3/4 MOA without much effort. When I settled-in, I was shooting groups just +/- 1/2" @ 100 yards. That's groundhog-accurate for sure. Rifle is a Tikka Varmint in 204.
I've owned 2 rifles in the caliber over the years, a Cooper that I got good groups with using handloads (kept for 1 season) and now this Tikka. Hands-down, the the Tikka is easier to shoot well, and was a little over 1/2 the price. (though not a cheap rifle itself) I can't recommend 204 without making a case for getting the same rifle, the Tikka is just excellent. Only upgrade was a trigger spring from YoDave. Being the only rifle I use to reach-out with, I wanted the scope to be pretty decent-mine wears a Sightron SIII.
Last point-I'd like to get another 223 one day and when I do, I hope it'll be a Tikka CTR. Until then, if I had to pick one...the 204 wins. I feel my success with it is tied to how good the rifle is to be honest. Never really loved the look of Tikkas and until this gun, really didn't like stainless barrels/actions much either. This gun has made me a believer.