223 would be quieter than the 22-250, which is quieter than the 243 (243 being the loudest). Assuming they have similar barrel lengths.
There is also an increase in recoil, not that it is substantial, but something to consider.
Also the range at which you want to shoot varmints is also something to consider, and the conditions (super windy?).
204 ruger might be worth a look, I don't own one, but they shoot flat, and are quieter than a 223 (allegedly).
Or get one of each, try them out, and see which ones the farmers grumble about.
Finally a question in this forum I might be able to help with.

First-off, it's non-farmer neighbors likely to be the issue...if the noise even proves to be one. I've owned varmint guns in 17 Hornet, .204R, .223 and .243...and have shot groundhogs with .270 and 22-250 as well. (belonging to buddies) Now this is anecdotal, but my answer would mostly echo what's been said before. OF the 3, .223 seems to have the lowest report, 22-250 a little more, .243 a big jump up from either of those two others. Since someone commented on .204, my opinion on that is that it's very close to .223, but the sound seems "choked" a bit...so the perceived level of sound just "seems" a little less than .223. If that makes any sense. Now, from behind the trigger...the experience sort of parallels the noise thing. 223 has the least recoil, 22-250 maybe a tiny bit more, 243 has enough recoil to cause you to lose your sight picture through the scope. Meaning, you won't see the impact. Zero pain of course, but enough jump to move the gun.

I like my hunting buddies too much to ever add a muzzle brake.
.204R is really a hand-loaders caliber, as I've found them the hardest to get good results with, using factory ammo. Granted, that's based on experiences with 2 rifles..only one of which I owned. For varmints up to coyote size, it might be the best tool for the job in my personal opinion...simply because of how easy it is to shoot well. (very flat trajectory) I don't currently own a .204, but it's the rifle next on the list for me. Goal is to have it up and running for spring of 2020.

Most of my varmint hunting is centered around groundhogs, and even though farmers have no use for these marmots...I still believe it's important to deliver a quick demise to any living thing you put in your crosshairs. For that reason, I might also suggest that you focus on what ammo shoots best in the gun you choose, and try to use varmint bullets whenever possible. I only load with Hornady V-Max, which seem to almost always dump their energy inside their target. (vs blast through, and keep going)
I'll wrap my comments up buy recounting an incident last spring. I'd bought a CZ 527 Varmint in 17 Hornet, mostly because I was super curious about the caliber...but also thought it might be a good fit for groundhogs where I like to hunt them. As an added bonus, I figured the lower report would be appreciated by non-farmer neighbors. Anyhow, wasn't long before I had a complaint from one of these neighbors~using my 17 Hornet! I'd hunted that same farm with a .243! lol Not even in the same league from a power/report POV. (The farmer told me that neighbor squawks about everything, so ignore...and keep shooting

) Suffice to say, I never did pick calibers based on noise level...and the one time I thought it would be appreciated was the one and only time I ever got a complaint.
So, pick what works best...and have at 'er! If there is anything else I can help with, send me a PM.
Happy shooting!