Yeah. Start waving that one around. Next thing you know, if you cleanly miss your game you will be charged with careless discharge.
Look, just because someone else shows up on public land and shoots at a decoy does not mean they did anything legally wrong. Proving they disturbed a legal hunt would be next to impossible unless you could prove they knew you were there. Just as proving any careless firearms handling.
Do I condone the actions of these guys? No. All I'm saying is if you hunt on public land, you better darn well expect at least some of the public to show up. If they actually did something illegal, then by all means report them. Whining and making up reasons why they should not be there solves nothing. Maybe next time walk up to them and see if they want to talk about it.
So yes, harden the f**k up.