What if there were a bunch of tree stands on the few areas that were good on the property? Are you just going to pack up and leave because someone put out a few stands in October and has hunted out of them once or twice for a couple weekends? Its nice of you to move on, but if they aren't utilizing a prime area there's no reason for you not to either.
Confronting people on public land vs private land are two different things. Regardless how they might retaliate, if they're on you're property and you don't want them there, get them off. Follow them to their vehicle and write down their licence plate and let them see that you're doing that. This way if they get pissed off and want to rob you they might think twice as they know that you know who they are. With any situation, if you approach people in a professional manor and be polite they're usually not going to do anything, if you yell at them and tell them to get in their car before you call the cops, they're going to leave with a bit of a grudge.
The local conservation authorities do have many small properties you can hunt on yes. For the population as large as Southern Ontario, there isn't much public land though no. And for the very large hunting population of Southern Ontario that doesn't have land to call their own, some of the places are pretty chaotic during different hunting seasons. The LPRCA property near mine had 6 cars in front of it last year an hour before sunrise on opening morning for the controlled hunt for example. Also a lot of the properties have a lot of joggers, dog walkers, cyclists, horseback riders, ATVers, ect going through them during the day, so with small parcels of land it easily pushes off anything in them. So yes you're right that there is public land, just most of the blocks from my experience seem like they're either pounded or have a very high use for other recreational values. In my eyes that's not ideal for quality deer hunting.