Do yourself a huge favor and totally stay away from the that whole area. It is inundated with every A$$hole from Toronto that owns a gun or any other piece of motorized equipment that he or she is attempting to kill themselves on. Hence why you heard the OPP and the CO's make frequent patrols there, it's because they do a wonderful business in that area.
I hunted up the Swift Road once or twice a fall back in the 70's. You would see absolutely no one. My wife and I took a little nostalgic drive up the Swift Road about 5 years ago. It was hard to believe what we seen. Basically it has become a haven for drunken red necks from the city. People camped all over. ATV's, UTV's, Dirt Bikes, Rock climbing jeeps, folks with guns doing a little target practice. Drinking is a huge part of the activities. If this is what your looking for then this is where you want to be. If not stay well clear.
I lived in the southern Georgian Bay area for 41 years so I know of what I speak.