The purpose of camo isnt to make you invisible, it is to break up your silhouette and make you harder to see. So if you think of it from that perspective, wearing camo will help. If you move while wearing camo, it becomes apparent to them that something is there.
They can see hunter orange but not hunter orange as we see it. Hunter orange over top of camo is a strange thing. Whatever color it presents as to the deer, it is not natural to have a square-ish blob of it in nature. It will stand out. I think that if we were allowed to wear a camo patterned hunter orange (Ontario does not allow for it), it would eliminate that issue.
Maybe if we had tigers in North America, they would adapt to figure out that orange means danger but based on the predators we have.....orange has been given a back seat on the bus and is not a priority.