http://laws.justice.gc.ca/eng/regulations/SOR-98-462/page-1.html
They refer to modifications as "altering" or "re-manufacturing" in the regulations.
Upon reading this, it appears that modifications are allowed, as in the case of modifying for the purpose of limiting the capacity to 5 or 10 rounds of the ammunition for which the magazine was designed, and the type of firearm for which it was designed (5 rounds for a rifle, 10 rounds for a pistol).
Where one might run into legal trouble is if the magazine was a 10 round pistol magazine and it was modified or "re-manufactured" specifically so that it could function in a semi-automatic rifle. One could argue that it was modified and intended to be used in a pistol, but just happens to also work in the rifle. In court, if it could be demonstrated to the judge or jury that the magazine would not normally fit the rifle, your modifications allowed it to fit, and then in fact you did put it in a rifle, I think you'd have a hard time convincing them that it was not your intent to use the magazine in a rifle when the modifications were made.
In other words, if your REAL reason for modifying the magazine was for the purpose of allowing it to fit in a semi-automatic rifle, then you have a prohibited device on your hands, unless you pin it to 5 rounds. Essentially you would be re-manufacturing a magazine, and it has to be limited to the appropriate capacity for the type of firearm that you intended to use it in.
So, having said all that, I think if you have a 10 round pistol magazine in a rifle, it'll raise eyebrows with those who don't know that it is okay, but, as long as you haven't modified the magazine, you're in the clear. It cannot be argued that you re-manufactured the magazine with the intent to use it in a rifle. On the other hand, if you did modify it, the matter could end up in court, and even if you win, which is not guaranteed, you still lose because of wasting your time and legal costs.
One thing I was wrong about, I think, is modifying a 5 round magazine. Looks like that is totally fine, as long as it holds no more than 5 rounds and cannot be easily adapted to hold more, etc, etc.... It's only 10 rounders that can get you into trouble, if you modify them and then put them in a rifle. I wouldn't modify a pistol magazine if the real intent is to use it in a rifle. Basically, what the other guy said earlier was exactly correct.
Why couldn't they just make the limit 10 rounds for semi-auto rifles and pistols? Would be so much simpler for everyone!