The only thing you have to worry about (that I can think of) is moisture. If you store your ammunition in a hermetically sealed container that has been purged of moisture prior to sealing, then you do not have to worry about internal condensation from fluctuating temperatures. This can be done using a desiccant such as silica gel packs or calcium chloride.
To answer your questions, yes you can and no it will not harm your ammo when done properly.
Add: Like others have already mentioned, MTM ammo cans are built for this purpose.