44. The Colt improves somewhat with hand loads, but then so does the 44 mag. In a strong single shot action (think T/C Encore not a SAA style revolver) the Colt can be pushed to very similar velocities to what the 44 makes in a six gun. But throw that 44 in the same single shot action and it leaps ahead again by a couple hundred FPS. You might be able to boost performance in say a Freedom Arms single action through careful experimentation, but at that point you may as well just switch to 454 Casull and leave the 44 Rem mag in the dust.
Long story short, between the two, get the 44 magnum if you're looking for maximum power.