It was a tough call for me which caliber to go with for my dangerous game rifle, the .375 H&H or .404 Jeffery (glad you spell 'Jeffery' correctly Wood! Few do.). I ended up with a .375, and have put it to good use in its intended roll, as I wanted an all 'round rifle, with lots of plains gaming, moose and deer hunting, etc where ranges stretch to where the slight improvement in trajectory matters.
Frankly though, I've yet to shoot an animal with my .375 at a range I couldn't have made with the .404 with ease. I also like that clean, soft shouldered beltless case. Bullet availability is a slight concern, I have a pair of barrels for a double rifle build in .416" with a reamer for .450-400 3", the .404's rimmed ballistic twin, but on a .416" bore, rather than .411". If .404 Jeffery were to use the .411" bore you'd have the same complimentary situation, where you could custom build yours on the much more common .416" bore for bullet availability, and in a pinch could still shoot standard .411" ammo with no risk. Unfortunately for the .404, you're stuck with its oddbawl bullet spec. Good news is one can cast for it, as the velocities are moderate. Recoils less than .416 Rigby, little more than .375 H&H, so in other words you should find it completely manageable.
Overall, a custom Model 70 with a drop box mag to hold 3-4 'Jeffs', forged to straight bolt handle, in .404 is something that jumps into my dreams at night. Let us know what you choose if you go for one. The CZ550 would be the most obvious, and easiest pick as it's factory offered. The Chadwick Ram of North American hunting lore was taken with a .404 as well for an interesting aside.