Well fortunatly I think guys who shoot at 500 yards are just plain silly. Even 400 is too far for me. 300 with a good rest maybe. That's why it's called hunting not shooting, get closer.
I don't pay much attention to energy either. Your bullet may have 4000 ft pounds of energy according to a ballistic table (Hornady website) but if you think about it, wouldn't 4000 pounds pick up a deer and knock him clean into the next county? These numbers are too decieving and give shooters false information. The old standby is 1000-1100 FT pounds for a deer at your impact range. Right? Lets go back a 140 years...
The late 1800's were the days of the buffalo hunters with their ultra long range, big bore rifles. Cartridges like the .45-70, .45-90, or any of the Sharpes cartridges are great examples. These cartridges look awesome just by their shear size and nobody these days would even think of saying any of them are not powerful enough to shoot anything we have to hunt. But, look at their energy levels on tables; Our standby, the .45-70 shouldn't be able to kill a deer much past 100 yards, and nobody should even think about hunting an elk or a moose with one as it doesn't have the kinetic energy at any range that our modern cartridges produce out to several hundred yards. Yet the buffalo hunters routinely killed bison at ranges I would never consider shooting. And they did it with rainbow tradgectories to boot.
How is it then that these old cartridges were and are still able to do the job? Well obviously it's because they were invented prior to ballistics charts. Had they come along after, they would habe been deemed obsolite by the 3000+ FPS crowd. But also it goes to show that downrange energy is over rated. Theoretical energy of several tons would, if it were real, have the power to move massive objects. It doesn't happen. Now you may say that a bullet can't dump it's energy in a deer to knock it around because it may go through first. Nope, watch this:
http://youtube.com/watch?v=bvbsWTCovCE
I made this bullet trap out of wet phonebooks. It weighed 50 or 60 pounds I guess. The bullet barely caused the bucket to move, and I recovered the bullet on the last book. For a 3/4 or so ton of energy, it doesn't look too impressive, does it?
What you need is a rifle that you can shoot comfortably and accuratly. It has to be a rifle that you like in order for you to enjoy it. It has to be of a cartridge legal for the game you hunt, accurate enough so your happy in the end and it has to shoot a good bullet. Premium quality bullets are what join the gaps between these fine cartridges. Obviously the higher the weight of the overall bullet contributes to the final on game performance. That's why we use bigger rifles for bigger game. Bigger bullets are built stronger, are likely to penetrate deeper and do as much damage as possible. But then we have the wild cards there as well. A 6.5 bullet is very long and very lean. They are famous for their penetration. In fact, I would say it's possible for that little bullet to out penetrate alot of bullets from the bigger and heavier bullets. That's why they have such a strong following here and abroad. Load a good bullet in a 6.5 somthing and heck, lets go to Jurassic Park!