I love my 25-06 to death, but for long range shooting, I'd choose the caliber that was renowned for long , wind bucking bullet design (which is the 6.5)
My best friend owns a 25-06 (where I got the idea to get one) and I have witnessed him taking a caribou at 700 yards with it .......in the barrens WAY further up north, (browning a-bolt) using a 100 gr moly coated X bullet from barnes, but in my opinion, that is not the norm. He is by far the best rifleman I have ever personally met, and my own comfort zone is 300 or less.
You'll be asking the bullets to do something they aren't quite designed to do, which is go long, buck the wind. I , in the past, owned a 6.5 swede, and although I did like it when punching paper at 300, I ended up trading it, because I don't do serious paper punching, and I had an offer that I couldn't refuse. Plus you can go heavy with 160's and load data exists for 180 gr bullets in 6.5 (although I've never seen any for sale, but maybe they exist from "Boutique" bullet makers. I haven't seen bullets over 120 grains in the 25 cal offerings. So, its purely academic that you "Need the Swede" if you are gonna punch paper at long range.
For what you want, I'd go with a 6.5 swede or even one of the newer 6.5's , like the creedmore, or 260 rem, as the velocity isn't as important as the accuracy for your venture.