Well, that depends, my friend. If one is shooting only factory ammo, and only up to 165 grains, then you are basically correct.
However, a reloader has a distinct advantage with the 30-06, particularly with the heavier bullets.
From 180 grains up, the '06 will deliver an additional 150 - 200 FPS, which is definitely worth having if chasing Elk, particularly.
Not knocking the 308 here, just pointing out a realistic view of the two chamberings.
A 338 delivers recoil above the tolerance level of some shooters, so the OP needs to consider that. [A T3 Lite in 338 will get your attention!!]
Regards, Dave.