The downsides are that the barrel will effectively be several inches longer due to the brake, making the rifle both heavier and more akward. Also, depending on the bullet and load, the point of impact may change when you turn the muzzle brake off. The fluted barrel will weigh more. And don't even think of trying to shoot the rifle with muzzle brake on WITHOUT hearing protection while hunting. I learned the hard way that this is extremely dumb.
In my opinion this would make a poor hunting rifle. The savages are great rifles, but you'd be better off with the standard barrelled and non-muzzle breaked version for hunting. 30-06 has very manageable recoil, so it shouldn't be an issue. If it is, you can always buy the reduced-recoil loads that remington makes.
