About 10 years ago I did a massive project on the gambit of shotgun rounds; from #9 birdshot through BBb, then through all the various sizes of buckshot, and then abut a half a dozen different types of slugs. I chronographed them all at 5 yards and then again at 20 yards(went through a couple chronographed in the process, d'oh!!) and shot them all into calibrated big ballistics gel blocks to collect comparative info on their terminal performance.
My interpretation of my observations would be that for your stated purpose of killing a deer at various ranges, how your gun patterns the buckshot is a much more important variable than which one penetrates better. Put another way....what is the max range your shotgun can put all of those pellets within an 18 to 24 inch circle (height of deer from top of thorax to bottom of thorax)? Once your pattern opens up past this, the probability of passing a pellet through the deer's vital organs goes down significantly, especially if you are talking about the 9 pellets in a standard 2&3/4 shell of 00 buck.
From a penetration perspective, I would not go anything smaller than #4 buck, with #1 buck being my preference (if you can find it!). BBB might be tempting, but IMHO it does not generate the penetration you would need at ranges any further than 5 or 7 yards.
To give you a bit of flavor on where my opinion comes from, I'll post below a sampling of some of my ballistics gel tests from 10 years ago. The penetration you see below is at a range of approximately 5 yards. It is not meaningfully different at 15 or 20 yards, however the pattern is much more open.
As an aside, over the years there has been sustained interest in my old library, which i took down a couple years ago for a couple different reasons. since removing them I've received alot of correspondence asking me to re-post the complete library. I had been thinking abut it, however I've recently decided to re-do them all and address all of the shortcomings of the previous tests. I'm half way through getting organized to re-do the complete project again, and this time will do it in HD video and will also include a lot of 20 gauge rounds.
Number 8 birdshot:
Number 5 birdshot:
Number 4 birdshot:
Number 2 birdshot:
Number 1 birdshot:
Lead BB:
#4 buckshot:
#1 buckshot:
00 buckshot:
Remington reduced recoil foster slug:
Winchester super-x 1600fps foster slug:
Brennake R10:
Cheers,
Brobee