Choosing the caliber is easier than the rifle. The .30-06 is the best choice for a first center-fire rifle - regardless of game.
When it comes to choice of rifle, there is no free ride. Some of the ex-military rifles are horrid - others are pretty good, but unless you know what to look for it is probably better to stick with a sporter. When money is an issue there is a temptation to go the used route, and very often excellent values can be realized, but again you must know what to look for, and even gun savvy people sometimes end up disappointed. My preference is a bolt action, but Remington pumps are surprisingly accurate, dependable, and easy to carry.
The beginners rifle does not need to be match rifle accurate, but then again it should be able to hold 1.5 MOA with good ammo. There is no point in the beginner becoming disillusioned by a rifle that cannot hold 4" at 100 yards. A rifle which fits, feeds flawlessly, has a good trigger, and good sights is far more important than .25 MOA, because the intrinsic accuracy of the rifle does not matter if it cannot be utilized by the shooter.
A word about scopes - the mounts are not the place to skimp, and do not use see through rings. It is also far better to put an expensive scope on a cheap rifle than the reverse. The scope itself does not need to be outrageously expensive, but $250.00 is about the cheapest you should consider. Often a fixed power from a big name maker is better than an equally priced or cheaper variable, and good shooting can be done with a fixed 4X or 6X.
Do not over look obtaining the proper cleaning tools for your rifle. This should include a one piece rod, a jointed steel rod for field use, a bore brush and jag, solvent and light oil, and cotton patches of an appropriate size.
Once the rifle is purchased and the shooter has become familiar with it's operation, initial shooting should be from some sort of bench to sight-in the piece, and to determine it's accuracy potential. After that more attention should be paid to shooting from field positions.
From here on out, the major cost of the rifle will be in ammunition, and depending on the amount of shooting you intend to do, you can decide whether to purchase factory ammo or handload.