There are 3 problems to overcome. The first is learning to shoot well, the second is learning to tolerate the recoil and blast of a centerfire rifle that is suitable for big game hunting, and the third is basic gun handling so you can load, unload, assume firing positions, and carrying the rifle in a comfortable and natural manner.
Cost always plays a part in our decision making process, and the purchase of the first rifle is no different, except that a poor choice can cost us more money down the road. In order to get good at something, you must do it often. "Shooting often" means that you are spending money on ammunition. Handloading can certainly soften the financial burden of shooting, but not everyone wants to take that on. Should you be unwilling or unable to handload, you should consider purchasing a rifle chambered for a round that you can purchase inexpensive surplus ammunition for. The rifle and ammunition should produce a level of accuracy that boosts confidence, and some rifles and ammo are so poor as to leave one with the feeling that becoming a good shot is simply beyond their ability. Very often chopped up Lee Enfield .303 rifles fall into this category, and although there are good ones, they tend to difficult to find, and once found they tend to be as expensive as a good off the rack sporter.
Check the EE for a suitable rifle, but if nothing jumps out at you, Tradex has a large selection of Husqvarna rifles, both commercial sporters and ex-military rifles. These rifles come in a variety of calibers for which inexpensive surplus ammunition is available. There are also a few other manufacturers such as Sako, Brno , and Saiga, but the vast majority is Husqvarna. I would look for a 6.5X55, a .308 (7.62X51), 7.62X54R, 8X57, and maybe even a .30/06. Surplus .30/06 ammo is scarce enough now that it should probably be considered a handloading proposition.
Go to Districorp's site and make sure that there is inexpensive ammo available for the rifle you choose. If you can afford it, buy a case of 1000-1200 rounds. You'll use it up faster than you think. When you have to buy soft point ammo for hunting, it won't break the bank. You'll need a box for checking your zero and a box to hunt with. Districorp sometimes has deals on soft point ammo as well, and the Hungarian stuff seems to be pretty good. When you shoot, save your brass if it's boxer primed. You never know, someday you might want to start reloading.
A used commercial sporter, with a Mauser action, in good condition for less than $500.00 is IMHO superior to a cheaply made new sporting rifle. Some will disagree. The Husqvarna sporters are drilled and tapped for scope mounting, and the bolt handles will clear a low mounted scope, something that is seldom true of a military rifle. If possible, buy a rifle that also comes with metallic sights. Do not buy cheap flimsy scope mounts. Quality mounts aren't cheap, but they are important. If you can't afford a good quality scope, shoot the rifle with the irons until you can afford one. Check the EE for good quality used scopes, I've bought a couple of Leupold's from sellers at 30-50% off the retail price, and those scopes have looked new when I got them. Leupold is a good choice due to the fact that there is a Canadian warranty facility, and these guys (Korth Agencies) give good service and produce good work. Fixed power scopes tend to be cheaper than variables, and there is an argument to be made that they stand up to hard use better. Don't go over 6X if you choose a fixed power scope for a big game rifle, and if you choose a variable don't go much over 3X or 3.5X at the bottom end. In most big game situations lower magnifications are more useful.
