The typical hunting rifle is designed to be used by normal people. With respect to moderate cartridges, recoil is mostly a non-issue for anyone fit enough to play contact sports. If you don't fall into that category a .22 centerfire is probably the place to start. For everyone else, cartridges from the 6.5X55 Swede to the .30/06 have much to offer, and although you do not seek an all around rifle, a sporter chambered for a moderate cartridge is exactly that. Choose your rifle carefully, I would lean toward commercial sporters rather than to military rifles or sporterized rifles with military actions.
As to your budget, while there are new budget priced rifle available, you sound like a good candidate to be a Tradex customer. On the Tradex site you can choose from dozens of high quality used rifles, mostly Husqvarnas although others do come up from time to time. Don't choose a rifle simply because it comes equipped with a scope. Bore condition is of primary importance with the cosmetic finish being a secondary consideration. Last year I acquired a Husqvarna 1600 in .30/06 for my 13 year old nephew, and this year he used it to kill a spike mule deer with a single shot, with a 130 gr TSX loaded to 2600 fps.
Once you choose your rifle, have it sent to a gunsmith to resolve any issues, often these will have cracks behind the tang, which is easily fixed. The stock should be cut to length so that you can easily reach the bolt handle when the rifle is at your shoulder; the common measurement is a length equal to the length from the inside of your elbow when bend at 90 degrees to the first pad of your index finger. However, this measurement should be considered with the addition of bulky clothing if you intend to do much cold weather shooting. A good quality recoil pad is important as even light recoil can be objectionable with a hard plastic butt plate. If the rifle does not come with sling swivel studs, they should be installed, and a good quality sling acquired. I like a sling that can be used as a shooting sling as well as a carry strap, but I find that this is a minority opinion these days. Consider though that when shooting slung up from a supported position, the chance of making a first round hit on your target improves by about a third.
I for one wouldn't be in hurry to scope the rifle. Good shooting can be done with irons, and the purchase of a good ghost ring receiver sight and a post front sight is a cheaper option than is a good scope, poor scopes are not worth consideration. The ghost ring and post will teach you much about marksmanship, and will probably surprise you with its precision as your skills improve. The thin rim of the aperture ghosts out of your vision, and once placed on the target, your entire concentration is on the front sight. The flat top of the rectangular post provides a better index of elevation, not unlike the horizontal wire of a scope's cross hair, than does the indistinct round top of a bead sight. The eye quickly finds the center of the post which is the vertical index and with that combination of vertical and horizontal indexes, very precise shooting can be accomplished In the meantime you can save towards the purchase of an appropriate scope and mounts.
As soon as possible begin to handload your ammunition. This will allow you to shoot more for the dollars you have to spend on ammo. Speaking of ammo, choosing a .223, .308, .30/06, 8mm, 6.5X55 etc can allow you to purchase surplus ammo when its available. Check with Districorp from time to time for the best prices and availability, and this stuff will get you by until you can handload.
Much can be learned through dry firing. Work on your natural point of aim so that it can be acquired quickly from fields positions. When you allow your body to go slack, if the sights swing off target, your position needs to be adjusted. Work on your sight picture and trigger control. When you press the trigger, what the front sight covers when the firing pin falls is where the bullet will go.
