You need to have an honest idea of what you are trying to achieve and the expectations in terms of results.
For recreational shooting, there is no need to spend huge sums of money on customizing. If you use factory ammo, don't bother customizing either. I'm assuming you are thinking in terms of 308 caliber too.
Understand that the differences in price of a Remington 700 relate not to function, but to appearance. A 700 action is a 700 action - period. A 700 barrel is a 700 barrel. The only two notable exceptions are the 5R, which is just formed by beating the barrel over a different shaped mandrel, or the Sendero, which uses a different process, but they shoot no better and no worse.
Some Remingtons shoot very well, in fact if you believe some of the stories on-line here, guys get quarter minute groups from them. I'm skeptical, but hey, nobody EVER bull$hits on the internet...
The 700 action - the guts of that gun - is the safest factory action out there. It is the design that is copied by virtually all custom action makers in one form or another. (You never hear about a "Savage Clone"). They are well made and easily customizable. Furthermore, there is no doubt that the biggest selection of aftermarket accessories exists to fit a Remington 700.
So... depending on what type of shooting you plan on doing, and depending on whether you reload or not, I suggest that you find a 700 you can live with. If you want to spend big money on a factory gun, at least find one with a stock you like, since the rest of the guts will be essentially the same.
You can customize as funds, skill and circumstances dictate.
A top-end true match barrel will run 400-450. Installation will be about another $250. A match trigger will run about $300. Accurization of the action to make it true, and a real long-distance contender will run another 2-300.
I am going to tease you and a few others reading this with a few tantalizing details... there is a couple of gunsmith/machinists in BC that are currently ordering the materials to begin producing a true custom action that will incorporate design features of an R700, Barnard and Sportco action and will sell for under a grand. It will be 100% Canadian too! I will have my hands on one of the first as I will be "campaigning" it in competition.
So... it is entirely possible to buy a factory rifle and have great results. You can add custom features as you go along. The biggest contributor to accuracy is the barrel; there is no reason why you cannot add a custom barrel to your gun at a later date.
I can assure you that any perceptions you have regarding Remington QC are related to the number of rifles they sell. They sell a huge number of guns, and their percentage of "monday" guns is no better or worse that any other brand.
