Factory ammo is not loaded with powder available to reloaders at retail. They purchase massive lots of bulk powders with a specific burn characteristic. Factory ammo is loaded to give "good enough" results in as many different firearms that shoot that calibre as possible. If you find a specific factory load that works well in your rifle it can be duplicated or improved upon with handloading. I've never had that hard of a time finding really accurate loads for my rifles. At most I've had to try three different powders before finding a very accurate load.
I sight in with factory ammo and then test handloads from there. They may have a different point of impact but I've always had them stay on the same paper at 50yds (and almost always at 100yds) except when playing with reduced loads, cast bullets, or start to get over pressure and then the bullets scatter a lot sometimes.
Handloads don't really get "erratic" except in some situations (undersized bullets, reduced loads, really bad bores, etc.). They just have a different point of impact and different accuracy. Once you develop a load that punches the smallest groups you can sight in your rifle for that load.
I'd say on average I will shoot 4-6 five shot groups to find the most accurate among those and then go up and down from the most accurate of those by half the increment I used before. So if I test 34, 35, 36, and 37 grains of X powder and 35 grains proves the most accurate I will then test 34.5, 35.0 (again), and 35.5 grains and use the most accurate among those. Using this method I can usually get a scoped rifle shooting at least 1 MOA if not less. I could probably tweak it past that and get even tighter groups but I haven't had a need for that (yet). If I can get a rifle shooting 1 MOA I wont complain.