Even a mediocre or poor load will occasionally show a small grouping. Having a 4 shot group with 3 holes touching and a flyer means you might be on the right track but sure doesn't prove that load. Can you repeat it? Is the flyer due to the loading, or the shooter? Flyers shouldn't automatically discounted.
Shoot at least 3 x 5 shot groups (or 5 x 3 shot groups or 1 x 15 shot group) and take the average group size from that and you'll have a better idea. DON'T LEAVE THE "FLYERS" OUT unless you are certain you messed up the shot.
Jethunter, you're a pretty knowledgeable fellow, especially when it comes to handloading.
Your advice is good but most modern rifles just don't shoot decent five shot groups. For many, three shot groups are the new norm. Especially with rifles like the Rem Mod 7 with their short pencil thin barrels.
Personally, I like to shoot 5-10 shot strings but never more than two quick shots following each other before allowing the barrel to go back to ambient temps. Yes, this does take much more time. However, by allowing the barrel to cool, it is much more consistent from shot to shot and it closely mimics the in field conditions, where the accuracy of those shots really counts.
Most rifles can be shot until they reach a sweet spot in the harmonics as the heat builds up. The real problem with that type of shooting is that the same accuracy can't be duplicated under field conditions.
Another big mistake, IMHO, is that many folks will shoot off several strings for groups then clean the rifle as soon as they get home. The next morning, they take the rifle out hunting with a clean barrel with no fouling shots down it to ensure the rifle is at its sweet spot for the next several shots.
I've had people laugh at me when I mention this to them, mostly because they purchase one box of ammo per year and it has to last the season, including sight in.
Even competitive shooters don't blast away as fast as they can work the bolt. Most good competitive shooters know where the sweet spot is on their rifle when it comes to temps and amount of rounds needed to get there after cleaning.
Back in the day, we had ten minutes to get off our shots during HBR competition. My particular rifle, when I first started wouldn't put the first two shots where they were supposed to go right after cleaning. Plenty accurate enough for hunting purposes but not good enough if I wanted to win. I always fired the first three shots onto the sighter target. The first two would always be in the 7-8 rings and the third shot would go into the 10 ring, if I did my part. Then, I would put the next 5 shots onto their individual targets for score. Every rifle is different. This rifle had a heavy profile tapered barrel to stay within the confines of HBR rules
Hunting rifles are usually much lighter and fussier. Every once in a blue moon, one will come along and will shoot everything well, from a freshly cleaned barrel to a fouled barrel. Those rifles are real gems and I'm lucky enough to have three such rifles. The thing is, I had to go through literally hundreds of rifles to get them. One I built as a custom 338-06 on a Win Mod 70 action, one is an off the shelf Tikka T3 chambered in 6.5x55 and the other came out of an estate sale and I had my eye on it for close to a decade. It's chambered in 30-06.
Jethunter is right, but, you need to use your head when sighting in your rifle. Shoot it enough to figure out what you're going to use it for and sight in accordingly. Remember, the vast majority don't shoot well from a freshly cleaned barrel. Yes there are a few that will. Most will shoot well, from a COLD barrel, after 3-4 fouling rounds, then you will have 15-20 more consistent shots before you need to clean and foul again.
Contrary to popular belief, if you're using modern components or modern ammo, the bores won't be damaged by not cleaning them right away after shooting. It's also an old wives tale that a freshly cleaned bore will shoot the best groups.