They happen, usually shooter error, but they can also be an issue with a load, or even a barrel that is past it's accuracy life. I have noticed more fliers with rimfire loads, and with centerfire barrels that started throwing fliers, after they had a lot of rounds down the barrel.
When it comes to Varmint rifles, their barrels usually have twist rates tailored to a specific bullet diameter/length/profile.
I can give you an example of such a rifle, that I loaded with hand swaged 130 grain, flat base ELD ogive.
The barrel was made by Hart and had a 1-15 twist rate. The Chamber was cut for the 308 Palma Match with a special below SAAMI spec outside neck diameter.
The barrel was a reject from a batch I picked up from Nobby Uno. Not sure why it was rejected, it was never chambered or threaded to mount and after it was profiled for easier carrry in the field, then mounted onto a Remington 700LA receiver and mounted in a McMillan stock, it shot better than I could hold in the field.
The bullets came from the US by a Bench Rest specialist, Randy Robinette. They weren't commonly available because they were specifically intended for hunting Varmints. You would be surprised how many Match shooters don't hunt and don't believe in it.
Anyway, this rifle was a proven shooter and set up specifically for the above mentioned bullets.
I always thought that the very few and far between misses or flyers were my fault. Well, that would have been partially right.
Randy admitted that there were no guarantees about flyers in any of his bullets but the chances of it were extremely small.
The very odd flyers I got with that rifle, happened because the bullets were just barely stabilized, which for accuracy purposes is usually an ideal condition. When the temps dropped to far, the powders I was using at the time, BLC2 and W748 were very temperature intolerant. I just wasn't getting the pressures and resulting velocities to properly stabilize those bullets at -25C temps and lower.