If just shooting groups, you want the distance of the two farthest apart holes, minus the diameter of one hole (or the average diameter of the "sighters"), to give a center-to-center measurement.
It is somewhat common practice to overlook a flyer, if you caused it, not the rifle of the ammo. This only applies when you are testing the accuracy of the gun and/or ammo.
If shooting targets as a measure of skill, you measurement includes all shots taken.
There is a program out there somewhere that measures and calculates your shot targets and give all sorts of information.
There are many varieties of targets and challenges that are scored differently.