Is the .321" measurement with a micrometer or caliper? For anything with a tolerance under .005" (+/- .0025") I always use a micrometer. Even the best caliper can have an error range up to +/- .002"
If they indeed measure .321" with a known, calibrated micrometer I would have no issue loading them if the bore is the proper diameter.
Edit:
That is assuming they are cup and core bullets, not solids. Solids is a totally different story.
If you slug the bore you may find it's a bit worn or over spec anyway. I've shot .310" lead core bullets in .308" bores without issue. Just start low and work up.