OP, your rifle was originally built to consistently shoot "issue" ammunition into a FOUR inch group.
Your rifle may not be capable of shooting tighter groups than you would like.
Have you checked the bedding? Is the King Screw spacer still in place? Is your bore on the large side of acceptable?
I've seen these rifles change from shooting shotgun like patterns to shooting sub 2 moa and even sub 1 moa groups when going from .311 to .312 diameter bullets.
Trying to compare an issue No4 sporter with a commercial off the shelf sporter, especially of recent manufacture is like comparing apples to oranges.
Some No4 Rifles will shoot extremely well, most will shoot well, some barely qualified for acceptance. It's luck of the draw what you get.