This is a difficult subject to address.
First challenge is how to measure "accuracy". A well tuned #4 will string rounds vertically, because of compensation. The slower rounds will print higher. The string will come together around 900-1000 yards. Until then, it looks like it is a poor shooter.
You need the barrel to be bearing down on the forend. You should notice it take some thumb pressure to push the barrel up, off the wood.
The actions has to bear evenly on the two draws, to take the recoil. The torque on the main action screw can be adjusted slightly to give some control of the compensation distance.
In the 83 I built a pair of #4s for target rifle competition. The first one shot so well that I feared that it would be shot out by the time I got to Bisley. The second one used a Sportco barrel instead of a Shultz Larsen (cheaper) and I practised exclusively with that one.
By the time I got to Bisley, I was afraid to switch back to the "main" rifle, so I used the practice rifle. I did well with it, dropping only 2 points at 900 and 1000 yards that week.
I still have the main one. Might have 500 rounds through it. have not used it since 1984.