OP, the information you give on your particular rifle isn't specific enough. The general rule of thumb is 5-7 pounds of upward pressure on the barrel created by the tip of the fore arm. If the rifle is a No1 there should be a spring/plunger arrangement about the middle of the fore end to do the job.
If your fore stock is loose you should take it off and check under the king screw (front of trigger guard) to make sure the spacer is still there. The spacer stops the wood in that area from being compressed and creating just the problem you mention. It also helps to keep even pressure on the ways where the receiver fits into the bedding and helps to keep the very narrow space needed between the fore stock and the butt socket flat. This is very important.
One other thing, not all No4 rifles shoot well with upward pressure on the barrel. That pressure is a cheap and easy method to correct minor bedding issues.
I have four No4 rifles that shoot extremely well. One is a No 4 MkI* Longbranch made in 1950 and another is a No4 MkII built by Pakistani Ordnance Factory. Both are in excellent condition inside and out. Both have fore end pressure of around 5 pounds. There is also a been there done that Savage mixmaster that has an excellent 2 groove bore with a 44 date that also shoots very well. The barrel on this rifle does the same thing as your rifle, it rattles around in the fore end. Actually it takes a bit of pressure to make it touch anywhere. The last rifle which is capable of very consistent good accuracy with decent ammo is a ROF N04 made in 43 with a five groove barrel. There isn't any detectable fore end pressure on it either.
It's a toss up as far as the fore end pressure goes. In reality many commercial makers use a fore end pressure pad. It's a trick as old as the first firearms to help gain consistent accuracy. The thing is, it's also a very controversial issue. I can see it on a military rifle, especially one made to have a bayonet attached. Bench rest rifles and accurized sporting rifle and even some military match rifle don't have any fore end pressure or any pressure against any part of the barrel from the receiver to the muzzle.
My first suggestion is to make sure the spacer is in place and torque up the king screw properly. Then go and shoot it. If it shoots acceptably leave it alone. If the bore is worn don't expect really good accuracy. Also, remember these rifles were built to a different standard of acceptable accuracy. Between the rifle and the ammo, 4 inches was acceptable.