OP, the lighter bullets shoot better because they are spinning fast enough to stabilize, which is what the other replies are alluding to.
The heavier/longer bullets need to spin faster to stabilize adequately.
According to Ganderite, who is a CGN member and did some tests for I believe Savage to determine the slowest twist rate that would stabilize heavier bullets, say up to 77 grains was determined to be and at what velocities.
They determined that the 1-9 twist was a good all around twist rate for most purposes.
With the exception of the 40 grain VMax, your load is to mild to produce enough velocity, which also increases the spin of the bullet.
I have a couple of 223 rifles. One is an older Ruger No1, with a 1-14 twist rate and it will not shoot any bullet well over 55 grains. Even the 55 grain monometal bullets won't stabilize when fired from that lovely rifle, no matter how fast you can safely shoot them.
The other rifle I have is a bolt action. Not spectacularly accurate with anything but good enough for hunting coyotes and ground squirrels at reasonable ranges, out to around 300 yards.
I wanted to shoot heavier bullets, simply because I picked up a couple of thousand 77 grn Sierra match bullets very cheap. At the velocities I was loading them to, they were all over the place, doing the same thing as the bullets from your rifle did.
Ganderite informed me that the 77 grain Sierra bullets I was shooting should stabilize in my 1-9 twist rifle, if I pushed the bullets to 2900+fps. This is easily done safely. He was absolutely correct with this information and those bullets shoot better than any others out of my rifle.
Since that time, I've played with a couple of other 224 caliber rifles and have started using 2900fps as a base velocity to start with.
They don't have to be pushed that fast with rifles that have a 1-7 or 1-8 twist rate as more spin is induced at lower velocities.
IMHO, if you get your bullets moving around 2900 to 3000 feet per second, all should be well.
