Are you crimping the shorter bullets? Crimping will uniform the bullet pull weight, something that is built in when a bullet is seated very close to or touching the rifling. .458 Winchester ammo can be fired in rifles with Lott chambers, and I've never heard about any accuracy issues when doing so, although pressure will be lower, so bullet jump, in this instance, does not seem to impact accuracy in a real practical sense when bullets are crimped.
If you were shooting lead bullets, I would say that better accuracy could be realized with longer bullets because the longer bullet with a long bearing surface better stabilizes relative to the center line of the bore (it's less prone to wobble or yaw in the barrel), but I don't believe this follows with jacketed bullets. Target shooters however, who choose VLD's have more difficulty getting those bullets to shoot than they do standard match bullets, so there must be a point of diminishing returns when the bearing surface becomes very short relative to the overall length of the bullet. In the case of the 350/.458, there is a very long bearing surface relative to it's length. Perhaps you just need to experiment with different loads or powders to find the sweet spot for those 350's. Choose a powder with a medium burning rate (3031, 748, RL-7 etc) and if you are trying to drive the bullet very fast slow it down a mite, and conversely if you are driving the bullet very slowly, try speeding it up.