You are looking for low velocity and high pressure (to cycle the action). I think a heavy bullet is required. I would start with 80 and 75 gr bullets.
You'd be right at the edge for max OAL and still being able to stuff the rounds in the magazine.
While I may not even move forward with this project; I'm an engineer, and I like a challenge; and the challenge is to make the rifle at hand function. While others may have done the ground work; they are few that are willing to share the info. I will be doing a lot of research to establish a starting point and when I'm reasonably certain that it could be done, then I would proceed. I need a baseline.
Please make sure to remove any muzzle device prior to your R&D.
You can destroy your flash hider/muzzle device with an unstable, tumbling, projectile.
People have a hard time even with a 1:7 twist, so you can imagine that with a 1:12 it will be a nice engineering challenge.
Keep me posted Sir.
Bullets don't tumble in the barrel. Id suggest unless your muzzle device is more than 12" long, I don't think you'd have to worry about tumbling bullets damaging a muzzle device.
To the OP, Question for you, is your intent to make this ammo work with a standard un-modified AR, or will you permit modification of the gun in order to make this work?
IF you are unwilling to modify the gun, then I suspect the heaviest bullet you can use is going to be a given. I'm torn between predicting that a fast burning or a slow burning powder would be ideal.
Slow burning powder in a 16> " barrel will likely result in unburnt powder going out of the barrel with no contribution to cycling the gun. I suspect faster burning powder is going to generate the GASES that you need fast enough to drive the action before the bullet exits the barrel, but I'd be concerned about a relative small window of success before you start encountering excessive chamber pressures.
Definitely an interesting experiment to see if modifications to the ammunition alone can get you there.
If you are willing to modify the gun, then I suggest you consider any of the following modifications that will improve your chances of success.
1) Rechambering the gun in .300 blackout. Its basically what this round was invented for.
2) Installing a variable gas block to increase the amount of gas permitted through the system to drive the action.
3) Moving to a pistol length carbine system on as long a barrel as you would tolerate. This should probably be used in conjunction with a variable gas block.