IMR4320 for some reason is a bit of an orphan powder; you don't see much loading data for it nor do you see a whole lot of people using it, even though on paper it looks like a perfectly fine companion powder to 4895, 4064, Varget, etc.
I suspect you will get very good results with IMR4320 with your Hornady 168 BTHPs, 180SMKs, etc.
To the OP - it is good that you are approaching this very prudently and working your loads up exactly like the reloading manuals instruct you to. Keep it up.
Oftentimes you'll find that a very mild load gives extremely good accuracy at short range (100 yards). If you were to chrono such a load, you'll usually find that it is very slow, and the variation in speed from one shot to the next is quite large (having a velocity spread of 100fps or even more). There's nothing wrong with this, if this fulfills the entirely of your shooting needs. It is worth knowing though, that working your load all the way to max can give you a load that gives you very good accuracy and very good velocity uniformity; this sort of load would give you much better results at 600 yards (and possibly further), as well as at 100 yards, in case that is of any interest/use to you.