Consider these changes. Use a match primer as this really helps to reduce stringing at LR. CCI BR2 or Fed 210M is ideal. Use a hodgdon extruded powder like h4831SC and H1000. Collet neck size your cases to eliminate any runout during sizing. Measure runout on your loaded ammo. There should be no more then 4 thou - less is better.
With a 165gr/180gr SST, 155gr/168/178gr Amax, you will have no issue with a load that can shoot MOA or better at 500m. Try and test at least 200yds away. 300yds is ideal but not always available. You should be able to shoot 1 to 1.5" at 200yds in calm air.
Bed that action especially around the lug. Should be fully supported. Get the trigger as light as possible. Free float that barrel. I will bed under the nut and first bit of barrel but not critical. Controlling the lug and around the action screws is critical. You will not need to he man the action screws once the bedding is properly done.
There is little wrong with your scope as long as it holds zero. In fact, shooting further might be advantages due to parallax (unless you can dial it out with a AO).
Use a target with a big cross which you staple up level. The cross should be large and thick enough so your reticle just covers it at 500m. There will be enough aiming error due to mirage without making it tougher following the 'bouncing ball'. You can draw a 5" circle around the center of the X so you know you are in the zone.
To start, aim dead center and work on consistency. If you can, have a look every few shots where they have landed relative to your POA, wind, mirage. If all is well, you will have a horizontal group with little vertical. Try for 3" of vertical. The horizontal is not important until you learn consistent form.
if the load is good, your shooting form will control vertical.
Once you are consistently shooting vertical 3" and under, you can work on your doping skills to bring the groups into center.
You have a rig that can shoot MOA as far as you want to go. Good ammo and lots of practise will get you to your goal.
Jerry