Despise the 180.
I really do not like any 180 grain cast or factory bullets for 40. Recoil is almost comparable to to 45acp so what's the attraction? Shoot 45! I have seen more guys with fast misses with the 180 combination than most nines or 45's. My far, far away pal Rick from R&RBullets who now is so freakin far away that I cannot get his Magma Engineering 155grain cast SWC that my tears are hitting the keyboard!
If you must use the unsuitable 180 grain bullet, do not exceed 4.0 grains of Bullseye. You will only increase the recoil. Four grains will work in the M&P, but a softer load that will function is 3.8 or 3.9.
Switch to a much lighter bullet and you will notice a big improvement. I'm not kidding...