Sold 'em in the shop I worked in, long ago. The shop was the Canadian distributor for a while too. No M1A rifle(long dollars then too) ever came back damaged by the ammo. However, if your rifle is still under warrantee, the ammo you use matters.
SA Inc.'s M1A manual says NATO Spec FMJ, BT or HP ammo only. No SP's. Handloads void the warrantee. It's a CYA thing for them. I wouldn't worry about 180's, myself. 180's won't bother your rifle(seriously doubt an SP will either. Fired a lot of 'em out of my M1 and Winchester semi'd M14 with no fuss), but a 165 will very likely shoot a lot better. Mind you, a National Match likes match grade bullets and ammo.
The M1A manual says this. Nothing about bullet weights or any other internet myth about ammo damaging rifles
"The M1A is designed and built to specifications to shoot standard factory .308 made to SAAMI specifications or 7.62x51 NATO ammunition."