Try peening the metal with it in place.
Yep. That is how it was originally installed.
Take some care while fitting it back in place, so as to make sure the threaded stub seats full in the bottom of the socket. This may require some care on displacing the metal a wee bit, outwards, around the edges of the socket.
You can use a flat nose punch, though I would try to use a round nosed one, and carefully work around the edge of the socket, using light hammer taps on the punch, to carefully displace the surface metal inwards around the tapered stub, to secure it in place.
The basic idea is to only move metal at the surface of the part, so it's a lot of light strikes, using a small hammer, while walking the punch nose around the rim of the hole.
If you have a buffer or a hard felt buffing wheel on a Dremel type tool, you can round over and polish the end of a center punch.