It looks like it's held in with a pin but it's not, so don't try hammering on it anymore... Watch this:
What year/country is yours? The rear sight on my '51 russian was definitely not a pin and it came out as shown in the video, but other SKS variations probably do have a few differences.
It looks like it's held in with a pin but it's not, so don't try hammering on it anymore... Watch this:




























