The bolt pin is a tight fit. Also, it is only meant to come out one way....you have to drive out by hitting the round end .... not the rectangular end with a proper size punch.
Also, you need to put it on a very sturdy base. It helps to put it on a piece of metal, like a vise. I place mine spanning the jaws of my vise .... opened up just enough so the pin has a place to fall once I get it moving.
Finally, if it hasn't been removed before and you have shot a lot of corrosive ammo, it might be a bit corroded. You could put some 'fluid film' or other penetrating oil to loosen it up a bit.
Keep trying.