So, if you are "sighted in", that means the notch on rear sight (not the dovetail on the barrel) and the front sight bead, line up with where that barrel is throwing bullets - you say at 50 yards. Could be a bent barrel - only need perhaps one bullet length at muzzle - is where the bullets will go - bore centre line can be bent like banana and still throw bullets to same place. Could be ding in muzzle causing bullets to deflect. Could be front sight offset the other way in its dovetail? As mentioned above in Post #3 and #5, could be a bent rear sight body. Could be the adjustable plate for rear sight notch is not centered in the rear sight body?
From your picture - is as if that rear sight body is twisted on that barrel - so perhaps the rear sight body connection to the male dovetail has been bent?
I have a Schultz and Larsen Model 61 .22 Long Rifle Target Rifle - was made back in the day for 50 meter bullseye competition - the front and rear aperture sights are deliberately off set to the left of the bore line. So it will have vertical divergence for the bullet's trajectory, and also windage divergence between sight line and bore line - hence can only be sighted in at one range - 50 meters as designed. Shorter range than that, bullets hit right of point of aim; longer range than that, bullets hit left of point of aim. The "x" ring, or "V-bull" is smaller than .22 bullet hole - 10 point ring is about .44" - those shooters, I presume, were unusually fussy about very precise "sighting in".
You might want to try a couple or three shots at 100 or 150 yards, and see where your group is. Your group might be centered, or it might not be. I would expect the group to be somewhat lower - I would not worry about elevation - it is the left/right change that you might want to find out whether that exists or not. And once you increase the range, also will start to see wind effects - that might want to blow your bullets one way or the other - so might want a wind flag or two - or pick a noticeably calm time of day?