If you want to try sabots .... Hornady SST Sabot Slugs or Remington AccuTips have a reputation to shoot well out of many different barrels/rifled shotguns.
With respect to your cast Lee slugs .... those slugs are not full bore .... they are undersized ... and in your gun/barrel ... and how you load them ... they might not engage the rifling correctly.
1.) You can try to reduce the powder load ... and that might help to gain some accuracy.
2.) You can try a thicker wad ... or a Teflon wrap within the shot cup ... or even a paper wrap around the slug (but within the shotcup) ... and to increase the diameter .... and so that the wad/slug engages the rifling better.
Also, try to find a wad that was shot with one of those Lee slugs .... the wad will tell you what is happening ...
But to be honest .... those Lee slugs are not known for stellar accuracy ..... they are more known as a cheap plinking solution ...