First thing with the handguard. It clips onto the barrel AFTER insertion into the housing on the gas block assembly. It should be free to "wiggle" and not make contact with top of that assembly, particularly with the stock inserted. When I replaced my handguard (with the wood stock), I thought the fit had to be snug, because that was how it was with the original. When I replaced the stock with the new one, that snug fit proved to be an epiphany for me as I experienced the exact issue as yourself.
So, firstly - with the stock removed, check the handguard and undertake some light sanding and or trimming if there is excessive contact. The handguard (when affixed to the barrel) should have some play when it is set into the gas block assembly. Next comes the tricky part.
The Hogue stock is basically a pliable rubber coating over a plastic skeleton. There is a metal liner, which fits over the stock and is inserted into the gas block assembly as well. You will note that it is a stamped contraption and has marks all over it from compression (pressure) as opposed to friction from movement. My rifle also had friction marks where the slide assembly made contact with this liner. When fitting the stock into the gas block, it fit quite snugly sliding into the gas block even before the receiver was in contact with the stock. At this point in time, the gas block was acting like a fulcrum and pressure from the stock/liner assembly was being exerted. This pressure is relieved as the block warms up after a few firings and metal starts to expand.
To relieve this, I took a pair of adjustable pliers, and gently "squeezed" the liner tighter to the stock and filed away excessive points of impact. After repeating this process several times until the stock would insert into the gas block assembly without undue pressure. I also lightly taped the liner tabs (making direct contact with the gas block as I noted the lower one (in particular) was making almost full contact with the gas block. I lightly sanded the surface of liner tabs and lightly greased - just because I like to - to ensure that there was not any uneven pressure.
Lastly, I undertook some light trimming on the stock to ensure that there was no binding or pressure points between the handguard and the stock. When the trigger guard is affixed, my handguard will "wiggle" slightly in vertical movement.
My groupings improved immensely and the bullet wander went away. That was my solution. I also discovered that the rifle adapts very well to the Winchester 64 gr psp bullets with about 25 grains of Win 748 powder.