The rifle's recoil is expressed in terms of energy. To absorb the energy, the shooter puts up a resistance (inertia and bracing your muscles against the hit) over a distance (the amount your shoulder/body moves). For all shooters the energy to be absorbed is the same (for the same rifle), so either distance traveled or the force applied to your shoulder must change.
A heavy shooter doesn't move much, so the force applied to his shoulder is high. A light shooter moves more, so the force applied to his shoulder is lower. In both cases the energy absorbed is the same, but the force felt is different.
I agree with 2bad4u2 that stock fit and other factors play at least as much or more important roles than the size of the person shooting, though.
Just as a personal observation from years of trapshooting, the guys with the flinching issues were almost always the big guys, while the skinny guys just seemed to roll with the recoil instead of eating it.
Mark