I don't claim to fully understand the health hazards related to blast overpressure caused by muzzle brakes as opposed to unbraked rifles, and I'd be happy to be corrected if wrong, but I'm not personally worried about much besides hearing damage from using insufficient hearing protection.
I have to compare braked rifles, which average in the 160-170db range, to military artillery such as the 105mm towed howitzer (charge 8) at 183db or the 60mm mortar (charge 4) at 185db. The men who operate this equipment frequently, even when double plugging, tend to report hearing loss and can report mTBI (mild Traumatic Brain Injury, ie minor concussion) due to the direct and repeated results of blast overpressure, which being 10-25db higher according to how the decibel scale works equates to 10 to 150 times more intense or 2 to 6 times louder in scale. (The db scale is a funky scale)
I'm sure there may be some hazards relating to sitting fairly close to a muzzle braked rifles for extended periods, but with how the intensity of sound works, I wouldn't be overly worried about getting a concussion or hearing damage, more like a headache and crap in my eyes. If there is other damage, I'd like to hear what that is so I can learn, but that seems to be the most common hazard.
With all that said, the sound walls would still be the best solution to staying away from blast overpressure in my opinion. Cheap, effective, lets everybody use their toys, and keeps everybody happy.