It's simple physics. Any time you remove material you remove strength and rigidity. Fact, end of story, no exceptions.
There's a bit more to it than that. Remove materials in the right area, and you can, in fact, improve the overall strength and rigidity of an object. Distribution of stresses is paramount!
However, when it comes to rifle barrels, the geometry limits what you can do. In this situation, you cannot add rigidity by removing material. Unless you happen to remove material so exactly that you improve the barrel harmonics, you will see no gain in accuracy. Even then, you will likely see a gain with one particular load, and a decrease with a different load. This is why every gun shoots differently with different loads - its all about harmonics. More than likely you will see an overall decrease in accuracy, if anything.
The best way to flute a barrel without significantly reducing the rigidity is to cut an odd number of flutes. Simple analysis of the moment of inertia shows that you reduce weight at a greater rate while better retaining rigidity of the barrel vs cutting an even number of flutes.
tl;dr - Flutes don't do much for the accuracy, but they look freaking sweet