The technical drawing package owned by the government(or Colt for the M4) is the standard, with given tolerance. Since FN and Colt got the TDP from the government, that is the military specification as the bid specified the TDP is to be used. The specification is not a guideline - it is a set of condition the end product must satisfy for contractual purposes.
In the case of the dimension of a part, it makes no sense that you do not follow the specification because you believe you can go "beyond and better than the milspec". it is a dimension, with a given tolerance, that a part needs to conform to achieve parts interchangability. If the dimension in the TDP does not mean anything, I am not quite sure what does!
The Technical "Data" Package (TDP), is a military specification for the M4, which was put forth by the military to Colt as an
"outline" for the manufacturing process, materials, tolerances, assembly, finishes, proof testing and dimensions needed for manufacture of the weapon by Colt to meet their contract with the U.S. military.
Though the military specifications “milspecs" and the military standard “milstds” which Colt must abide by to satisfy their Military contract consist of rigorous inspections, tolerances, endurance and interchangeability of parts, are not adhered to religiously by every firearms manufacturer out there. Therefore, parts may vary drastically as opposed to what is designated by the Military. Could be more, or could be less, especially with worn parts from questionable manufacture.
And so, I say to only procure top shelf components from known manufacturers and avoid the stacking effect with parts tolerances which may cause catastrophic failures when flipping and flopping parts of your gun without proper quality assurance. Especially with questionably "used" components.
Quality assurance with milspecs and milstandards are maintained by an onsite U.S. government inspector who keeps an office at Colt’s factory and by a number of Colt’s own inspectors. I doubt such government inspectors keep offices at other firearms manufacturing facilities that cater to the civilian market, or other agencies around the world.