Warfighter Support: Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding
Highlights
The Army has issued soldiers in Iraq and Afghanistan personal body armor, comprising an outer protective vest and ceramic plate inserts. GAO observed Preliminary Design Model testing of new plate designs, which resulted in the Army's awarding contracts in September 2008 valued at a total of over $8 billion to vendors of the designs that passed that testing. Between November and December 2008, the Army conducted further testing, called First Article Testing, on these designs. GAO is reporting on the degree to which the Army followed its established testing protocols during these two tests. GAO did not provide an expert ballistics evaluation of the results of testing. GAO, using a structured, GAO-developed data collection instrument, observed both tests at the Army's Aberdeen Test Center, analyzed data, and interviewed agency and industry officials to evaluate observed deviations from testing protocols. However, independent ballistics testing expertise is needed to determine the full effect of these deviations.
During Preliminary Design Model testing the Army took significant steps to run a controlled test and maintain consistency throughout the process, but the Army did not always follow established testing protocols and, as a result, did not achieve its intended test objective of determining as a basis for awarding contracts which designs met performance requirements. In the most consequential of the Army's deviations from testing protocols, the Army testers incorrectly measured the amount of force absorbed by the plate designs by measuring back-face deformation in the clay backing at the point of aim rather than at the deepest point of depression. Army testers recognized the error after completing about a third of the test and then changed the test plan to call for measuring at the point of aim and likewise issued a modification to the contract solicitation. At least two of the eight designs that passed Preliminary Design Model testing and were awarded contracts would have failed if measurements had been made to the deepest point of depression. The deviations from the testing protocols were the result of Aberdeen Test Center's incorrectly interpreting the testing protocols. In all these cases of deviations from the testing protocols, the Aberdeen Test Center's implemented procedures were not reviewed or approved by the Army and Department of Defense officials responsible for approving the testing protocols. After concerns were raised regarding the Preliminary Design Model testing, the decision was made not to field any of the plate designs awarded contracts until after First Article Testing was conducted. During First Article Testing, the Army addressed some of the problems identified during Preliminary Design Model testing, but GAO observed instances in which Army testers did not follow the established testing protocols and did not maintain internal controls over the integrity and reliability of data, raising questions as to whether the Army met its First Article Test objective of determining whether each of the contracted designs met performance requirements. The following are examples of deviations from testing protocols and other issues that GAO observed: (1) The clay backing placed behind the plates during ballistics testing was not always calibrated in accordance with testing protocols and was exposed to rain on one day, potentially impacting test results. (2) Testers improperly rounded down back-face deformation measurements, which is not authorized in the established testing protocols and which resulted in two designs passing First Article Testing that otherwise would have failed. Army officials said rounding is a common practice; however, one private test facility that rounds told GAO that they round up, not down. (3) Testers used a new instrument to measure back-face deformation without adequately certifying that the instrument could function correctly and in conformance with established testing protocols. The impact of this issue on test results is uncertain, but it could call into question the reliability and accuracy of the measurements. (4) Testers deviated from the established testing protocols in one instance by improperly scoring a complete penetration as a partial penetration. As a result, one design passed First Article Testing that would have otherwise failed. With respect to internal control issues, the Army did not consistently maintain adequate internal controls to ensure the integrity and reliability of test data. In one example, during ballistic testing, data were lost, and testing had to be repeated because an official accidentally pressed the delete button and software controls were not in place to protect the integrity of test data. Army officials acknowledged that before GAO's review they were unaware of the specific internal control problems we identified.
Recommendations
Matter for Congressional Consideration
| Matter | Status | Comments |
|---|---|---|
| DOD did not concur with our recommendation for an independent evaluation of First Article Testing results and accordingly plans to take no action to provide such an assessment. DOD asserted that the issues we identified do not alter the effects of testing. However, based on our analysis and findings there is sufficient evidence to raise questions as to whether the issues we identified had an impact on testing results. As a result, we continue to believe it is necessary to have an independent external expert review these test results and the overall effect of the testing deviations we observed on those results before any armor is fielded to military personnel. Without such an independent review, the First Article Test results remain questionable, undermining the confidence of the public and those who might rely on the armor for protection. Consequently, Congress may wish to consider directing the Office of the Secretary of Defense to either require that an independent external review of these body armor test results be conducted or that DOD officially amend its testing protocols to reflect any revised test procedures and repeat First Article Testing to ensure that only properly tested designs are fielded. |
Closed – Implemented
|
The Chairman and Ranking Minority Member of the Subcommittee on Air and Land Forces of the House Armed Services Committee sent a letter to DOD encouraging it to address GAO's recommendation. In response, the Army contracted with the National Academies to conduct an independent assessment of the body armor test protocols, procedures, and results. Based on that assessment, the Army revised its test protocols and procedures to improve internal controls and managements practices. |
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | To determine what effect, if any, the problems we observed had on the test data and on the outcomes of First Article Testing, the Secretary of Defense should direct the Secretary of the Army to provide for an independent evaluation of the First Article Testing results by ballistics and statistical experts external to DOD before any armor is fielded to soldiers under this contract solicitation and that the Army report the results of that assessment to the office of the Director of Operational Test and Evaluation and the Congress. In performing this evaluation, the independent experts should specifically evaluate the effects of the following practices observed during First Article Testing: (1) the rounding of back-face deformation measurements; (2) not scoring penetrations of material through the plate as a complete penetration unless broken fibers are observed in the Kevlar backing behind each plate; (3) the use of the laser scanner to measure back-face deformations without a full evaluation of its accuracy as it was actually used during testing, to include the use of the software modifications and operation under actual test conditions; (4) the exposure of the clay backing material to rain and other outside environmental conditions as well as the effect of high oven temperatures during storage and conditioning; and (5) the use of an additional series of clay calibration drops when the first series of clay calibration drops does not pass required specifications. |
Closed – Not Implemented
DOD commissioned the National Academies to review its body armor testing practices to determine which practices should be used in future testing. In April 2010, the National Academies issued their report and recommended testing practices that should be used in future Body Armor testing. While this review identified improvements for future testing, it did not specifically evaluate the First Article Testing results as we recommended.
|
| Department of Defense | To better align actual test practices with established testing protocols during future body armor testing, the Secretary of the Defense should direct the Secretary of the Army to document all key decisions made to clarify or change the testing protocols. |
Closed – Implemented
In April 2010, the Army Developmental Test Command Revised its Test Operations Procedures, which documented improvements in its testing protocols and procedure as we recommended.
|
| Department of the Army | With respect to the specific inconsistencies we identified between the test practices and testing protocols, the Secretary of the Army, based on the results of the independent expert review of the First Article Test results, should (1) determine whether those practices that deviated from established testing protocols during First Article Testing will be continued during future testing and change the established testing protocols to reflect those revised practices; and (2) evaluate and re-certify the accuracy of the laser scanner to the correct standard with all software modifications incorporated and include in this analysis a side-by-side comparison of the laser measurements of the actual back-face deformations with those taken by digital caliper to determine whether laser measurements can meet the standard of the testing protocols. |
Closed – Not Implemented
While the National Academies completed an independent review of first article testing for the Army, the Army did not use that information to determine whether its body armor testing practices had deviated from established testing protocols during First Article Testing. Further, the Army did not evaluate and re-certify the accuracy of the laser scanner to the correct standard with all software modifications incorporated and include in this analysis a side-by-side comparison of the laser measurements of the actual back-face deformations with those taken by digital caliper to determine whether laser measurements can meet the standard of the testing protocols.
|
| Department of Defense | To improve internal controls over the integrity and reliability of test data for future testing as well as provide for consistent test conditions and comparable data between tests, the Secretary of Defense should direct the Secretary of the Army to provide for an independent peer review of Aberdeen Test Center's body armor testing protocols, facilities, and instrumentation to ensure that proper internal controls and sound management practices are in place. This peer review should be performed by testing experts external to the Army and DOD. |
Closed – Implemented
DOD commissioned the National Academies to review its body armor testing practices. In April 2010, the National Academies completed its review of testing protocols, facilities, and instruments, making a number of recommendations to improve internal controls and management practices.
|