Best Practices:

A More Constructive Test Approach Is Key to Better Weapon System Outcomes

NSIAD-00-199: Published: Jul 31, 2000. Publicly Released: Jul 31, 2000.

Contact:

Paul L. Francis
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Pursuant to a congressional request, GAO provided information on the role of testing and evaluation in product development, focusing on: (1) how the conduct of testing and evaluation affects commercial and Department of Defense (DOD) program outcomes; (2) how best commercial testing and evaluation practices compare with DOD's; and (3) what factors account for the differences in these practices.

GAO noted that: (1) for the leading commercial firms GAO visited, the proof of testing and evaluation lies in whether a product experiences what one firm called "late cycle churn," or the scramble to fix a significant problem discovered late in development; (2) late cycle churn has been a fairly common occurrence on DOD weapon systems; (3) often, tests of a full system identify problems that could have been found earlier; (4) leading commercial firms GAO visited use testing and other techniques to expose problems earlier than the DOD programs GAO reviewed; (5) the firms focus on validating that their products have reached increasing levels of product maturity at given points in time; (6) the firms' products have three maturity levels in common--components work individually, components work together as a system in a controlled setting, and components work together as a system in a realistic setting; (7) the key to minimizing late surprises is to reach the first two levels early, limiting the burden on the third level; (8) by concentrating on validating knowledge rather than the specific technique used commercial firms avoid skipping key events and holding hollow tests that do not add knowledge; (9) on the weapon programs, system level testing carried a greater share of the burden; (10) earlier tests were delayed, skipped, or not conducted in a way that advanced knowledge; (11) the differences in testing practices reflect the different demands commercial firms and DOD impose on program managers; (12) leading commercial firms insist that a product satisfy the customer and make a profit; (13) success is threatened if managers are unduly optimistic or if unknowns about a product are not resolved early, when costs are low and more options are available; (14) the role of testing under these circumstances is constructive, for it helps eliminate unknowns; (15) product managers view testers and realistic test plans as contributing to a product's success; (16) success for a weapon system program is different--it centers on attempting to provide a superior capability within perceived time and funding limits; (17) success is influenced by the competition for funding and the quest for top performance--delivering the product late and over cost does not necessarily threaten success; (18) testing plays a less constructive role in DOD because a failure in a key test can jeopardize program support; (19) specifically, test results often become directly linked to funding and other key decisions for programs; and (20) such a role creates a more adversarial relationship between testers and program managers.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To lessen the dependence on testing late in development and foster a more constructive relationship between program managers and testers, the Secretary of Defense should instruct the managers and testers of weapon system programs to work together to define levels of product maturity that need to be validated, structure test plans around reaching increasing levels of product maturity, and orchestrate the right mix of tools to validate these levels. Acquisition strategies should then be built and funded to carry out this approach. Such a focus on attaining knowledge, represented by product maturity levels, can guard against the pressures to forego valuable tests to stay on schedule or to hold tests that do not add value to the product. This approach, which creates common ground between testers and product managers in leading commercial firms without compromising independence, still demands that the product or weapon system being matured meet the needs of the customer.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD concurred with this recommendation, and addressed it in the latest revision to DOD Directive 5000.1 and DOD Instruction 5000.2.

    Recommendation: The Secretary of Defense should not let the validation of lower levels of product maturity--individual components or systems in a controlled setting--be deferred to the higher level of system testing in a realistic setting. Although the mix of testing and evaluation tools may change and the acquisition strategy may be altered during the course of a weapon system development, the focus on attaining product maturity levels should not change. This discipline should also help guard against the practice of setting cost and schedule constraints for programs without considering the time and money it takes to sensibly validate maturity.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD concurred with this recommendation, and addressed it in the latest revision to DOD Directive 5000.1 and DOD Instruction 5000.2.

    Recommendation: The Secretary of Defense should require weapon systems to demonstrate a specified level of product maturity before a major programmatic approvals. In doing so, the Secretary may also need to establish interim indicators of product maturity to inform budget requests, which are made well in advance of programmatic decisions. Testing and evaluation could then be cast in a more constructive role of helping a weapon system reach these levels and would ease some of the burden currently placed on program managers to rely on judgment, rather than demonstrated product maturity, in promising success at times when major funding commitments have to be made.

    Agency Affected: Department of Defense

    Status: Closed - Not Implemented

    Comments: DOD did not concur with this recommendation, and has not taken any action since the report was issued in fiscal year 2000.

    Apr 23, 2014

    Apr 22, 2014

    Apr 18, 2014

    Apr 16, 2014

    Apr 11, 2014

    Apr 10, 2014

    Apr 9, 2014

    Looking for more? Browse all our products here