Defense Acquisitions:

Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy

GAO-06-368: Published: Apr 14, 2006. Publicly Released: Apr 14, 2006.

Additional Materials:

Contact:

Michael J. Sullivan
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Defense (DOD) is planning to invest $1.3 trillion between 2005 and 2009 in researching, developing, and procuring major weapon systems. How DOD manages this investment has been a matter of congressional concern for years. Numerous programs have been marked by cost overruns, schedule delays, and reduced performance. Over the past 3 decades, DOD's acquisition environment has undergone many changes aimed at curbing cost, schedule, and other problems. In order to determine if the policy DOD put in place is achieving its intended goals, we assessed the outcomes of major weapons development programs initiated under the revised policy. Additionally, we assessed whether the policy's knowledge-based, evolutionary principles are being effectively implemented, and whether effective controls and specific criteria are in place and being used to make sound investment decisions.

Changes made in DOD's acquisition policy over the past 5 years have not eliminated cost and schedule problems for major weapons development programs. Of the 23 major programs we assessed, 10 are already expecting development cost overruns greater than 30 percent or have delayed the delivery of initial operational capability to the warfighter by at least 1 year. The overall impact of these costly conditions is a reduction in the value of DOD's defense dollars and a lower return on investment. Poor execution of the revised acquisition policy is a major cause of DOD's continued problems. DOD frequently bypasses key steps of the knowledge-based process outlined in the policy, falls short of attaining key knowledge, and continues to pursue revolutionary--rather than evolutionary or incremental--advances in capability. Nearly 80 percent of the programs GAO reviewed did not fully follow the knowledge-based process to develop a sound business case before committing to system development. Most of the programs we reviewed started system development with immature technologies, and half of the programs that have held design reviews did so before achieving a high level of design maturity. These practices increase the likelihood that problems will be discovered late in development when they are more costly to address. Furthermore, DOD's continued pursuit of revolutionary leaps in capability also runs counter to the policy's guidance. DOD has not closed all of the gaps in the policy that GAO identified nearly 3 years ago, particularly with regard to adding controls and criteria. Effective controls require decision makers to measure progress against specific criteria and ensure that managers capture key knowledge before moving to the next acquisition phase. However, DOD's policy continues to allow managers to approach major investment decisions with many unknowns. Without effective controls that require program officials to satisfy specific criteria, it is difficult to hold decision makers or program managers accountable to cost and schedule targets. In this environment, decision-making transparency is crucial, but DOD is lacking in this area as well.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Production commitment (Milestone C): Initiation of low-rate production: (1) demonstrate manufacturing processes; (2) build production-representative prototypes; (3) test production-representative prototypes to achieve reliability goal; (4) test production-representative prototypes to demonstrate product in operational environment; (5) collect statistical process control data; (6) demonstrate that critical processes are capable and in statistical control; and (7) conduct decision review to begin production.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD has revised its defense acquisition policy and issued a Manufacturing Readiness Level (MRL) Deskbook. While DOD's revised policy, issued in December 2008, does not strictly require programs to collect or use statistical process control metrics, it has placed increased emphasis on the importance of building and testing production representative prototypes, and on the importance of assessing and measuring manufacturing readiness before beginning production. DOD's policy states that entrance into the production and deployment phase at Milestone C depends on the following criteria: acceptable performance in developmental test and evaluation and operational assessment; mature software capability; no significant manufacturing risks; manufacturing processes under control; an approved Initial Capabilities Document; an approved Capability Production Document; a refined integrated architecture; acceptable interoperability; acceptable operational supportability; and demonstration that the system is affordable throughout the life cycle, fully funded, and properly phased for rapid acquisition. In a briefing given after the revised policy was issued, DOD officials detailed more specific metrics that they noted are required by the DOD Acquisition process and will be applied to new and on-going programs. The metrics related to Milestone C are: (1) Joint Requirements Oversight Council-validation of the program requirements documentation related to production; (2) demonstration of manufacturing processes; and (3) demonstration that production-representative test articles have met the approved requirements in the system's intended environment. Subsequently, in May 2011, the Department issued its MRL Deskbook, which is intended to provide best practices for conducting assessments of manufacturing readiness. DOD expects the Deskbook to be used as a tool and resource to help defense programs with manufacturing related reporting and oversight. The collective impact of these efforts to place added emphasis on the need for manufacturing maturity before committing to a production program is sufficient to meet the intent of our recommendation.

    Recommendation: DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Design readiness review: Beginning of system demonstration: (1) complete 90 percent of design drawings; (2) complete subsystem and system design reviews; (3) demonstrate with prototype that design meets requirements; (4) obtain stakeholders' concurrence that drawings are complete and producible; (5) complete the failure modes and effects analysis; (6) identify key system characteristics; (7) identify critical manufacturing processes; (8) establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems; and (9) conduct decision review to enter system demonstration.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD has initiated a number of acquisition reform pilot programs and issued several policy memorandums, which ultimately resulted in the revision of DOD's acquisition policy, known as the DOD 5000 series. Although more could be done to further strengthen the policy, we believe that the most recent revision, issued in December 2008, essentially addresses the intent of our recommendation. The revised policy requires program managers to hold a Preliminary Design Review (PDR) prior to initiating system development--now referred to as Engineering and Manufacturing Development (EMD). According to the policy a successful PDR will inform requirements trades; improve cost estimation; and identify remaining design, integration, and manufacturing risks. In addition, the policy also emphasizes the importance of using key metrics to assess design maturity at the system-level Critical Design Review (CDR) before continuing into the demonstration phase of EMD--now referred to as System Capability and Manufacturing Process Demonstration. The metrics emphasized are: successful completion of subsystem CDRs; the percentage of hardware and software product build-to specifications and drawings completed and under configuration management; planned corrective actions to hardware/software deficiencies; adequate developmental testing; an assessment of environment, safety and occupational health risks; a completed failure modes and effects analysis; the identification of key system characteristics; the maturity of critical manufacturing processes; and an estimate of system reliability based on demonstrated reliability rates. In a briefing given after the revised policy was issued, DOD officials detailed more specific metrics that they noted are required by the DOD Acquisition process, and will be applied to new and on-going programs. The metrics related to CDR are (1) completion of System-level CDR, (2) completion of 90 percent of design drawings, (3) successful milestone decision review conducted, (4) completion of failure modes and effects analysis, and (5) establishment of reliability targets and identification of key manufacturing processes.

    Recommendation: DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Program start (Milestone B): Start of product development: (1) demonstrate technologies to high readiness levels; (2) ensure that requirements for the product are informed by the systems-engineering process; (3) establish cost and schedule estimates for product on the basis of knowledge from preliminary design using system-engineering tools; and (4) conduct decision review for program start.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). In addition, the USD (AT&L) issued a policy memorandum in September 2007 that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. We believe that the certification requirements in the law and the new prototyping policy establish controls that meet the intent of our recommendation.

    Recommendation: Furthermore, to ensure that major decisions are transparent and that program officials and decision makers are held accountable, the Secretary of Defense should require decision makers to include written rationale for each major decision in acquisition decision documentation. The rationale should address the key knowledge-based criteria appropriate for milestone decisions, explain why a program's level of knowledge in each area was deemed acceptable if criteria have not been met and provide a plan for achieving the knowledge necessary to meet criteria within a given time frame.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: The Department of Defense agreed that they should be accountable for program decisions and appropriately record the considerations in moving from one stage of development to the next. Milestone decision authorities are to employ the decision memorandums to document decisions and their rationale. Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). We believe that these certification requirements establish controls that meet the intent of our recommendation. Therefore, we are closing this recommendation as implemented.

    Jul 31, 2014

    Jul 30, 2014

    Jul 28, 2014

    Jul 17, 2014

    Jul 14, 2014

    Looking for more? Browse all our products here