Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes

DOD concurred with this recommendation, and immediately after publication of the report the Quadrennial Defense Review proposed a strategy to improve decision making by implementing more agile, transparent and open processes and advocated reaching investment decisions through collaboration among the warfighters, acquisition, and resource communities. Subsequently, Congress amended title 10, United States Code, to require the leaders of both the acquisition and requirements communities certify, before beginning a new technology development program, that the program addresses a valid need, is necessary and appropriate, and that the estimated cost is consistent with the stated priority level of the need. In December 2008, DOD issued revisions to the department's acquisition regulation that reflect the certification legislation and capture in policy a series of other changes intended to improve program stability and transparency. In addition, the Weapon System Acquisition Reform Act of 2009 established the Director of Cost Assessment and Program Evaluation as the principle official within DOD senior management for overseeing, reviewing, and in some cases conducting analyses to assist the Secretary of Defense with prioritizing weapon system investments. On April 6, 2009 the Secretary of Defense testified before Congress and outlined sweeping changes aimed at reducing or eliminating poor performing programs in DOD's weapon system acquisition portfolio. These proposed changes, which include restructuring the Army's Future Combat Systems program, indicate that the department is taking steps in the right direction. While more could be done to further improve, we believe that these actions to reform the defense acquisition system and balance current acquisition program investments with available resources effectively address the intent of our recommendation.

In December 2008, DOD revised its acquisition policy, DODD 5000.02, which more clearly defined incremental product development for achieving desired capabilities through evolutionary acquisitions as the preferred DOD investment approach. The policy clearly specifies that each increment should be managed like its own program with separate decision milestones, cost, and schedules. It further states that upgrades, product improvements, and similar efforts that provide signficant increases in operational capabilities should be managed as separate increments.

DOD concurred with the recommendation. On December 2, 2008, DOD issued a revised version of the defense acquisition policy, DODD 5000.02, containing several requirements that, in concert with the controls established in previously enacted legislation, essentially meet the intent of our recommendation. For example, the revised policy establishes a mandatory materiel development decision review as the formal entry point into the acquisition process for all programs. At that review, representatives from the requirements, acquisition, and funding communities are expected to work together to provide input to the decision maker: Joint Staff present the recommendations of the Joint Requirements Oversight Council, the DOD component provides the initial capabilities document, OSD's Program Analysis and Evaluation propose study guidance for the Analysis of Alternatives, and the Comptroller provides funding estimates. With this data in hand, the milestone decision authority (MDA) certifies to Congress that, among other things, a valid need is being addressed, the right people are addressing the need, and a cost estimate for the system has been submitted. The policy further notes that if the cost estimate established at Milestone A increases by 25 percent or more prior to Milestone B, the program manager must notify the MDA. The MDA, in consultation with the requirements community, shall then determine whether the capability need is truly worth the estimated level of resources.

In December 2008 DOD revised its acquisition policy to increase its emphasis on the technology development phase preceding an acquisition effort. The policy emphasizes that programs develop technology development strategies, along with associated funding, that provide for two or more competing teams producing prototypes of the system and/or key system elements prior to beginning product development. The policy requires critical technologies for the product be demonstrated in a relevant environment to be considered mature enough for use in product development. Concurrent with the policy revision, DOD also established configuration steering boards to review technical requirements to ensure available resources are not exceeded and to identify options to reduce costs and improve delivery. In addition Title 10 2366(b) now requires a DOD certification to Congress as part of the Milestone B business case certification that technologies have reached this maturity level before proceeding into development. DOD has recently emphasized technology maturity for programs that have proceeded past milestone B, or already in development, by restructuring programs. For example, DOD recently proposed canceling some ballistic missile defense projects and restructured the Future Combat Systems, due to concerns about the program's continued immature technologies despite being more than 5 years into product development. Concurrently with the implementation of the revised 5000.02, DOD implemented milestone A certification and established Configuration Steering Boards to review technical requirements and balance funding needs.

DOD oncurred with this recommendation. Subsequently, Congress amended title 10, United States Code, to require the leaders of the acquisition, requirements, and resource communities certify, before beginning a new technology development program, that the program addresses a valid national security need and is a necessary and appropriate material solution. The revised 5000.02 acquisition policy provides for the milestone decision authority's (MDA's) approval of the business case and certification for the program to move forward. Key inputs to that decision are the recommendations of the Joint Requirements Oversight Board and the initial capabilities document (ICD) prepared by the responsible Service component. The ICD describes the needed capability, the operational risk, and the basis for determining that non-materiel approaches will not sufficiently mitigate the capability gap. The revised policy aslo requires an analysis of alternatives (AoA). The AoA assesses preliminary materiel solutions, identifies key technologies, and estimates life-cycle costs to satisfy the capability need documented in the approved ICD.

Section 2366 of title 10 United States Code, as amended, and DOD's 5000.02 acquisition regulation, as revised in December 2008, requires the Milestone Decision Authority (MDA) to certify that an acquisition program is approved to start at milestone B, engineering and manufacturing development. The milestone decision authority (MDA) is required to include a statement in the decision memorandum that the program's business case has been reviewed and the entrance criteria met, including successful completion of an Analysis of Alternatives (AoA). The MDA approves the AoA guidance prepared by the Director, Program Analysis and Evaluation, and the study accomplished by the assigned DOD component. The AoA assesses preliminary materiel solutions, identifies key technologies, and estimated life-cycle costs for satisfying the documented capability need.

Section 2366 of title 10 United States Code, as amended, and DOD's 5000.02 acquisition regulation, as revised in December 2008, requires certifications at Milestones A and B to continue weapon system development depending on the amount of technology development required. Following the materiel development decision, the Milestone Decision Authority (MDA) can authorize entry into either Milestone A "Technology Development" or Milestone B "Engineering and Manufacturing Development" consistent with phase-specific entrance criteria and statutory requirements. Progress through the acquisition management system depends on obtaining sufficient technical knowledge to continue to the next phase of development. The MDA's decision memorandum documents review of the program and its business case. This includes successful completion of the analysis of alternatives (AoA) to assess the critical technology elements associated with each proposed materiel solution, technology maturity, integration risk, manufacturing feasibility, and, where necessary, technology maturation and demonstration needs. Entrance into Milestone B also requires evidence that critical technologies have been demonstrated in a relevant environment, i.e. a technology readiness level of 6 or greater. The revised regulation states that the requests for proposals for the engineering and manufacturing development phase advise prospective offerors that the government will not award a contract to a developer whose proposal is based on technologies that have not been demonstrated in a relevant environment and that offerors will be required to specify the technology readiness level on which the proposal is based and to provide reports documenting how the technologies have been demonstrated in a relevant environment.

10 USC 2366a, as amended, and the recently revised acquisition regulation, DOD 5000.02, provides for both milestone A and B certification to continue development, including review and approval of the business case. The revised policy increases the emphasis on demonstrating a knowledge-based business case prior to beginning development, in large part by requiring program managers to use competitive prototyping and hold a Preliminary Design Review (PDR) to reduce technical risk, validate designs and cost estimates, evaluate manufacturing processes, and refine requirements prior to beginning system development. In addition, the policy emphasizes the importance of using key metrics to assess design and manufacturing maturities before continuing into system demonstration. Specifically, DOD officials have noted that the metrics required by the acquisition process include (1) completion of System-level Critical Design Review; (2) completion of 90 percent of design drawings; (3) successful milestone decision review conducted; (4) completion of failure modes and effects analysis; and (5) establishment of reliability targets and identification of key manufacturing processes. Further, the revised policy places increased emphasis on the importance of demonstrating that there are no significant manufacturing risks and that manufacturing processes under control, among other things, before beginning production. In concert with these policy revisions, title 10, United States Code, now requires the Secretary of Defense to certify at both milestone A and milestone B that a program's business case meets certain knowledge-based criteria largely focused on program cost, schedule, and technology that are intended to place the program on a sound footing to achieve high levels of design and production maturity at key points in the acquisition process. Although more could be done, we believe that the most recent acquisition policy revision in concert with statutory requirements established after the issuance of our report essentially address the intent of our recommendation.

The department concurred with the recommendation and, in December 2008, issued its revised acquisition policy 5000.02. The regulation provides for certifications at Milestones A and B that the Milestone Decision Authority has reviewed the program and the business case. It also requires that the program has a cost estimate for executing the product's development and production plan along with an independent cost estimate. For those programs in technology development, DOD 5000.02 requires provision of two or more teams to produce prototypes up to or through Milestone B. It states that appropriate prototyping shall be employed to reduce technical risk, validate designs and cost estimates, evaluate manufacturing processes, and refine requirements. In addition, Title 10 2366(b) now requires a DOD certification to Congress as part of the Milestone B business case certification that reasonable cost and schedule estimates have been developed to execute the product development and production plan under the program. In 2009, the Congress passed the Weapons Systems Acquisition Reform Act of 2009 to help improve the organization and procedures for the acquisition of major weapon systems. A key element of this Act was the establishment of director of independent cost assessment who would study weapon program costs estimates to ensure that they are reasonable prior to committing to product development and production.

The Department of Defense concurred with the reccommendation. In December 2008, DOD revised its acquisition policy placing a greater emphasis that funding is available to fully execute the program's plan. Milestone A and B entrance requirements include a requirement to secure funding for the upcoming phase. The policy now states that transition into product development requires full funding and this funding should be programmed in anticipation of starting product development. In addition, Title 10 2366(b)now requires a DOD certification to Congress as part of the Milestone B business case certification that funding is available to execute the product development and production plan under the program, through the period covered by the future-years defense program.

In response to GAO's report, the Authorization Act fiscal year 2007 (Public Law 109-364) Section 853 addressed empowerment and accountability for program managers. The following were recommended: The Act requires the Secretary of Defense to establish a comprehensive strategy to enhance the role of DOD program managers. The strategy was to include enhanced training, mentoring, career path identification, incentives for recruitment and retention, improved resources and technical support, collection and dissemination of best practices and lessons learned, common templates and tools for data gathering and analysis for program management and oversight purposes, increased accountability for results, and enhanced monetary and non monetary incentives. The legislation directed the Secretary of Defense to revise guidance for major defense acquisition programs to address the qualifications, resources, responsibilities, tenure and accountability of program managers for the program development period (pre-milestone B or Key Decision Point B). In response DOD has adopted a policy on tenure for program managers and established a program management agreement as a tool for increasing the accountability of program managers.

In response to GAO's report, the Authorization Act fiscal year 2007 (Public Law 109-364) Section 853 addressed empowerment and accountability for program managers. The following were recommended: The Act requires the Secretary of Defense to establish a comprehensive strategy to enhance the role of DOD program managers. The strategy was to include enhanced training, mentoring, career path identification, incentives for recruitment and retention, improved resources and technical support, collection and dissemination of best practices and lessons learned, common templates and tools for data gathering and analysis for program management and oversight purposes, increased accountability for results, and enhanced monetary and non monetary incentives. The legislation directed the Secretary of Defense to revise guidance for major defense acquisition programs to address the qualifications, resources, responsibilities, tenure and accountability of program managers for the program development period (pre-milestone B or Key Decision Point B). In response DOD has adopted a policy on tenure for program managers and established a program management agreement as a tool for increasing the accountability of program managers.

In response to GAO's report, the Authorization Act fiscal year 2007 (Public Law 109-364) Section 853 addressed empowerment and accountability for program managers. The following were recommended: The Act requires the Secretary of Defense to establish a comprehensive strategy to enhance the role of DOD program managers. The strategy was to include enhanced training, mentoring, career path identification, incentives for recruitment and retention, improved resources and technical support, collection and dissemination of best practices and lessons learned, common templates and tools for data gathering and analysis for program management and oversight purposes, increased accountability for results, and enhanced monetary and non monetary incentives. The legislation directed the Secretary of Defense to revise guidance for major defense acquisition programs to address the qualifications, resources, responsibilities, tenure and accountability of program managers for the program development period (pre-milestone B or Key Decision Point B). In response DOD has adopted a policy on tenure for program managers and established a program management agreement as a tool for increasing the accountability of program managers.

In response to GAO's report, the Authorization Act fiscal year 2007 (Public Law 109-364) Section 853 addressed empowerment and accountability for program managers. The following were recommended: The Act requires the Secretary of Defense to establish a comprehensive strategy to enhance the role of DOD program managers. The strategy was to include enhanced training, mentoring, career path identification, incentives for recruitment and retention, improved resources and technical support, collection and dissemination of best practices and lessons learned, common templates and tools for data gathering and analysis for program management and oversight purposes, increased accountability for results, and enhanced monetary and non monetary incentives. The legislation directed the Secretary of Defense to revise guidance for major defense acquisition programs to address the qualifications, resources, responsibilities, tenure and accountability of program managers for the program development period (pre-milestone B or Key Decision Point B). In response DOD has adopted a policy on tenure for program managers and established a program management agreement as a tool for increasing the accountability of program managers.