Defense Acquisitions:

DOD's Revised Policy Emphasizes Best Practices, but More Controls Are Needed

GAO-04-53: Published: Nov 10, 2003. Publicly Released: Nov 10, 2003.

Additional Materials:

Contact:

Michael J. Sullivan
(202) 512-4383
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Defense's (DOD) investment in new weapon systems is expected to exceed $1 trillion from fiscal years 2003 to 2009. To reduce the risk of cost and schedule overruns, DOD revamped its acquisition policy in May 2003. The policy provides detailed guidance on how weapon systems acquisitions should be managed. The Senate report accompanying the National Defense Authorization Act for Fiscal Year 2004 required GAO to determine whether DOD's policy supports knowledge-based, evolutionary acquisitions and whether the policy provides the necessary controls for DOD to ensure successful outcomes, such as meeting cost and schedule goals. The report also required GAO to assess whether the policy is responsive to certain requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003 concerning DOD's management of the acquisition process.

DOD's new policy supports knowledge-based, evolutionary acquisitions by adopting lessons learned from successful commercial companies. One of those lessons is a knowledge-based approach, which requires program managers to attain the right knowledge at critical junctures--also known as knowledge points--so they can make informed investment decisions throughout the acquisition process. The policy also embraces an evolutionary or phased development approach, which sets up a more manageable environment for attaining knowledge. The customer may not get the ultimate capability right away, but the product is available sooner and at a lower cost. Leading firms have used these approaches--which form the backbone of what GAO calls the best practices model--to determine whether a project can be accomplished with the time and money available and to reduce risks before moving a product to the next stage of development. By adopting best practices in the acquisition policy, DOD's leadership has taken a significant step forward. The next step is to provide the necessary controls to ensure a knowledge-based, evolutionary approach. Implementing the necessary controls at all three knowledge points along the acquisition process helps decision makers ensure a knowledge-based approach is followed. Without controls in the form of measurable criteria that decision makers must consider, DOD runs the risk of making decisions based on overly optimistic assumptions. Each successive knowledge point builds on the preceding one, and having clearly established controls helps decision makers gauge progress in meeting goals and ensuring successful outcomes. DOD was responsive to the requirements in the Bob Stump National Defense Authorization Act for Fiscal Year 2003. DOD's responses reflected the committee's specific concerns about the application of certain statutory and regulatory requirements to the new evolutionary acquisition process, for more guidance for implementing spiral development, and about technology readiness at program initiations.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In response to GAO's report, DOD stated that it agreed in principle with the advantages of using knowledge-based controls at key points in the acquisition process to assess risk and ensure readiness to proceed into the next phase of the acquisition process. Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). We believe that the these certification requirements establish controls that meet the intent of our recommendation. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for program launch (milestone B) should ensure the capture of knowledge about the cost and schedule estimates based on knowledge from a preliminary design using systems engineering tools.

    Agency Affected: Department of Defense

  2. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB)to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will be able to complete 90 percent of their engineering drawings by the design readiness review. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the completion of 90 percent of engineering drawings.

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will capture knowledge from sub-system and system design reviews before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the completion of subsystem and system design reviews.

    Agency Affected: Department of Defense

  4. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will be able to obtain agreement from all stakeholders that drawings are complete and the design is producible before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the agreement from all stakeholders that drawings are complete and the design is producible.

    Agency Affected: Department of Defense

  5. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will capture knowledge about failure modes and effects before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the completion of failure modes and effects analysis.

    Agency Affected: Department of Defense

  6. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will capture knowledge about key system characteristics before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the identification of key system characteristics.

    Agency Affected: Department of Defense

  7. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will identify critical manufacturing processes before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the identification of critical manufacturing processes.

    Agency Affected: Department of Defense

  8. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that programs will capture knowledge system reliability before transitioning into the system demonstration phase. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for transitioning from system integration to system demonstration (design readiness review) should ensure the capture of knowledge about the reliability targets and a reliability growth plan based on demonstrated reliability rates of components and subsystems.

    Agency Affected: Department of Defense

  9. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that DOD will be in a position to complete development of production representative prototypes before committing to production - particularly given the department's current prototyping initiative. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for the production commitment (milestone C) should ensure the capture of knowledge about the completion of production representative prototypes.

    Agency Affected: Department of Defense

  10. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that DOD will be in a position to complete development of production representative prototypes, demonstrate those prototypes in an operational environment, and capture knowledge about system reliability prior to committing to production - particularly given the department's current prototyping initiative. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for the production commitment (milestone C) should ensure the capture of knowledge about the availability of production representative prototypes to achieve reliability goal and demonstrate the product in an operational environment.

    Agency Affected: Department of Defense

  11. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L) issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that DOD will capture knowledge about manufacturing processes before committing to production - particularly given the stated intent of department's current prototyping initiative. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for the production commitment (milestone C) should ensure the capture of knowledge about the collection of statistical process control data.

    Agency Affected: Department of Defense

  12. Status: Closed - Implemented

    Comments: DOD has several acquisition reform initiatives and pilot programs that we believe address the intent of our recommendation. In July 2007, the Under Secretary of Acquisition, Technology, and Logistics issued a policy memorandum directing the military departments to establish Configuration Steering Boards (CSB) to control changes in program requirements and technical content after milestone B. In September 2007, the USD (AT&L)issued another policy memorandum that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. In addition to these two policy memos, DOD is also in the process of establishing metrics that would be used in a knowledge-based acquisition system. In a briefing presented to OMB, DOD identified process metrics and controls, and also identified the information these metrics are expected to produce. Given our discussions with pertinent DOD officials, it is highly likely that these metrics will be implemented into the upcoming revision of DOD's acquisition policy, the DOD 5000 series. We believe that these new policies address the intent of our recommendation by increasing the likelihood that DOD will capture knowledge about manufacturing processes before committing to production - particularly given the stated intent of department's current prototyping initiative. Therefore, we are closing this recommendation as implemented.

    Recommendation: DOD must design and implement necessary controls to ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require additional controls for capturing knowledge at three key points--program launch, design readiness review for transitioning from system integration to system demonstration, and production commitment. The additional controls for the production commitment (milestone C) should ensure the capture of knowledge about the demonstration that critical manufacturing processes are capable and in statistical control.

    Agency Affected: Department of Defense

  13. Status: Closed - Implemented

    Comments: The Department of Defense agreed that they should be accountable for program decisions and appropriately record the considerations in moving from one stage of development to the next. Milestone decision authorities are to employ the decision memorandums to document decisions and their rationale. Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). We believe that the these certification requirements establish controls that meet the intent of our recommendation. Therefore, we are closing this recommendation as implemented.

    Recommendation: Because knowledge about technology, design, and manufacturing at critical junctures can lower DOD's investment risk, decisions that do not satisfy knowledge-based criteria should be visible and justified. Therefore, the Secretary of Defense should document the rationale for any decision to move a program to the next stage of development without meeting the knowledge-based criteria, including those listed in the first recommendation. The responsible milestone decision authority should justify the decision in the program's acquisition decision memorandum and in a report to Congress.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Sep 30, 2014

Sep 25, 2014

Sep 23, 2014

Sep 19, 2014

Sep 18, 2014

Sep 10, 2014

Sep 9, 2014

Sep 8, 2014

Looking for more? Browse all our products here