Skip to main content

Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes

GAO-02-701 Published: Jul 15, 2002. Publicly Released: Jul 15, 2002.
Jump To:
Skip to Highlights

Highlights

This report examines how best practices offer improvements to the way the Department of Defense (DOD) develops new weapons systems, primarily the design and manufacturing aspects of the acquisition process. Knowledge about a product's design and producibility facilitates informed decisions about whether to significantly increase investments and reduces the risk of costly design changes later in the program. Leading commercial companies employ practices to capture design and manufacturing knowledge in time to make key decisions during product development. First, the companies kept the degree of the design challenge manageable before starting a new product development program by using an evolutionary approach. Second, the companies captured design and manufacturing knowledge before the two critical decision points in product development: when the design was demonstrated to be stable--the second knowledge point--and when the product was demonstrated to be producible at an affordable cost--the third knowledge point. DOD has made changes to its acquisition policy in an attempt to improve its framework for developing weapons systems, but the policy does not require the capture of design or manufacturing knowledge or sufficient criteria to enter the system demonstration and production phases. In addition, it does not require a decision review to enter the demonstration phase of product development.

Recommendations

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include completed subsystem and system design reviews.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed subsystem and system design reviews was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include ninety percent of drawings completed.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed engineering drawings was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include demonstration that design meets requirements--prototype or variant testing.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Criteria for transitioning from system integration to system demonstration includes demonstrating the design with prototypes.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include stakeholders' (cross functional design team that includes design engineers, manufacturing, key supplier) assurance that drawings are complete.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. The policy does not include a requirement that stakeholders have assurance that drawings are complete. However, the capture of design knowledge in the form of completed drawings was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include completed failure modes and effects analysis.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of completed failure modes and effects analysis was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include identification of key system characteristics.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of the identification of key system characteristics was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system integration to system demonstration should include identification of critical manufacturing processes.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is a design readiness review to assess design maturity mid-way during System Development and Demonstration similar to best practices. Successful completion of the review ends system integration and continues System Development and Demonstration into system demonstration. While not required, the capture of design knowledge in the form of the identification of critical manufacturing processes was cited as evidence that could be used for decisionmaking.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include set reliability targets and growth plan.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstration that product reliability is acceptable.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include demonstrated manufacturing processes.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes the demonstrated control of manufacturing processes.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include built production representative prototypes.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. The policy includes entrance criteria into production of acceptable performance in test and evaluation with prototypes systems.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include tested prototypes to achieve reliability goal.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstration that product reliability is acceptable. The policy also includes entrance criteria into production of acceptable performance in test and evaluation with prototype systems.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include tested prototypes to demonstrate product in operational environment.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process this includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes acceptable performance in development, test and evaluation, and an operational assessment.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include collected statistical process control data.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process. This includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstrated control of the manufacturing process and the collection of statistical process control data.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require the capture of specific knowledge to be used as exit criteria for decision making at two key points--when transitioning from system integration to system demonstration and from system demonstration into production. The knowledge to be captured when moving from system demonstration into production should include demonstration that critical processes are capable and in control.
Closed – Implemented
In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process. This includes reducing manufacturing risk and demonstrating producibility prior to full rate production. According to the policy, available knowledge to support approval into this phase includes demonstrated control of the manufacturing process and the collection of statistical process control data.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should require that the interim process review, currently identified in DOD's policy as that point in the process between system integration and system demonstration, be a mandatory decision review. At this point, the design should be demonstrated to be stable so that during the next phase of development attention can be focused on demonstrating manufacturing processes and product reliability. The program manager should have proof--based on the exit criteria for moving out of system integration in the above recommendation--that the product design is stable. The exit criteria should be demonstrated and verified by the program manager before the program can make the substantial investments needed to begin manufacturing production representative prototypes in the next phase of development--system demonstration. To ensure visibility of demonstrated exit criteria to decision makers, the criteria and the program's status in achieving them should be included in each program's Defense Acquisition Executive Summary and Selected Acquisition Reports. If the program does not meet the exit criteria, investments should be delayed until such time as the criteria are satisfied. To proceed without completing the required demonstrations should require approval by the decision authority.
Closed – Not Implemented
This is now over 4 years old and has not been implemented. We have made similar recommendations in two subsequent reports, GAO-04-53 and GAO-06-368, and will use those reports to follow up on this recommendation.
Department of Defense DOD should take steps to close the gaps between its current acquisition environment and best practices. To do this, it should ensure that its acquisition process captures specific design and manufacturing knowledge, includes decisions at key junctures in the development program, and provides incentives to use a knowledge-based process. Such changes are necessary to obtain greater predictability in weapon system programs' cost and schedule, to improve the quality of weapon systems once fielded, and to deliver new capability to the war fighter faster. More specifically, the Secretary of Defense should expand exit criteria for the Milestone C decision to include the knowledge to be captured during the system demonstration phase as one identified in recommendation one. This will require that the program office demonstrate that the critical manufacturing processes are under statistical control and that product reliability has been demonstrated before entering production of the new weapon system. These are the best practices and indicate that the product design is mature and the program is ready to begin production of units for operational use that will meet the cost, schedule, and quality goals of the program.
Closed – Not Implemented
In commenting on GAO's report, DOD agreed that demonstration of critical manufacturing processes and reliability must occur prior to rate production. However, they stated that it is unlikely that the criteria could be satisfied by their milestone C decision (limited production). In May 2003, DOD revised its acquisition policy for major weapons systems acquisitions. Included in this policy is the intent of a knowledge based acquisition process where managers should provide knowledge about key aspects of a system at key points in the acquisition process, which includes prior to full rate production. The available knowledge to support approval includes demonstrated control of manufacturing processes and acceptable reliability, and the collection of statistical control data. Best practices suggest that the knowledge should be captured by production commitment, which is milestone C in DOD's process. While policy states limited production should be limited this is frequently not the case in DOD.
Department of Defense To ensure that contracts support a knowledge-based process, the Department of Defense should structure its contracts for major weapon system acquisitions so that (a) the capture and use of knowledge described in recommendation one for beginning system demonstration is a basis for DOD's decision to invest in the manufacturing capability to build initial prototypes and (b) the capture and use of manufacturing and reliability knowledge discussed in recommendation one for moving from system demonstration to production is a basis for DOD's decision to invest in production.
Closed – Not Implemented
This is now over 4 years old and has not been implemented. We have made similar recommendations in two subsequent reports, GAO-04-53 and GAO-06-368, and will use those reports to follow up on this recommendation.

Full Report

Office of Public Affairs

Topics

Best practicesBest practices reviewsDefense capabilitiesDefense cost controlDefense procurementProcurement practicesWeapons research and developmentWeapons systemsSystems designEngineering