GAO-09-543T, Defense Acquisitions: Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines


This is the accessible text file for GAO report number GAO-09-543T 
entitled 'Defense Acquisitions: Measuring the Value of DOD's Weapon 
Programs Requires Starting with Realistic Baselines' which was released 
on April 1, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Panel on Defense Acquisition Reform, Committee on Armed 
Services, House of Representatives: 

United States Government Accountability Office: 
GAO: 

For Release on Delivery: 
Expected at 7:30 a.m. EDT:
Wednesday April 1, 2009: 

Defense Acquisitions: 

Measuring the Value of DOD's Weapon Programs Requires Starting with 
Realistic Baselines: 

Statement of Michael J. Sullivan: 
Director Acquisition and Sourcing Management: 

GAO-09-543T: 

GAO Highlights: 

Highlights of GAO-09-543T, a testimony before the House Committee on 
Armed Services Panel on Defense Acquisition Reform. 

Why GAO Did This Study: 

In 2008, the cumulative cost growth in the Department of Defense’s 
portfolio of 96 major defense acquisition programs was $296 billion and 
the average delay in delivering promised capabilities to the warfighter 
was 22 months. These poor outcomes mean that other critical defense and 
national priorities may go unfunded and that warfighters may go without 
the equipment they need to counter the changing threats that they face. 

GAO has examined extensively the issues in DOD’s weapon system programs 
from the perspective of best practices in product development, and 
believes that significant improvements are possible. Because the 
ability to measure knowledge, processes, and outcomes is critical to 
achieving these improvements, GAO has used metrics to review the 
management and health of these programs from within the framework of 
best practices. 

This testimony discusses: 1) “knowledge metrics,” used to determine how 
well programs manage technology, design, and manufacturing risks; 2) 
outcome metrics--concerning cost, schedule, and capability--that serve 
as “health indicators” of how well programs are being executed in terms 
of predicted outcomes; and 3) the prerequisites that GAO believes must 
be met in order for a program’s plans and goals to be realistic. 

What GAO Found: 

GAO employs a set of knowledge metrics to determine whether programs 
have attained the right knowledge at critical points over the course of 
a weapon system acquisition, and facilitate the identification of 
potential problems that could lead to cost, schedule, or performance 
shortfalls. In essence, knowledge supplants risk over time. Key 
knowledge points and metrics include 1) achieving a high level of 
technology maturity at the start of program development, 2) reaching 
design stability at the system-level critical design review, and 3) 
demonstrating that critical manufacturing processes are in control 
before starting production. By applying these metrics to selected 
programs in DOD’s 2008 portfolio of major defense acquisitions, GAO 
found that most programs have started system development without mature 
technologies and moved into system demonstration with low levels of 
design stability. GAO has determined that programs with immature 
technologies and unstable designs have experienced significant cost and 
schedule growth. 

Program outcome metrics—quantitative measures of cost, schedule, and 
performance over time—provide useful indicators of the health of 
acquisition programs and whether they are meeting their intended goals. 
When assessed regularly for changes and the reasons that cause changes, 
these indicators can be valuable tools for improving insight into and 
oversight of individual programs as well as DOD’s total portfolio of 
major defense acquisitions. The collective performance of the programs 
in DOD’s portfolio is a key indicator of how well the acquisition 
system generates the return on investment that it promises to the 
warfighter, Congress and taxpayers. GAO recently reported that outcome 
metrics for DOD’s 2008 major defense acquisition portfolio show 
worsening performance when compared to the department’s 2003 portfolio. 
For example, total acquisition costs for programs in the 2008 portfolio 
increased 25 percent from first estimates compared to a 19-percent 
increase for programs in the 2003 portfolio. DOD is working with GAO 
and the Office of Management and Budget to develop a comprehensive set 
of outcome metrics to better assess its portfolio of programs. 

While knowledge and outcome metrics provide valuable information about 
the potential problems and health of programs, they are of limited 
value if DOD does not do a better job ensuring acquisitions begin with 
realistic plans and baselines prior to development start. GAO believes 
there is a clear set of prerequisites that must be met by each 
program’s acquisition strategy before a measurement of the program’s 
health will be of real value. These prerequisites include: 1) 
establishing an evolutionary, knowledge-based business case for each 
acquisition; 2) separating technology development from product 
development; 3) limiting time and requirements for product development 
to manageable levels; 4) employing systems engineering early on in the 
process to arrive at realistic cost and schedule estimates; 5) 
committing to fully funding a program once it is approved; and 6) 
setting priorities from the top to ensure that candidate programs are 
truly needed and have a solid plan for delivery. 

View [hyperlink, http://www.gao.gov/products/GAO-09-543T] or key 
components. For more information, contact Michael J. Sullivan at (202) 
512-4841 or sullivanm@gao.gov. 

[End of section] 

Mr. Chairman and Members of the Committee: 

I am pleased to be here this morning to discuss how best to measure and 
determine whether DOD's acquisition system is providing value to the 
warfighter. Earlier this week, we reported that the cumulative cost 
growth in DOD's portfolio of 96 major defense acquisition programs was 
$296 billion and the average delay in delivering promised capabilities 
to the warfighter was 22 months. These outcomes mean that other 
critical defense and national priorities go unfunded and warfighters go 
without the equipment they need to counter ever changing threats that 
they face. This condition is unacceptable. We believe that significant 
improvement in the acquisition of weapon systems is possible and that 
the ability to measure knowledge, processes, and outcomes is critical 
to achieving that improvement. It is important to note that not one 
single metric or set of metrics is enough to monitor acquisitions and 
gain efficiencies. Today, we would like to break our discussion about 
how to measure the department's acquisitions into 3 basic sections: 

* First, we would like to present a set of metrics that we refer to as 
"knowledge metrics" and use to determine how well acquisition programs 
are managing and retiring predictable technology, design, and 
manufacturing risks and gaining knowledge. These metrics are valuable 
because they can predict problems and identify causes. 

* Second, we would like to discuss a set of outcome measures-- 
concerning cost, schedule, and capability--that serve as health 
indicators. These indicators measure how well programs are being 
executed and achieving predicted outcomes in terms of meeting original 
baselines for cost, schedule, and performance. These metrics have 
intrinsic value as simple measurements, just as a thermometer can warn 
a parent that a child has a fever. 

* Third, there are certain indicators that we look for--based on the 
work we have done examining best practices for product development-- 
that are, perhaps, more important than these knowledge and health 
metrics because they determine from the outset how realistic the 
acquisition plans and strategies of programs are. For the sake of 
today's discussion, we will refer to them as "prerequisite indicators." 
These prerequisites are most important because we question the value of 
ANY metric when measuring from an unrealistic baseline. 

We know that the knowledge and program health metrics we use to measure 
programs' progress and outcomes are valuable when used in realistic, 
market-driven product development environments. We also know that ALL 
of these metrics are important indicators for decision makers. Our 
extensive body of work examining world-class enterprises and the way 
they operate has validated their value for programs that must deliver a 
new product to market at a certain time and within a certain investment 
cost or suffer significant consequences. These metrics work because 
they are measuring realistic plans and goals that are supported by 
doable requirements, accurate cost and schedule estimates, and stable 
funding. The company developing the products suffers dire consequences, 
such as loss of market share, if these programs do not succeed. 

This statement draws from our extensive body of work on DOD's 
acquisition of weapon systems. A list of our key products is provided 
at the end of this statement. This work was conducted in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 

"Knowledge" Metrics Identify Potential Problems and Their Likely 
Causes: 

We have conducted a body of work that examines weapon acquisition 
issues from a perspective that draws upon lessons learned from best 
practices in product development. Collectively, these practices 
comprise a process that is anchored in knowledge. Achieving the right 
knowledge at the right time enables leadership to make informed 
decisions about when and how best to move into various expensive 
acquisition phases. In essence, knowledge supplants risk over time. 
This building of knowledge consists of gathering information about 
technology, design, and manufacturing at three critical points over the 
course of a weapon system program (Figure 1). We have developed 
valuable "knowledge metrics" that measure this knowledge build and 
allow us to identify potential problems that could lead to cost, 
schedule, or performance shortfalls and their likely causes. The 
metrics can be described as: 

* Knowledge Point 1, evidenced by the balance between a product's 
required capabilities and the resources available to meet them. Focus 
should be on understanding technological and design implications and 
achieving a high level of technology maturity at the start of system 
development. This means that the critical technologies needed to meet 
essential product requirements must be demonstrated to work in their 
intended environment. The technology readiness level for each critical 
technology is the metric we use to measure technology maturity. 
[Footnote 1] 

* Knowledge point 2, evidenced by the development of engineering 
prototypes and the completion of engineering drawings for an integrated 
product at the system design review. This metric provides tangible 
evidence that the product's design is stable, meaning it has a high 
probability of meeting customer requirements, as well as cost, 
schedule, and reliability targets. A best practice is to achieve design 
stability at the system-level critical design review, usually held 
midway through development. Completion of at least 90 percent of 
engineering drawings is the metric we use to measure design stability. 

* Knowledge point 3, evidenced by the demonstration that critical 
manufacturing processes are in control--that is, they are repeatable, 
sustainable, and capable of consistently producing parts within the 
product's quality tolerances and standards--at the start of production. 
One hundred percent of critical manufacturing processes in control is 
the metric we use to evaluate manufacturing maturity. 

Figure 1: Knowledge-Based Acquisition Process: 

[Refer to PDF for image: illustration] 

Material development start: 
Technology Development (A); 
PDR: Knowledge point 1: Metric: technology readiness levels at 7 or 
higher (B); 
Product Development: 
- Integration; 
- CDR: Knowledge point 2: Metric: 90 percent of engineering drawings 
complete; 
- Demonstration; 
Production: (C): Knowledge point 3: Metric: 100 percent of critical 
manufacturing processes in control. 

Source: GAO analysis of commercial best practices. 

[End of figure] 

Each of these metrics gauges the point when the requisite level of 
knowledge has been attained for a product in relation to where that 
product is in its development. World-class firms we have visited work 
hard to establish metrics such as these and their decision makers are 
required to consider them before deciding to advance a program to the 
next level. Theses types of metrics also help decision makers gauge 
progress in meeting cost, schedule, and performance goals and ensure 
that managers will (1) conduct activities to capture relevant product 
development knowledge, (2) provide evidence that this knowledge has 
been captured, and (3) hold decision reviews to determine that 
appropriate knowledge has been captured before moving the product to 
the next phase. The result is a product development process that 
provides critical measurements of knowledge, holds decision makers 
accountable, and delivers the expected results in a predictable manner. 
Attachment 1 to this statement provides a detailed list of activities 
that would provide program managers with the requisite technology, 
design, and manufacturing knowledge at key points in time during 
development. 

We have used these metrics to identify problems on major weapon system 
acquisition programs and have found a strong correlation between each 
of them and cost and schedule outcomes. For example, for 47 weapon 
programs in DOD's 2008 portfolio of major defense acquisitions, we 
assessed the knowledge attained at key decision points in the 
acquisition process and found the following:[Footnote 2] 

* Most programs have started system development without mature 
technologies. Only 4 of the 36 programs that provided data on technical 
maturity at development start did so with fully mature critical 
technologies. Further, only 14 of 39 programs that provided data have 
or plan to have demonstrated all of their technologies in a realistic 
environment prior to system-level critical design review, at which 
point the system's design should be stable. The 5 newer programs--those 
initiated since 2003[Footnote 3]--have higher levels of technology 
maturity, with all 5 programs demonstrating their technologies in a 
relevant environment prior to development start, in accordance with DOD 
and statutory criteria. However only 1of these programs met the best 
practice standard of demonstrating critical technologies in an 
operational environment. Last year, we determined that programs with 
immature technologies at the start of system development experienced 44 
percent higher cost growth than programs that began with mature 
technologies. 

* Programs that have held design reviews in recent years reported 
higher levels of design knowledge. However, designs, on average, are 
still far from stable. For the 24 programs in our assessment that held 
a critical design review since 2003, the average percentage of total 
expected design drawings releasable at this review was 65 percent, 
compared to a best practice standard of 90 percent. We have found that 
programs moving forward into system demonstration with low levels of 
design stability are more likely than other programs to encounter 
costly design changes and parts shortages that, in turn, cause labor 
inefficiencies, schedule delays, and quality problems. 

Attachment 3 represents our notional depiction of the problems and 
outcomes that can typically be expected when these knowledge metrics 
are followed versus when they are not. Generally speaking, programs 
that move forward without retiring technology, design, and 
manufacturing risk at appropriate junctures will encounter a cascade of 
problems beginning with design changes and continuing with parts 
shortages, changes to manufacturing processes, labor inefficiencies, 
and quality problems. All of these problems delay programs and add to 
their development costs. We have found, for example, that a significant 
portion--about 70 percent--of the total development cost growth in 
programs typically occurs after the design review. 

Outcome Metrics Provide Insight into the Health and Performance of 
Individual Weapon System Programs and DOD's Total Portfolio: 

Program outcome metrics--quantitative measures of cost, schedule, and 
performance, and changes in these factors over time--provide useful 
indicators of the health of acquisition programs and facilitate 
analyses of how well programs are meeting cost, schedule, and 
performance goals. When assessed regularly for changes and the reasons 
that cause changes, such indicators can be valuable tools for improving 
insight and oversight of individual programs as well as DOD's total 
portfolio of major defense acquisitions. Over the years we have 
reported cost, schedule and performance data--good and bad--on numerous 
weapon systems. Our work continues to identify systemic and program- 
specific causes for cost, schedule, and performance problems and has 
led us to designate, since 1990, DOD's management of major weapon 
system acquisitions as a high risk area. 

To improve acquisition performance and address the factors that keep 
weapon acquisitions on the high risk list, DOD is working with us and 
the Office of Management and Budget to develop a comprehensive set of 
outcome metrics to provide better, comprehensive, and consistent 
measures of program cost and schedule performance. Last year, this 
cooperative effort resulted in agreement to track trends and changes in 
programs from their original baselines, from 5 years ago, and from the 
previous year, for the following data points: 

* Development cost; 

* Procurement cost; 

* Total program cost; 

* Quantities to be procured; 

* Procurement unit costs; 

* Total program unit costs; 

* Cycle time from Milestone B to Initial Operational Capability: 

DOD initiated a pilot study of 7 major defense programs to assess the 
adequacy of the proposed metrics and their value in analyzing 
performance, and the results proved promising. DOD approved the outcome 
metrics and intends to collect and report such data on an annual basis. 
Efforts to develop similar metrics on schedule performance continue. 

We believe that the metrics DOD plans to use are valuable for providing 
insight into the performance of weapon system programs. We have used 
similar metrics for many years in assessing programs. For example, we 
recently reported that ten of DOD's largest acquisition programs, 
commanding about half the overall acquisition dollars in the 
department's 2008 portfolio of major programs, have experienced 
significant cost growth and have seen quantities reduced by almost a 
third (see table 1). The two largest programs--the Joint Strike Fighter 
and the Future Combat System--represent significant cost risk moving 
forward and will dominate the portfolio for years. Since these programs 
consume such a large portion of the funding that DOD spends on research 
and development and procurement, their performance also affects other 
major weapon acquisitions, smaller acquisition programs, and DOD's 
ability to fund and acquire other supplies and equipment as well. 

Table 1: Changes in Costs and Quantities for Ten of the Highest Cost 
Acquisition Programs: 

Fiscal year 2009 dollars in millions: 

Program: Joint Strike Fighter; 
Total cost: First full estimate: $206,410; 
Total cost: Current estimate: $244,772; 
Total quantity: First full estimate: 2,866; 
Total quantity: Current estimate: 2,456; 
Acquisition unit cost: Percentage change: 38. 

Program: Future Combat System; 
Total cost: First full estimate: $89,776; 
Total cost: Current estimate: $129,731; 
Total quantity: First full estimate: 15; 
Total quantity: Current estimate: 15; 
Acquisition unit cost: Percentage change: 45. 

Program: Virginia Class Submarine; 
Total cost: First full estimate: $58,378; 
Total cost: Current estimate: $81,556; 
Total quantity: First full estimate: 30; 
Total quantity: Current estimate: 30; 
Acquisition unit cost: Percentage change: 40. 

Program: F-22A Raptor; 
Total cost: First full estimate: $88,134; 
Total cost: Current estimate: $73,723; 
Total quantity: First full estimate: 648; 
Total quantity: Current estimate: 184; 
Acquisition unit cost: Percentage change: 195. 

Program: C-17 Globemaster III; 
Total cost: First full estimate: $51,733; 
Total cost: Current estimate: $73,571; 
Total quantity: First full estimate: 210; 
Total quantity: Current estimate: 190; 
Acquisition unit cost: Percentage change: 57. 

Program: V-22 Joint Services Advanced Vertical Lift Aircraft; 
Total cost: First full estimate: $38,726; 
Total cost: Current estimate: $55,544; 
Total quantity: First full estimate: 913; 
Total quantity: Current estimate: 458; 
Acquisition unit cost: Percentage change: 186. 

Program: F/A-18E/F Super Hornet; 
Total cost: First full estimate: $78,925; 
Total cost: Current estimate: $51,787; 
Total quantity: First full estimate: 1,000; 
Total quantity: Current estimate: 493; 
Acquisition unit cost: Percentage change: 33. 

Program: Trident II Missile; 
Total cost: First full estimate: $49,939; 
Total cost: Current estimate: $49,614; 
Total quantity: First full estimate: 845; 
Total quantity: Current estimate: 561; 
Acquisition unit cost: Percentage change: 50. 

Program: CVN 21 Nuclear Aircraft Class Carrier; 
Total cost: First full estimate: $34,360; 
Total cost: Current estimate: $29,914; 
Total quantity: First full estimate: 3; 
Total quantity: Current estimate: 3; 
Acquisition unit cost: Percentage change: -13. 

Program: P-8A Poseidon Multi-mission Maritime Aircraft; 
Total cost: First full estimate: $29,974; 
Total cost: Current estimate: $29,622; 
Total quantity: First full estimate: 115; 
Total quantity: Current estimate: 113; 
Acquisition unit cost: Percentage change: 1. 

Source: GAO analysis of DOD data. 

[End of table] 

While program outcome metrics are good measures of individual program 
performance, the collective performance of DOD's portfolio of major 
defense acquisition programs is a key indicator of how well the 
department's acquisition system generates the return on investment it 
promises to the warfighter, Congress, and the taxpayer. Portfolio 
metrics also provide senior leaders and Congress with a snapshot of the 
cumulative impact of current investment decisions and poor program 
performance on future budgets. In our annual assessment of selected 
weapon programs, we analyzed the performance of DOD programs at the 
portfolio level by comparing programs' initial cost, schedule, and 
quantity estimates to their current estimates, based on data obtained 
from the Selected Acquisition Reports. This year's cumulative results, 
reported earlier this week, [Footnote 4] are shown in table 2. 

Table 2: Analysis of DOD Major Defense Acquisition Program 
Portfolios[A]: 

Fiscal year 2009 dollars. 

Portfolio size: Number of programs; 
Fiscal Year 2003: 77; 
Fiscal Year 2007: 95; 
Fiscal Year 2008: 96. 

Portfolio size: Total planned commitments; 
Fiscal Year 2003: $1.2 trillion; 
Fiscal Year 2007: $1.6 trillion; 
Fiscal Year 2008: $1.6 trillion. 

Portfolio size: Commitments outstanding; 
Fiscal Year 2003: $724.2 billion; 
Fiscal Year 2007: $875.2 billion; 
Fiscal Year 2008: $786.3 billion. 

Portfolio indicators: Change to total RDT&E costs from first estimate; 
Fiscal Year 2003: 37 percent; 
Fiscal Year 2007: 40 percent; 
Fiscal Year 2008: 42 percent. 

Portfolio indicators: Change to total acquisition cost from first 
estimate; 
Fiscal Year 2003: 19 percent; 
Fiscal Year 2007: 26 percent; 
Fiscal Year 2008: 25 percent. 

Portfolio indicators: Total acquisition cost growth; 
Fiscal Year 2003: $183 billion; 
Fiscal Year 2007: $301.3 billion[B]; 
Fiscal Year 2008: $296.4 billion. 

Portfolio indicators: Share of programs with 25 percent increase in 
program acquisition unit cost growth; 
Fiscal Year 2003: 41 percent; 
Fiscal Year 2007: 44 percent; 
Fiscal Year 2008: 42 percent. 

Portfolio indicators: Average schedule delay in delivering initial 
capabilities; 
Fiscal Year 2003: 18 months; 
Fiscal Year 2007: 21 months; 
Fiscal Year 2008: 22 months. 

Source: GAO analysis of DOD data. 

[A] Data were obtained from DOD's Selected Acquisition Reports (dated 
December 2002, 2006, and 2007). In a few cases data were obtained 
directly from program offices. The number of programs reflects the 
programs with Selected Acquisition Reports; however, in our analysis we 
have broken a few Selected Acquisition Reports programs into smaller 
elements or programs. Not all programs had comparable cost and schedule 
data and these programs were excluded from the analysis where 
appropriate. Portfolio performance data do not include costs of 
developing Missile Defense Agency elements or the DIMHRS program. 

[B] The acquisition cost growth for the 2007 portfolio was $295 billion 
in 2008 constant dollars. 

[End of table] 

Analyzing the data and comparing metrics from different time periods 
provides unique insights into the relative health of the portfolio and 
trends. From 2003 to 2008, the number of programs in DOD's major 
defense acquisition portfolio has grown from 77 to 96. Total costs for 
these programs now total $1.6 trillion with almost one-half of this 
amount still to be spent. Outcome metrics for 2008 show worsening 
performance in all categories compared to the 2003 portfolio and mixed 
performance--some better, some worse--compared to the 2007 data. While 
DOD is committing substantially more investment dollars to developing 
and procuring new weapon systems, the total acquisition costs for the 
2008 portfolio has grown by $296 billion over initial estimates and the 
average schedule delay in delivering capabilities to the warfighter 
averages 22 months. Implications for the future are obvious. Continued 
cost growth reduces DOD's buying power and results in less funding 
being available for other DOD priorities and programs. As program costs 
increase, DOD must request more funding to cover overruns, make trade- 
offs with existing programs, delay the start of new programs, take 
funds from other accounts, or reduce procurement quantities. Continued 
failure to deliver weapon systems on time delays providing critical 
capabilities to the warfighter and results in operating costly legacy 
systems longer than expected, finding alternatives to fill capability 
gaps, or going completely without the capability. 

Key Prerequisites Need to Be Met to Improve the Value of Metrics and 
Achieve Better Acquisition Program Outcomes: 

While the metrics discussed above can provide valuable knowledge about 
potential problems and additional information on the health of DOD's 
acquisition programs, metrics alone may not be sufficient if the 
department does not do a better job ensuring that acquisitions begin 
with realistic plans and baseline estimates for cost and schedules 
prior to development start. We believe there is a clear set of 
prerequisites that must be a part of any acquisition strategy before 
any measurement of the acquisition's health can be valuable. Otherwise, 
metrics measured against unrealistic estimates will do no good. These 
key prerequisites for obtaining realistic baselines include: 

* Establishing a clear, knowledge-based, evolutionary business case for 
the product. This business case must: validate that a need exists; 
determine that resources are available to develop a product that will 
meet the need; determine that the product developer has a knowledge- 
based plan and strategy to deliver the product; establish reasonable 
estimates for cost, delivery time and quantities; and ensure available 
funding for the product. All of these elements of the business case 
should also be agreed upon by major stakeholders across the 
requirements, funding, acquisition, and warfighting communities. 

* Separating technology development activities from product development 
activities. The process of developing technology culminates in 
discovery--the gathering of knowledge--and must, by its nature, allow 
room for unexpected results and delays. Leading firms do not ask their 
product managers to develop technology because they have learned the 
hard way that invention cannot be scheduled. When immature technologies 
are brought onto the critical path of product development programs too 
early, they often cause long delays in an environment where large 
workforces must be employed, complex tools, plants, and facilities must 
be operated, long and expensive supplier networks must be paid, and the 
product itself must sometimes be redesigned once the final form of the 
technologies is known. Successful programs give responsibility for 
maturing technologies to science and technology organizations, rather 
than the program or product development managers, because the science 
and technology environment is less costly. We have recommended in the 
past that DOD's risks should be taken in the science and technology 
arena and that more funding should be made available to this process to 
do so. 

* Limiting time and requirements for product development to manageable 
levels. Product developers should strive to deliver the best available 
capabilities within realistic timeframes and should expect to continue 
to develop new capabilities when they are technologically feasible. By 
limiting product development cycle times to 6 years or less, DOD could 
assimilate new technologies into weapon systems more frequently, 
accelerate delivery of new technology to the warfighter, hold program 
managers accountable, and make more frequent and predictable work in 
production, where contractors and the industrial base can profit by 
being efficient. Too many major acquisitions currently take the 
opposite approach by seeking to deliver a revolutionary "big bang" 
capability in one step. This means that programs are more risky, 
delivery takes as long as 15 years in some cases, and costs grow at 
exponential rates from the original baseline due to the risky nature of 
the acquisition strategy. We point to the private sector and some past 
defense acquisitions, such as the F-16 program, as models for this 
practice. 

* Employing early systems engineering discipline in order to develop 
realistic cost and schedule estimates prior to development start. Early 
systems engineering provides the knowledge a product developer needs to 
identify and resolve performance and resource gaps before product 
development begins either by reducing requirements, deferring them to 
the future, or increasing the estimated cost for the weapon system's 
development. Requirements that are too risky given the state of 
technology and design should not be allowed into this expensive 
environment. 

* Making a commitment to fully fund programs once they are approved. 
This would require the department to ensure that it does not have too 
many programs underway given the amount of available resources. 

* Setting priorities from the top to ensure that candidate programs are 
truly needed and have a solid plan for delivery. DOD will continue to 
experience poor acquisition outcomes until it begins making choices 
that reflect the most important needs of the joint warfighter and match 
requirements with available resources. The urge to accept all candidate 
programs and to go for the "big bang" capability without the knowledge 
to achieve it should be resisted. Only the best candidates--defined in 
terms of priorities, resource availability, and executability--should 
be approved. 

There is no doubt that the current state of the department's 
acquisition process is too expensive for the taxpayer and not timely 
enough for the warfighter. The following illustration reinforces this 
point. 

Figure 2: Cost Remaining Versus Annual Appropriations for Major Defense 
Acquisitions: 

[Refer to PDF for image: multiple line graph] 

Fiscal year: 1992; 
Annual RDTE and Procurement Appropriations: $100 billion; 
Costs Remaining for Major Defense Acquisitions: $324 billion. 

Fiscal year: 1993; 
Annual RDTE and Procurement Appropriations: $91 billion; 
Costs Remaining for Major Defense Acquisitions: $271 billion. 

Fiscal year: 1994; 
Annual RDTE and Procurement Appropriations: $79 billion; 
Costs Remaining for Major Defense Acquisitions: $253 billion. 

Fiscal year: 1995; 
Annual RDTE and Procurement Appropriations: $78 billion; 
Costs Remaining for Major Defense Acquisitions: $341 billion. 

Fiscal year: 1996; 
Annual RDTE and Procurement Appropriations: $78 billion; 
Costs Remaining for Major Defense Acquisitions: $345 billion. 

Fiscal year: 1997; 
Annual RDTE and Procurement Appropriations: $79 billion; 
Costs Remaining for Major Defense Acquisitions: $331 billion. 

Fiscal year: 1998; 
Annual RDTE and Procurement Appropriations: $82 billion; 
Costs Remaining for Major Defense Acquisitions: $280 billion. 

Fiscal year: 1999; 
Annual RDTE and Procurement Appropriations: $89 billion; 
Costs Remaining for Major Defense Acquisitions: $233 billion. 

Fiscal year: 2000; 
Annual RDTE and Procurement Appropriations: $94 billion; 
Costs Remaining for Major Defense Acquisitions: $309 billion. 

Fiscal year: 2001; 
Annual RDTE and Procurement Appropriations: $104 billion; 
Costs Remaining for Major Defense Acquisitions: $234 billion. 

Fiscal year: 2002; 
Annual RDTE and Procurement Appropriations: $111 billion; 
Costs Remaining for Major Defense Acquisitions: $586 billion. 

Fiscal year: 2003; 
Annual RDTE and Procurement Appropriations: $137 billion; 
Costs Remaining for Major Defense Acquisitions: $579 billion. 

Fiscal year: 2004; 
Annual RDTE and Procurement Appropriations: $148 billion; 
Costs Remaining for Major Defense Acquisitions: $722 billion. 

Fiscal year: 2005; 
Annual RDTE and Procurement Appropriations: $165 billion; 
Costs Remaining for Major Defense Acquisitions: $745 billion. 

Fiscal year: 2006; 
Annual RDTE and Procurement Appropriations: $157 billion; 
Costs Remaining for Major Defense Acquisitions: $841 billion. 

Fiscal year: 2007; 
Annual RDTE and Procurement Appropriations: $160 billion; 
Costs Remaining for Major Defense Acquisitions: $767 billion. 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

This figure depicts an investment strategy for major weapon systems 
that continues to increase the costs to develop our existing weapons 
well into the future while the funding available to retire those costs 
appears capped at a very low level. While costs continue to rise as the 
result of more and more risky programs being added to the portfolio, 
our ability to allocate funds for these costs appears to be, at best, 
capped at very low percentages of the total cost. We could measure the 
risk of these acquisitions much better than we have in the past if we 
set the appropriate prerequisites for their initiation, measure the 
knowledge that must be in place at various points, and continue to 
monitor their health in terms of cost and schedule. 

Concluding Remarks: 

Measuring the performance of weapon system programs both individually 
and collectively is critical for determining whether the warfighter and 
the taxpayer are receiving the promised return on investment. No single 
metric, however, can capture the whole picture of how well programs are 
performing. It is important to look at knowledge and outcome metrics. 
Knowledge metrics provide key information for determining whether 
programs have the requisite knowledge to move from one phase of 
development to the next and are at risk of cost and schedule overruns. 
Outcomes metrics are also needed to provide temperature checks on the 
health and status of individual programs and the portfolio of programs 
as a whole. These metrics are vital for informing program decision 
making and helping to manage programs. 

Metrics by themselves do not solve problematic acquisitions. 
Ultimately, DOD still needs to do a better job planning and executing 
programs to achieve better outcomes. Critical to achieving successful 
outcomes is establishing knowledge-based, realistic program baselines. 
Without realistic baselines, there is no foundation for accurately 
measuring the knowledge and health of programs. Over the past several 
years, our work has highlighted a number of underlying causes for why 
DOD does not effectively manage the acquisition of weapon system 
programs. DOD recently revised its acquisition policy to provide a 
better foundation for developing weapon systems, however, reform will 
not be achieved without fundamental changes to the overall acquisition 
culture and environment that exists in DOD. I would be pleased to 
discuss these causes and issues with the Committee at a future time. 

Mr. Chairman, this concludes my prepared statement. I would be happy to 
answer any questions you may have at this time. 

Contacts and Acknowledgements: 

For further information about this statement, please contact Michael J. 
Sullivan (202) 512-4841 or sullivanm@gao.gov. Contact points for our 
Office of Congressional Relations and Public Affairs may be found on 
the last page of this statement. Individuals who made key contributions 
to this statement include Cheryl Andrew, Ridge Bowman, Bruce Fairbairn, 
Susan Neill, John Oppenheim, and Ron Schwenn. 

[End of section] 

Attachment 1: Knowledge-Based Activities: 

Knowledge Point 1: Start of product development activities: Best 
practice metric: Technology readiness level 7 (indicating technologies 
work in an operational environment): 

* Demonstrate technologies to high readiness levels. 

* Ensure that requirements for the product increment are informed by 
preliminary design using systems engineering process (such as 
prototyping of preliminary design). 

* Establish cost and schedule estimates for product on the basis of 
knowledge from preliminary design using system engineering tools (such 
as prototyping of preliminary design). 

* Constrain development phase (5 to 6 years or less) for incremental 
development. 

* Ensure development phase fully funded (programmed in anticipation of 
milestone). 

* Align program manager tenure to complete development phase. 

* Contract strategy that separates system integration and system 
demonstration activities. 

* Conduct independent cost estimate. 

* Conduct independent program assessment. 

* Conduct major milestone decision review for development start. 

Knowledge Point 2: Transition from system integration to system 
demonstration activities; Best practice metric: 90 percent of design 
drawings are complete by the critical design review: 

* Complete system critical design review. 

* Complete 90 percent of engineering design drawing packages. 

* Complete subsystem and system design reviews. 

* Demonstrate with system integration prototype that design meets 
requirements. 

* Complete the failure modes and effects analysis. 

* Identify key system characteristics. 

* Identify critical manufacturing processes. 

* Establish reliability targets and growth plan on the basis of 
demonstrated reliability rates of components and subsystems. 

* Conduct independent cost estimate. 

* Conduct independent program assessment. 

* Conduct major milestone decision review to enter system 
demonstration. 

Knowledge Point 3: Initiation of producing a product to be delivered to 
customer; Best practice metric: 100 percent of critical manufacturing 
processes are in control: 

* Demonstrate manufacturing processes. 

* Build and test production-representative prototypes to demonstrate 
product in operational environment. 

* Test production-representative prototypes to achieve reliability 
goal. 

* Collect statistical process control data. 

* Demonstrate that critical processes are capable and in statistical 
control. 

* Independent cost estimate. 

* Independent program assessment. 

* Conduct major milestone decision review to begin production. 

Source: GAO analysis of commercial best practices. 

[End of table] 

[End of section] 

Attachment 2: Technology Readiness Levels: 

Technology readiness level: 1. Basic principles observed and reported; 
Description: Lowest level of technology readiness. Scientific research 
begins to be translated into applied research and development. Examples 
might include paper studies of a technology's basic properties; 
Hardware: Software: None (paper studies and analysis); 
Demonstration Environment: None. 

Technology readiness level: 2. Technology concept and/or application 
formulated; 
Description: Invention begins. Once basic principles are observed, 
practical applications can be invented. The application is speculative 
and there is no proof or detailed analysis to support the assumption. 
Examples are still limited to paper studies; 
Hardware: Software: None (paper studies and analysis); 
Demonstration Environment: None. 

Technology readiness level: 3. Analytical and experimental critical 
function and/or characteristic proof of concept; 
Description: Active research and development is initiated. This 
includes analytical studies and laboratory studies to physically 
validate analytical predictions of separate elements of the technology. 
Examples include components that are not yet integrated or 
representative; 
Hardware: Software: Analytical studies and demonstration of nonscale 
individual components (pieces of subsystem); 
Demonstration Environment: Lab. 

Technology readiness level: 4. Component and/or breadboard. Validation 
in laboratory environment; 
Description: Basic technological components are integrated to establish 
that the pieces will work together. This is relatively "low fidelity" 
compared to the eventual system. Examples include integration of "ad 
hoc" hardware in a laboratory; 
Hardware: Software: Low fidelity breadboard; Integration of nonscale 
components to show pieces will work together. Not fully functional or 
form or fit but representative of technically feasible approach 
suitable for flight articles; 
Demonstration Environment: Lab. 

Technology readiness level: 5. Component and/or breadboard validation 
in relevant environment; 
Description: Fidelity of breadboard technology increases significantly. 
The basic technological components are integrated with reasonably 
realistic supporting elements so that the technology can be tested in a 
simulated environment. Examples include "high fidelity" laboratory 
integration of components; 
Hardware: Software: High fidelity breadboard; Functionally equivalent 
but not necessarily form and/or fit (size weight, materials, etc). 
Should be approaching appropriate scale. May include integration of 
several components with reasonably realistic support elements/ 
subsystems to demonstrate functionality; 
Demonstration Environment: Lab demonstrating functionality but not form 
and fit. May include flight demonstrating breadboard in surrogate 
aircraft. Technology ready for detailed design studies. 

Technology readiness level: 6. System/subsystem model or prototype 
demonstration in a relevant environment; 
Description: Representative model or prototype system, which is well 
beyond the breadboard tested for TRL 5, is tested in a relevant 
environment. Represents a major step up in a technology's demonstrated 
readiness. Examples include testing a prototype in a high fidelity 
laboratory environment or in simulated realistic environment; 
Hardware: Software: Prototype; Should be very close to form, fit and 
function. Probably includes the integration of many new components and 
realistic supporting elements/subsystems if needed to demonstrate full 
functionality of the subsystem; 
Demonstration Environment: High-fidelity lab demonstration or limited/ 
restricted flight demonstration for a relevant environment. Integration 
of technology is well defined. 

Technology readiness level: 7. System prototype demonstration in a 
realistic environment; 
Description: Prototype near or at planned operational system. 
Represents a major step up from TRL 6, requiring the demonstration of 
an actual system prototype in a realistic environment, such as in an 
aircraft, vehicle or space. Examples include testing the prototype in a 
test bed aircraft; 
Hardware: Software: Prototype. Should be form, fit and function 
integrated with other key supporting elements/subsystems to demonstrate 
full functionality of subsystem; 
Demonstration Environment: Flight demonstration in representative 
realistic environment such as flying test bed or demonstrator aircraft; 
Technology is well substantiated with test data. 

Technology readiness level: 8. Actual system completed and "flight 
qualified" through test and demonstration; 
Description: Technology has been proven to work in its final form and 
under expected conditions. In almost all cases, this TRL represents the 
end of true system development. Examples include developmental test and 
evaluation of the system in its intended weapon system to determine if 
it meets design specifications; 
Hardware: Software: Flight qualified hardware; 
Demonstration Environment: Developmental Test and Evaluation (DT&E) in 
the actual system application. 

Technology readiness level: 9. Actual system "flight proven" through 
successful mission operations; 
Description: Actual application of the technology in its final form and 
under mission conditions, such as those encountered in operational test 
and evaluation. In almost all cases, this is the end of the last "bug 
fixing" aspects of true system development. Examples include using the 
system under operational mission conditions; 
Hardware: Software: Actual system in final form; 
Demonstration Environment: Operational Test and Evaluation (OT&E) in 
operational mission conditions. 

Source: GAO and its analysis of National Aeronautics and Space 
Administration data. 

[End of table] 

[End of section] 

Attachment 3: Notional Illustration Showing the Different Paths That a 
Product's Development Can Take: 

[Refer to PDF for image: illustration] 

This illustration is a graph plotting schedule/time against 
cost/investment. Two paths are shown: 
1) Mature product and; 
2) Continual changes needed to reach maturity. 

Mature product: takes less time and has a lower cost due to: 
* stable design; 
* Demonstrated; 
* Built and tested; 
* Manufacturing processes in control. 

Continual changes needed to reach maturity: takes more time and has a 
greater cost due to: 
* Unstable design; 
* Late drawings and design changes; 
* Tooling and design changes (Fully integrated aircraft not built or 
tested); 
* Production manufacturing process not in control; 
* Labor inefficiencies and quality issues. 

Source: GAO. 

[End of figure] 

[End of section] 

Related GAO Products: 

Defense Acquisitions: Assessment of Major Weapon Programs. [hyperlink, 
http://www.gao.gov/products/GAO-09-326SP]. Washington, D.C.: March 30, 
2009. 

Defense Acquisitions: DOD Must Prioritize Its Weapon System 
Acquisitions and Balance Them with Available Resources. [hyperlink, 
http://www.gao.gov/products/GAO-09-501T]. Washington, D.C.: March 18, 
2009. 

Defense Acquisitions: DOD Must Balance Its Needs with Available 
Resources and Follow an Incremental Approach to Acquiring Weapon 
Systems. [hyperlink, http://www.gao.gov/products/GAO-09-431T]. 
Washington, D.C.: March 3, 2009. 

Defense Acquisitions: Perspectives on Potential Changes to DOD's 
Acquisition Management Framework. [hyperlink, 
http://www.gao.gov/products/GAO-09-295R]. Washington, D.C.: February 
27, 2009. 

Defense Management: Actions Needed to Overcome Long-standing Challenges 
with Weapon Systems Acquisition and Service Contract Management. 
[hyperlink, http://www.gao.gov/products/GAO-09-362T]. Washington, D.C.: 
February 11, 2009. 

Defense Acquisitions: Fundamental Changes Are Needed to Improve Weapon 
Program Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-08-1159T]. Washington, D.C.: September 
25, 2008. 

Defense Acquisitions: DOD's Requirements Determination Process Has Not 
Been Effective in Prioritizing Joint Capabilities. [hyperlink, 
http://www.gao.gov/products/GAO-08-1060]. Washington, D.C.: September 
25, 2008. 

Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve 
Major Weapon System Program Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-08-619]. Washington, D.C.: July 2, 
2008. 

Defense Acquisitions: Better Weapon Program Outcomes Require 
Discipline, Accountability, and Fundamental Changes in the Acquisition 
Environment. [hyperlink, http://www.gao.gov/products/GAO-08-782T]. 
Washington, D.C.: June 3, 2008. 

Defense Acquisitions: Assessments of Selected Weapon Programs. 
[hyperlink, http://www.gao.gov/products/GAO-08-467SP]. Washington, 
D.C.: March 31, 2008. 

Best Practices: Increased Focus on Requirements and Oversight Needed to 
Improve DOD's Acquisition Environment and Weapon System Quality. 
[hyperlink, http://www.gao.gov/products/GAO-08-294]. Washington, D.C.: 
February 1, 2008. 

Cost Assessment Guide: Best Practices for Estimating and Managing 
Program Costs. [hyperlink, 
http://www.gao.gov/products/GAO-07-1134SP]. Washington, D.C.: July 
2007. 

Defense Acquisitions: Assessments of Selected Weapon Programs. 
[hyperlink, http://www.gao.gov/products/GAO-07-406SP.Washington, D.C.: 
March 30, 2007. 

Best Practices: An Integrated Portfolio Management Approach to Weapon 
System Investments Could Improve DOD's Acquisition Outcomes. 
[hyperlink, http://www.gao.gov/products/GAO-07-388]. Washington, D.C.: 
March 30, 2007. 

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost 
and Schedule Problems under DOD's Revised Policy. [hyperlink, 
http://www.gao.gov/products/GAO-06-368]. Washington, D.C.: April 13, 
2006. 

DOD Acquisition Outcomes: A Case for Change. [hyperlink, 
http://www.gao.gov/products/GAO-06-257T]. Washington, D.C.: November 
15, 2005. 

Best Practices: Better Support of Weapon System Program Managers Needed 
to Improve Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-06-110]. Washington, D.C.: November 1, 
2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. [hyperlink, 
http://www.gao.gov/products/GAO-04-393]. Washington, D.C.: March 1, 
2004. 

Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, 
but More Controls Are Needed. [hyperlink, 
http://www.gao.gov/products/GAO-04-53]. Washington, D.C.: November 10, 
2003. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. [hyperlink, 
http://www.gao.gov/products/GAO-03-57. Washington, D.C.: February 11, 
2003. 

Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept 
Technology Demonstration. [hyperlink, 
http://www.gao.gov/products/GAO-03-52]. Washington, D.C.: December 2, 
2002. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-02-701]. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. [hyperlink, http://www.gao.gov/products/GAO-02-469T]. 
Washington, D.C.: February 27, 2002. 

[End of section] 

Footnotes: 

[1] Technology readiness levels, originally developed by the National 
Aeronautics and Space Administration, are measured on a scale of 1 to 
9, beginning with paper studies of a technology's feasibility and 
culminating with a technology fully integrated into a completed 
product. See Attachment 2 for the definitions of technology readiness 
levels. 

[2] We did this by collecting data directly from program offices using 
a questionnaire. These programs are primarily in development and, 
therefore, most relevant to current decisions about which programs 
should receive substantial investments of research and development 
funding now and large amounts of procurement funding in the future. 
Defense Acquisitions: Assessment of Selected Weapons Programs. 
[hyperlink, http://www.gao.gov/products/GAO-09-326SP]. Washington, 
D.C.: March 30, 2009. 

[3] In 2003, DOD revised its primary acquisition policy to state that 
technologies should be demonstrated in a relevant environment prior to 
starting an acquisition program. In 2006, this standard became a 
statutory requirement for all major defense acquisition programs in the 
National Defense Authorization Act for Fiscal Year 2006, Pub. L. No. 
109-163, § 801, codified at 10 U.S.C. § 2366b. 

[4] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, 
[hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington, 
D.C.: Mar. 30, 2009). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: