This is the accessible text file for GAO report number GAO-09-665 
entitled 'Defense Acquisitions: Many Analyses of Alternatives Have Not 
Provided a Robust Assessment of Weapon System Options' which was 
released on October 26, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Subcommittee on National Security and Foreign 
Affairs, Committee on Oversight and Government Reform, House of 
Representatives: 

United States Government Accountability Office:
GAO: 

September 2009: 

Defense Acquisitions: 

Many Analyses of Alternatives Have Not Provided a Robust Assessment of 
Weapon System Options: 

GAO-09-665: 

GAO Highlights: 

Highlights of GAO-09-665, a report to the Chairman, Subcommittee on 
National Security and Foreign Affairs, Committee on Oversight and 
Government Reform, U.S. House of Representatives. 

Why GAO Did This Study: 

Department of Defense (DOD) weapon programs often experience 
significant cost and schedule problems because they are allowed to 
start with too many technical unknowns and not enough knowledge about 
the development and production risks they entail. GAO was asked to 
review the department’s Analysis of Alternatives (AOA) process—a key 
first step in the acquisition process intended to assess the 
operational effectiveness, costs, and risks of alternative weapon 
system solutions for addressing a validated warfighting need. This 
report (1) examines whether AOAs have been effective in identifying the 
most promising options and providing a sound rationale for weapon 
program initiation, (2) determines what factors have affected the scope 
and quality of AOAs, and (3) assesses whether recent DOD policy changes 
will enhance the effectiveness of AOAs. To meet these objectives, GAO 
efforts included collecting information on AOAs from 32 major defense 
acquisition programs, reviewing guidance and other documents, and 
interviewing subject matter experts. 

What GAO Found: 

Although an AOA is just one of several inputs required to initiate a 
weapon system program, a robust AOA can be a key element to ensure that 
new programs have a sound, executable business case. Many of the AOAs 
that GAO reviewed did not effectively consider a broad range of 
alternatives for addressing a warfighting need or assess technical and 
other risks associated with each alternative. For example, the AOA for 
the Future Combat System program, one of DOD’s large and most complex 
development efforts, analyzed the operational performance and cost of 
its alternatives but failed to compare the technical feasibility and 
risks, assuming that the technologies would perform as forecasted. 
Without a sufficient comparison of alternatives and focus on technical 
and other risks, AOAs may identify solutions that are not feasible and 
decision makers may approve programs based on limited knowledge. While 
many factors can affect cost and schedule outcomes, we found that 
programs that had a limited assessment of alternatives tended to have 
poorer outcomes than those that had more robust AOAs. 

The narrow scope and limited risk analyses in AOAs can be attributed in 
part to program sponsors choosing a solution too early in the process, 
the compressed timeframes that AOAs are conducted under, and the lack 
of guidance for conducting AOAs. While AOAs are supposed to provide a 
reliable and objective assessment of viable weapon solutions, we found 
that service sponsors sometimes identify a preferred solution or a 
narrow range of solutions early on, before an AOA is conducted. The 
timing of AOAs has also been problematic. Some AOAs are conducted under 
compressed timeframes in order to meet a planned milestone or weapon 
system fielding date and are conducted concurrently with other key 
activities required to become a program of record. This can short-
change a comprehensive assessment of risks and preclude effective cost, 
schedule, and performance trade offs from taking place prior to 
beginning development. Furthermore, while DOD has an opportunity to 
influence the scope and quality of AOAs, it has not always provided 
guidance for conducting individual AOAs. 

Recognizing the need for more discipline in weapon systems acquisition, 
DOD recently revised its overall acquisition and requirements policies. 
If implemented properly, the revised policies could provide a better 
foundation for planning and starting new programs with sound, knowledge-
based business cases. Included in the revised acquisition policy are 
several mechanisms to improve the AOA process. For example, the policy 
revisions should help ensure that DOD direction is provided before AOAs 
are started and that they are conducted at an early point in the 
acquisition process where their results can inform decisions affecting 
program initiation. While these policy changes are promising, DOD must 
ensure that they are consistently implemented and reflected in 
decisions on individual programs. Furthermore, more specific criteria 
and guidance for how AOAs should be conducted may need to be developed 
to ensure they meet their intended objectives and provide an in-depth 
assessment of alternatives. 

What GAO Recommends: 

GAO is recommending that DOD establish criteria and guidance for how 
AOAs should be conducted and ensure AOAs are completed before 
requirements are set. DOD concurred, and stated that recently revised 
policies improve AOAs. We believe that the policy revisions do not go 
far enough to address our recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-09-665] or key 
components. For more information, contact Michael J. Sullivan at (202) 
512-4841 or sullivanm@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Most Programs Have Not Conducted a Robust Assessment of Alternatives: 

Choosing an Alternative Too Early and Conducting AOAs under Compressed 
Time Frames and without Effective Guidance Limit the Scope And Quality 
of AOAs: 

Recent DOD Policy Changes Could Improve AOA Effectiveness: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: Key Characteristics of AOAs: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Comparison of the Scope of Alternatives Considered with 
Program Cost and Schedule Growth: 

Table 2: Programs That Did Not Conduct AOAs: 

Table 3: Scope of Alternatives Considered in AOAs: 

Table 4: Risk Assessment of AOAs by Program: 

Table 5: Comparison of Quality of AOA Risk Assessments and Program Cost 
and Schedule Growth: 

Table 6: Factors Associated with AOAs That Had Different Scope and Risk 
Assessments: 

Table 7: Policy Revisions That Affect AOAs: 

Table 8: Key Characteristics of Programs That Conducted AOAs: 

Figures: 

Figure 1: Where AOAs Fit within the Requirements and Acquisition 
Processes: 

Figure 2: Alternatives Considered for the ARH Program: 

Figure 3: Milestones for WIN-T Program: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

September 24, 2009: 

The Honorable John F. Tierney: 
Chairman: 
Subcommittee on National Security and Foreign Affairs: 
Committee on Oversight and Government Reform: 
House of Representatives: 

Dear Mr. Chairman: 

Cost, schedule, and performance problems in the Department of Defense's 
(DOD) weapon system programs are serious. Recently, we reported that 
the department's 2008 portfolio of 96 major defense acquisition 
programs experienced cost growth of $296 billion, experienced an 
average delay in delivering initial capabilities of 22 months, and have 
delivered fewer quantities and capabilities to the warfighter than 
originally planned.[Footnote 1] Over the past several years, our work 
has highlighted a number of underlying causes for why poor outcomes 
have occurred in weapon programs. One key cause is that DOD allows 
programs to begin without a sound match between requirements and the 
resources needed to achieve them. That is, programs enter the 
acquisition process with requirements that are not fully understood, 
cost and schedule estimates that are based on optimistic assumptions, 
and a lack of sufficient knowledge about technology, design, and 
manufacturing. 

With the growing fiscal pressures now facing the nation, DOD needs to 
get the best value for every dollar it invests in weapon system 
programs. The department's management of its weapon system programs has 
been a matter of congressional concern for many years. In 2008, the 
Subcommittee requested that we study how DOD makes trade offs in 
requirements, costs, and technical risks before approving programs to 
start development. Specifically, the Subcommittee asked us to review 
the department's Analysis of Alternatives (AOA) process--a key first 
step in the acquisition process intended to assess the operational 
effectiveness, costs, and risks of alternative weapon system solutions 
for addressing a validated warfighting need. Determining what type of 
weapon system to pursue is critical because, according to a recent 
estimate, about three-quarters of a program's total life-cycle cost is 
influenced by decisions made before it is approved to start development 
[Footnote 2]. This report assesses (1) whether AOAs have been effective 
in identifying the most promising options and providing a sound 
rationale for weapon program initiation, (2) the factors that affect 
the scope and quality of AOAs, and (3) whether recent DOD policy 
changes will enhance the effectiveness of AOAs. 

To assess DOD's AOA process, we reviewed relevant DOD and military 
service policy and guidance, and interviewed officials from the Joint 
Staff, Office of Program Analysis & Evaluation, Air Force Office of 
Aerospace Studies, Army Training and Doctrine Command Analysis Center, 
and other subject matter experts. We also collected and analyzed 
information on AOAs from 32 major defense acquisition programs[Footnote 
3] that had started since fiscal year 2003.[Footnote 4] We obtained 
information from program officials on how these AOAs were conducted and 
whether the AOAs contributed to changes in the program's weapon system 
concept. Furthermore, we reviewed AOA documents and DOD and service 
guidance for conducting these AOAs. In reviewing AOAs, we examined the 
scope of alternatives that were considered and the extent of risk 
assessments conducted for each alternative. This work was conducted 
from June 2008 to September 2009 in accordance with generally accepted 
government auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to provide 
a reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a reasonable 
basis for our findings and conclusions based on our audit objectives. 
Additional information about our scope and methodology is provided in 
appendix I; appendix III contains information about the programs and 
AOAs we reviewed. 

Background: 

Before service or joint sponsors can initiate major defense acquisition 
programs and begin system development at Milestone (MS) B, they are 
required by DOD's acquisition policy to conduct an AOA.[Footnote 5] The 
AOA is an analytical study that is intended to compare the operational 
effectiveness, cost, and risks of a number of alternative potential 
solutions to address valid needs and shortfalls in operational 
capability. The basis for conducting an AOA begins when a capability 
need is validated or approved through the department's requirements 
determination process--the Joint Capabilities Integration and 
Development System (JCIDS) (See figure 1). 

Figure 1: Where AOAs Fit within the Requirements and Acquisition 
Processes: 

[Refer to PDF for image: illustration] 

Requirements: 
Initial Capabilities Document (ICD): Identifies capability gaps and 
proposes solutions; 

Acquisition: 
Analysis of Alternatives; defines system capabilities; leads to: 

Requirements: 
Milestone A; 
Capability development Document (CDD); 
Milestone B. 

Source: GAP presentation of DOD policies. 

[End of figure] 

A sponsor, usually a military service, submits a capability proposal-- 
called an Initial Capabilities Document (ICD)--through JCIDS, which 
identifies the existence of a capability gap, the operational risks 
associated with the gap, and a recommended solution or preferred set of 
solutions for filling the gap. When a capability proposal is validated, 
before a major defense acquisition program begins, an AOA is undertaken 
to compare potential solutions and determine the most promising and 
cost-effective weapon system to acquire. The AOA is a key input to 
defining the system capabilities of the major defense acquisition 
program, which are established in a capability development document 
(CDD). 

Most AOAs are sponsored by a single military service, but some may be 
conducted jointly by more than one service, in which case, the 
Milestone Decision Authority (MDA) designates a lead service as the 
sponsor. AOAs are conducted by study teams, the composition of which 
depends on the service--most of the Army's AOAs are conducted by the 
Army's Training and Doctrine Command Analysis Center, most of the Air 
Force's AOAs are conducted by the Air Force's major commands, such as 
the Air Combat Command, and most of the Navy's AOAs are contracted out 
to federally funded research and development centers and the Navy's 
various study centers. Both the Office of the Secretary of Defense 
(OSD) and the services are responsible for issuing study guidance to 
scope the AOA, which provide a minimum set of alternatives to analyze 
and shape the analysis through a series of study questions. Conducting 
an AOA may take anywhere from a few months to several years and cost 
from a few hundred thousand to several million dollars depending on its 
scope and complexity. The final results and recommendations of the AOA 
are then presented to decision makers, who decide on which alternative 
to select for program initiation. According to the Air Force's manual 
[Footnote 6] on conducting AOAs, some of the key questions that 
decision makers need the AOA to answer include: 

* What alternatives provide validated capabilities? 

* Are the alternatives operationally suitable and effective? 

* Can the alternatives be supported? 

* What are the risks (technical, operational, programmatic) for each 
alternative? 

* What are the life-cycle costs for each alternative? 

* How do the alternatives compare to one another? 

The Office of the Secretary of Defense, Program Analysis and Evaluation 
(OSD PA&E), plays a central role in the AOA process because it is 
responsible for providing initial guidance to the AOA study team, 
reviewing the proposed AOA study plan, and assessing the completed AOA. 
In carrying out these functions, OSD PA&E provides a DOD enterprise- 
level perspective to AOAs and encourages service sponsors to consider 
all viable concepts to fill a capability need, even if they were not 
initially considered by the service sponsors, and to assess technical 
risks and costs of each alternative. 

The AOA is one of several inputs required for a program's initiation at 
MS B,[Footnote 7] and it is a key element in planning and establishing 
a sound business case for a weapon system program. We have frequently 
reported on the importance of using a solid, executable business case 
before committing resources to a new product development effort. 
[Footnote 8] The business case in its simplest form is demonstrated 
evidence that (1) the warfighter's needs are valid and that they can 
best be met with the chosen concept, and (2) the chosen concept can be 
developed and produced within existing resources--that is, proven 
technologies, design knowledge, adequate funding, and adequate time to 
deliver the product when it is needed. The AOA addresses the first 
point of a business case by providing a foundation for developing and 
refining the operational requirements for a weapon system program. An 
AOA also addresses the second point of a business case by providing 
insight into the technical feasibility and costs of alternatives. By 
contributing to business cases, AOAs should provide programs with a 
sound basis for program initiation. 

Most Programs Have Not Conducted a Robust Assessment of Alternatives: 

Most of the programs we reviewed either did not conduct an AOA or 
conducted an AOA that focused on a narrow scope of alternatives and did 
not adequately assess and compare technical and other risks of each 
alternative. While many factors can affect program cost and schedule 
outcomes, we found that programs that conducted a limited assessment of 
alternatives before the start of system development tended to 
experience poorer outcomes than the programs that conducted more robust 
AOAs. According to several DOD and program officials, AOAs have often 
simply validated a concept selected by the sponsor and are not used as 
intended to make trade offs among performance, cost, and risks to 
achieve an optimal weapon system concept that satisfies the 
warfighter's needs within available resource constraints. 

Most Programs Analyzed a Narrow Scope of Alternatives before They 
Started: 

Most of the programs we reviewed considered a narrow scope of 
alternatives to support program start. Ten of the 32 programs did not 
conduct AOAs and focused on an already selected weapon system solution. 
Of the 22 programs that had AOAs, 13 of them examined a limited number 
of alternatives within a single weapon system concept such as 
helicopters or specific classes of ships, while 9 considered a 
relatively broad range of alternatives, by assessing many alternatives 
within a single weapon concept or alternatives across multiple 
concepts, such as comparing ships to aircraft. We found that the 
programs that considered a broad range of alternatives tended to have 
better cost and schedule outcomes than the programs that looked at a 
narrow scope of alternatives (see table 1). For example, 1 of the 9 
programs that examined a broad set of alternatives experienced high 
cost or schedule growth whereas 8 of the 13 programs that considered 
only a limited number of alternatives experienced high cost or schedule 
growth. 

Table 1: Comparison of the Scope of Alternatives Considered with 
Program Cost and Schedule Growth: 

Scope of alternatives[B]: No AOA conducted; 
Number of programs with cost or schedule growth[A]: 
Low: 7; 
Moderate: 0; 
High: 3. 

Scope of alternatives[B]: AOA included broad scope of alternatives; 
Number of programs with cost or schedule growth[A]: 
Low: 7; 
Moderate: 1; 
High: 1. 

Scope of alternatives[B]: AOA included narrow scope of alternatives; 
Number of programs with cost or schedule growth[A]: 
Low: 4; 
Moderate: 1; 
High: 8. 

Source: GAO. 

[A] Cost growth: High = 25 percent or greater growth in development 
cost (or procurement costs for nondevelopmental programs) from initial 
baseline to current estimates, Moderate = 10 to 24 percent growth in 
development cost (or procurement costs for nondevelopmental programs) 
from initial baseline to current estimates, Low = less than 10 percent 
growth in development cost (or procurement costs for nondevelopmental 
programs) from initial baseline to current estimates. 

Schedule growth: High = greater than 12 month delay for the initial 
operational capability date or acquisition cycle, Moderate = 7 to 12 
month delay for the initial operational capability date or acquisition 
cycle, Low = less than 7 month delay for the initial operational 
capability date or acquisition cycle. 

[B] Narrow scope of alternatives = 2-5 alternatives within one concept; 
broad scope of alternatives = 8-26 alternatives within one concept, or 
alternatives within multiple concepts. 

[End of table] 

Programs That Did Not Conduct AOAs: 

For various reasons, 10 of the 32 weapon programs we reviewed did not 
have formal AOAs to support program start (see table 2). For 7 of these 
programs, it may have been appropriate not to conduct the AOA because 
the programs involved a planned modernization to an existing weapon 
system or there was support from other analyses to warrant the chosen 
concept. This was the case, for example, with the Navy's Standard 
Missile 6 (SM-6) program. Because the missile was the next planned 
increment in a long history of missile development efforts and an AOA 
had been conducted for the previous standard missile increment, a 
separate AOA for SM-6 was considered repetitive and waived. The program 
started development in 2004 and has remained on track with its planned 
cost and schedule objectives. Similarly, an AOA was not conducted for 
the Air Force's Global Positioning System IIIA program because there 
was a body of analysis available that served the purpose of an AOA and 
the proposed program was considered a follow-on increment to a multi-
program effort to modernize global positioning system capabilities. 
Since it started development in 2008, the program has remained on cost 
and schedule. 

Table 2: Programs That Did Not Conduct AOAs: 

Program: B-2 RMP - B-2 Radar Modernization Program; 
Reasons for not conducting an AOA: The Radar Modernization Program 
(RMP) only changes the operating frequency of the radar with no 
functional upgrades. The B-2 system program office considers the 
analysis leading to the radar modernization to be an informal AOA; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: B-2 EHF INCREMENT I - B-2 Advanced Extremely High Frequency 
SatCom Capability; 
Reasons for not conducting an AOA: The program had previous analyses on 
processors and antennas that fulfilled the intent of an AOA. The 
milestone decision authority for the program waived the AOA 
requirement; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: F-22A Modernization Program; 
Reasons for not conducting an AOA: The F-22 program had completed a 
cost and operational effectiveness analysis in 1991. No analysis was 
conducted for the modernization component of the F-22 program; 
Cost or schedule growth (low, moderate, or high)[A]: High. 

Program: Global Positioning Satellite III (GPS IIIA); 
Reasons for not conducting an AOA: The Air Force had conducted multiple 
studies and analyses that together fulfilled the intent of an AOA; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: JTRS HMS - Joint Tactical Radio System Handheld, Manpack, and 
Small Form Fit Radios; 
Reasons for not conducting an AOA: The Army conducted a study which 
represents the results of a mission needs analysis, an operational 
requirements analysis, and a cost-effectiveness and affordability 
analysis. There was no comparison of alternatives. This report served 
as their AOA; 
Cost or schedule growth (low, moderate, or high)[A]: High. 

Program: MP RTIP - Multi-Platform Radar Technology Insertion Program; 
Reasons for not conducting an AOA: The Radar Technology Insertion 
Program was initially a Previously Planned Product Improvement for the 
Joint STARS aircraft; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: NMT - Advanced Extremely High Frequency Navy Multiband 
Terminal Satellite Program; 
Reasons for not conducting an AOA: This NMT program began as an ACAT 
II, and ACAT II programs are not required to have AOAs at program 
start; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: SBSS B10 - Space-Based Space Surveillance Block 10; 
Reasons for not conducting an AOA: An AOA was not conducted because of 
the Air Force's direction to quickly develop and field a replacement 
for the current satellite surveillance system in order to prevent an 
operational capability gap; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Program: SKY WARRIOR - Unmanned Aircraft System; 
Reasons for not conducting an AOA: The Army had already made the 
decision to procure an unmanned aerial system. The Army argued, among 
other things, that the source selection process would drive competitive 
selection of alternatives; 
Cost or schedule growth (low, moderate, or high)[A]: High. 

Program: SM-6 - Standard Missile-6; 
Reasons for not conducting an AOA: No AOA was conducted because several 
analyses were completed for previous iterations of the program; 
Cost or schedule growth (low, moderate, or high)[A]: Low. 

Source: GAO analysis of data provided by DOD program offices. 

[A] Cost growth: High = 25 percent or greater growth in development 
cost (or procurement costs for nondevelopmental programs) from initial 
baseline to current estimates, Moderate = 10 to 24 percent growth in 
development cost (or procurement costs for nondevelopmental programs) 
from initial baseline to current estimates, Low = less than 10 percent 
growth in development cost (or procurement costs for nondevelopmental 
programs) from initial baseline to current estimates. 

Schedule growth: High = greater than 12 month delay for the initial 
operational capability date or acquisition cycle, Moderate = 7 to 12 
month delay for the initial operational capability date or acquisition 
cycle, Low = less than 7 month delay for the initial operational 
capability date or acquisition cycle. 

[End of table] 

However, in the other 3 programs that did not have AOAs, the 
requirements and development effort proved to be more demanding and 
cost and schedule growth occurred. In the case of the Army's Sky 
Warrior Unmanned Aerial System (UAS) program, an Army executive waived 
the AOA requirement because the Army believed, among other things, that 
the source selection process would provide an adequate way to compare 
alternatives. However, when the Air Force and Joint Staff were 
reviewing the Sky Warrior's draft requirements and acquisition 
documentation, they raised concerns that the requirements potentially 
duplicated capability provided by the Air Force's Predator UAS. The 
Army cited the urgent need of battlefield commanders for the capability 
and gained approval to proceed to source selection. Three years after 
the Sky Warrior AOA was waived, the Deputy Secretary of Defense 
directed that the two UAS programs be combined into a single 
acquisition program to achieve efficiencies in areas such as common 
development, procurement, and training activities. However, the Army 
and Air Force have continued to pursue unique systems. In the meantime, 
the Sky Warrior UAS has experienced a 138 percent increase in total 
cost and 47-month schedule delay from original plans. By relying on 
industry-provided information in source selection and not conducting an 
independent AOA, the Army missed an opportunity to gain a better 
understanding of the other services' UAS capabilities, and pursue an 
acquisition strategy that would have taken advantage of commonalities 
and used resources more efficiently. 

Programs That Conducted AOAs: 

Of the 22 programs that conducted AOAs, 13 focused on a limited number 
of alternatives within a single weapon system concept while 9 focused 
on many alternatives[Footnote 9] (see table 3). According to DOD and 
service officials, the scope of an AOA can be different for each 
program and dependent upon many factors, including the nature of the 
capability need, the proposed time frame for fielding the capability, 
and the type of program being pursued-whether it is a new development 
start, a modification of a commercially available system, or an upgrade 
to an existing system. As a result, AOAs that focus on a limited number 
of alternatives within a single weapon system concept may be 
appropriate in some cases. For instance, when the capability need was 
defined in terms of upgrading an existing weapon system, AOAs focused 
on refining a single platform concept and its system-level 
specifications and attributes. The AOA for the Army's Apache Block III 
program is an example of an appropriately, but narrowly scoped AOA. It 
examined various block upgrade options for the existing Longbow Apache 
helicopter to improve interoperability and other shortcomings in the 
helicopter. The program started development in 2006 and has remained on 
track with its planned cost and schedule objectives. 

Table 3: Scope of Alternatives Considered in AOAs: 

AOAs with narrow scope of alternatives[A]: AGM-88E AARGM - AGM-88E 
Advanced Anti-Radiation Guided Missile (AARGM) Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with broad scope of alternatives[A]: BAMS - Broad Area Maritime 
Surveillance; 
Cost or schedule growth (low, moderate, or high)[B]: High. 

AOAs with narrow scope of alternatives[A]: AB3 - Apache Block III; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with broad scope of alternatives[A]: EA-18G - Electronic attack 
variant of the F/A-18 aircraft; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: ARH - Armed Reconnaissance 
Helicopter; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: CH-53K - Heavy Lift 
Replacement Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: CVN 21 - Next Generation 
Nuclear Aircraft Carrier; 
Cost or schedule growth (low, moderate, or high)[B]: Moderate; 
AOAs with broad scope of alternatives[A]: JCA - Joint Cargo Aircraft; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: DDG 1000 - ZUMWALT CLASS 
Destroyer; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: JLENS - Joint Land Attack 
Cruise Missile Defense Elevated Netted Sensor System; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: E-2D AHE - E-2D Advanced 
Hawkeye; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: LUH - Light Utility 
Helicopter; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: FCS - Future Combat Systems; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: MUOS - Mobile User Objective 
System; 
Cost or schedule growth (low, moderate, or high)[B]: Moderate. 

AOAs with narrow scope of alternatives[A]: JTRS AMF - Joint Tactical 
Radio System Airborne, Maritime/Fixed Station; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with broad scope of alternatives[A]: P-8A - Poseidon Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: LCS - Littoral Combat Ship; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: SDB I - Small Diameter Bomb 
Increment I; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with narrow scope of alternatives[A]: LHA REPLACEMENT - New 
Amphibious Assault Ship; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with broad scope of alternatives[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

AOAs with narrow scope of alternatives[A]: MPS - Mission Planning 
System; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

AOAs with narrow scope of alternatives[A]: VH-71 - Presidential 
Helicopter Fleet Replacement Program; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

AOAs with narrow scope of alternatives[A]: WIN-T - Warfighter 
Information Network - Tactical; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with broad scope of alternatives[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

Source: GAO analysis of data provided by DOD program offices. 

[A] Narrow scope of alternatives = 2-5 alternatives within one concept; 
broad scope of alternatives = 8-26 alternatives within one concept, or 
alternatives within multiple concepts. 

[B] Cost growth: High = 25 percent or greater growth in development 
cost (or procurement costs for nondevelopmental programs) from initial 
baseline to current estimates, Moderate = 10 to 24 percent growth in 
development cost (or procurement costs for nondevelopmental programs) 
from initial baseline to current estimates, Low = less than 10 percent 
growth in development cost (or procurement costs for nondevelopmental 
programs) from initial baseline to current estimates. 

Schedule growth: High = greater than 12 month delay for the initial 
operational capability date or acquisition cycle, Moderate = 7 to 12 
month delay for the initial operational capability date or acquisition 
cycle, Low = less than 7 month delay for the initial operational 
capability date or acquisition cycle. 

[End of table] 

In a few of the other AOAs that had a narrow scope, the capability need 
involved the replacement of an aging weapon system and the AOAs 
presumed that the concept of the aging weapon system was the 
appropriate starting point for analysis rather than examining whether 
other concepts could also meet the need. For example, the AOA for the 
Army's Armed Reconnaissance Helicopter (ARH) program, which was 
intended to replace the aging Kiowa helicopter fleet and improve attack 
and reconnaissance capabilities, examined two options: improving the 
legacy Kiowa helicopter or procuring nondevelopmental helicopters. The 
AOA did not explore other potential solutions, such as developing 
unmanned aerial systems, increasing the purchase of existing attack 
helicopters, increasing the purchase of other reconnaissance assets, or 
relying on a mix of solutions. After 3 years of development, the ARH 
program's research and development costs increased from about $360 
million to $940 million. A Center for Naval Analyses report 
commissioned by the Army after the ARH program began having execution 
problems identified several factors that contributed to the significant 
cost growth, including questionable requirements, an aggressive 
schedule, limited oversight, and a perceived preference for one 
helicopter model. As a result of the cost growth and other problems, 
DOD canceled the program in 2008 after determining that at least one 
alternative could provide equal or greater capability at less cost. 

Most of the programs (7 of 9) that examined a broad scope of 
alternatives have tracked well with their planned cost and schedule 
targets. The AOA for the Navy's P-8A Multi-mission Maritime Aircraft, 
which is a program designed to replace the P-3C aircraft and provide 
maritime patrol and reconnaissance for the Navy, explored multiple 
concepts and many alternatives in response to study guidance issued by 
OSD PA&E, including several nonmanned aircraft alternatives such as 
submarines, helicopters, and UAS. The AOA concluded that a manned 
aircraft would still be the best option to replace the P-3C. However, 
the AOA also helped the Navy to recognize that a UAS could perform some 
of the maritime patrol missions as an adjunct platform, eventually 
leading to the Broad Area Maritime Surveillance (BAMS) UAS AOA and 
program. The P-8A program has not experienced cost growth over its 4 
years of development and remains on schedule. Similarly, the Joint Land 
Attack Cruise Missile Defense Elevated Netted Sensor System (JLENS), 
which is designed to provide over the horizon detecting and tracking of 
land attack cruise missile and other targets, had an AOA that explored 
alternatives across multiple concepts, including aerostat sensors, sea- 
based sensors, and nonaerostat elevated sensors. The Army chose the 
aerostat concept and has developed an incremental program that has 
experienced low cost and schedule growth since starting development in 
2005. 

Many AOAs Have Not Adequately Assessed Risks for the Alternatives: 

DOD acquisition policy requires that AOAs assess the technical risk of 
alternatives, but it does not provide criteria and guidance for how and 
to what extent technical risks should be addressed and it does not 
specify that other types of risks should be assessed.[Footnote 10] 
Risks are important to assess because there may be technical, 
programmatic, or operational uncertainties associated with different 
alternatives that should be considered in determining the best weapon 
system approach. For example, it may be the case that one alternative 
is more effective than another in meeting a capability need but has 
more technical or other risks that may make the alternative infeasible 
to develop. Many of the AOAs we reviewed (12 of the 22) conducted 
limited assessments of the risks of each alternative presented (see 
table 4).[Footnote 11] Some AOAs we reviewed did not examine risks at 
all, focusing only on the operational effectiveness and costs of 
alternatives. Other AOAs had relatively limited risk assessments. For 
example, several AOAs did not discuss integration risks even though 
they were examining modified commercial systems that required the 
integration of subsystems or equipment packages, while other AOAs did 
not examine the schedule risks of the various alternatives, despite 
accelerated schedules and fielding dates for the programs. We found 
that programs with AOAs that conducted a more comprehensive assessment 
of risks tended to have better cost and schedule outcomes than those 
that did not (see table 5). 

Table 4: Risk Assessment of AOAs by Program: 

AOAs with no or limited assessment of risks[A]: AGM-88E AARGM - AGM 88E 
Advanced Anti-Radiation Guided Missile (AARGM) Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with adequate assessment of risks[A]: BAMS - Broad Area Maritime 
Surveillance; 
Cost or schedule growth (low, moderate, or high)[B]: High. 

AOAs with no or limited assessment of risks[A]: AB3 - Apache Block III; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with adequate assessment of risks[A]: DDG 1000 - ZUMWALT CLASS 
Destroyer; 
Cost or schedule growth (low, moderate, or high)[B]: High. 

AOAs with no or limited assessment of risks[A]: ARH - Armed 
Reconnaissance Helicopter; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: EA-18G - Electronic attack 
variant of the F/A-18 aircraft; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: CVN-21 - Next 
Generation Nuclear Aircraft Carrier; 
Cost or schedule growth (low, moderate, or high)[B]: Moderate; 
AOAs with adequate assessment of risks[A]: CH-53K - Heavy Lift 
Replacement Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: E-2D AHE - E-2D 
Advanced Hawkeye; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: JCA - Joint Cargo Aircraft; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: FCS - Future Combat 
Systems; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: JTRS AMF - Joint Tactical 
Radio System Airborne & Maritime/Fixed Station; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: JLENS - Joint Land 
Attack Cruise Missile Defense Elevated Netted Sensor System; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with adequate assessment of risks[A]: LUH - Light Utility 
Helicopter; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: LCS - Littoral Combat 
Ship; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: MUOS - Mobile User Objective 
System; 
Cost or schedule growth (low, moderate, or high)[B]: Moderate. 

AOAs with no or limited assessment of risks[A]: LHA Replacement - New 
Amphibious Assault Ship; 
Cost or schedule growth (low, moderate, or high)[B]: Low; 
AOAs with adequate assessment of risks[A]: P-8A - Poseidon Program; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: MPS - Mission Planning 
System; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: SDB I - Small Diameter Bomb 
Increment I; 
Cost or schedule growth (low, moderate, or high)[B]: Low. 

AOAs with no or limited assessment of risks[A]: VH-71 - Presidential 
Helicopter Fleet Replacement Program; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

AOAs with no or limited assessment of risks[A]: WIN-T - Warfighter 
Information Network - Tactical; 
Cost or schedule growth (low, moderate, or high)[B]: High; 
AOAs with adequate assessment of risks[A]: [Empty]; 
Cost or schedule growth (low, moderate, or high)[B]: [Empty]. 

Source: GAO analysis of data provided by DOD program offices. 

[A] No assessment = no discussion in AOA of technical, integration, or 
other risks; limited assessment = only technical risk addressed or not 
all of the alternatives are assessed for technical and other risks; 
adequate assessment = all alternatives are assessed for multiple 
factors such as technical, integration, and other risks. 

[B] Cost growth: High = 25 percent or greater growth in development 
cost (or procurement costs for nondevelopmental programs) from initial 
baseline to current estimates, Moderate = 10 to 24 percent growth in 
development cost (or procurement costs for nondevelopmental programs) 
from initial baseline to current estimates, Low = less than 10 percent 
growth in development cost (or procurement costs for nondevelopmental 
programs) from initial baseline to current estimates. 

Schedule growth: High = greater than 12 month delay for the initial 
operational capability date or acquisition cycle, Moderate = 7 to 12 
month delay for the initial operational capability date or acquisition 
cycle, Low = less than 7 month delay for the initial operational 
capability date or acquisition cycle. 

[End of table] 

Table 5: Comparison of Quality of AOA Risk Assessments and Program Cost 
and Schedule Growth: 

Quality of risk assessment[B]: AOAs with no or limited assessment of 
risks; 
Number of programs with cost or schedule growth[A]: 
Low: 4; 
Moderate: 1; 
High: 7. 

Quality of risk assessment[B]: AOAs with adequate assessment of risks; 
Number of programs with cost or schedule growth[A]: 
Low: 7; 
Moderate: 1; 
High: 2. 

Source: GAO. 

[A] Cost growth: High = 25 percent or greater growth in development 
cost (or procurement costs for nondevelopmental programs) from initial 
baseline to current estimates, Moderate = 10 to 24 percent growth in 
development cost (or procurement costs for nondevelopmental programs) 
from initial baseline to current estimates, Low = less than 10 percent 
growth in development cost (or procurement costs for nondevelopmental 
programs) from initial baseline to current estimates. 

Schedule growth: High = greater than 12 month delay for the initial 
operational capability date or acquisition cycle, Moderate = 7 to 12 
month delay for the initial operational capability date or acquisition 
cycle, Low = less than 7 month delay for the initial operational 
capability date or acquisition cycle. 

[B] No assessment = no discussion in AOA of technical, integration, or 
other risks; limited assessment = only technical risk addressed or not 
all of the alternatives are assessed for technical and other risks; 
adequate assessment = all alternatives are assessed for multiple 
factors such as technical, integration, and other risks. 

[End of table] 

AOAs that do not examine risks could provide overly optimistic 
assessments of alternatives, which do not provide for sound business 
case decisions. Comparing risks across alternatives is especially 
critical for new development programs, which rely on breakthrough 
technologies and assume that technology will be achieved as planned. Of 
the 22 programs that had AOAs, 8 were new development starts involving 
technology development. Of the 8 new development starts, only 4 had 
AOAs that performed adequate risk analyses. The other 4 AOAs did not 
assess technical, integration, or other risks as criteria for comparing 
the alternatives or neglected to analyze these risks altogether. For 
example, the AOA for the Future Combat Systems (FCS), one of most 
complex and technically challenging programs ever undertaken according 
to the Army, assessed the technical risks of each of the new 
development concepts for FCS, but did not assess and compare the risks 
with those of the other alternatives. The AOA concluded that the new 
FCS development option was more costly but more operationally effective 
than the baseline and improved baseline alternatives.[Footnote 12] By 
not comparing the risks of the alternatives, the FCS AOA missed an 
opportunity to provide the Army with a meaningful trade off among 
operational effectiveness, costs, and risks. Now, after 6 years of 
development, some of the critical technologies for the FCS program are 
still immature. The latest estimates for the program show that 
development costs have grown 38 percent or about $8 billion, and the 
fielding date has been delayed 57 months. As a result, DOD recently 
proposed canceling the FCS acquisition program. 

Also, the AOA for the Army's Warfighter Information Network-Tactical 
(WIN-T) program, which involves development of new on-the-move 
networking capabilities, did not address technical or programmatic 
risks. Army officials stated that WIN-T was largely based on a concept 
that did not have well-defined requirements of the proposed network and 
operations, and the WIN-T development alternative in the AOA was based 
on preliminary design concepts, from two competing contractors, which 
were blended together by the Army. The AOA did not take these risks 
into account and concluded that the new WIN-T alternative was the most 
operational and cost-effective solution available. In March of 2007, 
the WIN-T program had a Nunn McCurdy cost breach (25 percent or more 
unit cost growth) and was subsequently restructured by DOD. 
Insufficient technical readiness was cited as one of the key factors 
leading to the cost breach. 

Assessing risks is also important for programs based on commercial 
products that require significant modifications. Based upon a recent 
Defense Science Board report on buying commercially-based defense 
systems, programs that do not assess the systems engineering and 
programmatic risks of alternatives do not understand the true costs 
associated with militarizing commercial platforms or integrating 
various commercial components.[Footnote 13] As a result of this 
incomplete understanding of inherent technical and integration risks of 
programs, DOD fails to fully take advantage of efficiencies and cost 
savings from commercially available technologies. Several of the 
programs we reviewed that involved modified commercial products had 
AOAs with weak risk assessments. For example, the AOA for the Marine 
Corps' replacement for the Presidential Helicopter, VH-71, failed to 
assess the technical, integration, and schedule risks associated with 
its three alternatives. It instead compared alternatives based on costs 
and performance attributes, such as cabin size, deployability, and 
performance. One program official stated that the focus of the VH-71 
AOA was to merely identify platforms that had the best probability of 
meeting the requirements. According to a statement by the Secretary of 
Defense, the program's costs have nearly doubled, increasing from $6.5 
billion to $13 billion, and the schedule has fallen behind by several 
years. DOD recently canceled the program. The Defense Science Board, 
which assessed the VH-71 program, concluded that some of the program's 
requirements plainly exceeded the limits of the available technology 
and schedule. 

Choosing an Alternative Too Early and Conducting AOAs under Compressed 
Time Frames and without Effective Guidance Limit the Scope and Quality 
of AOAs: 

We identified several factors that may have limited the effectiveness 
of AOAs and their ability to identify the most promising option and 
contribute to a sound business case for starting a weapon system 
program: (1) service sponsors lock into a solution early on when a 
capability need is first validated through DOD's requirements process 
and before an AOA is conducted; (2) AOAs are conducted under compressed 
time frames in order to meet a planned milestone review or fielding 
date and their results come too late to inform key trade off decisions; 
and (3) DOD does not always provide guidance for conducting individual 
AOAs. The AOAs with one or more of these factors tended to be AOAs that 
had a limited scope and assessment of risks (see table 6). 

Table 6: Factors Associated with AOAs That Had Different Scope and Risk 
Assessments: 

AOAs with limited scope and risk assessments: 
Choosing an alternative too early: 
ARH - Armed Reconnaissance Helicopter; 
CVN 21 - Next Generation Nuclear Aircraft Carrier; 
FCS - Future Combat Systems; 
LCS - Littoral Combat Ship; 
Problematic timing: 
AB3 - Apache Block III; 
ARH - Armed Reconnaissance Helicopter; 
CVN 21 - Next Generation Nuclear Aircraft Carrier; 
FCS - Future Combat Systems; 
LCS - Littoral Combat Ship; 
VH-71 - Presidential Helicopter Fleet Replacement Program; 
WIN-T - Warfighter Information Network-Tactical; 
No or late PA&E guidance: 
AGM-88E AARGM - AGM-88E Advanced Anti-Radiation Guided Missile (AARGM) 
Program; 
E-2D AHE - E-2D Advanced Hawkeye; 
FCS - Future Combat Systems; 
LCS - Littoral Combat Ship; 
MPS - Mission Planning System; 
VH-71 - Presidential Helicopter Fleet Replacement Program. 

AOAs with broad scope and adequate risk assessments: 
Choosing an alternative too early: 
CH-53K - Heavy Lift Replacement Program; 
LUH - Light Utility Helicopter; 
Problematic timing: 
CH-53K - Heavy Lift Replacement Program; 
LUH - Light Utility Helicopter; 
No or late PA&E guidance: 
BAMS - Broad Area Maritime Surveillance; 
CH-53K - Heavy Lift Replacement Program; 
SDB I - Small Diameter Bomb Increment I. 

Source: GAO analysis of data provided by DOD program offices. 

[End of table] 

Service Sponsors Lock Into a Program Solution or Establish Requirements 
Prior to the AOA: 

In developing a capability proposal, sponsors not only justify the need 
to fill an existing capability gap, but also conduct an assessment-- 
called a functional solutions analysis (FSA)--to identify a potential 
concept or set of solutions to fill the gap. The identification of a 
potential concept is intended to provide a general approach for 
addressing the gap and set the stage for a more in-depth assessment of 
alternatives to be conducted in the AOA. In four cases, AOAs were 
limited because program sponsors had decided on a preferred solution 
prior to the AOA, when a capability need was first proposed through the 
department's requirements determination process. Approval of the 
capability proposal then led to a narrowly scoped AOA that supported or 
refined the preferred solution. According to DOD officials, the 
analysis supporting a capability proposal is generally conducted by the 
operational requirements community within a military service and 
contains only rudimentary assessments of the costs and technical 
feasibility of the solutions identified. 

With the Armed Reconnaissance Helicopter program, for example, the Army 
proposed acquiring an armed reconnaissance helicopter after the 
termination of the Comanche helicopter program, which had experienced 
significant cost and schedule problems.[Footnote 14] While the initial 
capability proposal submitted to JCIDS for the ARH considered 
nonhelicopter concepts, such as unmanned aerial systems, the Army 
concluded that a modified version of an existing armed reconnaissance 
helicopter was the preferred solution. According to Army officials, the 
modified helicopter solution was pushed in part because there was a 
desire to field a system within a relatively short time frame, a 
similar helicopter variant was in use by the special operations forces, 
and funding available from the terminated Comanche helicopter program 
needed to be used quickly. Because the Army effectively locked into a 
solution in this early stage, the AOA primarily focused on comparing 
the performance and costs of existing helicopter alternatives (see 
figure 2). 

Figure 2: Alternatives Considered for the ARH Program: 

[Refer to PDF for image: illustration] 

Armed Reconnaissance Helicopter (ARH): 

Alternatives Considered Before AOA (Initial Capabilities Document): 

* Upgrade of existing ARH; 
* Increase use of attack helicopters; 
* Initiate a new-start ARH program; 
* Increased use of Reconnaissance Surveillance Target Acquisition 
assets; 
* Unmanned Combat Armed Rotorcraft;
* Unmanned Aerial Vehicle Systems; 

Upgrade of existing ARH is preferred solution. 

Alternative Examined in AOA: 

Upgrade of existing ARH alternatives: 
* Improved Kiowa (legacy helicopter); 
* Non-developmental helicopters: 
1) OH 6/MH-6 MELB; 
2) Eurocopter EC-635; 
3) Eurocopter AS-550 C3; 
4) Mil Bell 407; 

No recommended solution. 

Platform Chosen: 

Mil Bel 407 helicopter. 

Source: GAO presentation of DOD data. 

[End of figure] 

Similarly, we have previously reported[Footnote 15] that the Navy began 
the Littoral Combat Ship (LCS) program before fully examining 
alternatives. Beginning in 1998, the Navy conducted a series of 
wargames and studies to test new concepts for surface combatant ships 
that could address known threats in littoral areas. Following these 
efforts, the Navy began an analysis of multiple concepts study[Footnote 
16] in 2002 to further refine the Navy's preferred solution--a new 
warship along the lines of LCS. Concurrently, the Navy established an 
LCS program office and issued a request for proposal to industry to 
submit LCS concepts. The Office of the Secretary of Defense and the 
Joint Staff were concerned that the Navy's focus on a single solution 
did not adequately consider other ways to address littoral capability 
gaps. Based on these concerns in late 2003, the Navy was directed to 
consider alternatives to surface ships such as submarines and manned 
aircrafts in the ongoing analysis of multiple concepts. The analysis, 
which was led by the Naval Surface Warfare Center, compared nonship 
alternatives to LCS-concept ships and concluded that the LCS concept 
remained the best solution to provide capabilities in the littorals. 
However, the estimated costs for the various LCS ship alternatives 
developed in the analysis far exceeded the $220 million (fiscal year 
2005 dollars) target that the Navy had set for the program. The Navy 
stated that because the cost estimates were rough-order-of-magnitude 
estimates and were based on preliminary concept designs, those costs 
were not used to make cost decisions for LCS. However, since starting 
development in 2004, the LCS program has experienced a 151 percent 
growth in development costs and its costs are closer to the cost 
estimates from the analysis of multiple concepts than the target cost 
set by the Navy. 

DOD and service officials responsible for conducting AOAs indicated 
that often capability requirements are proposed that are so specific 
that they effectively eliminate all but the service sponsor's preferred 
concepts instead of considering other alternatives. For example, in 
recent proposals to address a global strike capability need, two 
components of the Air Force--the Air Combat Command and Space Command--
defined initial performance requirements that required two different 
approaches. The Air Force Air Combat Command defined the requirement as 
the ability to strike a target within 1 day, which meant that bombers, 
which fall under the Air Force Air Combat Command's portfolio, could 
address the gap. However, the Air Force Space Command defined the 
requirement in the capability proposal as the ability to strike a 
target within a certain number of hours, which meant only missiles, 
which fall under the Air Force Space Command, could fulfill the need. 
Although OSD PA&E attempted to get the Air Force to consider both 
bombers and missiles in the same analysis, the major commands argued 
that their requirements were different enough to require two separate 
analyses. As a result, the Air Force Air Combat Command initiated the 
Next Generation Long-Range Strike AOA for a new bomber, while the Air 
Force Space Command initiated the Prompt Global Strike AOA separately. 

Similarly, for the ARH AOA, the Army called for very specific 
deployability requirements. These requirements included the ability to 
fit two helicopters into a C-130 aircraft and for the helicopter to be 
"fightable" within 15 minutes of arrival. The Center for Naval 
Analyses, in its report on the factors that led to significant cost and 
schedule growth in the ARH program, noted that it was not clear whether 
these requirements were needed to fulfill the operational gap. 
Furthermore, the Center for Naval Analyses noted that due to the 
stringent deployability requirements, the program had effectively 
eliminated other potentially feasible and cost-effective alternatives, 
such as twin-engine helicopters, and limited the analysis to single 
engine alternatives. 

Timing of AOAs May Not Be Conducive to Informing Trade Offs: 

Many AOAs are also conducted under compressed time frames--6 months or 
less--or concurrently with other key activities that are required for 
program initiation, in order to meet a planned milestone decision or 
weapon system fielding date. Consequently, AOAs may not have enough 
time to assess a broad range of alternatives and their risks, or be 
completed too late in the process to inform effective trade discussions 
prior to beginning development. In 9 of the 22 programs we reviewed 
that had AOAs, the timing of the AOAs was compressed or concurrent with 
other planning activities. In 7 of these 9 programs, the AOAs were 
limited. For instance, the AOA for the Future Combat Systems program 
was a complex undertaking; however, according to the authors of the 
AOA, it was conducted in half the time that a less complex AOA would 
typically be conducted. In addition, due to schedule constraints 
imposed to meet a preset milestone review date, the AOA was performed 
concurrently with concept development, requirements determination, and 
system definition documents. Ultimately, the Future Combat Systems AOA 
was completed 1 month after the operational requirements were validated 
and the same month that the program was approved to begin system 
development, which precluded trade off discussions among cost, 
performance, and risks from taking place. In addition, although AOAs 
are required to be done for a Milestone B decision, the Army's 
Warfighter Information Network-Tactical (WIN-T) program was approved to 
begin without one. The milestone decision authority for the program 
waived the AOA requirement until a later date. The WIN-T AOA was 
completed approximately 16 months after the program started (see figure 
3). 

Figure 3: Milestones for WIN-T Program: 

[Refer to PDF for image: illustration] 

July 2002: Initial requirements approval; 
November 2002: AOA Started;
April 2003: Final requirements approval; 
July 2003: MS B program initiation; 
November 2004: AOA completed. 

Source: GAO presentation of DOD data. 

[End of figure] 

DOD Has Not Consistently Provided Guidance for Conducting Individual 
AOAs: 

While DOD acquisition policy[Footnote 17] requires that major defense 
acquisition programs conduct an AOA prior to program initiation at 
Milestone B, the policy does not specify criteria or guidance for how 
AOAs should be conducted. According to the policy, OSD PA&E is to 
provide guidance to programs prior to, during, and after their AOA has 
been completed. The guidance is intended to ensure that the services 
are examining a sufficient number of alternatives that take into 
consideration joint plans and interoperability, but to also ensure that 
AOAs are analyzing key risks such as technology, cost, and schedule. In 
9 of the 22 programs we reviewed that had AOAs, OSD PA&E either 
provided late guidance or did not provide formal guidance when AOAs 
were started. In 6 of these 9 programs, the AOAs were limited. For 
instance, OSD PA&E did not provide guidance for the AOA that supported 
initiation of the VH-71 Presidential Helicopter program. In this AOA, 
the service had very specific performance requirements that narrowed 
the scope of the alternatives examined. In addition, the service 
conducted the AOA under a compressed schedule to meet a previously 
planned milestone, which may not have allowed for robust analyses of 
technology and integration risks. These factors most likely played a 
part in the AOA examining only 3 alternatives and eliminating 19 other 
alternatives early on. 

DOD officials have also stated that when OSD PA&E guidance is provided, 
it is sometimes late. For example, the LCS program AOA had been 
underway for about a year before OSD PA&E provided guidance to the 
Navy. Officials also explained that guidance is often informal, 
sometimes provided over the telephone, or if written, remains in draft 
form for long periods, preventing the services from formulating and 
having analysis plans approved. However, according to PA&E officials, 
sometimes guidance is never formalized or written because the services 
do not have a validated capability proposal or do not agree with the 
scope and direction provided. By not providing timely formal guidance 
before AOAs are started, DOD is missing an opportunity to ensure AOAs 
examine an appropriate scope of alternatives and conduct robust risk 
assessments. 

Recent DOD Policy Changes Could Improve AOA Effectiveness: 

In December 2008, DOD revised its acquisition policy and introduced 
several initiatives based in part on direction from Congress that could 
provide a better foundation for establishing knowledge-based business 
cases for initiating weapon system programs. The revised policy 
strengthens the front end of the acquisition process by requiring key 
systems engineering activities and early prototyping, and establishing 
required milestone reviews to assess whether programs are acquiring the 
requisite knowledge as they move towards the start of system 
development. In addition, in March 2009, DOD revised its policy 
governing the JCIDS process, to help streamline the determination of 
capability needs and improve the integration between JCIDS and the 
acquisition process. In revising these policies, DOD elevated the role 
of AOAs in determining weapon system concepts and strengthened how they 
are to be implemented. Improving the effectiveness of AOAs will depend 
on DOD's ability to ensure that its policy changes are consistently 
implemented and reflected in decisions on individual weapon system 
programs. We have reported in the past that inconsistent implementation 
of existing policies has hindered DOD's efforts to plan and execute 
programs effectively.[Footnote 18] The key revisions to the policies 
that impact AOAs are summarized in table 7. 

Table 7: Policy Revisions That Affect AOAs: 

DOD 5000.2 (May 2003 Revision): AOA required by regulation for Major 
Defense Acquisition Programs; 
DOD 5000.02 (December 2008 Revision): AOA required by statute for 
Milestone B certification; 
Differences: Policy now reflects congressional legislation passed in 
2006 that requires the milestone decision authority to certify to 
Congress that programs approved at MS B have, among other things, 
conducted an AOA. 

DOD 5000.2 (May 2003 Revision): AOA to be conducted during concept 
refinement phase leading up to Milestone A review that may be waived; 
DOD 5000.02 (December 2008 Revision): AOA to be conducted during 
Materiel Solution phase leading up to Milestone A review that must be 
certified by the milestone decision authority; 
Differences: Policy now places greater emphasis on Milestone A. This 
should help ensure AOAs are conducted earlier. 

DOD 5000.2 (May 2003 Revision): Focus of the AOA is to refine the 
selected concept documented in the approved capability proposal 
document; 
DOD 5000.02 (December 2008 Revision): AOA is to assess potential 
materiel solutions to satisfy the capability need; 
Differences: AOA intended now to conduct broader examination of 
alternatives. 

DOD 5000.2 (May 2003 Revision): PA&E to provide guidance prior to the 
start of AOA; 
DOD 5000.02 (December 2008 Revision): PA&E to provide AOA guidance for 
approval by the milestone decision authority; 
Differences: AOA guidance now required to be approved by milestone 
decision authority. 

DOD 5000.2 (May 2003 Revision): AOA study plan to be prepared for 
review by PA&E; 
DOD 5000.02 (December 2008 Revision): AOA study plan to be approved by 
PA&E; 
Differences: AOA study plan must now be approved by PA&E. 

DOD 5000.2 (May 2003 Revision): AOA shall assess the critical 
technologies associated with concepts, including technology maturity, 
technical risk, and, if necessary, technology maturation and 
demonstration needs; 
DOD 5000.02 (December 2008 Revision): The AOA shall assess the critical 
technology elements associated with each proposed materiel solution, 
including technology maturity, integration risk, manufacturing 
feasibility, and, where necessary, technology maturation and 
demonstration needs; 
Differences: Policy broadens the focus of AOA risk assessments to 
include integration risk and manufacturing feasibility. 

Chairman of the Joint Chiefs of Staff Instruction 3170.01F (May 2007 
Revision): If a materiel solution is needed, sponsor identifies a 
potential solution or set of solutions to fill a capability gap; 
Chairman of the Joint Chiefs of Staff Instruction 3170.01G (March 2009 
Revision): If a materiel solution is needed, sponsor will make 
recommendation on the broad type of materiel solution preferred--
information system, evolutionary development of an existing capability, 
or a transformational approach; 
Differences: Policy change should help mitigate sponsors identifying a 
preferred solution early on in a proposed capability document. 

Source: GAO analysis of DOD 5000 and CJCSI 3170 policies. 

[End of table] 

DOD's revised policies, for example, may help mitigate service sponsors 
from locking into a solution too early in the process by eliminating 
the functional solutions analysis in a capability proposal, which 
identified a preferred solution and influenced the scope of 
alternatives in an AOA. In the revision, the capability proposal will 
only identify a broad category of the type of materiel solution that 
should be considered; for example, whether it should be an incremental 
or transformational development approach. The AOA will then assess 
potential solutions as determined by the milestone decision authority 
and within the broad category recommended. This change integrates 
essentially what had been two separate trade space analyses into one 
analysis. In doing so, it sets up a better opportunity for a more 
robust analysis of alternatives. 

DOD's revised acquisition policy also now imposes early milestone 
reviews which should help resolve the timing issues we found with 
several AOAs in the past. Under the previous policy, AOAs were required 
for program initiation at Milestone B, which may have led to some AOAs 
being completed just prior or even after program initiation. Under the 
revised policy, AOAs are generally required earlier in the process. 
Furthermore, DOD PA&E is required to be involved much earlier in the 
process by providing requisite guidance at the Materiel Development 
Decision as well as approving AOA study plans before an AOA is started. 
These additional reviews with required guidance earlier in the 
acquisition process should help mitigate conducting AOAs under 
compressed time frames. However, while the revised policy strengthens 
the front end of the acquisition process, the AOA is still constrained 
to a given set of requirements that may be unfeasible and could lead to 
unsuccessful program outcomes, such as with the Armed Reconnaissance 
Helicopter and Future Combat Systems. 

Conclusions: 

With increased demand and competition for funding, it is critical that 
DOD weapon system programs provide the best value to the warfighter and 
to the taxpayer. Yet in too many cases, DOD programs do not accomplish 
this and experience significant cost, schedule, and performance 
problems. Many of these problems could be avoided if programs started 
with sound, knowledge-based business cases. A key to developing such 
business cases is having effective AOAs that analyze and compare the 
performance, costs, and risks of competing solutions, and identify the 
most promising weapon system solution to acquire. The majority of AOAs 
we reviewed were limited and thus did not sufficiently inform the 
business case for starting new programs. 

DOD's recent policy revisions are positive steps that could, if 
implemented properly, provide a better foundation for conducting AOAs 
and establishing sound business cases for starting acquisition 
programs. The revisions, for example, should help ensure that DOD 
direction is provided before AOAs are started and that AOAs are 
conducted at an early point in the acquisition process where their 
results can inform key decisions affecting program initiation. However, 
these policy changes alone will not be sufficient to ensure AOAs 
achieve their intended objectives. Unless mechanisms are established to 
ensure policy is followed, specific guidance and criteria are developed 
for how AOAs should be conducted, and AOAs are completed before program 
requirements are set, AOAs will not provide effective in-depth analyses 
and DOD will continue to struggle to make informed trade offs and start 
executable programs. 

Recommendations for Executive Action: 

To further strengthen the effectiveness of AOAs in helping DOD 
establish sound business cases for major weapon programs, we recommend 
that the Secretary of Defense take the two following actions: 

* Establish specific criteria and guidance for how AOAs should be 
conducted, including how technical and other programmatic risks should 
be assessed and compared. 

* Ensure that AOAs are completed and approved before program 
requirements--key performance parameters and attributes--are finalized 
and approved. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, DOD concurred with our 
recommendations. DOD's response is reprinted in appendix II. DOD stated 
in response to our first recommendation that it had made significant 
progress in establishing criteria and guidance for conducting AOAs, and 
in defining the relationship/role of the AOA in both the acquisition 
and capabilities determination processes. DOD indicated that the role 
of the AOA has been defined in recently revised acquisition policy 
(Department of Defense Instruction 5000.02, dated Dec. 2, 2008) and 
capabilities policy (Chairman, Joint Chiefs of Staff Instruction 
3170.01G, dated Mar. 1, 2009). While we agree that promising 
improvements have been made in revising the policies, they do not go 
far enough in providing specific criteria and guidance for how AOAs 
should be conducted. Without such direction, there is a risk that AOAs 
will continue to provide limited assessments of weapon system options, 
and DOD will initiate programs without sound, executable business 
cases. In concurring with our second recommendation--that AOAs be 
completed before requirements are finalized--DOD pointed out that under 
its revised acquisition policy, AOAs are now required to be completed 
before the formal initiation of an acquisition program. We agree that 
the policy should help improve the timing of AOAs so that they are 
conducted at an early point in the acquisition process and provide an 
opportunity for trade offs to take place. However, establishing and 
approving requirements is another key step required for initiating an 
acquisition program and this is done under a separate process--the 
Joint Capabilities Integration and Development System. We believe that 
DOD needs to take steps to ensure that program requirements are not 
finalized before the AOA is completed and that the results of the AOA 
are used to inform the setting of requirements. 

DOD also provided technical comments, which we incorporated where 
appropriate. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to the Secretary of Defense; the Secretaries of the Air Force, Army, 
and Navy; and interested congressional committees. This report will 
also be available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you have any questions about this report or need additional 
information, please contact me at (202) 512-4841 or sullivanm@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. GAO staff who 
made major contributions to this report are listed in appendix IV. 

Sincerely yours, 

Signed by: 

Michael J. Sullivan, Director: 
Acquisition and Sourcing Management: 

[End of section] 

Appendix I: Scope and Methodology: 

To assess whether analyses of alternatives (AOA) have been effective in 
identifying the most promising options and providing a sound rationale 
for program initiation, we analyzed data and documents for Acquisition 
Category (ACAT) I programs that have been initiated between fiscal 
years (FY) 2003 and 2008 and were in the Department of Defense's (DOD) 
FY 2008 Major Defense Acquisition Program (MDAP) list. The relevant 
policy that governs the AOA process for these programs, DOD Instruction 
5000.2 (Operation of the Defense Acquisition System), was revised by 
DOD in May of 2003 and revised again in December of 2008 to become DOD 
Instruction 5000.02. As a result, we used the May 2003 DOD Instruction 
to assess the AOAs. Using DOD's FY 2008 MDAP list and Milestone B dates 
provided by DOD, we identified 34 ACAT I programs that had been 
initiated, or started system development and production, between 2003 
and 2008. Programs that had been initiated between 2003 and 2008 but 
were not in the FY 2008 MDAP list, such as programs terminated before 
2008, were not included in the analysis. We collected AOA full reports, 
executive summaries, guidance documents, and study plans when 
available, from program officials. Program officials also responded to 
data collection surveys we distributed through service action officers 
to gather information about their programs' AOA, guidance, capability 
documents, and how the AOA led to changes to the program concept. An 
official for the Cobra Judy Replacement program responded to the 
survey, but officials did not respond to several phone calls and e- 
mails requesting additional documentation, so this program was not 
included in the analysis. In addition, because the Combat Search and 
Rescue Replacement Vehicle (CSAR-X) program did not start development. 

Of the remaining 32 programs, 10 programs did not have AOAs. Whether a 
program had an AOA or not was determined through analysis of program 
documents and survey responses. For the 22 programs that had AOAs, 
program documents and survey data were reviewed to determine the scope 
of the AOAs and whether the AOA assessed technology and integration 
risks. An AOA's scope was assessed to be narrow if the AOA examined 2 
to 5 alternatives within a single concept and assessed to be broad if 
the AOA examined 8 to 26 alternatives within a single concept or 
multiple concepts. An AOA was assessed to have not completed any risk 
analyses for its alternatives when it made no mention of risks in the 
entire AOA report; assessed to be limited if the risk analyses were not 
completed for all of the alternatives, if integration risks were not 
examined, or if the risk analyses were not emphasized in the 
conclusions and recommendations; and assessed to be adequate if 
technical and integration risks were analyzed and compared for all of 
the alternatives. We followed up with some program officials through 
phone calls and e-mails for additional information. To assess how the 
quality of AOAs correlates with programs' outcomes, we also collected 
program and cost data from DOD's Selected Acquisition Reports and GAO's 
Annual Assessments of Selected Weapon Programs. Programs with less than 
10 percent cost growth were considered to have low cost growth, 
programs with 10 to 24 percent cost growth were considered to have 
moderate cost growth, and programs with 25 percent or more cost growth 
were considered to have high cost growth. Programs with less than 7 
months of delay in initial operational capability or acquisition cycles 
were considered to have low schedule growth, programs with 7 to12 
months of delay in initial operational capability or acquisition cycles 
were considered to have moderate schedule growth, and programs with 
greater than 12 months of delay in initial operational capability or 
acquisition cycles were considered to have high schedule growth. The 32 
programs we reviewed accounted for one third of the 96 programs in 
DOD's 2008 Major Defense Acquisition Program portfolio and 
approximately 22 percent of the total planned funding commitments. 

To identify the factors that have affected the scope and quality of 
AOAs, we reviewed program documents, analyzed data from the survey, and 
reviewed DOD policy. We reviewed Initial Capabilities Documents (ICD) 
gathered from the Joint Staff's Knowledge Management/Decision Support 
tool and AOAs to determine how preferred solutions were carried from 
the requirements-generation process to the acquisition process. To 
determine how program schedules affected AOA scope and methodology, we 
analyzed AOA documents, program milestone dates, and AOA completion 
dates. To assess how DOD study guidance affected the quality of AOAs, 
we analyzed whether DOD provided guidance through survey responses and 
followed up with DOD to confirm those responses. We also reviewed 
regulations and policies issued by the Joint Staff, the military 
services, and DOD, as well as other DOD-produced documentation related 
to AOAs. 

To determine what additional actions may be needed to address the 
limitations in the AOA process, we analyzed relevant DOD policies and 
federal statutes, including DOD Instruction 5000.2 (May 2003), DOD 
Instruction 5000.02 (December 2008), the Chairman of the Joint Chiefs 
of Staff Manual (CJCSM) 3170.01C (May 2007), CJCSM 3170.01 (March 
2009), and Section 2366a of Title 10, United States Code. 

In researching all three objectives, we interviewed officials from the 
U.S. Army G3; U.S. Army Training and Doctrine Command Analysis Center 
(TRAC); U.S. Army Capabilities Integration Center (ARCIC); U.S. Air 
Force Office of Aerospace Studies; Office of the Assistant Secretary 
for Acquisition, Deputy Assistant Secretary of the Air Force for 
Science, Technology, and Engineering; Air Force Acquisitions - Global 
Reach; Deputy Assistant Secretary of the Navy, Acquisition and 
Logistics Management (A&LM); Deputy Directorate for Antiterrorism and 
Homeland Defense, J-34, Joint Staff; Office of the Secretary of 
Defense, Acquisition, Technology & Logistics; Office of the Secretary 
of Defense, Program Analysis and Evaluation; Office of the Deputy Under 
Secretary of Defense for Science and Technology (Acquisition and 
Technology)/Systems and Software Engineering; Armed Reconnaissance 
Helicopter Product Manager's Office; U.S. Army Aviation Center; Deputy 
Assistant Secretary of the Navy, Ship Programs; Littoral Combat Ship 
Program Office; Marine Corps Combat Development Command; Office of the 
Chief of Naval Operations, Deputy Chief of Naval Operations, 
Integration of Capabilities and Resources (N8), Director of Warfare 
Integration (N8F), Director of Surface Warfare (N86); Air Combat 
Command/A8I (Requirements), Secretary of the Air Force Technical and 
Analytical Support; and the Naval Surface Warfare Center, Dahlgren. 

We conducted this performance audit from June 2008 to September 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
Acquisition, Technology And Logistics: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 

September 3, 2009: 

Mr. Michael J. Sullivan: 
Director, Acquisition and Sourcing Management: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Mr. Sullivan: 

This is the Department of Defense (DoD) response to the GAO draft report
GAO-09-665, "Defense Acquisitions: Many Analyses of Alternatives Have 
Not Provided a Robust Assessment of Weapon System Options," dated 
August 5, 2009 (GAO Code 120747). 

The DoD concurs with both recommendations. The rationale for the DoD's 
position is enclosed. 

We appreciate the opportunity to comment on the draft report. My point 
of contact for this effort is Mr. Ron Woods, 703-697-8183, 
ronald.woods@osd.mil. 

Sincerely, 

Signed by: 

David G. Ahern: 
Director: 
Portfolio Systems Acquisition: 

Enclosure: As stated: 

[End of letter] 

GAO Draft Report Dated August 5, 2009: 
GAO-09-665 (GAO Code 120747): 

"Defense Acquisitions: Many Analyses of Alternatives Have Not
Provided a Robust Assessment of Weapon System Options" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
establish specific criteria and guidance for how analyses of 
alternatives should be conducted, including how technical and other 
programmatic risks should be assessed and compared. 

DOD Response: Concur. DoD has already made significant progress in 
establishing the criteria and guidance for the conduct of an Analysis 
of Alternatives (AoA), and in defining the relationship/role of the AoA 
in both the acquisition and capabilities validation processes. The 
emergent role of the AoA has been defined as providing the key 
analytical foundation for assessing the utility of options available to 
the Department in addressing validated capability requirements by 
conducting an analysis that trades off cost, schedule, performance and 
risk (both mission and programmatic — to include technical risk) among 
the various alternative options. This role has been codified in the 
recently signed Department of Defense Instruction 5000.02, "Operation 
of the Defense Acquisition System," dated December 2, 2008, and in the 
Chairman, Joint Chiefs of Staff Instruction, "Joint Capabilities 
Integration and Development System," dated March 1, 2009. The 
Department is continuing to revise policy and procedures regarding the 
AoA in response to the recently enacted Weapon System Acquisition 
Reform Act of 2009. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
ensure that analyses of alternatives are completed and approved before 
program requirements-key performance parameters and attributes-are 
finalized and approved. 

DOD Response: Concur. Guidance for the AoA, with the Director/Cost 
Assessment and Program Evaluation (D,CAPE) leading the development of 
the guidance, has become an accepted aspect of DoD guidance. The 
Material Development Decision (MDD) is now the mandatory entry point 
into the acquisition system with an approved Initial Capabilities 
Document and D,CAPE drafted AoA guidance as necessary entry conditions. 
The MDD occurs before the formal initiation of an acquisition program. 
The completed AoA is now required before the initiation of an 
acquisition program and should provide an analytically relevant process 
for assessing military utility, affordability and risk. 
[End of section] 

Appendix III: Key Characteristics of AOAs: 

We surveyed 32 major defense acquisition programs on their analyses of 
alternatives process and outputs. Ten of the programs did not conduct 
AOAs. The following table provides characteristics of the 22 programs 
that conducted AOAs.[Footnote 19] 

Table 8: Key Characteristics of Programs That Conducted AOAs: 

Program: AGM-88E AARGM - AGM-88E Advanced Anti-Radiation Guided Missile 
(AARGM) Program; 
Acquisition category: IC; 
Service sponsor: Navy; 
Organization conducting AOA: Center for Naval Analyses (CNA); Naval Air 
Warfare Center, China Lake; 
Type of organization: FFRDC and Service; 
Length of AOA: (in months): Cannot be Determined; 
Joint: No; 
Months between AOA completion and program start: Cannot be Determined. 

Program: AB3 - Apache Block III; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 6; 
Joint: No; 
Months between AOA completion and program start: 18. 

Program: ARH - Armed Reconnaissance Helicopter; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 6; 
Joint: No; 
Months between AOA completion and program start: 6. 

Program: BAMS - Broad Area Maritime Surveillance; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: MITRE Corporation; 
Type of organization: FFRDC; 
Length of AOA: (in months): 15; 
Joint: No; 
Months between AOA completion and program start: 55. 

Program: CVN 21 - Next Generation Nuclear Aircraft Carrier; 
Acquisition category: ID; 
Service sponsor: OSD with Navy Support; 
Organization conducting AOA: Center for Naval Analyses (CNA); 
Type of organization: FFRDC; 
Length of AOA: (in months): 46; 
Joint: Yes; 
Months between AOA completion and program start: 52. 

Program: DDG 1000 - ZUMWALT CLASS Destroyer; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Center for Naval Analyses (CNA); 
Type of organization: FFRDC; 
Length of AOA: (in months): 39; 
Joint: No; 
Months between AOA completion and program start: 92. 

Program: E-2D AHE - E-2D Advanced Hawkeye; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Naval Research Lab and Whitney, Bradley & 
Brown, Inc.; 
Type of organization: Service and; Contractor; 
Length of AOA: (in months): 32; 
Joint: No; 
Months between AOA completion and program start: 7. 

Program: EA-18G - Electronic attack variant of the F/A-18 aircraft; 
Acquisition category: ID; 
Service sponsor: OSD with Navy Support; 
Organization conducting AOA: Various, including military and government 
agencies as well as the Johns Hopkins University Applied Physics Lab; 
Type of organization: Multiple Services, FFRDC and Contractor; 
Length of AOA: (in months): 29; 
Joint: Yes; 
Months between AOA completion and program start: 24. 

Program: FCS - Future Combat Systems; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 15; 
Joint: No; 
Months between AOA completion and program start: 0. 

Program: CH-53K - Heavy Lift Replacement Program; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Booz Allen Hamilton; 
Type of organization: Contractor; 
Length of AOA: (in months): 5; 
Joint: No; 
Months between AOA completion and program start: 27. 

Program: JCA - Joint Cargo Aircraft; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 9; 
Joint: No; 
Months between AOA completion and program start: 24. 

Program: JLENS - Joint Land Attack Cruise Missile Defense Elevated 
Netted Sensor System; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 17; 
Joint: No; 
Months between AOA completion and program start: 34. 

Program: JTRS AMF - Joint Tactical Radio System Airborne & Maritime/ 
Fixed Station; 
Acquisition category: ID; 
Service sponsor: ASD NII and the Joint Staff; 
Organization conducting AOA: Institute for Defense Analyses (IDA) and 
the Johns Hopkins Applied Physics Lab; 
Type of organization: FFRDC and University Affiliated Research Center; 
Length of AOA: (in months): 13; 
Joint: Yes; 
Months between AOA completion and program start: 26. 

Program: LCS - Littoral Combat Ship; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Naval Surface Warfare Center, Dahlgren; 
Type of organization: Service; 
Length of AOA: (in months): 10; 
Joint: No; 
Months between AOA completion and program start: 1. 

Program: LHA REPLACEMENT - New Amphibious Assault Ship; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Center for Naval Analyses (CNA); 
Type of organization: FFRDC; 
Length of AOA: (in months): 15; 
Joint: No; 
Months between AOA completion and program start: 40. 

Program: LUH - Light Utility Helicopter; 
Acquisition category: IC; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 4; 
Joint: No; 
Months between AOA completion and program start: 18. 

Program: MPS - Mission Planning System; 
Acquisition category: ID; 
Service sponsor: Air Force; 
Organization conducting AOA: Air Force Air Combat Command; 
Type of organization: Service; 
Length of AOA: (in months): 14; 
Joint: No; 
Months between AOA completion and program start: 54. 

Program: MUOS - Mobile User Objective System; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Aerospace Corporation; 
Type of organization: FFRDC; 
Length of AOA: (in months): 28; 
Joint: No; 
Months between AOA completion and program start: 30. 

Program: P-8A - Poseidon Program; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: Center for Naval Analyses (CNA); 
Type of organization: FFRDC; 
Length of AOA: (in months): 19; 
Joint: No; 
Months between AOA completion and program start: 28. 

Program: SDB I - Small Diameter Bomb Increment I; 
Acquisition category: IC; 
Service sponsor: Air Force; 
Organization conducting AOA: Air Combat Command with support from 
Boeing, Raytheon, Lockheed Martin, and Northrup Grumman; 
Type of organization: Service; contractors; 
Length of AOA: (in months): 28; 
Joint: Yes; 
Months between AOA completion and program start: 35. 

Program: VH-71 - Presidential Helicopter Fleet Replacement Program; 
Acquisition category: ID; 
Service sponsor: Navy; 
Organization conducting AOA: 
Center for Naval Analyses (CNA); 
Type of organization: FFRDC; 
Length of AOA: (in months): 17; 
Joint: No; 
Months between AOA completion and program start: 15. 

Program: WIN-T - Warfighter Information Network-Tactical; 
Acquisition category: ID; 
Service sponsor: Army; 
Organization conducting AOA: Army's TRADOC Analysis Center (TRAC); 
Type of organization: Service; 
Length of AOA: (in months): 25; 
Joint: No; 
Months between AOA completion and program start: 16 months after MS B. 

Source: GAO. 

[End of table] 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, the following individuals made 
key contributions to this report: John Oppenheim (Assistant Director), 
Martin G. Campbell, James Kim, John Krump, Claire Li, Guisseli Reyes- 
Turnell, and Tatiana Winger. 

[End of section] 

Footnotes: 

[1] GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, 
[hyperlink, http://www.gao.gov/products/GAO-09-326SP] (Washington, 
D.C.: Mar. 30, 2009). 

[2] National Research Council, Pre-Milestone A and Early Phase Systems 
Engineering: A Retrospective Review and Benefits for Future Air Force 
Systems Acquisition (Washington, D.C.: 2008). 

[3] Major defense acquisition programs, referred to as acquisition 
category (ACAT) I programs, are defined as those that are expected to 
exceed $365 million (in fiscal year 2000 constant dollars) in research, 
development, test, and evaluation (RDT&E) or $2.19 billion (in fiscal 
year 2000 constant dollars) in procurement costs. For ACAT ID programs, 
the Under Secretary of Defense for Acquisition, Technology, and 
Logistics serves as the Milestone Decision Authority (MDA) and 
determines whether the program can proceed to the next phase of 
acquisition while, for ACAT IC programs, the MDA is the head of the DOD 
component or, if delegated, the service Component Acquisition 
Executive. 

[4] We selected programs that started after 2003, to correspond with 
significant revisions that had been made to DOD's acquisition policy at 
that time. 

[5] DOD's acquisition policy covering the time frame of our analysis, 
including guidance for conducting AOAs, is contained in DOD Instruction 
5000.2, May 2003 revision to the Operation of the Defense Acquisition 
System. If a program does not require development, it may enter at 
Milestone C. DOD Instruction 5000.2 was revised and reissued as DOD 
Instruction 5000.02 in December 2008. 

[6] While DOD Instruction 5000.2 provides general policy for conducting 
AOAs, it does not specify how AOAs should be conducted. The Air Force 
has developed detailed guidance for conducting AOAs. 

[7] Other key elements required for Milestone B include the technology 
readiness assessment, independent cost estimate, acquisition strategy, 
and Capability Development Document (CDD). 

[8] GAO, Defense Acquisitions: Sound Business Case Needed to Implement 
Missile Defense Agency's Targets Program, [hyperlink, 
http://www.gao.gov/products/GAO-08-1113] (Washington, D.C.: Sept. 26, 
2008), Defense Acquisitions: Improved Business Case Is Needed for 
Future Combat System's Successful Outcome, [hyperlink, 
http://www.gao.gov/products/GAO-06-367] (Washington, D.C.: Mar. 14, 
2006), and Tactical Aircraft: Air Force Still Needs Business Case to 
Support F/A-22 Quantities and Increased Capabilities, [hyperlink, 
http://www.gao.gov/products/GAO-05-304] (Washington, D.C.: Mar. 15, 
2005). 

[9] Although DOD has no specific criteria for how many alternatives are 
to be considered in an AOA, Air Force guidance indicates that AOAs 
should consider a comprehensive set of alternatives representing all 
reasonable solutions. For the AOAs we reviewed, we categorized 
"limited" number of alternatives as AOAs that examined two to five 
alternatives within a single weapon system concept and "many" 
alternatives as AOAs that examined eight or more alternatives within a 
single weapon system concept or alternatives across multiple weapon 
system concepts. 

[10] While DOD acquisition policy (DODI 5000.2) indicates that an AOA 
shall assess technology risk and maturity, Air Force guidance for 
conducting AOAs advocates that programmatic and operational risks 
should be considered as well. The Air Force guidance indicates that AOA 
study teams should determine, as a first step in conducting a risk 
analysis, what technical, programmatic, and operational factors are 
relevant to each alternative. For example, open architecture, schedule, 
overarching dependencies, and maintenance requirements may be important 
risk factors for some AOAs. 

[11] We defined "limited" risk assessments as AOAs that did not examine 
risks at all or examined only one risk factor such as technology 
maturity. We defined AOAs as having a more robust risk assessment if 
they examined multiple risk factors such as technology, integration, 
and schedule. 

[12] The AOA for the FCS program examined the following seven 
alternatives: Base Case - Heavy Force, Base Case - Light Force, Base 
Case - Interim Force, FCS - Block II, FCS - Block I, FCS - Increment 1, 
Product Improved Stryker Brigade Combat Team. 

[13] Defense Science Board, Buying Commercial: Gaining the Cost/ 
Schedule Benefits for Defense Systems. Report of the Defense Science 
Board Task Force on Integrating Commercial Systems into the DOD, 
Effectively and Efficiently (Washington, D.C.: February 2009). 

[14] The termination of the Comanche helicopter program provided the 
Army the opportunity to redirect its resources to fund a broad range of 
programs and activities, including the Armed Reconnaissance Helicopter, 
which is intended to address numerous shortcomings in the 
reconnaissance helicopter fleet. 

[15] GAO, Defense Acquisitions: Plans Need to Allow Enough Time to 
Demonstrate Capability of First Littoral Combat Ships, [hyperlink, 
http://www.gao.gov/products/GAO-05-255] (Washington, D.C.: Mar. 1, 
2005). 

[16] The Navy conducted an Analysis of Multiple Concepts, one of the 
studies required by DOD acquisition policy in 2002 for program 
initiation, which was subsequently approved by OSD PA&E as a tailored 
AOA. 

[17] DOD's acquisition policy is contained in DOD Instruction 5000.2, 
the Operation of the Defense Acquisition System. 

[18] GAO, Defense Acquisitions: DOD Must Balance Its Needs with 
Available Resources and Follow an Incremental Approach to Acquiring 
Weapon Systems, [hyperlink, http://www.gao.gov/products/GAO-09-431T] 
(Washington, D.C.: Mar. 3, 2009). 

[19] We removed the CSAR-X program from our analysis because the 
program was terminated before contracts were awarded. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: