This is the accessible text file for GAO report number GAO-11-150 
entitled 'Information Technology: Better Informed Decision Making 
Needed on Navy's Next Generation Enterprise Network Acquisition' which 
was released on March 11, 2011. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

March 2011: 

Information Technology: 

Better Informed Decision Making Needed on Navy's Next Generation 
Enterprise Network Acquisition: 

GAO-11-150: 

GAO Highlights: 

Highlights of GAO-11-150, a report to congressional requesters. 

Why GAO Did This Study: 

The Department of the Navy (DON), a major component of the Department 
of Defense (DOD), has launched its Next Generation Enterprise Network 
(NGEN) program to replace the Navy Marine Corps Intranet (NMCI) 
program. NGEN capabilities, such as secure transport of voice and 
data, data storage, and e-mail, are to be incrementally acquired 
through multiple providers. As planned, the first increment is 
expected to provide comparable NMCI capabilities, additional 
information assurance, and greater DON network control, at a cost of 
about $50 billion through fiscal year 2025. Given the size, 
importance, and complexity of NGEN, GAO was asked to determine whether 
DON has sufficiently analyzed alternative acquisition approaches and 
has a reliable schedule for executing the program, and whether program 
acquisition decisions have been performance- and risk-based. To do 
this, GAO reviewed the NGEN analysis of alternatives, integrated 
master schedule, and key milestone decisions. 

What GAO Found: 

What GAO Found
DON did not sufficiently analyze alternative acquisition approaches 
for NGEN because the alternatives analysis contained key weaknesses, 
and none of the alternatives assessed match the current acquisition 
approach. Specifically, the cost estimates for the respective 
alternatives were not reliable because they were not substantially 
accurate, and they were neither comprehensive nor credible. Further, 
the operational effectiveness analysis, the other key aspect of an 
analysis of alternatives, did not establish and analyze sufficient 
measures for assessing each alternative’s ability to achieve program 
goals and deliver program capabilities. Moreover, the acquisition 
approach that DON is actually pursuing was not one of the alternatives 
assessed in the analysis, and it is riskier and potentially costlier 
than the alternatives analyzed because it includes a higher number of 
contractual relationships. According to program officials, the 
analysis reflects the most that could be done in the time that was 
available to complete it, and they do not view the alternative 
selected as materially different from the assessed alternatives, even 
though it is about $4.7 billion more costly. (See table for comparison 
of alternatives.) 

Table: Comparison of NGEN Alternative Approaches: 

Contractual relationships: 
Status quo: 3; 
Alt. 2: 3; 
Alt. 3 variant: 10; 
Alt. 3: 15; 
Current approach: 21. 

Estimated cost[A]: 
Status quo: $10.3; 
Alt. 2: $10.8; 
Alt. 3 variant: $10.8; 
Alt. 3: $10.7; 
Current approach: $15.6. 

Relative risk: 
Status quo: Least; 
Alt. 2: More; 
Alt. 3 variant: Greater; 
Alt. 3: Greatest; 
Current approach: Undetermined. 

Sources: DON data (status quo and alternatives 2, 3 variant, and 3); 
GAO analysis of DON data (current approach). 

[A] Fiscal year 2011-2015 in billions. 

[End of table] 

DON does not have a reliable schedule for executing NGEN. Only two of 
the four subschedules that GAO reviewed, each of which help form the 
master schedule, adequately satisfied any of the nine practices that 
are associated with developing and maintaining a reliable schedule. 
These weaknesses have contributed to delays in key program milestones. 
During the course of GAO’s review, DON stated that action was taken to 
address some, but not all, of these weaknesses. According to program 
officials, schedule estimating was constrained by staffing limitations. 

NGEN acquisition decisions were not always performance- and risk-
based. In particular, the program was approved in the face of known 
performance shortfalls and risks. For example, the program was 
approved at a key acquisition review despite the lack of defined 
requirements, which was recognized as a risk that would impact the 
completion of other key documents, such as the test plan. This risk 
was later realized as a critical issue. According to program 
officials, the decisions to proceed were based on their view that they 
had sufficiently mitigated known risks and issues. 

Collectively, these weaknesses mean that DON does not have a 
sufficient basis for knowing that it is pursuing the best approach for 
acquiring NGEN capabilities and the program’s cost and schedule 
performance is unlikely to track to estimates. 

What GAO Recommends: 

GAO is recommending that DOD limit further investment in NGEN until it 
conducts an interim review to reconsider the selected acquisition 
approach and addresses issues discussed in this report. In its 
comments, DOD stated that it did not concur with the recommendation to 
reconsider its acquisition approach; GAO maintains that without doing 
so, DOD cannot be sure it is pursuing the most cost-effective approach. 

View [hyperlink, http://www.gao.gov/products/GAO-11-150] or key 
components. For more information, contact Valerie C.Melvin at (202) 
512-6304 or melvinv@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Current NGEN Acquisition Approach Is Not Grounded in a Reliable AOA: 

NGEN's Schedule Does Not Provide a Reliable Basis for Program 
Execution, and Program Delays Are Occurring: 

NGEN Acquisition Decisions Were Not Always Performance-and Risk-Based: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Early Transition Activities: 

Table 2: Planned NGEN Program Contracts: 

Table 3: Organizations Responsible for NGEN Acquisition Oversight and 
Management: 

Table 4: Description of DAS Phases: 

Table 5: Purpose of DON Gate Reviews: 

Table 6: Summary of Four Alternatives in the NGEN AOA: 

Table 7: Measures Related to Overseeing, Operating, and Managing NGEN: 

Table 8: DON's Qualitative Assessment of Alternatives' Ability to Meet 
NGEN Capabilities and Goals: 

Table 9: Comparison of NGEN AOA Alternatives and Current Approach: 

Table 10: Description of Scheduling Best Practices: 

Table 11: Summary Results of Selected NGEN Schedules' Satisfaction of 
Schedule Estimating Practices: 

Table 12: Delays in Key NGEN Program Events and Milestones, as of 
August 2010: 

Table 13: Naval Probability of Program Success Metric Definitions: 

Figures: 

Figure 1: Simplified View of OSD DAS and DON Gate Review Process: 

Figure 2: NGEN Review Schedule: 

Figure 3: Contractual Relationships in Current NGEN Approach: 

Figure 4: NGEN Gate 4 Probability of Program Success Assessment 
(November 2009): 

Figure 5: NGEN Gate 5 Probability of Program Success Assessment 
(October 2010): 

Abbreviations: 

AOA: analysis of alternatives: 

CAPE: Cost Assessment and Program Evaluation: 

CIO: chief information officer: 

DAS: Defense Acquisition System: 

DOD: Department of Defense: 

DON: Department of the Navy: 

IT: information technology: 

MAIS: Major Automated Information System: 

NGEN: Next Generation Enterprise Network: 

NMCI: Navy Marine Corps Intranet: 

OSD: Office of the Secretary of Defense: 

USD (AT&L): Under Secretary of Defense (Acquisition, Technology, and 
Logistics): 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

March 11, 2011: 

The Honorable Claire McCaskill: 
Chairman: 
The Honorable Kelly Ayotte: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

The Honorable Richard Burr: 
United States Senate: 

The Department of the Navy (DON) established the Next Generation 
Enterprise Network (NGEN) program in 2007 to replace the Navy Marine 
Corps Intranet (NMCI), which is provided through a DON-wide network 
services contract. As envisioned, NGEN capabilities, including secure 
transport of voice and data, data storage, and e-mail, are to be 
incrementally acquired through multiple providers. The first increment 
is to include capabilities comparable to NMCI, as well as enhanced 
information assurance and increased government control over network 
operations. To date, NGEN has reportedly spent about $432 million on 
work associated with the transition from NMCI. The first increment is 
to be fully operational in March 2014 and is to cost approximately $50 
billion through 2025. 

As agreed, our objectives were to determine whether (1) DON has 
sufficiently analyzed alternative approaches for acquiring NGEN, (2) 
DON has a reliable schedule for executing NGEN, and (3) acquisition 
decisions have been performance-and risk-based. To accomplish our 
objectives, we analyzed the NGEN alternatives analysis report and 
underlying support, the program's integrated master schedule, program 
performance assessments and risk reports, and executive acquisition 
decision briefings and meeting minutes, among other things. We 
conducted this performance audit from October 2009 to February 2011, 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. Additional 
details on our objectives, scope, and methodology are discussed in 
appendix I. 

Background: 

DON is a major component of the Department of Defense (DOD), 
consisting of the Navy and the Marine Corps. It is a large and complex 
organization, whose primary mission is to organize, train, maintain, 
and equip combat-ready naval forces capable of winning wars, deterring 
aggression by would-be foes, preserving freedom of the seas, and 
promoting peace and security. To support this mission, DON performs a 
variety of interrelated and interdependent information technology (IT)-
dependent functions. In fiscal year 2010, DON's IT budget was 
approximately $7.4 billion, for 971 investments. NGEN is one such 
system investment. 

Overview of NGEN: 

NGEN is to provide secure data and IT services, such as data storage, 
e-mail, and video-teleconferencing, to the Navy and the Marine Corps. 
NGEN is also intended to provide the foundation for DON's future Naval 
Networking Environment.[Footnote 1] DON is acquiring NGEN through 
multiple providers (contractors) to replace and improve the enterprise 
network and services provided by NMCI. It is to be developed 
incrementally, with the first increment to provide comparable NMCI 
capabilities,[Footnote 2] additional information assurance, and 
increased government control of the network. Future increments have 
yet to be defined. The program's preliminary life cycle cost estimate 
(through fiscal year 2025) for the first increment is about $50 
billion. As of September 30, 2010, the NGEN program had reportedly 
spent about $432 million. 

To bridge the time frame between the end of the NMCI contract and the 
full transition to NGEN, DON awarded a $3.7 billion continuity of 
services contract in July 2010 to the NMCI service provider, Hewlett 
Packard Enterprise Services. In addition to providing continuity of 
network services, the contract includes transition services and 
transfer to DON of NMCI infrastructure and intellectual property, 
[Footnote 3] as the NGEN contracts are to require use of the NMCI 
infrastructure and access to processes, procedures, and technical 
data. The continuity of services contract is scheduled to run from 
October 2010 through April 2014. 

To reduce the risk during the transition period from NMCI to NGEN, DON 
is currently performing eight early transition activities. The 
activities are discrete efforts intended to establish government 
management capabilities, allow for greater participation in 
operational decisions, and help expedite the transition time. Table 1 
describes each of these activities. 

Table 1: Early Transition Activities: 

Early transition activity: Information Technology Service Management 
Process Development; 
Start date: October 2008; 
End date: May 2011; 
Cost: $20.5 million; 
Description: Develop Information Technology Infrastructure Library[A] 
version 3-based service strategies, processes, and procedures. 

Early transition activity: Contractor Technical Representative 
Workforce Reconstitution; 
Start date: January 2009; 
End date: April 2011; 
Cost: $3.3 million; 
Description: Conduct job task analysis and assess learning tools for 
contractor technical representatives; develop enterprise-wide position 
descriptions and occupational standards for training, advancement, 
criteria, and performance objectives. 

Early transition activity: Comprehensive Facilities and Infrastructure 
Inventory; 
Start date: January 2009; 
End date: December 2010; 
Cost: $12.0 million; 
Description: Evaluate, analyze, and validate current NMCI 
infrastructure inventory consisting of technical data, assets, 
configuration items, and system components. 

Early transition activity: Defense Information System Network Core 
Extension Phase 1 and Maritime Operation Center Implementation; 
Start date: April 2008; 
End date: May 2011; 
Cost: $6.7 million; 
Description: Bring consistent wide area network connectivity from the 
Defense Information System Network to eight major nodes at fleet 
headquarters (Norfolk, Virginia, and Pearl Harbor, Hawaii). 

Early transition activity: Global Network Operations Command and 
Control Workforce Establishment; 
Start date: October 2008; 
End date: July 2011; 
Cost: $14.3 million; 
Description: Develop the personnel, processes, and tool requirements, 
and organizational analysis and alignment. 

Early transition activity: Wide Area Network and Enterprise Services 
Upgrade; 
Start date: April 2009; 
End date: August 2012; 
Cost: $46.0 million; 
Description: Demonstrate network operational control capability and 
validate the NGEN System Design baseline through early implementation. 

Early transition activity: Enterprise Tools Strategy and 
Implementation/Integration; 
Start date: April 2009; 
End date: May 2011; 
Cost: $56.6 million; 
Description: Analyze current tool capabilities to support information 
technology service management processes, and develop design 
requirements and tool integration specifications. 

Early transition activity: Nonclassified Internet Protocol Router 
Network Migration - Marine Air-Ground Task Force IT Support Centers 
East Pilot; 
Start date: October 2009; 
End date: August 2011; 
Cost: $12.9 million; 
Description: Assess the transition of base area network, local area 
network, and end-user equipment for about 1,200 users from the 
continuity of services contract to the government-owned/government-
operated NGEN environment. 

Source: GAO analysis of DON data. 

[A] Information Technology Infrastructure Library is a framework 
intended to align an organization's IT services with its 
organizational needs. It consists of service strategy, design, 
transition, operation and continual improvement best practice guidance. 

[End of table] 

To deliver NGEN capabilities, DON plans to award five contracts. 
[Footnote 4] See table 2 for a description of these contracts. 

Table 2: Planned NGEN Program Contracts: 

Contract: Independent Security Operations, Oversight and Assessment 
Support; 
Award date: April 2011; 
Purpose: Provide independent third-party security assessments of NMCI 
and NGEN. The contract is planned to include the option to support all 
DON networks within the naval network environment. 

Contract: Transport Services; 
Award date: May 2012; 
Purpose: Provide for the operation and sustainment of the transport 
infrastructure, associated services, and level-of-effort support for 
those services. The contract is planned to include technology refresh 
of cable plant, routers, and switches; some leasehold improvements; 
and moveable infrastructure associated with local network operations. 

Contract: End-User Hardware; 
Award date: December 2011; 
Purpose: Provide end-user equipment such as computers, monitors, and 
keyboards. 

Contract: Enterprise Software Licenses; 
Award date: March 2012; 
Purpose: Provide software licenses to meet DON-wide requirements. 

Contract: Enterprise Services/Service Coordination; 
Award date: August 2012; 
Purpose: Provide the Enterprise Service Desk, seat services supporting 
end-user devices, and data center services such as storage and e-mail 
along with hardware and software specific to Enterprise Services that 
are not covered under the End-User Hardware and Enterprise Software 
Licenses contracts. As the service coordinator, this contract is 
planned to cover coordination across all NGEN vendors for the 
successful delivery of NGEN services. 

Source: GAO analysis of DON data. 

[End of table] 

According to the NGEN Acquisition Strategy, DON plans to complete the 
Marine Corps' initial transition to NGEN in January 2012 and final 
transition in February 2013. The Navy's initial and final transition 
to NGEN are scheduled to be completed in December 2012 and March 2014, 
respectively.[Footnote 5] 

NGEN Oversight and Management Roles and Responsibilities: 

To manage the acquisition and deployment of NGEN, DON established a 
program management office within the Program Executive Office for 
Enterprise Information Systems. The program office manages the 
program's cost, schedule, and performance and is responsible for 
ensuring that the program meets its objectives. In addition, various 
DOD and DON organizations share program oversight and review 
responsibilities. Table 3 lists key entities and their roles and 
responsibilities. 

Table 3: Organizations Responsible for NGEN Acquisition Oversight and 
Management: 

Entity: Under Secretary of Defense for Acquisition, Technology, and 
Logistics; 
Roles and responsibilities: Serves as the milestone decision 
authority, with overall responsibility for the program, to include 
approving the program to proceed through its acquisition cycle on the 
basis of, for example, the acquisition strategy, an independently 
evaluated economic analysis, and the Acquisition Program Baseline. 

Entity: Director, Cost Assessment and Program Evaluation, Office of 
the Secretary of Defense (OSD CAPE); 
Roles and responsibilities: Issued NGEN analysis of alternatives (AOA) 
guidance, reviewed the AOA study plan, and approved the AOA results. 
Verifies and validates the reliability of cost and benefit estimates 
found in economic analyses. 

Entity: NGEN AOA Advisory Group; 
Roles and responsibilities: Oversaw the conduct of the NGEN AOA. Led 
by the DON Chief Information Officer. 

Entity: Assistant Secretary of the Navy, Research, Development, and 
Acquisition; 
Roles and responsibilities: Serves as DON's acquisition oversight 
organization for the program, to include enforcement of Under 
Secretary of Defense for Acquisition, Technology, and Logistics 
policies and procedures. Determines when all key milestones are ready 
to be submitted to the milestone decision authority. 

Entity: Chief of Naval Operations and Commandant of the Marine Corps; 
Roles and responsibilities: Serve together as resource sponsors for 
NGEN requirements and funding. 

Entity: Department of the Navy, Program Executive Office for 
Enterprise Information Systems; 
Roles and responsibilities: Oversees a portfolio of large-scale 
projects and programs designed to enable common business processes and 
provide standard capabilities. Reviews the acquisition strategy, 
economic analysis, and the Acquisition Program Baseline prior to 
approval by the milestone decision authority. 

Entity: Department of the Navy Chief Information Officer (CIO); 
Roles and responsibilities: Supports DON's planning, programming, 
budgeting, and execution processes by ensuring that the program has 
achievable and executable goals and conforms to financial management 
regulations, and DON, DOD, and federal IT policies in several areas 
(e.g., security, architecture, and investment management); 
works closely with the program office during milestone review 
assessments. Oversaw NGEN AOA, which was led by the Center for Naval 
Analyses and supported by Deloitte Consulting. 

Entity: NGEN Program Management Office; 
Roles and responsibilities: Performs day-to-day program management 
and, as such, is the single point of accountability for managing the 
program's objectives. 

Entity: Space and Naval Warfare Systems Command; 
Roles and responsibilities: Reviewed the NGEN AOA cost analysis. 
Reviews the NGEN program life cycle cost estimates and economic 
analysis. Serves jointly with the Marine Corps Systems Command as the 
NGEN technical authority. 

Entity: Marine Corps Systems Command; 
Roles and responsibilities: Serves jointly with the Space and Naval 
Warfare Systems Command as the NGEN technical authority. 

Source: GAO analysis of DON data. 

[End of table] 

Overview of DOD and DON Acquisition Processes and NGEN Status: 

NGEN is subject to both Office of the Secretary of Defense (OSD) and 
DON Major Automated Information System (MAIS) acquisition policy and 
guidance,[Footnote 6] which require it to comply with Defense 
Acquisition System (DAS)[Footnote 7] requirements. According to these 
requirements, all MAIS programs require a Materiel Development 
Decision prior to entering the first DAS phase. In making this 
decision, the milestone decision authority is to review the Initial 
Capabilities Document, which defines operational goals and needed 
capabilities, and authorizes the phase in which the program is to 
enter the DAS. The system consists of five key program life cycle 
phases and three related milestone decision points. Table 4 provides a 
description of each DAS phase. 

Table 4: Description of DAS Phases: 

Phase: Materiel Solution Analysis; 
Description: Prior to entering this phase the milestone decision 
authority completes a Materiel Development Decision. The purpose of 
the phase is to assess, through an AOA, potential solutions to satisfy 
an approved capability need. 

Phase: Technology Development; 
Description: Prior to entering this phase, the milestone decision 
authority conducts a Milestone A decision review of the proposed 
solution and a draft Technology Development Strategy. The purpose of 
the phase is to determine the appropriate set of technologies to be 
integrated into the solution and to refine user requirements. 

Phase: Engineering and Manufacturing Development; 
Description: Most acquisition programs enter the DAS in this phase. 
Prior to entering the phase, the milestone decision authority 
completes a Milestone B review to approve the Acquisition Strategy and 
Acquisition Program Baseline.[A] The purpose of the phase is to 
develop a system or system increment. The phase has two major efforts: 
Integrated System Design and System Capability and Manufacturing 
Process Demonstration. 
Integrated System Design: This effort includes a post-Critical Design 
Review assessment of design maturity, and issues and risks that could 
result in a breach to the Acquisition Program Baseline or 
substantively impact cost, schedule, or performance. 
System Capability and Manufacturing Process Demonstration: This effort 
is to demonstrate the ability of the system to operate according to 
the approved Key Performance Parameters[B] and ends when the system or 
system increment meets approved requirements. 

Phase: Production and Deployment; 
Description: Prior to entering this phase, the milestone decision 
authority conducts a Milestone C review of the Capability Production 
Document and the Test and Evaluation Master Plan, among other things, 
and authorizes limited deployment to support operational testing. The 
purpose of the phase is to achieve an operational capability that 
satisfies the mission needs, verified through independent operational 
test and evaluation, and to implement the system at all applicable 
locations. During this phase, the milestone decision authority also 
conducts a Full Deployment Decision Review. 

Phase: Operations and Support; 
Description: The purpose of this phase is to operationally sustain the 
system in the most cost-effective manner over its life cycle. 

Source: GAO analysis of DOD documentation. 

[A] According to DOD's acquisition guidebook, an Acquisition Program 
Baseline is a program manager's estimated cost, schedule, and 
performance goals. Goals consist of objective values, which represent 
what the user desires and expects, and threshold values, which 
represent acceptable limits. When the program manager determines that 
a current cost, schedule, or performance threshold value will not be 
achieved, the milestone decision authority must be notified, and a new 
baseline developed, reviewed by decision makers and, if the program is 
to continue, approved by the milestone decision authority. 

[B] Key Performance Parameters are attributes or characteristics of a 
system that are considered critical or essential to the development of 
an effective military capability. 

[End of table] 

In addition to Defense Acquisition System requirements, according to 
DON guidance and policy, all DON MAIS and pre-MAIS programs are 
required to go through a "Two-Pass/Six-Gate" acquisition review 
process.[Footnote 8] The first pass, which consists of Gates 1 through 
3, is focused on requirements development and validation and is led by 
the Chief of Naval Operations or the Commandant of the Marine Corps. 
The second pass, which consists of Gates 4 through 6, is focused on 
developing and delivering a solution via systems engineering and 
acquisition and is led by the Assistant Secretary of the Navy 
(Research, Development, and Acquisition). In addition to meeting 
specific criteria for passing a given gate and proceeding to the next 
gate, all gate reviews are to consider program health (i.e., 
satisfactory cost and schedule performance, known risks, and budget 
adequacy) in deciding whether to proceed. Table 5 lists the key 
purpose of each gate review. 

Table 5: Purpose of DON Gate Reviews: 

Gate: 1; 
Purpose: To approve the Initial Capabilities Document and to validate 
AOA guidance and assumptions. 

Gate: 2; 
Purpose: To review the AOA and approve the preferred alternatives 
resulting from the analysis. 

Gate: 3; 
Purpose: To authorize the Capability Development Document, which 
defines the system's Key Performance Parameters and includes 
information necessary to develop an affordable system or system 
increment, to be submitted to Joint Staff for review, and to approve 
the Concept of Operations. 

Gate: 4; 
Purpose: To approve the System Design Specification, which specifies 
the system requirements and is derived from the Capability Development 
Document. 

Gate: 5; 
Purpose: To approve the release of a request for proposal to industry. 

Gate: 6; 
Purpose: To review overall program health, following award of a 
contract and satisfactory completion of an Integrated Baseline 
Review.[A] Also, to approve the Capability Production Document and 
program health prior to and after Milestone C and the Full Deployment 
Decision Review. 

Source: GAO analysis of DON documentation. 

[A] An Integrated Baseline Review is performed to obtain stakeholder 
agreement on a contractor's Performance Measurement Baseline, which 
represents the cumulative value of planned work and serves as the 
baseline against which variances are calculated. 

[End of table] 

The DAS and DON acquisition phases and decision points for MAIS 
programs are illustrated in figure 1. 

Figure 1: Simplified View of OSD DAS and DON Gate Review Process: 

[Refer to PDF for image: illustration] 

Materiel solution analysis: 
OSD: 
MDD; 
DON: 
1 ICD; 
2 AOA. 

Technical development: 
OSD: 
Milestone A; 
DON: 
3 CDD; 
4 SDS; 
5 RFP. 

Engineering and manufacturing development: 
OSD: 
Post-CDR assessment; 
Milestone C; 
DON: 
6 IBR; 
6 CPD. 

Production and deployment: 
OSD: 
FDDR; 
DON: 
6 Program health; 
6 Program health. 

Operations and support: 

Source: GAO analysis of DOD and DON policy and guidance. 

Note: 
MDD = Materiel Development Decision; 
CDR = Critical Design Review; 
FDDR = Full Deployment Decision Review; 
ICD = Initial Capabilities Document; 
AOA = analysis of alternatives; 
CDD = Capability Development Document; 
SDS = System Design Specification; 
RFP = request for proposal; 
IBR = Integrated Baseline Review; 
CPD = Capability Production Document. 

[End of figure] 

As depicted in figure 2, DON completed a Gate 3 review of NGEN 
requirements in April 2008.[Footnote 9] In April 2009, the DON CIO 
completed the AOA for NGEN increment 1, and at the Gate 2 review the 
same month, the Deputy Chief of Naval Operations (Integration of 
Capabilities and Resources) and the Deputy Marine Corps Commandant for 
Programs and Resources approved the AOA to be submitted to the NGEN 
AOA Advisory Group.[Footnote 10] The advisory group subsequently 
approved the analysis and forwarded it in April 2009 to OSD Cost 
Assessment and Program Evaluation (CAPE), which approved it in 
December 2009. DON conducted a Gate 4 review of its System Design 
Specification in November 2009, and a Gate 5 review of its Transport 
Services request for proposal in October 2010. DON plans to conduct a 
Gate 6 review in July 2011. 

In May 2010, the USD (AT&L) completed the NGEN Materiel Development 
Decision, which designated the first increment of NGEN as a MAIS and 
authorized the program to enter the DAS in the production and 
deployment phase. A Milestone C review is currently planned for August 
2011. In June 2010, the USD (AT&L) approved the current acquisition 
approach. 

Figure 2: NGEN Review Schedule: 

[Refer to PDF for image: illustrated schedule] 

2008: 

January-June: 
DON: 3 Requirement (complete). 

2009: 

February-June: 
DON: 2 AOA (complete). 

September-December: 
DON: 4 SDS (complete). 

2010: 
March-July: 
OSD: MDD (complete). 

August-December: 
DON: 5 RFP (in process). 

2011: 

May-September: 
DON: 6 CPD (planned). 

July-October: 
OSD: Milestone C (planned). 

Source: GAO analysis of DON data. 

Note: 
MDD = Materiel Development Decision; 
AOA = analysis of alternatives; 
SDS = System Design Specification; 
RFP = request for proposal; 
CPD = Capability Production Document. 

[End of figure] 

Current NGEN Acquisition Approach Is Not Grounded in a Reliable AOA: 

An AOA is intended to help identify the most promising acquisition 
approach by comparing alternative solutions' costs and operational 
effectiveness. The NGEN AOA contained key weaknesses in its cost 
estimates and operational effectiveness analysis that impaired its 
ability to inform investment decision making. Further, none of the 
alternatives in this analysis match the current acquisition approach, 
and these differences have not been analyzed to determine the breadth 
of risk that exists. According to DON officials, the AOA reflects the 
most that could be accomplished in the time available to meet an 
imposed deadline. In addition, OSD officials stated that the 
differences between the current approach and the alternatives that 
were assessed are, in their view, not significant. However, the 
current approach is estimated to cost at least $4.7 billion more than 
any of the AOA alternatives. Without sufficient information to 
understand the differences in the relative costs and operational 
effectiveness among alternatives, decision makers lack assurance that 
their selected approach is the most promising and cost-effective 
course of action. 

NGEN AOA Was Not Based on Reliable Cost Estimates: 

According to relevant guidance,[Footnote 11] a key component of an AOA 
is a cost analysis that provides for cost estimates of each 
alternative. As such, cost estimates should be reliable in order to 
provide the basis for informed investment decision making, realistic 
budget formulation, meaningful progress measurement, and 
accountability for results. Our research has identified four 
characteristics of a high-quality, reliable cost estimate: well- 
documented, comprehensive, accurate, and credible.[Footnote 12] 

The NGEN AOA assessed four alternatives.[Footnote 13] All alternatives 
were assumed to deliver the same NMCI capabilities and the technology 
across alternatives was assumed to be substantially the same. The 
primary differences among the alternatives were how NGEN was to be 
acquired, managed, and operated. Table 6 below provides a description 
of each alternative. 

Table 6: Summary of Four Alternatives in the NGEN AOA: 

Alternative: Alternative 1 (status quo); 
Description: A recompete of the NMCI contract, in which the contractor 
was to be responsible for end-to-end integration of services and 
control of the network. 

Alternative: Alternative 2 (enhanced status quo); 
Description: A single contract similar to NMCI but with upgraded terms 
and conditions that were to address known NMCI deficiencies.[A] The 
contractor was to be responsible for end-to-end integration of 
services, and the government would control the network. 

Alternative: Alternative 3 Variant (3V) (segmented[B] approach); 
Description: Multiple contracts with different vendors. The government 
was to be responsible for end-to-end integration of services and 
control of the network. 

Alternative: Alternative 3 (segmented approach); 
Description: A greater number of contracts than alternative 3V. The 
government was to be responsible for end-to-end integration of 
services and control of the network. 

Source: NGEN Analysis of Alternatives report, Center for Naval 
Analyses, April 2009. 

[A] See, for example, GAO, Information Technology: DOD Needs to Ensure 
That Navy Marine Corps Intranet Program Is Meeting Goals and 
Satisfying Customers, GAO-07-51 (Washington, D.C.: Dec. 8, 2006), for 
information on NMCI deficiencies. 

[B] Segments are components of IT services to be delivered for NGEN 
through multiple contracts. 

[End of table] 

The four alternatives' estimated costs for increment 1 from fiscal 
year 2011 to fiscal year 2015 ranged from $10.25 billion (alternative 
1) to $10.84 billion (alternatives 2 and 3V). However, the estimates 
were not reliable because they substantially met only one of the 
characteristics of reliable cost estimates. Specifically, 

* The AOA cost estimates were substantially well-documented. To be 
well-documented, the cost estimates should state the purpose of the 
estimate; provide program background, including a system description; 
provide a schedule for developing the estimates; specify the scope of 
the estimate (in terms of time and what is and is not included); 
disclose key ground rules and assumptions, data sources, calculations 
performed and their results, the estimating methodology and rationale, 
and the results of a risk analysis; and provide a conclusion about 
whether the cost estimate is reasonable. Moreover, this information 
should be captured in such a way that the data used to derive the 
estimate can be traced to their sources. Finally, the cost estimates 
should be reviewed and accepted by management. 

Although the AOA did not sufficiently document the schedule, scope, 
and results of the risk analysis, it defined the purpose of the 
estimate; provided program background (e.g., system description); and 
disclosed ground rules and assumptions, data sources, calculations 
performed and their results, and the estimating methodology. Also, the 
data used to derive the estimates were captured in such a way that 
they could largely be traced to their sources, and the final AOA was 
reviewed and accepted by DON and OSD oversight entities. 

* The AOA cost estimates were not comprehensive. To be comprehensive, 
the cost estimates should include all government and contractor costs 
over the program's full life cycle, from program inception through 
design, development, deployment, and operation and maintenance to 
retirement. They should also provide sufficient detail and reflect all 
cost-influencing ground rules and assumptions. 

However, the cost estimates were not full life cycle costs. Instead, 
they only included government and contractor costs for a 5-year period 
from fiscal year 2011 to fiscal year 2015, covering 2 years of 
continued NMCI services with the current provider, 2 years of 
transition to the new provider(s), and 1 year of NGEN operation and 
maintenance. DON and OSD CAPE officials attributed this to the 
assumption that NGEN increment 1 contracts would have a 5-year period 
of performance and that future NGEN increments might be introduced 
after that period. 

Further, while the estimates were based on a cost element structure 
that was decomposed to a sufficient level of detail, and the 
documentation largely identified ground rules and assumptions, the 
cost estimates did not reflect all assumptions identified in the AOA, 
such as schedule and performance risks associated with (1) 
implementing IT processes, (2) expanding the government workforce, and 
(3) formulating the NGEN contracts. These were significant cost-
influencing risks and thus should have been incorporated into the 
estimates. 

* The AOA cost estimates were not substantially accurate. To be 
accurate, the cost estimates should not be overly conservative or 
optimistic and should be, among other things, based on an assessment 
of the most likely costs, and adjusted properly for inflation. In 
addition, steps should be taken to minimize mathematical mistakes and 
to ground the estimate in documented assumptions that can be verified 
by supporting data and a historical record of actual cost and schedule 
experiences on comparable programs. 

To DON's credit, the cost estimates were developed based on NMCI 
historical cost data, were adjusted properly for inflation, contained 
few mathematical mistakes, and were largely grounded in documented 
assumptions. However, the supporting data for key assumptions were not 
verified. For example, all estimates assumed that transition activity 
costs would amount to about 18 percent of the estimated cost of NGEN 
in its first year of operation, and alternative 3's estimate assumed 
that total cost would be reduced by 10 percent due to increased 
competition from its multicontract approach. However, the supporting 
data used by Deloitte Consulting for these assumptions were not 
provided to DON or the independent government review team for 
verification because the data were proprietary to the contractor. 
Further, NMCI historical data were only available at an aggregate 
level, so the team had to rely on subject-matter experts and other 
sources to estimate costs at a finer level of detail. 

* The AOA cost estimates were not credible. To be credible, the cost 
estimates should discuss any limitations in the analysis due to 
uncertainty or biases surrounding the data and assumptions. Major 
assumptions should be varied and other outcomes computed to determine 
how sensitive the estimate is to changes in the assumptions. Risk and 
uncertainty inherent in the estimate should be assessed and disclosed. 
Further, the estimate should be properly verified by, for example, 
comparing the results with an independent cost assessment. 

While the AOA identified limitations in the cost analysis, such as the 
use of NMCI data that did not reflect prices of other service 
providers, and evaluated the impact on costs of using different 
transition timeline scenarios, it did not include a sensitivity 
analysis of the key cost driver (i.e., the number of personnel needed 
to manage NGEN), despite concerns that the Navy's estimates of these 
numbers were not stabilized at the time of the AOA. In addition, while 
each cost estimate included a cost risk analysis based on the quality 
of data used, there were discrepancies in how the analysis was 
conducted and reported. For example, the cost for local area network 
facilities was estimated based on the contractor's experience, which 
was considered by the cost team to be a less credible data source, but 
it was scored higher on their risk scale, indicating that the data 
source was more credible. Also, schedule and performance risks were 
not quantified and reflected in the estimates, which is significant 
because a qualitative assessment of schedule and performance risks 
among alternatives revealed increased risk in implementing a segmented 
approach. If such risks had been quantified and reflected in the 
estimates, the results would have shown higher costs associated with 
alternatives 3 and 3V. Nevertheless, the AOA concluded that there was 
no significant cost difference among the alternatives. 

In addition, the cost estimates were not validated by the independent 
team responsible for reviewing the cost analysis. Specifically, 
independent review team officials told us that they participated in a 
line-by-line review of the cost model where they raised comments and 
questions to the cost team. However, about 69 percent of the team's 
comments were not fully addressed and included notable concerns, such 
as the questionable use of certain industry-based assumptions that may 
not be comparable to a program as large as NGEN or to the government 
environment. Independent review team officials attributed the comments 
not being closed to the fact that the team did not have authority over 
the cost model to ensure that its comments were addressed. Further, 
these officials told us that they were not asked to review the final 
version of the cost model, which was the version that first introduced 
alternative 3V, and their review of the final version of the AOA 
report occurred after the DON CIO had submitted it to OSD CAPE for 
final approval. 

According to officials responsible for developing the AOA, the 
weaknesses in the AOA cost estimates largely exist because there was 
not enough time to conduct a more thorough analysis. Specifically, 
they told us that the AOA schedule was constrained because the program 
wanted to get requests for proposals for NGEN contracts out by a 
predetermined date. This position was also supported by various 
management meeting minutes and other artifacts that we reviewed. 
However, DOD and DON officials disagreed with this position and told 
us that the time allotted to conduct the AOA did not negatively impact 
its quality or scope. A time-constrained approach is not consistent 
with DOD guidance, which states that the scope of an alternatives 
analysis should be proportionate to the amount of resources affected 
by the decision, with more significant programs receiving more 
analytical attention. The combination of the AOA weaknesses we 
identified, and the fact that NGEN has a preliminary program life 
cycle cost estimate of about $50 billion for increment 1 and is 
intended to provide the foundation for DON's future networking 
environment, suggest the need for considerable analytical attention to 
alternative approaches. 

Without reliable cost estimates for each alternative, decision makers 
did not have a sound basis for making an informed decision on an NGEN 
solution. Most notably, since the estimates did not reflect increased 
risks associated with the segmented approach, the differences in the 
alternatives' costs were understated, and the amount of risk and costs 
accepted by proceeding with a segmented approach were not fully 
understood. 

AOA Did Not Sufficiently Assess Operational Effectiveness: 

In addition to including reliable cost estimates, an AOA should assess 
how well each alternative satisfies required capabilities or goals. 
According to DOD guidance,[Footnote 14] such an analysis should (1) 
identify the operational capabilities and goals to be achieved with 
the system solution, (2) establish quantitative or qualitative 
measures for evaluating the operational effectiveness of each 
alternative, and (3) assess the ability of each alternative to achieve 
these measures. 

While the AOA identified program capabilities and goals, it did not 
sufficiently assess the alternatives' operational effectiveness, 
making it unclear how well the alternatives would actually satisfy 
NGEN capabilities and goals. Specifically, 

* The AOA identified capabilities and goals that the system solution 
should achieve. Among other things, these included addressing NMCI 
capability limitations identified based on 8 years of operational 
experience, as well as capabilities needed to support DOD and DON 
networking strategies for DOD's Global Information Grid Network 
Operations and DON's future Naval Networking Environment. (See table 8 
for these capabilities and goals.) 

* The AOA did not establish quantitative or qualitative measures for 
assessing the alternatives' ability to achieve the identified NGEN 
capabilities and goals, as shown in table 8. For example, one of the 
capabilities was visibility into root causes for major network 
outages, which the AOA merely concluded that alternatives 2, 3V, and 3 
were equally effective in addressing, even though no quantitative or 
qualitative measures of the alternatives' respective ability to 
provide visibility into root causes were defined. Further, the AOA did 
not discuss the methodology for assessing the alternatives. Rather, it 
simply states that it was a qualitative assessment. 

While the AOA did not establish measures for assessing the 
alternatives' ability to achieve NGEN capabilities and goals, it did 
establish several quantitative measures to differentiate among the 
alternatives' respective approaches to acquiring, managing, and 
delivering NMCI capabilities. However, these measures alone do not 
provide insight into how they would influence the operational 
effectiveness of each alternative because they were not linked to NGEN 
capabilities and goals, and they did not provide sufficient insight 
for selecting a preferred alternative. For example, while the AOA 
recognized that an increase in the number of contractual relationships 
would result in more complexity and risk in implementing the 
alternative, it did not include measures for quantifying how much more 
risk is introduced as the number of contractual relationships 
increases. (See table 7 for the measures that were provided in the 
AOA.) 

Table 7: Measures Related to Overseeing, Operating, and Managing NGEN: 

Government personnel as a percentage of total NGEN personnel; 
Status quo: 28; 
Alternative 2: 39; 
Alternative 3V: 40; 
Alternative 3: 41. 

Number of seams[A] managed by the government; 
Status quo: 1; 
Alternative 2: 1; 
Alternative 3V: 36; 
Alternative 3: 36. 

Number of contractual relationships managed by the government; 
Status quo: 3; 
Alternative 2: 3; 
Alternative 3V: 10; 
Alternative 3: 15. 

Source: NGEN Analysis of Alternatives report, Center for Naval 
Analyses, April 2009. 

[A] Seams are the relationships between segments (e.g., local area 
networks, service desks, and enterprise hardware). 

[End of table] 

In addition, the AOA included a separate assessment of the likelihood 
of each alternative to successfully implement IT best practices for 
end-to-end IT service delivery (i.e., IT Service Management 
framework).[Footnote 15] To DON's credit, the approach used to measure 
the alternatives in this assessment was more structured and better 
documented. Specifically, the AOA team conducted table-top exercises 
with subject-matter experts representing each of the communities that 
will contribute to the acquisition, operation, and oversight of NGEN, 
and it worked through scenarios, such as everyday operations and 
responding to a computer network incident, to determine the extent to 
which each alternative could employ IT best practices to address a 
given scenario. The team captured comments made by participants and 
used them to infer rankings that resulted in numerical scores for each 
alternative. 

The AOA did not assess the alternatives' ability to address 
capabilities and goals using defined measures of operational 
effectiveness because, as stated previously, no measures were 
established. Instead, it compared the alternatives based on 
qualitative determinations of whether the capability or goal was 
either met or partially met. (See table 8 for the results of DON's 
assessment.) 

Table 8: DON's Qualitative Assessment of Alternatives' Ability to Meet 
NGEN Capabilities and Goals: 

Capabilities: 

NMCI capabilities and services as of September 2010; 
Status quo: [Check];
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

Address NMCI deficiencies: 

* Solve problem with out-of-scope government directed action; 
Status quo: [Empty]; 
Alternative 2: [Check-]; 
Alternative 3V: [Check-]; 
Alternative 3: [Check]. 

* Sufficient visibility/situational awareness of network operations; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Visibility into root causes; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Adequate log keeping; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Technology refresh/architecture upgrades; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Empty]; 
Alternative 3: [Check]. 

Network Operations Concept of Operations: 

* Support Network Operations Concept of Operations; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Empty]; 
Alternative 3: [Check]. 

* Proactive control/defense of network; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

Goals: 

Supports Naval Networking Environment: 

* Enterprise network interoperability; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Government operational control; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Support transformation to service-oriented architecture; 
Status quo: [Check];
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Open architecture and standards; 
Status quo: [Check];
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Empty]. 

* Implement IT services management; 
Status quo: [Empty]; 
Alternative 2: [Check-]; 
Alternative 3V: [Check-]; 
Alternative 3: [Check]. 

* Implement portfolio management process; 
Status quo: [Empty]; 
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

* Active monitor/report of service level agreements; 
Status quo: [Check];
Alternative 2: [Check];
Alternative 3V: [Check];
Alternative 3: [Check]. 

Source: NGEN Analysis of Alternatives report, Center for Naval 
Analyses, April 2009. 

Note: A checkmark indicates that the government's goal or concern 
would be addressed by the alternative; a check minus indicates that 
the government's goal or concern would be partially addressed by the 
alternative. 

[End of table] 

As with the cost estimates, officials responsible for developing the 
AOA told us that the operational effectiveness analysis was subject to 
time constraints so that requests for proposals could be issued on 
time. Although DOD and DON officials told us that the time allotted to 
conduct the AOA did not negatively impact its quality or scope, our 
review suggests otherwise. Further, the time-constrained approach is 
not consistent with DOD guidance, which states that the scope of an 
alternatives analysis should be proportionate to the resources 
affected by the decision, with more significant programs receiving 
more analytical attention. 

Without a more thorough effectiveness analysis, decision makers did 
not have a sound basis for making an informed decision on the best 
NGEN alternative to pursue. Instead, DON has selected a segmented 
approach on the basis that it would provide increased flexibility in 
meeting NGEN capabilities and goals with no additional cost, even 
though the degree of increased flexibility among the alternatives 
remains unclear. 

Current Acquisition Approach Was Not Addressed in the AOA and Is 
Riskier and Costlier Than Other Approaches in the Analysis: 

According to DOD guidance,[Footnote 16] an AOA should examine viable 
solutions with the goal of identifying the most promising option, 
thereby informing acquisition decision making. However, the segmented 
approach currently being pursued by DON was not one of the 
alternatives assessed in the AOA. Specifically, the current approach 
has more contracts, a different segmentation scheme, and a different 
transition timeline than the analyzed alternatives. Further, the 
impact of these differences in terms of how they compare to the 
original alternatives was not assessed. 

The approach that is being pursued by the program office includes a 
higher number of contracts than those analyzed in the AOA. Given that 
the AOA highlighted greater schedule and performance risks as the 
number of contracts and contractual relationships in the approach 
increase, the relative schedule and performance risks for the current 
approach are likely greater than those for alternative 3, and 
therefore are likely to result in greater costs. In support of this 
likelihood, DON's November 2009 risk-adjusted preliminary program life 
cycle cost estimate for the current approach for fiscal year 2011 
through fiscal year 2015 shows that the current approach will cost at 
least an estimated $4.7 billion more than any of the alternatives in 
the AOA. (See table 9 for a comparison of the current approach to the 
approaches assessed in the AOA and figure 3 for an illustration of the 
contractual relationships associated with DON's current approach.) 

Table 9: Comparison of NGEN AOA Alternatives and Current Approach: 

Contracts[A]; 
Status quo: 2; 
Alternative 2: 2; 
Alternative 3V: 4; 
Alternative 3: 5; 
Current approach: 6. 

Contractual relationships; 
Status quo: 3; 
Alternative 2: 3; 
Alternative 3V: 10; 
Alternative 3: 15; 
Current approach: 21. 

Fiscal year 2015 operating cost (billions); 
Status quo: $2.25; 
Alternative 2: $2.62; 
Alternative 3V: $2.64; 
Alternative 3: $2.65; 
Current approach: $2.94. 

Fiscal years 2011 to 2015 total cost (billions); 
Status quo: $10.3; 
Alternative 2: $10.8; 
Alternative 3V: $10.8; 
Alternative 3: $10.7; 
Current approach: $15.6. 

Relative schedule risk; 
Status quo: Least; 
Alternative 2: More; 
Alternative 3V: Greater; 
Alternative 3: Greatest; 
Current approach: (not determined). 

Relative risk in delivering planned performance; 
Status quo: Least; 
Alternative 2: More; 
Alternative 3V: Greater; 
Alternative 3: Greatest; 
Current approach: (not determined). 

Transition timeline; 
Status quo: 24 months of continued NMCI services before 24 months of 
transitioning to new provider, and then 12 months of NGEN steady-state; 
Alternative 2: Same timeline as alternative 1; 
Alternative 3V: Same timeline as alternative 1; 
Alternative 3: Same timeline as alternative 1; 
Current approach: 14 months of continued NMCI services before 29 
months of transitioning to new providers in staggered phases, and then 
17 months of NGEN steady-state. 

Sources: NGEN Analysis of Alternatives report, Center for Naval 
Analyses, April 2009 (status quo and alternatives 2, 3V, and 3); GAO 
analysis of DON data (current approach). 

Note: The difference between the total fiscal year 2011 to fiscal year 
2015 cost for the current approach and alternative 3V is about $4.7 
billion due to rounding. 

[A] In all alternatives, wide area network services will be provided 
by the Defense Information Systems Agency. DON's relationship with the 
Defense Information Systems Agency will be contract-like in that it is 
an interagency agreement, in which DON must pay for services and will 
have to manage its relationship with the service providers. 

[End of table] 

Figure 3: Contractual Relationships in Current NGEN Approach: 

[Refer to PDF for image: illustration] 

The figure depicts relationships among and between the following: 

Government; 
Enterprise Services/Service Coordination; 
End-Users Hardware; 
Enterprise Software; 
Security; 
Transport; 
Wide Area Network. 

Source: GAO analysis of DON data. 

[End of figure] 

OSD CAPE officials told us that they believe the differences between 
the current approach and alternatives assessed in the AOA are not 
significant because DON is still pursing a segmented approach and that 
the differences were the result of "an appropriate evolution of the 
segmented approach." They further said that the increased risks in the 
current approach are offset by mitigating factors, such as the use of 
staggered phases to implement NGEN and the use of more efficient 
segmentation schemes. However, we have yet to receive any analysis to 
support their positions, and the current approach is estimated to cost 
about $4.7 billion more. As a result, DON cannot demonstrate that it 
is pursuing the most cost-effective approach for acquiring NGEN 
capabilities and meeting NGEN goals. 

NGEN's Schedule Does Not Provide a Reliable Basis for Program 
Execution, and Program Delays Are Occurring: 

The success of a large-scale acquisition program depends in part on 
having a reliable schedule that defines, among other things, when work 
activities and milestone events will occur, how long they will take, 
and how they are related to one another. As such, the schedule not 
only provides a road map for systematic program execution but also 
provides the means by which to gauge progress, identify and address 
potential problems, and promote accountability. Without a reliable 
schedule, it is likely that established program milestones will slip. 
In the case of NGEN, such delays are already being experienced. 

Our work has identified nine best practices associated with developing 
and maintaining a reliable schedule. These are (1) capturing all 
activities, (2) sequencing all activities, (3) assigning resources to 
all activities, (4) establishing the duration of all activities, (5) 
integrating schedule activities horizontally and vertically, (6) 
establishing the critical path[Footnote 17] for all activities, (7) 
identifying reasonable "float"[Footnote 18] between activities, (8) 
conducting a schedule risk analysis, and (9) updating the schedule 
using logic and durations. See table 10 for a description of each of 
these best practices. 

Table 10: Description of Scheduling Best Practices: 

Practice: Capturing all activities; 
Description: The schedule should reflect all activities (steps, 
events, outcomes, etc.) as defined in the program's work breakdown 
structure to include activities to be performed by both the government 
and its contractors. 

Practice: Sequencing all activities; 
Description: The schedule should sequence activities in the order that 
they are to be implemented. In particular, activities that must finish 
prior to the start of other activities (i.e., predecessor activities), 
as well as activities that cannot begin until other activities are 
completed (i.e., successor activities) should be identified. 

Practice: Assigning resources to all activities; 
Description: The schedule should reflect who will do the work 
activities, whether all required resources will be available when they 
are needed, and whether any funding or time constraints exist. 

Practice: Establishing the duration of all activities; 
Description: The schedule should reflect the duration of each 
activity. These durations should be as short as possible and have 
specific start and end dates. 

Practice: Integrating schedule activities horizontally and vertically; 
Description: The schedule should be horizontally integrated, meaning 
that it should link the products and outcomes associated with 
sequenced activities. The schedule should also be vertically 
integrated, meaning that traceability exists among varying levels of 
activities and supporting tasks and subtasks. 

Practice: Establishing the critical path for all activities; 
Description: The critical path represents the chain of dependent 
activities with the longest total duration in the schedule. 

Practice: Identifying reasonable float between activities; 
Description: The schedule should identify a reasonable amount of 
float--the time that a predecessor activity can slip before the delay 
affects successor activities--so that schedule flexibility can be 
determined. As a general rule, activities along the critical path 
typically have the least amount of float. 

Practice: Conducting a schedule risk analysis; 
Description: A schedule risk analysis is used to predict the level of 
confidence in the schedule, determine the amount of time contingency 
needed, and identify high-priority schedule risks. 

Practice: Updating the schedule using logic and durations; 
Description: The schedule should use logic and durations in order to 
reflect realistic start and completion dates, be continually monitored 
to determine differences between forecasted completion dates and 
planned dates, and avoid logic overrides and artificial constraint 
dates. 

Source: GAO. 

[End of table] 

In December 2009, NGEN established a baseline integrated master 
schedule[Footnote 19] composed of over 25 separate underlying 
schedules (or subschedules) to capture program milestones and the 
expected completion dates for activities leading up to them. However, 
the most current version of this schedule (May 2010) that was 
available at the time we began our review was not reliable because 
only two of the four subschedules[Footnote 20] that we analyzed 
substantially met any of the nine practices. The results of our 
analysis of the four subschedules are summarized in table 11. 

Table 11: Summary Results of Selected NGEN Schedules' Satisfaction of 
Schedule Estimating Practices: 

Schedule practice: Capturing all activities; 
Transition Integrated Product Team: Partially Met; 
IT Service Management Process Development: Partially Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Partially Met; 
Contract Technical Representative Workforce Reconstitution: Partially 
Met. 

Schedule practice: Sequencing all activities; 
Transition Integrated Product Team: Minimally Met; 
IT Service Management Process Development: Substantially Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Minimally Met; 
Contract Technical Representative Workforce Reconstitution: Minimally 
Met. 

Schedule practice: Assigning resources to all activities; 
Transition Integrated Product Team: Not Met; 
IT Service Management Process Development: Not Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Not Met; 
Contract Technical Representative Workforce Reconstitution: Not Met. 

Schedule practice: Establishing the duration of all activities; 
Transition Integrated Product Team: Minimally Met; 
IT Service Management Process Development: Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Minimally Met; 
Contract Technical Representative Workforce Reconstitution: Met. 

Schedule practice: Integrating schedule activities horizontally and 
vertically; 
Transition Integrated Product Team: Partially Met; 
IT Service Management Process Development: Substantially Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Partially Met; 
Contract Technical Representative Workforce Reconstitution: Partially 
Met. 

Schedule practice: Establishing the critical path for all activities; 
Transition Integrated Product Team: Minimally Met; 
IT Service Management Process Development: Minimally Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Minimally Met; 
Contract Technical Representative Workforce Reconstitution: Minimally 
Met. 

Schedule practice: Identifying reasonable float between activities; 
Transition Integrated Product Team: Minimally Met; 
IT Service Management Process Development: Partially Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Minimally Met; 
Contract Technical Representative Workforce Reconstitution: Partially 
Met. 

Schedule practice: Conducting a schedule risk analysis; 
Transition Integrated Product Team: Not Met; 
IT Service Management Process Development: Not Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Not Met; 
Contract Technical Representative Workforce Reconstitution: Not Met. 

Schedule practice: Updating the schedule using logic and durations; 
Transition Integrated Product Team: Partially Met; 
IT Service Management Process Development: Partially Met; 
Comprehensive Facilities Inventory Phase 2 Plan of Action and 
Milestones: Partially Met; 
Contract Technical Representative Workforce Reconstitution: Partially 
Met. 

Source: GAO analysis of DON data. 

"Met" = DON provided complete evidence that satisfied the criterion. 

"Substantially Met" = DON provided evidence that satisfied more than 
one-half of the criterion. 

"Partially Met" = DON provided evidence that satisfied about one-half 
of the criterion. 

"Minimally Met" = DON provided evidence that satisfied less than one- 
half of the criterion. 

"Not Met" = DON provided no evidence that satisfied any portion of the 
criterion. 

Note: The Transition Integrated Product Team subschedule contains work 
associated with transition planning, such as development of the 
Transition Management Strategy and Transition Concept of Operations. 
The IT Service Management Process Development subschedule contains 
early transition activity work focused on developing service 
management processes and procedures. The Comprehensive Facilities 
Inventory Phase 2 Plan of Action and Milestones subschedule contains 
early transition activity work associated the evaluation, analysis, 
and validation of the current NMCI infrastructure inventory. The 
Contract Technical Representative Workforce Reconstitution subschedule 
contains early transition activity work associated with job task 
analysis and the assessment of learning tools for contract technical 
representatives. 

[End of table] 

* Capturing all activities. All four subschedules partially met this 
practice. Specifically, the majority of the activities contained in 
these subschedules could be mapped back to the program's NGEN work 
breakdown structure. However, this structure is defined at a high 
level and is not expected to be further decomposed into planned work 
products and deliverables until the program enters the deployment 
phase when NGEN contracts are awarded. Until this structure is 
sufficiently defined, it cannot be determined whether the program 
schedules capture all work needed to accomplish program objectives. 
For example, we identified risk mitigation activities for 10 active 
risks that should have been, but were not, captured as scheduled work. 

During our review, program officials told us that they had since taken 
steps to ensure that all risk mitigation activities are added to the 
schedule. However, until NGEN work is sufficiently defined, the 
program does not have complete assurance that the activities currently 
captured in the various schedules support NGEN increment 1. 

* Sequencing all activities. One subschedule substantially met this 
practice while the other three minimally met it. The subschedule that 
substantially met this practice had less than 1 percent of activities 
missing a predecessor or successor dependency. Of the remaining three 
subschedules, two did not identify predecessor or successor activities 
for over half of the activities in their schedules. This is of concern 
because if an activity that has no logical successor slips, the 
schedule will not reflect the effect of these slips on the critical 
path, float, or scheduled start dates of "downstream" (i.e., later) 
activities. Additionally, one subschedule had "constraints" placed on 
about 73 percent of its activities, meaning that these activities 
cannot begin earlier even if upstream work is completed ahead of 
schedule. According to program officials, they are working to reduce 
the number of constraints in the schedule. However, until activities 
are properly sequenced, these issues reduce the credibility of the 
dates calculated by the scheduling tool. 

* Assigning resources to all activities. Program officials told us 
that they do not assign resources to any of the program schedules. 
They stated that the effort necessary to assign resources within the 
schedules would be significant and that they did not have the staff 
available to do this. However, without proper allocation of resources 
in the schedule, the program office cannot accurately forecast the 
likelihood that activities will be completed by their projected end 
dates, and the risk that key milestones will slip increases. 

* Establishing the duration of all activities. Two subschedules met 
this practice while two only minimally met it. The two subschedules 
that met this practice had established activities with reasonable 
durations--the majority of which were under 30 days. The remaining two 
did not establish reasonable durations for their activities. For 
example, the majority of the activities that were in progress for the 
Transition Integrated Product Team subschedule had durations ranging 
from 50 days to 1000 days. When such long durations are assigned to 
work activities, it is likely that the activity is not defined to the 
necessary level to identify all the work that must be performed. 

* Integrating schedule activities vertically and horizontally. One of 
the subschedules substantially met and the other three partially met 
this practice. The subschedule that substantially met the practice is 
horizontally aligned, meaning activities are logically sequenced, and 
vertically aligned, meaning that detailed activities roll up into 
larger summary tasks. The other three subschedules are also vertically 
aligned; however, they are unable to demonstrate horizontal 
integration because, as previously discussed, activities were not all 
logically sequenced. 

The integration issues identified on these subschedules also impact 
the NGEN master schedule. Because of the high number of missing 
dependencies, the number of in-progress activities with durations 
exceeding 30 days, and the high number of constraints, the master 
schedule is likely not fully horizontally integrated. 

Further, one of the subschedules is not vertically aligned with the 
master schedule because none of the key work activities in the 
subschedule were included in the master schedule. In addition, the 
master schedule was not integrated with the approved NGEN acquisition 
strategy. Program officials told us they did not revise the dates in 
the master schedule until after the continuity of services contract 
was awarded (July 2010), and that the dates in the acquisition 
strategy reflected the current information. By using a source other 
than the program office's working schedule, oversight officials' 
expectations about when milestones will be met may not be realistic. 

* Establishing the critical path for all activities. None of the four 
subschedules fully met this practice. Specifically, the scheduling 
tool was unable to generate a valid critical path for the subschedules 
due to the extent of issues associated with the sequencing of 
activities, integration of activities, and identification of 
reasonable float (discussed below). Program officials stated that they 
do not manage a critical path generated by the scheduling tool. 
Instead, these officials stated that they track activities associated 
with the deployment phase decision (Milestone C), which they have 
designated as being critically important to them. However, such 
practice does not allow the program to have immediate insight into the 
full sequence of activities (both critical and noncritical) that, if 
delayed, would impact the planned completion date of Milestone C, as 
well as a projected completion date should one of these activities be 
delayed. 

* Identifying reasonable float between activities. Two subschedules 
partially met this practice, while the remaining two minimally met it. 
Each of these subschedules identified float; however, the amount of 
excessive float varied. Both the Contract Technical Representative 
Workforce Reconstitution and IT Service Management Process Development 
subschedules partially met this practice because only 25 percent and 
41 percent of their work activities had float of 100 days or greater, 
respectively. The two remaining subschedules minimally met this 
practice because over 60 percent of their activities contained float 
of 100 days or greater. 

Excessive float values are indicative of schedule logic that is 
flawed, broken, or absent. As such, these float values are of limited 
value to mitigate risk by reallocating resources from tasks that can 
safely slip to tasks that must be completed on time. 

* Conducting a schedule risk analysis. The program has not performed a 
schedule risk analysis. Instead, according to program officials, 
schedule risks are considered during risk management board meetings 
and program health assessments. However, without this analysis, it is 
not possible to determine a level of confidence in meeting program 
milestones. A schedule risk analysis will calculate schedule reserve, 
which can be set aside for those activities identified as high-risk. 
Without this reserve, the program faces the risk of delays if they 
were to occur on critical path activities. 

* Updating the schedule using logic and durations. All four 
subschedules partially met this practice. According to program 
officials, status updates are performed on the subschedules once a 
week. However, despite status updates, date anomalies exist. For 
example, the Contract Technical Representative Workforce 
Reconstitution subschedule included five activities with an actual 
start date in the future. Furthermore, the subschedules' inability to 
produce a valid critical path indicates that the sequencing of 
activities is not appropriate, thus impairing the scheduling tool's 
ability to generate realistic start and end dates. 

According to program officials, they were aware of some of these 
schedule weaknesses based on a May 2010 assessment of the schedule 
performed by a support contractor. Among other things, the 
contractor's assessment found that the schedule did not provide for 
stakeholder review of most of the major acquisition documents or steps 
to mitigate known risks, and that it lacked a valid critical path due 
to network logic issues and activity constraints. Officials told us 
that they plan to address these issues. 

In addition, program officials stated that they hold monthly program 
management reviews to discuss schedule quality issues, as well as 
risks or issues that might affect the schedule. However, these reviews 
are not addressing key schedule issues. Specifically, the NGEN 
schedule management plan calls for the schedule to be resource-loaded 
from a centralized resource pool approved by the program manager, 
activities beginning within 90 days to have durations of no more than 
20 days, and activities for mitigating approved program risks to be 
added to the schedule. However, our analysis of the schedule showed 
that: 

* resources are not assigned within the schedule, 

* activities that are to begin within 90 days have durations that 
exceed 20 days, and: 

* activities for mitigating 10 approved program risks were not 
included. 

Collectively, the weaknesses in implementing the nine key practices 
for the program's integrated master schedule increase the risk of 
schedule slippages and related cost overruns and make meaningful 
measurement and oversight of program status and progress, as well as 
accountability for results, difficult to achieve. Moreover, they 
undermine the schedule's ability to produce credible dates for planned 
NGEN milestones and events. In the case of increment 1, this risk has 
already been realized. Specifically, the NGEN master schedule was 
rebaselined in August 2010, resulting in delays in a number of key 
dates, including a 5-month delay of the Milestone C decision. See 
table 12 for a summary of key event and milestone delays. 

Table 12: Delays in Key NGEN Program Events and Milestones, as of 
August 2010: 

Milestone decisions: 

Event/milestone: Materiel Development Decision; 
Status of event/milestone: Completed; 
Delay in months[A]: 3. 

Event/milestone: DON Gate 5 Review; 
Status of event/milestone: In progress; 
Delay in months[A]: 4. 

Event/milestone: DON Gate 6 Review; 
Status of event/milestone: Not yet occurred; 
Delay in months[A]: 7. 

Event/milestone: Milestone C; 
Status of event/milestone: Not yet occurred; 
Delay in months[A]: 5. 

Documents complete: 

Event/milestone: Acquisition Strategy; 
Status of event/milestone: Completed; 
Delay in months[A]: 4. 

Event/milestone: Economic Analysis; 
Status of event/milestone: In progress; 
Delay in months[A]: 0. 

Event/milestone: Cost Analysis Requirements Description; 
Status of event/milestone: In progress; 
Delay in months[A]: 4. 

Event/milestone: Preliminary Program Life Cycle Cost Estimate; 
Status of event/milestone: In progress; 
Delay in months[A]: 4. 

Event/milestone: Capability Production Document; 
Status of event/milestone: In progress; 
Delay in months[A]: 9. 

Event/milestone: Systems Engineering Plan; 
Status of event/milestone: In progress; 
Delay in months[A]: 10. 

MIlestone decisions: 

Event/milestone: Test and Evaluation Master Plan; 
Status of event/milestone: In progress; 
Delay in months[A]: 14. 

Event/milestone: Acquisition Program Baseline; 
Status of event/milestone: In progress; 
Delay in months[A]: 3. 

Contracts: 

Event/milestone: Independent Security Operations, Oversight and 
Assessment Support Contract Award; 
Status of event/milestone: Not yet awarded; 
Delay in months[A]: 4. 

Event/milestone: Continuity of Services Contract Award; 
Status of event/milestone: Completed; 
Delay in months[A]: 5. 

Event/milestone: Transport Request for Proposal Release; 
Status of event/milestone: Not yet released; 
Delay in months[A]: 4. 

Event/milestone: Transport Contract Award; 
Status of event/milestone: Not yet awarded; 
Delay in months[A]: 5. 

Event/milestone: Hardware Contract Award; 
Status of event/milestone: Not yet awarded; 
Delay in months[A]: 0. 

Event/milestone: Software Contract Award; 
Status of event/milestone: Not yet awarded; 
Delay in months[A]: 0. 

Event/milestone: Enterprise Services Request for Proposal Release; 
Status of event/milestone: Not yet released; 
Delay in months[A]: 0. 

Event/milestone: Enterprise Services Contract Award; 
Status of event/milestone: Not yet awarded; 
Delay in months[A]: 2. 

Source: GAO analysis of DON data. 

[A] Delays were calculated by comparing data in the December 2009 
schedule with the data in the August 2010 schedule. Delays were 
rounded to the nearest month. 

[End of table] 

While officials stated that they have addressed some of the weaknesses 
identified above in the August 2010 rebaselined integrated master 
schedule, they conceded that this schedule does not assign resources 
to work activities, and the scheduling tool is unable to generate a 
valid critical path. Because these key scheduling practices are not 
being performed, the schedule is still not reliable. 

Without a fully integrated and reliably derived schedule for the 
entire NGEN program, the program office cannot identify when and how 
it will proceed through Milestone C and ultimately transition from 
NMCI to NGEN, and it cannot adequately manage and measure its progress 
in executing the work needed to do so. 

NGEN Acquisition Decisions Were Not Always Performance-and Risk-Based: 

Successful execution of system acquisition programs depends in part on 
effective executive-level governance, to include having organizational 
executives review these programs at key milestones in their life 
cycles and make informed performance-and risk-based decisions as to 
how they should proceed.[Footnote 21] DON policy recognizes the 
importance of such milestone reviews. According to this policy, 
acquisition programs must proceed through a series of gate reviews (as 
discussed above), during which program performance is assessed and 
satisfactory program health must be demonstrated prior to moving 
forward. 

Currently, program performance and health at each gate are assessed 
using the Naval Probability of Program Success assessment methodology, 
which was established in September 2008. This assessment addresses 
four aspects of a program: (1) requirements, (2) resources, (3) 
planning/execution, and (4) external influencers. Associated with each 
aspect are two or more metrics, each of which is scored based on 
underlying criteria that are unique to each gate. (See table 13 for a 
description of each metric.) At a given gate review, the criteria are 
rated in terms of green, yellow, or red.[Footnote 22] Further, the 
metrics can be designated as critical, meaning that any significant 
issues that are associated with these metrics must be resolved before 
the gate can be exited. 

Table 13: Naval Probability of Program Success Metric Definitions: 

Program aspect: Requirements; 
Metric: Parameter status; 
Definition: Progress toward defining capability requirements and 
meeting those requirements; requirements traceability and validity of 
the threat assessment. 

Program aspect: Requirements; 
Metric: Scope evolution; 
Definition: Stability of performance parameters from the established 
baseline and the impact of requirements changes on total program cost 
and schedule. 

Program aspect: Requirements; 
Metric: Concept of operations; 
Definition: Progress toward developing and scoping the concept of 
operations, using it to inform program requirements, acquisition 
approaches, and strategies, and the validity of the concept of 
operations over time. 

Program aspect: Resources; 
Metric: Budget and planning; 
Definition: Sufficiency of funding based on last approved budget 
controls and degree of deviation from the current cost estimate. Also 
assesses how well acquisition, systems development, and sustainment 
strategies are evolving in ways intended to mitigate Total Ownership 
Cost growth. 

Program aspect: Resources; 
Metric: Manning; 
Definition: Stability and adequacy of Resource Sponsor and Program 
Office staffing (availability, skills, experience, certification, and 
training). 

Program aspect: Planning/execution; 
Metric: Acquisition management; 
Definition: Completeness of the program master schedule, status of 
milestone documentation development, and progress toward defining 
derived requirements in the System Design Specification. 

Program aspect: Planning/execution; 
Metric: Industry/company assessment; 
Definition: Assesses market research activities, industrial base 
health, and an understanding of industrial implications for cost, 
schedule, and technical risks. For major contracts, assesses each 
company's financial health, financial systems, and manufacturing/ 
production capabilities. 

Program aspect: Planning/execution; 
Metric: Total ownership cost estimating; 
Definition: Adequacy of the elements required to produce sound cost 
estimates: program description information, cost data, cost estimating 
process, cost estimate stability and comparisons, and cost estimate 
measures. 

Program aspect: Planning/execution; 
Metric: Test and evaluation; 
Definition: Progress toward defining and executing the Test and 
Evaluation Strategy and Test and Evaluation Master Plan. 

Program aspect: Planning/execution; 
Metric: Technical maturity; 
Definition: Maturity of the system and subsystems design, as well as 
the technical maturity of critical technology elements in accordance 
with an approved Technology Development Strategy. 

Program aspect: Planning/execution; 
Metric: Sustainment; 
Definition: Progress toward defining and executing the sustainment 
strategy, and the resource adequacy applied toward the life cycle 
sustainment activities. 

Program aspect: Planning/execution; 
Metric: Software; 
Definition: Status of software management and engineering activities 
by government agencies and/or contractors that are integral to program 
deliverables. 

Program aspect: Planning/execution; 
Metric: Contract planning and execution; 
Definition: Status of procurement activities and achievement of 
contracting milestones against the planned schedule; 
performance of major contractors and/or government performers. 

Program aspect: Planning/execution; 
Metric: Government program office performance; 
Definition: Progress toward defining and executing intragovernment 
requirements; 
responsiveness to deliverable submissions; 
delivery of facilities, funding, and government furnished 
equipment/information in accordance with scheduled requirements; 
and effectiveness of configuration management and risk management 
boards. 

Program aspect: Planning/execution; 
Metric: Technology protection[A]; 
Definition: Progress of safeguarding the research, technology 
information, and applied knowledge associated with the program. 

Program aspect: External influencers; 
Metric: Fit in vision; 
Definition: Alignment with current documented OSD guidance and Navy/ 
Marine Corps strategies. 

Program aspect: External influencers; 
Metric: Program advocacy; 
Definition: Support from key stakeholders, such as USD (AT&L) (or 
equivalent), CAPE, international partners, and other military services. 

Program aspect: External influencers; 
Metric: Interdependencies; 
Definition: Status of dependent programs' delivery of the requisite 
capability or quantity on schedule. 

Source: DON Naval Probability of Program Success Guidebook version 2.2. 

[A] This was a new metric introduced in the Naval Probability of 
Program Success version 2 framework. 

[End of table] 

As noted earlier, a Gate 1 review was not held because the gate-review 
process was not established when the program began. In lieu of a Gate 
1 review, according to the NGEN Acquisition Strategy, the Chief of 
Naval Operations Executive Board met to confirm NGEN requirements 
during the winter of 2007/2008 and these meetings were "nominally" a 
Gate 1 review. Subsequent to the establishment of the DON gate 
process, an NGEN Gate 2 review--intended to focus on an analysis of 
alternatives--was waived in early 2008 because the department planned 
to continue the use of existing NMCI technology, and NGEN entered the 
DON review process at Gate 3 in April 2008. OSD later identified the 
program as a pre-MAIS acquisition, resulting in the direction to 
conduct an analysis of acquisition alternative approaches. As such, 
DON held a Gate 2 review in April 2009, one year after the Gate 3 
review. Since then, DON held a Gate 4 review in November 2009, as well 
as a Gate 5 review in October 2010. As discussed below, the extent to 
which each of the gate reviews was performance-and risk-based varied. 

* Gate 3 review. At the time of this review, which was in April 2008, 
the Probability of Program Success assessment methodology was not yet 
in place. Therefore, program review documentation focused on, for 
example, program activities that had been completed, were under way, 
and were planned. However, these activities were not presented 
relative to any benchmark or goal, and thus program performance was 
not apparent in the documentation. Further, while program 
documentation shows that risks were disclosed, such as funding 
shortfalls for fiscal years 2008 and 2009, as well as workforce and 
training challenges, the scope and nature of the information presented 
did not extend to the level that the assessment methodology provides. 
For example, the information presented did not address the realism and 
achievability of the program master schedule and the confidence level 
associated with the current cost estimate, including the difference 
between the program office and independent cost estimates, which are 
both relevant criteria under the assessment methodology for the gate. 
Notwithstanding these gaps in information that would have limited 
informed program decision making, the program was approved to proceed. 

* Gate 2 review. At the time of this review, which was in April 2009, 
the Probability of Program Success assessment methodology was in 
place. However, it was not used to inform program decision making. 
Instead, the review focused on the AOA, next steps, and the overall 
program timeline. While briefing documentation shows that cost 
estimates for the alternatives exceeded planned funding, the 
documentation did not disclose the range of AOA and integrated master 
schedule weaknesses discussed earlier in this report, and the risks 
associated with these limitations. This is significant because the 
Gate 2 assessment criteria focus on, among other things, whether the 
AOA cost estimates and master program schedule are reliable and 
whether program execution is on schedule. Notwithstanding these 
weaknesses, the program was approved to proceed. 

* Gate 4 review. For this review, DON used its Probability of Program 
Success methodology and assessed the health of the program against 
each of the 17 metrics,[Footnote 23] including 3 that DON designated 
as potentially critical--parameter status, budget and planning, and 
acquisition management. According to the program health assessment 
used at this gate, 8 of the 17 metrics were rated as red, meaning that 
the program had identified significant issues that would inhibit 
delivery of capability within approved cost and schedule constraints 
and that mitigation strategies had not been identified. Moreover, the 
8 metrics rated as red included 3 that were designated as critical, 
meaning that these issues needed to be resolved before exiting the 
gate. Specifically, the parameter status metric was rated as red 
because NGEN requirements that increment 1 is to meet had not yet been 
defined; the budget and planning metric was rated as red because the 
program was not fully funded; and the acquisition management metric 
was rated as red because the USD (AT&L) had yet to authorize the 
milestone at which the program would enter the Defense Acquisition 
System. (See figure 4 for the assessment results for all 17 metrics.) 
Moreover, the gate briefing document highlighted a number of risks 
facing the program. For example, it faced the risk that key program 
documentation, such as the System Engineering Plan and the Test and 
Evaluation Master Plan, would not be completed until NGEN requirements 
were defined. Further, it faced the risk that insufficient funding 
would impact the program office's ability to acquire NMCI assets. 
Nevertheless, the program was approved to proceed. 

Figure 4: NGEN Gate 4 Probability of Program Success Assessment 
(November 2009): 

[Refer to PDF for image] 

[Refer to PDF for image: illustration] 

NGEN Program Health: Red; 
Program requirements: Red; 
Parameter Status[A]: Red, critical; 
Scope Evolution: Green; 
Concept of Operations: Yellow; 
Program resources: Red; 
Budget and Planning[A]: Red, critical; 
Manning: Red; 
Program planning/execution: Red; 
Acquisition Management[A]: Red, critical; 
Industry/Company Assessment: Red; 
Total Ownership Cost Estimating: Red; 
Test and Evaluation: Yellow; 
Technical Maturity: Yellow; 
Sustainment: Red; 
Software: Yellow; 
Contract Planning and Execution: Red; 
Government Program Office Performance: Green; 
External influencers: Green; 
Fit in Vision: Green; 
Program Advocacy: Green; 
Interdependencies: Green. 

Source: GAO analysis of DON data. 

Note: 

Green means that the program is on track to provide capability within 
approved cost and schedule constraints. 

Yellow means that the program has identified some significant issues 
with providing capability within approved cost and schedule 
constraints, but mitigation strategies are being executed. 

Red means that the program has identified significant issues that will 
inhibit delivery of capability within approved cost and schedule 
constraints and mitigation strategies have not been identified; and 
"critical" means that a potential nonexecutable situation exists for 
the program that must be remedied. 

[A] These metrics are considered critical when they are scored red. 

[End of figure] 

* Gate 5 review. For this review, which was conducted in October 2010, 
DON again used its Probability of Program Success methodology and 
assessed program performance and risk against all 18 metrics, 
including 9 that DON designated as potentially critical. Three metrics 
were rated as red; 1, test and evaluation, was deemed critical. 
According to the assessment, the test and evaluation metric was rated 
as red because the Test and Evaluation Master Plan was not complete; 
the budget and planning metric was rated as red because of significant 
NGEN funding reductions; and the manning metric was rated as red 
because of inadequate program office contracting, engineering and 
logistics personnel. Further, according to the assessment, the Test 
and Evaluation Master Plan was not complete because the requirements 
were not defined. As discussed above, the program recognized, at Gate 
4, the risk that a delay in defining NGEN requirements would impact 
the completion of this plan. (See figure 5 for the assessment results 
for all 18 metrics.) According to the gate briefing document, these 
red ratings introduced a number of risks, such as the risk that the 
program would not be able to execute its current acquisition approach 
and meet program milestones. 

In addition, even though the assessment rated the acquisition 
management metric as green, this rating is not consistent with our 
findings in this report about the NGEN integrated master schedule. 
Specifically, the rationale for the green rating was that the August 
2010 rebaselined schedule was viewed as realistic and achievable by 
key stakeholders. However, as stated earlier, program officials 
conceded that the schedule does not assign resources, and the 
scheduling tool is unable to generate a valid critical path, which are 
key scheduling practices; thus the August 2010 schedule was not 
reliable. The approval of the Assistant Secretary of the Navy 
(Research, Development and Acquisition) for NGEN to proceed beyond 
Gate 5 was made conditional on the program satisfactorily completing 
action items focused on releasing the request for proposals for the 
Transport Services contract (scheduled for January 2011) and resolving 
its funding shortfall. 

Figure 5: NGEN Gate 5 Probability of Program Success Assessment 
(October 2010): 

[Refer to PDF for image: illustration] 

NGEN Program Health: Yellow; 
Program requirements:Yellow; 
Parameter Status[A]: Yellow; 
Scope Evolution: Green; 
Concept of Operations: Yellow; 
Program resources: Red; 
Budget and Planning[A]: Red; 
Manning: Red; 
Program planning/execution: Yellow; 
Acquisition Management: Green; 
Industry/Company Assessment[A]: Green; 
Total Ownership Cost Estimating[A]: Yellow; 
Test and Evaluation[A]: Red, critical; 
Technical Maturity[A]: Yellow; 
Sustainment: Green; 
Software: Green; 
Contract Planning and Execution[A]: Yellow; 
Government Program Office Performance[A]: Yellow; 
Technology protection: Yellow; 
External influencers: Green; 
Fit in Vision: Green; 
Program Advocacy[A]: Yellow; 
Interdependencies: Green. 

Source: GAO analysis of DON data. 

[A] These metrics are considered critical when certain underlying 
criteria are rated red. This approach is different from the Gate 4 
assessment because DON used a revised version of the Probability of 
Program Success framework when it conducted the Gate 5 assessment 

[End of figure] 

As shown above, DON has demonstrated a pattern of approving NGEN at 
key acquisition review gates in the face of both limited disclosure of 
the program's health and risks and known program risks and shortfalls 
in performance. According to DON officials, the decisions to pass the 
gates and proceed were based on their view that they had sufficiently 
mitigated known risks and issues. By not fully ensuring that NGEN gate 
decisions sufficiently reflected program challenges, DON has increased 
the likelihood that the NGEN acquisition alternative that it is 
pursuing is not the most cost-effective course of action, and that the 
program will cost more and take longer to complete than planned. 

Conclusions: 

Given the enormous size, complexity, and mission importance of NGEN, 
it is vital that DON and DOD assure decision makers, including the 
congressional defense committees, that the approach to acquiring 
needed capabilities is the most cost-effective and that its execution 
is guided by a well-defined schedule and informed milestone decision 
making. To date, this has not occurred to the degree that it should. 
Most notably, while DON produced substantially well-documented cost 
estimates, the NGEN acquisition approach currently being followed is 
not grounded in a reliable analysis of alternative approaches, and the 
selected approach was not even assessed and is about $4.7 billion 
costlier and introduces more risk than the alternatives that were 
assessed. Further, the program's execution to date has not been based 
on the kind of reliably derived integrated master schedule that is 
essential to program success. While the program office is aware of 
some of the schedule weaknesses and intends to address them, 
additional work is needed to ensure that the schedule can produce 
credible dates for planned NGEN milestones and events. Exacerbating 
this is an equally troubling pattern of missed milestones and delays 
in key program documentation, as well as gate review decisions that 
have allowed the program to proceed in the face of significant 
performance shortfalls and risks. 

While NGEN is scheduled for an OSD-level milestone review in August 
2011, the above schedule limitations make it likely that this review 
date will slip. It is thus imperative, given the scope and nature of 
the program's problems, that swift and immediate action be taken to 
ensure that the most cost-effective acquisition approach is pursued 
and that a reliable schedule and performance-and risk-based decision 
making are employed. To do less increases the chances that needed NGEN 
capabilities will be delivered late and be more costly than necessary. 

Recommendations for Executive Action: 

To ensure that NGEN capabilities are acquired in the most cost- 
effective manner, we recommend that the Secretary of Defense take the 
following two actions: 

* direct the Under Secretary of Defense for Acquisition, Technology, 
and Logistics to conduct an interim NGEN milestone review, and: 

* direct the Secretary of the Navy to immediately limit further 
investment in NGEN until this review has been conducted and a decision 
on how best to proceed has been reported to the Secretary of Defense 
and congressional defense committees. 

At a minimum, this review should ensure that DON pursues the most 
advantageous acquisition approach, as evidenced by a meaningful 
analysis of all viable alternative acquisition approaches, to include 
for each alternative reliably derived cost estimates and metrics-based 
operational effectiveness analyses. In addition, the review should 
consider existing performance shortfalls and known risks, including 
those discussed in this report. 

To facilitate implementation of the acquisition approach resulting 
from the above review, we further recommend that the Secretary of 
Defense direct the Secretary of the Navy to take the following two 
actions: 

* ensure that the NGEN integrated master schedule substantially 
reflects the key schedule estimating practices discussed in this 
report, and: 

* ensure that future NGEN gate reviews and decisions fully reflect the 
state of the program's performance and its exposure to risks. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Deputy 
Assistant Secretary of Defense (C3, Space and Spectrum), and reprinted 
in appendix II, DOD stated that it concurred with one of our four 
recommendations, did not concur with one recommendation, and partially 
concurred with two. The department's comments are discussed below. 

* The department partially concurred with our recommendation to 
conduct an interim milestone review that provides assurance that DON 
is pursuing the most advantageous acquisition approach. Specifically, 
the department stated that it intended to leverage the next OSD-
chaired NGEN Overarching Integrated Product Team meeting in February 
2011 for the review and that following this meeting, the USD(AT&L) 
will conduct a Milestone Decision Authority review of the current NGEN 
approach, along with risks. According to the department, this approach 
balances the review processes already in place, resource constraints, 
and the need for an additional milestone review. Further, the 
department said it had concluded that DON's AOA was sufficient and 
that the analysis had been approved by CAPE. DOD added that it will 
complete an economic analysis--a post AOA-activity--for the August 
2011 Milestone C review, which will include a follow-on independent 
cost estimate and an updated determination of the most cost-effective 
solution. 

While these are important steps, DOD's planned actions do not appear 
to fully address our recommendation. Specifically, the department did 
not indicate any intent to reevaluate whether the current solution is 
indeed the most advantageous approach, despite the weaknesses 
contained in the AOA identified in this report and the fact that the 
current approach was not included in its analysis. According to the 
September 2010 draft NGEN economic analysis development plan, only the 
status quo and the current approach are to be analyzed, not the other 
three alternatives that were included in the AOA. Without a meaningful 
analysis of alternatives, DOD will be unable to determine the most 
cost-effective solution in its two upcoming key reviews. 

* The department did not concur with our recommendation that it limit 
further investment in NGEN until a decision has been made on how best 
to proceed based on an interim review that considers all viable 
alternative acquisition approaches and this decision has been reported 
to the Secretary of Defense and to congressional defense committees. 
The department stated that DON's NGEN acquisition strategy and program 
management have been approved by the milestone decision authority, and 
that adequate oversight is in place to ensure regulatory and statutory 
compliance. Further, the department said that the limitation on NGEN 
investments will impact future DON business operations and, 
ultimately, Naval warfighting capabilities. The department added that 
it will make adjustments to NGEN investments if it determines they are 
required; however, it also said it must continue to execute the 
investments within the time frame of the continuity of services 
contract. 

* While oversight is in place for the NGEN program, it is not 
effective. Specifically, as discussed in this report, DON's past 
reviews have resulted in decisions that were not always performance-
and risk-based. Given that DON is continuing to proceed in the face of 
the problems we are reporting, it is even more important that adequate 
oversight be provided by the Secretary and congressional defense 
committees. 

Moreover, we maintain that limiting further investment in NGEN, 
thereby delaying the Milestone C event and its associated activities, 
is the most prudent action at this time. By not evaluating all viable 
acquisition approaches before proceeding with further investment in 
NGEN, the department cannot be assured that it is pursuing the most 
cost-effective approach. Further, by selecting an approach that, as 
discussed in this report, carries greater relative schedule and 
performance risks than other alternatives and is being executed 
against an unreliable program schedule, the department increases the 
risk that its approach will lead to future cost overruns, requiring it 
to expend additional resources that could otherwise be used to provide 
other warfighting capabilities. Furthermore, even if the department 
proceeds along its current course, the issues we have identified with 
the program's schedule, along with the delays already experienced, 
raise concerns that it will be unable to complete the transition as 
planned within the time frames of the current continuity of services 
contract. 

* The department partially concurred with our recommendation that the 
Secretary of Defense direct the Secretary of the Navy to ensure that 
the NGEN integrated master schedule substantially reflects the key 
schedule estimating practices discussed in this report. DOD stated 
that the integrated master schedule was developed in accordance with 
industry best practices. However, as discussed in this report, none of 
the subschedules that we analyzed reflected all the practices that our 
work has identified as necessary to develop and maintain a reliable 
schedule. To its credit, DOD also said it would seek ways to improve 
schedule performance and that DON will review the scheduling practices 
discussed in this report and incorporate those found to be beneficial. 
We continue to believe that the Secretary of the Navy should ensure 
that the NGEN integrated master schedule incorporates all of the best 
practices for schedule estimating discussed in this report to help 
manage and measure its progress in executing the work needed to 
proceed through Milestone C and ultimately transition from NMCI to 
NGEN. 

* The department concurred with our recommendation to ensure that 
future NGEN gate reviews and decisions fully reflect the state of the 
program's performance and its exposure to risks. In this regard, the 
department stated that it plans to continue to conduct monthly risk 
management board meetings and program health reviews, and report the 
results to program leadership. It will be critical that decisions on 
NGEN fully reflect the state of the program's performance and exposure 
to risks. 

We are sending copies of this report to the appropriate congressional 
committees; the Director, Office of Management and Budget; the 
Congressional Budget Office; the Secretary of Defense; and the 
Secretary of the Navy. The report also is available at no charge on 
the GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staff members have any questions on matters discussed 
in this report, please contact me at (202) 512-6304 or 
melvinv@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. GAO staff who made major contributions to this report are 
listed in appendix III. 

Signed by: 

Valerie C. Melvin: 
Director, Information Management and Human Capital Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine whether (1) the Department of the 
Navy (DON) sufficiently analyzed alternative approaches for acquiring 
its Next Generation Enterprise Network (NGEN), (2) DON has a reliable 
program schedule for executing NGEN, and (3) acquisition decisions 
have been performance-and risk-based. 

To address the first objective, we evaluated the analysis of 
alternatives (AOA) report and its supporting documentation against 
relevant Department of Defense (DOD) guidance[Footnote 24] and GAO's 
Cost Estimating and Assessment Guide[Footnote 25] and compared the 
alternatives in the AOA final report with the NGEN Acquisition 
Strategy. More specifically, 

* For the cost analysis, we compared the AOA cost estimating 
documentation, such as the cost model spreadsheet, supporting 
documentation for the cost model, and the final NGEN AOA report, 
against the four characteristics of a reliable estimate[Footnote 26] 
in GAO's Cost Estimating and Assessment Guide to determine the extent 
to which the cost estimates reflected each of the four characteristics. 

* For the operational effectiveness analysis, we compared an NGEN 
alternatives performance assessment report and the AOA final report 
against the relevant DOD guidance to determine the extent to which the 
analysis was sufficient. In addition, we reviewed NGEN AOA Advisory 
Group meeting minutes and documentation containing the results of a 
Space and Naval Warfare Systems Command review of the cost analysis. 
We also interviewed cognizant DON and Office of the Secretary of 
Defense officials about the AOA's development and results. 

To address the second objective, we first reviewed the integrated 
master schedule and 4 of the 29 subschedules that existed when we 
began our review and that comprised the early transition activities 
intended to address key program risks, as well as high-level plans for 
postdeployment.[Footnote 27] Accordingly, we focused on assessing the 
May 2010 subschedules against the nine key schedule estimating 
practices[Footnote 28] in GAO's Cost Estimating and Assessment Guide 
using commercially available software tools to determine the extent to 
which each subschedule reflected each of the nine practices (e.g., a 
logical sequence of activities and reasonable activity durations). 
Further, we characterized the extent to which each subschedule 
satisfied each of the practices as either met, substantially met, 
partially met, minimally met, or not met.[Footnote 29] In addition, we 
compared the baseline schedule, established in December 2009, to the 
rebaselined schedule, established in August 2010, to identify whether 
key event and milestone dates had slipped. We also interviewed 
cognizant officials about development and management of the integrated 
master schedule and underlying subschedules. We also reviewed program 
documentation, such as the NGEN schedule management plan, program 
performance reports, program management reviews, and the acquisition 
strategy. 

To address the third objective we compared program review 
documentation, such as briefings, program performance assessments, and 
meeting minutes, to DON acquisition review policies and procedures, as 
well as to other programmatic documents, such as risk registers and 
risk management board briefings and meeting minutes. We also 
interviewed cognizant program officials regarding NGEN performance and 
program risks. 

To assess the reliability of the data that we used to support the 
findings in this report, we reviewed relevant program documentation to 
substantiate evidence obtained through interviews with agency 
officials. We determined that the data used in this report are 
sufficiently reliable. We have also made appropriate attribution 
indicating the sources of the data. 

We conducted this performance audit at DOD offices in the Washington, 
D.C., metropolitan area and at the Space and Naval Warfare Systems 
Command in San Diego, California, from October 2009 to February 2011 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Assistant Secretary Of Defense: 
Networks And Information Integration: 
6000 Defense Pentagon: 
Washington, DC 20301-6000: 

February 15, 2011: 

Ms. Valerie C. Melvin: 
Director, Information Management and Human Capital Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Melvin: 
This is the Department of Defense (DoD) response to the GAO Draft 
Report, GAO-11150, 'Information Technology: Better Informed Decision 
Making Needed on Navy's Next Generation Enterprise Network 
Acquisition,' dated December 15, 2010 (GAO Code 310679). We 
acknowledge receipt of the draft report and note that the Department's 
written comments are included. 

Our point of contact is Mr. Mark Godino at (703) 602-2720, ext 142. 
Please contact him with any questions or if you need clarification. 

Sincerely, 

Signed by: 

Dr. Ronald C. Jost: 
Deputy Assistant Secretary of Defense (C3, Space and Spectrum): 

Enclosure: As Stated: 

[End of letter] 

GAO Draft Report Dated December 15, 2010: 
GAO-11-150 (GAO Code 310679): 

"Information Technology: Better Informed Decision Making Needed On 
Navy's Next Generation Enterprise Network Acquisition" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Under Secretary of Defense for Acquisition, Technology and 
Logistics (USD(AT&L)) to conduct an interim Next Generation Enterprise 
Network (NGEN) milestone review. 

DOD Response: Partially Concur. The Department concurs with the 
finding that an interim milestone review may provide additional 
benefit, even though a thorough review was conducted by the Office of 
the Secretary of Defense (OSD) chaired, NGEN Overarching
Integrated Product Team (OIPT) in November 2010. The Department 
intends to further leverage the next upcoming OSD chaired NGEN OIPT 
scheduled for 17 February 2011, for this purpose. Following this OIPT 
meeting, the USD(AT&L) will conduct a Milestone Decision Authority 
(MDA) review of the current NGEN acquisition approach, along with the 
risks. This approach reconciles the balance between recognition of 
vetted consensus review processes already in place, resource 
constraints, and the need for an additional milestone review. The 
Department has concluded the Department of the Navy (DON) Analysis of 
Alternatives (AoA) was sufficient. The Office of the Director of Cost 
Assessment and Program Evaluation (CAPE) approved the NGEN Increment 1 
AoA in December 2009. Per Clinger-Cohen Certification and DoD 
requirements, a follow-on independent cost estimate and an updated 
determination of the most cost effective solution are completed as 
part of the Economic Analysis. The Economic Analysis is a post-AoA 
activity and will be completed at Milestone C, when the NGEN program 
is baselined (MDA approves Acquisition Program Baseline (APB)) in 
August 2011. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the Secretary of the Navy to immediately limit further 
investment in NGEN until this review has been conducted and a decision 
on how best to proceed has been reported to the Secretary and 
Congressional defense committees. 

DOD Response: Nonconcur. The DON's Next Generation Enterprise Network 
(NGEN) program management and investment strategy is well defined in 
its Acquisition Strategy, which was approved by the MDA and further 
clarified in subsequent Acquisition Decision Memorandums (ADMs). 
Additionally, the NGEN program has definitive, dedicated, and 
qualified acquisition oversight in place to ensure its regulatory and 
statutory compliance. The DON is presently in the midst of a limited-
period Continuity of Service Contract (CoSC) that is providing bridge 
services during the transition from the present Navy Marine Corps 
Intranet (NMCI) network to the future NGEN state. If, during the 
upcoming interim review by the USD(AT&L), the Department feels 
adjustments are required to the current or planned NGEN investments, 
actions will be taken at that time. Limitation of NGEN investments, at 
this time, will significantly impact future DON business operations 
and ultimately, Naval warfighting capabilities. In the near term, such 
limitations would impair Program Management Office activities critical 
to support planned Milestone C events in the Fourth Quarter of FY1 1. 
Timely execution of NGEN investments must continue in order to 
successfully transition from CoSC within the contractually obligated 
timeframe allotted. 

Recommendation 3: To facilitate implementation of the acquisition 
approach resulting from the above review, the GAO recommends that the 
Secretary of Defense direct the Secretary of the Navy to ensure that 
the (NGEN) integrated master schedule substantially reflects the key 
schedule estimating practices discussed in this report. 

DOD Response: Partially Concur. The NGEN Integrated Master Schedule 
(IMS) was developed in accordance with industry best practices for 
scheduling and was recently finalized and approved by the Program 
Manager (PM) on 26 August 2010. The NGEN IMS will continue to evolve 
as we move forward to Milestone C and an approved Acquisition Program 
Baseline (APB). As the NGEN IMS evolves, the Department will continue 
to seek out ways to improve our schedule performance. The Department 
will review the key schedule estimating practices discussed in this 
report and will consider incorporating those found to be beneficial. 

Recommendation 4: To facilitate implementation of the acquisition 
approach resulting from the above review, the GAO recommends that the 
Secretary of Defense direct the Secretary of the Navy to ensure that 
future NGEN gate reviews and decisions fully reflect the state of the 
program's performance and its exposure to risks. 

DOD Response: Concur. The NGEN PM will continue to conduct monthly 
formal Risk Management Board (RMB) and Probability of Program Success 
(POPS) (V2.0) reviews. Results of both of these formal meetings are 
regularly reported in the NGEN Program Dashboard, which is distributed 
weekly to program leadership and monthly to the Navy Senior 
Integration Board (NSIB) chaired by Principal Civilian Deputy 
Assistant Secretary of the Navy (Research, Development, and 
Acquisition) (PCDASN(RDA)). Additionally, program risk status is 
briefed at the OSD chaired OIPT meetings. Finally, both past and 
future NGEN gate reviews have and will continue to fully reflect the 
state of the program's performance and its exposure to risk. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie C. Melvin, (202) 512-6304, or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, key contributors to this 
report were Randolph C. Hite, Director; Carol Cha, Assistant Director; 
Monica Anatalio; Mathew Bader; Neil Doherty; Cheryl Dottermusch; James 
Houtz; Kaelin Kuhn; Neela Lakhmani; Lee McCracken; Jeanne Sung; and 
Adam Vodraska. 

[End of section] 

Footnotes: 

[1] The Naval Networking Environment is to be an iterative set of 
integrated, phased programs that share a common enterprise 
architecture and standards. It includes NGEN and the Consolidated 
Afloat Networks and Enterprise Services program, among others, and is 
to be in place by 2016. 

[2] NMCI provides about 382,000 workstations to approximately 700,000 
users across 2,500 Navy and Marine Corps locations around the world. 
NMCI is composed of transport infrastructure, such as cables, routers, 
and switches; end-user equipment, such as computers, monitors, and 
keyboards; and software. It provides, among other things, data 
storage, e-mail, transport of voice and data, and video-
teleconferencing. 

[3] The NMCI contractor will deliver in place the NMCI infrastructure 
and provide DON with a Government Purpose Rights license for NMCI 
technical data, computer software, and computer software documentation. 

[4] In addition to these contracts, DON plans to enter into an 
interagency agreement with the Defense Information Systems Agency to 
provide wide area network services. 

[5] NGEN program documentation defines initial transition for the Navy 
as execution and operation of NGEN services for 5 percent of the Navy 
management domain, and for the Marine Corps as execution and operation 
of 5 percent of Marine Corps seats. Final transition is defined as 
complete when increased government operational control and visibility 
are established and test and evaluation requirements are verified and 
validated for 80 percent of the Navy management domain and when 80 
percent of Marine Corps seats are transitioned and increased operation 
control of the network and visibility activities are implemented at 
the global, regional, and local level. 

[6] Department of Defense Instruction 5000.02, Operation of the 
Defense Acquisition System, Dec. 8, 2008, and Secretary of the Navy 
Instruction 5000.2D, Implementation and Operation of the Defense 
Acquisition System and the Joint Capabilities Integration and 
Development System (Oct. 16, 2008). 

[7] The DAS five-phase acquisition process is intended to translate 
mission needs and requirements into stable, affordable, and well- 
managed acquisition programs. 

[8] Secretary of the Navy Instruction 5000.2D, Implementation and 
Operation of the Defense Acquisition System and the Joint Capabilities 
Integration and Development System (Oct. 16, 2008). 

[9] The gate process had not been established at the time. DON 
reported that Gate 1 was nominally held in a series of Chief of Naval 
Operations Executive Board meetings during the winter of 2007/2008. 

[10] Gate 2 was initially waived because DON's original acquisition 
approach was to continue to use the existing NMCI technology and, 
therefore, a traditional analysis of alternative technologies was not 
planned. Subsequently, OSD identified the program as a pre-Major 
Automated Information System acquisition, and as a result an analysis 
of acquisition alternatives was directed. A Gate 2 review was 
conducted to review the AOA. 

[11] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 
2009); Defense Acquisition University, Defense Acquisition Guidebook, 
Section 3.3 "Analysis of Alternatives" (accessed Mar. 19, 2010). 

[12] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[13] DON officials stated that the NGEN AOA was not intended to be a 
traditional analysis to determine a system solution; rather, it was an 
analysis of alternative acquisition approaches. 

[14] Defense Acquisition Guidebook, Section 3.3 "Analysis of 
Alternatives" (Mar. 19, 2010); DON Acquisition and Capabilities 
Guidebook (December 2008); and Air Force Analysis of Alternatives 
Handbook (July 2008). 

[15] To support DON's Naval Networking Environment goals, NGEN plans 
to implement an IT service management framework to measure and report 
on all aspects of end-to-end IT service delivery. DON plans to use the 
IT Infrastructure Library best practices to implement this framework. 

[16] Defense Acquisition Guidebook, Section 3.3 "Analysis of 
Alternatives" (Mar. 19, 2010). 

[17] The critical path represents the chain of dependent activities 
with the longest total duration in the schedule. 

[18] Float is the amount of time a task can slip before affecting the 
critical path. 

[19] This schedule, considered to be an informal baseline schedule, 
was approved by the program manager to allow for later comparison to 
the current plan and to manage the overall scope of work to be 
conducted within the program. An official baseline schedule is 
expected to be approved by the USD (AT&L) as part of the Acquisition 
Program Baseline at Milestone C. 

[20] These schedules were the Transition Integrated Product Team, IT 
Service Management Process Development, Comprehensive Facilities 
Inventory Phase 2 Plan of Action and Milestones, and Contract 
Technical Representative Workforce Reconstitution. These subschedules 
represented the ongoing and/or planned work on NGEN to be completed 
prior to the deployment phase, as well as high-level plans for 
remaining work to be completed postdeployment. 

[21] GAO, Information Technology: Federal Agencies Need to Strengthen 
Investment Board Oversight of Poorly Planned and Performing Projects, 
[hyperlink, http://www.gao.gov/products/GAO-09-566] (Washington, D.C.: 
June 30, 2009). 

[22] Criteria are rated green, yellow, or red based on the extent to 
which the program meets necessary elements associated with that 
criteria. These measures can vary and are defined specifically for 
each criterion. For example, it could be based on the status of a 
deliverable (i.e., green if completed, yellow if in progress, or red 
if not yet started), the significance of issues (i.e., green if no 
issues, yellow if some issues, or red if significant issues), a 
percentage (i.e., green if less than 10 percent, yellow if between 10 
percent and 30 percent, or red if greater than 30 percent cost 
growth), or schedule (i.e., green if less than 6 months, yellow if 
between 6 and 12 months, or red if greater than 12 months schedule 
slip). 

[23] At the time of the Gate 4 review, DON was using the Probability 
of Program Success version 1 framework that consisted of 17 metrics. 
Technology Protection was added in version 2 of the framework. 

[24] Defense Acquisition University, Defense Acquisition Guidebook, 
Section 3.3 "Analysis of Alternatives" (accessed Mar. 19, 2010); DON 
Acquisition and Capabilities Guidebook (December 2008); and Air Force 
Analysis of Alternatives Handbook (July 2008). 

[25] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[26] These are (1) comprehensive, (2) well-documented, (3) accurate, 
and (4) credible. 

[27] These schedules were the Transition Integrated Product Team, IT 
Service Management Process Development, Comprehensive Facilities 
Inventory Phase 2 Plan of Action and Milestones, and Contract 
Technical Representative Workforce Reconstitution. 

[28] These are (1) capturing all activities, (2) sequencing all 
activities, (3) assigning resources to all activities, (4) 
establishing the duration of all activities, (5) integrating 
activities horizontally and vertically, (6) establishing the critical 
path for all activities, (7) identifying reasonable "float" between 
activities, (8) conducting a schedule risk analysis, and (9) updating 
the schedule using logic and durations. 

[29] "Met" means DON provided complete evidence that satisfies the 
entire criterion. "Substantially Met" means DON provided evidence that 
satisfies a large portion of the criterion. "Partially Met" means DON 
provided evidence that satisfies about half of the criterion. 
"Minimally Met" means DON provided evidence that satisfies a small 
portion of the criterion. "Not Met" means DON provided no evidence 
that satisfies any of the criterion. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: