This is the accessible text file for GAO report number GAO-12-223 
entitled 'Air Traffic Control Modernization: Management Challenges 
Associated with Program Costs and Schedules Could Hinder NextGen 
Implementation' which was released on February 16, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

February 2012: 

Air Traffic Control Modernization: 

Management Challenges Associated with Program Costs and Schedules 
Could Hinder NextGen Implementation: 

GAO-12-223: 

GAO Highlights: 

Highlights of GAO-12-223, a report to congressional committees. 

Why GAO Did This Study: 

The Federal Aviation Administration (FAA), partnering with other 
federal agencies and the aviation industry, is implementing the Next 
Generation Air Transportation System (NextGen), a new satellite-based 
air traffic management system that will replace the current radar-
based system and is expected to enhance the safety and capacity of the 
air transport system by 2025. Concurrently, FAA continues to maintain 
and upgrade existing air traffic control (ATC) systems that will also 
be needed for NextGen. This involves acquiring and implementing new 
software and hardware. 

GAO was asked to determine (1) how, if at all, costs and schedules of 
FAA ATC acquisitions programs, including those related to NextGen, 
have changed since they were first submitted to Congress, (2) the 
reasons for any such changes, and (3) the extent that selected ATC 
programs adhere to cost and schedule best practices. To do its work, 
GAO reviewed 30 programs and conducted cost and schedule analysis on 
four programs that had an approved baseline and were NextGen related. 
GAO reviewed acquisition documents and interviewed FAA officials. 

What GAO Found: 

In a review of 30 major ATC acquisition programs, all of which will 
contribute to the transition to NextGen, GAO found that costs for 11 
of the 30 programs have increased from their initial estimates by a 
total of $4.2 billion and 15 programs experienced delays. The 11 
acquisitions that experienced cost increases account for over 60 
percent of FAA’s total acquisition costs ($11 billion of $17.7 
billion) for the 30 programs. The 15 acquisitions that experienced 
schedule delays, of which 10 also had cost increases, ranged from 2 
months to more than 14 years and averaged 48 months. 

Cost increases and schedule delays occurred due to several factors, 
many of which have been longstanding challenges for FAA. Specifically, 
these have involved (1) additional or unanticipated system 
requirements; (2) insufficient stakeholder involvement (such as 
controllers’ input) throughout system development; (3) underestimating 
the complexity of software development; and (4) unanticipated events 
including funding shortfalls or work stoppages. These challenges, if 
they persist, will impede the implementation of NextGen, especially in 
light of the interdependencies among many acquisition programs, where 
cost increases or delays in one program can affect the costs and 
schedules of other programs. 

For the four programs GAO selected to analyze in depth, FAA is not 
consistently following the characteristics of high-quality cost 
estimates and scheduling best practices that GAO previously 
identified. Regarding cost estimates, GAO found that although all four 
of the programs generally provided well documented and comprehensive 
estimates, which are two of the four characteristics, no program fully 
met the two other characteristics. Specifically, each program estimate 
was not credible because each lacked an independent cost estimate, 
which provides a check against FAA’s estimate and three programs 
lacked risk or uncertainty analysis. The estimates also lacked 
accuracy because they were not updated regularly or based on 
comparable programs. Regarding scheduling practices, most programs did 
not substantially or fully meet the majority of the 9 best practices 
GAO previously identified including developing a fully integrated 
master schedule of all program activities and performing a schedule 
risk analysis. For example, without a schedule risk analysis, FAA is 
unable to predict, with any degree of confidence, if the estimated 
completion dates are realistic. FAA is implementing new processes and 
organizational changes to better manage acquisitions. However, by not 
consistently following the characteristics of high-quality cost 
estimate and scheduling best practices, FAA cannot provide reasonable 
assurance to Congress and other stakeholders that NextGen and other 
ATC programs will avoid additional cost increases or schedule delays. 

What GAO Recommends: 

To better estimate the cost and completion dates for major 
acquisitions, FAA should, among other things, require cost and 
schedule risk analysis, independent cost estimates and integrated 
master schedules. FAA did not comment on whether or not it agreed with 
the recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-12-223] or key 
components. For more information, contact Gerald L. Dillingham, Ph.D., 
at (202) 512-2834 or dillinghamg@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Most Ongoing ATC Programs Remain within Budget and on Schedule, but a 
Few Have Significantly Exceeded Initial Estimates: 

Schedules for Roughly Half of ATC Programs Are on Track, but Half of 
ATC Programs Are Delayed: 

Several Factors Contributed to Cost Increases and Schedule Delays, and 
Some Could Hamper NextGen Implementation: 

For Programs We Reviewed, FAA Did Not Consistently Meet All 
Characteristics of Quality Cost Estimates or Schedule Best Practices: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I Objectives, Scope and Methodology 57: 

Appendix II Detailed Purpose and Status for 30 Acquisitions GAO 
Reviewed: 

Appendix III Assessments of Four FAA Program Cost Estimates: 

Appendix IV Assessments of Four FAA Program Schedules: 

Appendix V GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: FAA Acquisition Life Cycle: 

Table 2: Characteristics of High-Quality Cost Estimates and Steps 
Related to Each Characteristic: 

Table 3: Eleven ATC Programs That Have Experienced Cost Increases as 
of August 2011: 

Table 4: Fifteen ATC Programs That Have Experienced Delays Based on 
Original Scheduled Completion Date: 

Table 5: Key Factors Contributing to Cost Growth, Schedule Delays, or 
Both for ATC Systems: 

Table 6: GAO Analysis of the Extent FAA Acquisition Cost Estimates Met 
the Characteristics of High-Quality and Reliable Cost Estimates: 

Table 7: Extent Program Schedules Met Best Practices: 

Table 8: GAO's Analysis of the FAA's ADS-B Cost Estimates: 

Table 9: GAO's Analysis of FAA's CATMT Cost Estimates: 

Table 10: GAO's Analysis of FAA's SWIM Cost Estimates: 

Table 11: GAO's Analysis of FAA's WAAS Cost Estimates: 

Table 12: Extent to Which the ADS-B Schedule Met Best Practices: 

Table 13: Extent to Which the CATMT Schedule Met Best Practices: 

Table 14: Extent to Which the SWIM Schedule Met Best Practices: 

Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met Best 
Practices: 

Table 16: Extent to Which the WAAS Contractor-Prepared Schedule Met 
Best Practices: 

Table 17: Probability of Project Completion: 

Table 18: Prioritized Risks at the 80th Percentile: 

Figures: 

Figure 1: Acquisition Programs Reported by FAA in the CIP for 2012 
through 2016: 

Figure 2: Improvements to Phases of Flight Expected under NextGen: 

Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B): 

Figure 4: Advanced Technologies and Oceanic Procedures (ATOP): 

Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh 
Segment 1: 

Figure 6: Aviation Surface Weather Observation Network (ASWON) and 
Automated Surface Observing System (ASOS) P31: 

Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6): 

Figure 8: Collaborative Air Traffic Management Technologies (CATMT), 
Work Packages 2 and 3 (WP2 and WP3): 

Figure 9: En Route Communication Gateway (ECG) Tech Refresh: 

Figure 10: En Route Automation Modernization (ERAM): 

Figure 11: Integrated Display Systems (IDS) Replacement: 

Figure 12: International Flight Inspection Aircraft (IFIA): 

Figure 13: Instrument Flight Procedure Automation (IFPA): 

Figure 14: Integrated Terminal Weather System (ITWS): 

Figure 15: Next Generation Air/Ground Communication System (NEXCOM) 
Segment 1a: 

Figure16: Next Generation Weather Radar (NEXRAD) Dual Polarization: 

Figure 17: Power Systems Sustained Support (PS3): 

Figure 18: Regulation and Certification Infrastructure for System 
Safety (RCISS) Segment 2: 

Figure 19: Runway Status Lights (RWSL): 

Figure 20: Standard Terminal Automation Replacement System (STARS) 
(TAMR Phase 1): 

Figure 21: System Wide Information Management (SWIM) Segment 1: 

Figure 22: Terminal Automation Modernization and Replacement (TAMR)--
Phase 2: 

Figure 23: Tower Training Simulator Systems: 

Figure 24: Trajectory Management--Arrival Tactical Flow Time Based 
Flow Management: 

Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life 
Extension Program (SLEP): 

Figure 26: Traffic Flow Management System (TFMS) Tech Refresh: 

Figure 27: Ultra High Frequency (UHF) Radio Replacement: 

Figure 28: Next Generation Voice Recorder Replacement Program (VRRP): 

Figure 29: Voice Switching and Control Switching System (VSCS) Tech 
Refresh Phase 2: 

Figure 30: Wide Area Augmentation System (WAAS): 

Figure 31: Weather and Radar Processor (WARP) Sustain: 

Figure 32: Weather Camera Program (WCP): 

[End of section] 

Abbreviations: 

ABRR: Airborne Reroute Execution: 

ADS-B: Automatic Dependent Surveillance-Broadcast: 

AirNav: Airports and Navigations Aids database: 

AMS: Acquisition Management System: 

APB: acquisition planning baseline: 

APTS: Automated Procedures Tracking System: 

ASDP: Advanced Signal Data Processor: 

ASOS: Automated Surface Observing System: 

ASR: Airport Surveillance Radar: 

ASWON: Aviation Surface Weather Observation Network: 

ATC: air traffic control: 

ATCBI-6: Air Traffic Control Radar Beacon Interrogator: 

ATCSCC: Air Traffic Control System Command Center: 

ATO: Air Traffic Organization: 

ATOP: Advanced Technologies and Oceanic Procedures: 

AUM: Arrival Uncertainty Management: 

CACR: Collaborative Airspace Constraint Resolution: 

CATMT: Collaborative Air Traffic Management Technologies: 

CIP: Capital Investment Plan: 

Cost Guide: GAO Cost Estimating and Assessment Guide: 

DataComm: Data Communications: 

DOD: Department of Defense: 

EBUS: Enhanced Back-up Surveillance: 

ECG: En Route Communication Gateway: 

ERAM: En Route Automation Modernization: 

ERIDS: En Route Information Display System: 

FAA: Federal Aviation Administration: 

F&E: facilities and equipment: 

I2I: Idea to In-Service Management: 

IDA: Investment Decision Authority: 

IDS: Integrated Display Systems: 

IFIA: International Flight Inspection Aircraft: 

IFP: Instrument Flight Procedures: 

IFPA: Instrument Flight Procedure Automation: 

IP&A: Investment Planning and Analysis: 

IPDS: Instrument Procedure Development System: 

IT: information technology: 

ITWS: Integrated Terminal Weather System: 

JRC: Joint Resources Council: 

NAS: National Airspace System: 

NextGen: Next Generation Air Transportation System: 

NEXCOM: Next Generation Air/Ground Communication System: 

NEXRAD: Next Generation Weather Radar: 

OE: Obstacle Evaluation: 

OMB: Office of Management and Budget: 

P3I: Pre-Planned Product Improvements: 

PAMRI: Peripheral Adapter Module Replacement Item: 

PS3: Power Systems Sustained Support: 

RCISS: Regulation and Certification Infrastructure for System Safety: 

RWSL: Runway Status Lights: 

SIP: SWIM implementation program: 

SLEP: Service Life Extension Program: 

SRA: schedule risk analysis: 

STARS: Standard Terminal Area Replacement System: 

SWIM: System Wide Information Management: 

TAMR: Terminal Automation Modernization and Replacement: 

TDWR: Terminal Doppler Weather Radar: 

Tech Center: FAA Technical Center: 

TERPS: Terminal Instrument Procedures: 

TFM: Traffic Flow Management: 

TFMS: Traffic Flow Management System: 

TRACON: Terminal Radar Approach Control: 

UHF: Ultra High Frequency: 

VRRP: Next Generation Voice Recorder Replacement Program: 

VSCS: Voice Switching and Control Switching System: 

WAAS: Wide Area Augmentation System: 

WAM: Colorado Wide Area Multilateration: 

WARP: Weather and Radar Processor: 

WCP: Weather Camera Program: 

WP2: Work Package 2: 

WP3: Work Package 3: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

February 16, 2012: 

Congressional Committees: 

To accommodate anticipated increases in air passenger traffic over 
time,[Footnote 1[ the Federal Aviation Administration (FAA) has 
expanded its acquisitions program to sustain current—-or legacy-—air 
traffic control (ATC) facilities and systems while simultaneously 
replacing or supplementing those systems through transition to the 
satellite-based Next Generation Air Transportation System (NextGen). 
This modernization effort involves acquiring and implementing new, 
advanced air traffic management systems, including hardware and 
software, to dramatically change the way the current aviation system 
is operated. As the agency transitions to NextGen, which has 
significantly increased the number, cost, and complexity of FAA’s 
acquisition programs, it is imperative that these programs remain on 
time and within budget, particularly given current budget constraints 
and the interdependencies of many NextGen-related ATC programs. FAA 
has taken several steps to improve its acquisition management––
including implementing a cost estimating methodology, a cost 
accounting system, and a business process, and developing an 
enterprise architecture––which resulted in the removal of its 
acquisition management from the GAO High-Risk list in 2009.[Footnote 
2] However, recent cost and scheduling problems among some major 
acquisition programs, such as the En Route Automation Modernization 
(ERAM), which is integral to ATC modernization, have renewed concerns 
about the agency’s ability to manage complex multibillion-dollar 
procurement programs. 

In response to your request, this report (1) describes how, if at all, 
the planned costs and schedules of current FAA ATC acquisition 
programs, including those related to NextGen, have changed since they 
were first submitted to Congress, (2) examines the reasons for any 
changes in planned costs and schedules, and (3) assesses the extent to 
which selected ATC programs adhere to best practices for determining
acquisition costs and schedules. 

To determine the changes, if any, in the costs and schedules of FAA’s
ATC programs, we gathered and analyzed agency data on the estimated
costs and schedules of the 30 ATC programs[Footnote 3] that had 
baselines-—that is, those programs whose estimated budget and schedule 
had received FAA executive approval.[Footnote 4] Eighteen of these 
programs are directly related to the implementation of NextGen, and 
all are needed to maintain and modernize the existing ATC system in 
order for it to operate in the NextGen environment. We also drew upon 
past GAO work in which we undertook detailed reviews of the status of 
ATC and other acquisition programs[Footnote 5] and obtained updated 
documentation as necessary from FAA. We interviewed FAA officials to 
obtain information on their programs’ past and current challenges and 
current status and summarized the status of all 30 ATC programs, 
including their original and current cost estimates and completion 
dates. For each program, we compared its initial estimated cost at the 
time of its submission to Congress for approval against its current 
cost estimate and compared its planned and actual schedules. 

To examine the reasons contributing to any changes in cost estimates
and schedules in the 30 baselined ATC programs, we interviewed FAA
officials and contractors and reviewed program documentation. We
analyzed information on cost increases and delays for the 30 baselined
programs to determine the factors that contributed to cost and schedule
changes. 

To assess the extent to which FAA’s cost estimating and scheduling
processes aligned with best practices, we conducted an in-depth review
of 4 of the 30 baselined programs: the Automatic Dependent
Surveillance-Broadcast (ADS-B) system, the Collaborative Air Traffic
Management Technologies (CATMT) system, the System Wide
Information Management (SWIM) system, and the Wide Area
Augmentation System (WAAS).[Footnote 6] We selected these programs 
based on the following criteria: the program had reached sufficient 
maturity such that risks could be identified and the program was key 
to the modernization of the air traffic control system. We conducted 
in-depth interviews with FAA acquisition program managers for the 
programs selected. In addition to interviews, we obtained and analyzed 
the most current cost and schedule estimates for these programs. To 
assess the extent to which FAA’s acquisition practices for these 
programs were consistent with best practices, we used the 2009 GAO 
Cost Estimating and Assessment Guide (Cost Guide).[Footnote 7] In 
assessing each program’s cost estimate, we used the Cost Guide to 
evaluate FAA’s estimating methodologies, assumptions, and results to 
determine whether the cost estimate was well-documented, 
comprehensive, accurate, and credible. We also determined the extent 
to which the schedule was prepared in accordance with best practices 
that are fundamental to having a reliable schedule. In addition, we 
performed a schedule risk analysis on a WAAS schedule prepared by the 
contractor to determine the high-priority risks that the program may 
encounter that could affect the schedule and the likelihood of the 
program finishing on time. We selected the WAAS contractor schedule 
because we determined that it was the only schedule of the four 
programs we reviewed in detail that was sufficiently reliable for a 
risk analysis to be performed. 

We conducted this performance audit from August 2010 to February 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Appendix I contains more
detailed information on our objectives, scope, and methodology. 

Background: 

FAA catalogs its acquisition programs in its annually updated Capital 
Investment Plan (CIP). The CIP identifies planned capital investment 
in the National Airspace System (NAS) for the next 5 years consistent 
with the amount requested in the agency's annual budget submission. 
Appendix C of the CIP, which identifies the anticipated budget line 
items, is divided into five activities: (1) Engineering, Development, 
Test, and Evaluation; (2) ATC Facilities and Equipment; (3) Non-ATC 
Facilities and Equipment; (4) Facilities and Equipment Mission 
Support; and (5) Personnel Compensation, Benefits and Travel. The CIP 
for fiscal years 2012 through 2016 contains 106 funded acquisition 
programs with estimated total budgets (through 2016) of more than $14 
billion. Of these 83 acquisition programs FAA considers ATC related, 
18 involve Engineering, Development, Test and Evaluation and 65 
involve ATC Facilities and Equipment. The 83 programs include 30 that 
have had program baselines approved by FAA's Joint Resources Council 
(JRC),[Footnote 8] which is responsible for approving major programs. 
[Footnote 9] These 30 baselined programs include communications, 
navigation, and surveillance systems that are key to ATC operations. 
FAA considers 5 of the programs to be foundational parts of NextGen, 
and all are key to modernizing the existing ATC system. Figure 1 
illustrates the universe of FAA acquisitions for fiscal years 2012-
2016. 

Figure 1: Acquisition Programs Reported by FAA in the CIP for 2012 
through 2016: 

[Refer to PDF for image: illustration] 

All acquisition programs: 106; 
ATC programs: 83; 
Major baselined programs: 30. 

Source: GAO analysis of FAA data. 

[End of figure] 

FAA has developed and uses its Acquisition Management System (AMS) to 
provide policies and guidance for managing ATC system programs through 
all phases of the programs' life cycles (see table 1). The Air Traffic 
Organization (ATO) within FAA is responsible for operating, 
maintaining, and modernizing the nation's current ATC system.[Footnote 
10] 

Table 1: FAA Acquisition Life Cycle: 

Needs and solution identification: 

Phase/activity: Mission analysis; 
What occurs during this phase or activity: Needs and solution 
identification: FAA identifies a capability shortfall and determines 
that it needs an investment to better carry out its mission. Recently, 
FAA began analyzing its mission needs within the context of its 
overall goals for the NAS. 

Phase/activity: Investment analysis; 
What occurs during this phase or activity: Needs and solution 
identification: FAA, using an investment analysis team, evaluates 
alternatives, selects practical and affordable solutions, and develops 
a baseline of cost, schedule, and performance requirements. This 
document is called the acquisition program baseline. 

Solution implementation: 

Phase/activity: System integration; 
What occurs during this phase or activity: Needs and solution 
identification: Both hardware and software components and subsystems 
are integrated into a product. Also, intra-and inter-system 
compatibility are tested and analyzed. 

Phase/activity: System demonstration; 
What occurs during this phase or activity: Needs and solution 
identification: Tests show that the product can work as required and 
be manufactured within targets. 

Phase/activity: System production; 
What occurs during this phase or activity: Needs and solution 
identification: All activities are carried out to produce needed 
quantities. Each end item is tested before it leaves the factory to 
verify that it conforms to specifications and is free from 
manufacturing defects. 

Phase/activity: In-service management; 
What occurs during this phase or activity: Needs and solution 
identification: All required activities are carried out, including 
directly operating, providing maintenance functions (both scheduled 
and unscheduled), and furnishing technical and logistics requirements. 

Phase/activity: Decommission; 
What occurs during this phase or activity: Needs and solution 
identification: All unneeded assets are decommissioned and removed 
from service at the end of their service lives. 

Source: FAA. 

[End of table] 

The acquisition program baseline defines the cost, schedule and 
performance baselines for the investment program. The JRC which 
determines whether to approve a cost and schedule baseline also 
approves rebaselining, through which the agency documents and approves 
major changes to a program's previously approved budget or schedule. 
Rebaselining resets the estimated costs and schedule used to determine 
how the program will be held accountable and can occur before the 
program is deployed. Once a program is rebaselined, FAA reports on the 
performance of the program based on the revised cost and schedule. 
Although the rationale for rebaselining can be reasonable, as, for 
example, when a program's scope has been expanded, reporting a 
program's performance based on a rebaselined cost or schedule can also 
skew or conceal from Congress and other stakeholders the program's 
actual total costs or overall timeline. We previously reported 
[Footnote 11] that the absence of this information on rebaselining in 
ATO's performance reporting could cause managers and other 
stakeholders, including Congress, to think that performance was better 
than it actually was. We recommended that FAA regularly report on the 
overall, long-term performance in acquiring ATC systems by providing 
in FAA's annual Performance and Accountability Report the original 
budget and schedule baselines for each rebaselined program and the 
reasons for the rebaselining. In response to our recommendation, FAA 
currently provides this information in Appendix D of its CIP, where it 
details baseline cost and schedule information for major acquisition 
programs. 

NextGen involves changes to every aspect of air transportation (see 
figure 2). NextGen requires the acquisition of new integrated systems 
(both software and hardware), flight procedures, aircraft performance 
capabilities, and supporting infrastructure to transform the current 
air transportation system into one that uses satellite-based 
surveillance and navigation operations instead of ground-based radar. 
[Footnote 12] These changes are intended to increase the efficiency 
and capacity of the air transportation system while maintaining safety 
and accommodating anticipated future growth. The planning for NextGen 
began in 2003[Footnote 13] and is now focused on implementing 
improvements in the midterm (by 2018) and in the far term (by 2025). 
[Footnote 14] 

Figure 2: Improvements to Phases of Flight Expected under NextGen: 

[Refer to PDF for image: illustration] 

Flight planning: 
            
Integrated flight planning: 
Allows immediate access to weather information through one data 
source.  

Push back/Taxi/Takeoff: 

Enhanced surface traffic operations: 
Data communications expedite clearances and reduce communication 
errors.  

Surface traffic management: 
Automation optimizes taxi routing by reducing taxi times and enhancing 
safety.  

Streamlined departure management: 
Allows multiple departure paths from each runway, thereby increasing 
departure capacity.  

Domestic/oceanic cruise: 

Efficient cruise: 
Standards for reduced separation between aircraft and consideration of 
weather conditions allow aircraft to fly most optimal path.  

Descent/Final approach/Landing: 

Streamlined arrival management: 
Equipped aircraft fly precise paths at reduced power from descent 
point to final approach. Time, fuel, emissions and holding are 
reduced.  

Enhanced surface traffic management: 
Detailed taxi route information sent via data communications to pilots 
prior to approach. Pilot and controller workload reduced and safety   
improved.  

Source: GAO analysis of FAA information. 

[End of figure] 

As noted previously, we selected 4 programs for in-depth review to 
determine the extent to which their cost estimates and schedules 
aligned with best practices: 

* ADS-B will enable aircraft to continually broadcast flight data such 
as position, air speed, and altitude, among other types of 
information, to air traffic controllers and other aircraft. The 
program was baselined in 2007, and FAA is currently installing the 
hardware and software at approximately 800 sites across the nation. 
The program is scheduled to be completed in 2014. 

* CATMT will provide new functionality and other enhancements to the 
existing Traffic Flow Management System (TFMS), such as automated 
reroutes and improved data exchanges between ATC facilities. CATMT was 
first baselined in 2008 and is scheduled to be completed in 2015. 

* SWIM will provide an information technology infrastructure that will 
enable the multiple systems that make up the NAS to share information. 
As such, SWIM is a portfolio of capabilities that will be implemented 
by other systems. SWIM was first baselined in 2009 and is scheduled to 
be completed in 2020. 

* WAAS will provide aircraft with more accurate aircraft position 
information to facilitate more direct flight paths and precision 
approaches to airports. The initial WAAS program was baselined in 
1998. The third segment of the program is scheduled to be completed in 
2013. 

Successful acquisition program management depends, in part, upon 
effective cost estimation. The cost estimate is the basis for 
establishing a program's detailed schedule (see following discussion 
of schedules), as well as identifying the bounds for how much program 
costs can be expected to vary. We have defined cost estimates as the 
summation of individual cost elements, using established methods and 
valid data, to estimate future program costs based on what is 
currently known.[Footnote 15] As such, cost estimating requires both 
science and judgment because answers are seldom--if ever--entirely 
precise. The goal is to find a reasonable estimate.[Footnote 16] 
Reliable cost estimating is a critical function without which agencies 
are at risk of experiencing cost overruns, missed deadlines, and 
performance shortfalls. 

Our Cost Guide identifies 12 steps consistently applied by cost-
estimating organizations throughout the federal government and 
industry and considered best practices for developing cost estimates. 
For the purposes of this review, we grouped these steps into four 
characteristics of high-quality and reliable estimates: well-
documented; comprehensive; accurate, and credible. (See table 2) 

Table 2: Characteristics of High-Quality Cost Estimates and Steps 
Related to Each Characteristic: 

Characteristic: Well-documented; 
Explanation: The documentation should address the purpose of the 
estimate, the program background, a description of the system, the 
system's schedule, the scope of the estimate (in terms of time and 
what is and is not included), the ground rules and assumptions, all 
data sources, the estimating and rationale, the results of the risk 
analysis, and a conclusion about whether the cost estimate is 
reasonable. Therefore, a good cost estimate--while taking the form of 
a single number--is supported by detailed documentation that describes 
how it was derived and how the expected funding will be spent in order 
to achieve a given objective. For example, the documentation should 
capture in writing such things as the source data used and their 
significance, the calculations performed and their results, and the 
rationale for choosing a particular estimating method or reference. 
Moreover, this information should be captured in such a way that the 
data used to derive the estimate can be traced back to and verified 
against their sources, allowing for the estimate to be easily 
replicated and updated. Finally, the cost estimate should be reviewed 
and accepted by management to ensure that there is a high level of 
confidence in the estimating process and in the estimate itself; 
Step identified in Cost Guide: 
Step 1: Define the estimate's purpose, scope, and schedule; 
Step 3: Define the program characteristics; 
Step 5: Identify ground rules and assumptions; 
Step 6: Obtain the data; 
Step 10: Document the estimate; 
Step 11: Present the estimate to management for approval. 

Characteristic: Comprehensive; 
Explanation: The cost estimates should include both government and 
contractor costs of the program over its full life cycle, from 
inception through design, development, deployment, and operation and 
maintenance to retirement. The estimates should also completely define 
the program, reflect its current schedule, and be technically 
reasonable. Comprehensive cost estimates should provide a level of 
detail appropriate to ensure that cost elements are neither omitted 
nor double-counted, and they should document all cost-influencing 
ground rules and assumptions. Establishing a product-oriented work 
breakdown structure is a best practice because it allows a program to 
track cost and schedule by defined deliverables, such as a hardware or 
software component; 
Step identified in Cost Guide: 
Step 2: Develop the estimating plan; 
Step 4: Determine the estimating structure; 
Step 5: Identify ground rules and assumptions. 

Characteristic: Accurate; 
Explanation: The cost estimates should provide for results that are 
unbiased, and they should not be overly conservative or optimistic. 
Estimates are accurate when they are based on an assessment of most 
likely costs, adjusted properly for inflation, and contain few, if 
any, minor mistakes. In addition, the estimates should be updated 
regularly to reflect material changes in the program, such as when 
schedules or other assumptions change, and actual costs so that the 
estimate is always reflecting the program's current status. Among 
other things, the estimate should be grounded in documented 
assumptions and a historical record of cost estimating and actual 
experiences on other comparable programs; 
Step identified in Cost Guide: 
Step 7: Develop the point estimate; 
Step 12: Update the estimate to reflect actual costs and changes. 

Characteristic: Credible; 
Explanation: The cost estimates should discuss any limitations of the 
analysis because of uncertainty or biases surrounding data or 
assumptions. Major assumptions should be varied, and other outcomes 
recomputed to determine how sensitive they are to changes in the 
assumptions. Risk and uncertainty analysis should be performed to 
determine the level of risk associated with the estimate. Furthermore, 
the estimate's results should be crosschecked, and an independent cost 
estimate conducted by a group outside the acquiring organization 
should be developed to determine whether other estimating methods 
produce similar results. For management to make good decisions, the 
program estimate must reflect the degree of uncertainty, so that a 
level of confidence can be given about the estimate. Having a range of 
costs around a point estimate is more useful to decision makers 
because it conveys the level of confidence in achieving the most 
likely cost and also informs them on cost, schedule, and technical 
risks; 
Step identified in Cost Guide: 
Step 7: Compare the point estimate to an independent cost estimate; 
Step 8: Conduct sensitivity analysis; 
Step 9: Conduct risk and uncertainty analysis. 

Source: GAO. 

[End of table] 

The success of a program also depends in part on having an integrated 
and reliable schedule, which defines, among other things, when work 
activities will occur, how long they will take, and how they are 
related to one another. As such, the program schedule not only 
provides a road map for systematic program execution, but also 
provides the means by which to gauge progress, identify and address 
potential problems, and promote accountability. Accordingly, a 
schedule helps ensure that all stakeholders understand both the dates 
for major milestones and the activities that drive the schedule. If 
changes occur within a program, the schedule helps decision makers 
analyze how those changes affect the program. We have previously 
identified nine best practices that help ensure a reliable program 
schedule (these best practices are discussed later in this report). 
The reliability of the schedule will determine the credibility of the 
program's forecasted dates, which are used for decision making. FAA is 
currently developing an integrated master schedule[Footnote 17] for 
the NextGen initiative that will be built in part on individual 
program schedules. The NextGen integrated master schedule is intended 
to be a comprehensive framework to support planned and actual work, 
providing decision makers with the information needed to manage the 
overall effort effectively. 

Most Ongoing ATC Programs Remain within Budget and on Schedule, but a 
Few Have Significantly Exceeded Initial Estimates: 

Most Cost Estimates Remain on Target, but Cost Overruns for Three Key 
Programs in Total Exceeded $4 Billion: 

Of the 30 baselined FAA ATC programs we reviewed,19 have not increased 
in cost, but 11 have experienced cost increases ranging from $2 
million to over $2 billion. And of the 19 programs whose costs have 
not increased, 7 experienced a cost decrease while the remainder have 
not changed significantly.[Footnote 18] However, the 11 programs that 
exceeded their initial estimated costs account for over 60 percent of 
total program costs for the 30 baselined programs--$11 billion of 
$17.7 billion. These 11 programs are among the most complex of FAA's 
major acquisitions in that each involves a large amount of software 
engineering. (See table 3) 

The 3 programs with the largest cost increases--totaling more than $4 
billion--are key to ATC modernization. Several factors contributed to 
cost overruns for the Standard Terminal Automation Replacement System 
(STARS), WAAS, and ERAM programs and required additional congressional 
appropriations or reductions in program scope. 

* Our previous work disclosed that the near tripling of the STARS's 
budget resulted from insufficient involvement of stakeholders and 
requirements growth.[Footnote 19] 

* The WAAS program began in 1998 with an initial cost estimate of $1 
billion[Footnote 20] and a current estimate of $3 billion. We reported 
previously that FAA's lack of scientific and technical expertise 
resulted in unplanned work and contributed to cost increases as well 
as delays in the deployment schedule. Additionally, FAA changed how it 
accounted for certain costs in the capital budget in 1999,[Footnote 
21] which further raised the cost estimate to $3.3 billion. FAA 
recently revised that estimate down to the current $3 billion during 
the 2009 rebaselining because, according to FAA officials, they had 
met certain program requirements in 2006. 

* As previously mentioned, ERAM is a key modernization system and will 
be the backbone of the NextGen system. FAA originally submitted to 
Congress an estimated cost of $2.1 billion in 2003, and the program is 
now expected to cost about $2.4 billion--an increase of about $330 
million. According to FAA, various software issues (e.g., unsuccessful 
transmission messages and inaccurate data pairing of aircraft and 
traffic display), as well as problems interfacing with other 
facilities and systems, have contributed to the cost increases and 
delays. The extent to which unanticipated requirements, unplanned 
work, and underestimates of the complexity of software development, 
among other factors, have contributed to other FAA ATC acquisition 
program cost overruns and scheduling delays is discussed later in this 
report. 

Table 3: Eleven ATC Programs That Have Experienced Cost Increases as 
of August 2011: 

Dollars in millions: 

Program: Wide Area Augmentation System[A]; 
Original start date: 1998; 
Initial estimate: $1,001; 
Current estimate: $3,008; 
Cost increase: $2,007; 
Percentage increases: 199%. 

Program: Standard Terminal Automation Replacement System; 
Original start date: 1996; 
Initial estimate: $940; 
Current estimate: $2,719; 
Cost increase: $1,779; 
Percentage increases: 189%. 

Program: En route Automation Modernization; 
Original start date: 2003; 
Initial estimate: $2,154; 
Current estimate: $2,484; 
Cost increase: $330; 
Percentage increases: 15%. 

Program: Automated Dependent Surveillance--Broadcast[C]; 
Original start date: 2007; 
Initial estimate: $1,682; 
Current estimate: $1,726; 
Cost increase: $44; 
Percentage increases: 3%. 

Program: Aviation Surface Observation Weather Network[D]; 
Original start date: 2001; 
Initial estimate: $351; 
Current estimate: $384; 
Cost increase: $33; 
Percentage increases: 10%. 

Program: Runway Status Lights; 
Original start date: 2010; 
Initial estimate: $327; 
Current estimate: $352; 
Cost increase: $25; 
Percentage increases: 8%. 

Program: UHF Replacement; 
Original start date: 2002; 
Initial estimate: $85; 
Current estimate: $93; 
Cost increase: $8; 
Percentage increases: 9%. 

Program: International Flight Inspection Aircraft[B]; 
Original start date: 2003; 
Initial estimate: $27; 
Current estimate: $34; 
Cost increase: $6; 
Percentage increases: 23%. 

Program: Integrated Terminal Weather System[B]; 
Original start date: 1997; 
Initial estimate: $276; 
Current estimate: $282; 
Cost increase: $6; 
Percentage increases: 2%. 

Program: Terminal Doppler Weather Radar; 
Original start date: 2003; 
Initial estimate: $75; 
Current estimate: $77; 
Cost increase: $2; 
Percentage increases: 3%. 

Program: Tower Training Simulators; 
Original start date: 2007; 
Initial estimate: $34; 
Current estimate: $36; 
Cost increase: $2; 
Percentage increases: 7%. 

Program: Total; 
Original start date: [Empty]; 
Initial estimate: $6,952; 
Current estimate: $11,195; 
Cost increase: $4,243; 
Percentage increases: [Empty]. 

Source: GAO analysis of FAA data. 

[A] According to FAA, the WAAS program is currently divided into three 
phases--two have been completed and the third is projected to be 
completed by 2013. However, in the CIP for fiscal years 2012 through 
2016, the WAAS program is listed as a single program costing $3 
billion. 

[B] Program completed during GAO review. 

[C] According to the FAA, the ADS-B current revised budget includes 
congressional directed spending of $9.4 million in fiscal year 2008 
and $6.8 million in fiscal year 2009. 

[D] This program includes the ongoing pre planned product improvement 
segment. 

[End of table] 

Schedules for Roughly Half of ATC Programs Are on Track, but Half of 
ATC Programs Are Delayed: 

Half of ATC Programs Are on or Ahead of Schedule and Half Are Delayed: 

We found that 15 of the 30 baselined programs either have experienced 
no change in schedule or were completed early or on time; however, the 
other 15 programs are projected to be completed later than originally 
estimated. These delayed programs range from the Integrated Display 
System, which will consolidate information from several weather 
subsystems into a single display, which FAA expects to be completed 2 
months after its initial estimated completion date, to WAAS, which FAA 
estimates will be completed in 2013--more than 14 years after its 
initial estimated completion date (see table 4). Ten of the 15 
programs with schedule delays also experienced cost increases. 
However, even if a schedule delay does not result in a direct cost 
increase to that program, the delay can lead to increased costs for 
FAA because FAA staff must continue to manage the acquisition over the 
longer term as it is being implemented, as well as maintain any legacy 
system that the program is replacing. Because of program 
interdependencies, a schedule delay can also affect how and when other 
programs will be implemented. 

Table 4: Fifteen ATC Programs That Have Experienced Delays Based on 
Original Scheduled Completion Date: 

Program: Wide Area Augmentation System[A]; 
Original start date: January 1998; 
Original planned completion date: August 1999; 
Projected completion date: September 2013; 
Delay or projected delay (in months): 169. 

Program: Air Traffic Control Beacon Interrogator-6; 
Original start date: August 1997; 
Original planned completion date: September 2004; 
Projected completion date: January 2012; 
Delay or projected delay (in months): 88. 

Program: Integrated Terminal Weather System[B]; 
Original start date: June 1997; 
Original planned completion date: July 2003; 
Projected completion date: Aug 2010; 
Delay or projected delay (in months): 85. 

Program: Next Generation Air/Ground Communication System Segment 1; 
Original start date: September 1998; 
Original planned completion date: September 2008; 
Projected completion date: September 2013; 
Delay or projected delay (in months): 60. 

Program: Airport Surveillance Radar-11[D]; 
Original start date: November 1997; 
Original planned completion date: September 2005; 
Projected completion date: June 2010; 
Delay or projected delay (in months): 57. 

Program: Terminal Doppler Weather Radar; 
Original start date: February 2003; 
Original planned completion date: December 2013; 
Projected completion date: September 2017; 
Delay or projected delay (in months): 4. 

Program: En route Automation Modernization; 
Original start date: June 2003; 
Original planned completion date: December 2010; 
Projected completion date: August 2014; 
Delay or projected delay (in months): 44. 

Program: Aviation Surface Weather Observation Network[C]; 
Original start date: August 2001; 
Original planned completion date: September 2009; 
Projected completion date: September 2012; 
Delay or projected delay (in months): 36. 

Program: UHF Replacement; 
Original start date: November 2002; 
Original planned completion date: September 2010; 
Projected completion date: September 2013; 
Delay or projected delay (in months): 36. 

Program: International Flight Inspection Aircraft; 
Original start date: December 2003; 
Original planned completion date: August 2009; 
Projected completion date: May 2012; 
Delay or projected delay (in months): 33. 

Program: Voice Switching and Control System (Tech Refresh) Phase 2; 
Original start date: August 2006; 
Original planned completion date: June 2012; 
Projected completion date: December 2014; 
Delay or projected delay (in months): 30. 

Program: Standard Terminal Automation Replacement System; 
Original start date: February 1996; 
Original planned completion date: October 2005; 
Projected completion date: June 2007; 
Delay or projected delay (in months): 20. 

Program: Tower Training Simulator[B]; 
Original start date: December 2007; 
Original planned completion date: September 2009; 
Projected completion date: August 2010; 
Delay or projected delay (in months): 11. 

Program: Runway Status Lights; 
Original start date: January 2010; 
Original planned completion date: October 2015; 
Projected completion date: June 2016; 
Delay or projected delay (in months): 8. 

Program: Integrated Display System Replacement; 
Original start date: September 2008; 
Original planned completion date: October 2015; 
Projected completion date: December 2015; 
Delay or projected delay (in months): 2. 

Source: GAO analysis of FAA data. 

[A] According to FAA, the WAAS program is divided into three phases--
two have been completed and the third is projected to be completed by 
2013. However, in the CIP for fiscal years 2012 through 2016, the WAAS 
program is listed as a single program costing $3 billion. 

[B] Program completed during GAO review. 

[C] Program includes the ongoing pre planned product improvement 
segment. 

[D] The ASR-11 Program was re-baselined in September 2005 with a 
planned completion of September 2009. 

[End of table] 

Several Factors Contributed to Cost Increases and Schedule Delays, and 
Some Could Hamper NextGen Implementation: 

A Number of Factors Have Contributed to Cost Increases and Delays: 

Cost increases and schedule delays occurred because of several 
factors, all of which have been long-standing challenges for FAA and 
some of which continue to affect programs despite FAA efforts to 
mitigate the factors. Specifically, these factors include (1) 
additional, unanticipated system requirements work; (2) insufficient 
stakeholder involvement throughout system development; (3) 
underestimates of the complexity of software development; and (4) 
unanticipated events, including funding decreases or work stoppages 
(see table 5). Of the 30 programs we reviewed, 15 experienced cost 
increases, schedule delays, or both,[Footnote 22] and we were able to 
determine that cost increases or schedule delays for 11 were 
attributable to one or more of these factors.[Footnote 23] 

Table 5: Key Factors Contributing to Cost Growth, Schedule Delays, or 
Both for ATC Systems: 

Name of ATC system: Airport Surveillance Radar-11; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Name of ATC system: EnRoute Automation Modernization; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Check]; 
Underestimating the complexity of software development: [Check]; 
Unanticipated events: [Empty]. 

Name of ATC system: Next Generation Air-Ground Communications; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Name of ATC system: Runway Status Lights; 
Unanticipated requirements or unplanned work: [Empty]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Name of ATC system: Voice Switching and Control System-Tech Refresh 
Phase 2; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Name of ATC system: Air Traffic Control Beacon Intergrorator-6; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Name of ATC system: Integrated Weather System; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Check]; 
Unanticipated events: [Check]. 

Name of ATC system: International Flight Inspection Aircraft; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Empty]. 

Name of ATC system: Wide Area Augmentation System; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Check]; 
Underestimating the complexity of software development: [Check]; 
Unanticipated events: [Empty]. 

Name of ATC system: Standard Terminal Automation Replacement System; 
Unanticipated requirements or unplanned work: [Check]; 
Insufficient stakeholder involvement: [Check]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Empty]. 

Name of ATC system: UHF Radio Replacement; 
Unanticipated requirements or unplanned work: [Empty]; 
Insufficient stakeholder involvement: [Empty]; 
Underestimating the complexity of software development: [Empty]; 
Unanticipated events: [Check]. 

Source: GAO analysis of FAA data. 

[End of table] 

Following are some examples of how these contributing factors led to 
cost increases or schedule delays in some of FAA's ATC baselined 
programs: 

* Unanticipated requirements or work: For nine of the programs in 
table 5, FAA has had to undertake substantially more development work 
than planned because FAA program officials originally misjudged the 
extent to which commercial off-the-shelf nondevelopmental solutions, 
such as those procured by another agency, would meet FAA's needs. For 
example, although WAAS was being developed by an integrated product 
team that included representatives from several FAA offices, the team 
did not effectively resolve problems in meeting a required performance 
capability--that pilots be warned in a timely manner when a system may 
be providing them with potentially misleading and possibly hazardous 
information. These actions resulted in unanticipated work and 
contributed to the rise in WAAS's cost from the original estimate of 
$509 million in 1994 to about $2 billion in 2005. 

* Insufficient stakeholder involvement: As we previously reported, 
[Footnote 24] ERAM was designed at a time when air traffic controllers 
did not participate in efforts to design and test new systems. Because 
active users of the system from different locations could not provide 
insight early on, issues that could have been addressed early in the 
design phase were not addressed. In response, FAA has taken steps to 
improve the testing of new systems in order to reduce the likelihood 
of larger-than-anticipated software issues arising during system 
implementation. For example, FAA and the controllers' union recently 
entered into a memorandum of understanding to bring controllers into 
the testing and evaluation phase of ERAM.[Footnote 25] Under this 
agreement, the controllers' union will have ERAM technical, 
evaluation, and training representatives, as well as a team of 16 
controllers (including 12 from en route facilities and 4 from terminal 
facilities), who will be detailed to test and validate software fixes 
with contractor engineers at the FAA Technical Center (Tech Center). 
In addition, our previous work disclosed that the near tripling of the 
Standard Terminal Automation Replacement System's budget resulted from 
insufficient involvement of stakeholders and requirements growth--two 
systemic factors that we found led to acquisitions missing their 
budget and schedule targets. This, in turn, contributed to cost 
growth, schedule delays, and eventually a reduction in the number of 
systems to be deployed.[Footnote 26] 

* Underestimates of the complexity of software development: This 
factor contributed to cost increases and schedule delays for ERAM, as 
well as issues with costs, scheduling, or both for two other programs. 
In 2010, FAA tested ERAM at two key sites (the Seattle and Salt Lake 
en route centers) on live air traffic, usually late at night when air 
traffic volume was low. During this testing, FAA encountered both 
anticipated and unanticipated software issues, which prompted the test 
sites, at times, to revert to using FAA's legacy en route computer 
system. Specifically, software instructions to a controller in one 
sector to hand off control of an aircraft to a controller in an 
adjacent sector failed, and flight data were lost or reassigned to 
another flight. While some testing at FAA's Tech Center preceded 
testing at the two key sites, the Tech Center could only test limited 
scenarios, and none of the scenario testing identified this software 
error. In addition, as discussed earlier, ERAM was designed during a 
time when air traffic controllers did not participate in efforts to 
design and initially test new systems. FAA anticipated the potential 
for software issues at the outset of the program and initially 
scheduled approximately 6 to 9 months of contingency time between the 
time it achieved initial operating capability[Footnote 27] and 
operational readiness demonstrations at these sites, leaving little 
buffer for any potential delays. FAA worked with its contractor to 
correct a number of software issues, but further testing on live air 
traffic at the two test sites continued to produce critical safety 
errors. As a result, in March 2010, FAA decided, with the support of 
the air traffic controllers' union, to halt all ERAM testing on live 
traffic and to revise the deployment schedule. The program was 
rebaselined in June 2011, and the program's completion date was 
extended from December 2010 to August 2014. As a result of the 
schedule delays, the rebaselined cost estimate increased from $ 2.1 
billion to $2.4 billion.[Footnote 28] 

* Unanticipated events: Unanticipated events at implementation sites 
and unanticipated funding issues have delayed several programs' 
schedules and increased costs. For example, Airport Surveillance Radar-
11 was originally scheduled to be completed in June 2009 but was 
delayed to June 2010. FAA reports indicated that the delay was due to 
an unusually protracted real estate acquisition at one site and issues 
involving validating performance during seasonal radar operations from 
other another site. Similarly, FAA's Runway Status Lights program--
which involves installing airport lighting equipment that visually 
signals to pilots when it is unsafe to enter, cross, or begin takeoff 
on a runway--has experienced schedule delays because of construction 
issues at five sites (Charlotte, North Carolina; Fort Lauderdale, 
Florida; Las Vegas; Minneapolis; and Washington-Dulles). FAA officials 
attributed some of these delays to the furlough of some FAA employees 
in July 2011 and a freeze on contractor funding during the furlough, 
which resulted in work stoppage orders for several projects--including 
Runway Status Lights. FAA program managers will need to assess the 
impact of the furlough on other programs that had experienced work 
stoppage orders, including ADS-B, the Standard Terminal Automation 
Replacement System, SWIM, WAAS, and various weather programs.[Footnote 
29] 

Interdependencies among NextGen ATC Programs Could Slow NextGen 
Implementation: 

The interdependencies of ATC acquisition programs have become more 
prominent as the NextGen program shifts from planning to 
implementation, so that cost increases and schedule delays in one 
program could have a cascading effect on other programs. As discussed 
earlier, due to the integrated nature of NextGen, the development and 
delivery of many of its component programs are mutually dependent on 
the development and delivery of one or more other programs. For 
example, ERAM, FAA's new en route computer system, is critical to the 
delivery of ADS-B capabilities such as broadcasting flight 
information. ERAM is also pivotal to the on-time implementation of two 
other key NextGen programs--Data Communications (DataComm),[Footnote 
30] which is estimated to cost about $3 billion, and the NextGen 
information technology architecture, SWIM, which is estimated to cost 
over $550 million. Due in part to ERAM's delay, FAA was forced to 
delay the Data Communications baseline date by approximately 6 months, 
rebaseline SWIM segment 1, and delay the SWIM segment 2 baseline date 
to 2012. The longer-term effects of these delays are unclear, but 
certain SWIM capabilities could be delayed for several years, and the 
progress of other programs that are dependent on SWIM's system 
integration could be hindered, as well. Thus, looking more broadly, 
the implementation of NextGen--both midterm (now through 2018) and far-
term (2019-2025) schedules--will be affected by how well FAA manages 
program interdependencies. 

As we reported in 2010,[Footnote 31] individual FAA program offices 
understand their programs' dependence on ERAM's implementation, but 
FAA has not developed a full listing of how ERAM schedule slippages, 
or slippages in other programs that are critical to NextGen, could 
either affect other programs' implementation schedules or delay the 
implementation of capabilities and improvements.[Footnote 32] In 2008, 
we recommended that FAA improve the usefulness of ATO's acquisition 
performance reporting by including information in the agency's 
Performance and Accountability report or elsewhere on the potential 
effect of any budget or schedule slippages on the overall transition 
to NextGen.[Footnote 33] This recommendation remains open, as FAA has 
not definitively indicated how it will track slippages that will 
affect other dependent NextGen programs. FAA's acquisition management 
system was not designed for managing NextGen programs in an integrated 
way. To assist in managing NextGen portfolios, FAA is starting to 
monitor all the activities of a particular operational improvement to 
ensure integration is on track. As we noted in the 2010 report, as 
this approach is more fully implemented, it will likely clarify the 
impact of slippages in one program's schedule on the implementation 
status of other NextGen programs and operational capabilities. In 
addition, as we will discuss in the next section, FAA is developing an 
Integrated Master Schedule for the entire NextGen initiative that is, 
in part, intended to show how changes in program schedules affect 
other programs and the timelines for the NextGen initiative as a 
whole. However, as we discuss later, the schedules for the four 
programs we reviewed in detail are not reliable. Reliable schedules at 
the program level will be needed to develop a reliable Integrated 
Master Schedule for NextGen. 

Program Management Changes Are Aimed at Addressing Past Challenges: 

According to FAA, it is taking actions to address the factors that 
have contributed to cost increases and schedule delays. In 2011, FAA 
assessed the NextGen effort as part of its Foundation for Success 
initiative[Footnote 34] and has implemented the "Idea to In-Service 
Management" (I2I) process, which it believes will result in 
improvements in the way the FAA develops, acquires, and implements new 
NextGen capabilities from conception through implementation. The I2I 
concept is intended to improve collaboration early in the acquisition 
process, resulting in better defined capabilities and an early 
indication of cost and benefits. These enhancements are intended to 
resolve many of the challenges associated with overall program 
management and enable FAA to focus on program management best 
practices. FAA believes that I2I will also result in improvements in 
specific areas that have presented challenges in the past, such as 
cost estimating, anticipating requirements and work, stakeholder 
collaboration, software development, and systems integration. 

Also in 2011, FAA implemented a reorganization of the NextGen 
Operations and Planning Office and ATO which FAA believes will support 
the I2I process and improve acquisitions of NextGen programs. 
Specifically, FAA created a NAS Lifecycle Planning Division within the 
NextGen Operations and Planning Office to focus the integration of 
NextGen programs from a cost, schedule, and systems capability 
perspective. Within ATO, FAA established a new Program Management 
Office, which puts the responsibility for the program management of 
all NextGen and other major ATC acquisitions within a single 
organization. By combining program managers into one organization, FAA 
hopes to create a stronger acquisition program and improve the 
consistency and implementation of best practices. According to FAA, 
these organizational changes allow responsibilities for acquisitions 
to be better defined to more efficiently set strategic direction, 
define operational requirements, ensure system integration, oversee 
implementation processes, and ensure accountability throughout the 
acquisition life cycle. 

To improve the acquisitions management process, FAA has also divided 
large acquisition programs into segments. A segmented or phased 
approach is being taken with programs like SWIM and CATMT. This 
approach breaks a larger program into smaller and more manageable 
pieces to lower the risk. In the past, we noted that this approach can 
improve management by providing for midcourse corrections and, thus, 
help FAA avoid costly late-stage changes. However, this approach can 
also increase the duration and possibly the cost of the 
program.[Footnote 35] According to FAA officials, a segmented approach 
allows the agency to more effectively manage acquisitions at both the 
program and enterprise architecture level. An enterprise architecture 
approach provides the structure to relate organizational mission, 
vision, and goals to business processes and the technical or 
information technology infrastructure required to execute them. FAA 
officials stated that many of the factors we identified that 
contributed to cost increases and schedule delays highlight the need 
for an enterprise-level perspective throughout the acquisition 
process. The I2I process is intended to provide an enterprise-level 
focus and improve collaboration across related programs. 

For Programs We Reviewed, FAA Did Not Consistently Meet All 
Characteristics of Quality Cost Estimates or Schedule Best Practices: 

Selected Cost Estimates Were Generally Comprehensive and Well-
Documented, but Accuracy and Credibility Need to Be Improved: 

Our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates showed 
that while each program followed at least some of the four 
characteristics of high-quality and reliable cost estimates--well-
documented, comprehensive, accurate, and credible--none of the 
programs adhered closely enough to those characteristics to create a 
reliable cost estimate. As previously noted, these characteristics 
incorporate the 12 steps consistently applied by cost-estimating 
organizations throughout the federal government and industry and 
considered best practices for developing cost estimates. The results 
of our review of the ADS-B, CATMT, SWIM, and WAAS cost estimates, 
which are summarized in table 6, show that they were most aligned with 
the characteristic of comprehensive cost estimates but need 
improvement in the other three areas, particularly with the 
characteristics of accurate and credible estimates. Imprecise 
estimates can result in Congress unnecessarily authorizing and 
appropriating millions of dollars for programs. As noted, in some 
cases, FAA, in order to stay within the original cost estimate, 
modified a program's requirements and Congress had to appropriate more 
funds or reduce the scope to allow FAA to finish the program. 

FAA could better ensure that the cost estimates for these four 
programs, as well as its other major acquisition programs, are 
reliable. Our work shows that an assessment of these cost estimates 
for these programs, as well as FAA's other major acquisition programs, 
would allow FAA to better understand if its cost estimation guidelines 
and our characteristics of high-quality cost estimates are in fact 
being followed (See table 6). 

Table 6: GAO Analysis of the Extent FAA Acquisition Cost Estimates Met 
the Characteristics of High-Quality and Reliable Cost Estimates: 

Characteristic: Well-documented; 
Characteristic description: The cost estimate should be supported by 
detailed documentation that describes the purpose of the estimate, the 
program background and system description, the scope of the estimate, 
the ground rules and assumptions, all data sources, estimating 
methodology and rationale, and the results of the risk analysis. 
Moreover, this information should be captured in such a way that the 
data used to derive the estimate can be traced back to, and verified 
against, the sources; 
ADS-B: Partially met; 
CATMT: Partially met; 
SWIM: Substantially met; 
WAAS: Substantially met. 

Characteristic: Comprehensive; 
Characteristic description: The cost estimates should include costs of 
the program over its full life cycle, provide a level of detail 
appropriate to ensure that cost elements are neither omitted nor 
double counted, and document all cost-influencing ground rules and 
assumptions; 
ADS-B: Substantially met; 
CATMT: Substantially met; 
SWIM: Substantially met; 
WAAS: Substantially met. 

Characteristic: Accurate; 
Characteristic description: The cost estimate should be based on an 
assessment of most likely costs (adjusted for inflation), documented 
assumptions, historical cost estimates, and actual experiences on 
other comparable programs. Estimates should be cross-checked against 
an independent cost estimate for accuracy, double counting, and 
omissions. In addition, the estimate should be updated to reflect any 
changes; 
ADS-B: Partially met; 
CATMT: Partially met; 
SWIM: Partially met; 
WAAS: Partially met. 

Characteristic: Credible; 
Characteristic description: The cost estimates should discuss any 
limitations of the analysis because of uncertainty, or biases 
surrounding data or assumptions. Risk and uncertainty analysis should 
be performed to determine the level of risk associated with the 
estimate. Further, the estimate's results should be cross-checked 
against an independent estimate; 
ADS-B: Partially met; 
CATMT: Minimally met; 
SWIM: Minimally met; 
WAAS: Partially met. 

Source: GAO analysis of FAA documents. 

Note: "Not met" means the program provided no evidence that satisfies 
any of the best practice criteria. "Minimally met" means the program 
provided evidence that satisfies a small portion of the criteria. 
"Partially met" means the program provided evidence that satisfies 
about half of the criteria. "Substantially met" means the program 
provided evidence that satisfies a large portion of the criteria. 
"Met" means the program provided evidence that completely satisfies 
the criteria. 

[End of table] 

Because the four programs were generally similar in the extent to 
which they met each of the four characteristics, the following 
discussion summarizes the strengths and weaknesses we found for each 
characteristic across the four programs. A more detailed discussion of 
our findings is contained in appendix IV. 

* Well-documented. Two of the four cost estimates we analyzed 
substantially met the characteristic of being well-documented; the 
other two partially met this characteristic. A well-documented cost 
estimate is thoroughly documented, including identifying specific 
source data and their significance, detailing calculations and 
results, and explaining why particular cost estimating methods were 
chosen. In other words, sufficient documentation exists such that an 
unfamiliar analyst could recreate the cost estimate and arrive at the 
same results. For example, the SWIM estimate provided detailed 
documentation describing the program, in addition to the methodology, 
calculations, and quantities used to develop the estimate. However, 
none of the four estimates sufficiently captured the entire source 
data used, addressed its reliability, or described how various forms 
of data from disparate sources were normalized (i.e., the data were 
described in like terms). For example, the WAAS estimate was based, in 
part, on actual labor costs from a previous contract, but the program 
office provided no evidence that these data were evaluated for 
reliability or accuracy. Similarly, the CATMT estimate routinely 
relied on subject matter expertise as a source for assumptions, such 
as the cost of labor, but the estimate did not document the experts' 
qualifications, background, underlying assumptions, or data sources. 
Moreover, we noted that three of the four estimates often 
substantially relied on expert opinion rather than on data. While 
expert opinion can be useful in the absence of data, it is subjective 
and generally should be used sparingly for cost estimates. Since data 
are the foundation of every cost estimate, data quality affects the 
overall quality of the estimate. In addition, because data are 
gathered from a variety of sources and take many different forms, 
normalization helps to improve consistency with other cost information 
and enable valid comparisons and projections. 

* Comprehensive. All four cost estimates we analyzed substantially met 
the characteristic of being comprehensive. For an estimate to be 
comprehensive, it should include full life-cycle costs, completely 
define the program with sufficient detail, include cost elements that 
are traceable to the statement of work or objective to ensure they are 
neither omitted nor double counted, and document all cost-influencing 
ground rules and assumptions. We found that the ADS-B, CATMT, and SWIM 
cost estimates included all life-cycle costs, regardless of program 
phase or funding source, and the ADS-B and SWIM cost estimates 
completely defined the program with an appropriate level of detail. In 
particular, the ADS-B cost estimate included cost estimates for both 
government and contractor costs, and the WAAS cost estimate thoroughly 
defined the program and reflected the current schedule. The four 
estimates did not fully meet the comprehensive characteristic because 
they lacked evidence that all cost influencing ground rules and 
assumptions were considered. 

* Accurate. None of the four cost estimates met or substantially met 
the characteristic of being accurate. The estimates generally adjusted 
costs for inflation and contained few computation or mathematical 
mistakes, but they were not regularly updated to reflect schedule and 
requirement changes, did not provide evidence of documenting or 
reviewing differences between planned and actual costs, and were not 
based on historical cost data from comparable programs. For example, 
the ADS-B, CATMT, and SWIM cost estimates provided no evidence that 
they were updated to reflect program changes, such as schedule 
slippages or varying assumptions, and did not include the current 
actual costs of the program. Although the WAAS estimate included 
evidence that it was updated to reflect major changes in technical and 
program requirements, such as the four rebaselinings the program has 
undergone since its 1998 inception, it did not include evidence that 
estimated costs were replaced with actual costs as the program 
advanced. Cost estimates that are not regularly updated with current 
information cannot provide decision makers with accurate information 
that is necessary, for example, when new system requirements are 
called for under tight budget conditions. In addition, comparing 
planned and actual costs enables cost estimators to measure the 
accuracy of their estimates and refine their processes. In addition, 
none of the four programs more than minimally used historical data to 
develop their cost estimates. Had historical data been used, the 
estimators would have had additional insight into actual costs on 
programs that used similar technologies, which could be used, for 
example, to challenge overly optimistic assumptions and bring more 
realism to the cost estimate. 

* Credible. None of the four cost estimates met or substantially met 
the characteristic of being credible, which includes obtaining an 
independent cost estimate from a group outside the acquiring 
organization, and cross-checking the major cost elements in that 
estimate against cost drivers identified through sensitivity and risk 
analyses. The ADS-B, CATMT, SWIM, and WAAS estimates lacked 
credibility largely because FAA did not obtain an independent cost 
estimate for any of the programs. In addition the CATMT, SWIM, and 
WAAS estimates provided little evidence that it conducted sensitivity 
or risk analyses. Instead, each program received independent cost 
reviews as part of the investment decision process--even though such 
reviews are not required by FAA policy. FAA stated that the 
Investment, Planning and Analysis (IP&A) Office in the FAA Finance 
Organization does not prepare independent estimates, but it is 
organizationally independent of the acquisition programs and conducts 
independent reviews of all cost estimates. However, an independent 
cost review is less rigorous than an independent cost estimate. 
According to our cost guide, an independent cost estimate is often 
more accurate because the estimating team is further removed from the 
program office and less prone to accept overly optimistic assumptions 
or be burdened by organizational bias.[Footnote 36] Other federal 
agencies, including the Department of Defense, require independent 
cost estimates. Had an independent cost estimate been completed, the 
estimating team and program team could have identified the major 
differences between their estimates, reconciled those differences 
where possible, and provided a synopsis of the two estimates and their 
differences to acquisition program management. In addition, without 
sensitivity and risk analyses, cost estimators cannot measure the 
effects of varying assumptions, and managers cannot determine, for 
example, the rational level of contingency reserves necessary to cover 
increased costs that may result from uncertainties such as unexpected 
design complexity, changes in requirements, or budget shortfalls--all 
of which FAA ATC programs, and in particular NextGen programs, have 
experienced in recent years. We found evidence that some level of risk 
analysis was conducted for ADS-B, CATMT, and SWIM, although the 
analysis was not sufficiently robust. For example, key cost drivers 
were not identified, and additional context about how the estimate 
could be affected by software design and development issues was not 
included. 

Selected Schedules Did Not Substantially Meet Most Best Practices and 
Are, Therefore, Unreliable: 

We determined that the schedules for the four programs we reviewed are 
unreliable because none met or substantially met all nine of the best 
practices for developing a reliable schedule. (see table 7). For 
example, none of the schedules fully met best practices for capturing 
all activities in an integrated master schedule, identifying critical 
paths and reasonable float for all activities, or assigning resources 
to those activities. Moreover, none of the schedules had documentation 
that provided more than minimal evidence that they conducted a 
schedule risk analysis. As was the case with our review of cost 
estimates for the four programs, our work regarding the schedules for 
these programs shows that an assessment of the schedules, as well as 
schedules for FAA's other major acquisition programs, would allow FAA 
to understand if the nine best practices for reliable schedules are 
being followed. 

Table 7: Extent Program Schedules Met Best Practices: 

Best practice: 1. Capturing all activities; 
Description: A schedule should reflect all activities defined in the 
program's work breakdown structure and include all activities to be 
performed by the government and contractor; 
ADS-B (FAA prepared): Partially met; 
CATMT (contractor prepared): Substantially met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Minimally met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 2. Sequencing all activities; 
Description: The schedule should be planned so that all activities are 
logically sequenced in the order they are to be carried out; 
ADS-B (FAA prepared): Minimally met; 
CATMT (contractor prepared): Partially met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Minimally met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 3. Assigning resources to all activities; 
Description: The schedule should realistically reflect the resources 
(i.e., labor material and overhead) needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
ADS-B (FAA prepared): Substantially met; 
CATMT (contractor prepared): Substantially met; 
SWIM (FAA prepared): Not met; 
WAAS (FAA prepared): Minimally met; 
WAAS (contractor prepared): Partially met. 

Best practice: 4. Establishing the duration of all activities; 
Description: The schedule should reflect how long each activity will 
take to execute; 
ADS-B (FAA prepared): Substantially met; 
CATMT (contractor prepared): Met; 
SWIM (FAA prepared): Partially met; 
WAAS (FAA prepared): Partially met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 5. Integrating schedule activities horizontally and 
vertically; 
Description: The schedule should be horizontally and vertically 
integrated--that is, it should link already sequenced activities with 
outcomes while also delineating the relation of supporting tasks and 
subtasks to upper-level milestones. Such mapping among levels enables 
different groups to work to the same master schedule; 
ADS-B (FAA prepared): Partially met; 
CATMT (contractor prepared): Partially met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Partially met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 6. Establishing the critical path; 
Description: The schedule should identify the critical path, or those 
activities that, if delayed, will negatively impact the overall 
project completion date. The critical path enables analysis of the 
effect delays may have on the overall schedule; 
ADS-B (FAA prepared): Not met; 
CATMT (contractor prepared): Partially met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Minimally met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 7. Identifying reasonable float; 
Description: The schedule should identify float--the amount of time an 
activity can slip in the schedule before it affects other activities--
so that flexibility in the schedule can be determined. As a general 
rule, activities along the critical path typically have the least 
amount of float; 
ADS-B (FAA prepared): Not met; 
CATMT (contractor prepared): Minimally met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Partially met; 
WAAS (contractor prepared): Substantially met. 

Best practice: 8. Conducting a schedule risk analysis; 
Description: The schedule should include a schedule risk analysis that 
uses statistical techniques to predict the probability of meeting a 
completion date. A schedule risk analysis can help management identify 
and understand the most important risks and focus on mitigating them; 
ADS-B (FAA prepared): Minimally met; 
CATMT (contractor prepared): Not met; 
SWIM (FAA prepared): Minimally met; 
WAAS (FAA prepared): Not met; 
WAAS (contractor prepared): Minimally met. 

Best practice: 9. Updating the schedule using logic and durations to 
determine dates; 
Description: The schedule should use realistic durations for 
activities and be monitored to determine when forecasted completion 
dates differ from the planned dates. This analysis can be used to 
assess whether schedule variances will affect future work; 
ADS-B (FAA prepared): Substantially met; 
CATMT (contractor prepared): Substantially met; 
SWIM (FAA prepared): Partially met; 
WAAS (FAA prepared): Partially met; 
WAAS (contractor prepared): Partially met. 

Source: GAO analysis of FAA documents. 

Note: "Not met" means the program provided no evidence that satisfies 
any of the best practice criteria. "Minimally met" means the program 
provided evidence that satisfies a small portion of the criteria. 
"Partially met" means the program provided evidence that satisfies 
about half of the criteria. "Substantially met" means the program 
provided evidence that satisfies a large portion of the criteria. 
"Met" means the program provided evidence that completely satisfies 
the criteria. 

[End of table] 

Because the scheduling best practices are interrelated in such a way 
that deficiencies in one best practice will cause deficiencies in the 
others, a schedule must meet or substantially meet all nine practices 
to be reliable. For example, preparing a schedule that is program-
wide--including an integrated breakdown of the work to be performed by 
both the government and its contractors over the expected life of the 
program--is a best practice. If the schedule does not capture all 
activities, then there will be uncertainty about whether activities 
are sequenced correctly or if the schedule properly reflects the 
resources needed to accomplish the work, which is also a best 
practice. Logic and durations (that is, the time it takes to complete 
a specific activity) should be used and maintained to ensure realistic 
start and completion dates and to reflect the true status of the 
project--a necessary condition for conducting follow-on schedule risk 
analyses. Moreover, if activities are not properly sequenced with 
logical links, it will not be certain if the critical path--which 
represents the chain of dependent activities with the longest total 
duration--is valid. Collectively, the weaknesses in not fully meeting 
or substantially meeting all nine key practices increases the risk of 
schedule slippages and cost overruns since a well-defined schedule 
helps to identify the amount of human capital and fiscal resources 
that are needed to execute the program. Therefore, by not having 
reliable schedules, FAA cannot conduct meaningful oversight of an 
acquisition program's progress or determine whether the program is 
achieving the desired results. 

The following discussion summarizes the extent to which the schedules 
for the four programs we examined met best practices. More detailed 
information for each program regarding scheduling best practices is 
presented in appendix V. 

ADS-B: 

We reviewed the schedule prepared by FAA and found it did not fully 
meet any of GAO's nine scheduling best practices, resulting in an 
unreliable schedule. Evidence provided in the ADS-B schedule indicates 
that it substantially met three of the nine best practices and 
partially, minimally, or did not meet the other six. For example, 
although the ADS-B schedule provided evidence of periodic updating, it 
did not capture all of the effort currently called for in the approved 
baseline for the entire ADS-B program and, therefore, was not a fully 
integrated schedule. Without fully integrating government activities 
with contractor activities, and thereby capturing all key activities, 
the schedule will not reliably estimate the program's completion. In 
addition, the ADS-B schedule we reviewed did not identify critical 
paths or include a schedule risk analysis, which uses statistical 
techniques to predict a level of confidence in meeting a program's 
completion date; did not logically sequence all activities and 
establish their durations; and had excessive float[Footnote 37] on a 
majority of current and planned activities. According to program 
officials, a number of the issues our analysis identified were, in 
part, the result of the schedule's limited time frame, which covered 
only a defined transitional period (October 2010 through April 2011) 
during which responsibility for about a third of the effort passed 
from the FAA to its prime contractor. Officials also stated that 
although their schedule contains critical activities, it has not had a 
traditional critical path since its contractor began managing the 
deployment of deliverables. The FAA uses contract options to order the 
scope, sequence, and requirements for key milestones. Within those 
options, the contractor has the authority to implement the sequence of 
more discrete activities in the order they deem most appropriate. FAA 
program officials plan to rectify this problem, noting that with 
negotiations now completed, they will in the near future identify a 
critical path to span all program milestones. 

CATMT: 

Because the CATMT program did not prepare an FAA schedule and instead 
relied on its contractor schedule, we reviewed the contractor 
schedule, which we found to be unreliable. Our analysis found that the 
contractor's CATMT schedule substantially or fully met four of the 
nine best practices: capturing all activities, assigning resources, 
establishing the durations, and updating the schedule. For example, 
the CATMT contactor schedule pertains to the current phase of the 
program that is being implemented in software releases, or phases. 
However, there was no overarching FAA government owned schedule that 
accounts for all software releases for the entire program and would 
thus delineate the relation of current software release tasks to the 
upper-level milestones for the overall CATMT program. The CATMT 
schedule included detailed resource information, and the program 
office provided evidence that resources are tracked in detailed labor-
hour spreadsheets. We also found that 90 percent of the activities 
were of short duration and that the program office regularly reviews 
the schedule, which is in line with best practices. On the other hand, 
five of the nine best practices were either partially, minimally, or 
not met. Specifically, the CATMT schedule lacked evidence indicating 
that it established a critical path, accurately identified float 
between activities, integrated the schedule vertically and 
horizontally, sequenced all activities, or performed a schedule risk 
analysis. Regarding the critical path, our analysis determined that 
the CATMT schedule does not identify a critical path for the entire 
program. Instead, the program is being accomplished multiple 6-month 
spirals[Footnote 38]; thus, there is only a critical path for each 
software release, not for the program as a whole.[Footnote 39] Without 
a valid program-wide critical path FAA management cannot determine 
which tasks, if they slip, will have the most detrimental effects on 
the project finish date. We also found that 68 percent of the 
remaining activities to be completed had unreasonably high float 
exceeding 1,000 days, meaning that those activities could slip about 5 
work years[Footnote 40] without affecting the overall project finish 
date, a highly unlikely scenario.[Footnote 41] The accurate 
identification of critical paths and float are inextricably linked. 
For example, if the schedule is missing activities or they are not 
correctly linked, float estimates will be miscalculated, resulting in 
an invalid critical path. Without a schedule that can produce a true 
critical path, the program office will neither be positioned to 
provide reliable timeline estimates nor be able to identify when 
problems or changes may occur and determine the impact they may have 
on subsequent work. CATMT program officials acknowledged that the 
schedule did not include program-wide critical paths but noted that a 
critical path exist for individual segments of the program. They also 
noted that a schedule risk analysis was not performed because it was 
not a contractual deliverable. 

SWIM: 

Because the SWIM schedule did not fully or substantially meet any of 
GAO's nine scheduling best practices, we found it to be unreliable. 
The SWIM program differs from the others in that it is an aggregation 
of NextGen acquisition programs, each developing an aspect of the SWIM 
information sharing capability. Because SWIM program managers are 
reliant on schedule information from a number of other programs, SWIM 
schedule integration is particularly important. However, our analysis 
found that the SWIM schedule was not, by any measure, fully integrated 
because it provided only a synopsis of the individual system 
implementing program schedules and, thus, did not fully represent the 
work required to complete the overall SWIM program. This resulted in 
float calculations that were unrealistic, and the resulting critical 
path calculations were invalid. In addition, while the many missing 
activities negatively impacted the schedule logic and the accuracy of 
durations, it also made the accurate allocation of resources and 
comprehensive integration of schedule activities, both horizontally 
and vertically, impossible. We also noted that FAA made no effort to 
identify a program-wide critical path. Program officials said that 
because each of the system implementing program schedules has its own 
critical path, involve disparate capabilities, and are independent of 
one another their individual critical paths are not accessible through 
the SWIM schedule software. They therefore are not used for overall 
SWIM program management. We believe that the SWIM program itself 
should have its own critical path that includes, at a minimum, 
acceptance of major deliverables from the system implementing program 
schedules. Without a program-wide critical path, management does not 
have a clear picture of the underlying project tasks that must be 
performed to achieve the overall program target completion date. 
Finally, although there was no risk analysis conducted on this 
schedule, our analysis found that this best practice was minimally met 
because a risk analysis was conducted on a separate but related 
schedule, and the SWIM program office considered risk to some extent. 

WAAS: 

Like the other three programs, we found the WAAS program schedule 
prepared by FAA unreliable because it did not fully or substantially 
meet any of GAO's nine scheduling best practices; however, we also 
reviewed the contractor's schedule for the same segment and found it 
fully or substantially met six best practices.[Footnote 42] For 
example, FAA's WAAS program schedule did not fully sequence activities 
in the order in which they are to be carried out. More specifically, 
the WAAS program schedule showed nearly half of the remaining 
activities were missing sequenced logic, causing us to question the 
calculated dates of activities. Logic is necessary for a schedule to 
show program managers when activities are expected to start and 
finish; when logic is missing, activity dates cannot adjust correctly 
to changes in activities. To test the ability of the schedule to 
dynamically update its dates due to changes, we artificially extended 
the duration of an activity to 1,500 days, which changed the 
activity's finish date. However, the duration extension had no effect 
on successor activities because this activity is not tied to any 
successor activities. Extending the duration to 1,500 days also pushed 
the project planned finish date from September 22, 2016, to June 29, 
2017; however, because the logic links are not in place, we questioned 
whether the projected finish date under this scenario is reliable. 

Moreover, the WAAS program schedule had too many artificial 
constraints that were driving the start and finish dates for more than 
70 percent of the remaining activities.[Footnote 43] Constraints are 
usually substitutes for logic and can mean that the schedule is not 
well planned or feasible. Constraints also greatly reduce the ability 
of the program to take advantage of possible time savings. Further, 
our analysis found that the schedule did not fully capture or assign 
resources to all government and contractor activities; it also did not 
accurately allocate resources or consistently establish the duration 
of activities. In addition, while WAAS program officials told us that 
the schedule was integrated vertically and horizontally, we did not 
find evidence of such integration. Furthermore, we found the WAAS 
program office's schedule did not identify a critical path for the 
entire program. As noted earlier, critical path and float 
determinations are closely related. Our analysis of the WAAS program 
office schedule found that more than half of the remaining activities 
had float of more than 1,000 working days, which we believe to be 
unreasonably high. Without proper determination of float, management 
cannot properly reallocate resources from tasks to other tasks without 
adversely affecting the overall completion date. Although program 
officials said that they maintained a risk register listing the 
potential risks that could impact the schedule and adjusted the 
schedule for these risks, we did not find evidence that the program 
office had conducted a risk analysis of its schedule. 

While the schedule prepared by the contractor did not fully or 
substantially meet three of the scheduling best practices, it fully or 
substantially met six: capturing, sequencing, assigning resources to, 
and establishing the duration of all activities; establishing the 
critical path; and identifying reasonable float between activities. 
For example, our analysis found that all activity durations are 
consistently estimated in days and adhere to a standard 5-day workweek 
that accounts for holidays, and no activities were scheduled to begin 
on a weekend. Officials from the contractor said duration estimates 
for the schedule are based on historical information from past 
performance, comparable releases, lessons learned, similar work, and 
other data requirements. In addition, our analysis traced several 
critical paths in the schedule. Though we found minor interruptions in 
the various critical paths, the schedule's logic, reasonable 
durations, and low total float estimates allow the calculation a valid 
critical path. 

WAAS Schedule Risk Analysis Indicates Risks That Managers Could 
Mitigate to Avoid Delays: 

As noted, FAA did not perform a complete schedule risk analysis for 
any of the four programs we reviewed and, thus, cannot accurately 
estimate these programs' completion dates with confidence. A schedule 
risk analysis, which is one of our best practices for program 
scheduling, uses statistical techniques to predict a level of 
confidence in meeting a program's completion date. The objective of 
the analysis is to develop a probability distribution of possible 
completion dates that reflect the project and its identified risks. 
This analysis can help program managers both understand the most 
important risks to the program and focus on mitigating those risks. 
Other federal agencies, including the Department of Defense and the 
National Aeronautics and Space Administration, require schedule risk 
analysis for major acquisitions; the Department of Veterans Affairs, 
in response to a GAO recommendation,[Footnote 44] plans to require 
schedule risk analysis for major construction projects. 

We conducted a schedule risk analysis on the WAAS contractor prepared 
schedule, which we chose because it was relatively mature, it 
partially met or substantially met six of the nine best practices (see 
table 7), and it contained enough information to perform a schedule 
risk analysis.[Footnote 45] We reviewed the risk register that the 
contractor had developed, which showed four potential risks to the 
project. We then conducted interviews with FAA program and contractor 
staff and asked them to discuss other potential risks to the project, 
including how the risk would affect the project's timeline and the 
likelihood of the risk occurring. Using this information we identified 
an additional 16 risks for a total of 20 risks. The fact that our 
interviews identified a relatively large number of new risks could be 
an indication that the contractor did not systematically analyze the 
full range of risks when developing the program's risk register. We 
then consolidated the 20 risks into 14 broader risks and tested how 
each would impact the duration of specific activities in the schedule. 
We then ran a Monte Carlo simulation,[Footnote 46] which consisted of 
the computer-generated results of 3,000 estimates of the future 
schedule based on the activities in the schedule, the chance that some 
of the activities would be affected by some risks and the predicted 
effect of those risks on the duration of each activity. 

We then analyzed the potential impact of risks on the program 
schedule. Since risks can effect the schedule in various ways--for 
example, risks can have a large impact on the durations of activities 
they affect, or they can introduce critical paths that are different 
from the baseline critical path--we analyzed the marginal impact of 
each of the risks we identified to determine which would have the 
greatest effect on the overall schedule. We found the following three 
key risks to the program, only the first of which (limited WAAS 
program office resources) was originally identified by the contractor. 
The three risks are: 

* limited WAAS program office resources such as staffing; 

* delays in software yet to be released and additional changes to 
software already released and in use; and: 

* a potentially optimistic schedule completion date. 

Our schedule risk analysis showed the completion of the segment of the 
WAAS program covered by the schedule could slip as much as 2 months. 
Specifically, the analysis showed that there is less than a 5 percent 
probability that the program segment would be completed by September 
6, 2012, the current baselined date for completion. However, it 
appears that the segment will be completed close to the deadline since 
we found a 50 percent probability that the program segment will be 
completed by October 23, 2012 (about 1.5 months after the current 
estimated date for completion); and an 80 percent probability that the 
program will be completed by November 13, 2012 (about 2.25 months 
after the current estimated date for completion). 

Although we did not conduct a schedule risk analysis for other FAA 
programs, the result of our analysis provides examples of the types of 
risks that major acquisition programs face and the impact those risks 
can have on meeting acquisition program milestones, especially given 
the interrelation and interdependencies among NextGen acquisitions 
discussed earlier. More information on our schedule risk analysis can 
be found in appendix V. 

Lack of Reliable Program Schedules Will Hinder Development of an 
Overall NextGen Integrated Master Schedule: 

FAA has begun developing an integrated master schedule for the entire 
NextGen initiative that would, in part, capture related NextGen 
program schedules, governance activities, and other performance and 
financial data to provide real-time monitoring of the overall NextGen 
effort. However, the unreliability of the four program schedules for 
programs that are integral to the NextGen initiative puts this high-
level master schedule at risk. Having a reliable integrated master 
schedule would enable FAA to determine how delays in one program 
impact other programs and the overall NextGen implementation timeline. 
While it is encouraging that FAA is beginning to develop an integrated 
NextGen master schedule, the effort could be hampered by the lack of 
schedule integration at the program level, as well as the failure of 
individual program schedules to meet best practices. For example, 
since FAA does not perform schedule risk analysis on individual 
programs, it cannot predict with certainty if any of the programs will 
be completed on time. Therefore, the integrated master schedule for 
NextGen would be built on schedules that may not reflect accurate 
program completion dates. Similarly, none of the four schedules we 
reviewed, which were for segments of the entire program, had reflected 
how tasks for the segment affected milestones for the entire program. 
Without integrated schedules at the program level, an integrated 
master schedule at the NextGen initiative level would be problematic. 

FAA has Taken Steps to Improve Acquisition Program Cost Estimates and 
Schedules: 

In response to our review of the extent that the four selected 
acquisition programs met best practices for cost estimates and 
schedules, FAA provided information on steps it is taking to improve 
its processes for both cost estimates and schedules and noted that 
some of the cost estimates and schedules we reviewed were developed 
before the improvements were in place. FAA stated that strengthening 
its cost estimation process is part of the seven key acquisition 
processes it has developed, including program management, contractor 
management, requirements, risk management, measurement and analysis, 
verification and validation, and quality assurance. FAA stated that it 
has updated its Guidelines for FAA Cost Estimating to be consistent 
with the GAO Cost Guide, filling in gaps that it had identified during 
a comparison of its practices to those contained in the Cost Guide. As 
of November 2011, 11 of the 12 best practices are addressed in the 
guidelines. According to FAA officials, the remaining best practice--
involving the creation of independent cost estimates--is unlikely to 
be implemented at FAA in the foreseeable future because FAA believes 
the resources required to create independent estimates are prohibitive 
in current budget environments. FAA has more than tripled the number 
of cost estimators in the Investment Planning and Analysis 
organization, many of which work with the acquisition program offices 
to provide guidance on preparing estimates. Additionally, as part of 
FAA's effort to improve acquisition certification and training, the 
agency is preparing to launch a cost estimating certification program. 
Coupled with a competency-based training program, FAA believes the 
certification program will enhance and improve consistency of the 
skills of FAA cost estimators. 

In describing its efforts to improve schedules, FAA stated that it 
views the development and maintenance of integrated schedules as an 
inherent and critical part of its seven key acquisition functions. FAA 
noted that included in its standard process for acquisition schedules 
are toolkits that require programs to develop integrated program 
schedules that address all nine of GAO's best practices. FAA stated 
that the current procedures for developing best practices were not 
fully in place when the four programs we reviewed began the 
implementation phase. 

Conclusions: 

FAA has made improvements in its management of air traffic control 
modernization acquisitions, and most of the 30 we reviewed are 
currently within the original cost estimate and half are on schedule. 
FAA is also taking steps to address past issues to ensure cost 
estimates and schedules are more accurate in the future, including 
incorporating best practices in its acquisitions guidance and 
policies. Nevertheless, our review of the FAA acquisitions found that 
it has yet to fully implement several GAO-identified best practices or 
follow others. Following best practices is particularly important for 
FAA, which must manage large, complex, and interdependent acquisitions 
associated with NextGen. Cost estimates that are imprecise can result 
in Congress appropriating millions of dollars for projects based on 
estimates that prove to be inaccurate, and program schedule delays can 
increase costs and affect the implementation of interdependent 
programs. In such cases, FAA will be forced to reduce the scope of the 
programs to stay within the original estimates or Congress will need 
to appropriate unanticipated funds to complete the programs. Delays 
and cost increases in individual programs could have a cascading 
effect on other programs and ultimately affect FAA's timelines and 
goals for NextGen implementation. 

Our analysis of the cost estimates and schedules for the four programs 
we reviewed indicates that FAA needs to further develop requirements 
for critical cost estimation and schedule procedures. Independent cost 
estimates can improve the accuracy and credibility of cost estimates 
and better ensure that programs will be completed within budget. A 
schedule risk analysis can help FAA determine the likelihood that a 
program will be completed on time. FAA stated that it has no immediate 
plans to conduct independent cost estimates due to current budgetary 
constraints. We recognize that conducting independent cost estimates 
and schedule risk analysis takes both financial resources and some 
time and that it may be appropriate to limit one or both of these 
analyses to instances where a program is particularly costly, complex, 
or on a compressed schedule. However, conducting independent cost 
estimates, schedule risk analyses, and other analyses called for in 
our best practices can not only help minimize the risk of cost 
overruns and schedule delays, but also provide FAA, congressional 
decision makers, and other stakeholders with important information 
about these critical acquisitions. 

It is also important that FAA develop master schedules at the 
individual acquisition program level. FAA's lack of a fully integrated 
master schedule for the programs we reviewed hampers its ability to 
provide accurate information on the schedule for these programs. This 
information will be needed as FAA simultaneously works to develop an 
integrated master schedule for the overall NextGen initiative. The use 
of an integrated master schedule can assist FAA in monitoring a 
program, identifying problems that could affect later stages of the 
program's implementation, improving the accuracy of cost estimates and 
schedules for individual programs, and improving the accuracy of 
information FAA is compiling to monitor the costs and schedules for 
the NextGen initiative. 

FAA has incorporated 11 of our 12 steps that are associated with the 
characteristics of a high-quality and reliable cost estimate into 
their acquisition guidelines. However, our analysis of the four major 
programs indicates that FAA has not adequately integrated all of the 
steps for these programs into its cost estimation processes, and thus 
the estimates are not reliable. Similarly, although FAA addresses our 
nine scheduling best practices in its acquisition guidelines, our 
analysis of the schedules for the four programs indicates that the 
schedules are not adequately following these best practices and are 
not reliable. Although the cost estimates and schedules for some of 
the four programs were developed prior to FAA's revision of 
acquisition guidelines, our work shows that FAA needs to assess its 
major acquisition programs to understand if its guidelines and other 
best practices are, in fact, being followed. Such an assessment would 
then allow FAA to better ensure that best practices for cost estimates 
and schedules are being applied. 

Recommendations for Executive Action: 

To improve cost estimates and schedules for NextGen and other major 
air traffic control acquisition programs, GAO recommends that the 
Secretary of Transportation direct FAA to take the following three 
actions when appropriate for major acquisition programs based on a 
program's cost, schedule, complexity, and risk: 

* Conduct independent cost estimates and schedule risk analysis for 
major acquisition programs. 

* Require a fully integrated master schedule for each major 
acquisition program, including those that are components of NextGen. 
An integrated master schedule should horizontally and vertically link 
all program activities and milestones, including government and 
contractor schedules and program segments. 

* Conduct an assessment of major acquisition programs to ensure they 
meet all of the established best practices for cost estimates and 
schedules contained in GAO guidance. 

Given constrained budgets, FAA should determine which programs should 
be subject to these recommendations, such as those that are 
particularly costly, complex, or on a compressed schedule. 

Agency Comments: 

We provided a draft of this report to the Department of Transportation 
for review and comment. DOT and FAA responded by email and did not 
comment whether or not they agreed or disagreed with our 
recommendations. DOT provided comments on the results of our analysis 
of the cost estimates and schedules for the four programs we reviewed 
in depth. In response to our finding that the ADS-B, CATMT, SWIM, and 
WAAS estimates lacked credibility largely because FAA did not obtain 
an independent cost estimate for any of the cost estimates, and 
provided little evidence that they conducted sensitivity or risk 
analyses, FAA stated it is not convinced that an independent 
organization will reduce the uncertainty of cost estimates. FAA noted 
that it does not have an independent organization such as the 
Department of the Navy's Center for Cost Analysis. However, FAA stated 
that the Finance Organization within ATO assessed the ADS-B program 
office's Basis of Estimate as part of the JRC Decision and that this 
level of independence, combined with specific entry and exit criteria, 
allowed the program offices to manage these acquisitions so that costs 
were controlled, risks mitigated, and technical parameters achieved, 
while adhering to the planned milestone schedule. We agree that the 
Finance Organization assessment of the two cost estimates provided 
some degree of independence and may have improved the accuracy of the 
ADS-B estimates, but it is not clear that such an independent review 
will guarantee similar results for other programs. As we stated in the 
report, such an independent cost review is less rigorous than an 
independent cost estimate. According to our cost guide, an independent 
cost estimate is often more accurate because the estimating team is 
further removed from the program office and less prone to accept 
overly optimistic assumptions or be burdened by organizational bias. 
DOT also provided technical clarifications, which we incorporated into 
the report as appropriate. 

We are sending copies of this report to interested congressional 
committees, the Secretary of Transportation, and the Acting 
Administrator of FAA. In addition, the report is available at no 
charge on the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-2834 or dillinghamg@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. GAO staff who made key 
contributions to this report are listed in appendix V. 

Signed by: 

Gerald L. Dillingham, Ph.D. 
Director, Physical Infrastructure Issues: 

List of Committees: 

The Honorable Jay Rockefeller:
Chairman:
The Honorable Kay Bailey Hutchison:
Ranking Member:
Committee on Commerce, Science, and Transportation:
United States Senate: 

The Honorable Ralph Hall:
Chairman:
The Honorable Eddie Bernice Johnson:
Ranking Member:
Committee on Science, Space, and Technology:
House of Representatives: 

The Honorable Thomas Petri:
Chairman:
The Honorable Jerry Costello:
Ranking Member:
Subcommittee on Aviation:
Committee on Transportation and Infrastructure:
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope and Methodology: 

In response to a congressional request, we examined the Federal 
Aviation Administration's (FAA) ability to modernize, upgrade, and 
replace the National Airspace System's (NAS) facilities and equipment 
to meet projected increases in traffic volumes, enhance the system's 
safety, and increase the efficiency of the air traffic control (ATC) 
system--a principal component of the NAS. FAA's ATC acquisitions are 
critical to maintaining the NAS and transitioning to the Next 
Generation Air Transportation System (NextGen) over the next 10 years. 
Given that some key legacy and NextGen acquisitions have experienced 
schedule delays and cost overruns, which may risk the timely 
implementation of NextGen, we (1) determined whether the planned costs 
and schedules of current FAA ATC acquisition programs have changed 
since they were first submitted to Congress; (2) examined the reasons 
for any changes in planned costs and schedules; and (3) assessed the 
extent to which select ATC programs adhered to best practices for 
determining acquisition costs and schedules. 

To describe any changes in costs and schedules of the current 30 FAA 
capital ATC acquisitions, we gathered and analyzed agency data on the 
estimated cost and schedules of these ATC acquisitions.[Footnote 47] 
We drew upon past work in which we undertook detailed reviews of the 
status of ATC and other acquisition programs[Footnote 48] and obtained 
updated documentation as necessary from FAA. We interviewed FAA 
officials to obtain information on FAA's acquisition process and 
summarized the status of all acquisitions, including FAA's original 
and current cost estimates and completion dates. For baselined 
acquisitions,[Footnote 49] we compared estimated costs when they were 
submitted to Congress for approval against their current estimates, 
and we analyzed planned and actual schedules. 

To determine the reasons for changes in cost estimates and schedules, 
we interviewed FAA officials and FAA contractors and reviewed 
acquisition documentation. We analyzed information on cost increases 
and delays to determine if systematic issues exist that have effects 
on other FAA acquisitions. 

To determine the extent to which select ATC programs adhered to best 
practices for determining acquisition costs and schedules, we 
conducted an in-depth review of 4 of the 30 acquisitions programs: The 
Automatic Dependent Surveillance-Broadcast (ADS-B) system, the 
Collaborative Air Traffic Management Technologies (CATMT) system, the 
System Wide Information Management (SWIM) system, and the Wide Area 
Augmentation System (WAAS). We selected these four acquisitions based 
on the following criteria: (1) existence of baselining, (2) the 
acquisition is at a point in the acquisition process where risks can 
be identified, and (3) the acquisition is key to NextGen and legacy 
systems. In addition to interviews, we collected documentation, and we 
analyzed and summarized the views and information collected. We also 
identified best practices that FAA could adopt or strengthen to 
improve its acquisitions cost estimation and scheduling and ensured 
that acquisitions follow cost and schedule best practices outlined in 
our Cost Estimating and Assessment Guide.[Footnote 50] We also 
performed a schedule risk analysis of the WAAS program to determine 
the likelihood of the project finishing on schedule. 

We used our Cost Estimating and Assessment Guide[Footnote 51] (Cost 
Guide) as a source of criteria for analyzing cost estimates. As noted 
earlier in our report, our Cost Guide identifies 12 steps consistently 
applied by cost-estimating organizations throughout the federal 
government and industry and considered best practices for developing 
cost estimates. For the purposes of this review, we grouped these 
steps into four characteristics of high-quality and reliable 
estimates--well-documented, comprehensive, accurate, and credible--
which can be summarized as follows: 

Well-documented: The documentation should address the purpose of the 
estimate, the project background and system description, its schedule, 
the scope of the estimate (in terms of time and what is and is not 
included), the ground rules and assumptions, all data sources, the 
estimating methodology and rationale, the results of the risk 
analysis, and a conclusion about whether the cost estimate is 
reasonable. Therefore, a good cost estimate--while taking the form of 
a single number--is supported by detailed documentation that describes 
how it was derived and how the expected funding will be spent in order 
to achieve a given objective. For example, the documentation should 
capture in writing such things as the source data used and their 
significance, the calculations performed and their results, and the 
rationale for choosing a particular estimating method or reference. 
Moreover, this information should be captured in such a way that the 
data used to derive the estimate can be traced back to and verified 
against their sources. Finally, the cost estimate should be reviewed 
and accepted by management to ensure there is a high level of 
confidence in the estimate and the estimating process. 

Comprehensive: The cost estimates should include both government and 
contractor costs of the project over its full life cycle, from 
inception through design, development, deployment, operation, and 
maintenance to retirement of the project. The cost estimate should be 
structured in sufficient detail to ensure that cost elements are 
neither omitted nor double counted, and they should document all cost-
influencing ground rules and assumptions. 

Accurate: The cost estimates should provide for results that are 
unbiased, and they should not be overly conservative or optimistic. 
Estimates are accurate when they are based on an assessment of most 
likely costs, adjusted properly for inflation, and contain few, if 
any, minor mistakes. In addition, the estimates should be updated 
regularly to reflect material changes in the project, such as when 
schedules or other assumptions change so that the estimate is always 
reflecting the project's current status. Among other things, the 
estimate should be grounded in documented assumptions and a historical 
record of cost estimating and actual experiences on other comparable 
projects. 

Credible: The cost estimates should discuss any limitations of the 
analysis because of uncertainty or biases surrounding data or 
assumptions. Major assumptions should be varied, and other outcomes 
recomputed to determine how sensitive they are to changes in the 
assumptions. Risk and uncertainty analysis should be performed to 
determine the level of risk associated with the estimate. Furthermore, 
the estimate's results should be cross-checked, and an independent 
cost estimate conducted by a group outside the acquiring organization 
should be developed to determine whether other estimating methods 
produce similar results. 

After reviewing documentation submitted by FAA and information 
obtained during interviews, we determined the extent to which the cost 
estimates met the characteristics of cost-estimating best practices 
for the four projects we reviewed. 

Our review of project schedules was based on research that identified 
a range of best practices associated with effective schedule 
estimating. In addition, we obtained the consulting services of David 
Hulett, Ph.D.,[Footnote 52] to assist in our risk analysis of the WAAS 
project schedule.[Footnote 53] We also conducted multiple interviews 
with project managers, contractors, and schedulers to determine the 
extent to which current project schedules met the best practices 
criteria. These nine practices are: 

Capturing all activities: The schedule should reflect all activities 
(steps, events, outcomes, and other factors) as defined in the 
project's work breakdown structure, including activities to be 
performed by both the government and its contractors. 

Sequencing all activities: The schedule should be planned so that it 
can meet project-critical dates. To meet this objective, activities 
need to be logically sequenced in the order that they are to be 
carried out. In particular, activities that must finish prior to the 
start of other activities (i.e., predecessor activities) and 
activities that cannot begin until other activities are completed 
(i.e., successor activities) should be identified. Identifying 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be used as a basis for 
guiding work and measuring progress. 

Assigning resources to all activities: The schedule should 
realistically reflect what resources (i.e., labor, material, and 
overhead) are needed to do the work, whether all required resources 
will be available when they are needed, and whether any funding or 
time constraints exist. 

Establishing the duration of all activities: The schedule should 
reflect how long each activity will take to execute. In determining 
the duration of each activity, the same rationale, data, and 
assumptions used for cost estimating should be used for preparing the 
schedule. Furthermore, these durations should be as short as possible 
and should have specific start and end dates. Excessively long periods 
needed to execute an activity should prompt further decomposition of 
the activity so that shorter execution durations will result. 

Integrating schedule activities horizontally and vertically: The 
schedule should be horizontally integrated, meaning that it should 
link the products and outcomes associated with already sequenced 
activities. These links are commonly referred to as "hand-offs" and 
serve to verify that activities are arranged in the right order to 
achieve aggregated products or outcomes. The schedule should also be 
vertically integrated, meaning that traceability exists among varying 
levels of activities and supporting tasks and subtasks. Such mapping 
or alignment among levels can enable different groups to work to the 
same master schedule. 

Establishing the critical path for all activities: With the use of 
scheduling software, the critical path--the longest-duration path 
through the sequenced list of activities--should be identified. The 
establishment of a project's critical path is necessary for examining 
the effects of delays in any activity along this path. Potential 
problems that may occur on or near the critical path should also be 
identified and reflected in the scheduling of the time for high-risk 
activities (see the next activity, "Identifying float"). 

Identifying reasonable float: The schedule should identify float--the 
time that a predecessor activity can slip before the delay affects 
successor activities--so that schedule flexibility can be determined. 
As a general rule, activities along the critical path typically have 
the least amount of float. 

Conducting a schedule risk analysis: A schedule risk analysis uses a 
good critical path method schedule and data about project schedule 
risks, as well as Monte Carlo simulation techniques, to predict the 
level of confidence in meeting a project's completion date, the amount 
of time contingency needed for a level of confidence, and the 
identification of high-priority risks. This analysis should focus not 
only on critical path activities but also on other schedule paths that 
may become critical. A schedule/cost risk assessment recognizes the 
interrelationship between schedule and cost and captures the risk that 
schedule durations and cost estimates may vary for a variety of 
reasons, including limited data, optimistic estimating, technical 
challenges, lack of qualified personnel, and other external factors. 
As a result, the baseline schedule should include a buffer or a 
reserve of extra time. A reserve of extra time for contingencies 
should be calculated by performing a schedule risk analysis. As a 
general rule, the reserve should be held by the project manager and 
applied as needed to those activities that take longer than scheduled 
because of the identified risks. Reserves of time should not be 
apportioned in advance to any specific activity since the risks that 
will actually occur and the magnitude of their impact are not known in 
advance. 

Updating the schedule using logic and durations to determine the 
dates: The schedule should use logic and durations in order to reflect 
realistic start and completion dates for project activities. The 
schedule should be continually monitored to determine when forecasted 
completion dates differ from the planned dates. This information can 
be used to determine whether schedule variances will affect downstream 
work. Maintaining the integrity of the schedule logic is not only 
necessary to reflect the project's true status but is also required 
before conducting a schedule risk analysis. The schedule should avoid 
logic overrides and artificial constraint dates that are chosen to 
create a certain result on paper. Individuals trained in critical path 
method scheduling should be responsible for updating the schedule. 

Based on our work, we determined the extent to which estimates and 
schedules for the four projects we selected met each best practices 
criterion: 

* Not Met--project officials provided no evidence that satisfies any 
portion of the criterion. 

* Minimally Met--project officials provided evidence that satisfies a 
small portion of the criterion. 

* Partially Met--project officials provided evidence that satisfies 
about half of the criterion. 

* Substantially Met--project officials provided evidence that 
satisfies a large portion of the criterion. 

* Met--project officials provided evidence that satisfies the entire 
criterion. 

We conducted this performance audit from August 2010 to February 2012 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Detailed Purpose and Status for 30 Acquisitions GAO 
Reviewed: 

* This appendix contains detailed information for 30 individual air 
traffic control programs. Each overview presents information and data 
that was provided by FAA. The overviews provide a description of the 
program and the cost and schedule status. The overviews are based on 
program office reported information as of August 2011. In most cases, 
we did not validate the data provided, but reviewed the data and 
performed various checks to determine they were reliable enough for 
our purposes. 

Figure 3: Automatic Dependent Surveillance-Broadcast (ADS-B): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts ADS-B environment] 

Purpose and status: 
The ADS-B system is based primarily on providing three fundamental 
broadcast services to support ADS-B-enabled applications: (1) ADS-B, 
which provides highly accurate aircraft-derived ADS-B reports that 
contain identification, state vector, and status/intent information 
about the aircraft; (2) Traffic Information Services, which provides 
ADS-Bequipped aircraft with surveillance data about non-ADS-B-equipped 
aircraft; and (3) Flight Information Services, which provides ground-
to-air broadcast of noncontrol, advisory information that provides 
users with near-real-time information to operate safely and 
efficiently. 

Contractor: ITT Corporation. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of ADS-B service volumes: 306; 
Start date: Aug. 2007; 
Completion date: Sept. 2014; 
Estimated cost: $1,681.50.   

Current target: 
Number of ADS-B service volumes: 306; 
Start date: Aug. 2007; 
Completion date: Sept. 2014; 
Estimated cost: $1,726.20.    

As of August 2011, this program is on target in terms of schedule, but 
it is about $45 million, or about 2.66 percent, over its original 
budget. According to FAA, the current revised budget includes 
congressional earmarks of $9.3 million in fiscal year 2008 and $6.8 
million in fiscal year 2009. Also included in the current revised 
budget is an additional $15 million for ADS-B related modifications to 
Terminal Software. The earmarks are not included in the baseline 
tracking and reporting. In March 2011, the Investment Decision 
Authority (IDA) approved the baseline schedule replan and strategic 
decision to incorporate Colorado Wide Area Multilateration (WAM) Phase 
II (previously baselined in December 2009) as part of the ADS-B 
baseline. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 4: Advanced Technologies and Oceanic Procedures (ATOP): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Air traffic controllers] 

Purpose and status: 
The ATOP program has replaced existing oceanic ATC systems and 
procedures with a single integrated system and modernizes facilities 
responsible for managing more than 24 million square miles of airspace 
over the Atlantic and Pacific Oceans. ATOP integrates flight and radar 
data processing, detects conflicts between aircraft, provides data 
link and surveillance capabilities, and automates previously manual 
processes. The ATOP program is in the Pre-Planned Product Improvements 
phase, deploying two software releases per year that deliver software 
efficiency and safety enhancements to meet agency commitments to the 
three oceanic en route air traffic control centers. 

Contractors: Lockheed Martin Information Systems and Global Solutions 
(IS&GS) - Civil. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ATOP Tech Refresh: 4; 
Start date: N/A; 
Completion date: Dec. 2014; 
Estimated cost: $548.20.  

Current target: 
Number of facilities receiving ATOP Tech Refresh: 4; 
Start date: N/A; 
Completion date: Dec. 2014; 
Estimated cost: $524.15.  

As of August 2011, this program is below its budget. According to FAA, 
the acquisition phase of this program was completed in March 2006. 
ATOP is currently in the Tech Refresh phase, with funding through 
fiscal year 2014. There were no acquisition planning baseline (APB) 
milestones for the Tech Refresh phase approved during the May 2001 
final investment decision. However, milestones that contribute to 
FAA's annual business plan goals are developed each fiscal year.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 5: Airport Surveillance Radar (ASR) Model 11 Tech Refresh 
Segment 1: 

[Refer to PDF for image: illustration and associated data] 

[Figure: ASR-11 equipment] 

Purpose and status: 
The ASR-11 program provides for the replacement and upgrade of out-of-
date ASR-11 commercial off-the-shelf hardware and software to ensure 
the continued operation of the radar system through its designated 
lifecycle. The Low Overhead Array Processors, which are used in the 
signal processor cabinet, are 1980s technology and are no longer in 
production. Current processor and memory utilization of some of these 
processor cards run at 80 percent to 90 percent. There is no 
possibility for expansion using these cards, and adding additional 
processor cards to distribute the processing would require major 
software modification and recoding. The Advanced Signal Data Processor 
(ASDP) has been developed to replace the existing signal data 
processor. ASDP was implemented in the production systems available in 
fiscal year 2009 and beyond from the vendor. ASDP modification kits 
are being acquired to retrofit the signal data processor cabinet for 
all the systems procured by FAA. 

Contractors: Raytheon, ITT, MCR, Regulus, and SAIC.
   
Baseline changes over time: 
    
Schedule and cost targets (dollars in millions): 
   
Baseline target: 
Number of facilities receiving ASR-11 Pre-planned Product Improvement 
(P3I) phase: 68;  
Start date: May 2008; 
Completion date: June 2015; 
Estimated cost: $20.90.   

Current target: 
Number of facilities receiving ASR-11 Pre-planned Product Improvement 
(P3I) phase: 68;  
Start date: May 2008; 
Completion date: June 2015; 
Estimated cost: $20.90.   

As of August 2011, this program is within its original budget and 
schedule. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 6: Aviation Surface Weather Observation Network (ASWON) and 
Automated Surface Observing System (ASOS) P31: 

[Refer to PDF for image: illustration and associated data] 

[Figure: ASWON equipment] 

Purpose and status: 
ASWON is an umbrella program that consists of multiple weather sensor 
system programs. The only ASWON program currently receiving facilities 
and equipment (F&E) funding is ASOS Pre-Planned Product Improvements 
(P31). All other ASWON systems are in service. The ASOS P31 program 
consists of five upgrades/enhancements to ASOS. Three upgrades are 
complete (Processor Upgrade, Dewpoint Sensor Replacement, and Ice-Free 
Wind Sensor), and one is active (Ceilometer Replacement). The ASOS P31 
program will upgrade or sustain the performance of 571 ASOS systems 
with the Ceilometer Replacement. The Ceilometer Replacement will 
replace an obsolete sensor to measure the height and amount of cloud 
coverage. 

Contractor: NWS/Vaisala Inc. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ASWON ASOS P3I: 571; 
Start date: Aug. 2001; 
Completion date: Sept. 2009; 
Estimated cost: $54.80. 

Current target: 
Number of facilities receiving ASWON ASOS P3I: 571; 
Start date: Aug. 2001; 
Completion date: Sept. 2012; 
Estimated cost: $55.90. 

As of August 2011, this program was 36 months behind its original 
schedule and about $1.1 million, or about 2 percent, over its original 
budget. The current cost and schedule estimates have been revised to 
reflect the removal of the EPI sensor from the ASWON P31 baseline   
because in May 2008, FAA senior management decided to remove the 
replacement of the EPI sensor from the program performance baseline. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 7: Air Traffic Control Radar Beacon Interrogator (ATCBI-6): 

[Refer to PDF for image: illustration and associated data] 

[Figure: ATCBI-6 equipment] 

Purpose and status: 
The ATCBI-6 Replacement program replaces existing en route ATCBI-4/5 
equipment and establishes new beacon-only sites. The ATCBI-6 is a 
secondary radar used for en route and oceanic air traffic control. The 
ATCBI-6 provides aircraft position information and identification to 
ATC facilities for separation assurance and traffic management. 

Contractor: Raytheon. 

Baseline changes over time: 
   
Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ATCBI-6 Replacement: 127; 
Start date: Aug. 1997; 
Completion date: Sept. 2004; 
Estimated cost: $282.90.    

Current target: 
Number of facilities receiving ATCBI-6 Replacement: 139; 
Start date: Aug. 1997; 
Completion date: Jan. 2012; 
Estimated cost: $255.10.    

As of August 2011, this program was $27.80 million, or 10 percent, 
under its original budget and 88 months behind its original schedule. 
There is a 9 percent increase in the number of facilities receiving 
this program. According to FAA, the budget was revised downward after 
the Joint Resources Council (JRC) rebaselined the program in May 2008 
to reflect lower system procurement and installation costs.   
The program schedule was extended because of budget reductions and 
deferrals, as well as the addition of the new establishment sites. 
According to FAA, during the FAA furlough, key personnel were 
unavailable to complete the ATCBI-6 systems electronic installation; 
as a result, the operational readiness decision for Santa Fe (one of 
the last sites to be deployed) was delayed to January 2012. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 8: Collaborative Air Traffic Management Technologies (CATMT), 
Work Packages 2 and 3 (WP2 and WP3): 

[Refer to PDF for image: data] 

Purpose and status: 
The CATMT program implements enhancements that will improve the 
Traffic Flow Management (TFM) decision support tool suite. CATMT TWP2 
includes Arrival Uncertainty Management (AUM), Collaborative Airspace 
Constraint Resolution (CACR), and Airborne Reroute Execution (ABRR). 
CATMT WP3 includes enhancements that continue to provide decision 
support capabilities that leverage the latest technology and research, 
and enable more efficient communication and collaboration with 
aircraft operators. WP3 modernizes the TFM remote sites. The Traffic 
Flow Management System (TFMS) is the automation backbone for the Air 
Traffic Control System Command Center (ATCSCC) and the nationwide 
Traffic Management Units that assist the ATCSCC in strategic planning 
and management of air traffic. CATMT WP2 development started at the 
beginning of fiscal year 2010 and will continue into fiscal year 2014. 
CATMT WP3 development began in fiscal year 2011 and will continue 
through fiscal year 2015. 

Contractor: Computer Science Corporation (CSC).  

Baseline changes over time: 
  
Schedule and cost targets (dollars in millions): 

CATMT WP2: 

Baseline target: 
Number of facilities receiving ATCBI-6 CATMT: 81; 
Start date: Sept. 2008; 
Completion date: Sept. 2014; 
Estimated cost: $109.50.  

Current target: 
Number of facilities receiving ATCBI-6 CATMT: 81; 
Start date: Sept. 2008; 
Completion date: Sept. 2014; 
Estimated cost: $109.50.  

CATMT WP3: 

Baseline target: 
Number of facilities receiving ATCBI-6 CATMT: 81; 
Start date: Jan. 2010; 
Completion date: Dec. 2015; 
Estimated cost: $53.01.  

Current target: 
Number of facilities receiving ATCBI-6 CATMT: 81; 
Start date: Jan. 2010; 
Completion date: Dec. 2015; 
Estimated cost: $53.01.  

As of August 2011, this program was within its original schedule, but 
about $600,000, or about 1 percent, over its original budget. 

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 9: En Route Communication Gateway (ECG) Tech Refresh: 

[Refer to PDF for image: illustration and associated data] 

[Figure: ECG equipment] 

Purpose and status: 
ECG replaced the Peripheral Adapter Module Replacement Item (PAMRI) 
functionality and provides the communications interface for 
transmittal of surveillance data and other legacy systems data for 
processing with the Host Computer System. ECG supports future systems 
and applications such as the En Route Automation Modernization (ERAM) 
and has the capability to accommodate future messaging formats and 
protocols. The ECG system consists of commercial-off-the-shelf 
hardware with minimal nondevelopmental item software. Tech Refresh 
will be used to sustain the capability of the ECG system and to ensure 
that new capabilities or functionality can be incorporated. The ECG 
was a prerequisite to deploying ERAM software and hardware. 

Contractor: EnRoute Computer Solutions. 

Baseline changes over time:  

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ECG: 23; 
Start date: N/A;
Completion date: N/A; 
Estimated cost: $83.50.   

Current target: 
Number of facilities receiving ECG: 23; 
Start date: N/A; 
Completion date: N/A; 
Estimated cost: $40.50.   

As of August 2011, this program was about $43 million, or about 51.5 
percent, under budget. According to FAA, the ECG baseline was approved 
for the Tech Refresh phase in March 2002 and included both the 
acquisition phase and the Tech Refresh phase. There were no APB 
milestones approved at the March 2001 final investment decision. 
Instead, for the Tech Refresh phase, milestones are developed each 
fiscal year that contribute to that fiscal year's business plan goals. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 10: En Route Automation Modernization (ERAM): 

[Refer to PDF for image: illustration and associated data] 

[Figure: ERAM equipment] 

Purpose and status: 
ERAM will replace the current Host Computer software and hardware, 
direct access radar channel software and hardware and other associated 
interfaces, communications, and support infrastructure. ERAM is the 
key to FAA's ability to implement new services, concepts, and traffic 
flows to users. ERAM is being developed and deployed incrementally. 
The first segment was completed in 2006 and replaced the Host Back-up 
and added safety alerts through the Enhanced Back-up Surveillance 
(EBUS) effort. The second segment was completed in 2007 and 
established the En Route Information Display System (ERIDS) 
nationwide. The third segment will replace the Host Computer System 
with new software and hardware. The fourth segment is a software 
release intended to support ERAM's interface with other systems such 
as ADS-B and SWIM. 

Contractor: Lockheed Martin. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ERAM: 26 
Start date: June 2003 
Completion date: Dec. 2010 
Estimated cost: $2,154.60    

Current target: 
Number of facilities receiving ERAM: 26 
Start date: June 2003 
Completion date: Aug. 2014 
Estimated cost: $2,484.60    

As of August 2011, this program was 44 months behind its original 
schedule and about $330 million, or about 15.23 percent, over its 
original budget. According to FAA, the schedule variance and cost 
increase are associated with the following factors: (1) the project 
plan did not factor in risks associated with operational complexities 
at the selected sites, (2) an insufficient testing environment failed 
to identify software issues before deployment to key sites, (3) 
communication between the program office and field sites was 
insufficient, and (4) stakeholder engagement was uneven during 
development and deployment. The Joint Resources Council approved 
changes to the program's budget and schedule baselines in June 2011.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 11: Integrated Display Systems (IDS) Replacement: 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts IDS replacement system] 

Purpose and status: 
IDS is a local and wide-area network information collection, 
dissemination, and display system. It consolidates information from 
multiple operational NAS weather subsystems and other operational 
sources onto a single display platform and distributes the data to air 
traffic controllers and airspace managers at various ATC facilities. 
According to FAA, the existing system is unsupportable because of 
obsolescence issues with both the hardware components and the 
proprietary software, and it lacks the capacity to incorporate 
software updates. The replacement system will provide greater system 
availability to users, as well as enable information to be 
consolidated to enhance air traffic controllers' situational awareness. 

Contractor: All Weather Inc. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving IDS Replacement: 390; 
Start date: Sept. 2008; 
Completion date: Oct. 2015; 
Estimated cost: $50.80.      

Current target: 
Number of facilities receiving IDS Replacement: 390; 
Start date: Sept. 2008; 
Completion date: Dec. 2015; 
Estimated cost: $50.80.      

As of August 2011, this program was within its original budget and 2 
months behind its original schedule. According to FAA, acquisition 
activities were delayed because of the fiscal year 2009 continuing 
resolution's restriction on project new starts." Completion of two 
phases—Developmental Test and Evaluation and Operational Test and 
Evaluation—were delayed because test plans and procedures were 
delivered late. Additionally, the amount of testing required and the 
duration of the tests were underestimated.  

Sources' GAO (analysis); FAA (data and artwork). 

Figure 12: International Flight Inspection Aircraft (IFIA): 

[Refer to PDF for image: data] 

Purpose and status: 
The IFIA program acquires three Challenger 600 series aircraft 
(jointly funded by FAA and the U.S. Air Force) to upgrade FAA's 
capability to evaluate and certify both ground-based and space-based 
navigational equipment facilities for state and federal agencies, 
including the Department of Defense (DOD). These efforts are critical 
to the security of the United States and its allies. Before and during 
military deployments, FAA must certify the safety of runways, 
navigational aids, landing systems, and support. 

Contractor: Bombardier Aerospace Corporation. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of aircraft: 3 
Start date: Dec. 2003 
Completion date: Aug. 2009 
Estimated cost: $27.40 

Current target: 
Number of aircraft: 3 
Start date: Dec. 2003 
Completion date: May 2012 
Estimated cost: $33.70 

As of August 2011, this program was $6.3 million, or 23 percent, over 
its original budget and 33 months behind its original schedule. These 
schedule and budget variances are associated with acquiring the second 
and third aircraft and occurred because (1) a contract option was 
exercised 2 years later than planned as a result of a delay in 
appropriations, (2) the purchased aircraft model was costlier and 
configured differently from the planned model because the original 
model had been phased out, and (3) the trade-in value for the aircraft 
being replaced was much lower than planned. The IFIA program will not 
be rebaselined but will continue to its completion with a cost and 
schedule variance to the original baseline, according to a decision by 
the Investment Decision Authority in January 2011. 

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 13: Instrument Flight Procedure Automation (IFPA): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts aircraft using IFPA procedures] 

Purpose and status: 
The IFPA program will replace, modernize, and update IFPA systems in 
support of both visual and instrument flight procedure development, 
such as approaches, standard terminal automation replacement system, 
airways, and departures. IFPA is a suite of next generation 
information technology (IT) tools. These tools create products using 
fully integrated solutions for visual and instrument flight 
procedures. IFPA consists of the Instrument Procedure Development 
System (IPDS), Instrument Flight Procedures (IFP) database, Airports 
and Navigations Aids database (AirNav), Obstacle Evaluation (OE) 
system, and the Automated Procedures Tracking System (APTS). This 
program is in a Tech Refresh phase. 

Contractors: MDA Systems, Inc., and L-3-Titan. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving IFPA: 1; 
Start date: Sept. 2006; 
Completion date: Sept. 2011; 
Estimated cost: $50.80.    

Current target: 
Number of facilities receiving IFPA: 1; 
Start date: Sept. 2006; 
Completion date: Sept. 2012; 
Estimated cost: $50.80.    

As of August 2011, this program was 12 months behind its original 
schedule but within its original budget. According to FAA, in 
September 2008, the Investment Decision Authority approved the 
incorporation of expanded technical requirements associated with 
changed RNAV Order 8260.54A, and Terminal Instrument Procedures 
(TERPS) Order 8260.3B, and Flight Procedures & Airspace Order 8260.19 
CHG20, as issued by FAA's Flight Standards Service in late 2007. These 
additional requirements for geodetic modeling (enhanced ellipsoidal) 
and criteria affected several components of IFPA. In April 2010, IDA 
approved a 12-month increase in the baseline schedule duration that 
was implicit with the growth in scope.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 14: Integrated Terminal Weather System (ITWS): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts ITWS display] 

Purpose and status: 
ITWS is an air traffic management tool that provides air traffic 
managers with graphics and full-color displays of essential weather 
information at major U.S. airports. ITWS was developed to fill the 
needs of air traffic managers, controllers, and airlines for a tool 
that integrates weather data from a number of sources and provides 
customers with a single, easily used and understood display of weather 
support products. The ITWS program provides for the development, 
installation, testing, training, maintenance, and life-cycle 
operational support of 34 operational ITWSs. These 34 systems will 
provide ITWS service to 75 airports, of which 30 are major airports. 

Contractor: Raytheon. 
    
Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving ITWS: 38; 
Start date: June 1997; 
Completion date: July 2003; 
Estimated cost: $276.10.     

Current target: 
Number of facilities receiving ITWS: 38; 
Start date: June 1997; 
Completion date: Aug. 2010; 
Estimated cost: $282.10.     

When completed, this program was $6 million, or about 2 percent, over 
its original budget and 85 months behind its original schedule. 
According to FAA, the program was completed in August 2010 with the 
commissioning of the last system.  

Sources: GAO (analysis); FAA (data and artwork).  

[End of figure] 

Figure 15: Next Generation Air/Ground Communication System (NEXCOM) 
Segment 1a: 

[Refer to PDF for image: illustration and associated data] 

[Figure: NEXCOM equipment] 

Purpose and status: 
NEXCOM will implement a new air/ground voice communication system 
using the limited available radio frequency spectrum more efficiently. 
It will provide the operational flexibility required for NextGen and 
will be implemented in two segments. NEXCOM Segment 1a, which 
addresses the en route environment, began replacing the radios used 
for high and ultrahigh en route sectors with multimode digital radios 
in 2004. These radios can function in either analog or digital mode. 
Segment 2 was approved for a final investment in September 2011. The 
overall plan is to implement new generation UHF/VHF radios that will 
service high-density terminal areas and flight service operations and 
to upgrade emergency backup radios for use when the primary radios are 
not functioning. 

Contractor: ITT Aerospace Communications. 

Baseline changes over time: 
  
Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving NEXCOM Segment 1a: 1,437; 
Start date: Sept. 1998; 
Completion date: Sept. 2008; 
Estimated cost: $407.60.   

Current target: 
Number of facilities receiving NEXCOM Segment 1a: 1,437; 
Start date: Sept. 1998; 
Completion date: Sept. 2013; 
Estimated cost: $324.70.   

As of August 2011, this program (Segment 1a) is under its original 
budget by about $83 million, or about 20 percent. However, this 
program is 60 months behind its original schedule. The budget decrease 
reflects a May 2000 Joint Resources Council decision to reduce the 
program's scope to the acquisition of multimodal digital radios. In 
addition, the number of facilities scheduled to receive radios has 
decreased 15.2 percent. According to FAA, the schedule was extended 
because of resource issues associated with the radios' installation.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 16: Next Generation Weather Radar (NEXRAD) Dual Polarization: 

[Refer to PDF for image: illustration and associated data] 

[Figure: NEXRAD equipment] 

Purpose and status: 
NEXRAD improvements will replace the existing infrastructure and 
introduce required new capabilities to multiple interdependent weather 
systems. The NEXRAD Product Improvement updates NEXRAD technology, 
providing upgrades to various systems. This program also provides 
improved flash flood warnings, severe thunderstorm warnings, 
biological target identification, and various types of winter storm 
warnings. Aviation applications include new warnings for hail and 
icing conditions, turbulence, and bird strikes. 

Contractors: MIT Lincoln Labs, and L3. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving NEXRAD Dual Polarization: 12; 
Start date: Sept. 2008; 
Completion date: Sept. 2013; 
Estimated cost: $25.66.     

Current target: 
Number of facilities receiving NEXRAD Dual Polarization: 12; 
Start date: Sept. 2008; 
Completion date: Sept. 2013; 
Estimated cost: $25.70.     

As of August 2011, this program was within its original schedule and 
was $40,000 over its original budget.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 17: Power Systems Sustained Support (PS3): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts PS3 process] 

Purpose and status: 
The PS3 program is an infrastructure sustainment and renewal program. 
Electrical power systems are necessary to allow continued operation of 
air traffic control facilities when there is an interruption in power 
from commercial sources. These power systems also protect sensitive 
electronic equipment from commercial power surges and fluctuations. 
Projects are prioritized using NAS metrics of capacity, demand, 
passenger value of time, and other specific expert information. 

Contractor: none listed by FAA. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving PS3: N/A[A]; 
Start date: Mar. 2008; 
Completion date: Sept. 2018; 
Estimated cost: $1,351.80.     

Current target: 
Number of facilities receiving PS3: N/A[A]; 
Start date: Mar. 2008; 
Completion date: Sept. 2018; 
Estimated cost: $969.42.     

[A] Not applicable. 

As of August 2011, this program was about $332 million, or about 25 
percent, under its original budget, and within its original schedule. 
According to FAA, major affordability issues and resource constraints 
in the field could affect the deployment of the power systems. There 
are multiple program assets throughout NAS, and requirements change 
annually based on affordability. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 18: Regulation and Certification Infrastructure for System 
Safety (RCISS) Segment 2: 

[Refer to PDF for image: data] 

Purpose and status: 
The RCISS program provides the automation hardware, software, and 
communication infrastructure to support Aviation Safety information 
databases and access to them by the increasingly mobile FAA safety 
work force. RCISS is the next generation infrastructure, which will 
build upon the legacy infrastructure to improve fact-based decision 
making. RCISS's enterprise infrastructure plans provide the access 
methods to all Aviation Safety (AVS) national safety applications 
developed by Safety Approach for Safety Oversight, Aviation Safety 
Knowledge Management Environment, and all other national safety 
programs developed or currently deployed within AVS. The start date 
for RCISS's Segment 1 was July 2007.  

Contractors: Dell, eFAST, Harris, and SAVES. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving RICSS Segment 2: N/A[A]; 
Start date: Oct. 2010; 
Completion date: Sept. 2016; 
Estimated cost: $90.70. 

Current target: 
Number of facilities receiving RICSS Segment 2: N/A[A];  
Start date: Oct. 2010; 
Completion date: Sept. 2016; 
Estimated cost: $90.70. 

[A] Not applicable. 

As of August 2011, this program was within its original budget and 
schedule.  

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 19: Runway Status Lights (RWSL): 

[Refer to PDF for image: illustration and associated data] 

[Figure: RWSL equipment] 

Purpose and status: 
The RWSL system integrates airport lighting equipment with approach 
and surface surveillance systems to provide a visual signal to pilots 
indicating that it is unsafe to enter, cross, or begin takeoff on a 
runway. Airport surveillance sensor inputs are processed through 
safety logic that commands in-pavement lights to illuminate red when 
there is traffic on or approaching the runway. Runway Entrance Lights 
provide signals to aircraft crossing runways from intersecting 
taxiways. Takeoff Hold Lights provides signals to aircraft in position 
for takeoff. 

Contractors: SAAB and Sensis Corporation. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving RWSL: 26; 
Start date: Jan. 2010; 
Completion date: Oct. 2015; 
Estimated cost: $327.40.    

Current target: 
Number of facilities receiving RWSL: 26; 
Start date: Jan. 2010; 
Completion date: June 2016; 
Estimated cost: $352.40. 

As of August 2011, the program was about $25 million, or about 8 
percent, over budget and 8 months, or about 9 percent, behind its 
original schedule. According to FAA, the cost growth is associated 
with changes in construction methods, revised airport requirements, 
and requests for additional light arrays; the 8-month increase in 
duration is associated with delays in the start of construction at 
four sites (Denver, Atlanta, Boston, and San Diego). These delays 
reflect the deferral of $20 million in funding beyond fiscal year 
2013. According to FAA, this deferral resulted from the restructuring 
of FAA's Capital Investment Plan to achieve the lower funding targets 
set by the Office of Management and Budget for fiscal years 2012 
through 2016. In addition, FAA is assessing the effects of 
construction delays resulting from the FAA furlough on the program's 
cost and schedule. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 20: Standard Terminal Automation Replacement System (STARS) 
(TAMR Phase 1): 

[Refer to PDF for image: illustration and associated data] 

[Figure: STARS equipment] 

Purpose and status: 
STARS is a joint DOD and FAA program to modernize terminal ATC 
automation systems. STARS is a digital processing and display system 
that replaced the aging ATC equipment at Automated Radar Terminal 
System II IA and other high-activity Terminal Radar Approach Control 
(TRACON) facilities and airport traffic control towers. The STARS 
program's acquisition phase was completed with deployment to the last 
site in June 2007. The STARS program is now in its Technology Refresh 
and Software Enhancement phase. Software enhancements consist of two 
software builds each year, implementing system performance, 
efficiency, safety, and security modifications to the software 
baseline. Technology Refresh consists of upgrades of selected hardware 
components, including processors, operating systems, monitors, system 
architecture, local area network, and continuous data recording.
The STARS program is the first phase of the Terminal Automation 
Modernization and Replacement (TAMR) Program. 

Contractor: Raytheon (Prime). 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving STARS: 172; 
Start date: Feb. 1996; 
Completion date: Oct. 2005; 
Estimated cost: $940.20. 

Current target: 
Number of facilities receiving STARS: 47; 
Start date: Feb. 1996; 
Completion date: June 2007; 
Estimated cost: $2,719.20.   

Completed in June 2007, this program was about $1,779 million, or 
about 189 percent, over its original budget and about 20 months behind 
its original schedule. Moreover, the number of facilities receiving 
STARS decreased by about 73 percent. According to FAA, this program, 
which was rebaselined in 2004, was completed in June 2007, 6 months 
ahead of the rebaselined completion date. STARS is now in the 
Technology Refresh phase, with funding planned through fiscal year 
2031. The current baseline includes funding for Tech Refresh; however, 
the original baseline did not. There were no APB milestones for the 
Tech Refresh phase approved at the May 2004 final investment decision. 
Milestones are developed each fiscal year for efforts that contribute 
to that fiscal year's business plan goals. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 21: System Wide Information Management (SWIM) Segment 1: 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts SWIM system] 

Purpose and status: 
The SWIM program is intended to facilitate an open, flexible, modular, 
manageable, and secure information management and sharing architecture 
for NAS operational data and other data exchanged among service 
consumers and providers through SWIM information technology 
infrastructure. SWIM is intended to transform NAS application 
interfaces from a tightly coupled point-to-point model into a service-
oriented architecture. In Segment 2, Service-Oriented Architecture 
Core Services will be developed, deployed, and maintained by SWIM with 
assets under the control of the SWIM program office. 

Contractors: Lockheed Martin and Computer Sciences. 
             
Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving SWIM Segment 1: N/A[A];  
Start date: July 2009; 
Completion date: Sept. 2015; 
Estimated cost: $310.20.                

Current target: 
Number of facilities receiving SWIM Segment 1: N/A[A];  
Start date: July 2009; 
Completion date: Sept. 2015; 
Estimated cost: $310.20.                

[A] Not applicable. 

As of August 2011, segment 1 was within its original budget and 
schedule. According to FAA, segment 1 is expected to be rebaselined in 
2012. FAA is planning for a baseline decision for SWIM Segment 2 in 
2012. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 22: Terminal Automation Modernization and Replacement (TAMR)--
Phase 2: 

[Refer to PDF for image: data] 

Purpose and status: 
The first phase of the program—-TAMR Phase 1, completed in 2007—-
replaced the automated radar processing and display systems at 47 
TRACON facilities and their associated ATC towers. This program 
implements a phased approach to modernize or replace older radar 
terminals. Phase 2 of the TAMR program modernizes or replaces 
automation systems at nine FAA sites that pose a critical risk to 
service. 

Contractors: Raytheon (Prime) and Lockheed Martin (subcontractor: 
Transportation and Security Solutions). 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving TAMR Phase 2: 9; 
Start date: June 2005; 
Completion date: July 2008; 
Estimated cost: $139.50. 

Current target: 
Number of facilities receiving TAMR Phase 2: 9; 
Start date: June 2005; 
Completion date: July 2008; 
Estimated cost: $139.50. 

This program was on target as of August 2011, as it was within its 
original budget and schedule. According to FAA, the acquisition phase 
of the program was completed in July 2008. This program is now in the 
Tech Refresh phase, with funding authorized through fiscal year 2030. 
In addition, the TAMR Phase 2 baseline was approved in June 2005, 
including both the acquisition phase and the Tech Refresh phase.
No APB milestones were approved at the June 2005 final investment 
decision. Instead, for the Tech Refresh phase, milestones are 
developed each fiscal year for efforts that contribute to that fiscal 
year's business plan goals. In addition, TAMR Phase 3, which aims to 
consolidate automation at some sites and implement NextGen 
capabilities, is planned to receive a final investment decision in 
December 2011. TAMR Phase 2 Tech Refresh costs of $82.9 million are 
included in the original TAMR Phase 2 baseline of $139.5 million. 

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 23: Tower Training Simulator Systems: 

[Refer to PDF for image: illustration and associated data] 

[Figure: Photo courtesy of training simulators] 

Purpose and status: 
The FAA Academy conducts technical training for air traffic 
controllers, airways facilities technicians, aviation safety 
inspectors, and other specialists and is responsible for internal 
training infrastructure. Training on the new systems being installed 
requires updated simulators, training media, and communications 
equipment. The program provides funding to update the simulators, 
training media, and communications equipment and is intended to cut 
training costs and create a well-trained technical workforce. The NAS 
training simulator project acquires and deploys training simulators to 
selected air traffic facilities in the field and at the academy. This 
project focuses on using technology to assist FAA in training newly 
hired controllers during the next 10 years in response to projected 
staffing requirements. 

Contractor: Adacel. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving Tower Training Simulators: 24; 
Start date: Dec. 2007; 
Completion date: Sept. 2009; 
Estimated cost: $34.20.    

Current target: 
Number of facilities receiving Tower Training Simulators: 24; 
Start date: Dec. 2007; 
Completion date: Aug 2010; 
Estimated cost: $36.60.    

As of August 2011, this program was $2.1 million, or about 7 percent, 
over its original budget and 11 months behind its original schedule. 
According to FAA, due to the critical need to rapidly procure and 
deploy the training simulators in the field, site surveys were not 
conducted at the field sites to validate the waterfall schedule prior 
to the December 2007 Final Investment Decision, where the schedule was 
baselined. Further, there was no coordination with the Occupational 
Safety and Health Administration OSHA/Fire Life Safety organization to 
ensure compliance for the installed systems. As the site surveys were 
conducted after the final investment decision (FID), issues surfaced 
where sites needed additional time to locate and/or site prep the 
space for the simulators. All simulators have been installed and are 
being used. The $36.60 program value includes cost for Full Time 
Equivalent positions. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 24: Trajectory Management--Arrival Tactical Flow Time Based 
Flow Management: 

[Refer to PDF for image: data] 

Purpose and status: 
Time Based Flow Management (TBFM) will modernize and enhance the 
current Traffic Management Advisor (TMA) System. Traffic Management 
Advisor (TMA) is a vital part of the NAS and enhances air traffic 
operations by reducing delays and increasing efficiency of airline 
operations. TMA is an automation system currently available that 
enables the use of time-based metering to optimize the flow of 
aircraft as they approach and depart congested airspace and airports. 
TBFM is an evolution of the TMA Program and uses time-based metering 
software to optimize capacity in the NAS. 

Contractors: Lockheed Martin. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving Trajectory Management: 80; 
Start date: Apr. 2010; 
Completion date: Nov. 2014; 
Estimated cost: $115.00. 

Current target: 
Number of facilities receiving Trajectory Management: 80; 
Start date: Apr. 2010; 
Completion date: Nov. 2014; 
Estimated cost: $115.00. 

As of August 2011, this program was within its original budget and 
schedule. 

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 25: Terminal Doppler Weather Radar (TDWR) Service Life 
Extension Program (SLEP): 

[Refer to PDF for image: illustration and associated data] 

[Figure: TDWR equipment] 

Purpose and status: 
This program is an ongoing effort (started in 2002) to modernize the 
existing TDWR equipment, so that TDWR continues to provide benefits to 
the NAS and the National Weather Service. The overarching purpose of 
SLEP projects is to sustain the TDWR's windshear service, needed by 
pilots and air traffic controllers. SLEP encompasses the procurement 
and modernization of TDWR facilities and equipment, replacing 
equipment that has reached or soon will reach the end of its useful 
life or become obsolete. 

Contractor: FAA. 
 
Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving TDWR SLEP: 47; 
Start date: Feb. 2003; 
Completion date: Dec. 2013; 
Estimated cost: $74.50.   

Current target: 
Number of facilities receiving TDWR SLEP: 47; 
Start date: Feb. 2003; 
Completion date: Sept. 2017; 
Estimated cost: $77.04.   

As of August 2011, this program was about $2.54 million, or about 3 
percent, over its original budget and 45 months behind its original 
schedule. According to FAA, this cost increase reflected an executive 
committee decision in August 2010 to add $2.5 million for radomes 
(shelters for radar antennas). This program received approval from the 
Investment Decision Authority in March 2011 for a baseline change 
decision for schedule and cost. This cost increase is associated with 
eight of the nine projects that make up the TDWR SLEP: Delays during 
development and testing, technical issues, field resource constraints, 
and delays in obtaining an executive committee decision on the number 
of radomes contributed to the schedule delays.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 26: Traffic Flow Management System (TFMS) Tech Refresh: 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts TFMS] 

Purpose and status: 
The TFMS Tech Refresh program replaces workstations, servers, routers, 
and associated equipment of the TFMS core located at the William J. 
Hughes Technical Center, in Atlantic City, NJ; its backup at the 
Disaster Recovery Center is located at the prime contractor site; the 
TFMS legacy application hardware for National Traffic Management Log 
is at the same locations. TFMS is the automation backbone for the FAA 
Command Center and the nationwide Traffic Management Units that assist 
the FAA Command Center in the strategic planning and management of air 
traffic. TFMS hosts the software decision support systems that assist 
in managing and metering air traffic to reduce delays and make maximum 
use of system capacity to dynamically balance growing flight demands 
with NAS capacity. 

Contractor: Computer Sciences Corporation. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving TFMS Tech Refresh: 3; 
Start date: Mar. 2011; 
Completion date: Sept. 2015; 
Estimated cost: $39.60.   

Current target: 
Number of facilities receiving TFMS Tech Refresh: 3; 
Start date: Mar. 2011; 
Completion date: Sept. 2015; 
Estimated cost: $39.60.   

As of August 2011, this program was within budget and on schedule. 
According to FAA, the program was approved by the Investment Decision 
Authority in March 2011 for a final investment decision. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 27: Ultra High Frequency (UHF) Radio Replacement: 

[Refer to PDF for image: illustration and associated data] 

[Figure: UHF radio] 

Purpose and status: 
The UHF Radio Replacement program replaces radios at remote 
communications air and ground sites and backup emergency 
communications locations. The transmitters and receivers will be 
deployed concurrently with the NEXCOM very high frequency multimode 
digital radio to minimize implementation costs. The Federal Aviation 
Act of 1958 requires the FAA Administrator to provide necessary 
facilities for the regulation and protection of air traffic. For 
national security reasons, DOD must have access to the NAS. In order 
for air traffic controllers to communicate with some DOD aircraft, FAA 
must operate and maintain the ultra high frequency radio 
infrastructure. DOD reaffirmed this requirement in August 2001. 

Contractor: General Dynamics C4 Systems (GD4S). 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving UHF radio replacements: 1,437; 
Start date: Nov. 2002; 
Completion date: Sept. 2010; 
Estimated cost: $85.10.     

Current target: 
Number of facilities receiving UHF radio replacements: 1,437; 
Start date: Nov. 2002; 
Completion date: Sept. 2013; 
Estimated cost: $93.10.     

As of August 2011, this program was $8 million, or 9.4 percent, over 
its original budget and 36 months behind its original 2010 schedule. 
Since August 2011, the number of facilities receiving UHF radio 
replacements has decreased by 15 percent. According to FAA, the $8 
million increase in the program's budget was used to procure 
additional UHF radios to bridge the gap until new radios are delivered 
through the NEXCOM Segment 2 procurement. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 28: Next Generation Voice Recorder Replacement Program (VRRP): 

[Refer to PDF for image: data] 

Purpose and status: 
The VRRP provides new voice recorders for en route and terminal ATC 
facilities. The program is intended to replace obsolete and 
unsupportable digital voice recorders that have reached the end of 
their 10-year life. This program will provide recording capability 
between air traffic controllers, pilots, and ground-based air traffic 
facilities in all ATC domains. It will also be utilized in the 
investigation of accidents and incidents and routine evaluation of ATC 
operations, including operational errors and operational deviations. 
Additionally, the program is intended to reduce operations and 
maintenance costs to sustain recorder systems. 

Contractor: Nice Systems Inc. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving VRRP: 478; 
Start date: Apr. 2007; 
Completion date: May 2013; 
Estimated cost: $45.80. 

Current target: 
Number of facilities receiving VRRP: 478; 
Start date: Apr. 2007; 
Completion date: May 2013; 
Estimated cost: $45.80. 

As of August 2011, this program was within its original budget and 
schedule. 

Sources: GAO (analysis); FAA (data). 

[End of figure] 

Figure 29: Voice Switching and Control Switching System (VSCS) Tech 
Refresh Phase 2: 

[Refer to PDF for image: illustration and associated data] 

[Figure: VSCS system equipment] 

Purpose and status: 
The VSCS system will replace and upgrade hardware and software 
components for the voice switching systems in all 21 en route air 
traffic control centers. These upgrades are intended to ensure that 
the centers' air-to-ground communication capabilities are reliable and 
available for separating aircraft, coordinating flight plans, and 
transferring information between air traffic control facilities in the 
en route environment. Phase 1 of the equipment upgrade began in 2000 
and ended in 2006. Phase 2 consists of additional upgrades that will 
allow the system to remain in use beyond 2014. 

Contractor: Harris Corp. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving VSCS Tech Refresh Phase 2: 23; 
Start date: Aug. 2006; 
Completion date: June 2012; 
Estimated cost: $83.90.    

Current target: 
Number of facilities receiving VSCS Tech Refresh Phase 2: 23; 
Start date: Aug. 2006; 
Completion date: Dec. 2014; 
Estimated cost: $83.90.    

As of August 2011, this program was within its original budget but 30 
months behind its original schedule. According to FAA, the 30-month 
(42.9 percent) increase in schedule duration is associated with two of 
the projects in the VSCS Tech Refresh program: (1) The completion of 
the power supply refurbishment has slipped to December 2014. This 
revised end date reflects an expansion of the project to encompass the 
retrofit of all power supplies at all centers rather than a limited 
retrofit program via attrition (as power supply fails at the centers). 
The determination to make this change was based on an analysis of 
field failures that showed power supply capacitors have a limited 
lifetime; (2) The completion of the PLM software conversions has 
slipped as well. According to FAA, this schedule variance is due to 
resource constraints, both for personnel and for the test lab facility 
at the William J. Hughes Technical Center. The Investment Decision 
Authority approved a baseline change decision in May 2011. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 30: Wide Area Augmentation System (WAAS): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Artwork depicts WAAS equipment and locations] 

Purpose and status: 
WAAS, a satellite-based navigation technology, allows any qualifying 
airport in the NAS to have vertical and horizontal guidance without 
expensive legacy navigation hardware installed at each runway. WAAS is 
intended to increase safety and enhance capacity in the national 
airspace at a lower cost than other alternatives. WAAS continuously 
broadcasts a GPS-like signal in space for horizontal and vertical 
navigation across the NAS. WAAS is also currently supporting early 
opportunities for many of the NextGen capabilities in the Performance 
Based Navigation area using satellite-based navigation routes and 
terminal operations. Release 3 of this program is scheduled for July 
2012, release 4 for September 2012, and release 5 for September 2013. 

Contractors: Raytheon, AMTI, and Lockheed Martin. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving WAAS: 1; 
Start date: Jan. 1998; 
Completion date: Aug. 1999; 
Estimated cost: $1,000.60.   

Current target: 
Number of facilities receiving WAAS: 1; 
Start date: May 2009; 
Completion date: Sept. 2013; 
Estimated cost: $3,008.10.   

As of August 2011, this program was $2,007.50 million, or about 200 
percent, over its original budget and 169 months behind its original 
schedule. According to FAA, this program's budget was increased 
because funding for satellite communications was transferred to the 
Facilities and Equipment appropriation from the Operations and 
Maintenance appropriation and because the life-cycle of the baseline 
was extended. The schedule was extended to meet specifications and 
requirements. 

According to FAA, under the current contract, the WAAS program is 
divided into three phases. Phase 1 was completed with the achievement 
of Initial Operating Capability in 2003, and Phase 2 was completed 
with the achievement of Full Lateral Precision Approaches with 
Vertical Guidance (LPV) in 2008. Phase 3 is the current segment of 
WAAS, a mixed lifecycle of development, modernization, and 
enhancements in conjunction with steady state operations and 
maintenance and will provide a robust, reliable, and sustainable, LPV-
200 capability. The initial cost of $1 billion in 1998 was increased 
to $3 billion in 1999. This increase was due to developmental problems 
and issues with acquisition plans for obtaining WAAS satellites, and 
to reflect the transfer of the satellite leases from the operations 
and maintenance account to the facilities and equipment account. In 
2004, the baseline increased to $3.3 billion. This increase reflects 
the adjustment of scope to include LPV and extension of the program 
lifecycle. In 2009, the next useful segment, Phase 3, was approved 
with a decrease in the cost baseline to $3.0 billion. The decrease is 
associated with LPV-200 achieving Category 1 equivalent service 
without system changes. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 31: Weather and Radar Processor (WARP) Sustain: 

[Refer to PDF for image: illustration and associated data] 

[Figure: WARP radar display] 

Purpose and status: 
The WARP Sustain program's aging hardware and software infrastructure 
must be supported by incorporating limited changes to the supported 
operating system and hardware when parts become obsolescent. The 
existing architecture is being sustained until its replacement, the 
NextGen Weather Processor, is deployed. Sustaining WARP will ensure 
that its weather processing and distribution capabilities continue to 
provide data that support a broad range of users throughout the NAS. 
The WARP system is operational at FAA's 21 en route centers and FAA 
Command Center. Among other purposes, WARP is designed to integrate 
weather data onto air traffic control displays and to disseminate 
weather data to critical national airspace systems, such as ATOP. 

Contractor: Harris Corporation. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving WARP Sustain: 22; 
Start date: Apr. 2009; 
Completion date: Apr. 2012; 
Estimated cost: $27.50.     

Current target: 
Number of facilities receiving WARP Sustain: 22; 
Start date: Apr. 2009; 
Completion date: Apr. 2012; 
Estimated cost: $27.50.     

As of August 2011, this program was within its original budget and 
schedule. 

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

Figure 32: Weather Camera Program (WCP): 

[Refer to PDF for image: illustration and associated data] 

[Figure: Photos of camera displays] 

Purpose and status: 
A disproportionate number of all U.S. aircraft crashes occur in 
Alaska. Limited weather information in Alaska contributes to a higher 
risk of accidents and flight inefficiencies. A National Transportation 
Safety Board safety study recommended FAA assist the National Weather 
Service by evaluating the technical feasibility and aviation safety 
benefits of remote color video weather observing systems in Alaska. 
The mission of the WCP is to improve safety and efficiency by 
providing weather visibility information in the form of near-real-time 
camera images to aviation users. 

Contractors: NISC and TSSC. 

Baseline changes over time: 

Schedule and cost targets (dollars in millions): 

Baseline target: 
Number of facilities receiving WCP: 221; 
Start date: Dec. 2007; 
Completion date: Sept. 2014; 
Estimated cost: $25.10.  

Current target: 
Number of facilities receiving WCP: 221; 
Start date: Dec. 2007; 
Completion date: Sept. 2014; 
Estimated cost: $20.20.  

As of August 2011, this program was within its original schedule; 
however, its funding has been reduced by $4.90 million, or about 20 
percent.  

Sources: GAO (analysis); FAA (data and artwork). 

[End of figure] 

[End of section] 

Appendix III: Assessments of Four FAA Program Cost Estimates: 

This appendix provides the results of our analysis of the extent to 
which the processes and methodologies used to develop and maintain the 
four FAA cost estimates meet the characteristics of high-quality cost 
estimates. These characteristics incorporate the 12 steps consistently 
applied by cost-estimating organizations throughout the federal 
government and industry and considered best practices for developing 
cost estimates and that are listed in table 2 of the report. The 
following tables provide the detailed results of our analysis of the 
program cost estimates for Automatic Dependent Surveillance-Broadcast 
(ADS-B), Collaborative Air Traffic Management Technologies (CATMT), 
System Wide Information Management (SWIM), and Wide Area Augmentation 
System (WAAS). "Not met" means the program provided no evidence that 
satisfies any of the criteria. "Minimally met" means the program 
provided evidence that satisfies a small portion of the criterion. 
"Partially met" means the program provided evidence that satisfies 
about half of the criterion. "Substantially met" means the program 
provided evidence that satisfies a large portion of the criterion. 
"Fully met" means the program provided evidence that completely 
satisfies the criterion. 

[End of figure] 

Table 8: GAO's Analysis of the FAA's ADS-B Cost Estimates: 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Well-documented; 

* Captures the source data used, the reliability of the data, and how 
the data were made compatible with other data in the estimate. 
Data should be collected from primary sources. The source, content, 
time, and units should be adequately documented. Data should also be 
analyzed to determine accuracy and reliability, and to identify cost 
drivers; 
Key examples of rationale for assessment: 
* Lists data sources, but does not provide documentation of the source 
data, bringing into question the reliability of the data. (Minimally 
meets.) Data are the foundation of every cost estimate. Depending on 
data quality, an estimate can range anywhere from a mere guess to a 
highly defensible cost position. Data are often in many different 
forms and need to be adjusted before being used. The cost estimator 
needs information about the source and reliability of the data in 
order to know whether the data collected can be used directly or need 
to be modified; 

* Describes the calculations and the methodology used to derive each 
element's cost. 
Documentation should describe what calculation methods are used, as 
well as how they were applied, and explain any anomalies; 
Key examples of rationale for assessment: 
* Documentation does not fully explain how FAA derived estimated 
costs. For example, FAA relied on expert opinion to estimate several 
costs but provided no historical data to back up the opinions. There 
was also no supporting information for the software cost estimates. 
(Minimally meets.) Poorly documented cost estimates can cause a 
program's credibility to suffer because the documentation cannot 
explain the rationale for the methodology or the calculations. 
Estimates that lack sufficient documentation are not useful for 
updates or information sharing and can hinder understanding and proper 
use; 

* Describes how the estimate was developed. 
The data supporting the estimate should be available and adequately 
documented so that the estimate can be easily documented to reflect 
actual costs or program changes; 
Key examples of rationale for assessment: 
* Cost calculations are described at a high level, but the 
documentation does not provide enough detail so that a cost analyst 
unfamiliar with the program could understand what was done and 
replicate it. Furthermore, we found an inconsistency between dollar 
values stored in the cost estimating tool and the Excel spreadsheets 
used to report life-cycle costs. (Partially met.) Without good 
documentation, management and oversight officials will not be 
convinced that the estimate is credible; supporting data, lessons 
learned, and reasons why costs changed will not be available for 
future use; questions about the approach or data used to create the 
estimate cannot be answered; and the scope of the analysis cannot be 
thoroughly defined; 

* Discusses the technical baseline description. 
A technical baseline description provides a common definition of the 
program, including detailed technical, program, and schedule 
descriptions of the system, for a cost estimate to be built on. The 
data in the technical baseline should be consistent with the data used 
to develop the cost estimate; 
Key examples of rationale for assessment: 
* Technical details contained within the basis-of-estimate 
documentation are consistent with corresponding details in the 
technical baseline. (Substantially meets); 

* Provides evidence of management review and acceptance. 
There should be a briefing to management, including a clear 
explanation of how the cost estimate was derived. Management's 
acceptance of the cost estimate should be documented; 
Extent to which criterion was met: Partially met; 
Key examples of rationale for assessment: 
* The estimate was briefed to the Joint Resources Council. The 
briefing included a discussion of the scope, justification, and cost 
of the program. It also contained an overview of the program's 
technical requirements, life-cycle costs, assumptions, and results of 
risk and sensitivity analysis. (Met); 

Extent to which criterion was met: Partially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Comprehensive; 

* Includes all life-cycle costs. 
A life-cycle cost estimate provides a complete and structured 
accounting of all resources and associated cost elements required to 
develop, produce, deploy, and sustain a particular program. It should 
cover the program from its inception through its retirement; 
Key examples of rationale for assessment: 
* Includes costs from fiscal year 2007 through fiscal year 2035 for 
both government and contractor efforts across all phases of the 
program (Met); 

* Completely defines the program, reflects the current schedule, and 
is technically reasonable. 
The cost estimate should be based on a documented technical baseline 
description, which provides a common definition of the program--
including detailed technical, program, and schedule descriptions of 
the system; 
Key examples of rationale for assessment: 
* Reflects the current project schedule, but no single technical 
baseline was provided and there was no evidence that the technical 
documents had been updated to reflect changes. (Substantially met); 

* Has a product-oriented work breakdown structure, and is traceable to 
the program's technical scope, at an appropriate level of detail. 
A work breakdown structure provides a basic framework for a variety of 
related activities like estimating costs, developing schedules, 
identifying resources and potential risks, and providing the means for 
measuring program status using earned value management. It is product-
oriented if it allows a program to track cost and schedule by defined 
deliverables, such as a hardware or software component; 
Key examples of rationale for assessment: 
* Reflects the FAA standard work breakdown structure, which is not 
product-oriented; instead, it breaks work down into functional 
categories. While the cost estimate work breakdown structure matches 
the work breakdown structure for earned value management, it does not 
match the work breakdown structure for the schedule. Furthermore, 
there was no evidence that the work breakdown structure had been 
updated as the program became better defined. (Partially met.) Without 
a work breakdown structure, the program lacks a framework to develop a 
schedule and cost plan that can be used to easily track technical 
accomplishments. A standard product-oriented work breakdown structure 
facilitates the tracking of resource allocations and expenditures, 
which can give an agency insight to reliably estimate the cost of 
future similar programs; 

* Documents all cost-influencing ground rules and assumptions. 
Cost estimates are typically based on limited information and 
therefore need to be bound by ground rules and assumptions. Ground 
rules are a set of estimating standards that provide guidance and 
common definitions, while assumptions are judgments about past, 
present, or future conditions that may affect the estimate; 
Key examples of rationale for assessment: 
* While ground rules and assumptions were discussed, many of the 
assumptions did not include supporting data. In addition, details 
supporting risk assumptions were incomplete because the sources for 
the risk ranges were not provided. (Partially met.) Unless ground 
rules and assumptions are clearly documented, the cost estimate will 
not have a basis for assessing potential risks. Furthermore, the 
estimate cannot be reconstructed when the original estimators are no 
longer available; 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Accurate; 

* Produces unbiased results. 
Cost estimates should have an uncertainty analysis, which determines 
where the estimate falls against the range of all possible costs; 
Key examples of rationale for assessment: 
* The cost basis of estimate contains minimum, most likely, and 
maximum values to model uncertainties for each cost element and to 
provide a range of costs; however, no analysis has been performed to 
determine the confidence level of the estimate. (Partially met.) A 
cost estimate is biased if the estimated work is overly conservative 
or too optimistic. Unless the estimate is based on an assessment of 
the most likely costs and reflects the degree of uncertainty given all 
of the risks considered, management will not be able to make good 
decisions. 

* Is properly adjusted for inflation. 
Cost data should be adjusted for inflation to ensure that comparisons 
and projections are valid. Data should also be normalized to constant-
year dollars to remove the effects of inflation; 
Key examples of rationale for assessment: 
* The cost estimate was adjusted for inflation correctly, but the 
source data for the government salaries inflation index were not 
provided. (Substantially met); 

* Contains few mistakes. 
Results should be checked for accuracy, double counting, and omissions; 
Key examples of rationale for assessment: 
* We found no instances of incorrect formulas, double-counted costs, 
or omitted costs. Moreover, costs reported by fiscal year correctly 
summed to their life-cycle totals. (Met); 

* Is regularly updated to reflect significant program changes. 
The cost estimate should be updated to reflect significant program 
changes, such as changes to schedules or other assumptions. Updates 
should also reflect actual costs so that the estimate always reflects 
the current program status; 
Key examples of rationale for assessment: 
* There was no evidence that the cost estimate was updated to reflect 
changes or actual costs to reflect the current program status. (Not 
met.) A lack of cost estimate updates interferes with analysis of 
changes in program costs and hinders collection of cost and technical 
data to support future estimates. The cost estimate should be updated 
when the technical baseline changes; otherwise, it will lack 
credibility. A properly updated cost estimate can provide decision 
makers with accurate information for assessing alternative decisions; 

* Documents and explains variances between planned and actual costs. 
Variances between planned and actual costs should be documented, 
explained, and reviewed. For any elements whose actual costs or 
schedules differ from the estimate, the estimate should discuss 
variances and lessons learned; 
Key examples of rationale for assessment: 
* Variances between planned and actual costs, and explanations for the 
variance, were documented in earned value management data, but no 
variance for total cost at completion was included in the cost 
estimate documentation. (Substantially met.) Without a documented 
comparison between the current estimate (updated with actual costs) 
and the old estimate, cost estimators cannot determine the level of 
variance between the two estimates. That is, the estimators cannot see 
how well they are estimating and how the program is changing over time; 

* Reflects cost-estimating experiences from comparable programs. 
The estimate should be based on historical cost estimation data and 
actual experiences from other comparable programs. These data should 
be reliable and relevant to the new program; 
Key examples of rationale for assessment: 
* There is some evidence that the estimate was based on a historical 
record of cost-estimating and actual experiences from other analogous 
programs. However, the reliability, risks, and applicability of the 
analogous data were not addressed. Also, comparisons were not made 
between ADS-B and analogous programs. (Minimally met.) Historical data 
provide the cost estimator with insight into actual costs on similar 
programs, including any cost growth that occurred after the original 
estimate. As a result, historical data can be used to challenge 
optimistic assumptions and bring more realism to a cost estimate; 

Extent to which criterion was met: Partially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Credible; 

* Includes a sensitivity analysis that identifies a range of possible 
costs based on varying inputs. 

A sensitivity analysis examines how changes to key assumptions and 
inputs affect the estimate. The estimate should identify key cost 
drivers, examine their parameters and assumptions, and re-estimate the 
total cost by varying each parameter between its minimum and maximum 
range; 
Key examples of rationale for assessment: 
* While a high-level sensitivity analysis was provided to the Joint 
Resources Council for the equipage cost driver, other key cost 
drivers, assumptions, and data inputs were not varied. (Minimally 
met.) Because uncertainty cannot be avoided, it is necessary to 
identify the cost elements that represent the most risk. A sensitivity 
analysis reveals how the cost estimate is affected by a change in a 
single assumption, which helps the cost estimator understand the 
extent to which each variable affects the cost estimate. Any sources 
of variation should be well-documented and traceable; 

* Contains a risk and uncertainty analysis. 
A risk and uncertainty analysis recognizes the potential for error and 
attempts to quantify it by identifying the effects of changing key 
cost drivers; 
Key examples of rationale for assessment: 
* A risk analysis for each cost element was performed but the risk 
data were essentially the same for each element and there was no 
documentation of correlations of risks between the cost elements. 
(Substantially met); 

* Includes cross-checking of major cost elements. 
A cross-check is done by using a different method to see if it 
produces similar results; Includes a comparison to an independent cost 
estimate conducted by another organization; A second, independent cost 
estimate should be performed by an organization outside of the program 
office's influence. It should be based on the same technical baseline, 
ground rules, and assumptions as the original estimate; 
Key examples of rationale for assessment: 
* Contains no evidence that cross-checks were performed. (Not met) The 
main purpose of cross-checking is to determine whether alternative 
methods produce similar results. If so, then confidence in the 
estimate increases, leading to greater credibility; 
* A rough order-of-magnitude estimate was performed by another 
organization but addressed only a part of the total program. 
(Partially met) An independent cost estimate is considered one of the 
best and most reliable estimate validation methods. It provides an 
independent view of expected program costs that tests the program 
office's estimate for reasonableness. Without an independent cost 
estimate, decisions makers will lack insight into a program's 
potential costs because independent cost estimates frequently use 
different methods and are less burdened with organizational bias. 

Extent to which criterion was met: Partially met. 

Source: GAO analysis of FAA's ADS-B cost estimate. 

[End of table] 

Table 9: GAO's Analysis of FAA's CATMT Cost Estimates: 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Well-documented: 

* Captures the source data used, the reliability of the data, and how 
the data were made compatible with other data in the estimate. 
Data should be collected from primary sources. The source, content, 
time, and units should be adequately documented. The data should also 
be analyzed to determine accuracy and reliability, and to identify 
cost drivers; 
Key examples of rationale for assessment: 
* Lists data sources, the majority of which were subject matter 
experts. No details on the qualifications or background of these 
experts were provided, and there was no documentation about the 
reliability of the data. (Partially met.) Data are the foundation of 
every cost estimate. Depending on data quality, an estimate can range 
anywhere from a mere guess to a highly defensible cost position. Data 
are often in many different forms and need to be adjusted before being 
used. The cost estimator needs information about the source and 
reliability of the data in order to know whether the data collected 
can be used directly or need to be modified; 

* Describes the calculations and the methodology used to derive each 
element's cost. 
Documentation should describe what calculation methods are used, as 
well as how they were applied, and explain any anomalies; 
Key examples of rationale for assessment: 
* Documentation does not fully explain how FAA derived estimated 
costs. For example, FAA relied on expert opinion to estimate several 
costs based on a percentage of other cost elements but provided no 
historical data to back up the opinions. There was also no supporting 
information for the software cost estimates. (Partially met.) Poorly 
documented cost estimates can cause a program's credibility to suffer 
because the documentation cannot explain the rationale for the 
methodology or the calculations. Estimates that lack sufficient 
documentation are not useful for updates or information sharing and 
can hinder understanding and proper use; 

* Describes how the estimate was developed. 
The data supporting the estimate should be available and adequately 
documented so that the estimate can be easily documented to reflect 
actual costs or program changes; 
Key examples of rationale for assessment: 
* Cost calculations are described in detail for fiscal year 2014, but 
documentation was missing for fiscal years 2015 through 2022. As a 
result, the documentation does not provide enough detail so that a 
cost analyst unfamiliar with the program could understand what was 
done and replicate it. (Partially met.) Without good documentation, 
management and oversight officials will not be convinced that the 
estimate is credible; supporting data, lessons learned, and reasons 
why costs changed will not be available for future use; questions 
about the approach or data used to create the estimate cannot be 
answered; and the scope of the analysis cannot be thoroughly defined; 

* Discusses the technical baseline description. 
A technical baseline description provides a common definition of the 
program, including detailed technical, program, and schedule 
descriptions of the system, for a cost estimate to be built on. The 
data in the technical baseline should be consistent with the cost 
estimate; 
Key examples of rationale for assessment: 
* Many inconsistencies were found between the cost basis of the 
estimate document and the cost model. (Partially met.) Because the 
technical baseline is intended to serve as the basis for developing a 
cost estimate, it should be discussed in the cost estimate 
documentation. Without a technical baseline, the cost estimate will 
not be based on a comprehensive program description and will lack 
specific information about technical and program risks; 

* Provides evidence of management review and acceptance; 
There should be a briefing to management, including a clear 
explanation of how the cost estimate was derived. Management's 
acceptance of the cost estimate should be documented; 
Key examples of rationale for assessment:
* The estimate was presented to the Joint Resources Council management 
during an investment decision briefing. The briefing included a 
program overview, life-cycle costs over time, risk analysis, budget 
justification, and recommendation. The briefing did not include detail 
on the estimating methodology for each cost element; sensitivity, 
risk, and uncertainty analysis; or an affordability analysis. 
(Substantially met). 

Extent to which criterion was met: Partially met; 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Comprehensive: 

* Includes all life-cycle costs. 
A life-cycle cost estimate provides a complete and structured 
accounting of all resources and associated cost elements required to 
develop, produce, deploy, and sustain a particular program. It should 
cover the program from its inception through its retirement; 
Key examples of rationale for assessment: 
* Includes costs from fiscal year 2009 through fiscal year 2022 for 
both government and contractor efforts across all phases of the 
program. However, some government labor costs for in-service 
management were not included. (Substantially met); 

* Completely defines the program, reflects the current schedule, and 
is technically reasonable. 
The cost estimate should be based on a documented technical baseline 
description, which provides a common definition of the program--
including detailed technical, program, and schedule descriptions of 
the system; 
Key examples of rationale for assessment: 
* Reflects many detailed technical requirements, but no single 
technical baseline was provided. (Substantially met); 

* Has a product-oriented work breakdown structure, and is traceable to 
the program's technical scope, at an appropriate level of detail. 
A work breakdown structure provides a basic framework for a variety of 
related activities like estimating costs, developing schedules, 
identifying resources and potential risks, and providing the means for 
measuring program status using earned value management. It is product-
oriented if it allows a program to track cost and schedule by defined 
deliverables, such as a hardware or software component; 
Key examples of rationale for assessment: 
* Reflects FAA's standard work breakdown structure, which does not 
follow a product-oriented work breakdown structure. Instead, it breaks 
work down into functional categories. The cost estimate work breakdown 
structure matches the schedule work breakdown structure. However, we 
found no evidence that the work breakdown structure had been updated 
as the program became better defined. (Substantially met); 

* Documents all cost-influencing ground rules and assumptions. 
Cost estimates are typically based on limited information and 
therefore need to be bound by ground rules and assumptions. Ground 
rules are a set of estimating standards that provide guidance and 
common definitions, while assumptions are judgments about past, 
present, or future conditions that may affect the estimate; 
Key examples of rationale for assessment: 
* The estimate provides a limited set of cost-influencing ground rules 
and assumptions; however, many of the assumptions did not include 
supporting data. In addition, there was little evidence risks were 
identified if the assumption did not hold. (Partially met.) Unless 
ground rules and assumptions are clearly documented, the cost estimate 
will not have a basis for assessing potential risks. Furthermore, the 
estimate cannot be reconstructed when the original estimators are no 
longer available. 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Accurate: 

* Produces unbiased results. 
Cost estimates should have an uncertainty analysis, which determines 
where the estimate falls against the range of all possible costs; 
Key examples of rationale for assessment: 
* The cost basis of estimate contains low, likely, and high estimates 
for software lines of code, but the estimate did not have a similar 
analysis for every assumption in the cost model. In addition, we found 
no confidence level associated with the estimate. (Partially met.) A 
cost estimate is biased if the estimated work is overly conservative 
or too optimistic. Unless the estimate is based on an assessment of 
the most likely costs and reflects the degree of uncertainty given all 
of the risks considered, management will not be able to make good 
decisions; 

* Is properly adjusted for inflation. 
Cost data should be adjusted for inflation to ensure that comparisons 
and projections are valid. Data should also be normalized to constant-
year dollars to remove the effects of inflation; 
Key examples of rationale for assessment: 
* The cost estimate was adjusted for inflation correctly and the 
source data for the inflation indexes were provided. (Met); 

* Contains few mistakes. 
Results should be checked for accuracy, double counting, and omissions; 
Key examples of rationale for assessment: 
* While we found no apparent mistakes in the calculations, as stated 
earlier, there were inconsistencies in costs between the cost model 
and the cost basis of estimate documentation. (Partially met.) 
Validating that a cost estimate is accurate requires thoroughly 
understanding and investigating how the cost model was constructed. 
Without access to a detailed cost model and estimate details, 
calculations may not be accurate or expressed consistently; 

* Is regularly updated to reflect significant program changes. 
The cost estimate should be updated to reflect significant program 
changes, such as changes to schedules or other assumptions. Updates 
should also reflect actual costs so that the estimate always reflects 
the current program status; 
Key examples of rationale for assessment: 
* We found no evidence that the cost estimate was updated to reflect 
changes or actual costs. (Not met.) A lack of cost estimate updates 
interferes with analysis of changes in program costs and hinders 
collection of cost and technical data to support future estimates. The 
cost estimate should be updated when the technical baseline changes; 
otherwise, it will lack credibility. A properly updated cost estimate 
can provide decision makers with accurate information for assessing 
alternative decisions; 

* Documents and explains variances between planned and actual costs. 
Variances between planned and actual costs should be documented, 
explained, and reviewed. For any elements whose actual costs or 
schedules differ from the estimate, the estimate should discuss 
variances and lessons learned; 
Key examples of rationale for assessment: 
* There was no documentation of variances between planned and actual 
costs. (Not met.) Without a documented comparison between the current 
estimate (updated with actual costs) and the old estimate, the cost 
estimators cannot determine the level of variance between the two 
estimates. That is, the estimators cannot see how well they are 
estimating and how the program is changing over time; 

* Reflects cost-estimating experiences from comparable programs. 
The estimate should be based on historical cost estimation data and 
actual experiences from other comparable programs. These data should 
be reliable and relevant to the new program; 
Key examples of rationale for assessment: 
* There is some evidence that the estimate was based on a historical 
record of cost-estimating and actual experiences from other analogous 
programs. For example, software costs were estimated using function 
points and the Constructive Cost Model (COCOMO). However, no 
comparison was made between CATMT and other comparable FAA programs 
for the purpose of the validating the CATMT estimate. (Minimally met.) 
Historical data provide the cost estimator with insight into actual 
costs on similar programs, including any cost growth that occurred 
after the original estimate. As a result, historical data can be used 
to challenge optimistic assumptions and bring more realism to a cost 
estimate. 

Extent to which criterion was met: Partially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Credible: 

* Includes a sensitivity analysis that identifies a range of possible 
costs based on varying inputs. 
A sensitivity analysis examines how changes to key assumptions and 
inputs affect the estimate. The estimate should identify key cost 
drivers, examine their parameters and assumptions, and re-estimate the 
total cost by varying each parameter between its minimum and maximum 
range; 
Key examples of rationale for assessment: 
A sensitivity analysis was provided on the discount rate used in the 
economic analysis, but other key cost drivers like software design and 
development, as well as assumptions and data inputs, were not varied. 
(Minimally met.) Because uncertainty cannot be avoided, it is 
necessary to identify the cost elements that represent the most risk. 
A sensitivity analysis reveals how the cost estimate is affected by a 
change in a single assumption, which helps the cost estimator 
understand the extent to which each variable affects the cost 
estimate. Any sources of variation should be well-documented and 
traceable; 

* Contains a risk and uncertainty analysis. 
A risk and uncertainty analysis recognizes the potential for error and 
attempts to quantify it by identifying the effects of changing key 
cost drivers; 
Key examples of rationale for assessment: 
* The documentation discusses some aspects of a risk and uncertainty 
analysis, such as risk percents, but lacks some results, such as the 
confidence level for the cost estimate. (Partially met.) The program 
estimate should reflect the degree of uncertainty, so that a level of 
confidence can be provided to management about the estimate. An 
estimate without risk and uncertainty analysis is unrealistic because 
it does not assess the variability in the cost estimate from such 
effects as schedules slipping, missions changing, and proposed 
solutions not meeting users' needs; 

* Cross-checking of major cost elements. 
A cross-check is done by using a different method to see if it 
produces similar results; 
Key examples of rationale for assessment: 
* The documentation states that a function point analysis and a 
historical cost estimation method were used to cross-check the 
software cost estimate. However, a comparison of the model results was 
not provided, and we could not identify other examples of cross-
checks. (Minimally met.) The main purpose of cross-checking is to 
determine whether alternative methods produce similar results. If so, 
then confidence in the estimate increases, leading to greater 
credibility; 

* A comparison to an independent cost estimate conducted by another 
organization. 
A second, independent cost estimate should be performed by an 
organization outside of the program office's influence. It should be 
based on the same technical baseline, ground rules, and assumptions as 
the original estimate; 
Key examples of rationale for assessment: 
* There was no evidence that an independent cost estimate was 
conducted. (Not met.) An independent cost estimate is considered one 
of the best and most reliable estimate validation methods. It provides 
an independent view of expected program costs that tests the program 
office's estimate for reasonableness. Without an independent cost 
estimate, decision makers will lack insight into a program's potential 
costs because independent cost estimates frequently use different 
methods and are less burdened with organizational bias. 

Extent to which criterion was met: Minimally met> 

Source: GAO analysis of FAA's CATMT cost estimate. 

[End of table] 

Table 10: GAO's Analysis of FAA's SWIM Cost Estimates: 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Well-documented: 

* Captures the source data used, the reliability of the data, and how 
the data were made compatible with other data in the estimate; 
Data should be collected from primary sources. The source, content, 
time, and units should be adequately documented. Data should also be 
analyzed to determine accuracy and reliability, and to identify cost 
drivers; 
Key examples of rationale for assessment: 
* The documentation captures the majority of the data sources used. 
However, it does not provide specific backup of the data sources and 
does not capture the reliability or the accuracy of the data. 
(Partially met.) Data are the foundation of every cost estimate. 
Depending on data quality, an estimate can range anywhere from a mere 
guess to a highly defensible cost position. Data are often in many 
different forms and need to be adjusted before being used. The cost 
estimator needs information about the source and reliability of the 
data in order to know whether the data collected can be used directly 
or need to be modified; 

* Describes the calculations and the methodology used to derive each 
element's cost; 
Documentation should describe what calculation methods are used, as 
well as how they were applied, and explain any anomalies; 
Key examples of rationale for assessment: 
* The documentation describes in detail the methodology, calculations, 
basis of data, and quantities used to determine most of the cost 
estimate. However, there is little detail provided about the cost 
elements that used the analogy and parametric estimating 
methodologies. (Substantially met); 

* Describes how the estimate was developed; 
The data supporting the estimate should be available and adequately 
documented so that the estimate can be easily documented to reflect 
actual costs or program changes; 
Key examples of rationale for assessment: 
* The documentation provides enough detail so that a cost analyst 
unfamiliar with the program could understand what was done and 
replicate it; however, there was no sensitivity analysis performed. 
(Substantially met); 

* Discusses the technical baseline description; 
A technical baseline description provides a common definition of the 
program, including detailed technical, program, and schedule 
descriptions of the system, for a cost estimate to be built on. The 
data in the technical baseline should be consistent with the cost 
estimate; 
Key examples of rationale for assessment: 
* The technical details contained within the basis of estimate are 
consistent with the corresponding details in the technical baseline, 
but there is no mapping between the basis of estimate, the technical 
baseline, and the life-cycle cost estimate. (Substantially met); 

* Provides evidence of management review and acceptance; 
There should be a briefing to management, including a clear 
explanation of how the cost estimate was derived. Management's 
acceptance of the cost estimate should be documented; 
Key examples of rationale for assessment: 
* The estimate was presented to management during a briefing and was 
accepted by the issuance of a Joint Resources Council record of 
decision. The briefing included a program overview, a comparison 
between the 2007 and 2009 estimates, a breakout of cost drivers, and a 
business case analysis. In addition, a software estimate briefing 
discussed the estimating methodology used. However, more specific 
items, such as data sources, risk information, and a comparison to an 
independent cost estimate were not provided. (Partially met.) A cost 
estimate is not considered valid until management has approved it. It 
is imperative that management understand how the estimate was 
developed, including the risks associated with the underlying data and 
methods. 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Comprehensive: 

* Includes all life-cycle costs; 
A life-cycle cost estimate provides a complete and structured 
accounting of all resources and associated cost elements required to 
develop, produce, deploy, and sustain a particular program. It should 
cover the program from its inception through its retirement; 
Key examples of rationale for assessment: 
* Includes all life-cycle costs for each of the seven SWIM 
Implementing Programs (SIP) from fiscal year 2011 through fiscal year 
2033. However, costs for fiscal year 2009 and fiscal year 2010 were 
not included because these costs were already baselined. The cost 
estimate also includes both government and contractor efforts across 
all phases of the program. (Substantially met); 

* Completely defines the program, reflects the current schedule, and 
is technically reasonable; 
The cost estimate should be based on a documented technical baseline 
description, which provides a common definition of the program--
including detailed technical, program, and schedule descriptions of 
the system; 
Key examples of rationale for assessment: 
* Reflects many detailed technical requirements, including scope and 
schedule requirements, but no single technical baseline was provided. 
Also, there was no evidence that the technical baseline requirements 
had been updated. (Substantially met); 

* Has a product-oriented work breakdown structure, and is traceable to 
the program's technical scope, at an appropriate level of detail; 
A work breakdown structure provides a basic framework for a variety of 
related activities like estimating costs, developing schedules, 
identifying resources and potential risks, and providing the means for 
measuring program status using earned value management. It is product-
oriented if it allows a program to track cost and schedule by defined 
deliverables, such as a hardware or software component; 
Key examples of rationale for assessment: 
* Reflects FAA's standard work breakdown structure, which does not 
follow a product-oriented work breakdown structure. Instead, it breaks 
work down into functional categories. The cost estimate work breakdown 
structure also does not match either the schedule or the earned value 
management work breakdown structure. In addition, we found no evidence 
that the work breakdown structure had been updated as the program 
became better defined. (Partially met.) Without a work breakdown 
structure, the program lacks a framework to develop a schedule and 
cost plan that can be used to easily track technical accomplishments. 
A standard, product-oriented work breakdown structure facilitates the 
tracking of resource allocations and expenditures, which can give FAA 
insight into reliable estimates of the cost of similar programs in the 
future; 

* Documents all cost-influencing ground rules and assumptions; 
Cost estimates are typically based on limited information and 
therefore need to be bound by ground rules and assumptions. Ground 
rules are a set of estimating standards that provide guidance and 
common definitions, while assumptions are judgments about past, 
present, or future conditions that may affect the estimate; 
Key examples of rationale for assessment: 
* The estimate documents all cost-influencing ground rules and 
assumptions; however, in several instances, not all of the data 
sources supporting each were provided. In addition, while there was a 
risk adjustment section in the documentation, the majority of the risk 
adjustments were based on the judgment of the SWIM program office 
team. (Partially met.) Unless ground rules and assumptions are clearly 
documented, the cost estimate will not have a basis for assessing 
potential risks. Furthermore, the estimate cannot be reconstructed 
when the original estimators are no longer available. 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Accurate: 

* Produces unbiased results; 
Cost estimates should have an uncertainty analysis, which determines 
where the estimate falls against the range of all possible costs; 
Key examples of rationale for assessment: 
* The cost basis of estimate contains minimum, most likely and maximum 
values to model uncertainties for each cost element. No historical 
data were provided to support any of the risk adjustments, and no 
level of confidence was associated with the estimate. (Minimally met.) 
A cost estimate is biased if the estimated work is overly conservative 
or too optimistic. Unless the estimate is based on an assessment of 
the most likely costs and reflects the degree of uncertainty given all 
of the risks considered, its usefulness to management for making 
decisions is limited; 

* Is properly adjusted for inflation; 
Cost data should be adjusted for inflation to ensure that comparisons 
and projections are valid. Data should also be normalized to constant-
year dollars to remove the effects of inflation; 
Key examples of rationale for assessment: 
* The cost estimate was adjusted for inflation correctly and the 
source data for the inflation indexes were provided for all elements 
except for government salaries. (Substantially met); 

* Contains few mistakes; 
Results should be checked for accuracy, double counting, and omissions; 
Key examples of rationale for assessment: 
* The estimate contains no mistakes, and we found no evidence of 
double counting or omissions. Additionally, the documentation, 
equations, and results were consistent between the documentation and 
the cost model. (Met); 

* Is regularly updated to reflect significant program changes; 
The cost estimate should be updated to reflect significant program 
changes, such as changes to schedules or other assumptions. Updates 
should also reflect actual costs so that the estimate always reflects 
the current program status; 
Key examples of rationale for assessment: 
* There was no evidence that the cost estimate was recently updated to 
reflect changes or actual costs to reflect current program status. The 
cost estimate was updated in 2009 from the original 2007 estimate, 
which requested funding only for fiscal year 2009 through fiscal year 
2010. (Minimally met.) A lack of cost estimate updates interferes with 
analysis of changes in program costs and hinders collection of cost 
and technical data to support future estimates. The cost estimate 
should be updated when the technical baseline changes; 
otherwise, it will lack credibility. A properly updated cost estimate 
can provide decision makers with accurate information for assessing 
alternative decisions; 

* Documents and explains variances between planned and actual costs; 
Variances between planned and actual costs should be documented, 
explained, and reviewed. For any elements whose actual costs or 
schedules differ from the estimate, the estimate should discuss 
variances and lessons learned; 
Key examples of rationale for assessment: 
* Earned value management system reports for July 2010 through 
December 2010 contained actual costs and variances. However, no recent 
earned value management reports captured actual costs and variances. 
In addition, because the cost estimate work breakdown structure did 
not match the earned value management work breakdown structure, it 
would be difficult to track variances from the earned value management 
system back to the cost estimate. (Minimally met.) Without a 
documented comparison between the current estimate (updated with 
actual costs) and the old estimate, cost estimators cannot determine 
the level of variance between the two estimates. That is, the 
estimators cannot see how well they are estimating and how the program 
is changing over time; 

* Reflects cost estimating experiences from comparable programs; 
The estimate should be based on a historical cost estimation data and 
actual experiences from other comparable programs. These data should 
be reliable and relevant to the new program; 
Key examples of rationale for assessment: 
* There is some evidence that the estimate was based on a historical 
record of cost estimating. For example, per diem and hotel rates were 
obtained from a government website; however, no such comparable data 
were provided for other cost elements. (Minimally met.) Historical 
data provide the cost estimator with insight into actual costs on 
similar programs, including any cost growth that occurred after the 
original estimate. As a result, historical data can be used to 
challenge optimistic assumptions and bring more realism to a cost 
estimate. 

Extent to which criterion was met: Partially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Credible: 

* Includes a sensitivity analysis that identifies a range of possible 
costs based on varying inputs; 
A sensitivity analysis examines how changes to key assumptions and 
inputs affect the estimate. The estimate should identify key cost 
drivers, examine their parameters and assumptions, and re-estimate the 
total cost by varying each parameter between its minimum and maximum 
range; 
Key examples of rationale for assessment: 
* While an independent evaluation includes a breakout of cost drivers 
as a percentage of total cost, the cost estimate documentation does 
not show evidence that a sensitivity analysis was performed. 
(Minimally met.) Because uncertainty cannot be avoided, it is 
necessary to identify the cost elements that represent the most risk. 
A sensitivity analysis reveals how the cost estimate is affected by a 
change in a single assumption, which helps the cost estimator to 
understand the extent to which each variable affects the cost 
estimate. Any sources of variation should be well-documented and 
traceable; 

* Contains a risk and uncertainty analysis; 
A risk and uncertainty analysis recognizes the potential for error and 
attempts to quantify it by identifying the effects of changing key 
cost drivers; 
Key examples of rationale for assessment: 
* The documentation discusses some aspects of a risk and uncertainty 
analysis, and each element receives a risk adjustment. A risk briefing 
that showed cost risk ranges for all of the SIPs were also provided. 
However, the documentation did not discuss what level of confidence 
was associated with the cost estimate. (Partially met). The program 
estimate must reflect the degree of uncertainty, so that a level of 
confidence can be provided to management about the estimate. An 
estimate without risk and uncertainty analysis is unrealistic because 
it does not assess the variability in the cost estimate from such 
effects as schedules slipping, missions changing, and proposed 
solutions not meeting users' needs; 

* Includes cross-checking of major cost elements; 
A cross-check is done by using a different method to see if it 
produces similar results; 
Key examples of rationale for assessment: 
* There was no evidence that cross-checks were performed. (Not met) 
The main purpose of cross-checking is to determine whether alternative 
methods produce similar results. If so, then confidence in the 
estimate increases, leading to greater credibility; 

* Contains a comparison to an independent cost estimate conducted by 
another organization; 
A second, independent cost estimate should be performed by an 
organization outside of the program office's influence. It should be 
based on the same technical baseline, ground rules, and assumptions, 
as the original estimate; 

* There was no evidence that an independent cost estimate was 
conducted; however, there was an independent assessment briefing that 
showed a review of all SIP cost estimates. (Minimally met.) An 
independent cost estimate is considered one of the best and most 
reliable estimate validation methods. It provides an independent view 
of expected program costs that tests the program office's estimate for 
reasonableness. Without an independent cost estimate, decisions makers 
will lack insight into a program's potential costs because independent 
cost estimates frequently use different methods and are less burdened 
with organizational bias. 

Extent to which criterion was met: Minimally met. 

Source: GAO analysis of FAA's SWIM cost estimate. 

[End of table] 

Table 11: GAO's Analysis of FAA's WAAS Cost Estimates: 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Well-documented: 

* Captures the source data used, the reliability of the data, and how 
the data were made compatible with other data in the estimate; 
Data should be collected from primary sources. The source, content, 
time, and units should be adequately documented. Data should also be 
analyzed to determine accuracy and reliability, and to identify cost 
drivers; 
Key examples of rationale for assessment: 
* The documentation captures the majority of the data sources used; 
however, it does not capture the reliability or the accuracy of the 
data. (Partially met.) Data are the foundation of every cost estimate. 
Depending on data quality, an estimate can range anywhere from a mere 
guess to a highly defensible cost position. Data are often in many 
different forms and need to be adjusted before being used. The cost 
estimator needs information about the source and reliability of the 
data in order to know whether the data collected can be used directly 
or need to be modified; 

* Describes the calculations and the methodology used to derive each 
element's cost; 
Documentation should describe what calculation methods were used, as 
well as how they were applied, and explain any anomalies; 
Key examples of rationale for assessment: 
* The documentation describes in great detail the methodology, 
calculations, basis of data, and quantities used in the cost estimate. 
However, estimates for some elements were based on the judgment of 
subject matter experts. (Substantially met); 

* Describes how the estimate was developed; 
The data supporting the estimate should be available and adequately 
documented so that the estimate can be easily documented to reflect 
actual costs or program changes; 
Key examples of rationale for assessment: 
* The documentation provides enough detail so that a cost analyst 
unfamiliar with the program could understand what was done and 
replicate it. There was no sensitivity analysis performed, and some 
cost elements did not have a risk adjustment because they were 
considered ongoing efforts. (Substantially met); 

* Discusses the technical baseline description; 
A technical baseline description provides a common definition of the 
program, including detailed technical, program, and schedule 
descriptions of the system, for a cost estimate to be built on. The 
data in the technical baseline should be consistent with the cost 
estimate; 
Key examples of rationale for assessment: 
* The technical details contained within the basis of the estimate are 
consistent with the corresponding details in the technical baseline. 
However, the cost estimate does not reference the technical baseline 
documents, and there is no mapping between the basis of estimate, the 
technical baseline, and the cost estimate. (Substantially met); 

* Provides evidence of management review and acceptance; 
There should be a briefing to management, including a clear 
explanation of how the cost estimate was derived. Management's 
acceptance of the cost estimate should be documented; 
Key examples of rationale for assessment: 
* The estimate was presented to management during a briefing and was 
accepted by the issuance of a Joint Resources Council record of 
decision. The briefing included a program overview and specific 
details about the service area, previous baselines, and a cost benefit 
analysis. In addition, information about equipment procurement, the 
schedule, and the spending plan were also presented along with a risk 
analysis, an affordability analysis, and an independent evaluation. 
However, more specific items such as data sources for each cost 
element, a sensitivity analysis, an uncertainty analysis, and a 
comparison against an independent cost estimate were not provided. 
(Substantially met). 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Comprehensive: 

* Includes all life-cycle costs; 
A life-cycle cost estimate provides a complete and structured 
accounting of all resources and associated cost elements required to 
develop, produce, deploy, and sustain a particular program. It should 
cover the program from its inception through its retirement; 
Key examples of rationale for assessment: 
* Includes all life-cycle costs from fiscal year 2009 through fiscal 
year 2028 except sunk costs and disposition costs. The cost estimate 
covers both government and contractor efforts for the solution 
development, implementation, and in-service management phases of the 
program. (Partially met.) A life-cycle cost estimate should encompass 
all past, present, and future costs for every aspect of the program, 
regardless of funding source--including all government and contractor 
costs. Life-cycle cost estimates can enhance decision making by 
allowing for design trade-off studies to be evaluated on a total cost 
basis as well as on a technical and performance basis; 

* Completely defines the program, reflects the current schedule, and 
is technically reasonable; 
The cost estimate should be based on a documented technical baseline 
description, which provides a common definition of the program, 
including detailed technical, program, and schedule descriptions of 
the system; 
Key examples of rationale for assessment: 
* Reflects many detailed technical requirements, including scope and 
schedule requirements, but no single technical baseline was provided. 
Consistent with best practices, we found evidence that the technical 
baseline requirements had been updated to reflect technical, program, 
and schedule changes to the program. (Substantially met); 

* Has a product-oriented work breakdown structure, and is traceable to 
the program's technical scope, at an appropriate level of detail; 
A work breakdown structure provides a basic framework for a variety of 
related activities like estimating costs, developing schedules, 
identifying resources and potential risks, and providing the means for 
measuring program status using earned value management. It is product-
oriented if it allows a program to track cost and schedule by defined 
deliverables, such as a hardware or software component; 
Key examples of rationale for assessment: 
* Reflects FAA's standard work breakdown structure, which does not 
follow a product-oriented work breakdown structure. Instead, it breaks 
work down into functional categories. The cost estimate work breakdown 
structure also does not match either the schedule or the earned value 
management work breakdown structure; however, there was a mapping from 
the schedule work breakdown structure to the cost work breakdown 
structure. In addition, we found evidence that the work breakdown 
structure had been updated as the program became better defined. 
(Partially met.) Without a work breakdown structure, the program lacks 
a framework to develop a schedule and cost plan that can be used to 
easily track technical accomplishments. A standard, product-oriented 
work breakdown structure facilitates the tracking of resource 
allocations and expenditures, which can give the agency a basis to 
reliably estimate the cost of future similar programs; 

* Documents all cost-influencing ground rules and assumptions; 
Cost estimates are typically based on limited information and 
therefore need to be bound by ground rules and assumptions. Ground 
rules are a set of estimating standards that provide guidance and 
common definitions, while assumptions are judgments about past, 
present, or future conditions that may affect the estimate; 
Key examples of rationale for assessment: 
* The estimate documents all cost-influencing ground rules and 
assumptions. In addition, there were additional ground rules and 
assumptions found in the technical baseline's authoritative 
references. However, in several instances, not all of the data sources 
supporting the assumptions were provided. In addition, while there was 
a risk adjustment section in the documentation, the majority of the 
risk adjustments were blank, based on the assumption that the costs 
were well defined by this point. (Substantially met). 

Extent to which criterion was met: Substantially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Accurate: 

* Produces unbiased results; 
Cost estimates should have an uncertainty analysis, which determines 
where the estimate falls against the range of all possible costs; 
Is properly adjusted for inflation; 
Cost data should be adjusted for inflation to ensure that comparisons 
and projections are valid. Data should also be normalized to constant-
year dollars to remove the effects of inflation; 
Contains few mistakes; 
Results should be checked for accuracy, double counting, and omissions; 

* Is regularly updated to reflect significant program changes; 

* The cost estimate should be updated to reflect significant program 
changes, such as changes to schedules or other assumptions. Updates 
should also reflect actual costs so that the estimate always reflects 
the current program status; 

* Documents and explains variances between planned and actual costs; 
* Variances between planned and actual costs should be documented, 
explained, and reviewed. For any elements whose actual costs or 
schedules differ from the estimate, the estimate should discuss 
variances and lessons learned; 

* Reflects cost estimating experiences from comparable programs; 
The estimate should be based on a historical cost estimation data and 
actual experiences from other comparable programs. These data should 
be reliable and relevant to the new program; 

Key examples of rationale for assessment: 

* A minimal uncertainty analysis was performed for select cost 
elements with an 80 percent confidence level reported. No analysis was 
done to show the accuracy of the data sources used for this analysis. 
(Minimally met.) A cost estimate is biased if the work estimate is 
overly conservative or too optimistic. Unless the estimate is based on 
an assessment of the most likely costs and reflects the degree of 
uncertainty given all of the risks considered, management will not be 
able to make good decisions; 

* The cost estimate was adjusted for inflation correctly, and the 
source data for the inflation indexes were provided. (Met); 
* The estimate contains no mistakes, and we found no evidence of 
double counting or omissions. Additionally, the documentation, 
equations, and results were consistent between the documentation and 
the cost model. (Met); 

* There was evidence that the cost estimate was updated to reflect 
changes in technical and program requirements. For example, the WAAS 
program has undergone four rebaselines from fiscal year 1994 through 
fiscal year 2009. During each rebaseline, the cost estimate was 
updated. However, the cost model is not updated on a frequent basis. 
It is only updated to reflect major changes in technical and program 
requirements. There was no evidence that the cost estimate was updated 
with actual costs. (Partially met.) A lack of cost estimate updates 
interferes with analysis of changes in program costs and hinders 
collection of cost and technical data to support future estimates. The 
cost estimate should be updated when the technical baseline changes; 
otherwise, it will lack credibility. A properly updated cost estimate 
can provide decision makers with accurate information for assessing 
alternative decisions; 

* Earned value management system reports for November 2008 through 
February 2009 contained actual costs and variances. However, no recent 
earned value management reports captured actual costs and variances. 
In addition, because the cost estimate work breakdown structure did 
not match the earned value management work breakdown structure, 
updating the cost estimate would not be a straightforward effort. 
(Partially met.) Without a documented comparison between the current 
estimate (updated with actual costs) and the old estimate, cost 
estimators cannot determine the level of variance between the two 
estimates. That is, the estimators cannot see how well they are 
estimating and how the program is changing over time; 

* There is some evidence that the estimate was based on a historical 
record of cost estimating. For example, software source lines of code 
were estimated based on analogous historical data, but there was no 
documentation to support these costs. For the other cost elements, we 
did not find sufficient documentation of historical costs from 
comparable programs. (Minimally met) Historical data provide the cost 
estimator with insight into actual costs on similar programs, 
including any cost growth that occurred after the original estimate. 
As a result, historical data can be used to challenge optimistic 
assumptions and bring more realism to a cost estimate. 

Extent to which criterion was met: Partially met. 

Characteristic of high-quality cost estimates/assessment criterion/ 
explanation: Credible: 

* Includes a sensitivity analysis that identifies a range of possible 
costs based on varying inputs; 
A sensitivity analysis examines how changes to key assumptions and 
inputs affect the estimate. The estimate should identify key cost 
drivers, examine their parameters and assumptions, and re-estimate the 
total cost by varying each parameter between its minimum and maximum 
range; 
Key examples of rationale for assessment: 
* Minimal evidence was provided to show that a sensitivity analysis 
was provided for some cost elements, such as labor. However, we could 
not determine if key cost drivers were identified across the entire 
program or if input ranges were varied. (Minimally met.) Because 
uncertainty cannot be avoided, it is necessary to identify the cost 
elements that represent the most risk. A sensitivity analysis reveals 
how the cost estimate is affected by a change in a single assumption, 
which helps the cost estimator to understand the extent to which each 
variable affects the cost estimate. Any sources of variation should be 
well-documented and traceable; 

* A risk and uncertainty analysis; 
A risk and uncertainty analysis recognizes the potential for error and 
attempts to quantify it by identifying the effects of changing key 
cost drivers; 
Key examples of rationale for assessment: 
* The documentation discusses some aspects of a risk and uncertainty 
analysis, such as a risk adjustment; however, only a few (4 out of 40) 
of the cost elements contained detailed documentation to support the 
risk adjustments. The majority of the cost elements (36 out of 40) did 
not have any risk adjustments applied. In addition, the documentation 
did not discuss what level of confidence was associated with the cost 
estimate. (Partially met.) The program estimate must reflect the 
degree of uncertainty, so that a level of confidence can be provided 
to management about the estimate. An estimate without risk and 
uncertainty analysis is unrealistic because it does not assess the 
variability in the cost estimate from such effects as schedules 
slipping, missions changing, and proposed solutions not meeting users' 
needs; 

* Cross-checking of major cost elements; 
A cross-check is done by using a different method to see if it 
produces similar results; 
Key examples of rationale for assessment: 
* There was minimal evidence that cross-checks were identified. During 
the fiscal year 2004 rebaseline, the basis of estimate was reviewed by 
several independent staff, but there was no evidence to supporting 
specific elements being cross-checked. (Minimally met.) The main 
purpose of cross-checking is to determine whether alternative methods 
produce similar results. If so, then confidence in the estimate 
increases, leading to greater credibility; 

* A comparison to an independent cost estimate conducted by another 
organization; 
A second, independent cost estimate should be performed by an 
organization outside of the program office's influence. It should be 
based on the same technical baseline, ground rules, and assumptions, 
as the original estimate; 
Key examples of rationale for assessment: 
* There was no evidence that an independent cost estimate was 
conducted. However, as stated earlier, there was an independent review 
during the fiscal year 2004 rebaseline; since that time, the cost 
model has been reviewed several times. For example, in 2007 an 
independent evaluation analyzed eight cost issues. (Partially met.) An 
independent cost estimate is considered one of the best and most 
reliable estimate validation methods. It provides an independent view 
of expected program costs that tests the program office's estimate for 
reasonableness. Without an independent cost estimate, decisions makers 
will lack insight into a program's potential costs because independent 
cost estimates frequently use different methods and are less burdened 
with organizational bias. 

Extent to which criterion was met: Partially met. 

Source: GAO analysis of FAA's WAAS cost estimate. 

[End of table] 

[End of section] 

Appendix IV: Assessments of Four FAA Program Schedules: 

This appendix provides the results of our analysis of the extent to 
which the processes and methodologies used to develop and maintain 
four FAA integrated master schedules meet nine best practices 
associated with effective schedule estimating. The following tables 
provide the detailed results of our analyses of the schedules for the 
Automatic Dependent Surveillance-Broadcast (ADS-B), Collaborative Air 
Traffic Management Technologies (CATMT), System Wide Information 
Management (SWIM), and Wide Area Augmentation System (WAAS) programs 
compared to the nine best practices. 

"Not met" means the program provided no evidence that satisfies any 
portion of the criterion. "Minimally met" means the program provided 
evidence that satisfies a small portion of the criterion. "Partially 
met" means the program provided evidence that satisfies about half of 
the criterion. "Substantially met" means the program provided evidence 
that satisfies a large portion of the criterion. "Fully met" means the 
program provided evidence that satisfies the entire criterion. 

Table 12: Extent to Which the ADS-B Schedule Met Best Practices: 

Best practice: 1. Capturing all activities; 
Explanation: The schedule should reflect all activities as defined in 
the project's work breakdown structure, which defines in detail the 
work necessary to accomplish a project's objectives--including 
activities to be performed by both the owner and the contractors; 
Extent to which criterion was met: Partially met; 
GAO analysis: Our analysis found that detailed activities within the 
schedule are mapped to a four-level work breakdown structure, an 
organization breakdown structure, and unique control account numbers. 
Activities are further mapped to service areas, control account 
managers, and project leaders. Level-of-effort activities are included 
in the schedule and are clearly marked as such. In addition, several 
risk mitigation activities are included in the schedule. For example, 
activities associated with a contingency plan for the 2011 Mississippi 
River flood are included. 
The government owns and maintains the program schedule for the ADS-B 
program, which covers all government and nonprime contract work 
through fiscal year 2013. The schedule is planned in 6-month rolling 
wave periods. Officials stated that they have not requested a detailed 
schedule deliverable from the prime contractor because the contractor 
portion represents work performed in the fixed-price subscription 
phase. The contractor owns a separate schedule that is not a required 
deliverable to the government. The government integrated master 
schedule includes prime contractor touch points that are updated 
monthly according to the contractor's progress. However, without fully 
integrating all contractor and government activities, we cannot 
guarantee that the schedule has adequately captured all key activities 
necessary for the program's completion. 
Officials stated that work needs to be accomplished in and beyond 
fiscal year 2014, but the program schedule does not contain long-term 
planning activities beyond the current FAA-approved baseline. Effort 
from fiscal year 2015 to fiscal year 2020 is not included in the 
schedule because the program has no budget until Joint Resources 
Council approval of the next baseline. However, officials noted that 
the prime contract extends to 2025. Relevant guidance states that a 
comprehensive schedule should reflect all activities for a project and 
recognizes that there can be uncertainties and unknown factors in 
schedule estimates due to, among other things, limited data. 

Best practice: 2. Sequencing all activities; 
Explanation: The schedule should be planned so that critical project 
dates can be met. To meet this objective, activities need to be 
logically sequenced--that is, listed in the order in which they are to 
be carried out. In particular, activities that must be completed 
before other activities can begin (predecessor activities), as well as 
activities that cannot begin until other activities are completed 
(successor activities), should be identified. This helps ensure that 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be established and used as 
a basis for guiding work and measuring progress; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Our analysis found 207 remaining activities, or work 
that remains to be accomplished (19 percent), are missing predecessor 
logic and 289 remaining activities (27 percent) are missing successor 
logic. These counts include 50 remaining activities (5 percent) that 
are missing both predecessor and successor logic. 
In addition, our analysis found a number of activities with dangling 
logic. Each activity--other than the start and finish milestones--must 
have its start date driven by a previous activity, and each activity 
must drive the start date of another activity. Tasks with dangling 
logic can either carry on indefinitely without affecting downstream 
activities (finish date has no successor) or must start earlier in 
order to finish on time (start date has no predecessor). The schedule 
includes 372 remaining activities, or 34 percent, with dangling 
successors. 
The reliability of the schedule is also hindered by the presence of 
lags and leads (negative lags). 
Lags represent the passing of time between activities but are often 
misused to put activities on a specific date or to insert a buffer for 
risk. In particular, 22 percent of the remaining activities are 
affected by at least one lag or lead. Lags should be justified because 
they cannot have risk or uncertainty. Furthermore, lags often 
represent work or a delay whose duration is unknown, yet are hard-
coded into the schedule. Using lags rather than activities to account 
for work or to insert a buffer for risks causes problems because lags 
persist even when activities are delayed. As a result, when activities 
take longer than planned, the lags may actually distort the true logic 
of the schedule. Negative lags are generally not valid because 
negative time is not demonstrable; additionally, a lead implies the 
ability to predict the finish date of a predecessor activity with 
certainty. In general, leads can be replaced with straightforward 
finish-to-start logic once tasks are broken down further. 
FAA officials said that there is broken and incomplete logic for the 
detail planning period (Wave "G") that we assessed. They stated that 
the schedule we evaluated was in transition because of ongoing 
negotiations with the prime contractor over ADS-B service volume. 
Having all interdependencies between tasks identified is necessary for 
the schedule to properly calculate dates and predict changes in the 
future. Without the correct linkages, tasks that slip early in the 
schedule do not transmit delays to tasks that should depend on them. 
When this happens, the schedule will not provide a sufficient basis 
for understanding the program as a whole, and users of the schedule 
will lack confidence in the dates and the critical path. Unless 
complete network logic is established, the schedule cannot predict 
impacts on the project's planned finish date of, among other things, 
misallocated resources, delayed tasks, external events, and 
unrealistic deadlines. 

Best practice: 3. Assigning resources to all activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
materials, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
Extent to which criterion was met: Substantially met; 
GAO analysis: The ADS-B integrated master schedule is resource-loaded 
at the work package level by resource name, and contains approximately 
250 resources, including both government and nongovernment full-time-
equivalents. Each has responsibility for managing the resources at 
that level. The program office leadership teams define what work will 
be done in each control account at a high level and which resources 
are available. The control account manager is then responsible for 
assigning resources at the work package level. Once the rolling wave 
planning is approved, resources are assigned accordingly with resource 
leveling occurring outside of the schedule. Officials use the 
integrated master schedule to track all resources that are not related 
to the prime contract (approximately 70 percent of the baseline work). 
Allocation reports are run during the initial rolling wave planning 
process to ensure resources are neither underallocated nor 
overallocated, but are not assessed monthly to track resource 
allocations along with schedule progress updates. However, allocations 
are assessed whenever a change request is initiated to ensure 
resources are available for these additional activities. 

Best practice: 4. Establishing the duration of all activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from planned dates; this information can be used to determine 
whether schedule variances will affect subsequent work; 
Extent to which criterion was met: Substantially met; 
GAO analysis: Our analysis found that approximately 75 percent of the 
activities within the detail planning period of the schedule had 
original durations of 44 days or less, which generally meets best 
practices. Our analysis also found that all durations are consistently 
estimated in days and adhere to a standard 5-day workweek that 
accounts for holidays. Our analysis found no activities scheduled to 
begin on a weekend. Program officials stated that long-duration 
planning packages exist in the schedule for the period fiscal year 
2012 through fiscal year 2014 and that they hold high-level resources 
and budget meetings until officials are able to plan future rolling 
wave periods. 
Officials stated that duration estimates are based on estimates from 
responsible control account managers who follow earned value 
management guidance. However, these estimates are not formally 
justified or documented in a basis of estimate. The schedule aligns to 
an earned value management system, and the earned value technique used 
for each activity is stored in the schedule. 

Best practice: 5. Integrating the schedule horizontally and vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link products and outcomes associated with other 
sequenced activities. These links are commonly referred to as "hand-
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that the dates for starting and 
completing activities in the integrated master schedule should be 
aligned with the dates for supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Extent to which criterion was met: Partially met; 
GAO analysis: The schedule is vertically integrated, with low-level 
activities being traceable to higher-level summary activities in the 
work breakdown structure hierarchy. However, our analysis determined 
that the schedule is not horizontally integrated because of the issues 
described in Best Practice 2. Officials stated that all logic is not 
in place for the current detail planning period. Instead, the 
scheduler has used a combination of soft constraints, lags, and out-of-
sequence updates in an attempt to match as closely as possible actual 
execution of activities while the deployment phase is under 
negotiation. Horizontal integration demonstrates that the overall 
schedule is rational, planned in a logical sequence, accounts for the 
interdependencies of activities, and provides a way to evaluate 
current status. Therefore, if the schedule lacks horizontal 
integration, the effects of slipped tasks on downstream work cannot be 
determined and there will be little confidence in the calculated dates 
or critical path. 

Best practice: 6. Establishing the critical path for all activities; 
Explanation: Scheduling software should be used to identify the 
critical path, which represents the chain of dependent activities with 
the longest total duration. Establishing a project's critical path is 
necessary to examine the effects of any activity slipping along this 
path. Potential problems along or near the critical path should also 
be identified and reflected in scheduling the duration of high-risk 
activities; 
Extent to which criterion was met: Not met; 
GAO analysis: Officials stated that there is no critical path in the 
current schedule because of the incomplete logic noted in Best 
Practice 2. Officials stated that once the program began deploying 
service volumes, a critical path was difficult to derive because the 
contractor has the authority to implement service volumes in any order 
it chooses. However, now that negotiations for service volumes are 
complete, program officials plan to begin identifying a critical path 
to all Office of Management and Budget (OMB) Exhibit 300 (annual 
performance report) milestones. 
Without a valid program critical path, management cannot determine 
which slipped tasks will have detrimental effects on the project 
finish date and it cannot determine if float within the schedule can 
be used to mitigate critical tasks by reallocating resources from 
tasks that can safely slip to tasks that must be completed on time. 
Until the schedule can produce a true critical path, the program 
office will not be positioned to provide reliable timeline estimates 
and it will not be able to identify when problems or changes may occur 
and what effect they may have on downstream work. 

Best practice: 7. Identifying reasonable float; 
Explanation: The schedule should identify the float--the amount of 
time by which a predecessor activity can slip before the delay affects 
successor activities--so that a schedule's flexibility can be 
determined. As a general rule, activities along the critical path have 
the least float. Total float is the total amount of time by which an 
activity can be delayed without delaying the project's completion, if 
everything else goes according to plan; 
Extent to which criterion was met: Not met; 
GAO analysis: Officials stated that because of the missing logic in 
the current detail planning period, the float represented in the 
schedule is not realistic. Our analysis confirmed this, as we found 
many activities within the schedule with extremely high float; 
that is, unrealistically high float that is greater than 199 days. 
Specifically, there are 12 (1 percent) in-progress activities and 745 
planned activities (69 percent) that are able to slip more than 199 
days without affecting the end date of the program; 
Program officials stated they do not yet monitor float in the schedule 
because this particular version of the ADS-B schedule is in transition 
during deployment negotiations with the prime contractor. 

Best practice: 8. Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis (SRA) should be performed using 
statistical techniques to predict the level of confidence in meeting a 
project's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can affect the project's status; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Officials stated that a schedule risk analysis is not 
part of the rolling wave planning process. They perform uncertainty 
analysis on the cost estimate, but do not implement it on the 
schedule. Our analysis found this practice to be minimally met 
because, as noted in Best Practice 1, risk mitigation activities are 
entered into the schedule as baseline change requests. The program 
scheduler does not mark these as risk mitigation activities per se, 
but the documentation associated with a baseline change request will 
list the additional activities. An example of a risk mitigation 
activity is "Boxer Contingency Plan," which was added for the 2011 
flooding of the Mississippi River. 

Best practice: 9. Updating the schedule using logic and durations to 
determine dates; 
Explanation: The schedule should be continuously updated using logic 
and durations to determine realistic start and completion dates for 
program activities. The schedule should be analyzed continuously for 
variances to determine when forecasted completion dates differ from 
planned dates. This analysis is especially important for those 
variations that affect activities identified as being in a project's 
critical path and can influence a scheduled completion date; 
Extent to which criterion was met: Substantially met; 
GAO analysis: Our analysis found that the schedule has a valid and 
current status date. Our analysis also found there are no activities 
with start or finish dates in the past that are missing actual start 
and actual finish dates, and there are no activities with actual start 
and actual finish dates in the future. 
Contractor updates are received by each control account manager in the 
form of expected finish dates, so the program scheduler is aware when 
a task is running late. Because the contractor has the authority to 
deploy service volumes in any order, officials are not overly 
concerned about the effect of a late task on the program as a whole. 
They stated that the control account managers know each service volume 
negotiated deadline and are aware of the effects of the slipped tasks. 
However, too many activities slipping into the future may create a bow-
wave of effort that the contractor may not have the resources to 
complete near the expected deployment completion date in fiscal year 
2013. In addition, activities should be updated using actual duration 
or effort and estimates of remaining duration or effort. The 
scheduling software will calculate the percentage of completed 
activities and a forecasted finish date based on actual effort and 
planned effort remaining. 
Our analysis found 60 activities (6 percent) as occurring out of 
sequence. That is, activities have started before their predecessor 
was completed, contrary to the network logic. Out-of-sequence progress 
should be corrected, either by changing the logical relationships to 
reflect what has actually happened or stopping work on the successor 
activity until the predecessor activity is complete (called "retained 
logic"). FAA officials stated they expect many activities to be done 
out of sequence as they adjust to the negotiated deployment dates with 
the contractor. In future rolling waves, as logic is put in place and 
negotiated dates are firm, they expect less out-of-sequence logic. 

Source: GAO analysis of ADS-B schedule. 

[End of table] 

Table 13: Extent to Which the CATMT Schedule Met Best Practices: 

Best practice: 1. Capturing all activities; 
Explanation: The schedule should reflect all activities as defined in 
the project's work breakdown structure, which defines in detail the 
work necessary to accomplish a project's objectives--including 
activities to be performed by both the owner and the contractors; 
Extent to which criterion was met: Substantially met; 
GAO analysis: The scope of work for the CATMT project, which has no 
defined end state, is defined in 6-month increments. The schedule is 
owned and maintained by the prime contractor, who is responsible for 
capturing all activities authorized in the schedule. The schedule 
tracks government-furnished information and equipment deliverables and 
it covers all work in the statement of work. The work breakdown 
structure maps to a standardized FAA work breakdown structure but is 
functionally oriented rather than product-oriented in accordance with 
FAA standards. 
Our analysis was able to successfully track detail work activities in 
the schedule to statement of work requirements, contract line item 
numbers, and task orders. 

Best practice: 2. Sequencing all activities; 
Explanation: The schedule should be planned so that critical project 
dates can be met. To meet this objective, activities need to be 
logically sequenced--that is, listed in the order in which they are to 
be carried out. In particular, activities that must be completed 
before other activities can begin (predecessor activities), as well as 
activities that cannot begin until other activities are completed 
(successor activities), should be identified. This helps ensure that 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be established and used as 
a basis for guiding work and measuring progress; 
Extent to which criterion was met: Partially met; 
GAO analysis: For the 481 remaining activities, or work that remains 
to be completed, we found few issues affecting the sequencing logic of 
the schedule. For example, all activities in the schedule had at least 
one predecessor and one successor activity. 
However, our analysis found several issues that would affect the 
ability of the schedule to dynamically forecast future dates. For 
example, we found three Must Finish On constraints (1 percent of the 
remaining activities). A Must Finish On constraint means that the 
task, whether linked or not, must finish on a certain date. That is, 
it prevents the activity from finishing any earlier or later than a 
certain date, thereby overwriting network logic. The use of hard 
constraints is essentially the same as marking a date on a calendar 
and, therefore, defeats the purpose of using a dynamic scheduling 
tool. Officials stated that one Must Finish On constraint, the "Ready 
for Operations" milestone for Release 5, is used to create the 
critical path through Release 5, a viable reason that was not 
justified in documentation. Officials provided no justification for 
the remaining two Must Finish On constraints. We also found 24 Start 
No Earlier Than constraints (5 percent of the remaining activities). 
Start No Earlier Than constraints are considered "soft" in that they 
allow the activity to slip into the future if their predecessor 
activity is delayed. However, the activity cannot begin earlier than 
its constraint date. Officials stated that the constraints were 
justified but only provided justification for 3 of the 24 Start No 
Earlier Than constraints. For example, officials stated "Tech Refresh 
Development Tech Ops" is an obsolete task that will be deleted from 
the schedule and removed from the contract. In addition, officials 
said a Start No Earlier Than constraint was in place to prevent 
multiple level-of-effort tasks (management and other oversight related 
activities that continue until the detail activities they support have 
been completed) from being identified as critical activities. 
Activities with constraints typically are substitutes for logic and 
can mean that the schedule is not well planned and may not be feasible. 
Our analysis also found 33 tasks with lags (7 percent of the remaining 
activities), including 28 activities with positive lags and 5 
activities with leads (negative lags). According to officials, the 
lags are justified, but it is not their policy to document 
justifications. Lags should be justified because they cannot have risk 
or uncertainty. Lags often represent work or a delay whose duration is 
unknown, yet are hard-coded into the schedule. Activities represented 
by lags are not, in fact, risk free. Lags are often used to put 
activities on a specific date or to insert a buffer for risk. These 
lags persist even when activities are delayed and use up the buffer. 
When that happens, the lag may no longer be necessary. Leads are 
discouraged, as negative time is not demonstrable. Leads can often be 
replaced by additional tasks and linked using the appropriate 
scheduling logic. If used improperly, lags can distort float 
calculations in the schedule and corrupt the calculation of the 
critical path. 

Best practice: 3. Assigning resources to all activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
materials, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
Extent to which criterion was met: Substantially met; 
GAO analysis: Our analysis found 48 resources in the schedule; 
however, resources are actually tracked outside of the schedule in 
detailed labor hour spreadsheets. Officials said this way of managing 
resources meets agency criteria. Moreover, they stated that because of 
the time frame represented by the schedule, the program may have three 
software releases in progress at any one time, which results in an 
overlapping of resources. Staffing levels are reviewed during the 
monthly management review meetings to ensure resources identified by 
the contractor map to the associated activities in the schedule. 
Because resources are tracked outside the schedule rather than loaded 
within the schedule itself, management's ability to fully determine 
whether all required resources will be available when needed, and 
whether any funding or time constraints exist, is hampered. A resource-
loaded schedule implies that all required labor and significant 
materials, equipment, and other costs are assigned to the appropriate 
activities within the schedule. Assigning resources to activities 
ensures that resources are used to determine activity durations 
because resource requirements directly relate to the duration of an 
activity. 

Best practice: 4. Establishing the duration of all activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from planned dates; 
this information can be used to determine whether schedule variances 
will affect subsequent work; 
Extent to which criterion was met: Met; 
GAO analysis: Excluding level-of-effort activities, more than 90 
percent of the remaining detailed activities meet best practices for 
having durations of 44 days or less. In addition, all activities are 
assigned to standard calendars that account for holidays. 
In estimating durations, program officials stated that initially they 
used production rates for software lines of code to help determine 
durations. Now that the program is in the execution phase, officials 
have several years of history pertaining to production, industry 
standards, and defect rates on which to base duration estimates. 
Officials also said they try to keep the durations as short as 
possible, but activities such as coding are naturally 3-month-long 
tasks whose duration cannot artificially be shortened to 44 days. 

Best practice: 5. Integrating schedule activities horizontally and 
vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link products and outcomes associated with other 
sequenced activities. These links are commonly referred to as "hand-
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that the dates for starting and 
completing activities in the integrated master schedule should be 
aligned with the dates for supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Extent to which criterion was met: Partially met; 
GAO analysis: Vertical integration--that is, the ability to 
consistently trace work breakdown structure elements between detailed 
and summary master schedules--is demonstrated in the schedule because 
lower-level activities and milestones roll up into higher-level 
summary activities. In addition, the schedule shows the dependencies 
between CATMT and other NextGen programs, such as EnRoute Automation 
Modernization (ERAM) and SWIM. Officials said in order to help manage 
these dependencies on ERAM and SWIM, CATMT development leads attend 
ERAM and SWIM weekly meetings and discuss risk dependencies. In 
addition, officials stated they also track the ERAM and SWIM schedules 
and any related schedule slips. 
However, horizontal traceability is hampered by the reliance on date 
constraints and lags as discussed in Best Practice 2. For example, 
extending the durations of selected activities in the schedule more 
than 400 working days had no effect on the project finish date. If the 
schedule lacks horizontal integration, then the effects of slipped 
tasks on downstream work cannot be determined and there will be little 
confidence in the calculated dates or critical path. Any logic errors 
between detail, intermediate, and summary schedules will cause 
inconsistent dates between schedules and will cause different 
expectations between senior management and activity owners. 

Best practice: 6. Establishing the critical path for all activities; 
Explanation: Scheduling software should be used to identify the 
critical path, which represents the chain of dependent activities with 
the longest total duration. Establishing a project's critical path is 
necessary to examine the effects of any activity slipping along this 
path. Potential problems along or near the critical path should also 
be identified and reflected in scheduling the duration of high-risk 
activities; 
Extent to which criterion was met: Partially met; 
GAO analysis: Officials said because of the 6-month spiral development 
approach, the schedule cannot deliver a single critical path for the 
entire program. Instead, the critical paths are calculated and based 
on by software releases. To calculate a critical path by each software 
release, the prime contractor uses an end constraint on the key 
deliverable milestone for each software release. 
At the time of our analysis, officials stated that only work related 
to Release 5 was subject to a critical path analysis because that was 
the focus of management. We were able to trace a contiguous critical 
path for Release 5, beginning with the project status date and ending 
with the Release 5 finish milestone. However, the validity of the 
Release 5 critical path is hampered by the existence of seven in-
progress and remaining detail activities within Release 5 that have 
over 1,300 working days of total float. We also determined that a 
small number of activities outside of Release 5 were currently under 
way (for example, in other task orders). Because these activities fall 
outside of Release 5, and therefore outside the purview of a Release 5 
critical path analysis, management may not be fully aware of the 
effect of any delay on these activities. 
The critical path is directly related to the logical sequencing of 
events and float calculations. If the schedule is missing dependencies 
or if activities are not linked correctly, float estimates will be 
miscalculated. Incorrect float estimates will result in an invalid 
critical path and will hinder the ability of management to allocate 
resources from noncritical activities to those that must be completed 
on time. Without a valid critical path, there is no true program 
control because management cannot determine which slipped tasks will 
have detrimental effects on the project finish date. Until the 
schedule can produce a true critical path, the program office will not 
be positioned to provide reliable timeline estimates and it will not 
be able to identify when problems or changes may occur and what effect 
they may have on downstream work. 

Best practice: 7. Identifying reasonable float; 
Explanation: The schedule should identify the float--the amount of 
time by which a predecessor activity can slip before the delay affects 
successor activities--so that a schedule's flexibility can be 
determined. As a general rule, activities along the critical path have 
the least float. Total float is the total amount of time by which an 
activity can be delayed without delaying the project's completion, if 
everything else goes according to plan; 
Extent to which criterion was met: Minimally Met; 
GAO analysis: While officials said the schedule does not contain 
excessive float values and float values are monitored as part of the 
schedule's health assessment, our analysis found unreasonable amounts 
of total float throughout the schedule. For example, 325 (68 percent) 
of the 481 remaining activities have float values greater than 1,000 
days. These unreasonable float values are due mostly to activities 
being tied to the project finish milestone, which is constrained to 
start no earlier than July 1, 2016. Interim milestones that are 
scheduled to finish earlier, such as those marking the end of task 
orders, are tied to the project finish milestone as predecessors and 
are, therefore, showing large amounts of float that do not reflect 
actual flexibility in the schedule. The majority of high-float 
activities are level-of-effort activities, many of which are extended 
to the end of these interim milestones and are thus associated with 
unreasonable amounts of float as well. As noted in our analysis of the 
schedule's adherence to Best Practice 6, a hard constraint is in place 
to calculate a critical path through Release 5 activities; 
however, our analysis also found unreasonable amounts of total float 
in some of these activities. For example, several activities have more 
than 1,000 days of float, including "Test and Deploy"--which shows a 
total float value of 1,280 days. Because of this excessive float, 
delays in the activities above will have no effect on the finish date 
of Release 5. 
Float estimates are directly related to the logical sequencing of 
activities and, therefore, if the schedule is not properly sequenced, 
float calculations will be miscalculated. Without reliable float 
estimates, management may be unable to allocate resources from 
noncritical activities to activities that cannot slip without 
affecting the project finish date. 

Best practice: 8. Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
project's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can affect the project's status; 
Extent to which criterion was met: Not met; 
GAO analysis: A risk analysis of the schedule's vulnerability to 
slippages in the completion of activities has not been performed. 
Officials stated that a schedule risk analysis was not conducted 
because it is not a contractual deliverable. However, to address 
schedule risk with software development, the contractor performed 
cross-check and risk analyses on estimated lines of code using 
industry standard software development estimating tools. In addition, 
the officials stated they perform what-if analyses and discuss risks 
during their monthly earned value management analysis reviews. 
However, a comprehensive schedule risk analysis is an essential tool 
for decision makers. A schedule risk analysis can be used to determine 
a level of confidence in meeting the completion date or whether proper 
reserves have been incorporated into the schedule. A schedule risk 
analysis will calculate schedule reserve, which can be set aside for 
those activities identified as high risk. Without this reserve, the 
program faces the risk of delays to the scheduled completion date if 
any delays occur on critical path activities. 

Best practice: 9. Updating the schedule using logic and durations to 
determine the dates; 
Explanation: The schedule should be continuously updated using logic 
and durations to determine realistic start and completion dates for 
program activities. The schedule should be analyzed continuously for 
variances to determine when forecasted completion dates differ from 
planned dates. This analysis is especially important for those 
variations that affect activities identified as being in a project's 
critical path and can influence a scheduled completion date; 
Extent to which criterion was met: Substantially met; 
GAO analysis: The schedule is updated by the contractor weekly, and 
the contractor also provides a formal update of the schedule to the 
program office on a monthly or bimonthly basis as necessary. Officials 
stated that during the monthly schedule reviews, FAA is an active 
participant in discussions involving network logic and forecasted 
activity dates. 
We found 47 activities (10 percent of remaining activities) with start 
dates in the past or with no actual start dates and 49 activities (10 
percent of remaining activities) with finish dates in the past with no 
actual finish dates. The program office said the majority of these 
data anomalies are associated with "on-hold" activities. After 
reviewing the schedule, we did find that majority of these activities 
were labeled as "on-hold." We found no activities in the schedule with 
actual start dates in the future or actual finish dates in the future. 

Source: GAO analysis of CATMT schedule. 

[End of table] 

Table 14: Extent to Which the SWIM Schedule Met Best Practices: 

Best practice: 1. Capturing all activities; 
Explanation: The schedule should reflect all activities as defined in 
the project's work breakdown structure, which defines in detail the 
work necessary to accomplish a project's objectives--including 
activities to be performed by both the owner and the contractors; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Our analysis found that, despite a substantial amount of 
detail and clearly marked level-of-effort activities, the program 
schedule does not effectively represent the work required to complete 
SWIM, primarily because of the way in which the schedule was created 
and the limited authority of the SWIM program office. The Integrated 
Schedule is a synopsis of the individual SWIM implementation program 
(SIP) schedules and includes only activities whose timeliness is 
assumed to be crucial to meeting Joint Resources Council milestone 
dates, resulting in a schedule that likely excludes a portion of the 
program's scope. Because of missing activities, it is unlikely that 
this schedule will highlight opportunities for process improvement 
(e.g., identifying redundant activities), what-if analysis, and risk 
mitigation. 
The SWIM program office does not have purview over the content that is 
included in the individual SIP schedules. Instead, this responsibility 
is held by FAA's Joint Resources Council secretariat. Thus, the SWIM 
program office cannot enforce the use of a standard work breakdown 
structure for all SIP schedules or dictate the way in which the SIPs 
name their activities. Our analysis found repetitive naming of 
activities across SIP schedules that are not associated with specific 
products or phases. This makes communication difficult between teams, 
particularly between team members that are responsible for updating 
and integrating multiple schedules. 
If the schedule does not fully and accurately reflect the project's 
activities, it will not serve as an appropriate basis for analysis and 
may result in unreliable completion dates, time extension requests, 
and delays. Since the schedule is used for coordination, missing 
elements will hinder coordination efforts, increasing the likelihood 
of disruption and delays. In addition, if activities are missing from 
the schedule, then other best practices will not be met. Without 
accounting for all necessary activities, it is uncertain whether: all 
activities are scheduled in the correct order, resources are properly 
allocated, missing activities would appear on the critical path, or a 
schedule risk analysis accounts for all risk. 

Best practice: 2. Sequencing all activities; 
Explanation: The schedule should be planned so that critical project 
dates can be met. To meet this objective, activities need to be 
logically sequenced--that is, listed in the order in which they are to 
be carried out. In particular, activities that must be completed 
before other activities can begin (predecessor activities), as well as 
activities that cannot begin until other activities are completed 
(successor activities), should be identified. This helps ensure that 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be established and used as 
a basis for guiding work and measuring progress; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Our analysis found that the activities in the SWIM 
Integrated Schedule were not properly sequenced because of missing 
dependencies and the presence of dangling activities, date 
constraints, and lags. Dangling logic reduces the credibility of the 
calculated activity start and finish dates and the identity of the 
critical paths because the slip or elongation of an activity that has 
no logical successor will not affect the scheduled dates of successor 
activities. 
In addition, the schedule contains 23 constraints (6 percent of the 
remaining activities), or work that remains to be completed, including 
13 soft constraints and 10 hard constraints. Program officials stated 
that hard constraints were added to enable the creation of the 
critical path for the subprogram. However, officials stated that float 
is not monitored. If float is not monitored, the critical path may be 
invalid, and there is thus no credible reason to include these 
constraints in the schedule. 
Constraints jeopardize the ability of the schedule to remain dynamic--
that is, to compute the correct delivery dates and critical path if 
durations change. Constraints artificially set dates in the software 
and may mislead users of the schedule, since they do not force the 
actual project to do anything. A large number of activities with 
constraints typically are substitutes for logic and can mean that the 
schedule is not well planned and may not be feasible. 
The quality of the schedule is significantly degraded by the presence 
of lags and leads (negative lags). In particular, 61 percent of the 
remaining activities are affected by at least one lag or lead. Program 
officials explained that lags are used to represent collections of 
activities that were present in the original SIP schedules but deemed 
noncrucial during the Integrated Schedule creation process. The 
purpose of these lags is then to preserve the underlying network logic 
and timeline. However, in this case, a summary activity would be a 
more appropriate modeling choice, since it would allow for logic 
preservation, but also maintain the dynamic nature of the schedule. 
Using lags rather than activities to account for work or to insert a 
buffer for risks causes problems because lags persist even when 
activities are delayed. For this reason, lags should be eliminated and 
replaced with activities so they can be tracked. Furthermore, lags 
lessen the ability of the project schedule to dynamically respond to 
changes in the status of predecessor activities, and the lags' need to 
be constantly manually updated defeats the purpose of a dynamic 
schedule and can cause errors in the schedule when there are too many 
lags to maintain. 

Best practice: 3. Assigning resources to all activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
materials, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
Extent to which criterion was met: Not met; 
GAO analysis: The schedule is not resource-loaded. According to 
program officials, the SWIM program is somewhat atypical with respect 
to resource loading. In particular, the substantive effort for the 
program is handled by SIP support contractors and the SWIM program 
office serves to monitor the SIPs' progress and ensure that 
deliverables are completed on time. Moreover, in accordance with the 
program's federated approach, the SWIM program office does not dictate 
the content that is included in the individual SIP schedules. As a 
result, the SIPs are not required to resource-load their schedules 
and, thus, the Integrated Schedule itself is not resource loaded; 
Because this schedule does not have resource assignments, it cannot be 
used to monitor productivity or resource-constrained activities, 
allocate idle resources, or level resources across activities. 

Best practice: 4. Establishing the duration of all activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from planned dates; this information can be used to determine 
whether schedule variances will affect subsequent work; 
Extent to which criterion was met: Partially met; 
GAO analysis: The requirements for compliance with this best practice 
have been partially met, primarily because of the nature of the 
schedule creation and update processes. Durations for many of the 
activities in the schedule are significantly longer than 2 working 
months (44 working days). Specifically, 190 (55 percent) of the 
remaining activities have a duration longer than the threshold of 44 
days, and half of the activities are scheduled for longer than 66 
days. Furthermore, the average duration of all remaining activities is 
158 days, or roughly 7 working months. Such a schedule is likely to be 
very challenging to manage and is indicative of insufficient detail. 
Long-duration activities also make it difficult for management to 
gauge progress and can skew schedule risk analysis results. 
Officials stated that many of these lengthy durations are the result 
of the way in which the schedule was created. Those activities that 
are not level-of-effort were originally summary activities in the SIP 
schedule from which they were extracted and, thus, are composed of 
many detailed activities with shorter durations. 
In addition, because the method by which activity durations are 
estimated is not within the purview of the SWIM program office, there 
is little insight into how the durations are derived. However, as 
noted in the discussion of Best Practice 3, there is no requirement 
for the SIP contractor schedules to be resource-loaded. Because 
activity durations are based on many underlying assumptions, including 
the number of hours per workweek, number of full-time-equivalent 
personnel, productivity rates, and other factors, they should be 
estimated under normal conditions, not optimal conditions. If 
durations are not realistic, program management does not have the 
information necessary to properly allocate resources to tasks and 
understand how those tasks affect downstream work. 

Best practice: 5. Integrating the schedule horizontally and vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link products and outcomes associated with other 
sequenced activities. These links are commonly referred to as "hand-
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that the dates for starting and 
completing activities in the integrated master schedule should be 
aligned with the dates for supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Extent to which criterion was met: Minimally met; 
GAO analysis: This schedule suffers from a lack of horizontal 
integration for reasons identified in other Best Practice responses. 
The schedule failed several tests for horizontal integration, 
primarily because of the hard constraints on the SIP completion 
milestones. For example, the duration of a critical activity was 
increased by 50 days, changing its float (and that of its predecessor) 
from 0 to 50 days and its finish date from February 28, 2011, to May 
9, 2011, but without any effect on the project completion date. 
Further increasing this activity's duration by 1,500 days had the 
expected effect on the finish dates of the activity and its successor, 
but left the project's completion date unchanged and several years 
before the new finish date of the critical activity. Horizontal 
integration demonstrates that the overall schedule is rational, 
planned in a logical sequence, accounts for interdependent activities, 
and provides a way to evaluate current status. Therefore, if the 
schedule lacks horizontal integration, the effects of slipped tasks on 
subsequent work cannot be determined and there will be little 
confidence in the calculated dates or critical path. 
Schedules that are not horizontally integrated may not depict 
relationships between different program elements and product hand-
offs. When this happens, product hand-offs of project subcomponents 
cannot be fully traced to the end product, leading to less effective 
management of the project. According to officials, all parties 
involved in the creation and updating of the SWIM program office 
schedules meet routinely for status reviews and have signed off on all 
"touch points" and hand-offs. However, because of the structure of the 
schedule, it is difficult to determine where key hand-offs are made 
and how they are related to top-level milestone dates; 
As noted in the discussion of Best Practice 1, there are several 
inconsistencies between lower-and upper-level milestone dates. In 
addition, this schedule does not contain all lower-level activities 
but, rather, only a selection of activities from the lower-level SIP 
schedules. Any logic errors between detail, intermediate, and summary 
schedules will cause inconsistent dates between schedules, as well as 
different expectations between senior management and activity owners. 

Best practice: 6. Establishing the critical path for all activities; 
Explanation: Scheduling software should be used to identify the 
critical path, which represents the chain of dependent activities with 
the longest total duration. Establishing a project's critical path is 
necessary to examine the effects of any activity slipping along this 
path. Potential problems along or near the critical path should also 
be identified and reflected in scheduling the duration of high-risk 
activities; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Based on the description of critical path management 
practices provided by the SWIM program office, we analyzed this best 
practice from the traditional, top-level perspective, as well as from 
that of the individual SIPs. In both cases, we found that the 
requirements for compliance were only minimally met. For each analysis 
that we performed, we traced the paths of activities with the least 
amount of total float since they are considered "drivers" of a 
critical finish milestone. 
Program officials stated that, although the SIP schedules are 
integrated in the schedule, the SIPs are unrelated and, thus, each has 
its own critical path. However, the full individual critical paths are 
not accessible through the Integrated Schedule, and the SIP schedules 
are not used by the SWIM program office for management purposes, so 
their existence offers little advantage from a SWIM management 
perspective. Furthermore, the SWIM program itself should have its own 
critical path, which includes, at the very least, acceptance of all 
major deliverables from the SIPs. 
We first examined the SWIM-wide critical path identified by the 
scheduling software. This critical path was found to be invalid for a 
variety of reasons, including the inclusion of line-of-effort 
activities, date constraints, activities with excessive durations, and 
lags and leads. Additionally, issues identified during the analysis of 
other best practices--particularly Best Practices 1 and 2--prevent 
this critical path from being valid. Without a valid program critical 
path, management cannot determine which slipped tasks will have 
detrimental effects on the project finish date; 
We also analyzed the schedule from the perspective of each SIP to 
determine the validity of individual critical paths. In each case, the 
critical path failed to meet the requirements for this best practice 
because of the presence of one or more of the following: line-of-
effort activities, lags, leads, long durations, program management 
activities, and excessive float. 

Best practice: 7. Identifying reasonable float; 
Explanation: The schedule should identify the float--the amount of 
time by which a predecessor activity can slip before the delay affects 
successor activities--so that a schedule's flexibility can be 
determined. As a general rule, activities along the critical path have 
the least float. Total float is the total amount of time by which an 
activity can be delayed without delaying the project's completion, if 
everything else goes according to plan; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Program officials noted that total float is within 
acceptable limits for the project but that float is not monitored by 
the SWIM program office. However, our analysis found unreasonable 
float calculations in the schedule. In particular, 107 (28 percent) of 
the 393 remaining detail and milestone activities have float values 
greater than or equal to 100 working days; that is, 27 percent of all 
remaining activities can slip more than 4 working months without 
affecting successor activities. Additionally, two activities with 
excessive negative float values (95 working days) were discovered. 
The abnormal float values are primarily caused by the schedule 
integration process. In particular, these high values arise because of 
large differences between the finish dates of individual SIP 
capabilities and the SWIM program completion date. Program officials 
noted that adding links between the completion dates of the individual 
SIP capabilities and the project completion milestone (in order to 
lower the float values) is unnecessary since float management is not 
used by the SWIM program office. Officials stated that total float is 
not monitored because the schedule is not resource-loaded and the work 
is performed at the SIP level. In addition to the integration process, 
other possible causes of excessive float are the missing dependencies 
and constraints noted in the discussion of Best Practice 2. 
Float is directly related to the logical sequencing of events and the 
critical path. If the schedule is missing activities or dependencies, 
or links activities incorrectly, float estimates will not be accurate. 
Incorrect float estimates may result in an invalid critical path and 
an inaccurate assessment of project completion dates. In addition, 
inaccurate values of total float result in a false depiction of true 
project status, which could lead to decisions that may jeopardize the 
project. 

Best practice: 8. Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
project's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can affect the project's status; 
Extent to which criterion was met: Minimally met; 
GAO analysis: No schedule risk analysis was conducted for this 
schedule. However, our analysis found that this best practice was 
minimally met because a risk analysis was conducted on a separate 
higher-level SWIM schedule and risk management has been considered, to 
some extent, by the SWIM program office. 
The schedule risk analysis was performed on a high-level schedule, 
consisting of standard work packages designed to represent a generic 
software development program, in order to satisfy Joint Resources 
Council requirements. According to the SWIM program office, the SWIM 
Joint Resources Council met with each of the SIP program offices to 
develop high-level schedules that were risk adjusted and priced for 
the original cost estimate. These individual schedules were combined, 
resulting in a schedule with 182 activities. This risk analysis does 
not comply with this best practice because the underlying schedule was 
created using standard work packages, not specific to SWIM, and does 
not accurately capture the relationships between the SIP capabilities 
and the success of the SWIM program. Furthermore, even if a risk 
analysis had been performed on the Integrated Schedule, several issues 
have been identified that would call its validity into question. If 
the schedule risk analysis is to be credible, the program must have a 
quality schedule that reflects reliable logic and clearly identifies 
the critical path--conditions that the SWIM program's Integrated 
Schedule does not meet. 
Program officials have stated that it has not been practice to perform 
risk analysis on the Integrated Schedule because it is a derivative of 
several lower-level schedules and does not capture the work of the 
original program. However, a schedule risk analysis should be 
conducted on each schedule used for planning purposes, in order to 
provide a more realistic view of the expected project timeline. 

Best practice: 9. Updating the schedule using logic and durations to 
determine dates; 
Explanation: The schedule should be continuously updated using logic 
and durations to determine realistic start and completion dates for 
program activities. The schedule should be analyzed continuously for 
variances to determine when forecasted completion dates differ from 
planned dates. This analysis is especially important for those 
variations that affect activities identified as being in a project's 
critical path and can influence a scheduled completion date; 
Extent to which criterion was met: Partially met; 
GAO analysis: Our analysis found that the requirements for compliance 
with this best practice have been partially met. According to program 
officials, the schedule is updated regularly during monthly status 
meetings. This has been partially confirmed by the latest schedule 
status on January 31, 2011, shortly before the program office 
transmitted it to us. Despite the regularity of these meetings, 
however, the update process described by program officials is quite 
labor-intensive and likely prone to error. 
We discovered evidence of the inherent error in this process through 
our analysis. Although the schedule was recently updated, there were 
32 activities with start dates in the past but without a recorded 
actual start date, and 46 activities with finish dates in the past but 
without an actual finish date entered in the schedule. If these 
activities have started (finished), their actual start (finish) date 
should be updated to reflect this accomplishment. If not, the start 
(finish) date should be updated to reflect the current plan. 
Additionally, our analysis identified 28 activities in the schedule 
with finish dates that occurred before the status date, 58 activities 
with remaining durations exceeding their planned finish date, and 21 
activities updated out of sequence. The existence of these date 
anomalies indicates error in the duration-estimating methodology or 
neglect in the schedule-updating process, and a schedule with 
remaining out-of-sequence progress may have the wrong logic in place 
and hence have inaccurate critical paths and completion dates. 

Source: GAO analysis of SWIM schedule. 

[End of table] 

Table 15: Extent to Which the FAA-Prepared WAAS Schedule Met Best 
Practices: 

Best practice: 1. Capturing all activities; 
Explanation: The schedule should reflect all activities as defined in 
the project's work breakdown structure, which defines in detail the 
work necessary to accomplish a project's objectives--including 
activities to be performed by both the owner and the contractors; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Our analysis found that the program schedule is not 
fully integrated. Program officials said the government's integrated 
master schedule, which is maintained by FAA, is made up of 15 
subschedules; however, these subschedules are not linked to the 
government's integrated master schedule. Furthermore, identification 
of these subschedules in the government's integrated master schedule 
is difficult because that schedule does not contain any custom text 
fields, statements of work, or contract line Item mappings. Without 
these mappings, it is impossible to determine which activities in the 
government's integrated master schedule belong to which subschedules 
and whether all the work has been captured. 
In addition, we found that the work breakdown structure for the 
government's integrated master schedule does not match the work 
breakdown structure used for the cost estimate or the work breakdown 
structure used in the program's Earned Value Management System Control 
Plan. Because the work breakdown structure is not standardized, we 
cannot be sure that the schedule contains all of the work necessary to 
accomplish the program's objectives, and it is difficult, if not 
impossible, to identify activities across cost, schedule, and earned 
value management reporting. 
Without an integrated master schedule that accounts for all planned 
government and contractor effort, management is not able to reliably 
estimate planned dates beyond the current schedule's end date of 
September 2016. Moreover, unless all activities are accounted for, it 
is uncertain whether all activities are properly sequenced, resources 
are properly assigned, the critical path reflects all activities, or a 
schedule risk analysis accounts for all risk. Finally, if the schedule 
does not fully and accurately reflect the project's activities, it 
will not serve as an appropriate basis for analysis and may result in 
unreliable completion dates, time extension requests, and delays. 

Best practice: 2. Sequencing all activities; 
Explanation: The schedule should be planned so that critical project 
dates can be met. To meet this objective, activities need to be 
logically sequenced--that is, listed in the order in which they are to 
be carried out. In particular, activities that must be completed 
before other activities can begin (predecessor activities), as well as 
activities that cannot begin until other activities are completed 
(successor activities), should be identified. This helps ensure that 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be established and used as 
a basis for guiding work and measuring progress; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Our analysis found 116 remaining activities--or 
remaining work to be completed. (36 percent) missing predecessor logic 
and 162 activities (48 percent) missing successor logic. In addition, 
our analysis found a number of activities with constraints. The 
schedule includes 198 activities (49 percent) with Start No Earlier 
Than constraints. These constraints are considered "soft" in that they 
allow the activity to slip into the future if their predecessor 
activity is delayed. However, the activity cannot begin earlier than 
its constraint date. We also found 92 activities (23 percent) with 
Finish No Earlier Than constraints. These constraints are also 
considered "soft" date constraints because they prevent activities 
from finishing earlier than their constraint date. 
The reliability of the schedule is also hindered by the presence of 
lags and leads (negative lags). In particular, 38 activities (9 
percent) are affected by at least one lag or lead. Lags should be 
justified because they cannot have risk or uncertainty. Furthermore, 
lags often represent work or a delay whose duration is unknown, yet 
are hard-coded into the schedule. Using lags rather than activities to 
account for work or to inset a buffer for risks causes problems 
because lags persist even when activities are delayed. As a result, 
when activities take longer, the lags may actually distort the true 
logic of the schedule. Negative lags are generally not valid because 
time is not demonstrable; moreover, a lead implies the ability to 
predict the finish date of a predecessor activity with certainty. In 
general, leads can be replaced with straightforward finish-to-start 
logic once tasks are broken down further. 
Having all interdependencies between tasks identified is necessary for 
the schedule to properly calculate dates and predict changes. Without 
the right linkages, tasks that slip early in the schedule do not 
transmit delays to tasks that should depend on them. When this 
happens, the schedule will not provide a sufficient basis for 
understanding the program as a whole, and users of the schedule will 
lack confidence in the dates and the critical path. Unless complete 
network logic is established, the schedule cannot predict effects on 
the project's planned finish date of, among other things, misallocated 
resources, delayed tasks, external events, and unrealistic deadlines. 

Best practice: 3. Assigning resources to all activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
materials, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Program officials stated that the schedule is not 
resource-loaded, and our analysis confirmed that resources are not 
assigned to activities. Program officials said the general resource 
allocation process begins with the 5-year Strategic Plan that lays out 
the program's long-term goals. 
Program officials said given the current political and budget 
environment, it is very difficult to hire government employees, and, 
as a result, it is easier to fill resource gaps with contractor 
personnel than with government personnel.
Assigning resources to activities ensures that resources are used to 
determine activity durations because resource requirements directly 
relate to the duration of an activity. Labor, material, equipment, and 
funding requirements are examined to determine the feasibility of the 
schedule, so that resources provide a benchmark of the total and per-
period cost of the project. If the current schedule does not allow for 
insight into current or projected allocation of resources, then the 
risk of the program slipping is significantly increased. 

Best practice: 4. Establishing the duration of all activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from planned dates; this information can be used to determine 
whether schedule variances will affect subsequent work; 
Extent to which criterion was met: Partially met; 
GAO analysis: Our analysis found that more than 70 percent of the 
remaining activities within the detail planning period of the schedule 
had original durations of 44 days or less, which generally meets best 
practices. Forty-one activities had long durations, of between 200 and 
630 days that appear to be based on a fiscal year's period of work, 
without regard to deliverables either at the end or at an intermediate 
point of shorter duration. Usually activities in the schedule are 
organized around producing deliverables rather than on the end of the 
calendar or fiscal year, or of a 12-month period. 
Program officials said the team uses past experiences to estimate 
activity durations. Officials also said durations are determined in 
meetings between program engineers, control account managers, and 
upper management. We found no deadlines in the schedule, and program 
officials noted that there is no mandated date for the schedule. 
Although the activity durations in the program's schedule appear to be 
reasonable, not assigning resources in the schedule directly affects 
the estimated work and duration of activities. Assigning resources to 
activities inside the schedule ensures that resources are used to 
determine activity durations because resource requirements directly 
relate to the duration of an activity. Therefore, because the schedule 
does not include adequate resources, it is questionable that the 
activity durations realistically reflect how long each activity will 
take to execute. Furthermore, only one activity had a calendar 
assigned to it. All of the other activities had no calendar 
assignment, which means that holidays were not accounted for in this 
schedule, immediately calling into question the validity of the 
calculated dates. 

Best practice: 5. Integrating schedule activities horizontally and 
vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link products and outcomes associated with other 
sequenced activities. These links are commonly referred to as "hand-
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that the dates for starting and 
completing activities in the integrated master schedule should be 
aligned with the dates for supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Extent to which criterion was met: Partially met; 
GAO analysis: The results for Best Practice 2, Sequencing of All 
Activities, and Best Practice 7: Identifying Reasonable Float, on 
activities and paths indicate that the logic is not in place to 
demonstrate vertical integration in the integrated master schedule; 
The WAAS program office said vertical integration is demonstrated in 
the contractor schedules, because the lower-level schedules map to 
milestones in the integrated master schedule. However, we found that 
the dates for two key milestones in the detailed contractor schedule 
did not match the dates in the government's integrated master 
schedule, indicating a problem with vertical integration among the 
detailed and higher-level schedules. 
Program officials said horizontal integration is identified in the 
schedule when activities are handed off from givers to receivers, but 
our analysis could not identify these hand-offs in the schedule. In 
addition, the schedule does not fully demonstrate horizontal 
traceability because it does not include complete logic from program 
start to program finish and does not fully integrate the entire scope 
of work for all parties involved in the program. 
Program officials said the WAAS program uses both the acquisition 
program baseline and the Office of Management and Budget (OMB) Exhibit 
300 to track the program.[A] However, our analysis found that the WAAS 
integrated master schedule only reflects work approved through 2016, 
even though the acquisition program baseline and OMB Exhibit 300 have 
estimated work planned beyond that date. Without horizontal 
traceability, there is no assurance that the forecasted dates within 
the schedule will be determined by network logic and progress to date 
rather than artificial constraints. 
We also tested horizontal integrity by extending the durations of key 
activities for Release 3B, the current release during the GAO audit, 
to observe the effects on successor activity dates and the project 
finish date. Extending the durations of selected activities in the 
schedule by more than 1,000 working days had no effect on the 
successor activities or the project finish date. 
Unless a schedule is fully horizontally integrated, the effects of 
slipped tasks on succeeding work cannot be determined. Horizontal 
integration demonstrates that an overall schedule is rational, planned 
in a logical sequence, accounts for interdependencies between work and 
planning packages, and provides a way to evaluate a program's current 
status. When schedules are not horizontally integrated, relationships 
between different program teams cannot be seen and product hand-offs 
cannot be identified. Vertical integration traces the consistency of 
data between work breakdown structure elements within the layers of 
the schedule--master, intermediate, detailed. Without schedules that 
are vertically integrated, lower-level schedules cannot be clearly 
traced to upper-tiered milestones, and, as a result, total schedule 
integrity is compromised. This can hamper the ability of different 
teams to use the schedule to manage expectations. 

Best practice: 6. Establishing the critical path for all activities; 
Explanation: Scheduling software should be used to identify the 
critical path, which represents the chain of dependent activities with 
the longest total duration. Establishing a project's critical path is 
necessary to examine the effects of any activity slipping along this 
path. Potential problems along or near the critical path should also 
be identified and reflected in scheduling the duration of high-risk 
activities; 
Extent to which criterion was met: Minimally met; 
GAO analysis: Officials stated that there is no critical path in the 
current schedule, but they know what the critical activities are in 
the subschedules that make up the integrated master schedule. As 
evidence, they pointed out hard-copy versions of five detail schedule 
critical paths for the subschedules, including those for the safety 
computer. However, we were unable to find a valid "critical path" in 
the schedule itself. Both critical paths in the schedule start with a 
Start No Earlier Than constraint, which is not in line with best 
practices. Moreover, one of the paths includes several activities with 
lags that equate to 39 months (approximately 854 working days) of 
unexplained duration. 
Without a valid program critical path, management cannot determine 
which slipped tasks will have detrimental effects on the project 
finish date and whether float within the schedule can be used to 
mitigate delays in critical tasks by reallocating resources from tasks 
that can safely slip to tasks that must be completed on time. Until 
the schedule can produce a true critical path, the program office will 
not be positioned to provide reliable timeline estimates and it will 
not be able to identify when problems or changes may occur and how 
they may affect downstream work. 

Best practice: 7. Identifying reasonable float; 
Explanation: The schedule should identify the float--the amount of 
time by which a predecessor activity can slip before the delay affects 
successor activities--so that a schedule's flexibility can be 
determined. As a general rule, activities along the critical path have 
the least float. Total float is the total amount of time by which an 
activity can be delayed without delaying the project's completion if 
everything else goes according to plan; 
Extent to which criterion was met: Partially met; 
GAO analysis: Our analysis found excessive float values. We found 238 
activities (58 percent) with float values greater than 1,000 working 
days, or 46 months. Our analysis also found 160 activities (39 
percent) with float values greater than 148 working days, or 7 months. 
Float estimates are directly related to the logical sequencing of 
activities. Therefore, if the schedule is not properly sequenced, 
float estimates will be miscalculated. Without reliable float 
estimates, management may be unable to allocate resources from 
noncritical activities to activities that cannot slip without 
affecting the project finish date. Furthermore, because the critical 
path is directly related to the logical sequencing of events and float 
calculations, if the schedule is missing dependencies or if activities 
are incorrectly linked, float estimates will be miscalculated, 
resulting in an invalid critical path. 

Best practice: 8. Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
project's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can affect the project's status; 
Extent to which criterion was met: Not met; 
GAO analysis: Officials stated that a schedule risk analysis had not 
been performed on the integrated master schedule. Program officials 
said that although they do not have a WAAS contingency schedule, they 
do manage risk with a risk register with mitigation plans and that the 
integrated master schedule is also risk-adjusted. Our analysis found 
no evidence of risk mitigation activities in the schedule. 
Performing a schedule risk analysis on the integrated master schedule, 
using statistical analysis, would allow the program office to model 
the effects of work not yet on contract. One way to estimate the 
effects of the scope of work not on contract is through the use of a 
Monte Carlo simulation (used to approximate the probability outcomes 
of multiple trials by; generating random numbers) to estimate the 
durations and start dates. This would give the program office some 
visibility into risk impacts (understanding of the effects of risk). 
However, for a schedule risk analysis to be credible, the program must 
have a good schedule network that clearly identifies the critical path 
and represents all work in the schedule, neither of which is 
demonstrated in the program's schedule. 
A schedule risk analysis will calculate schedule reserve, which can be 
set aside for those activities identified as high risk. Without this 
reserve, the program's scheduled completion date faces the risk of 
delays, if any delays were to occur on critical path activities. 
However, if the schedule risk analysis is to be credible, the program 
must have a quality schedule that reflects reliable logic and clearly 
identifies the critical path--conditions that the schedule does not 
meet. 

Best practice: 9. Updating the schedule using logic and durations to 
determine the dates; 
Explanation: The schedule should be continuously updated using logic 
and durations to determine realistic start and completion dates for 
program activities. The schedule should be analyzed continuously for 
variances to determine when forecasted completion dates differ from 
planned dates. This analysis is especially important for those 
variations that affect activities identified as being in a project's 
critical path and can influence a scheduled completion date; 
Extent to which criterion was met: Partially Met; 
GAO analysis: Program officials said the status of the schedule is 
updated monthly by incorporating actual effort achieved by the 
contractors and the support team members. Whether the schedule is 
properly updated is questionable. During a January 2011 meeting with 
the program office, officials said they did not believe the schedule 
status date was correct and, for the purposes of the audit, GAO should 
use the date in the filename. The status date is an input into the 
calculations used to update and schedule remaining work. Neither the 
date on which someone is viewing the schedule nor the latest save date 
of the file should be used as a substitute for a valid status date. 
Program officials said they have hired a dedicated scheduler who is 
responsible for updating the schedule. 
Our analysis also found several data anomalies. For example, the 
schedule includes 19 remaining activities (5 percent) with start dates 
in the past with no recorded actual start dates; 17 remaining 
activities (4 percent) with finish dates in the past with no recorded 
actual finish; 15 remaining activities (4 percent) with actual start 
dates in the future; and 12 remaining activities (3 percent) with 
actual finish dates in the future. 
Despite the program office's efforts to keep the schedule updated, the 
integrated master schedule does not capture all activities and the 
schedule does not contain the complete network logic between all 
activities necessary for the schedule to correctly forecast the start 
and end dates of activities within the plan. As such, it is 
questionable that the schedule reflects the program's true status. 
As a best practice, the schedule should be continually monitored to 
determine when forecasted completion dates differ from their planned 
dates. This information can be used to determine whether schedule 
variances will affect downstream work. Maintaining the integrity of 
the schedule logic is not only necessary to reflect the program's true 
status, but is also required before conducting a schedule risk 
analysis. Without a documented, consistently applied schedule change 
control process, program managers can continually revise the schedule 
to match performance. Such adjustments hinder the project manager's 
understanding of the project's true performance. Good documentation 
helps with analyzing changes in the program schedule and identifying 
the reasons for variances between estimates and actual results, 
thereby contributing to the collection of cost, schedule, and 
technical data that can be used to support future estimates. 

Source: GAO analysis of WAAS schedule. 

[A] An Exhibit 300--also called a Capital Asset Plan and Business 
Case--is used to justify resource requests for major investments and 
is intended to enable an agency to demonstrate to its own management, 
as well as to OMB, that a major project is well planned. 

[End of table] 

Table 16: Extent to Which the WAAS Contractor-Prepared Schedule Met 
Best Practices: 

Best practice: 1. Capturing all activities; 
Explanation: The schedule should reflect all activities as defined in 
the project's work breakdown structure, which defines in detail the 
work necessary to accomplish a project's objectives--including 
activities to be performed by both the owner and the contractors. 
Extent to which criterion was met: Substantially met; 
GAO analysis: Our analysis found that the contractor's schedule is 
governed by a statement of work and the contract with FAA. Detailed 
activities are cross-referenced to elements in the project's work 
breakdown structure that map to the task order structure in the 
schedule using custom text fields. Because the contractor did not 
provide a work breakdown structure, although we found custom text 
fields in the schedule, we cannot tell without a work breakdown 
structure whether all the work is included in the schedule. 

Best practice: 2. Sequencing all activities; 
Explanation: The schedule should be planned so that critical project 
dates can be met. To meet this objective, activities need to be 
logically sequenced--that is, listed in the order in which they are to 
be carried out. In particular, activities that must be completed 
before other activities can begin (predecessor activities), as well as 
activities that cannot begin until other activities are completed 
(successor activities), should be identified. This helps ensure that 
interdependencies among activities that collectively lead to the 
accomplishment of events or milestones can be established and used as 
a basis for guiding work and measuring progress. 
Extent to which criterion was met: Substantially met; 
GAO analysis: Our analysis found 6 remaining activities (1 percent) 
missing predecessor or successor logic. In addition, our analysis 
found 6 remaining activities (1 percent) with dangling logic where the 
start of the activity had no predecessor (only F-F predecessors). Each 
activity--other than the start and finish milestones--must have its 
start date driven by a previous activity, and each activity must drive 
the start date of another activity. Tasks with dangling logic can 
either carry on indefinitely without affecting downstream activities 
(finish date has no successor) or must start earlier in order to 
finish on time (start date has no predecessor). Dangling activities 
are those activities with logic that do not give the right activity 
start and finish dates when durations change. 
Though we found just a few missing dependencies and dangling 
activities, we identified 46 activities (7 percent) that are affected 
by at least one lag or lead. Lags should be justified because they 
cannot have risk or uncertainty. They often represent work or a delay 
whose duration is unknown, yet are hard-coded into the schedule. 
Negative lags--also known as leads--are discouraged because negative 
time is not demonstrable. Leads can often be replaced by additional 
tasks and linked using the appropriate F-S logic. 

Best practice: 3. Assigning resources to all activities; 
Explanation: The schedule should reflect what resources (e.g., labor, 
materials, and overhead) are needed to do the work, whether all 
required resources will be available when needed, and whether any 
funding or time constraints exist. 
Extent to which criterion was met: Partially met; 
GAO analysis: Program officials stated that the schedule is not 
resource-loaded and our analysis confirmed that resources are not 
assigned to activities. Program officials said resources are managed 
via their earned value management system in which resources are 
specified by resource group and skill types and are tied to the 
schedule by the work breakdown structure. 
Although the contractor may be conducting this analysis of resources 
via its earned value management system or its critical chain process, 
it did not provide any supporting documentation to validate this 
assertion. 
Assigning resources to activities ensures that resources are used to 
determine activity durations because resource requirements directly 
relate to the duration of an activity. Labor, material, equipment, and 
funding requirements are examined to determine the feasibility of the 
schedule, so that resources provide a benchmark of the total and per-
period cost of the project. If the current schedule does not allow for 
insight into current or projected overallocation of resources, then 
the risk of the program slipping is significantly increased. 

Best practice: 4. Establishing the duration of all activities; 
Explanation: The schedule should realistically reflect how long each 
activity will take to execute. In determining the duration of each 
activity, the same rationale, historical data, and assumptions used 
for cost estimating should be used. Durations should be as short as 
possible and have specific start and end dates. The schedule should be 
continually monitored to determine when forecasted completion dates 
differ from planned dates; this information can be used to determine 
whether schedule variances will affect subsequent work. 
Extent to which criterion was met: Substantially Met; 
GAO analysis: Our analysis found that more than half of the 154 
remaining detailed activities (107 or 69 percent) meet best practices 
for durations being less than 44 days (or 2 working months). Our 
analysis found that all durations are consistently estimated in days 
and adhere to a standard 5-day workweek that accounts for holidays or 
uses calendars that account for FAA moratoriums. Our analysis found no 
activities scheduled to begin on a weekend. Program officials said 
duration estimates for the schedule are based on historical 
information from past performance, comparable releases, lessons 
learned, similar work, and other data requirements. 

Best practice: 5. Integrating schedule activities horizontally and 
vertically; 
Explanation: The schedule should be horizontally integrated, meaning 
that it should link products and outcomes associated with other 
sequenced activities. These links are commonly referred to as "hand-
offs" and serve to verify that activities are arranged in the right 
order to achieve aggregated products or outcomes. The schedule should 
also be vertically integrated, meaning that the dates for starting and 
completing activities in the integrated master schedule should be 
aligned with the dates for supporting tasks and subtasks. Such mapping 
or alignment among levels enables different groups to work to the same 
master schedule; 
Extent to which criterion was met: Substantially met; 
GAO analysis: FAA program officials said vertical integration is 
demonstrated in the contractor's schedule because the lower-level 
schedules map to milestones in the FAA integrated master schedule. 
However, we found that dates for the same milestone activities in the 
contractor and FAA schedule did not match. Horizontal integration is 
also hampered in the schedule. For example, extending the durations of 
selected activities in the schedule more than 500 working days had no 
effect on the project finish date. 
Unless the schedule is fully horizontally integrated, the effects of 
slipped tasks on succeeding work cannot be determined. Horizontal 
integration demonstrates that the overall schedule is rational, 
planned in a logical sequence, accounts for interdependencies between 
work and planning packages, and provides a way to evaluate current 
status. When schedules are not horizontally integrated, relationships 
between different program teams cannot be seen and product hand-offs 
cannot be identified. Vertical integration traces the consistency of 
data between work breakdown structure elements within the layers of 
the schedule--master, intermediate, detailed. Without schedules that 
are vertically integrated, lower-level schedules cannot be clearly 
traced to upper-tiered milestones, and, as a result, total schedule 
integrity is compromised. This can hamper the ability of different 
teams to use the schedule to manage expectations. 

Best practice: 6. Establishing the critical path for all activities; 
Explanation: Scheduling software should be used to identify the 
critical path, which represents the chain of dependent activities with 
the longest total duration. Establishing a project's critical path is 
necessary to examine the effects of any activity slipping along this 
path. Potential problems along or near the critical path should also 
be identified and reflected in scheduling the duration of high-risk 
activities. 
Extent to which criterion was met: Substantially met; 
GAO analysis: Program officials said the schedule contained a critical 
path. Our analysis traced several critical paths in the schedule. 
Though we found lags and deadlines, which create an interruption in 
the various critical paths, the schedule's logic, reasonable 
durations, and low total float estimates allow the scheduling software 
to calculate a valid critical path. 

Best practice: 7. Identifying reasonable float; 
Explanation: The schedule should identify the float--the amount of 
time by which a predecessor activity can slip before the delay affects 
successor activities--so that a schedule's flexibility can be 
determined. As a general rule, activities along the critical path have 
the least float. Total float is the total amount of time by which an 
activity can be delayed without delaying the project's completion, if 
everything else goes according to plan. 
Extent to which criterion was met: Substantially met; 
GAO analysis: Of the 154 remaining detailed tasks, 148 or 96 percent, 
have float values of less than 100 days. The other 6 activities, or 4 
percent, have float values of between 106 and 299 days. Though we 
found some negative float values in the schedule and a few activities 
with float values greater than 100 days, the logical sequencing of 
activities, realistic durations, and evidence of accurate updating 
demonstrate that the total float values in this schedule are 
reasonable. 

Best practice: 8. Conducting a schedule risk analysis; 
Explanation: A schedule risk analysis should be performed using 
statistical techniques to predict the level of confidence in meeting a 
project's completion date. This analysis focuses not only on critical 
path activities but also on activities near the critical path, since 
they can affect the project's status. 
Extent to which criterion was met: Minimally met; 
GAO analysis: Program officials said there is no contractual 
requirement to conduct a schedule risk analysis and said that FAA has 
never expressed an interest in conducting such an analysis. Regarding 
managing and assessing risks in the schedule, program officials said 
they use a monthly risk management board process through which they 
address schedule risk with FAA. Program officials said that, going 
forward, they plan to conduct a Monte Carlo analysis of their 
schedules. 
In our analysis, we found that the contractor's schedule includes a 22-
day "buffer" activity; this is a placeholder activity with a duration 
of 22 days that is intended to be used as contingency should delays 
occur within the release 3B effort. However, our analysis found that 
the buffer does not produce 22 days of savings when removed from the 
schedule. Removing the buffer from the schedule saves 3 days because 
the buffer is not on the current critical path. 
A comprehensive schedule risk analysis is an essential tool for 
decision makers. A schedule risk analysis can be used to determine a 
level of confidence in meeting the completion date or whether proper 
reserves have been incorporated into the schedule. A schedule risk 
analysis will calculate schedule reserve, which can be set aside for 
those activities identified as high-risk. Without this reserve, the 
program faces the risk of delays to the scheduled completion date if 
any delays occur in critical path activities. 

Best practice: 9. Updating the schedule using logic and durations to 
determine the dates; 
Explanation: The schedule should be continuously updated using logic 
and durations to determine realistic start and completion dates for 
program activities. The schedule should be analyzed continuously for 
variances to determine when forecasted completion dates differ from 
planned dates. This analysis is especially important for those 
variations that affect activities identified as being in a project's 
critical path and can influence a scheduled completion date. 
Extent to which criterion was met: Partially met; 
GAO analysis: Our analysis found that the schedule has a valid and 
current status date. Our analysis also found there are no activities 
with start or finish dates in the past that are missing actual start 
and actual finish dates, and there are no activities with actual start 
and actual finish dates in the future. Program officials said that 
schedule updates are provided by the control account managers. The 
scheduler, who is certified in earned value management and has been 
doing scheduling for 2 years, provides the control account manager 
with a status update template that provides the scheduler with in-
progress and percentage-complete activities. The scheduler then 
reviews this information with the control account manager for clarity 
and completeness. 
In addition to schedule updates, program officials said they maintain 
a list of logic changes, which include changes to the sequencing of 
activities or delays in starting planned work. Program officials said 
that they track both start and finish variances in the Microsoft 
Project file as well as those tasks whose status is unreported or that 
are late. Program officials also said the schedules are archived 
daily, weekly, and monthly. Though program officials provided 
testimonial evidence that they adhere to best practices when updating 
their schedule, they did not provide any documentation to validate 
these assertions. 
As a best practice, the schedule should be continually monitored to 
determine when forecasted completion dates differ from the planned 
dates. This information can be used to determine whether schedule 
variances will affect downstream work. Maintaining the integrity of 
the schedule logic is not only necessary to reflect true status, but 
is also required before conducting a schedule risk analysis. Without a 
documented, consistently applied schedule change control process, 
project performers can continually revise the schedule to match 
performance, thereby hindering the project manager's understanding of 
the project's true performance. Good documentation helps with 
analyzing changes in the program schedule and identifying the reasons 
for variances between estimates and actual results. It thus 
contributes to the collection of cost, schedule, and technical data 
that can be used to support future estimates. 

Source: GAO analysis of WAAS contractor-prepared schedule. 

[End of table] 

Schedule Risk Analysis: 

A best practice that the WAAS contractor schedule did not meet is 
conducting a schedule risk analysis, which is not required by FAA's 
schedule specifications. FAA officials told us that they do not 
conduct schedule risk analysis. In August and September 2011, we 
performed our own schedule risk analysis on the WAAS contractor 
schedule, through which we analyzed the latest version of the 
contractor's schedule available to us at the time of the analysis. 

A schedule risk analysis uses statistical techniques to predict a 
level of confidence in meeting a program's completion date. This 
analysis focuses on critical path activities and on near-critical and 
other activities since any activity may potentially affect the 
program's completion date. The objective of the simulation is to 
develop a probability distribution of possible completion dates that 
reflect the program and its quantified risks. From the cumulative 
probability distribution, the organization can match a date to its 
degree of risk tolerance. For instance, an organization might want to 
adopt a program completion date that provides a 70 percent probability 
that the program will finish on or before that date, leaving a 30 
percent probability that it will extend beyond, or overrun that date, 
given the schedule and the risks. The organization can thus adopt a 
plan consistent with its desired level of confidence in the overall 
integrated schedule. This analysis can give valuable insight into what-
if drills and quantify the effects of program changes. 

In developing a schedule risk analysis, probability distributions for 
each activity's duration have to be established. Furthermore, risk in 
all activities must be evaluated and included in the analysis. Some 
managers focus only on the critical path, but because we cannot be 
certain how long activities will take, we cannot know the true 
critical path. Consequently, it would be a mistake to focus only on 
the software-calculated critical path (those activities in which, if 
delayed, will negatively impact the overall project completion date) 
when some off-critical-path activity might become critical if a risk 
were to occur. Typically, three-point estimates--that is, best, most 
likely, and worst-case estimates--are used to develop the probability 
distributions for the duration of workflow activities. 

Once the distributions have been established, a Monte Carlo simulation 
uses random numbers to select specific durations from each activity 
probability distribution and calculates a new critical path and dates, 
including major milestone and program completion dates. The Monte 
Carlo simulation continues this random selection thousands of times, 
creating a new program duration estimate and critical path each time. 
The resulting frequency distribution displays the range of program 
completion dates along with the probabilities that these dates will 
occur. Table 17 provides a range of dates and the probability of the 
project's completion on those dates or earlier, based on our 3,000 
iterations that are chosen at random during the Monte Carlo 
simulation. For example, according to our schedule risk analysis, 
there is a less than 5 percent chance that the project will be 
finished on or before September 13, 2012. Likewise, there is an 80 
percent chance that the project will be finished on or before November 
13, 2012. 

Because completion on any date is uncertain, it is more realistic to 
show a range of possible completion dates than to focus on a single 
date. In deciding which percentile to use for prudent scheduling, 
there is no international best practice standard. The chosen 
percentile depends on the riskiness and maturity of the project. For 
some projects, we emphasize the 80th percentile as a conservative 
promise-of-completion date. While the 80th percentile may appear 
overly conservative, it is a useful promise-of-completion date if a 
number of new but currently unknown risks (i.e., "unknown unknowns") 
are anticipated. The 50th percentile date may expose the project to 
overruns. 

Table 17: Probability of Project Completion: 

Finish date; 
Baseline: Sept. 6, 2012; 
5th Percentile: Sept. 18, 2012; 
50th Percentile: Oct. 23, 2012; 
80th Percentile: Nov. 13, 2012; 
95th Percentile: Dec. 6, 2012. 

Calendar days beyond scheduled finish date; 
Baseline: [Empty]; 
5th Percentile: 12; 
50th Percentile: 47; 
80th Percentile: 68; 
95th Percentile: 91. 

Source: GAO analysis of contractor data. 

[End of table] 

In the case of the WAAS contractor schedule, our analysis concluded 
that management should realistically expect cutover, or completion, 
between October 23, 2012, and November 13, 2012, the 50th and 80th 
percentiles, respectively. The artificial must-finish date constraint 
of September 6, 2012, built into the schedule is unlikely. Our 
analysis shows the probability of completion by September 6, 2012, is 
less than 5 percent likely with the current schedule without risk 
mitigation. There are two reasons why the planned end completion date 
is not likely to occur, according to the results of our schedule risk 
analysis. First, most risks are threats only. Only two opportunities 
were identified during the analysis: (1) the estimating error of the 
schedule may be between -10 percent and 15 percent, and (2) there is a 
65 percent chance that the 11-day formal shadow test will not be 
needed. Second, there are parallel paths within the structure of the 
schedule that lead to merge points. If several paths converge to one 
milestone, the latest merging path determines the date. This "merge 
bias" cannot accelerate schedule dates and usually adds structural 
risk to the schedule. 

Identified Risks: 

The contractor supplied six different risks that are currently 
identified in the project's risk register. Using these risks as a 
basis for discussion, we interviewed 16 experts familiar with the 
project, including prime contractor officials, FAA officials, and 
technical FAA consultants to identify any other risks. Each 
interviewee was asked four questions to address four related points. 

* To estimate the probability that an identified risk will occur on 
the project in such a way that some activity durations are affected. 
The estimated probability is translated into the percentage of 
iterations that are chosen at random during the simulation. For 
example, if the expert estimates weather will have a 10 percent chance 
of affecting some activities, then, on average, weather risk will 
occur in 10 percent of the Monte Carlo iterations. 

* If the interviewee believed the identified risk was likely, the 
interviewee was asked to identify which activities' durations would be 
affected. For example, activities related to steel erection or 
concrete pouring may be affected if the weather risk is realized and 
bad weather occurs. 

* After the interviewee identified affected activities, the 
interviewee was asked to provide a three-point estimate of the risk's 
effect on duration--low, most likely, and high. Estimates were 
provided as percentages, which were applied to the activity durations 
in the Monte Carlo simulation if the risk occurred. For example, if 
bad weather occurs, the duration of a 10-day steel erection activity 
may be affected a minimum of 110 percent, a most likely of 150 
percent, or a maximum of 200 percent. These percentages translate into 
increases in the activity's duration of 11 days under a low-risk 
scenario, 15 days under the most likely risk scenario, and 20 days 
under the maximum risk scenario. If the risk is not realized, there is 
no change to the activity's original estimated duration. 

* Finally, the interviewee was asked to identify any unaccounted for 
risks. 

We began the interviews with 6 risks and, through the interview 
process, identified 16 more risks. During data analysis, some risks 
were consolidated with others or eliminated because of limited data. 
In all, 14 risks were identified and incorporated into the Monte Carlo 
simulation. These include 9 risk drivers, 4 existence risks, and 1 
schedule duration risk.[Footnote 54] 

The final risk drivers used in the schedule risk analysis are as 
follows: 

* Some software Release 3B problem reports require testing with a live 
geostationary satellite; software simulation environment may not be 
able to test all requirements. 

* Potential difficulty repairing Fullerton contractor's lab safety 
computers. 

* Software delays in software Release 3A and additional changes to 
software Release 2B are likely to delay the 3B schedule. 

* Parallel implementation of Releases 2B, 2C, 3A, and 3B create 
competition for FAA resources. 

* Release 3B problem report testing is more complex and subject to 
uncertainty. 

* FAA WAAS personnel move around, and inexperienced staff must be 
trained on WAAS. 

* Workshare strategy between FAA and its prime contractor affects the 
schedule for on-the-job training learning curve. 

* Changes to the simulation tools may be more difficult or more time-
consuming to implement than anticipated. 

The final existence risks are as follows: 

* FAA performs formal software release 3B shadow test. 

* Late problem report may come from FAA. 

* FAA program office is resource-limited. 

Inserting existence risk activities does not affect dates within the 
baseline schedule because the activities initially have zero duration. 
The activities have duration only if they happen to occur during an 
iteration of the simulation. 

The final uncertainty risk is schedule without buffer may be 
optimistic. 

Most risks were identified by multiple respondents during the 
interviews. During data analysis, data from the interviews are 
combined and analyzed to create ranges and probabilities for each of 
the 14 risks. 

Because risks are multiplicative, several risks occurring on the same 
activity may overestimate the true risk. That is, by default in the 
Monte Carlo simulation, risks occur in a series, one after another, so 
that an activity that has several risks may be unrealistically 
extended if all risks occur. In reality, an activity may recover from 
two or more risks simultaneously, so that the actual risk is not 
multiplicative. Therefore, to avoid overestimation of risk, risks can 
explicitly be defined as occurring in parallel rather than in series. 
Risks that occur in a series will occur one after the other and add 
(or subtract) their respective effects on duration to the affected 
activity. If risks occur in parallel, on the other hand, only the 
maximum effect of all risks will affect the duration. For example, if 
the risk of complexity adds 3 days to the duration of software 
development and the risk of staff shortages adds 4 days, then 
development will extend 7 days if the risks are defined in series. 
However, the duration will extend only 4 days if the risks are defined 
as parallel. This definition of parallel risks helps temper any risk 
overestimated by a multiplication of risk factors. We defined one risk 
in series: software delays in Release 3A and additional changes to 
Release 2B are likely to delay Release 3B schedule. All other risks 
were assumed to be parallel. 

Most risks were assigned directly to existing activities in the 
schedule. However, some risks required adjustments to the schedule. 
These adjustments involved replacing lags with activities and 
inserting existence risk activities. 

Lags: During our initial analysis of the contractor's schedule, we 
identified 25 remaining activities with lags and 21 remaining 
activities with leads (negative lags). While lags represent the 
passing of time between activities, they are often misused to put 
activities on a specific date or to insert a buffer for risk. That 
number was reduced to 20 activities with lags and 6 activities with 
leads with the "Rev 1" schedule that was altered by the prime 
contractor for our use in the schedule risk analysis. We replaced the 
33-day lag between the R3B Start Cutover and R3B End Cutover 
activities with an actual activity, ID 258 "Cutover Task." Replacing 
lags with actual activities does not affect dates within the baseline 
schedule because the activities have the same duration as the lags. 

Existence risk: We identified some risks that would add an 
indeterminate amount of time to the overall schedule if they were 
realized. For example, if the Tijuana facility is not ready because 
software requirements are misunderstood, it could add 4 to 26 days to 
the schedule in the form of additional facility work. 

Prioritizing risks and risk mitigation: Risks can affect the schedule 
in several ways: They can have a high probability of occurring; have a 
large-percentage impact on the durations of the activities they 
affect; or they can apply to risk-critical paths, which may differ 
from the baseline deterministic critical path. Beyond applying 14 
risks to the schedule, we were interested in identifying the marginal 
impact of each risk. That is, we were interested in identifying which 
risks have the largest impact on the schedule because these were the 
risks that should be targeted first for mitigation. 

To find the marginal impact of a risk on the total project risk at a 
certain percentile, the Monte Carlo simulation was performed with the 
risk removed. The difference between the finish dates of the 
simulation with all the risks and the simulation with the missing risk 
yielded the marginal impact of the risk. Table 18 gives the priority 
of risks at the 80th percentile and the marginal impact of each risk. 

Table 18: Prioritized Risks at the 80th Percentile: 

Priority level: 1; 
Risk: FAA program office is resource-limited; 
Marginal impact in calendar days: 13. 

Priority level: 2; 
Risk: Software delays in Release 3A and additional changes to Release 
2B are likely to delay Release 3B schedule; 
Marginal impact in calendar days: 7. 

Priority level: 3; 
Risk: Release 3B problem report development and test are more complex 
than planned and subject to uncertainty[A]; 
Marginal impact in calendar days: 7. 

Priority level: 4; 
Risk: FAA personnel move around and often get inexperienced people to 
train; 
Marginal impact in calendar days: 8. 

Priority level: 5; 
Risk: Workshare strategy between FAA and prime contractor affects 
schedule for on-the-job training learning curve; 
Marginal impact in calendar days: 15. 

Priority level: 6; 
Risk: Late problem report may come from FAA; 
Marginal impact in calendar days: 7. 

Priority level: 7; 
Risk: Some Release 3B problem reports require testing with a live 
geostationary satellite; 
software simulation environment may not be able to test all 
requirements; 
Marginal impact in calendar days: 6. 

Priority level: 8; 
Risk: Schedule without buffer may be optimistic; 
Marginal impact in calendar days: 5. 

Priority level: Total; 
Marginal impact in calendar days: 68. 

Source: GAO analysis of WAAS schedule: 

[A] These are two separate risks but are assigned the same priority 
because they are correlated 100 percent in our analysis. That is, we 
assumed if complexity affected one, it would necessarily affect the 
other. 

[End of table] 

The marginal impact directly translates to potential calendar days 
saved if the risk is mitigated. Once risks are prioritized at the 
percentile desired by management, a risk mitigation workshop can be 
implemented to deal with the high-priority risks in order. The 
prioritized list of risks will form the basis of the workshop, and 
risk mitigation plans can be analyzed using the risk model to 
determine how much time might be saved. Project managers cannot expect 
to completely mitigate any one risk, and it is not reasonable to 
expect to mitigate all risks. In addition, risk mitigation will add to 
the project budget. However, some opportunities may be available to 
partially mitigate risks. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Gerald Dillingham, Ph.D., (202) 512-2834 or d [Hyperlink, 
dillinghamg@gao.gov] illinghamg@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, individuals making key 
contributions to this report include Edward Laughlin and Karen Richey 
(Assistant Directors), Lindsey Bach, David Brown, Pamela Davidson, 
Tisha Derricotte, Kevin Egan, James Geibel, Bert Japikse, Delwen 
Jones, Jason Lee, Dominic Nadarski, Josh Ormond, and Brian Welsh. 

[End of section] 

Footnotes: 

[1] The nation's air traffic control system handles almost 30 million 
flights per year. FAA predicts that by 2025, the number of passengers 
will increase from about 700 million to about 1.1 billion per year, 
and the number of flights will increase from about 80,000 to more than 
95,000 every 24 hours. 

[2] GAO, High Risk Series: An Update, [hyperlink, 
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: January 
2009). 

[3] We requested the information on the programs in August 2010. 

[4] According to FAA, baselined programs are acquisition programs with 
an agreed-to description of the attributes and estimated costs at a 
particular point in time. The baseline is a formal, management-
approved document that serves as a starting point for tracking any 
changes in the program as it is developed and implemented for 
performance accountability. 

[5] GAO, Department of Homeland Security: Assessments of Selected 
Complex Acquisitions, [hyperlink, 
http://www.gao.gov/products/GAO-10-588SP], (Washington, D.C.: June 30, 
2010); Defense Acquisitions: Assessments of Selected Weapon Programs, 
[hyperlink, http://www.gao.gov/products/GAO-10-388SP] (Washington, 
D.C.: Mar. 30, 2010); Air Traffic Control: FAA Reports FAA Reports 
Progress in System Acquisitions, but Changes in Performance 
Measurement Could Improve Usefulness of Information, [hyperlink, 
http://www.gao.gov/products/GAO-08-42] (Washington, D.C.: Dec. 18, 
2007); and National Airspace System: FAA Has Made Progress but 
Continues to Face Challenges in Acquiring Major Air Traffic Control 
Systems, [hyperlink, http://www.gao.gov/products/GAO-05-331] 
(Washington, D.C.: June 10, 2005). 

[6] The four initial cost estimates we reviewed were developed at 
various points in the past. 

[7] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[8] The JRC is an FAA executive body consisting of associate and 
assistant administrators, acquisition program executives, the Chief 
Financial Officer, the Chief Information Officer, and legal counsel. 
The JRC makes executive-level decisions, including those that 
determine whether a program meets a mission need and should proceed. 

[9] As defined in Office of Management and Budget Circular A-11, Part 
7, major programs are assets that require special management attention 
because of their importance to the agency's mission. These include 
high development, operating, or maintenance costs; high risk; high 
return; or a significant role in the administration of agency 
programs, finances, property, or other resources. 

[10] FAA is implementing a reorganization of ATO that may alter the 
current alignment of program acquisition responsibilities within ATO. 

[11] [hyperlink, http://www.gao.gov/products/GAO-08-42]. 

[12] GAO, Next Generation Air Transportation System: Challenges with 
Partner Agency and FAA Coordination Continue, and Efforts to Integrate 
Near, Mid-, and Long-term Activities Are Ongoing, [hyperlink, 
http://www.gao.gov/products/GAO-10-649T] (Washington, D.C.: Apr. 21, 
2010). 

[13] Pub. L. No. 108-176, Vision 100--Century of Aviation 
Reauthorization Act Pub. L. No. 108-176, §§ 709-710, 117 Stat. 2490, 
2582 (2003) authorized FAA to begin the NextGen initiative. 

[14] Midterm includes capabilities that are planned to be implemented 
by 2018, such as improved aircraft operational procedures and 
automated data communication between aircraft and controllers. Far 
term refers to the complete implementation of NextGen. 

[15] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[16] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[17] An integrated master schedule is required at the acquisition 
program level in order to meet our best practices, which we discuss 
later in this report. 

[18] Some programs had neither cost increases not schedule delays, 
including: En route Communication Gateway, Integrated Display System, 
Regulation and Certification Infrastructure for System Safety, Time 
Based Flow Manager, and Weather and Radar Program. 

[19] [hyperlink, http://www.gao.gov/products/GAO-08-42]. 

[20] In [hyperlink, http://www.gao.gov/products/GAO-05-331], we noted 
that the original estimate for WAAS was $509 million for a terminated 
contract with Wilcox Electric. Subsequent estimates and baselines 
consider only the interim contract awarded to Hughes Aircraft in 1996. 

[21] FAA changed how it accounted for the costs of satellite leases 
from the operations account to the facilities and equipment account. 

[22] Ten programs experienced cost delays and schedule delays, and 
five programs only experienced schedule delays. 

[23] We did not categorize, or have available information for, the 
remaining four programs. 

[24] GAO, Next Generation Air Transportation System: FAA's Metrics Can 
Be Used to Report on Status of Individual Programs, but Not of Overall 
NextGen Implementation or Outcomes, [hyperlink, 
http://www.gao.gov/products/GAO-10-629] (Washington, D.C.: July 27, 
2010). 

[25] Under the new contract between FAA and the controllers union, the 
union is to have full participation in the development and 
implementation of air traffic modernization systems. 

[26] [hyperlink, http://www.gao.gov/products/GAO-08-42]. 

[27] Initial operating capability is the declaration by site personnel 
that the system is ready for conditional operational use in the NAS. 

[28] Recent reports from the Department of Transportation Inspector 
General and MITRE (a not-for profit organization chartered to work in 
the public interest) indicate that these cost and schedule estimates 
may be underestimated. 

[29] [hyperlink, http://www.faa.gov/news/media/workstop/]. 

[30] DataComm is planned provide capabilities for pilots and 
controllers to transmit digital messages and will eventually replace 
the analog voice communication system currently in use. 

[31] [hyperlink, http://www.gao.gov/products/GAO-10-629]. 

[32] FAA's Enterprise Architecture for the national airspace system 
shows the interdependencies and capabilities that may be affected by 
various programs, but this document cannot indicate specific 
milestones that could be affected. 

[33] [hyperlink, http://www.gao.gov/products/GAO-08-42]. 

[34] The Foundation for Success initiative is aimed at improving 
certain governance, shared services, human capital, and NextGen 
management structures to better manage FAA functions. 

[35] GAO, Air Traffic Control: FAA's Modernization Efforts Past, 
Present, and Future, [hyperlink, 
http://www.gao.gov/products/GAO-04-227T] (Washington, D.C.: Oct. 30, 
2003). 

[36] An independent cost estimate can be done by an independent cost 
estimating office within an organization or by an outside contractor. 

[37] The amount of time by which a predecessor activity can slip 
before the delay affects successor activities. 

[38] Software spiral development is an incremental approach to reduce 
risk, so that user needs and requirements are better defined. 

[39] FAA officials said that because of the 6-month spiral development 
approach, the schedule cannot deliver a single critical path for the 
entire program. Instead, the critical paths are calculated and based 
by software releases. To calculate a critical path by each software 
release, the prime contractor uses an end constraint on the key 
deliverable milestone for each software release. For further 
explanation, see table 13 in appendix IV. 

[40] A work year is approximately 200 days since every organization 
works to a different calendar. 

[41] These float values are due mostly to activities being tied to the 
project finish milestone, which is constrained to start no earlier 
than July 1, 2016. For further explanation, see table 13 in appendix 
IV. 

[42] We reviewed two schedules for the WAAS program: one produced by 
the FAA program office and the other produced by its prime contractor. 
We evaluated the contractor prepared schedule, which was the most 
current schedule available for the purposes of the schedule risk 
analysis, which is discussed more fully later in this report. 

[43] A constraint predefines the start, finish, or both dates of an 
activity. The schedule should use logic and durations in order to 
reflect realistic start and completion dates for activities. 

[44] GAO, VA Construction: VA Is Working to Improve Initial Project 
Cost Estimates, but Should Analyze Cost and Schedule Risks, 
[hyperlink, http://www.gao.gov/products/GAO-10-189] (Washington, D.C.: 
Dec. 14, 2009). 

[45] The other three schedules did not have the required information 
to conduct a schedule risk analysis. 

[46] A Monte Carlo simulation involves the use of random numbers and 
probability distributions to examine potential outcomes. 

[47] We requested the information on the programs in August 2010. 

[48] [hyperlink, http://www.gao.gov/products/GAO-10-588SP], 
[hyperlink, http://www.gao.gov/products/GAO-10-388SP], [hyperlink, 
http://www.gao.gov/products/GAO-08-42], and [hyperlink, 
http://www.gao.gov/products/GAO-05-331]. 

[49] According to FAA, baselined acquisitions, as opposed to 
nonbaselined acquisitions, are an agreed-to description of the 
attributes of a product, at a point in time, that serves as a basis 
for defining change; an approved and released document, or a set of 
documents, each of a specific revision--the purpose of which is to 
provide a defined basis for managing change; the currently approved 
and released configuration documentation; or a released set of files 
consisting of a software version and associated configuration 
documentation. 

[50] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[51] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[52] Hulett & Associates, LLC Los Angeles. Dr. Hulett is the author of 
"Practical Schedule Risk Analysis." 

[53] The WAAS contractor schedule was found reliable enough to conduct 
a schedule risk analysis. Because we found the other three schedules 
were unreliable, a schedule risk analysis could not be performed. 

[54] The schedule duration risk is applied to each activity's duration 
to represent the inherent inaccuracy of scheduling. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.