This is the accessible text file for GAO report number GAO-04-901 
entitled 'Air Traffic Control: System Management Capabilities Improved, 
but More Can Be Done to Institutionalize Improvements' which was 
released on September 20, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Committees: 

August 2004: 

AIR TRAFFIC CONTROL: 

System Management Capabilities Improved, but More Can Be Done to 
Institutionalize Improvements: 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-901]: 

GAO Highlights: 

Highlights of GAO-04-901, a report to congressional committees: 

Why GAO Did This Study: 

Since 1981, the Federal Aviation Administration (FAA) has been working 
to modernize its aging air traffic control (ATC) system. Individual 
projects have suffered cost increases, schedule delays, and performance 
shortfalls of large proportions, leading GAO to designate the program a 
high-risk information technology initiative in 1995. Because the 
program remains a high risk initiative, GAO was requested to assess 
FAA’s progress in several information technology management areas. This 
report, one in a series responding to that request, has two objectives: 
(1) to evaluate FAA’s capabilities for developing and acquiring 
software and systems on its ATC modernization program and (2) to 
assess the actions FAA has under way to improve these capabilities.

What GAO Found: 

FAA has made progress in improving its capabilities for acquiring 
software-intensive systems, but some areas still need improvement. GAO 
had previously reported in 1997 that FAA’s processes for acquiring 
software were ad hoc and sometimes chaotic. Focusing on four mission 
critical air traffic projects, GAO’s current review assessed system 
and software management practices in numerous key areas such as project 
planning, risk management, and requirements development. GAO found that 
these projects were generally performing most of the desired practices: 
of the 900 individual practices evaluated, 83 percent were largely or 
fully implemented. The projects were generally strong in several areas 
such as project planning, requirements management, and identifying 
technical solutions. However, there were recurring weaknesses in the 
areas of measurement and analysis, quality assurance, and verification. These weaknesses hinder FAA from consistently and effectively managing its mission critical systems and increase the risk of cost overruns, schedule delays, and performance shortfalls.

To improve its software and system management capabilities, FAA has 
undertaken a rigorous process improvement initiative. In response to 
earlier GAO recommendations, in 1999, FAA established a centralized 
process improvement office, which has worked to help FAA organizations 
and projects to improve processes through the use of a standard model, 
the integrated Capability Maturity Model. This model, which is a broad 
model that integrates multiple maturity models, is used to assess the 
maturity of FAA’s software and systems capabilities. The projects that 
have adopted the model have demonstrated growth in the maturity of 
their processes, and more and more projects have adopted the model. 
However, the agency does not require the use of this process 
improvement method. To date, less than half of FAA’s major ATC 
projects have used this method, and the recurring weaknesses we 
identified in our project-specific evaluations are due in part to the 
choices these projects were given in deciding whether to and how to 
adopt this process improvement initiative. Further, as a result of 
reorganizing its ATC organizations to a performance-based organization, 
FAA is reconsidering prior policies, and it is not yet clear that 
process improvement will continue to be a priority. Without a strong 
senior-level commitment to process improvement and a consistent, 
institutionalized approach to implementing and evaluating it, FAA 
cannot ensure that key projects will continue to improve systems 
acquisition and development capabilities. As a result, FAA will 
continue to risk the project management problems—including cost 
overruns, schedule delays, and performance shortfalls—that have 
plagued past acquisitions.

What GAO Recommends: 

GAO is making recommendations to the Secretary of Transportation to 
address specific weaknesses and to institutionalize FAA’s process 
improvement initiatives by establishing a policy and plans for 
implementing and overseeing process improvement initiatives.

In commenting on a draft of this report, agency officials generally 
agreed with GAO’s recommendations. They also provided technical 
corrections, which were incorporated as appropriate.

www.gao.gov/cgi-bin/getrpt?GAO-04-901.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Dave Powner at (202) 
512-9286 or pownerd@gao.gov.

[End of section]

Contents: 

Letter: 

Executive Summary: 

Purpose: 

Background: 

Results in Brief: 

Principal Findings: 

Recommendations for Executive Action: 

Agency Comments: 

Chapter 1: Introduction

Overview of ATC: 

FAA's ATC Modernization Is a High-Risk Initiative: 

Prior Report Noted Weaknesses in FAA's Software Acquisition 
Capabilities: 

FAA Is Changing Its Approach to ATC Management: 

The Capability Maturity Model Integration Provides a Means of Assessing 
an Organization's Ability to Manage Software and System Acquisition and 
Development: 

Objectives, Scope, and Methodology: 

Chapter 2: FAA Is Performing Most Project Planning Practices, but It Is 
Not Yet Fully Managing the Process: 

Chapter 3: FAA Is Performing Most Project Monitoring and Control 
Practices, but It Is Not Yet Fully Managing the Process: 

Chapter 4: FAA Is Performing Most Risk Management Practices, but It Is 
Not Yet Fully Managing the Process: 

Chapter 5: FAA Is Performing Requirements Development Practices, but It 
Is Not Yet Fully Managing the Process: 

Chapter 6: FAA Is Performing Requirements Management Practices, but It 
Is Not Yet Fully Managing the Process: 

Chapter 7: FAA Is Performing Most Technical Solution Practices, but It 
Is Not Yet Fully Managing the Process: 

Chapter 8: FAA Is Performing Product Integration Practices, but It Is 
Not Yet Fully Managing the Process: 

Chapter 9: FAA Is Not Performing Key Verification Practices or Fully 
Managing the Process: 

Chapter 10: FAA Is Performing Validation Practices, but It Is Not Yet 
Fully Managing the Process: 

Chapter 11: FAA Is Performing Most Configuration Management Practices, 
but It Is Not Yet Fully Managing the Process: 

Chapter 12: FAA Is Not Performing Key Process and Product Quality 
Assurance Practices or Managing the Process: 

Chapter 13: FAA Is Not Performing Most Measurement and Analysis 
Practices or Managing the Process: 

Chapter 14: FAA Is Performing Supplier Agreement Management Practices, 
but It Is Not Yet Fully Managing the Process: 

Chapter 15: FAA Is Performing Deployment, Transition, and Disposal 
Practices, but It Is Not Yet Fully Managing the Process: 

Chapter 16: FAA’s Process Improvement Initiative Has Matured, but It 
Is Not Yet Institutionalized: 

FAA’s Process Improvement Initiative Has Matured: 

FAA Has Not Yet Institutionalized Process Improvement in Its ATC 
Organizations: 

Chapter 17: Conclusions and Recommendations: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix: 

Appendix I: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Staff Acknowledgments: 

Tables: 

Table 1: CMMI Process Areas: 

Table 2: FAA Projects That GAO Assessed Using CMMI: 

Table 3: Process Areas Assessed on Four ATC Projects: 

Table 4: Four Projects' Appraisal Results in Project Planning: 

Table 5: VSCS Project Planning: Detailed Findings on Level 1 Goals and 
Practices: 

Table 6: VSCS Project Planning: Detailed Findings on Level 2 Goals and 
Practices: 

Table 7: ERAM Project Planning: Detailed Findings on Level 1 Goals and 
Practices: 

Table 8: ERAM Project Planning: Detailed Findings on Level 2 Goals and 
Practices: 

Table 9: ITWS Project Planning: Detailed Findings on Level 1 Goals and 
Practices: 

Table 10: ITWS Project Planning: Detailed Findings on Level 2 Goals and 
Practices: 

Table 11: ASDE-X Project Planning: Detailed Findings on Level 1 Goals 
and Practices: 

Table 12: ASDE-X Project Planning: Detailed Findings on Level 2 Goals 
and Practices: 

Table 13: Four Projects' Appraisal Results in Project Monitoring and 
Control: 

Table 14: VSCS Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices: 

Table 15: VSCS Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices: 

Table 16: ERAM Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices: 

Table 17: ERAM Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices: 

Table 18: ITWS Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices: 

Table 19: ITWS Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices: 

Table 20: ASDE-X Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices: 

Table 21: ASDE-X Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices: 

Table 22: Four Projects' Appraisal Results in Risk Management: 

Table 23: VSCS Risk Management: Detailed Findings on Level 1 Goals and 
Practices: 

Table 24: VSCS Risk Management: Detailed Findings on Level 2 Goals and 
Practices: 

Table 25: ERAM Risk Management: Detailed Findings on Level 1 Goals and 
Practices: 

Table 26: ERAM Risk Management: Detailed Findings on Level 2 Goals and 
Practices: 

Table 27: ITWS Risk Management: Detailed Findings on Level 1 Goals and 
Practices: 

Table 28: ITWS Risk Management: Detailed Findings on Level 2 Goals and 
Practices: 

Table 29: ASDE-X Risk Management: Detailed Findings on Level 1 Goals 
and Practices: 

Table 30: ASDE-X Risk Management: Detailed Findings on Level 2 Goals 
and Practices: 

Table 31: Four Projects' Appraisal Results in Requirements Development: 

Table 32: VSCS Requirements Development: Detailed Findings on Level 1 
Goals and Practices: 

Table 33: VSCS Requirements Development: Detailed Findings on Level 2 
Goals and Practices: 

Table 34: ERAM Requirements Development: Detailed Findings on Level 1 
Goals and Practices: 

Table 35: ERAM Requirements Development: Detailed Findings on Level 2 
Goals and Practices: 

Table 36: ITWS Requirements Development: Detailed Findings on Level 1 
Goals and Practices: 

Table 37: ITWS Requirements Development: Detailed Findings on Level 2 
Goals and Practices: 

Table 38: ASDE-X Requirements Development: Detailed Findings on Level 1 
Goals and Practices: 

Table 39: ASDE-X Requirements Development: Detailed Findings on Level 2 
Goals and Practices: 

Table 40: Four Projects' Appraisal Results in Requirements Management: 

Table 41: VSCS Requirements Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 42: VSCS Requirements Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 43: ERAM Requirements Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 44: ERAM Requirements Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 45: ITWS Requirements Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 46: ITWS Requirements Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 47: ASDE-X Requirements Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 48: ASDE-X Requirements Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 49: Four Projects' Appraisal Results in Technical Solution: 

Table 50: VSCS Technical Solution: Detailed Findings on Level 1 Goals 
and Practices: 

Table 51: VSCS Technical Solution: Detailed Findings on Level 2 Goals 
and Practices: 

Table 52: ERAM Technical Solution: Detailed Findings on Level 1 Goals 
and Practices: 

Table 53: ERAM Technical Solution: Detailed Findings on Level 2 Goals 
and Practices: 

Table 54: ITWS Technical Solution: Detailed Findings on Level 1 Goals 
and Practices: 

Table 55: ITWS Technical Solution: Detailed Findings on Level 2 Goals 
and Practices: 

Table 56: ASDE-X Technical Solution: Detailed Findings on Level 1 Goals 
and Practices: 

Table 57: ASDE-X Technical Solution: Detailed Findings on Level 2 Goals 
and Practices: 

Table 58: Three Projects' Appraisal Results in Product Integration: 

Table 59: VSCS Product Integration: Detailed Findings on Level 1 Goals 
and Practices: 

Table 60: VSCS Product Integration: Detailed Findings on Level 2 Goals 
and Practices: 

Table 61: ITWS Product Integration: Detailed Findings on Level 1 Goals 
and Practices: 

Table 62: ITWS Product Integration: Detailed Findings on Level 2 Goals 
and Practices: 

Table 63: ASDE-X Product Integration: Detailed Findings on Level 1 
Goals and Practices: 

Table 64: ASDE-X Product Integration: Detailed Findings on Level 2 
Goals and Practices: 

Table 65: Four Projects' Appraisal Results in Verification: 

Table 66: VSCS Verification: Detailed Findings on Level 1 Goals and 
Practices: 

Table 67: VSCS Verification: Detailed Findings on Level 2 Goals and 
Practices: 

Table 68: ERAM Verification: Detailed Findings on Level 1 Goals and 
Practices: 

Table 69: ERAM Verification: Detailed Findings on Level 2 Goals and 
Practices: 

Table 70: ITWS Verification: Detailed Findings on Level 1 Goals and 
Practices: 

Table 71: ITWS Verification: Detailed Findings on Level 2 Goals and 
Practices: 

Table 72: ASDE-X Verification: Detailed Findings on Level 1 Goals and 
Practices: 

Table 73: ASDE-X Verification: Detailed Findings on Level 2 Goals and 
Practices: 

Table 74: Four Projects' Appraisal Results in Validation: 

Table 75: VSCS Validation: Detailed Findings on Level 1 Goals and 
Practices: 

Table 76: VSCS Validation: Detailed Findings on Level 2 Goals and 
Practices: 

Table 77: ERAM Validation: Detailed Findings on Level 1 Goals and 
Practices: 

Table 78: ERAM Validation: Detailed Findings on Level 2 Goals and 
Practices: 

Table 79: ITWS Validation: Detailed Findings on Level 1 Goals and 
Practices: 

Table 80: ITWS Validation: Detailed Findings on Level 2 Goals and 
Practices: 

Table 81: ASDE-X Validation: Detailed Findings on Level 1 Goals and 
Practices: 

Table 82: ASDE-X Validation: Detailed Findings on Level 2 Goals and 
Practices: 

Table 83: Four Projects' Appraisal Results in Configuration Management: 

Table 84: VSCS Configuration Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 85: VSCS Configuration Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 86: ERAM Configuration Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 87: ERAM Configuration Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 88: ITWS Configuration Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 89: ITWS Configuration Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 90: ASDE-X Configuration Management: Detailed Findings on Level 1 
Goals and Practices: 

Table 91: ASDE-X Configuration Management: Detailed Findings on Level 2 
Goals and Practices: 

Table 92: Four Projects' Appraisal Results in Process and Product 
Quality Assurance: 

Table 93: VSCS Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices: 

Table 94: VSCS Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices: 

Table 95: ERAM Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices: 

Table 96: ERAM Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices: 

Table 97: ITWS Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices: 

Table 98: ITWS Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices: 

Table 99: ASDE-X Process and Product Quality Assurance: Detailed 
Findings on Level 1 Goals and Practices: 

Table 100: ASDE-X Process and Product Quality Assurance: Detailed 
Findings on Level 2 Goals and Practices: 

Table 101: Four Projects' Appraisal Results in Measurement and 
Analysis: 

Table 102: VSCS Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices: 

Table 103: VSCS Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices: 

Table 104: ERAM Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices: 

Table 105: ERAM Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices: 

Table 106: ITWS Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices: 

Table 107: ITWS Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices: 

Table 108: ASDE-X Measurement and Analysis: Detailed Findings on Level 
1 Goals and Practices: 

Table 109: ASDE-X Measurement and Analysis: Detailed Findings on Level 
2 Goals and Practices: 

Table 110: Two Projects' Appraisal Results in Supplier Agreement 
Management: 

Table 111: ERAM Supplier Agreement Management: Detailed Findings on 
Level 1 Goals and Practices: 

Table 112: ERAM Supplier Agreement Management: Detailed Findings on 
Level 2 Goals and Practices: 

Table 113: ASDE-X Supplier Agreement Management: Detailed Findings on 
Level 1 Goals and Practices: 

Table 114: ASDE-X Supplier Agreement Management: Detailed Findings on 
Level 2 Goals and Practices: 

Table 115: Two Projects' Appraisal Results in Deployment, Transition, 
and Disposal: 

Table 116: ITWS Deployment, Transition, and Disposal: Detailed Findings 
on Level 1 Goals and Practices: 

Table 117: ITWS Deployment, Transition, and Disposal: Detailed Findings 
on Level 2 Goals and Practices: 

Table 118: ASDE-X Deployment, Transition, and Disposal: Detailed 
Findings on Level 1 Goals and Practices: 

Table 119: ASDE-X Deployment, Transition, and Disposal: Detailed 
Findings on Level 2 Goals and Practices: 

Figures: 

Figure 1: Percentage of Fully, Largely, Partially and Not Implemented 
Practices for All Four Projects: 

Figure 2: Summary of the ATC over the Continental United States and 
Oceans: 

Figure 3: CMMI Capability Levels: 

Figure 4: Four Projects' Capability Levels in Project Planning: 

Figure 5: Four Projects' Capability Levels in Project Monitoring and 
Control: 

Figure 6: Four Projects' Capability Levels in Risk Management: 

Figure 7: Four Projects' Capability Levels in Requirements Development: 

Figure 8: Four Projects' Capability Levels in Requirements Management: 

Figure 9: Four Projects' Capability Levels in Technical Solution: 

Figure 10: Three Projects' Capability Levels in Product Integration: 

Figure 11: Four Projects' Capability Levels in Verification: 

Figure 12: Four Projects' Capability Levels in Validation: 

Figure 13: Four Projects' Capability Levels in Configuration 
Management: 

Figure 14: Four Projects' Capability Levels in Process and Product 
Quality Assurance: 

Figure 15: Four Projects' Capability Levels in Measurement and 
Analysis: 

Figure 16: Two Projects' Capability Levels in Supplier Agreement 
Management: 

Figure 17: Two Projects' Capability Levels in Deployment, Transition, 
and Disposal: 

Abbreviations: 

ASDE-X: Airport Surface Detection Equipment-Model X: 

ATC: air traffic control: 

CMMI: Capability Maturity Model Integration: 

ERAM: En Route Automation Modernization: 

FAA: Federal Aviation Administration: 

iCMM: integrated Capability Maturity Model: 

ITWS: Integrated Terminal Weather System: 

SEI: Software Engineering Institute: 

VSCS: Voice Switching and Control System: 

Letter August 20, 2004: 

The Honorable Tom Davis: 
Chairman, Committee on Government Reform: 
House of Representatives: 

The Honorable Adam Putnam: 
Chairman, Subcommittee on Technology, Information Policy, 
Intergovernmental Relations and Census: 
Committee on Government Reform: 
House of Representatives: 

The accompanying report is one in a series of reports responding to 
your request that we review the information technology management 
processes that the Federal Aviation Administration (FAA) uses to manage 
its air traffic control modernization. This report discusses our 
assessment of the department's capabilities to develop and acquire 
software and systems, as well as actions FAA has under way to improve 
these capabilities. We are making recommendations to the Secretary of 
Transportation to assist the agency in improving these capabilities.

As agreed with your staffs, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 30 days 
from the date on the report. At that time, we will send copies of the 
report to the Secretary of Transportation, the Administrator of the 
Federal Aviation Administration, and other congressional committees. 
Copies will also be made available at our Web site at [Hyperlink, 
http://www.gao.gov]. If you have any questions or wish to discuss the 
issues in this report, please contact David Powner at (202) 512-9286 
or by e-mail at [Hyperlink, PownerD@gao.gov], or Keith Rhodes at (202) 
512-6412 or by e-mail at [Hyperlink, RhodesK@gao.gov]. Additional GAO 
contact and staff acknowledgements are listed in appendix I.

Signed by: 

David A. Powner: 
Director, Information Technology Management Issues: 

Signed by: 

Keith Rhodes: 
Chief Technologist: 

[End of section]

Executive Summary: 

Purpose: 

In 1981, the Federal Aviation Administration (FAA) initiated an 
ambitious effort to modernize its aging air traffic control (ATC) 
system to enhance the safety, capacity, and efficiency of the national 
airspace system. Over the ensuing years, individual FAA modernization 
projects suffered cost increases, schedule delays, and performance 
shortfalls of large proportions. In 1995, GAO designated the 
modernization program as a high-risk information technology initiative 
because of its size, complexity, cost, and problem-plagued past.

Recognizing that FAA's ATC modernization remained a high-risk 
information technology initiative in 2003,[Footnote 1] the Chairman of 
the House Committee on Government Reform and the Chairman of that 
Committee's Subcommittee on Technology, Information Policy, 
Intergovernmental Relations and the Census asked GAO to assess FAA's 
progress in several information technology management areas. This 
report is one in a series of reports responding to that request. The 
objectives of this review are (1) to evaluate FAA's capabilities for 
developing and acquiring software and systems on its ATC modernization 
program and (2) to assess the actions FAA has under way to improve 
these capabilities.

Background: 

To accommodate forecasted growth in air traffic and to replace aging 
equipment, in 1981, FAA embarked on an ambitious ATC modernization 
program. This modernization effort includes the acquisition of new 
radar, navigation, communication, and information processing systems, 
as well as new ATC facilities, and is expected to cost $51 billion 
through 2007. FAA has spent over $35 billion on this effort to date.

Quality software and systems are essential to FAA's ATC modernization 
program, and the quality of this software and these systems is governed 
by the value and maturity of the processes involved in acquiring, 
developing, managing, and maintaining them. Carnegie Mellon 
University's Software Engineering Institute, recognized for its 
expertise in software and system processes, has developed the 
Capability Maturity Model® Integration (CMMISM)[Footnote 2] and a CMMI 
appraisal methodology[Footnote 3] to evaluate, improve, and manage 
system and software development processes. The CMMI model and 
appraisal methodology provide a logical framework for measuring and 
improving key processes needed for achieving quality software and 
systems.

In brief, the CMMI appraisal methodology calls for assessing up to 25 
different process areas--clusters of related activities such as project 
planning, requirements management, and quality assurance--by 
determining whether key practices are implemented and whether 
overarching goals are satisfied. Successful implementation of these 
practices and satisfaction of these goals result in the achievement of 
successive capability levels. CMMI capability levels range from 0 to 5, 
with level 0 meaning that the process is either not performed or 
partially performed; level 1 meaning that the basic process is 
performed; level 2 meaning that the process is managed; level 3 meaning 
that the process is defined throughout the organization; level 4 
meaning that the process is quantitatively managed; and level 5 meaning 
that the process is optimized. Using the CMMI model and appraisal 
methodology, GAO specialists, trained by the Software Engineering 
Institute, evaluated four FAA projects' capabilities in 12 to 14 
process areas.[Footnote 4] (See ch. 1 of this report for more detailed 
information on GAO's scope and methodology.) 

Results in Brief: 

FAA has made progress in improving its capabilities for acquiring 
software-intensive systems, but there are still areas that need 
improvement. In a 1997 report, GAO found that FAA's processes for 
acquiring software were ad hoc and sometimes chaotic.[Footnote 5] 
However, currently, FAA projects that GAO reviewed are performing most 
of the desired system and software management practices. GAO appraised 
four different air traffic projects in 12 to 14 different process areas 
through capability level 2 and found that capability levels varied from 
level 0 to level 2. Despite these varied capability levels, many 
practices are being performed. Of the 900 individual practices 
evaluated, 83 percent were largely or fully implemented. The projects 
generally performed key practices in project planning, project 
monitoring and control, risk management, configuration management, 
requirements development, requirements management, validation, 
identifying technical solutions, deployment, and managing supplier 
agreements.[Footnote 6] Nevertheless, there were several recurring 
weaknesses in the areas of measurement and analysis, quality assurance, 
and verification. These weaknesses prevent FAA from consistently and 
effectively managing its mission critical systems and increase the risk 
of cost overruns, schedule delays, and performance shortfalls.

FAA's process improvement initiative has matured in rigor and scope, 
but more can be done to ensure continued improvement in the ATC 
organization. In response to GAO's prior recommendations, in 1999, FAA 
established a centralized process improvement office that reports to 
the Chief Information Officer. This office developed an FAA integrated 
capability maturity model (iCMM), a model that is similar to the CMMI, 
but crafted to include international standards as well as multiple 
capability maturity models. Over the past several years, the process 
improvement office has revised and improved the iCMM model, trained 
over 7,000 FAA personnel in using the model, guided FAA organizations 
and projects in using the model, and evaluated the process maturity of 
projects that adopted the model. Further, since its inception, the 
number of projects that adopted the iCMM process improvement model has 
grown, and the projects that adopted the model have demonstrated growth 
in the maturity of their processes. However, the agency does not 
require the use of this process improvement method, nor has it mandated 
which core process areas to work on. The recurring weaknesses that GAO 
identified in its project-specific evaluations are due in part to the 
choices these projects were given in deciding whether to and how to 
adopt process improvement initiatives. Further, as a result of 
reorganizing its ATC organizations to a performance-based 
organization,[Footnote 7] FAA is reconsidering prior policies, and it 
is not yet clear that process improvement will continue to be a 
priority. Without a strong senior-level commitment to process 
improvement, and a consistent, institutionalized approach to 
implementing and evaluating process improvement, FAA cannot ensure that 
key projects will continue to make progress in improving systems 
acquisition and development capabilities, and the agency cannot proceed 
to more advanced capability levels, which focus on organizationwide 
management of processes. As a result, FAA continues to risk the project 
management problems--including cost overruns, schedule delays, and 
performance shortfalls--that have plagued past acquisitions.

We are recommending to the Secretary of Transportation that the FAA 
address specific weaknesses and institutionalize its process 
improvement initiatives by establishing a policy and plans for 
implementing and overseeing process improvement initiatives. In 
commenting on a draft of this report, agency officials generally agreed 
with our recommendations. Officials also provided technical 
corrections, which we have incorporated into this report as 
appropriate.

Principal Findings: 

FAA Projects Are Performing Many System Management and Acquisition 
Practices, but They Are Not Yet Fully Managing Most of the Processes: 

Since GAO's 1997 report, FAA has made considerable progress in 
improving its processes for acquiring and developing software and 
systems. In a prior report, GAO found that FAA's processes for 
acquiring software were ad hoc and sometimes chaotic. Currently, FAA 
projects are performing many of the desired system and software 
management practices. However, there are still areas for improvement.

GAO appraised four mission critical air traffic projects in 12 to 14 
different process areas and found that capability levels varied among 
projects and process areas. Each of the four projects achieved a rating 
of capability level 1, or "performed," in eight or more process areas 
but also achieved a capability level 0 in one or more process areas. 
Additionally, two projects achieved a rating of capability level 2, or 
"managed," in one or more process areas.

Despite these varied capability scores, many practices are being 
performed. Of the 900 individual practices evaluated, 83 percent were 
largely or fully implemented. The other 17 percent of the practices 
were either partially or not implemented. Although there was not 
complete consistency among projects, the projects GAO reviewed were 
strongest in project planning, project monitoring and control, risk 
management, configuration management, requirements development, 
requirements management, validation, identifying technical solutions, 
deployment, and managing supplier agreements. However, there were 
several recurring weaknesses in the areas of measurement and analysis, 
quality assurance, and verification. These weaknesses inhibit FAA's 
ability to consistently achieve a managed level of performance.

Figure 1 shows the number of practices that GAO found to be fully, 
largely, partially, and not implemented on all four projects.

Figure 1: Percentage of Fully, Largely, Partially and Not Implemented 
Practices for All Four Projects: 

[See PDF for image]

[End of figure]

FAA's Process Improvement Initiative Has Matured, but It Is Not Yet 
Institutionalized: 

Over the past several years, FAA has made considerable progress in 
improving its processes for acquiring and developing software and 
systems, but more can be done to ensure continued improvement in the 
new Air Traffic Organization.

Acting on GAO's prior recommendations, FAA established in 1999 a 
centralized process improvement office that reports directly to the 
Chief Information Officer. This office led the government in an effort 
to integrate various standards and models into a single maturity model, 
the iCMM. The Chief Information Officer's process improvement office 
also developed and sponsored iCMM-related training, and by late 2003, 
it had trained over 7,000 participants. This training ranges from 
overviews on how to use the model to more focused courses in such 
specific process areas as quality assurance, configuration management, 
and project management. The office also guides FAA organizations in 
using the model and leads appraisal teams in evaluating the process 
maturity of the projects and organizations that adopted the model.

In addition to the Chief Information Officer-sponsored process 
improvement efforts, several of FAA's business areas, including the 
business areas with responsibility for ATC system acquisitions and 
operations, endorsed and set goals for process improvement activities 
using the iCMM. As a result, there has been a continuing growth over 
the years in the number of individual projects and umbrella 
organizations that adopted process improvement and the iCMM model. 
Additionally, these projects and organizations have demonstrated 
improvements in process maturity. In internal surveys, the programs and 
organizations pursuing process improvement have consistently reported 
enhanced productivity, higher quality, increased ability to predict 
schedules and resources, higher morale, and better communication and 
teamwork.

However, more can be done to ensure continued improvement in the ATC 
organization. In order to achieve advanced system management 
capabilities and to gain the benefits of more mature processes, leading 
organizations institutionalize process improvement initiatives. 
Specifically, these organizations provide senior-level endorsement and 
enforcement of process improvement initiatives and strive for 
consistency in the adoption of process improvement efforts.

In recent years, FAA's air traffic control-related organizations have 
encouraged process improvement through the iCMM model, but individual 
projects' use of the model has been voluntary. That is, individual 
project teams could determine whether or not they would implement the 
model and which process areas to work on. In addition, project teams 
could decide when, if ever, to seek an appraisal of their progress in 
implementing the model. As a result, individual projects are making 
uneven progress. To date, less than half of the projects listed in 
FAA's system architecture have sought appraisals in at least one 
process area. Further, the recurring weaknesses GAO identified in its 
project-specific evaluations are due in part to the leeway these 
projects were given in adopting process improvement initiatives. 
Weaknesses in key processes could lead to systems that do not meet the 
users' needs, exceed estimated costs, or take longer than expected to 
complete.

Currently, FAA's commitment to its process improvement initiative is 
not certain because the agency is undergoing a reorganization of its 
ATC business areas. FAA recently moved its air traffic-related 
organizations into a single, performance-based organization under the 
direction of a Chief Operating Officer. The Chief Operating Officer is 
currently reevaluating all policies and processes, and he plans to 
issue new acquisition guidance in coming months. Unless the Chief 
Operating Officer demonstrates a strong commitment to process 
improvement and establishes a consistent, institutionalized approach to 
implementing and evaluating this process improvement, FAA risks taking 
a major step backwards in its capabilities for acquiring ATC systems 
and software. That is, FAA is unlikely to ensure that critical projects 
will continue to make progress in improving systems acquisition and 
development capabilities, and the agency will be unable to proceed to 
more advanced capability levels, which focus on organizationwide 
management of processes. Without such advancement in its capabilities, 
FAA will continue to be vulnerable to acquisition problems, including 
cost overruns, schedule delays, and performance shortfalls.

Recommendations for Executive Action: 

Given the importance of software-intensive systems to FAA's ATC 
modernization program, GAO recommends that the Secretary of 
Transportation direct the FAA Administrator to ensure that the 
following five actions take place: 

* The four projects that GAO appraised should take action to fully 
implement the practices that GAO identified as not implemented or 
partially implemented.

* The new Air Traffic Organization should establish: 

* a policy requiring organizations and project teams to implement iCMM 
or equivalent process improvement initiatives and: 

* a plan for implementing iCMM or equivalent process improvement 
initiatives throughout the organization. This plan should specify a 
core set of process areas for all projects, clear criteria for when 
appraisals are warranted, and measurable goals and time frames.

* The Chief Information Officer's process improvement office, in 
consultation with the Air Traffic Organization, should develop a 
strategy for overseeing all air traffic projects' progress to 
successive levels of maturity; this strategy should specify measurable 
goals and time frames.

* To enforce process improvement initiatives, FAA investment decision 
makers should take a project's capability level in core process areas 
into consideration before approving new investments in the project.

Agency Comments: 

In oral comments on a draft of this report, Department of 
Transportation and FAA officials generally concurred with GAO's 
recommendations, and they indicated that FAA is pleased with the 
significant progress that it has achieved in improving the processes 
used to acquire software and systems. Further, these officials noted 
that FAA has already started implementing changes to address issues 
identified in the report. They said that progress is evident both in 
the improved scores, compared with the prior GAO study, and also in the 
way FAA functions on a day-to-day basis. For example, these officials 
explained that FAA is now working better as a team because the agency 
is using cross-organizational teams that effectively share knowledge 
and best practices for systems acquisition and management. Agency 
officials also noted that the constructive exchange of information with 
GAO was very helpful to them in achieving progress, and they emphasized 
their desire to maintain a dialog with GAO to facilitate continued 
progress. FAA officials also provided technical corrections, which we 
have incorporated into this report as appropriate.

[End of section]

Chapter 1: Introduction: 

The primary mission of the Federal Aviation Administration (FAA) is to 
provide a safe, secure, and efficient global aerospace system that 
contributes to national security and the promotion of U.S. aerospace 
safety. FAA's ability to fulfill this mission depends on the adequacy 
and reliability of the nation's air traffic control (ATC) systems--a 
vast network of computer hardware, software, and communications 
equipment.

To accommodate forecasted growth in air traffic and to relieve the 
problems of aging ATC systems, FAA embarked on an ambitious ATC 
modernization program in 1981. FAA now estimates that it will spend 
about $51 billion to replace and modernize ATC systems through 2007. 
Our work over the years has chronicled many FAA problems in meeting ATC 
projects' cost, schedule, and performance goals.[Footnote 8] As a 
result of these issues as well as the tremendous cost, complexity, and 
mission criticality of the modernization program, we designated the 
program as a high-risk information technology initiative in 1995, and 
it has remained on our high-risk list since that time.

Overview of ATC: 

Automated information processing and display, communication, 
navigation, surveillance, and weather resources permit air traffic 
controllers to view key information--such as aircraft location, 
aircraft flight plans, and prevailing weather conditions--and to 
communicate with pilots. These resources reside at, or are associated 
with, several ATC facilities--ATC towers, terminal radar approach 
control facilities, air route traffic control centers (en route 
centers), flight service stations, and the ATC System Command Center. 
Figure 2 shows a visual summary of ATC over the continental United 
States and oceans.

Figure 2: Summary of the ATC over the Continental United States and 
Oceans: 

[See PDF for image]

Note: TRACON is a Terminal Radar Approach Control Facility. ARINC is a 
private organization funded by the airlines and FAA to operate radio 
stations.

[End of figure]

FAA's ATC Modernization Is a High-Risk Initiative: 

Faced with growing air traffic and aging equipment, in 1981, FAA 
initiated an ambitious effort to modernize its ATC system. This effort 
involves the acquisition of new surveillance, data processing, 
navigation, and communications equipment, in addition to new facilities 
and support equipment. Initially, FAA estimated that its ATC 
modernization effort would cost $12 billion and could be completed over 
10 years. Now, 2 decades and $35 billion later, FAA expects to need 
another $16 billion through 2007 to complete key projects, for a total 
cost of $51 billion.

Over the past 2 decades, many of the projects that make up the 
modernization program have experienced substantial cost overruns, 
schedule delays, and significant performance shortfalls. Our work over 
the years has documented many of these shortfalls.[Footnote 9] As a 
result of these problems, as well as the tremendous cost, complexity, 
and mission criticality of the modernization program, we designated the 
program as a high-risk information technology initiative in 1995, and 
it has remained on our high-risk list since that time.[Footnote 10]

Our work since the mid-1990s has pinpointed root causes of the 
modernization program's problems, including (1) immature software 
acquisition capabilities, (2) lack of a complete and enforced system 
architecture, (3) inadequate cost estimating and cost accounting 
practices, (4) an ineffective investment management process, and (5) an 
organizational culture that impaired the acquisition process. We have 
made over 30 recommendations to address these issues, and FAA has made 
substantial progress in addressing them. Nonetheless, in our most 
recent high-risk report, we noted that more remains to be done--and 
with FAA still expecting to spend billions on new ATC systems, these 
actions are as critical as ever.

Prior Report Noted Weaknesses in FAA's Software Acquisition 
Capabilities: 

In March 1997, we reported that FAA's processes for acquiring software, 
the most costly and complex component of its ATC systems, were ad hoc, 
sometimes chaotic, and not repeatable across projects.[Footnote 11] We 
also reported that the agency lacked an effective management structure 
for ensuring software process improvement. As a result, the agency was 
at great risk of not delivering promised software capabilities on time 
and within budget. We recommended that FAA establish a Chief 
Information Officer organizational structure, as prescribed in the 
Clinger-Cohen Act, and assign responsibility for software acquisition 
process improvement to this organization. We also recommended several 
actions intended to help FAA improve its software acquisition 
capabilities by institutionalizing mature processes. These included 
developing a comprehensive plan for process improvement, allocating 
adequate resources to ensure that improvement efforts were implemented, 
and requiring that projects achieve a minimum level of maturity before 
being approved.

FAA has implemented most of our recommendations. The agency established 
a Chief Information Officer position that reports directly to the 
administrator and gave this position responsibility for process 
improvement. The Chief Information Officer's process improvement office 
developed a strategy and led the way in developing an integrated 
framework for improving maturity in system acquisition, development, 
and engineering processes. Some of the business organizations within 
FAA, including the organizations responsible for ATC acquisitions and 
operations, adopted the framework and provided resources to process 
improvement efforts.

FAA did not, however, implement our recommendation to require that 
projects achieve a minimum level of maturity before being approved. 
Officials reported that rather than establish arbitrary thresholds for 
maturity, FAA intended to evaluate process areas that were most 
critical or at greatest risk for each project during acquisition 
management reviews.

FAA Is Changing Its Approach to ATC Management: 

Recent legislation and an executive order have led to major changes in 
the way that FAA manages its ATC mission. In April 2000, the Wendell H. 
Ford Aviation Investment and Reform Act for the 21st Century (Air-21) 
established the position of Chief Operating Officer for the ATC system. 
In December 2000, executive order 13180 instructed FAA to establish a 
performance-based organization known as the Air Traffic Organization 
and to have the Chief Operating Officer lead this organization under 
the authority of the FAA administrator. This order, amended in June 
2002, called for the Air Traffic Organization to enhance the FAA's 
primary mission of ensuring the safety, security, and efficiency of the 
National Airspace System and further improve the delivery of air 
traffic services to the American public by reorganizing air traffic 
services and related offices into: 

a performance-based, results-oriented organization.[Footnote 12] The 
order noted that as a performance-based organization, the Air Traffic 
Organization would be able to take better advantage of the unique 
procurement and personnel authorities currently used by FAA, as well as 
of the additional management reforms enacted by Congress under Air-21. 
In addition, the Air Traffic Organization is responsible for developing 
methods to accelerate ATC modernization, improving aviation safety 
related to ATC, and establishing strong incentives to agency managers 
for achieving results.

In leading the new Air Traffic Organization, the Chief Operating 
Officer's responsibilities include establishing and maintaining 
organizational and individual goals, a 5-year strategic plan including 
ATC system mission and objectives, and a framework agreement with the 
Administrator to establish the new organization's relationships with 
other FAA organizations. In August 2003, the first Chief Operating 
Officer joined the agency and initiated a reorganization combining the 
separate ATC-related organizations and offices into the Air Traffic 
Organization.

The Capability Maturity Model Integration Provides a Means of Assessing 
an Organization's Ability to Manage Software and System Acquisition and 
Development: 

An essential aspect of FAA's ATC modernization program is the quality 
of the software and systems involved, which is heavily influenced by 
the quality and maturity of the processes used to acquire, develop, 
manage, and maintain them. Carnegie Mellon University's Software 
Engineering Institute (SEI), recognized for its expertise in software 
and system processes, has developed the Capability Maturity Model 
Integration (CMMI) and a CMMI appraisal methodology[Footnote 13] to 
evaluate, improve, and manage system and software development and 
engineering processes. The CMMI model and appraisal methodology provide 
a logical framework for measuring and improving key processes needed 
for achieving high-quality software and systems. The model can help an 
organization set process improvement objectives and priorities and 
improve processes; the model can also provide guidance for ensuring 
stable, capable, and mature processes. According to SEI, organizations 
that implement such process improvements can achieve better project 
cost and schedule performance and higher quality products.[Footnote 
14]

In brief, the CMMI model identifies 25 process areas--clusters of 
related practices that, when performed collectively, satisfy a set of 
goals that are considered important for making significant improvements 
in that area. Table 1 describes these process areas.

Table 1: CMMI Process Areas: 

CMMI process area: Project planning; 
Purpose: Establish and maintain 
plans that define project activities.

CMMI process area: Project monitoring and control; 
Purpose: Provide an 
understanding of the project's progress so that appropriate corrective 
actions can be taken when the project's performance deviates 
significantly from the plan.

CMMI process area: Supplier agreement management; 
Purpose: Manage the 
acquisition of products from suppliers for which there exists a formal 
agreement.

CMMI process area: Integrated project management for integrated product 
and process development; 
Purpose: Establish and manage the product and 
the involvement of the relevant stakeholders, according to an 
integrated and defined process that is tailored from the organization's 
set of standard processes. For integrated product and process 
development, integrated project management also covers the 
establishment of a shared vision for the project and a team structure 
for integrated teams that will carry out the objectives of the 
project.

CMMI process area: Risk management; 
Purpose: Identify potential problems before they occur, so that risk-
handling activities may be planned and invoked as needed across the 
life of the product or project to mitigate adverse impacts on achieving 
objectives.

CMMI process area: Integrated teaming; 
Purpose: Form and sustain an integrated team for the development of 
work products.

CMMI process area: Integrated supplier management; 
Purpose: Proactively identify sources of products that may be used to 
satisfy the project's requirements and to manage selected suppliers 
while maintaining a cooperative project-supplier relationship.

CMMI process area: Quantitative project management; 
Purpose: Quantitatively manage the project's defined process to achieve 
the project's established quality and process-performance objectives.

CMMI process area: Requirements management; 
Purpose: Manage the requirements of the project's products and product 
components and to identify inconsistencies between those requirements 
and the project's plans and work products.

CMMI process area: Requirements development; 
Purpose: Produce and analyze customer, product, and product-component 
requirements.

CMMI process area: Technical solution; 
Purpose: Design, develop, and implement solutions to requirements. 
Solutions, designs, and implementations encompass products, product 
components, and product-related life-cycle processes either singly or 
in combinations as appropriate.

CMMI process area: Product integration; 
Purpose: Assemble the product from the product components; ensure that 
the product, as integrated, functions properly; and deliver the 
product.

CMMI process area: Verification; 
Purpose: Ensure that selected work products meet their specified 
requirements.

CMMI process area: Validation; 
Purpose: Demonstrate that a product or product component fulfills its 
intended use when placed in its intended environment.

CMMI process area: Configuration management; 
Purpose: Establish and maintain the integrity of work products using 
configuration identification, configuration control, configuration 
status accounting, and configuration audits.

CMMI process area: Process and product quality assurance; 
Purpose: Provide staff and management with objective insight into 
processes and associated work products.

CMMI process area: Measurement and analysis; 
Purpose: Develop and sustain a measurement capability that is used to 
support management information needs.

CMMI process area: Decision analysis and resolution; 
Purpose: Analyze possible decisions using a formal evaluation process 
that evaluates identified alternatives against established criteria.

CMMI process area: Organizational environment for integration; 
Purpose: Provide an integrated product and process development 
infrastructure and manage people for integration.

CMMI process area: Causal analysis and resolution; 
Purpose: Identify causes of defects and other problems and take action 
to prevent them from occurring in the future.

CMMI process area: Organizational process focus; 
Purpose: Plan and implement organizational process improvement based 
on a thorough understanding of the current strengths and weaknesses of 
the organization's processes and process assets.

CMMI process area: Organizational process definition; 
Purpose: Establish and maintain a usable set of organizational process 
assets.

CMMI process area: Organizational training; 
Purpose: Develop the skills and knowledge of people so they can 
perform their roles effectively and efficiently.

CMMI process area: Organizational process performance; 
Purpose: Establish and maintain a quantitative understanding of the 
performance of the organization's set of standard processes in support 
of quality and process-performance objectives, and provide the process 
performance data, baselines, and models to quantitatively manage the 
organization's projects.

CMMI process area: Organizational innovation and development; 
Purpose: Select and deploy incremental and innovative improvements 
that measurably improve the organization's processes and technologies. 
The improvements support the organization's quality and process- 
performance objectives as derived from the organization's business 
objectives. 

Source: SEI.

[End of table]

The CMMI model provides two alternative ways to view these process 
areas. One way, called continuous representation, focuses on improving 
capabilities in individual process areas. The second way, called staged 
representation, groups process areas together and focuses on achieving 
increased maturity levels by improving the group of process areas.

The CMMI appraisal methodology calls for assessing process areas by 
determining whether the key practices are implemented and whether the 
overarching goals are satisfied. Under continuous representation, 
successful implementation of these practices and satisfaction of these 
goals result in the achievement of successive capability levels in a 
selected process area. CMMI capability levels range from 0 to 5, with 
level 0 meaning that the process is either not performed or partially 
performed; level 1 meaning that the basic process is performed; level 2 
meaning that the process is managed; level 3 meaning that the processes 
is defined throughout the organization; level 4 meaning that the 
process is quantitatively managed; and level 5 meaning that the process 
is optimized. Figure 3 provides details on CMMIcapability levels.

Figure 3: CMMI Capability Levels: 

[See PDF for image]

[End of figure]

Objectives, Scope, and Methodology: 

The Chairman, House Committee on Government Reform, and the Chairman of 
that Committee's Subcommittee on Technology, Information Policy, 
Intergovernmental Relations and the Census requested that we evaluate 
FAA's software and system development processes used to manage its ATC 
modernization. Our objectives were (1) to evaluate FAA's capabilities 
for developing and acquiring software and systems on its ATC 
modernization program and (2) to assess the actions FAA has under way 
to improve these capabilities.

To evaluate FAA's capabilities for developing and acquiring software 
and systems, we applied the CMMI model (continuous representation) and 
its related appraisal methodology to four FAA projects. Our appraisers 
were all SEI-trained software and information systems specialists. In 
addition, we employed SEI-trained consultants as advisors on our first 
evaluation to ensure proper application of the model and appraisal 
methodology.

In consultation with FAA officials, we selected four FAA projects with 
high impact, visibility, and cost, which represented different air 
traffic domains and reflected different stages of life cycle 
development. The projects included the Voice Switching and Control 
System (VSCS), the Integrated Terminal Weather System (ITWS), the En 
Route Automation Modernization (ERAM) project, and the Airport Surface 
Detection Equipment-Model X (ASDE-X). The four projects are described 
in table 2.

Table 2: FAA Projects That GAO Assessed Using CMMI: 

Systems: Voice Switching and Control System (VSCS); 
Description: VSCS is a voice communication system that is in operation 
at 21 Air Route Traffic Control Centers in the continental United 
States and Alaska. VSCS provides air traffic controllers at these 
locations with air-to-ground and ground-to-ground voice communication 
capability. The project's acquisition cost was about $1.3 billion, and 
it has been fully deployed and operational for over 9 years. FAA is 
currently working to refresh the system's technologies and to extend 
the life of the system.

Systems: En Route Automation Modernization (ERAM); 
Description: ERAM is a critical effort to replace existing software 
and hardware in the En Route air traffic control automation (Host) 
computer system, its backup system, and other associated interfaces, 
communications, and support infrastructure. The contract was awarded in 
2002 to Lockheed Martin. ERAM is one of the most expensive and 
software-intensive ATC projects. The project is estimated to cost $2.1 
billion, with deployment at 27 facilities scheduled to begin in 2009 
and to end by 2010.

Systems: Integrated Terminal Weather System (ITWS); 
Description: ITWS is an automated weather information system expected 
to use a variety of weather sensors to efficiently utilize airport 
runways in all kinds of weather. It is expected to acquire and 
integrate weather data from multiple sensors and provide traffic 
managers with a graphic display of weather information. The project's 
cost is estimated at $286.1 million, with deployment scheduled to 
begin in 2003 and end by 2008.

Systems: Airport Surface Detection Equipment-Model X (ASDE-X); 
Description: ASDE-X is a radar system that provides surveillance to 
prevent runway incursions at a large number of airports. It is an 
important safety initiative, initially designed to provide a low-cost 
alternative to FAA's existing ASDE-3 radar systems. The project's cost 
is estimated at $505.2 million, with deployment scheduled to take place 
between 2003 and 2007.

Source: FAA.

[End of table]

In conjunction with FAA's process improvement organization, we 
identified relevant CMMI process areas for each appraisal. In addition, 
because system deployment is an important aspect of FAA systems 
management that is not included in CMMI, we used the deployment, 
transition, and disposal process area from FAA's integrated Capability 
Maturity Model, version 2. For consistency, we merged FAA's criteria 
with SEI's framework and added the standard goals and practices needed 
to achieve capability level 2.

In selected cases, we did not review a certain process area because it 
was not relevant to the current stage of a project's life cycle. For 
example, we did not evaluate supplier agreement management or 
deployment on VSCS because the system is currently in operation, and 
these process areas are no longer applicable to this system. Table 3 
displays the CMMI process areas that we reviewed for each project.

Table 3: Process Areas Assessed on Four ATC Projects:

Process areas: Project planning;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Project monitoring and control;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Risk management;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Requirements development;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Requirements management;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Technical solution;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Product integration;
Projects: VSCS: Yes;
Projects: ERAM: No;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Verification;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Validation;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Configuration management;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Process and product quality assurance;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Measurement and analysis;
Projects: VSCS: Yes;
Projects: ERAM: Yes;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Process areas: Supplier agreement management;
Projects: VSCS: No;
Projects: ERAM: Yes;
Projects: ITWS: No;
Projects: ASDE-X: Yes.

Process areas: Deployment, transition, and disposal;
Projects: VSCS: No;
Projects: ERAM: No;
Projects: ITWS: Yes;
Projects: ASDE-X: Yes.

Source: GAO.

[End of table]

For each process area reviewed, we evaluated project-specific 
documentation and interviewed project officials to determine whether 
key practices were implemented and goals were achieved. In accordance 
with CMMI guidance, we characterized practices as fully implemented, 
largely implemented, partially implemented, and not implemented, and 
characterized goals as satisfied or unsatisfied. After combining the 
practices and goals, the team determined if successive capability 
levels were achieved. According to the CMMI appraisal method, practices 
must be largely or fully implemented in order for a goal to be 
satisfied. Further, all goals must be satisfied in order to achieve a 
capability level. In order to achieve advanced capability levels, all 
preceding capability levels must be achieved. For example, a 
prerequisite for level 2 is the achievement of level 1. As agreed with 
FAA process improvement officials, we evaluated the projects through 
capability level 2.[Footnote 15] Consistent with the CMMI appraisal 
methodology, we validated our findings by sharing preliminary 
observations with the project team so that they were able to provide 
additional documentation or information as warranted.

To assess the actions FAA has under way to improve its system and 
software acquisition and development processes, we evaluated process 
improvement strategies and plans. We also evaluated the progress the 
agency has made in expanding its process improvement initiative, both 
through the maturity of the model and the acceptance of the model by 
project teams. We also interviewed officials from the offices of the 
Chief Information Officer and the Chief Operating Officer to determine 
the effect current changes in the ATC organization could have on the 
process improvement initiatives.

The Department of Transportation and FAA provided oral comments on a 
draft of this report. These comments are presented in chapter 17. We 
performed our work from September 2003 through July 2004 in accordance 
with generally accepted government auditing standards.

[End of section]

Chapter 2: FAA Is Performing Most Project Planning Practices, but It Is 
Not Yet Fully Managing the Process:

The purpose of project planning is to establish and maintain plans that 
define the project activities. This process area involves developing 
and maintaining a plan, interacting with stakeholders, and obtaining 
commitment to the plan.

As figure 4 shows, three of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. The fourth project would have achieved level 1 if it had 
performed one more practice (see the overview in table 4 for details). 
None of the four projects satisfied all criteria for the "managing" 
capability level (level 2). While all four projects had differing 
weaknesses that contributed to this result, common weaknesses across 
most of the projects occurred in the areas of monitoring and 
controlling the project planning process and in ensuring quality 
assurance of the process. As a result of these weaknesses, FAA is 
exposed to increased risks that projects will not meet cost, schedule, 
or performance goals and that projects will not meet mission needs.

Looked at another way, of the 96 practices we evaluated in this process 
area, FAA projects had 88 practices that were fully or largely 
implemented and 8 practices that were partially or not implemented.

Figure 4: Four Projects' Capability Levels in Project Planning:

[See PDF for image]

[End of figure]

Table 4: Four Projects' Appraisal Results in Project Planning:

Needed to achieve capability level 1:

Goal: Estimates of project planning parameters are established and 
maintained;
VSCS: Goal not satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish a top-level work breakdown structure to 
estimate the scope of the project;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain estimates of the attributes of 
the work products and tasks;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Define the project life-cycle phases upon which to 
scope the planning effort;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Estimate the project effort and cost for the work 
products and tasks based on estimation rationale;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: A project plan is established and maintained as the basis for 
managing the project;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain the project's budget and 
schedule;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Identify and analyze project risks;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Plan for the management of project data;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Plan for necessary resources to perform the project;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Plan for knowledge and skills needed to perform the 
project;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Plan the involvement of identified stakeholders;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the overall project plan 
content;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Commitments to the project plan are established and maintained;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Review all plans that affect the project to 
understand project commitments;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Reconcile the project plan to reflect available and 
estimated resources;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Obtain commitment from relevant stakeholders 
responsible for performing and supporting plan execution;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: No;
ERAM: Yes;
ITWS: Yes;
ASDE-X: Yes.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project planning process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project planning process;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the project 
planning process, developing the work products, and providing the 
services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project planning process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the project 
planning process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the project 
planning process under appropriate levels of configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project planning process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the project planning process 
against the plan for performing the process, and take appropriate 
corrective action;
VSCS: Practice not implemented;
ERAM: Practice not implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures, 
and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
project planning process with higher level management, and resolve 
issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 5 through 12. Specifically, 
tables 5 and 6 provide results for VSCS; tables 7 and 8 provide results 
for ERAM; tables 9 and 10 provide results for ITWS; and tables 11 and 
12 provide results for ASDE-X.

[End of table]

Table 5: VSCS Project Planning: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Estimates of project planning parameters are established and 
maintained;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish a top-level work breakdown structure to 
estimate the scope of the project;
Rating: Fully implemented.

Desired practice: Establish and maintain estimates of the attributes of 
the work products and tasks;
Rating: Partially implemented;
Comment: The project team has established estimates of the work 
products; however, it does not maintain information on attributes, such 
as size, function points, requirements, inputs, and outputs.

Desired practice: Define the project life-cycle phases upon which to 
scope the planning effort;
Rating: Fully implemented.

Desired practice: Estimate the project effort and cost for the work 
products and tasks based on estimation rationale;
Rating: Largely implemented;
Comment: The project team estimates the project effort and cost for the 
work products and tasks based on subject matter expert opinion, but it 
does not use objective rationale or historical data.

Goal: A project plan is established and maintained as the basis for 
managing the project;
Rating: Satisfied.

Desired practice: Establish and maintain the project's budget and 
schedule;
Rating: Fully implemented.

Desired practice: Identify and analyze project risks;
Rating: Fully implemented.

Desired practice: Plan for the management of project data;
Rating: Fully implemented.

Desired practice: Plan for necessary resources to perform the project;
Rating: Fully implemented.

Desired practice: Plan for knowledge and skills needed to perform the 
project;
Rating: Largely implemented;
Comment: Project personnel stated that knowledge and skills to perform 
the project are planned for; however, this information is not 
documented.

Desired practice: Plan the involvement of identified stakeholders;
Rating: Fully implemented.

Desired practice: Establish and maintain the overall project plan 
content;
Rating: Fully implemented.

Goal: Commitments to the project plan are established and maintained;
Rating: Satisfied.

Desired practice: Review all plans that affect the project to 
understand project commitments;
Rating: Fully implemented.

Desired practice: Reconcile the project plan to reflect available and 
estimated resources;
Rating: Fully implemented.

Desired practice: Obtain commitment from relevant stakeholders 
responsible for performing and supporting plan execution;
Rating: Fully implemented.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 6: VSCS Project Planning: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices is not 
implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project planning process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project planning process;
Rating: Largely implemented;
Comment: The project team conducted the activities required for the 
project planning process; however, it did not have a documented plan 
for performing the project planning process. For future use, the 
project team has created a handbook that addresses the elements of a 
plan.

Desired practice: Provide adequate resources for performing the project 
planning process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project planning process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
planning process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
planning process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project planning process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project planning process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the project 
planning process against the plan.

Desired practice: Objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
project planning process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 7: ERAM Project Planning: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Estimates of project planning parameters are established and 
maintained;
Rating: Satisfied.

Desired practice: Establish a top-level work breakdown structure to 
estimate the scope of the project;
Rating: Fully implemented.

Desired practice: Establish and maintain estimates of the attributes of 
the work products and tasks;
Rating: Fully implemented.

Desired practice: Define the project life-cycle phases upon which to 
scope the planning effort;
Rating: Fully implemented.

Desired practice: Estimate the project effort and cost for the work 
products and tasks based on estimation rationale;
Rating: Fully implemented.

Goal: A project plan is established and maintained as the basis for 
managing the project;
Rating: Satisfied.

Desired practice: Establish and maintain the project's budget and 
schedule;
Rating: Fully implemented.

Desired practice: Identify and analyze project risks;
Rating: Fully implemented.

Desired practice: Plan for the management of project data;
Rating: Fully implemented.

Desired practice: Plan for necessary resources to perform the project;
Rating: Fully implemented.

Desired practice: Plan for knowledge and skills needed to perform the 
project;
Rating: Fully implemented.

Desired practice: Plan the involvement of identified stakeholders;
Rating: Fully implemented.

Desired practice: Establish and maintain the overall project plan 
content;
Rating: Fully implemented.

Goal: Commitments to the project plan are established and maintained;
Rating: Satisfied.

Desired practice: Review all plans that affect the project to 
understand project commitments;
Rating: Fully implemented.

Desired practice: Reconcile the project plan to reflect available and 
estimated resources;
Rating: Largely implemented;
Comment: The project team has processes in place to reconcile the 
project plan to reflect available and estimated resources; however, the 
plan has not been reconciled to date.

Desired practice: Obtain commitment from relevant stakeholders 
responsible for performing and supporting plan execution;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 8: ERAM Project Planning: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices is not 
implemented and another is partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project planning process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project planning process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
planning process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project planning process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
planning process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
planning process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project planning process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project planning process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the project 
planning process against the plan for performing the process or take 
appropriate corrective action.

Desired practice: Objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining the quality 
assurance process that is expected to objectively evaluate adherence of 
the project planning process to the process description and standards 
and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project planning process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 9: ITWS Project Planning: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Estimates of project planning parameters are established and 
maintained;
Rating: Satisfied.

Desired practice: Establish a top-level work breakdown structure to 
estimate the scope of the project;
Rating: Fully implemented.

Desired practice: Establish and maintain estimates of the attributes of 
the work products and tasks;
Rating: Fully implemented.

Desired practice: Define the project life-cycle phases upon which to 
scope the planning effort;
Rating: Largely implemented;
Comment: The project team defined the project life-cycle phases upon 
which to scope the planning effort; however, the project team has not 
documented a specific project life cycle for this project.

Desired practice: Estimate the project effort and cost for the work 
products and tasks based on estimation rationale;
Rating: Fully implemented.

Goal: A project plan is established and maintained as the basis for 
managing the project;
Rating: Satisfied.

Desired practice: Establish and maintain the project's budget and 
schedule;
Rating: Fully implemented.

Desired practice: Identify and analyze project risks;
Rating: Fully implemented.

Desired practice: Plan for the management of project data;
Rating: Fully implemented.

Desired practice: Plan for necessary resources to perform the project;
Rating: Fully implemented.

Desired practice: Plan for knowledge and skills needed to perform the 
project;
Rating: Fully implemented.

Desired practice: Plan the involvement of identified stakeholders;
Rating: Fully implemented.

Desired practice: Establish and maintain the overall project plan 
content;
Rating: Fully implemented.

Goal: Commitments to the project plan are established and maintained;
Rating: Satisfied.

Desired practice: Review all plans that affect the project to 
understand project commitments;
Rating: Fully implemented.

Desired practice: Reconcile the project plan to reflect available and 
estimated resources;
Rating: Fully implemented.

Desired practice: Obtain commitment from relevant stakeholders 
responsible for performing and supporting plan execution;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 10: ITWS Project Planning: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices are 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project planning process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project planning process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
planning process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project planning process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
planning process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
planning process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
project planning process under some levels of configuration management; 
however, the controls on these products are not adequate because all 
support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
project planning process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project planning process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Partially implemented;
Comment: The project team collects contractor cost and schedule data, 
but it does not monitor and control the project planning process 
against the plan for performing the process or take appropriate 
corrective action.

Desired practice: Objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures 
and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project planning process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 11: ASDE-X Project Planning: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Estimates of project planning parameters are established and 
maintained;
Rating: Satisfied.

Desired practice: Establish a top-level work breakdown structure to 
estimate the scope of the project;
Rating: Fully implemented.

Desired practice: Establish and maintain estimates of the attributes of 
the work products and tasks;
Rating: Fully implemented.

Desired practice: Define the project life-cycle phases upon which to 
scope the planning effort;
Rating: Fully implemented.

Desired practice: Estimate the project effort and cost for the work 
products and tasks based on estimation rationale;
Rating: Fully implemented.

Goal: A project plan is established and maintained as the basis for 
managing the project;
Rating: Satisfied.

Desired practice: Establish and maintain the project's budget and 
schedule;
Rating: Fully implemented.

Desired practice: Identify and analyze project risks;
Rating: Fully implemented.

Desired practice: Plan for the management of project data;
Rating: Largely implemented;
Comment: The project team plans for the management of data, but 
internal documents, such as the project plan, have yet to be placed 
under formal configuration management control.

Desired practice: Plan for necessary resources to perform the project;
Rating: Largely implemented;
Comment: Although the project team provides resources to perform the 
project, such as staff and tools, there is no evidence that these are 
managed and controlled.

Desired practice: Plan for knowledge and skills needed to perform the 
project;
Rating: Fully implemented.

Desired practice: Plan the involvement of identified stakeholders;
Rating: Fully implemented.

Desired practice: Establish and maintain the overall project plan 
content;
Rating: Fully implemented.

Goal: Commitments to the project plan are established and maintained;
Rating: Satisfied.

Desired practice: Review all plans that affect the project to 
understand project commitments;
Rating: Fully implemented.

Desired practice: Reconcile the project plan to reflect available and 
estimated resources;
Rating: Fully implemented.

Desired practice: Obtain commitment from relevant stakeholders 
responsible for performing and supporting plan execution;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 12: ASDE-X Project Planning: Detailed Findings on Level 2 Goals 
and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project planning process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project planning process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
planning process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project planning process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the project 
planning process to various groups, but it does not assign duties to 
individuals.

Desired practice: Train the people performing or supporting the project 
planning process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
planning process under appropriate levels of configuration management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the project planning process under appropriate levels of 
configuration management; however, internal documents have yet to be 
placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
project planning process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project planning process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Largely implemented;
Comment: The project team monitors and controls the project planning 
process against the plan for performing the process; however, the 
metrics are collected by the contractor, and the project team does not 
maintain, record, or track the metrics on a monthly basis.

Desired practice: Objectively evaluate adherence of the project 
planning process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate the adherence of the project 
planning process to its process description, standards, and procedures 
and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project planning process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 3: FAA Is Performing Most Project Monitoring and Control 
Practices, but It Is Not Yet Fully Managing the Process:

The purpose of project monitoring and control is to provide an 
understanding of the project's progress so that appropriate corrective 
actions can be taken when the project's performance deviates 
significantly from the plan. Key activities include monitoring 
activities, communicating status, taking corrective action, and 
determining progress.

As shown in figure 5, three of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. The fourth project would have achieved level 1 if it had 
performed one more practice (see the overview in table 13 for details). 
None of the four projects satisfied all criteria for the "managing" 
capability level (level 2). While the projects had differing weaknesses 
that contributed to this result, a common weakness across most of the 
projects occurred in the area of ensuring quality assurance of the 
process. As a result of this weakness, FAA is exposed to increased 
risks that projects will not meet cost, schedule, or performance goals 
and that projects will not meet mission needs.

Looked at another way, of the 80 practices we evaluated in this process 
area, FAA projects had 74 practices that were fully or largely 
implemented and 6 practices that were partially or not implemented.

Figure 5: Four Projects' Capability Levels in Project Monitoring and 
Control:

[See PDF for image]

[End of figure]

Table 13: Four Projects' Appraisal Results in Project Monitoring and 
Control:

Goal: Actual performance and progress of the project are monitored 
against the project plan;
VSCS: Goal not satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Monitor the actual values of the project planning 
parameters against the project plan;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor commitments against those identified in the 
project plan;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor risks against those identified in the project 
plan;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor the management of project data against the 
project plan;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Monitor stakeholder involvement against the project 
plan;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Periodically review the project's progress, 
performance, and issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Review the accomplishments and results of the project 
at selected project milestones;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Corrective actions are managed to closure when the project's 
performance or results deviate significantly from the plan;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Collect and analyze the issues and determine the 
corrective actions necessary to address the issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Take corrective action on identified issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Manage corrective actions to closure;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: No;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project monitoring and control process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project monitoring and control process;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the project 
monitoring and control process, developing the work products, and 
providing the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project monitoring and control process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the project 
monitoring and control process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the project 
monitoring and control process under appropriate levels of 
configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project monitoring and control process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the project monitoring and 
control process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice not implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
project monitoring and control process with higher level management, 
and resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 14 through 21. Specifically, 
tables 14 and 15 provide results for VSCS; tables 16 and 17 provide 
results for ERAM; tables 18 and 19 provide results for ITWS; and tables 
20 and 21 provide results for ASDE-X.

Table 14: VSCS Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Actual performance and progress of the project are monitored 
against the project plan;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Monitor the actual values of the project planning 
parameters against the project plan;
Rating: Partially implemented;
Comment: The project team discusses project parameters at various 
meetings. However, it does not monitor the actual values of the 
parameters against the project plan.

Desired practice: Monitor commitments against those identified in the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor risks against those identified in the project 
plan;
Rating: Fully implemented.

Desired practice: Monitor the management of project data against the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor stakeholder involvement against the project 
plan;
Rating: Fully implemented.

Desired practice: Periodically review the project's progress, 
performance, and issues;
Rating: Fully implemented.

Desired practice: Review the accomplishments and results of the project 
at selected project milestones;
Rating: Fully implemented.

Goal: Corrective actions are managed to closure when the project's 
performance or results deviate significantly from the plan;
Rating: Satisfied.

Desired practice: Collect and analyze the issues and determine the 
corrective actions necessary to address the issues;
Rating: Fully implemented.

Desired practice: Take corrective action on identified issues;
Rating: Fully implemented.

Desired practice: Manage corrective actions to closure;
Rating: Fully implemented.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 15: VSCS Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project monitoring and control process;
Rating: Largely implemented;
Comment: The project team conducts project monitoring and control 
practices; however, there is no specific plan for performing the 
project monitoring and control process. In addition, the project team 
has developed a handbook that provides guidelines on what should be 
done. This handbook was not available for the current projects.

Desired practice: Provide adequate resources for performing the project 
monitoring and control process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
monitoring and control process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
monitoring and control process under appropriate levels of 
configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project monitoring and control process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project monitoring and 
control process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the project 
monitoring and control process against the plan.

Desired practice: Objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
project monitoring and control process with higher level management, 
and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 16: ERAM Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Actual performance and progress of the project are monitored 
against the project plan;
Rating: Satisfied.

Desired practice: Monitor the actual values of the project planning 
parameters against the project plan;
Rating: Fully implemented.

Desired practice: Monitor commitments against those identified in the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor risks against those identified in the project 
plan;
Rating: Fully implemented.

Desired practice: Monitor the management of project data against the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor stakeholder involvement against the project 
plan;
Rating: Largely implemented;
Comment: The project team stated that it monitors stakeholder 
involvement against the plan by recording stakeholder participation in 
program meetings, user team meetings, and in the review of documents. 
However, the team did not have a checklist at each meeting that stated 
which stakeholders should be present.

Desired practice: Periodically review the project's progress, 
performance, and issues;
Rating: Fully implemented.

Desired practice: Review the accomplishments and results of the project 
at selected project milestones;
Rating: Fully implemented.

Goal: Corrective actions are managed to closure when the project's 
performance or results deviate significantly from the plan;
Rating: Satisfied.

Desired practice: Collect and analyze the issues and determine the 
corrective actions necessary to address the issues;
Rating: Fully implemented.

Desired practice: Take corrective action on identified issues;
Rating: Fully implemented.

Desired practice: Manage corrective actions to closure;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 17: ERAM Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project monitoring and control process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
monitoring and control process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
monitoring and control process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
monitoring and control process under appropriate levels of 
configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
project monitoring and control process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project monitoring and 
control process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining the quality 
assurance process that is expected to objectively evaluate adherence of 
the project monitoring and control process to its process description, 
standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project monitoring and control process with higher level management, 
and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 18: ITWS Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Actual performance and progress of the project are monitored 
against the project plan;
Rating: Satisfied.

Desired practice: Monitor the actual values of the project planning 
parameters against the project plan;
Rating: Fully implemented.

Desired practice: Monitor commitments against those identified in the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor risks against those identified in the project 
plan;
Rating: Fully implemented.

Desired practice: Monitor the management of project data against the 
project plan;
Rating: Largely implemented;
Comment: The project team monitors the management of project data 
against the project plan; however, it does not identify and document 
significant issues and impacts.

Desired practice: Monitor stakeholder involvement against the project 
plan;
Rating: Fully implemented.

Desired practice: Periodically review the project's progress, 
performance, and issues;
Rating: Fully implemented.

Desired practice: Review the accomplishments and results of the project 
at selected project milestones;
Rating: Fully implemented.

Goal: Corrective actions are managed to closure when the project's 
performance or results deviate significantly from the plan;
Rating: Satisfied.

Desired practice: Collect and analyze the issues and determine the 
corrective actions necessary to address the issues;
Rating: Fully implemented.

Desired practice: Take corrective action on identified issues;
Rating: Fully implemented.

Desired practice: Manage corrective actions to closure;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 19: ITWS Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project monitoring and control process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
monitoring and control process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the project 
monitoring and control process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
monitoring and control process under appropriate levels of 
configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
project monitoring and control process under some levels of 
configuration management; however, the controls on these products are 
not adequate because all support contractors have full access to the 
system.

Desired practice: Identify and involve the relevant stakeholders of the 
project monitoring and control process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project monitoring and 
control process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team monitors the process, but it does not use the 
data to control the process and take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project monitoring and control process with higher level management, 
and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 20: ASDE-X Project Monitoring and Control: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Actual performance and progress of the project are monitored 
against the project plan;
Rating: Satisfied.

Desired practice: Monitor the actual values of the project planning 
parameters against the project plan;
Rating: Fully implemented.

Desired practice: Monitor commitments against those identified in the 
project plan;
Rating: Fully implemented.

Desired practice: Monitor risks against those identified in the project 
plan;
Rating: Fully implemented.

Desired practice: Monitor the management of project data against the 
project plan;
Rating: Largely implemented;
Comment: The project team monitors the contractor documents, but 
internal documents, such as the project plan, have yet to be placed 
under formal configuration management control.

Desired practice: Monitor stakeholder involvement against the project 
plan;
Rating: Fully implemented.

Desired practice: Periodically review the project's progress, 
performance, and issues;
Rating: Fully implemented.

Desired practice: Review the accomplishments and results of the project 
at selected project milestones;
Rating: Fully implemented.

Goal: Corrective actions are managed to closure when the project's 
performance or results deviate significantly from the plan;
Rating: Satisfied.

Desired practice: Collect and analyze the issues and determine the 
corrective actions necessary to address the issues;
Rating: Fully implemented.

Desired practice: Take corrective action on identified issues;
Rating: Fully implemented.

Desired practice: Manage corrective actions to closure;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 21: ASDE-X Project Monitoring and Control: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the project monitoring and control process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
project monitoring and control process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the project 
monitoring and control process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the project monitoring and control process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the project 
planning process to various groups; however, it does not assign duties 
to individuals.

Desired practice: Train the people performing or supporting the project 
monitoring and control process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the project 
monitoring and control process under appropriate levels of 
configuration management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the project monitoring and control process under 
appropriate levels of configuration management; however, internal 
documents have yet to be placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
project monitoring and control process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the project monitoring and 
control process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team monitors and controls the project monitoring 
and control process against the plan for performing the process; 
however, the metrics are collected by the contractor, and the project 
team does not maintain, record, or track the metrics on a monthly 
basis.

Desired practice: Objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the project 
monitoring and control process to its process description, standards, 
and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
project monitoring and control process with higher level management, 
and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 4: FAA Is Performing Most Risk Management Practices, but It Is 
Not Yet Fully Managing the Process:

The purpose of risk management is to identify potential problems before 
they occur, so that risk-handling activities may be planned and invoked 
as needed across the life of the product or project to mitigate adverse 
impacts on achieving objectives. Effective risk management includes 
early and aggressive identification of risks through the involvement of 
relevant stakeholders. Early and aggressive detection of risk is 
important, because it is typically easier, less costly, and less 
disruptive to make changes and correct work efforts during the earlier 
phases of the project.

As shown in figure 6, three of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. The fourth project would have achieved level 1 if it had 
performed one more practice (see the overview in table 22 for details). 
Two of the four FAA projects also satisfied all criteria for the 
"managed" capability level (level 2) in this process area. While the 
other projects had differing weaknesses that contributed to this 
result, common weaknesses across some of the projects occurred in the 
area of monitoring and controlling the risk management process and in 
ensuring quality assurance of the process. As a result of these 
weaknesses, FAA faces increased likelihood that project risks will not 
be identified and addressed in a timely manner--thereby increasing the 
likelihood that projects will not meet cost, schedule, or performance 
goals.

Looked at another way, of the 68 practices we evaluated in this key 
process area, FAA projects had 59 practices that were fully or largely 
implemented and 9 practices that were partially or not implemented.

Figure 6: Four Projects' Capability Levels in Risk Management:

[See PDF for image]

[End of figure]

Table 22: Four Projects' Appraisal Results in Risk Management:

Goal: Preparation for risk management is conducted;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Determine risk sources and categories;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Define the parameters used to analyze and categorize 
risks and the parameters used to control the risk management effort;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the strategy to be used for 
risk management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Risks are identified and analyzed to determine their relative 
importance;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Identify and document the risks;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Evaluate and categorize each identified risk using 
the defined risk categories and parameters, and determine its relative 
priority;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Risks are handled and mitigated, where appropriate, to reduce 
adverse impacts on achieving objectives;
VSCS: Goal not satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management strategy;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: No;
ERAM: Yes;
ITWS: Yes;
ASDE-X: Yes.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the risk management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
risk management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the risk 
management process, developing the work products, and providing the 
services of the process;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the risk management process;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Train the people performing or supporting the risk 
management process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the risk management 
process under appropriate levels of configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
risk management process as planned;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the risk management process 
against the plan for performing the process, and take appropriate 
corrective action;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Objectively evaluate adherence of the risk management 
process to its process description, standards, and procedures, and 
address noncompliance;
VSCS: Practice partially implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Review the activities, status, and results of the 
risk management process with higher level management, and resolve 
issues;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: Yes;
ITWS: No;
ASDE-X: Yes.

Sources: GAO, SEI.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 23 through 30. Specifically, 
tables 23 and 24 provide results for VSCS; tables 25 and 26 provide 
results for ERAM; tables 27 and 28 provide results for ITWS; and tables 
29 and 30 provide results for ASDE-X.

Table 23: VSCS Risk Management: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for risk management is conducted;
Rating: Satisfied.

Desired practice: Determine risk sources and categories;
Rating: Fully implemented.

Desired practice: Define the parameters used to analyze and categorize 
risks and the parameters used to control the risk management effort;
Rating: Fully implemented.

Desired practice: Establish and maintain the strategy to be used for 
risk management;
Rating: Fully implemented.

Goal: Risks are identified and analyzed to determine their relative 
importance;
Rating: Satisfied.

Desired practice: Identify and document the risks;
Rating: Fully implemented.

Desired practice: Evaluate and categorize each identified risk using 
the defined risk categories and parameters, and determine its relative 
priority;
Rating: Fully implemented.

Goal: Risks are handled and mitigated, where appropriate, to reduce 
adverse impacts on achieving objectives;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management strategy;
Rating: Largely implemented;
Comment: The project team has a high level risk mitigation plan that 
was developed as part of the project plan. However, this plan is not 
sufficiently detailed.

Desired practice: Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate;
Rating: Partially implemented;
Comment: While project officials stated that risks are discussed at 
meetings, there is little evidence that the project team monitors the 
status of each risk periodically or implements the risk mitigation 
plan.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 24: VSCS Risk Management: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because four of the practices below 
are partially implemented and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the risk management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
risk management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the risk 
management process, developing the work products, and providing the 
services of the process;
Rating: Partially implemented;
Comment: Resources for performing the risk management process are not 
adequate. Some practices for this process area are not being performed 
because of a lack of resources.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the risk management process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility and authority for 
performing the process, developing the work products, and providing the 
services of the risk management process. However, all of these 
responsibilities have been assigned to a single individual, who also 
has other duties, thereby increasing the risk that these activities 
will not be robust.

Desired practice: Train the people performing or supporting the risk 
management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the risk management 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
risk management process as planned;
Rating: Partially implemented;
Comment: The project team stated that risks are identified and 
discussed periodically during meetings with relevant stakeholders; 
however, meeting minutes did not always document that risks were 
discussed with relevant stakeholders.

Desired practice: Monitor and control the risk management process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the risk 
management process against the plan for performing the process or take 
appropriate corrective action.

Desired practice: Objectively evaluate adherence of the risk management 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team subjectively evaluates adherence to the risk 
management process. The project team does not objectively evaluate 
adherence of the risk management process to its process description, 
standards, and procedures, and it does not address noncompliance.

Desired practice: Review the activities, status, and results of the 
risk management process with higher level management, and resolve 
issues;
Rating: Partially implemented;
Comment: Project personnel stated that risk management activities are 
discussed with higher level management. However, the meeting minutes 
generally did not show that the project team reviews the activities, 
status, and results of the risk management process with higher level 
management and resolves issues.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 25: ERAM Risk Management: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for risk management is conducted;
Rating: Satisfied.

Desired practice: Determine risk sources and categories;
Rating: Fully implemented.

Desired practice: Define the parameters used to analyze and categorize 
risks and the parameters used to control the risk management effort;
Rating: Fully implemented.

Desired practice: Establish and maintain the strategy to be used for 
risk management;
Rating: Fully implemented.

Goal: Risks are identified and analyzed to determine their relative 
importance;
Rating: Satisfied.

Desired practice: Identify and document the risks;
Rating: Fully implemented.

Desired practice: Evaluate and categorize each identified risk using 
the defined risk categories and parameters, and determine its relative 
priority;
Rating: Fully implemented.

Goal: Risks are handled and mitigated, where appropriate, to reduce 
adverse impacts on achieving objectives;
Rating: Satisfied.

Desired practice: Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management strategy;
Rating: Fully implemented.

Desired practice: Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 26: ERAM Risk Management: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the risk management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
risk management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the risk 
management process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the risk management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the risk 
management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the risk management 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
risk management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the risk management process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Largely implemented;
Comment: The project team monitors the risk management process against 
the plan for performing the process and takes appropriate corrective 
action. However, the team is not using the available data on a monthly 
basis to control the process.

Desired practice: Objectively evaluate adherence of the risk management 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Largely implemented;
Comment: The project team recently performed an audit on the risk 
management process that objectively evaluated adherence of the process 
to its process description, standards, and procedures, and addressed 
noncompliance; however, the team has just started defining its quality 
assurance process.

Desired practice: Review the activities, status, and results of the 
risk management process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 achieved.

Sources: GAO, SEI.

[End of table]

Table 27: ITWS Risk Management: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for risk management is conducted;
Rating: Satisfied.

Desired practice: Determine risk sources and categories;
Rating: Fully implemented.

Desired practice: Define the parameters used to analyze and categorize 
risks and the parameters used to control the risk management effort;
Rating: Fully implemented.

Desired practice: Establish and maintain the strategy to be used for 
risk management;
Rating: Fully implemented.

Goal: Risks are identified and analyzed to determine their relative 
importance;
Rating: Satisfied.

Desired practice: Identify and document the risks;
Rating: Fully implemented.

Desired practice: Evaluate and categorize each identified risk using 
the defined risk categories and parameters, and determine its relative 
priority;
Rating: Fully implemented.

Goal: Risks are handled and mitigated, where appropriate, to reduce 
adverse impacts on achieving objectives;
Rating: Satisfied.

Desired practice: Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management strategy;
Rating: Fully implemented.

Desired practice: Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 28: ITWS Risk Management: Detailed Findings on Level 2 Goals and 
Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the risk management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
risk management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the risk 
management process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the risk management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the risk 
management process as needed;
Rating: Largely implemented;
Comment: Key members of the project team are trained on the risk 
management process; however, other project team members have yet to be 
trained.

Desired practice: Place designated work products of the risk management 
process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the risk 
management process under some levels of configuration management; 
however, the controls on these products are not adequate because all 
support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
risk management process as planned;
Rating: Largely implemented;
Comment: The project team has involved certain individuals in the 
monitoring of risks; however, stakeholders are not identified in the 
risk management plan, and the project lead has not approved the list of 
stakeholders.

Desired practice: Monitor and control the risk management process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Partially implemented;
Comment: The project team is using a risk management database; however, 
there is no evidence that the project team is monitoring and 
controlling the risk management process against the plan for performing 
the risk management process and taking appropriate corrective action.

Desired practice: Objectively evaluate adherence of the risk management 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the risk 
management process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
risk management process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 29: ASDE-X Risk Management: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Preparation for risk management is conducted;
Rating: Satisfied.

Desired practice: Determine risk sources and categories;
Rating: Fully implemented.

Desired practice: Define the parameters used to analyze and categorize 
risks and the parameters used to control the risk management effort;
Rating: Fully implemented.

Desired practice: Establish and maintain the strategy to be used for 
risk management;
Rating: Fully implemented.

Goal: Risks are identified and analyzed to determine their relative 
importance;
Rating: Satisfied.

Desired practice: Identify and document the risks;
Rating: Fully implemented.

Desired practice: Evaluate and categorize each identified risk using 
the defined risk categories and parameters, and determine its relative 
priority;
Rating: Fully implemented.

Goal: Risks are handled and mitigated, where appropriate, to reduce 
adverse impacts on achieving objectives;
Rating: Satisfied.

Desired practice: Develop a risk mitigation plan for the most important 
risks to the project, as defined by the risk management strategy;
Rating: Fully implemented.

Desired practice: Monitor the status of each risk periodically and 
implement the risk mitigation plan as appropriate;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 30: ASDE-X Risk Management: Detailed Findings on Level 2 Goals 
and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the risk management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
risk management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the risk 
management process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the risk management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the risk 
management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the risk management 
process under appropriate levels of configuration management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the risk management process under appropriate levels of 
configuration management; however, internal documents have yet to be 
placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
risk management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the risk management process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Largely implemented;
Comment: The project team monitors and controls the risk management 
process against the plan for performing the process; however, the 
metrics are collected by the contractor, and the project team does not 
maintain, record, or track the metrics on a monthly basis, therefore 
making it difficult to take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the risk management 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
risk management process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 achieved.

Sources: GAO, SEI.

[End of section]

Chapter 5: FAA Is Performing Requirements Development Practices, but It 
Is Not Yet Fully Managing the Process:

The purpose of requirements development is to produce and analyze 
customer, product, and product-component needs. This process area 
addresses the needs of relevant stakeholders, including those pertinent 
to various product life-cycle phases. It also addresses constraints 
caused by the selection of design solutions. The development of 
requirements includes elicitation, analysis, validation, and 
communication of customer and stakeholder needs and expectations.

As shown in figure 7, all four FAA projects satisfied all criteria for 
the "performing" capability level (level 1) in this process area. None 
of the four projects satisfied all criteria for the "managing" 
capability level (level 2). While all four projects had differing 
weaknesses that contributed to this result, common weaknesses across 
multiple projects occurred in the areas of training people and in 
ensuring quality assurance of the requirements development process, as 
shown in the overview in table 31. As a result of these weaknesses, FAA 
is exposed to increased risks that projects will not fulfill mission 
and user needs.

Looked at another way, of the 84 practices we evaluated in this key 
process area, FAA projects had 77 practices that were fully or largely 
implemented and 7 practices that were partially or not implemented.

Figure 7: Four Projects' Capability Levels in Requirements Development:

[See PDF for image]

[End of figure]

Table 31: Four Projects' Appraisal Results in Requirements Development:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Identify and collect stakeholder needs, expectations, 
constraints, and interfaces for all phases of the product life cycle;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Transform stakeholder needs, expectations, 
constraints, and interfaces into customer requirements;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Customer requirements are refined and elaborated to develop 
product and product-component requirements;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain product and product-component 
requirements that are based on the customer requirements;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Allocate the requirements for each product component;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Identify interface requirements;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain operational concepts and 
associated scenarios;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain a definition of required 
functionality;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Analyze requirements to ensure that they are 
necessary and sufficient;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Validate requirements to ensure that the resulting 
product will perform appropriately in its intended-use environment;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: Yes;
ITWS: Yes;
ASDE-X: Yes.

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Elicit stakeholder needs, expectations, constraints, 
and interfaces for all phases of the product life cycle;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Validate requirements to ensure that the resulting 
product will perform as intended in the user's environment using 
multiple techniques as appropriate;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements development process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements development process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements development process, developing the work products, and 
providing the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements development process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
requirements development process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Place designated work products of the requirements 
development process under appropriate levels of configuration 
management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements development process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the requirements development 
process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice partially implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
requirements development process with higher level management, and 
resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[A] This goal is repeated at both capability levels 1 and 2, but it 
comprises more advanced practices at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 32 through 39. Specifically, 
tables 32 and 33 provide results for VSCS; tables 34 and 35 provide 
results for ERAM; tables 36 and 37 provide results for ITWS; and tables 
38 and 39 provide results for ASDE-X.

Table 32: VSCS Requirements Development: Detailed Findings on Level 1 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Identify and collect stakeholder needs, expectations, 
constraints, and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Desired practice: Transform stakeholder needs, expectations, 
constraints, and interfaces into customer requirements;
Rating: Largely implemented;
Comment: The project team usually--but not always--transforms 
stakeholder needs, expectations, constraints, and interfaces into 
customer requirements.

Goal: Customer requirements are refined and elaborated to develop 
product and product-component requirements;
Rating: Satisfied.

Desired practice: Establish and maintain product and product-component 
requirements that are based on the customer requirements;
Rating: Largely implemented;
Comment: The project team has established and usually--but not always-
-maintains product and product-component requirements that are based on 
the customer requirements.

Desired practice: Allocate the requirements for each product component;
Rating: Largely implemented;
Comment: The project team usually--but not always--allocates the 
requirements for each product component.

Desired practice: Identify interface requirements;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Establish and maintain operational concepts and 
associated scenarios;
Rating: Fully implemented.

Desired practice: Establish and maintain a definition of required 
functionality;
Rating: Largely implemented;
Comment: The project team has established and usually--but not always-
-maintains a definition of required functionality.

Desired practice: Analyze requirements to ensure that they are 
necessary and sufficient;
Rating: Largely implemented;
Comment: The project team usually--but not always--analyzes 
requirements to ensure that they are necessary and sufficient.

Desired practice: Validate requirements to ensure that the resulting 
product will perform appropriately in its intended-use environment;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 33: VSCS Requirements Development: Detailed Findings on Level 2 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Elicit stakeholder needs, expectations, constraints, 
and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Validate requirements to ensure that the resulting 
product will perform as intended in the user's environment using 
multiple techniques as appropriate;
Rating: Largely implemented;
Comment: The project team usually--but not always--validates 
requirements to ensure that the resulting product will perform as 
intended in the user's environment using multiple techniques as 
appropriate.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements development process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements development process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements development process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements development process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements development process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the requirements 
development process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements development process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements development 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Partially implemented;
Comment: The project team does not consistently monitor and control the 
requirements development process against the plan for performing the 
process and take appropriate corrective action, although it does some 
of these activities.

Desired practice: Objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
requirements development process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 34: ERAM Requirements Development: Detailed Findings on Level 1 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Identify and collect stakeholder needs, expectations, 
constraints, and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Desired practice: Transform stakeholder needs, expectations, 
constraints, and interfaces into customer requirements;
Rating: Fully implemented.

Goal: Customer requirements are refined and elaborated to develop 
product and product-component requirements;
Rating: Satisfied.

Desired practice: Establish and maintain product and product-component 
requirements that are based on the customer requirements;
Rating: Fully implemented.

Desired practice: Allocate the requirements for each product component;
Rating: Fully implemented.

Desired practice: Identify interface requirements;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Establish and maintain operational concepts and 
associated scenarios;
Rating: Fully implemented.

Desired practice: Establish and maintain a definition of required 
functionality;
Rating: Fully implemented.

Desired practice: Analyze requirements to ensure that they are 
necessary and sufficient;
Rating: Fully implemented.

Desired practice: Validate requirements to ensure that the resulting 
product will perform appropriately in its intended-use environment;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 35: ERAM Requirements Development: Detailed Findings on Level 2 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Elicit stakeholder needs, expectations, constraints, 
and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Validate requirements to ensure that the resulting 
product will perform as intended in the user's environment using 
multiple techniques as appropriate;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements development process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements development process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements development process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements development process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements development process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the requirements 
development process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements development process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements development 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team monitors the requirements development process 
against the plan by reviewing periodic data; however, the project team 
does not always use these data to control the requirements development 
process.

Desired practice: Objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the requirements development process to its description, standards, 
and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements development process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 36: ITWS Requirements Development: Detailed Findings on Level 1 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Identify and collect stakeholder needs, expectations, 
constraints, and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Desired practice: Transform stakeholder needs, expectations, 
constraints, and interfaces into customer requirements;
Rating: Fully implemented.

Goal: Customer requirements are refined and elaborated to develop 
product and product-component requirements;
Rating: Satisfied.

Desired practice: Establish and maintain product and product-component 
requirements that are based on the customer requirements;
Rating: Fully implemented.

Desired practice: Allocate the requirements for each product component;
Rating: Fully implemented.

Desired practice: Identify interface requirements;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Establish and maintain operational concepts and 
associated scenarios;
Rating: Fully implemented.

Desired practice: Establish and maintain a definition of required 
functionality;
Rating: Fully implemented.

Desired practice: Analyze requirements to ensure that they are 
necessary and sufficient;
Rating: Fully implemented.

Desired practice: Validate requirements to ensure that the resulting 
product will perform appropriately in its intended-use environment;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 37: ITWS Requirements Development: Detailed Findings on Level 2 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Elicit stakeholder needs, expectations, constraints, 
and interfaces for all phases of the product life cycle;
Rating: Largely implemented;
Comment: The project team has outlined a process for eliciting 
stakeholder needs, expectations, constraints, and interfaces for all 
phases of the product life cycle through the system engineering plan; 
however, the team did not provide evidence that this practice is 
performed.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Validate requirements to ensure that the resulting 
product will perform as intended in the user's environment using 
multiple techniques as appropriate;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements development process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements development process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements development process, developing the work products, and 
providing the services of the process;
Rating: Largely implemented;
Comment: The project team has adequate personnel resources; however, 
the project team does not have adequate software resources for 
performing and providing the services of the requirements development 
process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements development process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements development process as needed;
Rating: Partially implemented;
Comment: The project team did not provide adequate supporting evidence 
that FAA personnel are trained on performing or supporting the 
requirements development process as needed.

Desired practice: Place designated work products of the requirements 
development process under appropriate levels of configuration 
management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
requirements development process under some levels of configuration 
management; however, the controls on these products are not adequate 
because all support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements development process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements development 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team is using metrics for monitoring the process, 
and project officials reported that the team was controlling the 
process; however, the team did not provide sufficient evidence that it 
is controlling the requirements development process against the plan 
and taking appropriate corrective action.

Desired practice: Objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements development process with higher level management, and 
resolve issues;
Rating: Largely implemented;
Comment: At the beginning of the project, the project team reviewed the 
status and the results of the requirements development process during 
the product team meetings. However, the project team did not provide 
adequate support that the status of the requirements development 
process was currently being reviewed.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 38: ASDE-X Requirements Development: Detailed Findings on Level 1 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Identify and collect stakeholder needs, expectations, 
constraints, and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Desired practice: Transform stakeholder needs, expectations, 
constraints, and interfaces into customer requirements;
Rating: Fully implemented.

Goal: Customer requirements are refined and elaborated to develop 
product and product-component requirements;
Rating: Satisfied.

Desired practice: Establish and maintain product and product-component 
requirements that are based on the customer requirements;
Rating: Fully implemented.

Desired practice: Allocate the requirements for each product component;
Rating: Fully implemented.

Desired practice: Identify interface requirements;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Establish and maintain operational concepts and 
associated scenarios;
Rating: Fully implemented.

Desired practice: Establish and maintain a definition of required 
functionality;
Rating: Fully implemented.

Desired practice: Analyze requirements to ensure that they are 
necessary and sufficient;
Rating: Fully implemented.

Desired practice: Validate requirements to ensure that the resulting 
product will perform appropriately in its intended-use environment;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 39: ASDE-X Requirements Development: Detailed Findings on Level 2 
Goals and Practices:

Goal: Stakeholder needs, expectations, constraints, and interfaces are 
collected and translated into customer requirements;
Rating: Satisfied.

Desired practice: Elicit stakeholder needs, expectations, constraints, 
and interfaces for all phases of the product life cycle;
Rating: Fully implemented.

Goal: The requirements are analyzed and validated, and a definition of 
required functionality is developed;
Rating: Satisfied.

Desired practice: Validate requirements to ensure that the resulting 
product will perform as intended in the user's environment using 
multiple techniques as appropriate;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements development process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements development process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements development process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements development process;
Rating: Largely implemented;
Comment: The project team assigns responsibility and authority to 
various groups to perform the requirements development process. 
However, there is no single document that identifies specific 
requirements development duties by project team individual.

Desired practice: Train the people performing or supporting the 
requirements development process as needed;
Rating: Partially implemented;
Comment: The project team is in the process of designing a training 
class, but it has not yet trained the people performing or supporting 
the requirements development process.

Desired practice: Place designated work products of the requirements 
development process under appropriate levels of configuration 
management;
Rating: Largely implemented;
Comment: The project team places designated work products of the 
requirements development process under appropriate levels of 
configuration management; however, internal documents have yet to be 
placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements development process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements development 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the requirements 
development process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements development process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 6: FAA Is Performing Requirements Management Practices, but It 
Is Not Yet Fully Managing the Process:

The purpose of requirements management is to manage the project's 
product components and to identify inconsistencies between requirements 
and the project's plans and work products. This process area includes 
managing all technical and nontechnical requirements and any changes to 
these requirements as they evolve.

As shown in figure 8, all four FAA projects satisfied all criteria for 
the "performing" capability level (level 1) in this process area, but 
none satisfied all criteria for achieving a "managed" capability level 
(level 2). While the projects had differing weaknesses that contributed 
to this result, a common weakness across most of the projects occurred 
in the area of ensuring quality assurance of the requirements 
management process, as shown in the overview in table 40. As a result 
of these weaknesses, FAA is exposed to increased risks that projects 
will not fulfill mission and user needs.

Looked at another way, of the 60 practices we evaluated in this key 
process area, FAA projects had 54 practices that were fully or largely 
implemented and 6 practices that were partially or not implemented.

Figure 8: Four Projects' Capability Levels in Requirements Management:

[See PDF for image]

[End of figure]

Table 40: Four Projects' Appraisal Results in Requirements Management:

Goal: Requirements are managed, and inconsistencies with project plans 
and work products are identified;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop an understanding with the requirements 
providers on the meaning of the requirements;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Manage changes to the requirements as they evolve 
during the project;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Identify inconsistencies between the project plans 
and work products and the requirements;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: Yes;
ITWS: Yes;
ASDE-X: Yes.

Goal: Requirements are managed, and inconsistencies with project plans 
and work products are identified.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Obtain commitment to the requirements from the 
project participants;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Maintain bidirectional traceability between the 
requirements and the project plans and work products;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements management process, developing the work products, and 
providing the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
requirements management process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Place designated work products of the requirements 
management process under appropriate levels of configuration 
management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements management process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the requirements management 
process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice partially implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
requirements management process with higher level management, and 
resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Source: GAO, SEI.

[A] This goal is repeated at both capability levels 1 and 2, but it 
comprises more advanced practices at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 41 through 48. Specifically, 
tables 41 and 42 provide results for VSCS; tables 43 and 44 provide 
results for ERAM; tables 45 and 46 provide results for ITWS; and tables 
47 and 48 provide results for ASDE-X.

Table 41: VSCS Requirements Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Develop an understanding with the requirements 
providers on the meaning of the requirements;
Rating: Fully implemented.

Desired practice: Manage changes to the requirements as they evolve 
during the project;
Rating: Fully implemented.

Desired practice: Identify inconsistencies between the project plans 
and work products and the requirements;
Rating: Largely implemented;
Comment: The project team usually--but not always--identifies 
inconsistencies between the project plans and work products and the 
requirements.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 42: VSCS Requirements Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Obtain commitment to the requirements from the 
project participants;
Rating: Fully implemented.

Desired practice: Maintain bidirectional traceability between the 
requirements and the project plans and work products;
Rating: Largely implemented;
Comment: The project team usually--but not always--maintains 
bidirectional traceability between the requirements and the project 
plans and work products.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the requirements 
management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Partially implemented;
Comment: The project team does not consistently monitor or control the 
requirements management process against the plan for performing the 
process or take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
requirements management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 43: ERAM Requirements Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Develop an understanding with the requirements 
providers on the meaning of the requirements;
Rating: Fully implemented.

Desired practice: Manage changes to the requirements as they evolve 
during the project;
Rating: Fully implemented.

Desired practice: Identify inconsistencies between the project plans 
and work products and the requirements;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 44: ERAM Requirements Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Obtain commitment to the requirements from the 
project participants;
Rating: Fully implemented.

Desired practice: Maintain bidirectional traceability between the 
requirements and the project plans and work products;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the requirements 
management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team usually--but not always--monitors and 
controls the requirements management process against the plan for 
performing the process and takes appropriate corrective action.

Desired practice: Objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the requirements management process to its process description, 
standards, and address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 45: ITWS Requirements Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Develop an understanding with the requirements 
providers on the meaning of the requirements;
Rating: Fully implemented.

Desired practice: Manage changes to the requirements as they evolve 
during the project;
Rating: Fully implemented.

Desired practice: Identify inconsistencies between the project plans 
and work products and the requirements;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 46: ITWS Requirements Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Obtain commitment to the requirements from the 
project participants;
Rating: Fully implemented.

Desired practice: Maintain bidirectional traceability between the 
requirements and the project plans and work products;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements management process, developing the work products, and 
providing the services of the process;
Rating: Largely implemented;
Comment: The project team has adequate resources to review requirements 
and assign requirements management responsibilities to resources; 
however, the project team does not have adequate software resources for 
performing and providing the services of the requirements management 
process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
requirements management process as needed;
Rating: Partially implemented;
Comment: The project team stated that training was conducted; however, 
the team did not provide sufficient evidence that FAA personnel are 
trained as needed on performing or supporting the requirements 
management process.

Desired practice: Place designated work products of the requirements 
management process under appropriate levels of configuration 
management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
requirements management process under some levels of configuration 
management; however, the controls on these products are not adequate 
because all of the support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The project team is using metrics for monitoring the process 
but did not provide sufficient evidence that it is controlling the 
requirements management process against the plan and taking appropriate 
corrective action.

Desired practice: Objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements management process with higher level management, and 
resolve issues;
Rating: Largely implemented;
Comment: At the beginning of the project, the project team reviewed the 
status and the results of the requirements management process during 
meetings. However, the project team did not provide adequate evidence 
that the status of the requirements management process was currently 
being reviewed.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 47: ASDE-X Requirements Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Develop an understanding with the requirements 
providers on the meaning of the requirements;
Rating: Fully implemented.

Desired practice: Manage changes to the requirements as they evolve 
during the project;
Rating: Fully implemented.

Desired practice: Identify inconsistencies between the project plans 
and work products and the requirements;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 48: ASDE-X Requirements Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: Requirements are managed and inconsistencies with project plans 
and work products are identified;
Rating: Satisfied.

Desired practice: Obtain commitment to the requirements from the 
project participants;
Rating: Fully implemented.

Desired practice: Maintain bidirectional traceability between the 
requirements and the project plans and work products;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the requirements management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
requirements management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
requirements management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the requirements management process;
Rating: Largely implemented;
Comment: The project team assigns responsibility and authority to 
various groups to perform the requirements management process. However, 
it does not assign duties to project team individuals.

Desired practice: Train the people performing or supporting the 
requirements management process as needed;
Rating: Largely implemented;
Comment: The project team is composed of individuals with previous 
requirements management process experience and is in the process of 
designing a requirements management training class; however, no project 
team members have received project-specific requirements management 
training.

Desired practice: Place designated work products of the requirements 
management process under appropriate levels of configuration 
management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the requirements management process under appropriate 
levels of configuration management; however, internal documents have 
yet to be placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
requirements management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the requirements management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the requirements 
management process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
requirements management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 7: FAA Is Performing Most Technical Solution Practices, but It 
Is Not Yet Fully Managing the Process:

The purpose of the technical solution process area is to design, 
develop, and implement products, product components, and product-
related life-cycle processes to meet requirements. This process 
involves evaluating and selecting solutions that potentially satisfy an 
appropriate set of allocated requirements, developing detailed designs, 
and implementing the design.

As shown in figure 9, three FAA projects satisfied all criteria for the 
"performing" capability level (level 1) in this process area. The 
fourth project would have achieved level 1 if it had performed two more 
practices (see the overview in table 49 for details). None of the four 
projects satisfied all criteria for the "managing" capability level 
(level 2). While all four projects had differing weaknesses that 
contributed to this result, common weaknesses across most of the 
projects occurred in the area of ensuring quality assurance of the 
technical solution process. As a result of this weakness, FAA is 
exposed to increased risks that projects will not meet mission needs.

Looked at another way, of the 72 practices we evaluated in this key 
process area, FAA projects had 62 practices that were fully or largely 
implemented and 10 practices that were partially or not implemented.

Figure 9: Four Projects' Capability Levels in Technical Solution:

[See PDF for image]

[End of figure]

Table 49: Four Projects' Appraisal Results in Technical Solution:

Goal: Product or product-component solutions are selected from 
alternative solutions;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop alternative solutions and selection criteria;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Select the product-component solutions that best 
satisfy the criteria established;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Goal: Product or product-component designs are developed;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop a design for the product or product 
component;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the solution for product-
component interfaces;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Product components, and associated support documentation, are 
implemented from their designs;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Implement the designs of the product components;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Develop and maintain the end-use documentation;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: Yes;
ITWS: No;
ASDE-X: Yes.

Additional needed to achieve capability level 2:

Goal: Product or product-component solutions are selected from 
alternative solutions.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop detailed alternative solutions and selection 
criteria;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Evolve the operational concept, scenarios, and 
environments to describe the conditions, operating modes, and operating 
states specific to each product component;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the technical solution process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
technical solution process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
technical solution process, developing the work products, and providing 
the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the technical solution process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Train the people performing or supporting the 
technical solution process as needed;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Place designated work products of the technical 
solution process under appropriate levels of configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
technical solution process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the technical solution process 
against the plan for performing the process, and take appropriate 
corrective action;
VSCS: Practice partially implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures, 
and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
technical solution process with higher level management, and resolve 
issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[A] This goal is repeated at both capability levels 1 and 2, but it 
comprises more advanced practices at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 50 through 57. Specifically, 
tables 50 and 51 provide results for VSCS; tables 52 and 53 provide 
results for ERAM; tables 54 and 55 provide results for ITWS; and tables 
56 and 57 provide results for ASDE-X.

Table 50: VSCS Technical Solution: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop alternative solutions and selection criteria;
Rating: Fully implemented.

Desired practice: Select the product-component solutions that best 
satisfy the criteria established;
Rating: Largely implemented;
Comment: The project team selects the product-component solutions that 
best satisfy the criteria established, but it does not provide 
sufficient detail regarding how it selected them.

Goal: Product or product-component designs are developed;
Rating: Satisfied.

Desired practice: Develop a design for the product or product 
component;
Rating: Largely implemented;
Comment: The project team usually--but not always--develops designs for 
the product and product components.

Desired practice: Establish and maintain the solution for product-
component interfaces;
Rating: Fully implemented.

Goal: Product components, and associated support documentation, are 
implemented from their designs;
Rating: Satisfied.

Desired practice: Implement the designs of the product components;
Rating: Fully implemented.

Desired practice: Develop and maintain the end-use documentation;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 51: VSCS Technical Solution: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop detailed alternative solutions and selection 
criteria;
Rating: Fully implemented.

Desired practice: Evolve the operational concept, scenarios, and 
environments to describe the conditions, operating modes, and operating 
states specific to each product component;
Rating: Largely implemented;
Comment: The project team usually--but not always--evolves the 
operational concept, scenarios, and environments to describe the 
conditions, operating modes, and operating states specific to each 
product component.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the technical solution process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
technical solution process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
technical solution process, developing the work products, and providing 
the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the technical solution process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
technical solution process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the technical 
solution process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
technical solution process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the technical solution process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Partially implemented;
Comment: The project team does not consistently monitor and control the 
technical solution process against the plan for performing the process 
or take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
technical solution process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 52: ERAM Technical Solution: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop alternative solutions and selection criteria;
Rating: Fully implemented.

Desired practice: Select the product-component solutions that best 
satisfy the criteria established;
Rating: Fully implemented.

Goal: Product or product-component designs are developed;
Rating: Satisfied.

Desired practice: Develop a design for the product or product 
component;
Rating: Fully implemented.

Desired practice: Establish and maintain the solution for product-
component interfaces;
Rating: Fully implemented.

Goal: Product components, and associated support documentation, are 
implemented from their designs;
Rating: Satisfied.

Desired practice: Implement the designs of the product components;
Rating: Largely implemented;
Comment: The project team is working on the product-component designs 
and has contracted the implementation of the designs in the statement 
of work. However, the project has not yet fully reached the design 
implementation stage.

Desired practice: Develop and maintain the end-use documentation;
Rating: Largely implemented;
Comment: The project team has defined and contracted the development 
and maintenance of the end-use documentation in the statement of work. 
However, the project has not reached the end-use documentation phase.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 53: ERAM Technical Solution: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop detailed alternative solutions and selection 
criteria;
Rating: Fully implemented.

Desired practice: Evolve the operational concept, scenarios, and 
environments to describe the conditions, operating modes, and operating 
states specific to each product component;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the technical solution process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
technical solution process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
technical solution process, developing the work products, and providing 
the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the technical solution process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
technical solution process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the technical 
solution process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
technical solution process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the technical solution process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which will objectively evaluate adherence of the 
technical solution process to its process description, standards, and 
procedures and address noncompliance.

Desired practice: Review the activities, status, and results of the 
technical solution process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 54: ITWS Technical Solution: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented.

Desired practice: Develop alternative solutions and selection criteria;
Rating: Partially implemented;
Comment: The project team developed alternative solutions; however, it 
did not develop alternative selection criteria.

Desired practice: Select the product-component solutions that best 
satisfy the criteria established;
Rating: Partially implemented;
Comment: Although the project team selected product components by 
approving critical design reviews, no criteria were established to 
choose between product-component alternatives.

Goal: Product or product-component designs are developed;
Rating: Satisfied.

Desired practice: Develop a design for the product or product 
component;
Rating: Fully implemented.

Desired practice: Establish and maintain the solution for product-
component interfaces;
Rating: Fully implemented.

Goal: Product components, and associated support documentation, are 
implemented from their designs;
Rating: Satisfied.

Desired practice: Implement the designs of the product components;
Rating: Fully implemented.

Desired practice: Develop and maintain the end-use documentation;
Rating: Fully implemented.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 55: ITWS Technical Solution: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Develop detailed alternative solutions and selection 
criteria;
Rating: Partially implemented;
Comment: The project team developed detailed alternative solutions; 
however, it did not develop detailed alternative selection criteria.

Desired practice: Evolve the operational concept, scenarios, and 
environments to describe the conditions, operating modes, and operating 
states specific to each product component;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because four of the practices below 
are partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the technical solution process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
technical solution process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
technical solution process, developing the work products, and providing 
the services of the process;
Rating: Partially implemented;
Comment: The project team provided adequate resources and documentation 
for performing the technical solution process in 1998; however, the 
documentation has not been updated since it was initially created.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the technical solution process;
Rating: Partially implemented;
Comment: The project team assigned responsibility and authority for 
performing the technical solution process and developing the work 
products in 1998; however, the documentation has not been updated since 
it was initially created.

Desired practice: Train the people performing or supporting the 
technical solution process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the technical 
solution process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
technical solution process under some levels of configuration 
management; however, the controls on these products are not adequate 
because all the support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
technical solution process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the technical solution process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures 
and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
technical solution process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 56: ASDE-X Technical Solution: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop alternative solutions and selection criteria;
Rating: Fully implemented.

Desired practice: Select the product-component solutions that best 
satisfy the criteria established;
Rating: Fully implemented.

Goal: Product or product-component designs are developed;
Rating: Satisfied.

Desired practice: Develop a design for the product or product 
component;
Rating: Fully implemented.

Desired practice: Establish and maintain the solution for product-
component interfaces;
Rating: Fully implemented.

Goal: Product components, and associated support documentation, are 
implemented from their designs;
Rating: Satisfied.

Desired practice: Implement the designs of the product components;
Rating: Fully implemented.

Desired practice: Develop and maintain the end-use documentation;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 57: ASDE-X Technical Solution: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Product or product-component solutions are selected from 
alternative solutions;
Rating: Satisfied.

Desired practice: Develop detailed alternative solutions and selection 
criteria;
Rating: Fully implemented.

Desired practice: Evolve the operational concept, scenarios, and 
environments to describe the conditions, operating modes, and operating 
states specific to each product component;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;

Finding: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the technical solution process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
technical solution process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
technical solution process, developing the work products, and providing 
the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the technical solution process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
technical solution process as needed;
Rating: Largely implemented;

Finding: The prime contractor's personnel are trained in design, 
development, and testing activities, with FAA personnel overseeing 
these activities; however, the FAA personnel are not trained in the 
technical solution process.

Desired practice: Place designated work products of the technical 
solution process under appropriate levels of configuration management;
Rating: Largely implemented;

Finding: The project team places designated work products of the 
technical solution process under appropriate levels of configuration 
management; however, internal documents are not yet placed under 
configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
technical solution process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the technical solution process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;

Finding: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the technical 
solution process to its process description, standards, and procedures 
and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
technical solution process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 8: FAA Is Performing Product Integration Practices, but It Is 
Not Yet Fully Managing the Process:

The purpose of the product integration process is to assemble the 
product components, ensure that the integrated product functions 
properly, and deliver the product. A critical aspect of this process is 
managing the internal and external interfaces of the products and 
product components, in one stage or in incremental stages.

For this process area, we did not perform an appraisal for the ERAM 
project, because it was at a stage in which product integration was not 
applicable. As shown in figure 10, the three remaining projects 
satisfied all criteria for the "performing" capability level (level 1) 
in this process area. None of the projects satisfied all criteria for 
the "managing" capability level (level 2). While the projects had 
differing weaknesses that contributed to this result, common weaknesses 
across most of the projects occurred in the areas of monitoring and 
controlling the product integration process and ensuring quality 
assurance of the process, as shown in the overview in table 58. As a 
result of this weakness, FAA is exposed to increased risk that product 
components will not be compatible, resulting in projects that will not 
meet cost, schedule, or performance goals.

Looked at another way, of the 54 practices we evaluated in this process 
area, FAA projects had 49 practices that were fully or largely 
implemented and 5 practices that were partially or not implemented.

Figure 10: Three Projects' Capability Levels in Product Integration:

[See PDF for image]

[End of figure]

Table 58: Three Projects' Appraisal Results in Product Integration:

Goal: Preparation for product integration is conducted;
VSCS: Goal satisfied;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Determine the product-component integration 
sequence;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The product-component interfaces, both internal and external, are 
compatible;
VSCS: Goal satisfied;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Review interface descriptions for coverage and 
completeness;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Manage internal and external interface definitions, 
designs, and changes for products and product components;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Verified product components are assembled, and the integrated, 
verified, and validated product is delivered;
VSCS: Goal satisfied;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Confirm, before assembly, that each product component 
required to assemble the product has been properly identified and 
functions according to its description, and that the product-component 
interfaces comply with the interface descriptions;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assemble product components according to the product 
integration sequence and available procedures;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Evaluate assembled product components for interface 
compatibility;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Package the assembled product or product component 
and deliver it to the appropriate customer;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: N/A;
ITWS: Yes;
ASDE-X: Yes.

Goal: Preparation for product integration is conducted.[A];
VSCS: Goal satisfied;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain the environment needed to 
support the integration of the product components;
VSCS: Practice largely implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: N/A;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the product integration process;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
product integration process;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the product 
integration process, developing the work products, and providing the 
services of the process;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the product integration process;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Train the people performing or supporting the product 
integration process as needed;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the product 
integration process under appropriate levels of configuration 
management;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
product integration process as planned;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the product integration process 
against the plan for performing the process, and take appropriate 
corrective action;
VSCS: Practice not implemented;
ERAM: N/A;
ITWS: Practice not implemented;
ASDE-X: Practice largely implemented.

Desired practice: Objectively evaluate adherence of the product 
integration process to its process description, standards, and 
procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
product integration process with higher level management, and resolve 
issues;
VSCS: Practice fully implemented;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: N/A;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

Note: N/A represents not applicable;

project not appraised in this process area.

[A] This goal is repeated at both capability levels 1 and 2, but it 
comprises more advanced practices at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 59 through 64. Specifically, 
tables 59 and 60 provide results for VSCS; tables 61 and 62 provide 
results for ITWS; and tables 63 and 64 provide results for ASDE-X.

Table 59: VSCS Product Integration: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Determine the product-component integration 
sequence;
Rating: Fully implemented.

Goal: The product-component interfaces, both internal and external, are 
compatible;
Rating: Satisfied.

Desired practice: Review interface descriptions for coverage and 
completeness;
Rating: Fully implemented.

Desired practice: Manage internal and external interface definitions, 
designs, and changes for products and product components;
Rating: Fully implemented.

Goal: Verified product components are assembled, and the integrated, 
verified, and validated product is delivered;
Rating: Satisfied.

Desired practice: Confirm, before assembly, that each product component 
required to assemble the product has been properly identified and 
functions according to its description and that the product-component 
interfaces comply with the interface descriptions;
Rating: Fully implemented.

Desired practice: Assemble product components according to the product 
integration sequence and available procedures;
Rating: Fully implemented.

Desired practice: Evaluate assembled product components for interface 
compatibility;
Rating: Fully implemented.

Desired practice: Package the assembled product or product component 
and deliver it to the appropriate customer;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 60: VSCS Product Integration: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support the integration of the product components;
Rating: Largely implemented;
Comment: The project team has established the environment needed to 
support the integration of the product components. However, the project 
team does not maintain this environment.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the product integration process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
product integration process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the product 
integration process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the product integration process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the product 
integration process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the product 
integration process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
product integration process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the product integration process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the product 
integration process against the plan for performing the process or take 
appropriate corrective action.

Desired practice: Objectively evaluate adherence of the product 
integration process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
product integration process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 61: ITWS Product Integration: Detailed Findings on Level 1 Goals 
and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Determine the product-component integration 
sequence;
Rating: Fully implemented.

Goal: The product-component interfaces, both internal and external, are 
compatible;
Rating: Satisfied.

Desired practice: Review interface descriptions for coverage and 
completeness;
Rating: Fully implemented.

Desired practice: Manage internal and external interface definitions, 
designs, and changes for products and product components;
Rating: Fully implemented.

Goal: Verified product components are assembled, and the integrated, 
verified, and validated product is delivered;
Rating: Satisfied.

Desired practice: Confirm, before assembly, that each product component 
required to assemble the product has been properly identified and 
functions according to its description and that the product-component 
interfaces comply with the interface descriptions;
Rating: Fully implemented.

Desired practice: Assemble product components according to the product 
integration sequence and available procedures;
Rating: Fully implemented.

Desired practice: Evaluate assembled product components for interface 
compatibility;
Rating: Fully implemented.

Desired practice: Package the assembled product or product component 
and deliver it to the appropriate customer;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 62: ITWS Product Integration: Detailed Findings on Level 2 Goals 
and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support the integration of the product components;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;

Finding: The goal is unsatisfied because two of the practices below are 
partially implemented and another is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the product integration process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
product integration process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the product 
integration process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;

Finding: The project team has conducted product integration activities, 
but the resources needed to support these activities were not defined.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the product integration process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the product 
integration process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the product 
integration process under appropriate levels of configuration 
management;
Rating: Partially implemented;

Finding: The project team places designated work products of the 
product integration process under some levels of configuration 
management; however, the controls on these products are not adequate 
because all the support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
product integration process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the product integration process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Not implemented;

Finding: The project team does not monitor and control the product 
integration process against the plan for performing the process or take 
appropriate corrective action.

Desired practice: Objectively evaluate adherence of the product 
integration process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;

Finding: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the product 
integration process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
product integration process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 63: ASDE-X Product Integration: Detailed Findings on Level 1 
Goals and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Determine the product-component integration 
sequence;
Rating: Fully implemented.

Goal: The product-component interfaces, both internal and external, are 
compatible;
Rating: Satisfied.

Desired practice: Review interface descriptions for coverage and 
completeness;
Rating: Fully implemented.

Desired practice: Manage internal and external interface definitions, 
designs, and changes for products and product components;
Rating: Fully implemented.

Goal: Verified product components are assembled, and the integrated, 
verified, and validated product is delivered;
Rating: Satisfied.

Desired practice: Confirm, before assembly, that each product component 
required to assemble the product has been properly identified and 
functions according to its description and that the product-component 
interfaces comply with the interface descriptions;
Rating: Fully implemented.

Desired practice: Assemble product components according to the product 
integration sequence and available procedures;
Rating: Fully implemented.

Desired practice: Evaluate assembled product components for interface 
compatibility;
Rating: Fully implemented.

Desired practice: Package the assembled product or product component 
and deliver it to the appropriate customer;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 64: ASDE-X Product Integration: Detailed Findings on Level 2 
Goals and Practices:

Goal: Preparation for product integration is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support the integration of the product components;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the product integration process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
product integration process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the product 
integration process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the product integration process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the product 
integration process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the product 
integration process under appropriate levels of configuration 
management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the product integration process under appropriate levels of 
configuration management; however, internal documents have yet to be 
placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
product integration process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the product integration process 
against the plan for performing the process, and take appropriate 
corrective action;
Rating: Largely implemented;
Comment: The project team monitors and controls the product integration 
process against the plan for performing the process; however, the 
metrics are collected by the contractor, and the project team does not 
maintain, record, or track the metrics on a monthly basis.

Desired practice: Objectively evaluate adherence of the product 
integration process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the project 
integration process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
product integration process with higher level management, and resolve 
issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 9: FAA Is Not Performing Key Verification Practices or Fully 
Managing the Process:

The purpose of verification is to ensure that selected work products 
meet their specified requirements. This process area involves preparing 
for and performing tests and identifying corrective actions. 
Verification of work products substantially increases the likelihood 
that the product will meet the customer, product, and product-component 
requirements.

As shown in figure 11, only one of four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. As shown in the overview in table 65, key weaknesses in 
preparing and conducting peer reviews prevented the other three 
projects from achieving level 1. None of the four projects satisfied 
all criteria for the "managing" capability level (level 2). While all 
four projects had differing weaknesses that contributed to this result, 
common weaknesses across most of the projects occurred in the areas of 
monitoring and controlling the verification process and in ensuring 
quality assurance of the process. As a result of these weaknesses, FAA 
is exposed to increased risk that the product will not meet the user 
and mission requirements, increasing the likelihood that projects that 
will not meet cost, schedule, or performance goals.

Looked at another way, of the 68 practices we evaluated in this process 
area, FAA projects had 51 practices that were fully or largely 
implemented and 17 practices that were partially or not implemented.

Figure 11: Four Projects' Capability Levels in Verification:

[See PDF for image]

[End of figure]

Table 65: Four Projects' Appraisal Results in Verification:

Goal: Preparation for verification is conducted;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Select the work products to be verified and the 
verification methods that will be used for each;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Goal: Peer reviews are performed on selected work products;
VSCS: Goal satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Prepare for peer reviews of selected work products;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice not implemented.

Desired practice: Conduct peer reviews on selected work products and 
identify issues resulting from the peer review;
VSCS: Practice largely implemented;
ERAM: Practice partially implemented;
ITWS: Practice not implemented;
ASDE-X: Practice not implemented.

Goal: Selected work products are verified against their specified 
requirements;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Perform verification on the selected work products;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: No;
ITWS: No;
ASDE-X: No.

Goal: Preparation for verification is conducted.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain the environment needed to 
support verification;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Goal: Peer reviews are performed on selected work products.[A];
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Analyze data about preparation, conduct, and results 
of the peer reviews;
VSCS: Practice partially implemented;
ERAM: Practice not implemented;
ITWS: Practice not implemented;
ASDE-X: Practice not implemented.

Goal: Selected work products are verified against their specified 
requirements.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Analyze the results of all verification activities 
and identify corrective action;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the verification process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
verification process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Provide adequate resources for performing the 
verification process, developing the work products, and providing the 
services of the process;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the verification process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
verification process as needed;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the verification 
process under appropriate levels of configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
verification process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the verification process against 
the plan for performing the process, and take appropriate corrective 
action;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice not implemented;
ASDE-X: Practice not implemented.

Desired practice: Objectively evaluate adherence of the verification 
process to its process description, standards, and procedures, and 
address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
verification process with higher level management, and resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[A] This goal is repeated at both capability levels 1 and 2, but it 
comprises more advanced practices at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 66 through 73. Specifically, 
tables 66 and 67 provide results for VSCS; tables 68 and 69 provide 
results for ERAM; tables 70 and 71 provide results for ITWS; and tables 
72 and 73 provide results for ASDE-X.

Table 66: VSCS Verification: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Select the work products to be verified and the 
verification methods that will be used for each;
Rating: Fully implemented.

Goal: Peer reviews are performed on selected work products;
Rating: Satisfied.

Desired practice: Prepare for peer reviews of selected work products;
Rating: Fully implemented.

Desired practice: Conduct peer reviews on selected work products and 
identify issues resulting from the peer review;
Rating: Largely implemented;
Comment: The project team usually--but not always--conducts peer 
reviews on selected work products and identifies issues resulting from 
the peer review.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Perform verification on the selected work products;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 67: VSCS Verification: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support verification;
Rating: Fully implemented.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the practice below is 
partially implemented.

Desired practice: Analyze data about preparation, conduct, and results 
of the peer reviews;
Rating: Partially implemented;
Comment: The project team analyzes data about the results of peer 
reviews, but it does not collect or record data about the preparation 
or conduct of the peer reviews.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Analyze the results of all verification activities 
and identify corrective action;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the verification process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
verification process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
verification process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the verification process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
verification process as needed;
Rating: Largely implemented;
Comment: The project team trains the people performing and supporting 
the process, but does not consistently document individuals' training 
to ensure that standards are met.

Desired practice: Place designated work products of the verification 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
verification process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the verification process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: The project team does not monitor and control the verification 
process against the plan in order to take appropriate corrective 
action.

Desired practice: Objectively evaluate adherence of the verification 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
verification process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 68: ERAM Verification: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Select the work products to be verified and the 
verification methods that will be used for each;
Rating: Largely implemented;
Comment: Although the project team does not verify internally generated 
documents, it selects other work products to be verified and the 
verification methods that will be used.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Prepare for peer reviews of selected work products;
Rating: Largely implemented;
Comment: The project team is preparing for peer reviews of software; 
however, other peer reviews for selected work products have not yet 
been addressed.

Desired practice: Conduct peer reviews on selected work products and 
identify issues resulting from the peer review;
Rating: Partially implemented;
Comment: The project has awarded a contract to conduct peer reviews, 
but has not conducted any peer reviews to date.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Perform verification on the selected work products;
Rating: Largely implemented;
Comment: The project team has performed verification on selected 
documents delivered under contract deliverable requirements, but not on 
internal documents. The project has not reached the verification 
testing phase.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 69: ERAM Verification: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support verification;
Rating: Largely implemented;
Comment: The project team has established and maintains a generic 
environment needed to support verification of work products, but 
detailed environments for verification will be developed and delivered 
later in the program.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the practice below is not 
implemented.

Desired practice: Analyze data about preparation, conduct, and results 
of the peer reviews;
Rating: Not implemented;
Comment: The project team has not yet received any software peer review 
data to analyze.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Analyze the results of all verification activities 
and identify corrective action;
Rating: Largely implemented;
Comment: The project team analyzes the results of the verification 
activities and identifies corrective actions for selected documents 
delivered under contract deliverable requirements; however, the team 
does not analyze or identify corrective actions on internal documents.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the verification process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
verification process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
verification process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team has provided most of the resources for 
performing the verification process, developing the work products, and 
providing the services of the process; however, the project team does 
not explicitly describe the resources required for the testing 
environment.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the verification process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
verification process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the verification 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
verification process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the verification process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Largely implemented;
Comment: The project team monitors the process by looking at periodic 
data on change request activity, but it does not always use these data 
to control the verification process.

Desired practice: Objectively evaluate adherence of the verification 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the verification process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
verification process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 70: ITWS Verification: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Select the work products to be verified and the 
verification methods that will be used for each;
Rating: Largely implemented;
Comment: The project team selects some work products, including 
contractor documents and software products, to be verified and the 
verification methods that will be used for each. The project team does 
not select or verify internally generated documents.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented and the other is not implemented.

Desired practice: Prepare for peer reviews of selected work products;
Rating: Partially implemented;
Comment: The project team has just established, but not yet 
implemented, a peer review plan that is expected to prepare for the 
peer review of selected products.

Desired practice: Conduct peer reviews on selected work products and 
identify issues resulting from the peer review;
Rating: Not implemented;
Comment: The project team has not yet conducted peer reviews on 
selected work products or identified issues resulting from the peer 
review.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Perform verification on the selected work products;
Rating: Largely implemented;
Comment: The project team performs verification on the software and 
contractor delivered documents; however, it does not verify internally 
generated work products.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 71: ITWS Verification: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support verification;
Rating: Largely implemented;
Comment: The project team has established and maintains the environment 
needed to support verification for the software and contractor 
products. However, the environment to support the verification of 
internally generated documents has not been established.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the practice below is not 
implemented.

Desired practice: Analyze data about preparation, conduct, and results 
of the peer reviews;
Rating: Not implemented;
Comment: The project team has not yet identified how it will analyze 
data about the preparation, conduct, and results of the peer reviews.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Analyze the results of all verification activities 
and identify corrective action;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the verification process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
verification process;
Rating: Largely implemented;
Comment: The project team has established and maintains the plan for 
performing the verification process for software and contractor 
delivered work products. However, the plan for internally generated 
work products has not been established or maintained.

Desired practice: Provide adequate resources for performing the 
verification process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate personnel resources for 
performing the verification process, developing the work products, and 
providing the services of the process for software and contractor 
delivered work products. However, there is no evidence of software 
resources (tools) for tracking the software verification efforts.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the verification process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
verification process as needed;
Rating: Partially implemented;
Comment: The project team reported that it hires trained contractors. 
However, the project team does not train the people performing or 
supporting the verification process as needed.

Desired practice: Place designated work products of the verification 
process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
verification process under some levels of configuration management; 
however, the controls on these products are not adequate because all 
support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
verification process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the verification process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: Although the project team monitors the verification progress 
through the program management review briefings and other mechanisms 
for documentation, there are no measures used by the project team to 
monitor or control the process against the plan for performing the 
process.

Desired practice: Objectively evaluate adherence of the verification 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the verification 
process to its process description, standards, and procedures and to 
address noncompliance.

Desired practice: Review the activities, status, and results of the 
verification process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 72: ASDE-X Verification: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Select the work products to be verified and the 
verification methods that will be used for each;
Rating: Largely implemented;
Comment: The project team performs verification on the software and 
contractor delivered documents; however, it does not verify internally 
generated work products.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the two practices below are 
not implemented.

Desired practice: Prepare for peer reviews of selected work products;
Rating: Not implemented;
Comment: The project team has not prepared for peer reviews of selected 
work products.

Desired practice: Conduct peer reviews on selected work products and 
identify issues resulting from the peer review;
Rating: Not implemented;
Comment: The project team has not conducted peer reviews on selected 
work products or identified issues resulting from the peer review.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Perform verification on the selected work products;
Rating: Largely implemented;
Comment: The project team performs verification on the software and 
contractor-delivered documents; however, it does not verify internally 
generated work products.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 73: ASDE-X Verification: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for verification is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support verification;
Rating: Largely implemented;
Comment: The project team has established and maintains the environment 
needed to support verification of the software at a high level; 
however, detailed environments for verification have not been 
established. In addition, the project team has not yet established an 
environment to support the verification of internally generated 
documents.

Goal: Peer reviews are performed on selected work products;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the practice below is not 
implemented.

Desired practice: Analyze data about preparation, conduct, and results 
of the peer reviews;
Rating: Not implemented;
Comment: The project team has not analyzed data about preparation, 
conduct, and results of the peer reviews.

Goal: Selected work products are verified against their specified 
requirements;
Rating: Satisfied.

Desired practice: Analyze the results of all verification activities 
and identify corrective action;
Rating: Largely implemented;
Comment: The project team analyzes the results of the software and 
contractor-delivered documentation verification activities and 
identifies corrective actions needed.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented and another is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the verification process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
verification process;
Rating: Largely implemented;
Comment: The project team has established and maintains the plan for 
performing the verification process for software and contractor 
delivered work products. However, the plan for internally generated 
work products has not been established or maintained.

Desired practice: Provide adequate resources for performing the 
verification process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate personnel resources for 
performing the verification process, developing the work products, and 
providing the services of the process for software and other work 
products. However, the project team did not provide evidence of 
adequate software resources for performing the verification process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the verification process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the 
verification process to various groups, but it does not assign duties 
to individuals.

Desired practice: Train the people performing or supporting the 
verification process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the verification 
process under appropriate levels of configuration management;
Rating: Largely implemented;
Comment: The project team places designated work products of the 
process under appropriate levels of configuration management; however, 
internal documents have yet to be placed under configuration management 
control.

Desired practice: Identify and involve the relevant stakeholders of the 
verification process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the verification process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: The project team does not monitor and control the verification 
process against the plan for performing the process or take appropriate 
corrective action.

Desired practice: Objectively evaluate adherence of the verification 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the verification 
process to its process description, standards, and procedures and to 
address noncompliance.

Desired practice: Review the activities, status, and results of the 
verification process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 10: FAA Is Performing Validation Practices, but It Is Not Yet 
Fully Managing the Process:

The purpose of validation is to demonstrate that a product or product 
component fulfills its intended use when placed in its intended 
environment. Validation activities are vital to ensuring that the 
products are suitable for use in their intended operating environment.

As shown in figure 12, all four FAA projects satisfied all criteria for 
the "performing" capability level (level 1) in this process area. None 
of the four projects satisfied all criteria for the "managing" 
capability level (level 2). While all four projects had differing 
weaknesses that contributed to this result, common weaknesses across 
most of the projects occurred in the areas of monitoring and 
controlling the validation process and in ensuring quality assurance of 
the process, as shown in the overview in table 74. As a result of these 
weaknesses, FAA is exposed to increased risk that the project will not 
fulfill its intended use, thereby increasing the likelihood that the 
projects will not meet cost, schedule, or performance goals.

Looked at another way, of the 56 practices we evaluated in this process 
area, FAA projects had 47 practices that were fully or largely 
implemented and 9 practices that were partially or not implemented.

Figure 12: Four Projects' Capability Levels in Validation:

[See PDF for image]

[End of figure]

Table 74: Four Projects' Appraisal Results in Validation:

Goal: Preparation for validation is conducted;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Select products and product components to be 
validated and the validation methods that will be used for each;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: The product or product components are validated to ensure that 
they are suitable for use in their intended operating environment;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Perform validation on the selected products and 
product components;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Analyze the results of the validation activities and 
identify issues;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: Yes;
ITWS: Yes;
ASDE-X: Yes.

Goal: Preparation for validation is conducted.[A];
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain the environment needed to 
support validation;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the validation process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
validation process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
validation process, developing the work products, and providing the 
services of the process;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the validation process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
validation process as needed;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the validation 
process under appropriate levels of configuration management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
validation process as planned;
VSCS: Practice largely implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the validation process against 
the plan for performing the process, and take appropriate corrective 
action;
VSCS: Practice not implemented;
ERAM: Practice not implemented;
ITWS: Practice not implemented;
ASDE-X: Practice not implemented.

Desired practice: Objectively evaluate adherence of the validation 
process to its process description, standards, and procedures, and 
address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
validation process with higher level management, and resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[A] This goal is repeated at both levels 1 and 2, but it requires a 
more advanced practice at level 2.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 75 through 82. Specifically, 
tables 75 and 76 provide results for VSCS; tables 77 and 78 provide 
results for ERAM; tables 79 and 80 provide results for ITWS; and tables 
81 and 82 provide results for ASDE-X.

Table 75: VSCS Validation: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Select products and product components to be 
validated and the validation methods that will be used for each;
Rating: Fully implemented.

Goal: The product or product components are validated to ensure that 
they are suitable for use in their intended operating environment;
Rating: Satisfied.

Desired practice: Perform validation on the selected products and 
product components;
Rating: Fully implemented.

Desired practice: Analyze the results of the validation activities and 
identify issues;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 76: VSCS Validation: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support validation;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the validation process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
validation process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
validation process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the validation process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
validation process as needed;
Rating: Largely implemented;
Comment: The project team generally trains the people performing and 
supporting the process but does not consistently document individuals' 
training to ensure standards are met.

Desired practice: Place designated work products of the validation 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
validation process as planned;
Rating: Fully implemented;
Comment: The project team identifies and involves the relevant 
stakeholders of the validation process as planned.

Desired practice: Monitor and control the validation process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: The project team does not monitor and control the validation 
process against the plan for performing the process or take appropriate 
corrective action.

Desired practice: Objectively evaluate adherence of the validation 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
validation process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 77: ERAM Validation: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Select products and product components to be 
validated and the validation methods that will be used for each;
Rating: Fully implemented.

Goal: The product or product components are validated to ensure that 
they are suitable for use in their intended operating environment;
Rating: Satisfied.

Desired practice: Perform validation on the selected products and 
product components;
Rating: Largely implemented;
Comment: The project team has started the validation on the selected 
product and product components through extensive validation plans. 
However, the project is not yet in the validation phase.

Desired practice: Analyze the results of the validation activities and 
identify issues;
Rating: Largely implemented;
Comment: The project team has started to analyze the results of the 
validation activities and identify issues. However, the project is not 
yet in the validation phase.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 78: ERAM Validation: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support validation;
Rating: Largely implemented;
Comment: The project team establishes a general environment needed to 
support validation of work products. However, detailed environments 
have not yet been developed.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the validation process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
validation process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
validation process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project has provided most of the resources for performing 
the validation process, developing the work products, and providing the 
services of the process; however, the project does not explicitly 
describe the resources required for the testing environment.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the validation process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
validation process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the validation 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
validation process as planned;
Rating: Largely implemented;
Comment: The project team has identified and plans to involve the 
relevant stakeholders of the validation process, but it has not yet 
done so.

Desired practice: Monitor and control the validation process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: The project team does not monitor and control its validation 
planning process.

Desired practice: Objectively evaluate adherence of the validation 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the validation process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
validation process with higher level management, and resolve issues;
Rating: Largely implemented;
Comment: The project team reviews test planning activities for the 
validation process with higher level management; however, the 
validation process has not yet reached the stage where validation 
issues need to be addressed and resolved.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 79: ITWS Validation: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Select products and product components to be 
validated and the validation methods that will be used for each;
Rating: Fully implemented.

Goal: The product or product components are validated to ensure that 
they are suitable for use in their intended operating environment;
Rating: Satisfied.

Desired practice: Perform validation on the selected products and 
product components;
Rating: Fully implemented.

Desired practice: Analyze the results of the validation activities and 
identify issues;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 80: ITWS Validation: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support validation;
Rating: Fully implemented.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented, and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the validation process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
validation process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
validation process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate personnel resources for 
performing the validation process, developing the work products, and 
providing the services of the process. However, team lacks the software 
resources (tools) to track the software validation efforts.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the validation process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
validation process as needed;
Rating: Partially implemented;
Comment: The project team reported that it hires trained contractors. 
However, the project team does not train the people performing or 
supporting the validation process as needed.

Desired practice: Place designated work products of the validation 
process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
validation process under some levels of configuration management; 
however, the controls on these products are not adequate because all 
support contractors have access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
validation process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the validation process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: Although the project team monitors the verification progress 
through the program management review briefings, there are no measures 
used by the project team to monitor or control the process against the 
plan for performing the process or to take appropriate corrective 
action.

Desired practice: Objectively evaluate adherence of the validation 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the validation 
process to its process description, standards, and procedures and to 
address noncompliance.

Desired practice: Review the activities, status, and results of the 
validation process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 81: ASDE-X Validation: Detailed Findings on Level 1 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Select products and product components to be 
validated and the validation methods that will be used for each;
Rating: Fully implemented.

Goal: The product or product components are validated to ensure that 
they are suitable for use in their intended operating environment;
Rating: Satisfied.

Desired practice: Perform validation on the selected products and 
product components;
Rating: Fully implemented.

Desired practice: Analyze the results of the validation activities and 
identify issues;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 82: ASDE-X Validation: Detailed Findings on Level 2 Goals and 
Practices:

Goal: Preparation for validation is conducted;
Rating: Satisfied.

Desired practice: Establish and maintain the environment needed to 
support validation;
Rating: Largely implemented;
Comment: The project team has established and maintains the environment 
needed to support validation of the software at a high level; however, 
detailed environments for validation are not yet established.

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented and another is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the validation process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
validation process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
validation process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate personnel resources for 
performing the validation process, developing the work products, and 
providing the services of the process. However, the project team did 
not provide evidence of software resources.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the validation process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility and authority for 
performing the process, developing the work products, and providing the 
services of the validation process; however, it does not assign duties 
to individuals.

Desired practice: Train the people performing or supporting the 
validation process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the validation 
process under appropriate levels of configuration management;
Rating: Largely implemented;
Comment: The project team places designated work products of the 
process under appropriate levels of configuration management; however, 
internal documents have yet to be placed under configuration management 
control.

Desired practice: Identify and involve the relevant stakeholders of the 
validation process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the validation process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Not implemented;
Comment: The project team does not monitor and control the validation 
process against the plan for performing the process or take appropriate 
corrective action.

Desired practice: Objectively evaluate adherence of the validation 
process to its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the validation 
process to its process description, standards, and procedures and to 
address noncompliance.

Desired practice: Review the activities, status, and results of the 
validation process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 11: FAA Is Performing Most Configuration Management Practices, 
but It Is Not Yet Fully Managing the Process:

The purpose of configuration management is to establish and maintain 
the integrity of work products. This process area includes both the 
functional processes used to establish and track work product changes 
and the technical systems used to manage these changes. Through 
configuration management, accurate status and data are provided to 
developers, end users, and customers.

As shown in figure 13, three of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. The fourth project would have achieved level 1 if it had 
performed two more practices (see the overview in table 83 for 
details). Only one of the four projects satisfied all criteria for the 
"managing" capability level (level 2). While all four projects had 
differing weaknesses that contributed to this result, common weaknesses 
across some of the projects occurred in the areas of monitoring and 
controlling the process and in ensuring the quality assurance of the 
configuration management process, as shown in the overview in table 83. 
As a result of these weaknesses, FAA is exposed to increased risk that 
the project teams will not effectively manage their work products, 
resulting in projects that do not meet cost, schedule, or performance 
goals.

Looked at another way, of the 68 practices we evaluated in this process 
area, FAA projects had 60 practices that were fully or largely 
implemented and 8 practices that were partially or not implemented.

Figure 13: Four Projects' Capability Levels in Configuration 
Management:

[See PDF for image]

[End of figure]

Table 83: Four Projects' Appraisal Results in Configuration Management:

Goal: Baselines of identified work products are established;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Identify the configuration items, components, and 
related work products that will be placed under configuration 
management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain a configuration management and 
change management system for controlling work products;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Create or release baselines for internal use and for 
delivery to the customer;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Goal: Changes to the work products under configuration management are 
tracked and controlled;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Track change requests for the configuration items;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Control changes to the configuration items;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice not implemented;
ASDE-X: Practice fully implemented.

Goal: Integrity of baselines is established and maintained;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain records describing 
configuration items;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Perform configuration audits to maintain integrity of 
the configuration baselines;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: Yes;
ITWS: No;
ASDE-X: Yes.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the configuration management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
configuration management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
configuration management process, developing the work products, and 
providing the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the configuration management process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Train the people performing or supporting the 
configuration management process as needed;
VSCS: Practice largely implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the configuration 
management process under appropriate levels of configuration 
management;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
configuration management process as planned;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the configuration management 
process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice not implemented;
ERAM: Practice fully implemented;
ITWS: Practice not implemented;
ASDE-X: Practice fully implemented.

Desired practice: Objectively evaluate adherence of the configuration 
management process to its process description, standards, and 
procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Review the activities, status, and results of the 
configuration management process with higher level management, and 
resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: Yes.

Sources: GAO, SEI.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 84 through 91. Specifically, 
tables 84 and 85 provide results for VSCS; tables 86 and 87 provide 
results for ERAM; tables 88 and 89 provide results for ITWS; and tables 
90 and 91 provide results for ASDE-X.

Table 84: VSCS Configuration Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Baselines of identified work products are established;
Rating: Satisfied.

Desired practice: Identify the configuration items, components, and 
related work products that will be placed under configuration 
management;
Rating: Fully implemented.

Desired practice: Establish and maintain a configuration management and 
change management system for controlling work products;
Rating: Fully implemented.

Desired practice: Create or release baselines for internal use and for 
delivery to the customer;
Rating: Fully implemented.

Goal: Changes to the work products under configuration management are 
tracked and controlled;
Rating: Satisfied.

Desired practice: Track change requests for the configuration items;
Rating: Fully implemented.

Desired practice: Control changes to the configuration items;
Rating: Fully implemented.

Goal: Integrity of baselines is established and maintained;
Rating: Satisfied.

Desired practice: Establish and maintain records describing 
configuration items;
Rating: Fully implemented.

Desired practice: Perform configuration audits to maintain integrity of 
the configuration baselines;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 85: VSCS Configuration Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the configuration management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
configuration management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
configuration management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the configuration management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
configuration management process as needed;
Rating: Largely implemented;
Comment: The project team generally trains the people performing and 
supporting the configuration management process, but does not document 
individuals' training to ensure standards are met.

Desired practice: Place designated work products of the configuration 
management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
configuration management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the configuration management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the 
configuration management process against the plan for performing the 
process.

Desired practice: Objectively evaluate adherence of the configuration 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
configuration management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 86: ERAM Configuration Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Baselines of identified work products are established;
Rating: Satisfied.

Desired practice: Identify the configuration items, components, and 
related work products that will be placed under configuration 
management;
Rating: Fully implemented.

Desired practice: Establish and maintain a configuration management and 
change management system for controlling work products;
Rating: Fully implemented.

Desired practice: Create or release baselines for internal use and for 
delivery to the customer;
Rating: Largely implemented;
Comment: The project team generally creates or releases baselines for 
internal use and for delivery to the customer; however, the team does 
not yet have baselines for the current project.

Goal: Changes to the work products under configuration management are 
tracked and controlled;
Rating: Satisfied.

Desired practice: Track change requests for the configuration items;
Rating: Fully implemented.

Desired practice: Control changes to the configuration items;
Rating: Fully implemented.

Goal: Integrity of baselines is established and maintained;
Rating: Satisfied.

Desired practice: Establish and maintain records describing 
configuration items;
Rating: Fully implemented.

Desired practice: Perform configuration audits to maintain integrity of 
the configuration baselines;
Rating: Largely implemented;
Comment: The project team performs configuration audits to maintain 
integrity of the configuration baselines; however, the current project 
has not yet been subject to audits.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 87: ERAM Configuration Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the configuration management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
configuration management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
configuration management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the configuration management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
configuration management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the configuration 
management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
configuration management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the configuration management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the configuration 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the configuration management process to its process description, 
standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
configuration management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 88: ITWS Configuration Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Baselines of identified work products are established;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Identify the configuration items, components, and 
related work products that will be placed under configuration 
management;
Rating: Fully implemented.

Desired practice: Establish and maintain a configuration management and 
change management system for controlling work products;
Rating: Partially implemented;
Comment: The project team has established and maintains a configuration 
management system for controlling work products; however, the system 
does not have adequate controls in place because all support 
contractors have full access to the system.

Desired practice: Create or release baselines for internal use and for 
delivery to the customer;
Rating: Fully implemented.

Goal: Changes to the work products under configuration management are 
tracked and controlled;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Track change requests for the configuration items;
Rating: Fully implemented.

Desired practice: Control changes to the configuration items;
Rating: Not implemented;
Comment: The project team does not control all changes to the 
configuration items.

Goal: Integrity of baselines is established and maintained;
Rating: Satisfied.

Desired practice: Establish and maintain records describing 
configuration items;
Rating: Fully implemented.

Desired practice: Perform configuration audits to maintain integrity of 
the configuration baselines;
Rating: Fully implemented.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 89: ITWS Configuration Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the configuration management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
configuration management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
configuration management process, developing the work products, and 
providing the services of the process;
Rating: Partially implemented;
Comment: The project team provides some resources, such as staffing, 
for performing the configuration management process, developing the 
work products, and providing the services of the process. However, the 
team does not provide sufficient software tools to support the 
configuration management process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the configuration management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
configuration management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the configuration 
management process under appropriate levels of configuration 
management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
configuration management process under configuration management; 
however, the controls on these products are not adequate because all 
support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
configuration management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the configuration management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the 
configuration management process against the plan for performing the 
process or take appropriate corrective action. The only metrics that 
the team records are start and stop dates of the change proposal 
process.

Desired practice: Objectively evaluate adherence of the configuration 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the 
configuration management process to its process description, standards, 
and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
configuration management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 90: ASDE-X Configuration Management: Detailed Findings on Level 1 
Goals and Practices:

Goal: Baselines of identified work products are established;
Rating: Satisfied.

Desired practice: Identify the configuration items, components, and 
related work products that will be placed under configuration 
management;
Rating: Fully implemented.

Desired practice: Establish and maintain a configuration management and 
change management system for controlling work products;
Rating: Fully implemented.

Desired practice: Create or release baselines for internal use and for 
delivery to the customer;
Rating: Fully implemented.

Goal: Changes to the work products under configuration management are 
tracked and controlled;
Rating: Satisfied.

Desired practice: Track change requests for the configuration items;
Rating: Fully implemented.

Desired practice: Control changes to the configuration items;
Rating: Fully implemented.

Goal: Integrity of baselines is established and maintained;
Rating: Satisfied.

Desired practice: Establish and maintain records describing 
configuration items;
Rating: Fully implemented.

Desired practice: Perform configuration audits to maintain integrity of 
the configuration baselines;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 91: ASDE-X Configuration Management: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the configuration management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
configuration management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
configuration management process, developing the work products, and 
providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the configuration management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
configuration management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the configuration 
management process under appropriate levels of configuration 
management;
Rating: Largely implemented;
Comment: The project team places designated contractor-related work 
products of the configuration management process under appropriate 
levels of configuration management; however, internal documents have 
yet to be placed under configuration management control.

Desired practice: Identify and involve the relevant stakeholders of the 
configuration management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the configuration management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the configuration 
management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
configuration management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 12: FAA Is Not Performing Key Process and Product Quality 
Assurance Practices or Managing the Process:

The purpose of process and product quality assurance is to provide 
staff and management with objective insights into processes and 
associated work products. This process area includes the objective 
evaluation of project processes and products against approved 
descriptions and standards. Through process and product quality 
assurance, the project is able to identify and document noncompliance 
issues and provide appropriate feedback to project members.

As shown in figure 14, only one of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. Weaknesses in the objective evaluation of designated 
performed processes, work products, and services against the applicable 
process descriptions, standards, and procedures prevented the projects 
from achieving level 1. None of the four projects satisfied all 
criteria for the "managing" capability level (level 2). Table 92 
provides an overview of our appraisal results. As shown in the table, 
while the four projects had differing weaknesses that contributed to 
this result, common weaknesses across multiple projects occurred in the 
areas of establishing a plan, providing resources, training people, 
providing configuration management, identifying stakeholders, 
monitoring and controlling the process, ensuring quality assurance, and 
reviewing the status of the quality assurance process with higher level 
managers. As a result of these weaknesses, FAA is exposed to increased 
risk that the projects will not effectively implement key management 
processes, resulting in projects that will not meet cost, schedule, or 
performance goals, and that will not meet mission needs.

Looked at another way, of the 56 practices we evaluated in this process 
area, FAA projects had 33 practices that were fully or largely 
implemented and 23 practices that were partially or not implemented.

Figure 14: Four Projects' Capability Levels in Process and Product 
Quality Assurance:

[See PDF for image]

[End of figure]

Table 92: Four Projects' Appraisal Results in Process and Product 
Quality Assurance:

Goal: Adherence of the performed process and associated work products 
and services to applicable process descriptions, standards, and 
procedures is objectively evaluated;
VSCS: Goal satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Objectively evaluate the designated performed 
processes against the applicable process descriptions, standards, and 
procedures;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Objectively evaluate the designated work products and 
services against the applicable process descriptions, standards, and 
procedures;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Goal: Noncompliance issues are objectively tracked and communicated, 
and resolution is ensured;
VSCS: Goal satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Communicate quality issues and ensure resolution of 
noncompliance issues with the staff and managers;
VSCS: Practice largely implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Establish and maintain records of the quality 
assurance activities;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Capability level 1 achieved?
VSCS: Yes;
ERAM: No;
ITWS: No;
ASDE-X: No.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the process and product quality assurance 
process;
VSCS: Practice fully implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
process and product quality assurance process;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Provide adequate resources for performing the process 
and product quality assurance process, developing the work products, 
and providing the services of the process;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice partially implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the process and product quality assurance process;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the process 
and product quality assurance process as needed;
VSCS: Practice largely implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the process and 
product quality assurance process under appropriate levels of 
configuration management;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
process and product quality assurance process as planned;
VSCS: Practice largely implemented;
ERAM: Practice largely implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice partially implemented.

Desired practice: Monitor and control the process and product quality 
assurance process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice not implemented;
ERAM: Practice not implemented;
ITWS: Practice not implemented;
ASDE-X: Practice not implemented.

Desired practice: Objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures, and address noncompliance;
VSCS: Practice fully implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
process and product quality assurance process with higher level 
management, and resolve issues;
VSCS: Practice fully implemented;
ERAM: Practice largely implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice partially implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 93 through 100. Specifically, 
tables 93 and 94 provide results for VSCS; tables 95 and 96 provide 
results for ERAM; tables 97 and 98 provide results for ITWS; and tables 
99 and 100 provide results for ASDE-X.

Table 93: VSCS Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices:

Goal: Adherence of the performed process and associated work products 
and services to applicable process descriptions, standards, and 
procedures is objectively evaluated;
Rating: Satisfied.

Desired practice: Objectively evaluate the designated performed 
processes against the applicable process descriptions, standards, and 
procedures;
Rating: Fully implemented.

Desired practice: Objectively evaluate the designated work products and 
services against the applicable process descriptions, standards, and 
procedures;
Rating: Fully implemented.

Goal: Noncompliance issues are objectively tracked and communicated, 
and resolution is ensured;
Rating: Satisfied.

Desired practice: Communicate quality issues and ensure resolution of 
noncompliance issues with the staff and managers;
Rating: Largely implemented;
Comment: The project team communicates quality issues and usually--but 
not always--ensures the resolution of noncompliance issues.

Desired practice: Establish and maintain records of the quality 
assurance activities;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 94: VSCS Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the process and product quality assurance 
process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
process and product quality assurance process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the process 
and product quality assurance process, developing the work products, 
and providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the process and product quality assurance process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the process 
and product quality assurance process as needed;
Rating: Largely implemented;
Comment: The project team generally trains the people performing and 
supporting the process and product quality assurance process, but it 
does not document individuals' training to ensure standards are met.

Desired practice: Place designated work products of the process and 
product quality assurance process under appropriate levels of 
configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
process and product quality assurance process as planned;
Rating: Largely implemented;
Comment: The project team has identified and involves the relevant 
stakeholders of the process and product quality assurance process, but 
it does not track noncompliance issues to closure.

Desired practice: Monitor and control the process and product quality 
assurance process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the process and 
product quality assurance process against the plan for performing the 
process or take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures, and address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
process and product quality assurance process with higher level 
management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 95: ERAM Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices:

Goal: Adherence of the performed process and associated work products 
and services to applicable process descriptions, standards, and 
procedures is objectively evaluated;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the two practices below are 
partially implemented.

Desired practice: Objectively evaluate the designated performed 
processes against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process, which is expected to objectively evaluate the designated 
performed processes against the applicable process descriptions, 
standards, and procedures.

Desired practice: Objectively evaluate the designated work products and 
services against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process, which is expected to objectively evaluate the designated work 
products and services against the applicable process descriptions, 
standards, and procedures.

Goal: Noncompliance issues are objectively tracked and communicated, 
and resolution is ensured;
Rating: Satisfied.

Desired practice: Communicate quality issues and ensure resolution of 
noncompliance issues with the staff and managers;
Rating: Largely implemented;
Comment: The project team communicates quality issues and ensures 
resolution of noncompliance issues with the staff and managers; 
however, the team has just started defining a quality assurance 
process.

Desired practice: Establish and maintain records of the quality 
assurance activities;
Rating: Largely implemented;
Comment: The project team has established and maintains records of the 
quality assurance activities; however, the team has just started 
defining a quality assurance process.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 96: ERAM Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented, and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the process and product quality assurance 
process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
process and product quality assurance process;
Rating: Largely implemented;
Comment: The project team has just started defining the quality 
assurance process and has established and maintains the plan for 
performing the process.

Desired practice: Provide adequate resources for performing the process 
and product quality assurance process, developing the work products, 
and providing the services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate resources for performing 
the process, developing the work products, and providing the services 
of the process; however, the team has just started defining the quality 
assurance process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the process and product quality assurance process. This 
responsibility should be performed by an independent party;
Rating: Largely implemented;
Comment: The project team has assigned responsibility and authority for 
performing the process, developing the work products, and providing the 
services of the process and product quality assurance process; however, 
the responsible party is not independent of the project team.

Desired practice: Train the people performing or supporting the process 
and product quality assurance process as needed;
Rating: Largely implemented;
Comment: The project team trains the people performing or supporting 
the process as needed; however, the team has just started defining the 
quality assurance process.

Desired practice: Place designated work products of the process and 
product quality assurance process under appropriate levels of 
configuration management;
Rating: Partially implemented;
Comment: The project team has just started defining a process that is 
expected to identify the work products of the quality assurance process 
that are to be placed under configuration control.

Desired practice: Identify and involve the relevant stakeholders of the 
process and product quality assurance process as planned;
Rating: Largely implemented;
Comment: The project team has identified and involves the relevant 
stakeholders as planned; however, the team has just started defining a 
quality assurance process.

Desired practice: Monitor and control the process and product quality 
assurance process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: Because the project team has just started defining a quality 
assurance process, there are no provisions in the current plan to 
monitor and control the process.

Desired practice: Objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process that is expected to objectively evaluate adherence of the 
process and product quality assurance process to its process 
description, standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
process and product quality assurance process with higher level 
management, and resolve issues;
Rating: Largely implemented;
Comment: The project team reviews the activities, status, and results 
of the process with higher level management and resolves issues; 
however, the team has just started defining a quality assurance 
process.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 97: ITWS Process and Product Quality Assurance: Detailed Findings 
on Level 1 Goals and Practices:

Goal: Adherence of the performed process and associated work products 
and services to applicable process descriptions, standards, and 
procedures is objectively evaluated;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the two practices below are 
partially implemented.

Desired practice: Objectively evaluate the designated performed 
processes against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team is planning, but has not yet implemented, a 
quality assurance process that is expected to objectively evaluate the 
designated performed processes against the applicable process 
descriptions, standards, and procedures.

Desired practice: Objectively evaluate the designated work products and 
services against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team is planning, but has not yet implemented, a 
quality assurance process that is expected to objectively evaluate the 
designated work products and services against the applicable process 
descriptions, standards, and procedures.

Goal: Noncompliance issues are objectively tracked and communicated, 
and resolution is ensured;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Communicate quality issues and ensure resolution of 
noncompliance issues with the staff and managers;
Rating: Partially implemented;
Comment: The project team communicates quality issues and ensures 
resolution of noncompliance issues with staff and managers. However, 
the actions are not centrally managed in the project action database.

Desired practice: Establish and maintain records of the quality 
assurance activities;
Rating: Largely implemented;
Comment: The project team has established and maintains records of the 
quality assurance activities; however, quality assurance activities are 
limited.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 98: ITWS Process and Product Quality Assurance: Detailed Findings 
on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because four of the practices below 
are partially implemented, and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the process and product quality assurance 
process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
process and product quality assurance process;
Rating: Partially implemented;
Comment: The project team has just started updating its quality 
assurance plan for performing the process and product quality assurance 
process. However, the plan is still in draft and has not yet been 
finalized.

Desired practice: Provide adequate resources for performing the process 
and product quality assurance process, developing the work products, 
and providing the services of the process;
Rating: Largely implemented;
Comment: The project team has provided adequate personnel resources for 
performing the process and product quality assurance process, 
developing the work products, and providing the services of the 
process. However, the team currently has no automated tools to assist 
in performing the process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the process and product quality assurance process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility and authority for 
performing the process, developing the work products, and providing the 
services of the process and product quality assurance process; however, 
the quality assurance point of contact is located within the project 
team and is not sufficiently objective.

Desired practice: Train the people performing or supporting the process 
and product quality assurance process as needed;
Rating: Partially implemented;
Comment: The project team trains the personnel supporting the process 
and product quality assurance process in process improvement, but not 
in product quality assurance.

Desired practice: Place designated work products of the process and 
product quality assurance process under appropriate levels of 
configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
process and product quality assurance process under some levels of 
configuration management; however, the controls on these products are 
not adequate because all support contractors have full access to the 
system.

Desired practice: Identify and involve the relevant stakeholders of the 
process and product quality assurance process as planned;
Rating: Largely implemented;
Comment: While the project team involves stakeholders in the process 
and product quality assurance process, the team did not identify a 
complete list of stakeholders.

Desired practice: Monitor and control the process and product quality 
assurance process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor or control the process and 
product quality assurance process against the plan for performing the 
process or take appropriate corrective action.

Desired practice: Objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
process and product quality assurance process with higher level 
management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 99: ASDE-X Process and Product Quality Assurance: Detailed 
Findings on Level 1 Goals and Practices:

Goal: Adherence of the performed process and associated work products 
and services to applicable process descriptions, standards, and 
procedures is objectively evaluated;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because the two practices below are 
partially implemented.

Desired practice: Objectively evaluate the designated performed 
processes against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team has just started updating a quality assurance 
process that is expected to objectively evaluate the designated 
performed processes against the applicable process descriptions, 
standards, and procedures.

Desired practice: Objectively evaluate the designated work products and 
services against the applicable process descriptions, standards, and 
procedures;
Rating: Partially implemented;
Comment: The project team has just started updating a quality assurance 
process that is expected to objectively evaluate the designated work 
products against the applicable process descriptions, standards, and 
procedures.

Goal: Noncompliance issues are objectively tracked and communicated, 
and resolution is ensured;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Communicate quality issues and ensure resolution of 
noncompliance issues with the staff and managers;
Rating: Partially implemented;
Comment: The project team has just started updating a quality assurance 
process that is expected to communicate quality issues and ensure 
resolution of noncompliance issues with the staff and managers.

Desired practice: Establish and maintain records of the quality 
assurance activities;
Rating: Largely implemented;
Comment: The project team has established and maintains records of the 
quality assurance activities; however, quality assurance activities are 
limited.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 100: ASDE-X Process and Product Quality Assurance: Detailed 
Findings on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because five of the practices below 
are partially implemented, and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the process and product quality assurance 
process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
process and product quality assurance process;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process and has begun to establish and maintain the plan for performing 
the process.

Desired practice: Provide adequate resources for performing the process 
and product quality assurance process, developing the work products, 
and providing the services of the process;
Rating: Partially implemented;
Comment: The project team has provided adequate personnel resources for 
performing the process, developing the work products, and providing the 
services of the process. However, the project team has just started 
defining a quality assurance process and currently has no tools to 
assist in performing the process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the process and product quality assurance process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the process 
and product quality assurance process to various groups, but it does 
not assign duties to individuals.

Desired practice: Train the people performing or supporting the process 
and product quality assurance process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the process and 
product quality assurance process under appropriate levels of 
configuration management;
Rating: Largely implemented;
Comment: The project team places designated work products of the 
process under appropriate levels of configuration management; however, 
internal documents have yet to be placed under configuration management 
control.

Desired practice: Identify and involve the relevant stakeholders of the 
process and product quality assurance process as planned;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process. The project team has identified stakeholders, but it has not 
yet established formal communication processes that involve the 
relevant stakeholders.

Desired practice: Monitor and control the process and product quality 
assurance process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team has just started defining a quality assurance 
process, and there are no provisions in the current plan to monitor and 
control the process.

Desired practice: Objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the process and 
product quality assurance process to its process description, 
standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
process and product quality assurance process with higher level 
management, and resolve issues;
Rating: Partially implemented;
Comment: The project team has just started defining a quality assurance 
process and has not established formal communication processes with 
higher level management.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 13: FAA Is Not Performing Most Measurement and Analysis 
Practices or Managing the Process:

The purpose of measurement and analysis is to develop and sustain a 
measurement capability that is used to support management information 
needs. This process area includes the specification of measures, data 
collection and storage, analysis techniques, and the reporting of these 
values. This process allows users to objectively plan and estimate 
project activities and identify and resolve potential issues.

As shown in figure 15, none of the four FAA projects satisfied all 
criteria for the "performing" capability level (level 1) in this 
process area. Weaknesses in managing and storing measurement data, 
measurement specifications, and analysis results kept the projects from 
achieving level 1. Further, none of the four projects satisfied all 
criteria for the "managing" capability level (level 2). As shown in the 
overview in table 101, while the four projects had differing weaknesses 
that contributed to this result, common weaknesses across multiple 
projects occurred in the areas of establishing an organizational 
policy, establishing a plan, providing resources, assigning 
responsibility, training people, configuration management, identifying 
stakeholders, monitoring and controlling the process, ensuring quality 
assurance, and reviewing status with higher level management of the 
measurement and analysis process. As a result of these weaknesses, FAA 
is exposed to increased risk that the projects will not have adequate 
estimates of work metrics or a sufficient view into actual performance. 
This increases the likelihood that projects will not meet cost, 
schedule, or performance goals, and that projects will not meet mission 
needs.

Looked at another way, of the 72 practices we evaluated in this process 
area, FAA projects had 30 practices that were fully or largely 
implemented and 42 practices that were partially or not implemented.

Figure 15: Four Projects' Capability Levels in Measurement and 
Analysis:

[See PDF for image]

[End of figure]

Table 101: Four Projects' Appraisal Results in Measurement and 
Analysis:

Goal: Measurement objectives and activities are aligned with identified 
information needs and objectives;
VSCS: Goal not satisfied;
ERAM: Goal satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain measurement objectives that 
are derived from identified information needs and objectives;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice not implemented;
ASDE-X: Practice fully implemented.

Desired practice: Specify measures to address the measurement 
objectives;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Specify how measurement data will be obtained and 
stored;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Specify how measurement data will be analyzed and 
reported;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Goal: Measurement results that address identified information needs and 
objectives are provided;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Obtain specified measurement data;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Analyze and interpret measurement data;
VSCS: Practice not implemented;
ERAM: Practice partially implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Manage and store measurement data, measurement 
specifications, and analysis results;
VSCS: Practice not implemented;
ERAM: Practice partially implemented;
ITWS: Practice not implemented;
ASDE-X: Practice partially implemented.

Desired practice: Report results of measurement and analysis activities 
to all relevant stakeholders;
VSCS: Practice not implemented;
ERAM: Practice partially implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Capability level 1 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Goal: The process is institutionalized as a managed process;
VSCS: Goal not satisfied;
ERAM: Goal not satisfied;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the measurement and analysis process;
VSCS: Practice not implemented;
ERAM: Practice fully implemented;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
measurement and analysis process;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice not implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
measurement and analysis process, developing the work products, and 
providing the services of the process;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the measurement and analysis process;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice not implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
measurement and analysis process as needed;
VSCS: Practice not implemented;
ERAM: Practice fully implemented;
ITWS: Practice not implemented;
ASDE-X: Practice largely implemented.

Desired practice: Place designated work products of the measurement and 
analysis process under appropriate levels of configuration management;
VSCS: Practice not implemented;
ERAM: Practice fully implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice largely implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
measurement and analysis process as planned;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice not implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the measurement and analysis 
process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: Practice not implemented;
ERAM: Practice not implemented;
ITWS: Practice not implemented;
ASDE-X: Practice partially implemented.

Desired practice: Objectively evaluate adherence of the measurement and 
analysis process to its process description, standards, and procedures, 
and address noncompliance;
VSCS: Practice not implemented;
ERAM: Practice largely implemented;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Review the activities, status, and results of the 
measurement and analysis process with higher level management, and 
resolve issues;
VSCS: Practice not implemented;
ERAM: Practice partially implemented;
ITWS: Practice largely implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: No;
ERAM: No;
ITWS: No;
ASDE-X: No.

Sources: GAO, SEI.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 102 through 109. Specifically, 
tables 102 and 103 provide results for VSCS; tables 104 and 105 provide 
results for ERAM; tables 106 and 107 provide results for ITWS; and 
tables 108 and 109 provide results for ASDE-X.

Table 102: VSCS Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices:

Goal: Measurement objectives and activities are aligned with identified 
information needs and objectives;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because all four of the practices 
below are not implemented.

Desired practice: Establish and maintain measurement objectives that 
are derived from identified information needs and objectives;
Rating: Not implemented;
Comment: The project team has neither established nor maintains 
measurement objectives that are derived from identified information 
needs and objectives for the project or organization.

Desired practice: Specify measures to address the measurement 
objectives;
Rating: Not implemented;
Comment: The project team does not specify measures to address the 
measurement objectives.

Desired practice: Specify how measurement data will be obtained and 
stored;
Rating: Not implemented;
Comment: The project team does not specify how measurement data will be 
obtained and stored. The project team periodically collects subjective 
data on the status of the project.

Desired practice: Specify how measurement data will be analyzed and 
reported;
Rating: Not implemented;
Comment: The project team does not specify how measurement data will be 
analyzed and reported.

Goal: Measurement results that address identified information needs and 
objectives are provided;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because all four of the practices 
below are not implemented.

Desired practice: Obtain specified measurement data;
Rating: Not implemented;
Comment: The project team does not obtain measurement data.

Desired practice: Analyze and interpret measurement data;
Rating: Not implemented;
Comment: The project team does not have any measurement data to analyze 
and interpret.

Desired practice: Manage and store measurement data, measurement 
specifications, and analysis results;
Rating: Not implemented;
Comment: The project team has no measurement data to manage, store, or 
analyze.

Desired practice: Report results of measurement and analysis activities 
to all relevant stakeholders;
Rating: Not implemented;
Comment: The project team does not report results of measurement and 
analysis activities to all relevant stakeholders.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 103: VSCS Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because none of the 10 practices below 
are implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the measurement and analysis process;
Rating: Not implemented;
Comment: The organization did not establish and maintain a policy for 
planning and performing the measurement and analysis process.

Desired practice: Establish and maintain the plan for performing the 
measurement and analysis process;
Rating: Not implemented;
Comment: The project team does not have an established plan for 
performing the measurement and analysis process.

Desired practice: Provide adequate resources for performing the 
measurement and analysis process, developing the work products, and 
providing the services of the process;
Rating: Not implemented;
Comment: The project team does not have adequate resources for 
performing the measurement and analysis process, developing the work 
products, and providing the services of the process.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the measurement and analysis process;
Rating: Not implemented;
Comment: The project team has not assigned responsibility for 
performing the measurement and analysis process.

Desired practice: Train the people performing or supporting the 
measurement and analysis process as needed;
Rating: Not implemented;
Comment: The project team has not been trained to perform the 
measurement and analysis process.

Desired practice: Place designated work products of the measurement and 
analysis process under appropriate levels of configuration management;
Rating: Not implemented;
Comment: The project team does not conduct measurement and analysis 
practices. There are no work products generated.

Desired practice: Identify and involve the relevant stakeholders of the 
measurement and analysis process as planned;
Rating: Not implemented;
Comment: The project team has not identified any stakeholders for the 
measurement and analysis process.

Desired practice: Monitor and control the measurement and analysis 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the measurement 
and analysis process. There is no tracking to the plan.

Desired practice: Objectively evaluate adherence of the measurement and 
analysis process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Not implemented;
Comment: The project team does not objectively evaluate adherence of 
the measurement and analysis process to its process description, 
standards, and procedures, and it does not address noncompliance.

Desired practice: Review the activities, status, and results of the 
measurement and analysis process with higher level management, and 
resolve issues;
Rating: Not implemented;
Comment: The project team does not review the activities, status, and 
results of the measurement and analysis process with higher level 
management.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 104: ERAM Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices:

Goal: Measurement objectives and activities are aligned with identified 
information needs and objectives;
Rating: Satisfied.

Desired practice: Establish and maintain measurement objectives that 
are derived from identified information needs and objectives;
Rating: Largely implemented;
Comment: The project team establishes and usually--but not always--
maintains measurement objectives that are derived from identified 
information needs and objectives.

Desired practice: Specify measures to address the measurement 
objectives;
Rating: Largely implemented;
Comment: The project team usually--but not always--specifies measures 
to address the measurement objectives.

Desired practice: Specify how measurement data will be obtained and 
stored;
Rating: Largely implemented;
Comment: The project team specifies how measurement data will be 
obtained and stored for each project area. However, these 
specifications were only recently identified, and the team is just 
beginning to implement them.

Desired practice: Specify how measurement data will be analyzed and 
reported;
Rating: Largely implemented;
Comment: The project team specifies how measurement data will be 
analyzed and reported for each project area. However, these 
specifications were only recently identified, and the team is just 
beginning to implement them.

Goal: Measurement results that address identified information needs and 
objectives are provided;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented.

Desired practice: Obtain specified measurement data;
Rating: Largely implemented;
Comment: The project team obtains specified measurement data for each 
project area. However, these data were only recently identified, and 
the team is just beginning to obtain them.

Desired practice: Analyze and interpret measurement data;
Rating: Partially implemented;
Comment: The project team does not consistently analyze and interpret 
measurement data for each project area.

Desired practice: Manage and store measurement data, measurement 
specifications, and analysis results;
Rating: Partially implemented;
Comment: The project team does not consistently manage and store 
measurement data, measurement specifications, and analysis results.

Desired practice: Report results of measurement and analysis activities 
to all relevant stakeholders;
Rating: Partially implemented;
Comment: The project team does not consistently report results of 
measurement and analysis activities to all relevant stakeholders.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 105: ERAM Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented, and one is not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the measurement and analysis process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
measurement and analysis process;
Rating: Largely implemented;
Comment: The project team establishes and usually--but not always--
maintains the plan for performing the process.

Desired practice: Provide adequate resources for performing the 
measurement and analysis process, developing the work products, and 
providing the services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate resources for performing 
the measurement and analysis process, developing the work products, and 
providing the services of the process. However, these resources were 
only recently identified.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the measurement and analysis process;
Rating: Largely implemented;
Comment: The project team assigns responsibility and authority for 
performing the process, developing the work products, and providing the 
services of the measurement and analysis process. However, these 
responsibilities were only recently identified, and the team is just 
beginning to carry them out.

Desired practice: Train the people performing or supporting the 
measurement and analysis process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the measurement and 
analysis process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
measurement and analysis process as planned;
Rating: Largely implemented;
Comment: The project team consistently identifies the relevant 
stakeholders of the measurement and analysis process; however, the team 
does not consistently involve the stakeholders in the process.

Desired practice: Monitor and control the measurement and analysis 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor and control the measurement 
and analysis process against the plan for performing the process or 
take appropriate action.

Desired practice: Objectively evaluate adherence of the measurement and 
analysis process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Largely implemented;
Comment: The project team recently evaluated the adherence of the 
measurement and analysis process to its process description, and 
standards, but it has just started defining its quality assurance 
process.

Desired practice: Review the activities, status, and results of the 
measurement and analysis process with higher level management, and 
resolve issues;
Rating: Partially implemented;
Comment: The project team does not consistently review the activities, 
status, and results of the measurement and analysis process with higher 
level management or resolve issues.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 106: ITWS Measurement and Analysis: Detailed Findings on Level 1 
Goals and Practices:

Goal: Measurement objectives and activities are aligned with identified 
information needs and objectives;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented, and one is not implemented.

Desired practice: Establish and maintain measurement objectives that 
are derived from identified information needs and objectives;
Rating: Not implemented;
Comment: The project team has not established or maintained measurement 
objectives that are derived from identified information needs and 
objectives.

Desired practice: Specify measures to address the measurement 
objectives;
Rating: Partially implemented;
Comment: The project team previously specified measures to address 
measurement objectives; however, the guidance for these measures is no 
longer used.

Desired practice: Specify how measurement data will be obtained and 
stored;
Rating: Partially implemented;
Comment: Although the project team is collecting data, it does not have 
procedures for data collection and storage.

Desired practice: Specify how measurement data will be analyzed and 
reported;
Rating: Partially implemented;
Comment: Although the project team collects and reports data, it has no 
procedures for analysis of these data.

Goal: Measurement results that address identified information needs and 
objectives are provided;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented, and one is not implemented.

Desired practice: Obtain specified measurement data;
Rating: Partially implemented;
Comment: Although the project team collects data, the data are not 
specified or linked to any measurement objectives.

Desired practice: Analyze and interpret measurement data;
Rating: Partially implemented;
Comment: The project team receives project data from the contractor and 
performs limited analysis on the data. However, the project team does 
not draw conclusions, conduct additional measurement and analysis as 
necessary, or refine criteria for future analysis.

Desired practice: Manage and store measurement data, measurement 
specifications, and analysis results;
Rating: Not implemented;
Comment: The project team does not manage or store measurement data to 
be used for future analysis.

Desired practice: Report results of measurement and analysis activities 
to all relevant stakeholders;
Rating: Largely implemented;
Comment: The project team communicates the collected data to all 
relevant stakeholders. However, the measurement and analysis data 
collected are limited to overall status, cost, and budget metrics.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 107: ITWS Measurement and Analysis: Detailed Findings on Level 2 
Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented, and five are not implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the measurement and analysis process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
measurement and analysis process;
Rating: Not implemented;
Comment: The project team has not established and does not maintain a 
plan for performing the measurement and analysis process.

Desired practice: Provide adequate resources for performing the 
measurement and analysis process, developing the work products, and 
providing the services of the process;
Rating: Partially implemented;
Comment: The project team has provided some personnel resources for 
performing the measurement and analysis process, developing the work 
products, and providing the services of the process. However, the 
software resources used for measurement and analysis are limited.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the measurement and analysis process;
Rating: Not implemented;
Comment: The project team has not assigned responsibility and authority 
for performing the process, developing the work products, or providing 
the services of the measurement and analysis process.

Desired practice: Train the people performing or supporting the 
measurement and analysis process as needed;
Rating: Not implemented;
Comment: The project team has not trained people performing or 
supporting the measurement and analysis process.

Desired practice: Place designated work products of the measurement and 
analysis process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
measurement and analysis process under some levels of configuration 
management; however, the controls on these products are not adequate 
because all support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
measurement and analysis process as planned;
Rating: Not implemented;
Comment: The project team has not identified or involved the relevant 
stakeholders of the measurement and analysis process as planned.

Desired practice: Monitor and control the measurement and analysis 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Not implemented;
Comment: The project team does not monitor or control the measurement 
and analysis process.

Desired practice: Objectively evaluate adherence of the measurement and 
analysis process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the measurement 
and analysis process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
measurement and analysis process with higher level management, and 
resolve issues;
Rating: Largely implemented;
Comment: The project team reviews the activities, status, and results 
of the measurement and analysis process with higher level management 
and resolves issues, but it does not collect or use a comprehensive set 
of metrics.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 108: ASDE-X Measurement and Analysis: Detailed Findings on Level 
1 Goals and Practices:

Goal: Measurement objectives and activities are aligned with identified 
information needs and objectives;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain measurement objectives that 
are derived from identified information needs and objectives;
Rating: Fully implemented.

Desired practice: Specify measures to address the measurement 
objectives;
Rating: Fully implemented.

Desired practice: Specify how measurement data will be obtained and 
stored;
Rating: Partially implemented;
Comment: Although the project team specifies how measurement data will 
be obtained, storage procedures for each project area do not provide 
ready access to the data for analysis and future use.

Desired practice: Specify how measurement data will be analyzed and 
reported;
Rating: Largely implemented;
Comment: The project team specifies how measurement data for each 
project area will be analyzed; however, documented reporting procedures 
are limited.

Goal: Measurement results that address identified information needs and 
objectives are provided;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Obtain specified measurement data;
Rating: Fully implemented.

Desired practice: Analyze and interpret measurement data;
Rating: Fully implemented.

Desired practice: Manage and store measurement data, measurement 
specifications, and analysis results;
Rating: Partially implemented;
Comment: Although the project team stores measurement data, it does not 
effectively manage this data. Specifically, the team cannot readily 
access the data for future use.

Desired practice: Report results of measurement and analysis activities 
to all relevant stakeholders;
Rating: Fully implemented.

Capability level 1 not achieved.

Sources: GAO, SEI.

[End of table]

Table 109: ASDE-X Measurement and Analysis: Detailed Findings on Level 
2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because two of the practices below are 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the measurement and analysis process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
measurement and analysis process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
measurement and analysis process, developing the work products, and 
providing the services of the process;
Rating: Largely implemented;
Comment: The project team provides resources for measurement and 
analysis activities, including data collection, analysis, and 
reporting, but it provides only limited resources for data storage and 
retrieval.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the measurement and analysis process;
Rating: Largely implemented;
Comment: The project team has responsibility and authority for 
performing the measurement and analysis process, but it has not 
assigned these duties to specific individuals.

Desired practice: Train the people performing or supporting the 
measurement and analysis process as needed;
Rating: Largely implemented;
Comment: The project team performs some training to support the 
measurement and analysis process, and more training is being 
implemented.

Desired practice: Place designated work products of the measurement and 
analysis process under appropriate levels of configuration management;
Rating: Largely implemented;
Comment: The project team places designated work products of the 
process under appropriate levels of configuration management; however, 
internal documents have yet to be placed under configuration management 
control.

Desired practice: Identify and involve the relevant stakeholders of the 
measurement and analysis process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the measurement and analysis 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Partially implemented;
Comment: Although the project team stated that it monitors and controls 
the process against the plan and takes action, the project has not yet 
resolved deficiencies in data storage, retrieval, and analysis.

Desired practice: Objectively evaluate adherence of the measurement and 
analysis process to its process description, standards, and procedures, 
and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just begun its quality assurance process, 
which is expected to objectively evaluate adherence of the measurement 
and analysis process to its process description, standards, and 
procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
measurement and analysis process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 14: FAA Is Performing Supplier Agreement Management Practices, 
but It Is Not Yet Fully Managing the Process:

The purpose of supplier agreement management is to manage the 
acquisition of products. This process area involves determining the 
type of acquisition that will be used for the products acquired, 
selecting suppliers, establishing, maintaining, and executing 
agreements, accepting delivery of acquired products, and transitioning 
acquired products to the project, among other items.

For this process area, we did not perform an appraisal for the VSCS or 
ITWS projects, because these projects were at stages in which supplier 
agreement management was not applicable. As shown in figure 16, both of 
the remaining FAA projects satisfied all criteria for the "performing" 
capability level (level 1) in this process area. One of the two 
projects satisfied all criteria for the "managing" capability level 
(level 2). In not consistently managing this process, FAA is exposed to 
increased risk that projects will not be performed in accordance with 
contractual requirements, resulting in projects that will not meet 
cost, schedule, or performance goals, and systems that will not meet 
mission needs.

Looked at another way, of the 34 practices we evaluated in this process 
area, FAA projects had 33 practices that were fully or largely 
implemented and 1 practice that was partially implemented. Table 110 
provides an overview of the appraisal results.

Figure 16: Two Projects' Capability Levels in Supplier Agreement 
Management:

[See PDF for image]

[End of figure]

Table 110: Two Projects' Appraisal Results in Supplier Agreement 
Management:

Goal: Agreements with the suppliers are established and maintained;
VSCS: N/A;
ERAM: Goal satisfied;
ITWS: N/A;
ASDE-X: Goal satisfied.

Desired practice: Determine the type of acquisition for each product or 
product component to be acquired;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Select suppliers based on an evaluation of their 
ability to meet the specified requirements and established criteria;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain formal agreements with the 
supplier;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Goal: Agreements with the suppliers are satisfied by both the project 
and the supplier;
VSCS: N/A;
ERAM: Goal satisfied;
ITWS: N/A;
ASDE-X: Goal satisfied.

Desired practice: Review candidate commercial, off-the-shelf (COTS) 
products to ensure they satisfy the specified requirements that are 
covered under a supplier agreement;
VSCS: N/A;
ERAM: Practice largely implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Perform activities with the supplier as specified in 
the supplier agreement;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Ensure that the supplier agreement is satisfied 
before accepting the acquired product;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Transition the acquired products from the supplier to 
the project;
VSCS: N/A;
ERAM: Practice largely implemented;
ITWS: N/A;
ASDE-X: Practice largely implemented.

Capability level 1 achieved?
VSCS: N/A;
ERAM: Yes;
ITWS: N/A;
ASDE-X: Yes.

Goal: The process is institutionalized as a managed process;
VSCS: N/ A;
ERAM: Goal not satisfied;
ITWS: N/A;
ASDE-X: Goal satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the supplier agreement management process;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
supplier agreement management process;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
supplier agreement management process, developing the work products, 
and providing the services of the process;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the supplier agreement management process;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
supplier agreement management process as needed;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice largely implemented.

Desired practice: Place designated work products of the supplier 
agreement management process under appropriate levels of configuration 
management;
VSCS: N/ A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
supplier agreement management process as planned;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/ A;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the supplier agreement management 
process against the plan for performing the process, and take 
appropriate corrective action;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice largely implemented.

Desired practice: Objectively evaluate adherence of the supplier 
agreement management process to its process description, standards, and 
procedures, and address noncompliance;
VSCS: N/A;
ERAM: Practice partially implemented;
ITWS: N/A;
ASDE-X: Practice largely implemented.

Desired practice: Review the activities, status, and results of the 
supplier agreement management process with higher level management, and 
resolve issues;
VSCS: N/A;
ERAM: Practice fully implemented;
ITWS: N/A;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: N/A;
ERAM: No;
ITWS: N/A;
ASDE-X: Goal satisfied.

Sources: GAO, SEI.

Note: N/A represents not applicable;

project not evaluated in this process.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 111 through 114. Specifically, 
tables 111 and 112 provide results for ERAM, and tables 113 and 114 
provide results for ASDE-X.

Table 111: ERAM Supplier Agreement Management: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Agreements with the suppliers are established and maintained;
Rating: Satisfied.

Desired practice: Determine the type of acquisition for each product or 
product component to be acquired;
Rating: Fully implemented.

Desired practice: Select suppliers based on an evaluation of their 
ability to meet the specified requirements and established criteria;
Rating: Fully implemented.

Desired practice: Establish and maintain formal agreements with the 
supplier;
Rating: Fully implemented.

Goal: Agreements with the suppliers are satisfied by both the project 
and the supplier;
Rating: Satisfied.

Desired practice: Review candidate COTS products to ensure they satisfy 
the specified requirements that are covered under a supplier agreement;
Rating: Largely implemented;
Comment: The project team has developed guidance for the review of COTS 
products to ensure they satisfy the specified requirements that are 
covered under a supplier agreement. However, the project team has 
deferred the selection of the products to later in the acquisition when 
it is more appropriate to do so.

Desired practice: Perform activities with the supplier as specified in 
the supplier agreement;
Rating: Fully implemented.

Desired practice: Ensure that the supplier agreement is satisfied 
before accepting the acquired product;
Rating: Fully implemented.

Desired practice: Transition the acquired products from the supplier to 
the project;
Rating: Largely implemented;
Comment: The project team has plans to transition acquired products 
from the supplier to the project.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 112: ERAM Supplier Agreement Management: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the supplier agreement management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
supplier agreement management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
supplier agreement management process, developing the work products, 
and providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the supplier agreement management process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
supplier agreement management process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the supplier 
agreement management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
supplier agreement management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the supplier agreement management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Fully implemented.

Desired practice: Objectively evaluate adherence of the supplier 
agreement management process against its process description, 
standards, and procedures, and address noncompliance;
Rating: Partially implemented;
Comment: The project team has just started defining its quality 
assurance process, which is expected to objectively evaluate adherence 
of the supplier agreement management process against its process 
description, standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
supplier agreement management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 113: ASDE-X Supplier Agreement Management: Detailed Findings on 
Level 1 Goals and Practices:

Goal: Agreements with the suppliers are established and maintained;
Rating: Satisfied.

Desired practice: Determine the type of acquisition for each product or 
product component to be acquired;
Rating: Fully implemented.

Desired practice: Select suppliers based on an evaluation of their 
ability to meet the specified requirements and established criteria;
Rating: Fully implemented.

Desired practice: Establish and maintain formal agreements with the 
supplier;
Rating: Fully implemented.

Goal: Agreements with the suppliers are satisfied by both the project 
and the supplier;
Rating: Satisfied.

Desired practice: Review candidate COTS products to ensure they satisfy 
the specified requirements that are covered under a supplier agreement;
Rating: Fully implemented.

Desired practice: Perform activities with the supplier as specified in 
the supplier agreement;
Rating: Fully implemented.

Desired practice: Ensure that the supplier agreement is satisfied 
before accepting the acquired product;
Rating: Fully implemented.

Desired practice: Transition the acquired products from the supplier to 
the project;
Rating: Largely implemented;
Comment: The project team is planning to transition the acquired 
products from the supplier to the project team; however, the transition 
phase has not yet started.

Capability level 1 achieved.

Sources: GAO, SEI.

[End of table]

Table 114: ASDE-X Supplier Agreement Management: Detailed Findings on 
Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the supplier agreement management process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
supplier agreement management process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
supplier agreement management process, developing the work products, 
and providing the services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the process, developing the work products, and providing the services 
of the supplier agreement management process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the project 
planning process to various groups, but it does not assign duties to 
individuals.

Desired practice: Train the people performing or supporting the 
supplier agreement management process as needed;
Rating: Largely implemented;
Comment: The project team has provided training as needed to some but 
not all of the people performing or supporting the process.

Desired practice: Place designated work products of the supplier 
agreement management process under appropriate levels of configuration 
management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
supplier agreement management process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the supplier agreement management 
process against the plan for performing the process, and take 
appropriate corrective action;
Rating: Largely implemented;
Comment: The team collects measurements and monitors the process 
against the plan for performing the process; however, the project is 
not archiving and using the data to control the process beyond the 
monthly snapshots.

Desired practice: Objectively evaluate adherence of the supplier 
agreement management process to its process description, standards, and 
procedures, and address noncompliance;
Rating: Largely implemented;
Comment: The project team objectively evaluates adherence of the 
products to its requirements and addresses noncompliance through the 
quality reliability officer. However, the project team has just begun 
its quality assurance process, which is expected to objectively 
evaluate adherence of the supplier agreement management process to its 
process description, standards, and procedures and to address 
noncompliance.

Desired practice: Review the activities, status, and results of the 
supplier agreement management process with higher level management, and 
resolve issues;
Rating: Fully implemented.

Capability level 2 achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 15: FAA Is Performing Deployment, Transition, and Disposal 
Practices, but It Is Not Yet Fully Managing the Process:

The purpose of the deployment, transition, and disposal process area is 
to place a product or service into an operational environment, transfer 
it to the customer and to the support organization, and deactivate and 
dispose of the replaced product or dispense with the service. This 
process area includes the design and coordination of plans and 
procedures for placement of a product or service into an operational or 
support environment and bringing it into operational use. It ensures 
that an effective support capability is in place to manage, maintain, 
and modify the supplied product or service. It further ensures the 
successful transfer of the product or service to the customer/ 
stakeholder and the deactivation and disposition of the replaced 
capability.

For this process area, we did not perform an appraisal for the VSCS or 
ERAM projects, because these projects were at stages in which 
deployment was not applicable. As shown in figure 17, both of the 
remaining FAA projects satisfied all criteria for the "performing" 
capability level (level 1) in this process area. Neither satisfied all 
criteria for the "managing" capability level (level 2). As shown in the 
overview in table 115, while the projects had differing weaknesses that 
contributed to this result, a common weakness across projects occurred 
in the area of monitoring and controlling the deployment process. As a 
result of this weakness, FAA is exposed to increased risk that the 
projects will not be delivered on time, resulting in projects that will 
not meet cost, schedule, or performance goals.

Looked at another way, of the 32 practices we evaluated in this process 
area, FAA projects had 28 practices that were fully or largely 
implemented and 4 practices that were partially implemented.

Figure 17: Two Projects' Capability Levels in Deployment, Transition, 
and Disposal:

[See PDF for image]

[End of figure]

Table 115: Two Projects' Appraisal Results in Deployment, Transition, 
and Disposal:

Goal: Customer/stakeholder operation and support facilities and 
personnel are prepared to accept the delivery, placement, and 
transition of the product or service into use; customer/stakeholder 
operation and support organizations demonstrate their capacity to 
support the product or service upon assumption of responsibility; and 
the continuity of operational performance is maintained.[A];
VSCS: N/A;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Develop a strategy for deployment, transition, and 
disposal, and perform activities in accordance with the strategy;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Establish the facility and infrastructure environment 
to receive and operate the product or service;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Verify that fielded configuration items reflect the 
product or service baseline, and manage change control;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Demonstrate the ability of the customer/stakeholder 
support organization to maintain, modify, and support the product or 
service;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Desired practice: Transfer the product or service to the customer/
stakeholder operation and support organizations;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice largely implemented;
ASDE-X: Practice largely implemented.

Goal: The replaced product or service is deactivated and disposed of 
and/or dispensed with, as appropriate;
VSCS: N/A;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Desired practice: Deactivate and dispose of replaced product and/or 
dispense with service;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Capability level 1 achieved?
VSCS: N/A;
ERAM: N/A;
ITWS: Goal satisfied;
ASDE-X: Goal satisfied.

Goal: The process is institutionalized as a managed process;
VSCS: N/ A;
ERAM: N/A;
ITWS: Goal not satisfied;
ASDE-X: Goal not satisfied.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the deployment process;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Establish and maintain the plan for performing the 
deployment process;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Provide adequate resources for performing the 
deployment process, developing the work products, and providing the 
services of the process;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Assign responsibility and authority for performing 
the deployment process, developing the work products, and providing the 
services of the process;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice largely implemented.

Desired practice: Train the people performing or supporting the 
deployment process as needed;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Place designated work products of the deployment 
process under appropriate levels of configuration management;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
deployment process as planned;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Desired practice: Monitor and control the deployment process against 
the plan for performing the process, and take appropriate corrective 
action;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice partially implemented;
ASDE-X: Practice partially implemented.

Desired practice: Objectively evaluate adherence of the deployment 
process against its process description, standards, and procedures, and 
address noncompliance;
VSCS: N/A;
ERAM: N/A;
ITWS: Practice partially implemented;
ASDE-X: Practice fully implemented.

Desired practice: Review the activities, status, and results of the 
deployment process with higher level management, and resolve issues;
VSCS: N/A;
ERAM: N/ A;
ITWS: Practice fully implemented;
ASDE-X: Practice fully implemented.

Capability level 2 achieved?
VSCS: N/A;
ERAM: N/A;
ITWS: No;
ASDE-X: No.

Sources: GAO, FAA, and SEI.

Note: N/A represents not applicable; project not appraised in this 
process area.

[A] FAA's iCMM model identifies this goal as four separate goals, each 
with underlying practices. We combined them into one goal for our 
reporting purposes, because some of the practices supporting individual 
goals were repeated in other goals and would have skewed our summary 
results.

[End of table]

Additional details on each project's appraisal results at successive 
capability levels are provided in tables 116 through 119. Specifically, 
tables 116 and 117 provide results for ITWS, and tables 118 and 119 
provide results for ASDE-X.

Table 116: ITWS Deployment, Transition, and Disposal: Detailed Findings 
on Level 1 Goals and Practices:

Goal: Customer/stakeholder operation and support facilities and 
personnel are prepared to accept the delivery, placement, and 
transition of the product or service into use; customer/stakeholder 
operation and support organizations demonstrate their capacity to 
support the product or service upon assumption of responsibility; and 
the continuity of operational performance is maintained;
Rating: Satisfied.

Desired practice: Develop a strategy for deployment, transition, and 
disposal, and perform activities in accordance with the strategy;
Rating: Fully implemented.

Desired practice: Establish the facility and infrastructure environment 
to receive and operate the product or service;
Rating: Fully implemented.

Desired practice: Verify that fielded configuration items reflect the 
product or service baseline, and manage change control;
Rating: Fully implemented.

Desired practice: Demonstrate the ability of the customer/ stakeholder 
support organization to maintain, modify, and support the product or 
service;
Rating: Largely implemented;
Comment: The contractor support organization has demonstrated the 
ability to maintain, modify, and support the product or service and has 
trained FAA personnel to support the product or service; however, the 
product or service has not yet been transitioned from the contractor to 
FAA personnel.

Desired practice: Transfer the product or service to the customer/
stakeholder operation and support organizations;
Rating: Largely implemented;
Comment: The project team is planning to transfer the product to the 
FAA operation and support organization but has not yet completed this 
activity.

Goal: The replaced product or service is deactivated and disposed of 
and/or dispensed with, as appropriate;
Rating: Satisfied.

Desired practice: Deactivate and dispose of the replaced product and/or 
dispense with the replaced service;
Rating: Fully implemented.

Capability level 1 achieved.

Sources: GAO, FAA, and SEI.

[End of table]

Table 117: ITWS Deployment, Transition, and Disposal: Detailed Findings 
on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because three of the practices below 
are partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the deployment process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
deployment process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
deployment process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Assign responsibility and authority for performing 
the deployment process, developing the work products, and providing the 
services of the process;
Rating: Fully implemented.

Desired practice: Train the people performing or supporting the 
deployment process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the deployment 
process under appropriate levels of configuration management;
Rating: Partially implemented;
Comment: The project team places designated work products of the 
deployment process under some levels of configuration management; 
however, the controls on these products are not adequate because all 
support contractors have full access to the system.

Desired practice: Identify and involve the relevant stakeholders of the 
deployment process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the deployment process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Partially implemented;
Comment: The project team reported, but provided no evidence, that it 
is monitoring and controlling the deployment process against the plan 
for performing the deployment process and taking corrective action.

Desired practice: Objectively evaluate adherence of the deployment 
process against its process description, standards, and procedures, and 
address noncompliance;
Rating: Partially implemented;
Comment: The project team is planning, but has not yet implemented, a 
quality assurance process that is expected to objectively evaluate 
adherence of the deployment process against its process description, 
standards, and procedures and to address noncompliance.

Desired practice: Review the activities, status, and results of the 
deployment process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

Table 118: ASDE-X Deployment, Transition, and Disposal: Detailed 
Findings on Level 1 Goals and Practices:

Goal: Customer/stakeholder operation and support facilities and 
personnel are prepared to accept the delivery, placement, and 
transition of the product or service into use; customer/stakeholder 
operation and support organizations demonstrate their capacity to 
support the product or service upon assumption of responsibility; and 
the continuity of operational performance is maintained;
Rating: Satisfied.

Desired practice: Develop a strategy for deployment, transition, and 
disposal, and perform activities in accordance with the strategy;
Rating: Largely implemented;
Comment: The project team has developed and is implementing strategies 
for deployment, but the transition and disposal plans are still in the 
early stages of development.

Desired practice: Establish the facility and infrastructure environment 
to receive and operate the product or service;
Rating: Fully implemented.

Desired practice: Verify that fielded configuration items reflect the 
product or service baseline, and manage change control;
Rating: Fully implemented.

Desired practice: Demonstrate the ability of the customer/ stakeholder 
support organization to maintain, modify, and support the product or 
service;
Rating: Largely implemented;
Comment: The project team has planned to demonstrate support capability 
for software projects in the in-service support stages. However, the 
project has not reached this stage of the life cycle and has not begun 
to implement the support required for this practice.

Desired practice: Transfer the product or service to the customer/
stakeholder operation and support organizations;
Rating: Largely implemented;
Comment: The project team has plans to transition the product to the 
customer/stakeholder operation and support organization. However, the 
transition from the prime contractor to the government organization has 
not yet begun.

Goal: The replaced product or service is deactivated and disposed of 
and/or dispensed with, as appropriate;
Rating: Satisfied.

Desired practice: Deactivate and dispose of the replaced product and/or 
dispense with the replaced service;
Rating: Largely implemented;
Comment: The project team has plans to deactivate and dispose of 
replaced products and dispense with services. However, the project has 
not reached this phase of the life cycle.

Capability level 1 achieved.

Sources: GAO, FAA, and SEI.

[End of table]

Table 119: ASDE-X Deployment, Transition, and Disposal: Detailed 
Findings on Level 2 Goals and Practices:

Goal: The process is institutionalized as a managed process;
Rating: Unsatisfied;
Comment: The goal is unsatisfied because one of the practices below is 
partially implemented.

Desired practice: Establish and maintain an organizational policy for 
planning and performing the deployment process;
Rating: Fully implemented.

Desired practice: Establish and maintain the plan for performing the 
deployment process;
Rating: Fully implemented.

Desired practice: Provide adequate resources for performing the 
deployment process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team provides adequate personnel resources and is 
planning to provide adequate tools and facilities for performing the 
process, developing the work products, and providing the services of 
the process. However, the project team has not yet begun the deployment 
phase, and the tools have not yet been put into use.

Desired practice: Assign responsibility and authority for performing 
the deployment process, developing the work products, and providing the 
services of the process;
Rating: Largely implemented;
Comment: The project team has assigned responsibility for the 
deployment process to various groups, but it has not yet assigned 
duties to individuals.

Desired practice: Train the people performing or supporting the 
deployment process as needed;
Rating: Fully implemented.

Desired practice: Place designated work products of the deployment 
process under appropriate levels of configuration management;
Rating: Fully implemented.

Desired practice: Identify and involve the relevant stakeholders of the 
deployment process as planned;
Rating: Fully implemented.

Desired practice: Monitor and control the deployment process against 
the plan for performing the process, and take appropriate corrective 
action;
Rating: Partially implemented;
Comment: The project team is monitoring the deployment process against 
the plan for performing the process. However, the team is not currently 
controlling the deployment process or taking corrective action.

Desired practice: Objectively evaluate adherence of the deployment 
process against its process description, standards, and procedures, and 
address noncompliance;
Rating: Fully implemented.

Desired practice: Review the activities, status, and results of the 
deployment process with higher level management, and resolve issues;
Rating: Fully implemented.

Capability level 2 not achieved.

Sources: GAO, SEI.

[End of table]

[End of section]

Chapter 16: FAA's Process Improvement Initiative Has Matured, but It Is 
Not Yet Institutionalized: 

Since our 1997 report, the Federal Aviation Administration's (FAA) 
process improvement initiative has grown tremendously in rigor and 
scope. In our earlier appraisal, we found that FAA's performance of key 
processes was ad hoc and sometimes chaotic, whereas current results 
show that FAA projects are performing most key practices. However, 
these process improvement activities are not required throughout the 
air traffic organizations, and the recurring weaknesses we identified 
in our project-specific evaluations are due in part to the choices 
these projects were given in deciding whether to and how to adopt 
process improvement initiatives. Further, because of a recent 
reorganization, the new Air Traffic Organization's commitment to this 
process improvement initiative is not certain. As a result, FAA is not 
consistent in its adoption and management of process improvement 
efforts, so that individual projects' costs, schedules, and performance 
remain at risk. Without agencywide adoption of process improvement 
initiatives, the agency cannot increase the maturity of its 
organizational capabilities.

FAA's Process Improvement Initiative Has Matured: 

Over the past several years, FAA has made considerable progress in 
improving its processes for acquiring and developing software and 
systems. Acting on our prior recommendations, in 1999, FAA established 
a centralized process improvement office that reports directly to the 
Chief Information Officer. This office led the government in an effort 
to integrate various standards and models into a single maturity model, 
called the integrated Capability Maturity Model (iCMM).[Footnote 16] In 
fact, FAA's iCMM served as a demonstration for the Software Engineering 
Institute's effort to integrate various models into its own Capability 
Maturity Model Integration (CMMI).

The Chief Information Officer's process improvement office also 
developed and sponsored iCMM-related training, and by late 2003, it had 
trained over 7,000 participants. The training offered ranges from 
overviews on how to use the model to more focused courses in such 
specific process areas as quality assurance, configuration management, 
and project management. The office also guides FAA organizations in 
using the model and leads appraisal teams in evaluating the process 
maturity of the projects and organizations that adopted the model.

In addition to the Chief Information Officer-sponsored process 
improvement efforts, several of FAA's business areas, including the 
business areas with responsibility for air traffic control (ATC) system 
acquisitions and operations, endorsed and set goals for process 
improvement activities using the iCMM. As a result, there has been a 
continuing growth over the years in the number of individual projects 
and umbrella organizations that adopted process improvement and the 
iCMM model. Specifically, the number of projects and organizations 
(which account for multiple projects) undergoing iCMM appraisals grew 
from 1 project in 1997, to 28 projects and 3 organizations by 2000, to 
39 projects and 11 organizations by 2003.

These projects and organizations have demonstrated improvements in 
process maturity. Under the iCMM model, in addition to achieving 
capability levels in individual process areas, entities can achieve 
successive maturity levels by demonstrating capabilities in a core set 
of process areas.[Footnote 17] FAA process improvement officials 
reported that by 2000, 10 projects and one organization had achieved 
iCMM maturity level 2. To date, 14 projects and three organizations 
have achieved iCMM maturity level 2, and one project and two 
organizations have achieved iCMM maturity level 3.[Footnote 18] 
Additionally, 13 projects and four organizations achieved capability 
levels 2 or 3 in one or more process areas.

Moreover, in internal surveys, the programs and organizations pursuing 
process improvement have consistently reported enhanced productivity, 
higher quality, increased ability to predict schedules and resources, 
higher morale, and better communication and teamwork. These findings 
are reiterated by the Software Engineering Institute in its recent 
study of the benefits of using the CMMI model for process 
improvement.[Footnote 19] According to that study, organizations that 
implement such process improvements can achieve better project cost and 
schedule performance and higher quality products. Specifically, of the 
12 cases that the Software Engineering Institute assessed, there were: 

* nine examples of cost related benefits, including reductions in the 
cost to find and fix a defect, and in overall cost savings;

* eight cases of schedule related benefits, including decreased time 
needed to complete tasks and increased predictability in meeting 
schedules;

* five cases of measurable improvements in quality, mostly related to 
reducing defects over time;

* three cases of improvements in customer satisfaction; and: 

* three cases showing positive return on investment from their CMMI-
based process improvements.

FAA Has Not Yet Institutionalized Process Improvement in Its ATC 
Organizations: 

Leading organizations have found that in order to achieve advanced 
system management capabilities and to gain the benefits of more mature 
processes, an organization needs to institutionalize process 
improvement. Specifically, to be effective, an organization needs 
senior-level endorsement of its process improvement initiatives and 
consistency in the adoption and management of process improvement 
efforts.

In recent years, FAA's ATC-related organizations have encouraged 
process improvement through the iCMM model. Specifically, FAA's 
acquisition policy calls for continuous process improvement and 
endorses the use of the iCMM model. Also, the former air traffic 
organizations set annual goals for improving maturity using the iCMM 
model in selected projects and process areas. For example, in 1997, the 
former ATC acquisition organization set a goal of having 11 selected 
projects achieve iCMM maturity level 2 by 1999 and maturity level 3 by 
2001. While the projects did not meet the 1999 goal, several projects 
achieved level 2 in 2000, and most made improvements in selected 
process areas.

However, FAA did not institutionalize the use of the iCMM model 
throughout the organization and, as a result, individual projects' use 
and application of the model has been voluntary. Individual project 
teams could determine whether or not they would implement the model and 
which process areas to work on. In addition, project teams could decide 
when, if ever, to seek an appraisal of their progress in implementing 
the model.

Because of this voluntary approach, to date less than half of the 
projects listed in FAA's system architecture have sought appraisals in 
at least one process area. Specifically, of the 48 systems listed in 
FAA's system architecture, only 18 have sought appraisals. Some of the 
mission critical systems that have not sought appraisals include an 
advanced radar system and air traffic information processing system.

Another result of this voluntary approach is that individual projects 
are making uneven progress in core areas. For example, the four 
projects that we appraised ranged from capability levels 0 to 2 in the 
risk management process area: in other words, projects varied from 
performing only part of the basic process, to performing the basic 
process, to actively managing the process. As another example, all four 
of the projects we appraised captured some metrics on their 
performance. However, these metrics varied greatly from project to 
project in depth, scope, and usefulness. Individual weaknesses in key 
processes could lead to systems that do not meet the users' needs, 
exceed estimated costs, or take longer than expected to complete.

While FAA encouraged process improvement in the past, the agency's 
current commitment to process improvement in its new Air Traffic 
Organization is not certain. FAA recently moved its air traffic-related 
organizations into a single, performance-based organization, the Air 
Traffic Organization, under the direction of a Chief Operating Officer. 
The Chief Operating Officer is currently reevaluating all policies and 
processes, and plans to issue new acquisition guidance in coming 
months. As a result, the Air Traffic Organization does not currently 
have a policy that requires organizations and project teams to 
implement process improvement initiatives such as the iCMM. It also 
does not have a detailed plan--including goals, metrics, and 
milestones--for implementing these initiatives throughout the 
organization, nor does it have a mechanism for enforcing compliance 
with any requirements--such as taking a project's capability levels 
into consideration before approving new investments. Further, because 
the Air Traffic Organization's commitment to the iCMM is not yet 
certain, FAA's centralized process improvement organization is unable 
to define a strategy for improving and overseeing process improvement 
efforts in the Air Traffic Organization.

Unless the Chief Operating Officer demonstrates a strong commitment to 
process improvement and establishes a consistent, institutionalized 
approach to implementing, enforcing, and evaluating this process 
improvement, FAA risks taking a major step backwards in its 
capabilities for acquiring ATC systems and software. That is, FAA may 
not be able to ensure that critical projects will continue to make 
progress in improving systems acquisition and development capabilities, 
and the agency is not likely to proceed to the more advanced capability 
levels which focus on organizationwide management of processes. 
Further, FAA may miss out on the benefits that process improvement 
models offer, such as better managed projects and improved product 
quality. Should this occur, FAA will continue to be vulnerable to 
project management problems including cost overruns, schedule delays, 
and performance shortfalls.

[End of section]

Chapter 17: Conclusions and Recommendations: 

Conclusions: 

The Federal Aviation Administration (FAA) has made considerable 
progress in implementing processes for managing software acquisitions. 
Key projects are performing most of the practices needed to reach a 
basic level of capability in process areas including risk management, 
project planning, project monitoring and control, and configuration 
management. However, recurring weaknesses in the areas of verification, 
quality assurance, and measurement and analysis prevented the projects 
from achieving a basic level of performance in these areas and from 
effectively managing these and other process areas. These weaknesses 
could lead to systems that do not meet the users' needs, exceed 
estimated costs, or take longer than expected to complete. Further, 
because of the recurring weaknesses in measurement and analysis, senior 
executives may not receive the project status information they need to 
make sound decisions on major project investments.

FAA's process improvement initiative has matured in recent years, but 
more can be done to institutionalize improvement efforts. The Chief 
Information Officer's centralized process improvement organization has 
developed an integrated Capability Maturity Model (iCMM) and 
demonstrated improvements in those using the model, but to date the 
agency has not ensured that projects and organizational units 
consistently adopt such process improvements. Specifically, the agency 
lacks a detailed plan--including goals, metrics, and milestones--for 
implementing these initiatives throughout the new Air Traffic 
Organization, and a mechanism for enforcing compliance with any 
requirements--such as taking a project's capability level into 
consideration before approving new investments. With the recent move of 
FAA's air traffic control-related organizations into a performance-
based organization, the agency has an opportunity to reiterate the 
value of process improvement and to achieve the benefits of more mature 
processes. In the coming months, it will be critical for this new 
organization to demonstrate its commitment to process improvement 
through its policies, plans, goals, oversight, and enforcement 
mechanisms. Without such endorsement, the progress that FAA has made in 
recent years could dissipate.

Recommendations for Executive Action: 

Given the importance of software-intensive systems to FAA's air traffic 
control modernization program, we recommend that the Secretary of 
Transportation direct the FAA Administrator to ensure that the 
following five actions take place: 

* The four projects that we appraised should take action to fully 
implement the practices that we identified as not implemented or 
partially implemented.

* The new Air Traffic Organization should establish: 

* a policy requiring organizations and project teams to implement iCMM 
or equivalent process improvement initiatives and: 

* a plan for implementing iCMM or equivalent process improvement 
initiatives throughout the organization. This plan should specify a 
core set of process areas for all projects, clear criteria for when 
appraisals are warranted, and measurable goals and time frames.

* The Chief Information Officer's process improvement office, in 
consultation with the Air Traffic Organization, should develop a 
strategy for overseeing all air traffic projects' progress to 
successive levels of maturity; this strategy should specify measurable 
goals and time frames.

* To enforce process improvement initiatives, FAA investment decision 
makers should take a project's capability level in core process areas 
into consideration before approving new investments in the project.

Agency Comments: 

In its oral comments on a draft of this report, Department of 
Transportation and FAA officials generally concurred with our 
recommendations, and they indicated that FAA is pleased with the 
significant progress that it has achieved in improving the processes 
used to acquire software and systems. Further, these officials noted 
that FAA has already started implementing changes to address issues 
identified in the report. They said that progress is evident in both 
the improved scores, compared with our prior study, and also in the way 
FAA functions on a day-to-day basis. For example, these officials 
explained that FAA is now working better as a team because the 
organization is using cross-organizational teams that effectively share 
knowledge and best practices for systems acquisition and management. 
FAA officials also noted that the constructive exchange of information 
with us was very helpful to them in achieving progress, and they 
emphasized their desire to maintain a dialog with us to facilitate 
continued progress. Agency officials also provided technical 
corrections, which we have incorporated into this report as 
appropriate.

[End of section]

Appendixes: 

Appendix I: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Colleen Phillips (202) 512-6326 David Powner (202) 512-9286 Keith 
Rhodes (202) 512-6412: 

Staff Acknowledgments: 

In addition to those named above, Jim Belford, Dan Bennett, Barbara 
Collier, John Dale, Season Dietrich, Thayne Hill, Deborah Lott, Tammi 
Nguyen, Madhav Panwar, Andrea Smith, and Carl Urie made key 
contributions to this report.

(310452): 

FOOTNOTES

[1] U.S. General Accounting Office, High-Risk Series: An Update, GAO-
03-119 (Washington, D.C.: January 2003).

[2] CMM®, Capability Maturity Model, and Capability Maturity Modeling 
are registered in the U.S. Patent and Trademark Office. CMMISM is a 
service mark of Carnegie Mellon University.

[3] Standard CMMI Appraisal Methodology for Process Improvement.

[4] Not all process areas are relevant to all stages of development. 
The number of process areas evaluated varied, depending on each 
project's stage of development. 

[5] U.S. General Accounting Office, Air Traffic Control: Immature 
Software Acquisition Processes Increase FAA Acquisition Risks, GAO/
AIMD-97-47 (Washington, D.C.: Mar. 21, 1997).

[6] These and other system management process areas are described in 
chapter 1.

[7] A performance-based organization is an organization that commits to 
clear management objectives, measurable goals, customer service 
standards, and specific targets for improved performance.

[8] U.S. General Accounting Office, Air Traffic Control: FAA's 
Modernization Efforts--Past, Present, and Future, GAO-04-227T 
(Washington, D.C.: Oct. 30, 2003).

[9] Examples include U.S. General Accounting Office, Air Traffic 
Control: Status of FAA's Modernization Program, GAO/RCED-95-175FS 
(Washington, D.C.: May 26, 1995); Air Traffic Control: Status of FAA's 
Modernization Program, GAO/RCED-99-25 (Washington, D.C.: Dec. 3, 1998); 
Air Traffic Control: FAA's Modernization Efforts--Past, Present, and 
Future, GAO-04-227T (Washington, D.C.: Oct. 30, 2003).

[10] U.S. General Accounting Office, High-Risk Series: An Overview, 
GAO/HR-95-1 (Washington, D.C.: February 1995); High-Risk Series: 
Information Management and Technology, GAO/HR-97-9 (Washington, D.C.: 
February 1997); High-Risk Series: An Update, GAO/HR-99-1 (Washington, 
D.C.: January 1999); High-Risk Series: An Update, GAO-01-263 
(Washington, D.C.: January 2001); High-Risk Series: An Update, GAO-03-
119 (Washington, D.C.: January 2003).

[11] U.S. General Accounting Office, Air Traffic Control: Immature 
Software Acquisition Processes Increase FAA Acquisition Risks, GAO/
AIMD-97-47 (Washington, D.C.: Mar. 21, 1997).

[12] A performance-based organization is an organization that commits 
to clear management objectives, measurable goals, customer service 
standards, and specific targets for improved performance.

[13] Standard CMMI® Appraisal Methodology for Process Improvement.

[14] Software Engineering Institute, Demonstrating the Impact and 
Benefits of CMMI: An Update and Preliminary Results, CMU/SEI-2003-SR-
009 (October 2003).

[15] We also appraised one system at level 3 because it had achieved 
maturity level 3 on FAA's iCMM. However, because the system did not 
meet the CMMI requirements for level 2, it could not achieve level 3.

[16] FAA's iCMM version 1.0 combined the features of three Software 
Engineering Institute capability maturity models: the Software 
Acquisition Capability Maturity Model, the Capability Maturity Model 
for Software, and the Systems Engineering Capability Maturity Model. 
FAA's iCMM version 2.0 integrates the following standards and models: 
International Organization for Standardization standards 9001: 2000, TR 
15504, 12207, and CD 15288; the National Institute of Standards and 
Technology's Malcolm Baldrige National Quality Award criteria; the 
Office of Personnel Management's President's Quality Award criteria; 
the Software Engineering Institute's Capability Maturity Model 
Integration Integrated Product and its Process Development and 
Capability Maturity Model Integration A Specification; and the 
Electronic Industries Association Interim Standard 731.

[17] Under iCMM, version 1, an entity must achieve capability level 2 
in 9 core process areas to attain maturity level 2, and it must achieve 
capability level 3 in 18 core process areas in order to attain maturity 
level 3. Version 2 is more stringent, requiring achievement of 
capability level 3 in 20 core process areas in order to attain maturity 
level 3. 

[18] All the organizations except one were assessed on iCMM v1.0. 

[19] Software Engineering Institute, Demonstrating the Impact and 
Benefits of CMMI: An Update and Preliminary Results, CMU/SEI-2003-SR-
009 (October 2003).

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548: 

To order by Phone: 



Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: