This is the accessible text file for GAO report number GAO-03-343 entitled 'National Airspace System: Better Cost Data Could Improve FAA's Management of the Standard Terminal Automation Replacement System' which was released on February 04, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to the Honorable Ellen O. Tauscher, House of Representatives: January 2003: NATIONAL AIRSPACE SYSTEM: Better Cost Data Could Improve FAA’s Management of the Standard Terminal Automation Replacement System: GAO-03-343: GAO Highlights: Highlights of GAO-03-343, a report to the Honorable Ellen O. Tauscher, House of Representatives. NATIONAL AIRSPACE SYSTEM Better Cost Data Could Improve FAA’s Management of the Standard Terminal Automation Replacement System. Why GAO Did This Study: To enhance the capacity and safety of the national airspace system, the Federal Aviation Administration (FAA), within the Department of Transportation, is acquiring 74 Standard Terminal Automation Replacement Systems (STARS). STARS will replace some outdated air traffic control equipment. Since 1996, when FAA initiated this major computer hardware and software acquisition, the scope and estimated costs of STARS have changed many times. FAA now estimates that STARS’s remaining costs will total about $2.54 billion. GAO was asked to assess the reliability of FAA’s life-cycle cost estimate for STARS, determine the impact of STARS’s estimated costs on future FAA budgets, and identify any alternatives to STARS that FAA is considering. GAO based its analysis on published FAA cost data and the guidance FAA uses for managing major acquisitions. What GAO Found: The reliability of FAA’s life-cycle cost estimate for STARS is uncertain. This estimate includes estimates of the program’s development, operation, and technology upgrade costs, shown for the next 6 fiscal years in the figure below. The development cost estimate is based on the contractor’s projections, which FAA has not yet independently analyzed as its guidance directs. Furthermore, baseline data in cost performance reports that FAA obtains from the contractor are not accurate because the data do not reflect the current status of the contract. As a result, FAA is limited in its ability to manage the contract effectively. FAA plans to address these problems. In addition, the program’s operation and technology upgrade cost estimates are based on limited experience with STARS and extend many years into the future. However, the estimates do not reflect these uncertainties. Estimated STARS Funding, Fiscal Years 2004-2009 [See PDF for Image] [End of Figure] For fiscal years 2004 through 2007, the years for which FAA provided budget information, STARS’s estimated costs should have a declining impact on FAA’s budgets because the program’s development is nearly over and its operations are still limited. For fiscal years 2008 through 2030, the impact of STARS’s estimated costs on FAA’s budgets is unknown because it is still too soon in the budget cycle for FAA to have developed detailed budgets for these years. After deploying STARS at the 74 terminal and support facilities included in the program, FAA could use STARS, another contractor’s technology, or a combination of the two technologies for the nearly 100 remaining facilities. FAA is exploring the feasibility of combining the technologies and expects to announce its plans in April 2003. What GAO Recommends: To improve FAA’s management of STARS and of subsequent terminal modernization programs and to provide the Congress with more reliable information for oversight, GAO recommends that the Secretary of Transportation direct the FAA Administrator to maintain accurate, current baseline data; review baseline data within 6 months of any major modification to ensure that the data reflect the current status of the contract; and prepare a rigorous life-cycle cost estimate that identifies the level of uncertainty. To view the full report, including the scope and methodology, click on the link above. For more information, contact Gerald L. Dillingham at (202) 512-5555 or Keith A. Rhodes at (202) 512-6412. Contents: Letter: Results in Brief: Background: Reliability of FAA’s Life-Cycle Cost Estimate for STARS Is Uncertain: Impact of STARS’s Estimated Costs on FAA’s Budgets Is Expected to Decline through Fiscal Year 2007: After Deploying STARS at 74 Facilities, FAA May Modify Its Approach for the Remaining Facilities: Conclusions: Recommendations for Executive Action: Agency Comments: Scope and Methodology: Appendix: Appendix I: GAO Contacts and Staff Acknowledgements: GAO Contacts: Staff Acknowledgments: Table: Table 1: Proposed Funding for FAA’s Total Facilities and Equipment Budget and for STARS’s Development and Deployment: Figures: Figure 1: Standard Terminal Automation Replacement System (STARS): Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years 2004-2030: Figure 3: Allocation of Facilities and Equipment Funding for Fiscal Year 2004 in FAA’s Capital Investment Plan: Abbreviations: ARTS: Automated Radar Terminal System: DCMA: Defense Contract Management Agency: DOD: Department of Defense: FAA: Federal Aviation Administration: STARS: Standard Terminal Automation Replacement System: This is a work of the U.S. Government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. It may contain copyrighted graphics, images or other materials. Permission from the copyright holder may be necessary should you wish to reproduce copyrighted materials separately from GAO’s product. Letter January 31, 2003: The Honorable Ellen O. Tauscher House of Representatives: Dear Ms. Tauscher: On November 17, 2002, the Federal Aviation Administration (FAA) began using the Standard Terminal Automation Replacement System (STARS) to control air traffic at the Philadelphia air route traffic control center, the first “busy” FAA facility to use this version of the new system.[Footnote 1] FAA’s current plan is to procure 74 STARS systems, including 70 for terminal facilities and 4 for support facilities.[Footnote 2] STARS will replace outdated computer equipment that is used to control air traffic within 5 to 50 nautical miles of an airport.[Footnote 3] With STARS, air traffic controllers at these facilities will receive new hardware and software that produce color displays of aircraft position and flight information. In the future, FAA will be able to upgrade the software to provide air traffic control tools, such as those that will allow better spacing of aircraft as they descend into airports. STARS is complex, costly, and software- intensive. Since 1996, when FAA initiated STARS, the number of systems scheduled to be procured has ranged from as many as 188 to as few as 74,[Footnote 4] and the program’s cost and schedule have also varied considerably. Over the years, we have reported on these changes, most recently in September 2002.[Footnote 5] As agreed with your office, this report addresses the following questions: * How reliable is FAA’s estimate of the life-cycle costs to develop and deploy, operate and maintain, and upgrade STARS? * What impact will STARS’s estimated cost have on future FAA budgets given competing demands for funds to enhance aviation safety, security, and capacity? * What alternatives to STARS is FAA considering? In addition, you asked whether STARS falls under the termination provisions of Public Law 104-264, as an acquisition that is more than 50 percent over its cost goal or behind schedule. Our analysis of the law and its legislative history indicates that STARS is not subject to these termination provisions. According to our analysis, the termination provisions apply to acquisitions initiated after October 9, 1996, the date of the law’s enactment. Because FAA approved the initial acquisition plan for STARS in March 1996 and signed the contract with Raytheon Corporation in September 1996, STARS is not subject to those provisions. This report covers cost and performance issues related to FAA’s procurement of STARS for 74 terminal and support facilities.[Footnote 6] To conduct our work, we reviewed FAA’s 5-year Capital Investment Plan, which proposes funding for programs to modernize the national airspace system. We also analyzed data from cost performance reports that the STARS contractor developed for FAA. However, we did not independently verify these cost and performance data. Results in Brief: The reliability of FAA’s life-cycle cost estimate for STARS is uncertain. According to FAA, the costs to develop and deploy,[Footnote 7] operate and maintain, and upgrade STARS will amount to about $2.54 billion for 74 systems for fiscal years 2004 through 2030. More specifically, the costs of completing STARS’s development and deployment will amount to about $153 million for fiscal years 2004 through 2008, the costs of operating and maintaining the systems at those facilities over their useful lives will add another $1.46 billion for fiscal years 2004 through 2030,[Footnote 8] and the costs of upgrading STARS technology will amount to about $930 million over the same period. FAA’s development cost estimate is based largely on the contractor’s recent proposals and projections, which incorporate the costs of new work specified in major modifications to the STARS contract. FAA has not yet independently analyzed these proposals and projections as its guidance directs and therefore does not know whether the development cost estimate of $153 million is reliable. Furthermore, FAA has not worked with the contractor to incorporate the requirements and costs of the new work into cost performance reports, which it receives from the contractor. These reports are intended to provide FAA with accurate, current data for monitoring and overseeing the contractor’s progress and for estimating the program’s remaining costs, but FAA is not using the reports because they do not reflect the current status of the contract. FAA is now analyzing the contractor’s cost data and working with the contractor to align the cost performance reports with the current status of the contract. FAA expects to complete these tasks in the spring of 2003 and then should be able to use the cost performance reports as intended. The reliability of FAA’s estimates of the life-cycle costs to operate, maintain, and upgrade STARS technology is unknown, primarily because FAA has limited experience with STARS equipment and the estimates extend nearly 30 years into the future. However, despite these uncertainties, FAA has expressed its cost estimates as point values, rather than as ranges. As a result, the estimates may imply more certainty than is appropriate. Moreover, in this instance, the use of point values limits disclosure of the program’s investment risks. We are making recommendations to strengthen FAA’s management of STARS, and FAA officials indicated that the recommendations were in line with the agency’s ongoing and planned efforts. These recommendations, we believe, will also help FAA better manage the planned modernization of terminal facilities that are no longer included in the STARS program. According to FAA’s latest budget planning documents, the impact of STARS’s estimated costs on FAA’s budgets will decline for fiscal years 2004 through 2007; for later years, however, the impact of these estimated costs is unknown. Since FAA has nearly finished developing STARS, the program’s development costs are expected to decrease over the next 4 fiscal years, while its operation and maintenance costs are expected to grow with increased deployment. According to FAA’s documents, the funding proposed to develop and deploy STARS represents 4 percent of FAA’s proposed $3.06 billion facilities and equipment budget account for fiscal year 2004 and smaller percentages for fiscal years 2005 through 2007. This account funds the development, procurement, and installation of equipment to help increase the capacity and improve the safety, efficiency, and security of the national airspace system. Given these small and declining percentages for STARS, the program’s estimated costs should, over this period, have a small and declining impact on other aviation safety, security, and capacity efforts funded from the facilities and equipment account. FAA’s budget planning documents also show that as FAA deploys STARS at more facilities, more funding will come from the agency’s operations account, which supports training and compensation for the controllers and technicians who operate and maintain STARS. However, the impact of STARS’s estimated costs on FAA’s budgets for later years is currently unknown because it is too soon in the budget cycle for FAA to have developed detailed budgets beyond fiscal year 2007. In addition, as we previously noted, FAA has limited operational experience for projecting STARS’s budgetary impact, and long-range cost estimates are inherently uncertain. FAA is committed to deploying STARS at the 74 terminal and support facilities included in the STARS program, but for the nearly 100 other facilities that remain to be modernized, the agency could deploy STARS, another contractor’s hardware and software, or a combination of STARS and the other contractor’s technologies that are currently being used in FAA terminal facilities. FAA knows that each contractor’s technology works independently, and FAA is assessing the feasibility of operating Raytheon’s STARS display system with the processing system from Lockheed Martin Corporation’s Common Automated Radar Terminal System, which was recently installed at some terminal facilities. Combining the two technologies could be cost-effective because it would allow FAA to use both the customized display system that accounted for a substantial portion of STARS’s development costs and the recently acquired processing system. FAA plans to announce the results of this assessment in April 2003. Background: STARS will replace controller workstations with new color displays, processors, and computer software at FAA and Department of Defense terminal air traffic control facilities. (See fig. 1.) FAA’s goal for STARS is to provide an open, expandable terminal automation platform that can accommodate future air traffic growth and allow for the introduction of new hardware-and software-based tools to promote safety, maximize operational efficiency, and improve controllers’ productivity. FAA believes that STARS will facilitate efforts to optimally configure the terminal airspace around the country, exchange digital information between pilots and controllers, and introduce new position and surveillance capabilities for pilots. Figure 1: Standard Terminal Automation Replacement System (STARS): [See PDF for image] - graphic text: [End of figure] - graphic text: FAA has given high priority to STARS. Both the past and the current FAA Administrator have emphasized the program’s importance to enhancing the capacity of the national airspace system. In addition, STARS is 1 of 19 programs on FAA’s list of top programs. Recently, FAA gave priority consideration to STARS so that it could meet its commitment to the Congress to deploy STARS at the Philadelphia terminal, the first busy facility, on November 17, 2002. Furthermore, according to FAA officials, the agency remains committed to funding STARS until completion, even if that means postponing the funding for other programs. Agency officials have indicated, for example, that if the fiscal year 2004 budget falls below the level shown in FAA’s planning documents, funds from other programs would likely be cut to fully fund STARS. For each acquisition program that FAA undertakes, it officially estimates the program’s life-cycle costs (from development and deployment through operations and maintenance), schedule, benefits, and performance in a formal budget document called the acquisition program baseline. FAA uses this document--which its acquisition decision-making body, the Joint Resources Council, must approve--to decide whether to fund the program and, if it is funded, to monitor its progress. FAA also uses the approved acquisition program baseline to develop a 5-year budget-planning document, called the Capital Investment Plan. Program managers rely on the acquisition program baseline to oversee the program’s progress and to ensure that no action is taken that would breach the approved baseline. To support their oversight, program managers typically require contractors to deliver cost performance reports, each of which includes a performance measurement baseline for assessing the contractor’s progress in meeting the contract’s cost, schedule, and technical performance goals. For STARS, FAA has one contract with Raytheon, which accounts for 82 percent of the funding approved in the acquisition program baseline. Most of the remaining funding is used for contracts that support FAA’s internal program management. For guidance in managing its major acquisition programs, FAA relies largely on two documents--its own Acquisition System Toolset, a “one- stop acquisition information system” on FAA’s Web site that contains the agency’s official acquisition policy and guidance, and the Defense Contract Management Agency’s (DCMA) guide on implementing earned value management.[Footnote 9] The Acquisition System Toolset includes policy, guidance, instructions, examples, best practices, lessons learned, references, and other related information tailored to each type of procurement contract. The earned value management guide provides information on how to use cost and performance measurement to manage acquisition programs. FAA funds STARS primarily through two of its budget accounts: (1) facilities and equipment and (2) operations. The facilities and equipment account covers the costs to develop, procure, and place the new equipment in operation.[Footnote 10] After the equipment has been fully operating for at least a year, the funding source shifts to the operations account, which covers the costs to support and maintain the equipment over its life cycle. Operations costs are, in large part, personnel costs--for the controllers who operate and the technicians who maintain the equipment. Planned product improvements and technology upgrades are primarily funded from the facilities and equipment account. Since 1996, when FAA initiated STARS, it has spent approximately $1.2 billion, or about 86 percent of the funding budgeted for the program, and it has twice approved major changes to the program’s cost and schedule estimates. First, in October 1999, FAA modified its acquisition approach (from off-the-shelf software only to a combination of customized and off-the-shelf software) and increased to 188 the number of facilities scheduled to receive STARS. We reported on these changes in September 2002.[Footnote 11] At that time, FAA also concluded that it did not have adequate funding to deploy STARS to all 188 facilities with the remainder of the STARS funding. Instead, FAA decided to deploy STARS to 74 terminal and support facilities. The selected facilities had frequent equipment failures, were new, or had the digital radar needed to operate STARS. FAA has since reduced the total number of facilities to about 170. The agency is currently studying options for modernizing display systems at the nearly 100 remaining facilities and is identifying the additional costs of upgrading the other facilities with STARS or an alternative system. The STARS program office stated that these additional costs, together with the funds already committed to STARS, would more than likely exceed the $1.4 billion originally planned. The program office is in the process of developing options and estimates for these additional sites. Reliability of FAA’s Life-Cycle Cost Estimate for STARS Is Uncertain: FAA has estimated the life-cycle cost for STARS, including the costs to develop and deploy systems to 74 terminal and support facilities and to operate, maintain, and upgrade the systems. However, the reliability of these cost estimates is uncertain. If FAA’s estimates are not reliable, both the agency and the Congress will be limited in their ability to project and compare the costs and benefits of completing STARS and modernizing other facilities as well as in their ability to budget realistically for other capital investments. FAA Has Estimated STARS’s Remaining Life-Cycle Costs: According to FAA’s estimates, the agency will need about $2.54 billion for STARS over the remaining life of the program. As indicated in figure 2, FAA will need about (1) $153 million for fiscal years 2004 through 2008 to complete the development and deployment of 74 systems; (2) $1.46 billion for fiscal years 2004 through 2030--or about $54 million a year, on average--to operate and maintain the systems;[Footnote 12] and (3) $930 million to upgrade STARS technology over the same period. FAA’s operating cost estimate assumes that STARS’s operating costs will grow with deployment through about fiscal year 2007 and will subsequently keep pace with inflation, at least through fiscal year 2009. FAA expects the costs of technology upgrades to more than triple between fiscal years 2007 and 2008 because of a technology update cycle, which will peak when the hardware and software are due for replacement. This projection assumes that STARS software will be upgraded every 3 years. Figure 2: Life-Cycle Cost Estimate for STARS, Fiscal Years 2004-2030: [See PDF for image] - graphic text: [End of figure] - graphic text: Note: These data are from FAA’s March 2002 acquisition program baseline for STARS and have been adjusted to reflect the effects of inflation. Numbers may not add because of rounding. FAA does not yet know to what extent its estimate of STARS’s remaining development costs is reliable. The estimate is derived from the contractor’s proposals and projections, which reflect the costs of major modifications to the contract made since May 2000. However, FAA has not yet independently analyzed and validated the proposals and projections and therefore cannot assess the reliability of the development cost estimate. FAA obtains monthly cost performance reports from the contractor, which the agency should be able to use both to oversee the contractor’s performance and to estimate the program’s remaining development costs. However, FAA does not use these reports because they are not current. More specifically, their central component, the performance management baseline--which establishes performance, cost, and schedule milestones for the contract--has not been updated since May 2000 and therefore does not incorporate the effects of major contract modifications approved since that date. For example, the September 2002 cost performance report does not reflect FAA’s March 2002 reduction in the scope of STARS from 188 to 74 systems, and the report does not include the costs of new work that FAA authorized between May 2000 and September 2002. Consequently, the report indicates that STARS is on schedule and within 1 percent of budget, even though, compared with the program envisioned in May 2000, FAA is now under contract to modernize fewer than half as many facilities at more than twice the cost per facility. Cost Performance Reports Do Not Meet FAA’s Criteria: The cost performance reports for STARS do not meet the criteria for such reports established in the guidance for managing major acquisitions that FAA has adopted from DCMA.[Footnote 13] According to this guidance, cost performance reports are valid only when: * a reliable performance measurement baseline is established and maintained (i.e., regularly updated and validated), * changes to the baseline are carefully controlled (i.e., negotiated and approved before being authorized), * an integrated baseline review[Footnote 14] takes place within 6 months of a contract’s award or significant modification, and: * contract oversight occurs regularly. FAA and the contractor have established a performance measurement baseline for the STARS contract, but they have not satisfied the remaining criteria. The baseline has not been updated since May 2000, and FAA has not validated it. For example, FAA has not asked an independent organization, such as the Defense Contract Audit Agency, to verify the costs in the contractor’s cost performance reports by tracing the costs back to the contractor’s accounting system. In addition, FAA has not controlled changes to the baseline because, between May 2000 and September 2002, it approved up to $179 million in authorized, unpriced work--that is, additional work that FAA agreed to let the contractor perform without first negotiating or independently verifying the costs. Furthermore, the additional tasks and costs have not been incorporated in the baseline, even though the DCMA guidance calls for processing contract modifications expeditiously and incorporating them in a timely manner to maintain the baseline’s integrity. Although the DCMA guidance does not define “a timely manner,” experts generally agree that this term means no longer than 3 months. FAA has not maintained and controlled the baseline because, according to program officials, it has been “schedule driven”-- committed to deploying STARS at the Philadelphia terminal by November 17, 2002. FAA is currently analyzing the contractor’s cost data and working with the contractor to incorporate modifications in the performance measurement baseline. Although the DCMA guidance calls for performing an integrated baseline review within 6 months of awarding or significantly modifying a contract, FAA has not performed such a review of STARS since August 2000, even though it has subsequently made two major modifications to the contract. An integrated baseline review is important to ensure that the contract’s cost data are aligned with the current status of the program after a major contract modification. According to the Manager of Terminal Automation, who oversees STARS, FAA had planned to initiate an integrated baseline review of the STARS contract in November 2002 and expects to begin this effort as soon as funding is available. To provide regular contract oversight as the DCMA guidance requires, FAA and Raytheon meet monthly to, among other things, discuss the cost performance reports. However, discussions of these reports do not, in our view, constitute regular contract oversight because, without a current, valid performance measurement baseline, FAA cannot compare what the contractor has done with what the contractor agreed to do in the contract. With a current, valid baseline, the reports would indicate when cost or schedule thresholds had been exceeded, and FAA could then require the contractor to explain the reasons for the variances and to identify and take appropriate corrective actions. But because the baseline has not been maintained and is not aligned with the current status of the program, the reports are not useful for evaluating the contractor’s performance or for projecting the contract’s remaining costs. The Manager of Terminal Automation agreed that the current cost performance reports are not useful for these purposes and said that the agency therefore uses the monthly meetings with the contractor to discuss other program control and financial issues. FAA’s Cost Estimates Do Not Reflect Levels of Risk: FAA has limited information for determining the reliability of the costs it has estimated to operate and maintain STARS and to upgrade its technology. The agency first projected the costs of operating and maintaining STARS in March 2002 and said that if these costs were not funded, essential STARS maintenance would not be completed. However, FAA has just begun to operate STARS and has limited experience for projecting future operating and maintenance costs. In addition, FAA did not perform a risk assessment, as its Acquisition System Toolset guidance specifies, to identify the minimum, most likely, and maximum expected costs for the entire program. Although FAA performed such an assessment for one facility, it did not extend this effort--again because of time pressures, according to FAA officials. Because FAA has just begun to operate STARS, it also has limited experience for projecting the costs of technology upgrades over the life of the program. Additionally, both the operating and the technology upgrade cost estimates are uncertain because they extend many years into the future. Despite the uncertainty inherent in estimates--especially long-range ones--FAA has expressed its cost estimates for STARS as point values. As we reported in 1997,[Footnote 15] point values imply certainty and therefore are not suitable for expressing estimates. Instead, ranges or numbers with confidence levels would be more appropriate. For instance, a cost estimate of $1 million could be presented either as a range of $900,000 to $1.1 million or as $1 million with a confidence interval of 90 percent, indicating that there is a 10 percent chance that the cost will exceed the estimate.[Footnote 16] Because FAA did not perform a risk assessment for the entire STARS program, it did not develop ranges or confidence levels that it could use to express its cost estimates for STARS. And because FAA used point values instead, it limited its disclosure of the program’s investment risks. The reliability of FAA’s development and operating cost estimates is important not only for managing STARS effectively but also for planning appropriately for other terminal modernization efforts and other FAA capital investments. Without reliable estimates for comparing STARS’s costs and benefits and for anticipating future expenditures, FAA and the Congress cannot make informed decisions about how best to modernize the nearly 100 facilities that are not currently scheduled to receive STARS. Furthermore, because FAA has given high priority to completing STARS and has said that it will fund the program even if it has to postpone the funding for other programs, FAA’s estimates for STARS will influence the agency’s plans for funding other capital investments. Impact of STARS’s Estimated Costs on FAA’s Budgets Is Expected to Decline through Fiscal Year 2007: According to our analysis of budgetary data for fiscal years 2004 through 2007 that we obtained from FAA, the estimated costs of STARS should have a declining impact on FAA’s budgets during these years because the program’s development phase is nearly over and its operations will still be limited. For later years, the impact of these estimated costs on FAA’s budgets is unknown because it is too soon in the budget cycle for FAA to have developed detailed budgets beyond fiscal year 2007. Because FAA has nearly completed the development of STARS and has begun to install the equipment, the agency is budgeting less from the facilities and equipment account for STARS. Therefore, the proposed funding for STARS does not have a significant impact on the funding for other efforts to improve the safety, security, and capacity of the national airspace system. For fiscal year 2004, the $94.1 million planned to develop and deploy STARS represents 3 percent of the total funding for facilities and equipment proposed in FAA’s Capital Investment Plan. When the $19.9 million planned for technology upgrades is included in the projections for fiscal year 2004, the estimated cost for STARS is about 4 percent of FAA’s planned total facilities and equipment budget (see fig. 3). Figure 3: Allocation of Facilities and Equipment Funding for Fiscal Year 2004 in FAA’s Capital Investment Plan: [See PDF for image] - graphic text: [End of figure] - graphic text: Notes: GAO analysis of data from FAA. Beginning with the fiscal year 2003 budget, FAA showed its facilities and equipment budget in broad performance (mission) outcome areas. Individual programs support these broad performance areas. For example, STARS supports several mission areas, such as safety, security, and capacity. To illustrate the impact of STARS on the other broad mission areas, this figure consolidates the allocation for STARS in one percentage and compares this percentage with the allocations to the broad mission areas. According to FAA’s Capital Investment Plan, the proposed funding for FAA’s total facilities and equipment budget will increase slightly during fiscal years 2005 through 2007, while the proposed funding for STARS’s development and deployment will continue to decline (see table 1). Therefore, the impact of these estimated STARS costs on the facilities and equipment budget for safety, security, and capacity would be even less during these fiscal years. The estimated costs of technology upgrades, if included in the facilities and equipment cost estimates for STARS for these 3 fiscal years, would likewise have a similarly small impact on the facilities and equipment budget for these fiscal years. Table 1: Proposed Funding for FAA’s Total Facilities and Equipment Budget and for STARS’s Development and Deployment: Dollars in millions. Total facilities and equipment budget; Dollars in millions: Fiscal year 2005: $3,129; Dollars in millions: Fiscal year 2006: $3, 202; Dollars in millions: Fiscal year 2007: $3, 277. STARS development and deployment; Dollars in millions: Fiscal year 2005: 43.5; Dollars in millions: Fiscal year 2006: 12.5; Dollars in millions: Fiscal year 2007: 2.0. STARS technology upgrades; Dollars in millions: Fiscal year 2005: 13.6; Dollars in millions: Fiscal year 2006: 12.6; Dollars in millions: Fiscal year 2007: 16.5. [End of table] Source: FAA. Note: These data are from the Capital Investment Plan that FAA submitted with its fiscal year 2003 budget request and FAA’s March 2002 acquisition program baseline for STARS. As FAA begins to operate STARS equipment at more terminal facilities, the primary funding source for the program will shift from the facilities and equipment budget account to the operations budget account. Although FAA has projected that it will increasingly need funding from (1) the operations account to cover the costs of operating and maintaining STARS and (2) the facilities and equipment account to upgrade STARS technology, it has not yet developed detailed estimates of the program’s operations and maintenance and technology upgrade costs beyond fiscal year 2007. FAA officials noted that FAA, like most federal agencies, develops its budget in 5-year plans and has not yet begun to develop detailed budgets beyond fiscal year 2007. Consequently, FAA does not currently know what impact the estimated costs of operating, maintaining, and upgrading STARS will have on the agency’s future budgets. Furthermore, as we previously noted, FAA has limited operational experience with STARS to use in estimating and budgeting for the costs of the program’s operations, maintenance, and technology upgrades. Finally, the uncertainty of long-range estimates makes it difficult to determine the impact of STARS’s estimated costs on FAA’s budgets for outlying fiscal years. After Deploying STARS at 74 Facilities, FAA May Modify Its Approach for the Remaining Facilities: FAA is committed to deploying STARS at the 74 terminal and support facilities included in the STARS program, but for the nearly 100 other facilities that remain to be modernized, the agency has at least three options: It could deploy Raytheon’s STARS, it could procure Lockheed Martin Corporation’s Common Automated Radar Terminal System (Common ARTS) technology, or it could combine Raytheon’s STARS display system with Lockheed Martin’s Common ARTS processing system. Between 1997 and 1999, FAA deployed Common ARTS to 131 small to medium-sized facilities and to 5 larger facilities. From its experience with both systems, FAA knows that each works independently. However, by combining the two technologies, FAA could both (1) take advantage of the customized software developed to resolve complex computer-human (controller and technician) interface issues that accounted for a significant portion of STARS’s development costs and (2) continue to use portions of the Common ARTS equipment that it recently deployed. In October 2002, FAA asked Raytheon to consider the feasibility of merging the STARS display system with the Common ARTS processing system. According to the Manager of Terminal Automation, Raytheon finished negotiating a subcontract with Lockheed Martin in January 2003, and FAA anticipates that the two contractors will now begin analyzing the feasibility of a merger of components of the two systems. After FAA receives the results of this analysis, which it expects in April 2003, it plans to determine the cost and schedule for modernizing the remaining terminal facilities. Conclusions: FAA lacks accurate, valid, current data on the STARS program’s costs and progress. Without such data, FAA is limited in its ability to effectively oversee the contractor’s performance and reliably estimate future costs. FAA cannot use the contractor’s cost performance reports for these purposes until the contract’s performance measurement baseline has been revised to incorporate the results of contract modifications, and FAA has verified that the revised baseline is aligned with the current status of the contract. Furthermore, without performing a risk assessment to identify the program’s minimum, most likely, and maximum expected costs, FAA cannot reliably determine the level of uncertainty inherent in its cost estimates. Finally, by using point values, rather than ranges or other appropriate measures, to express its cost estimates, FAA is not revealing the extent of their uncertainty. Its current estimates, expressed as point values, are misleading because they convey undue certainty and limit disclosure of the program’s investment risks. As such, the estimates’ usefulness to program managers and congressional overseers is limited. The earned value management guidance that FAA has adopted from DCMA and the Acquisition System Toolset that it compiled for itself establish clear procurement management policies and procedures that are applicable to STARS and to subsequent terminal modernization programs. In light of FAA’s incomplete or inconsistent application of this guidance to the STARS program thus far and the resulting cost overruns and schedule delays, we believe it is essential that the agency revisit the guidance, not only when it updates its baseline and performs an integrated baseline review this spring, but also throughout the remainder of STARS and throughout subsequent terminal modernization programs. Recommendations for Executive Action: To improve FAA’s management of STARS and of subsequent terminal modernization programs and to provide the Congress with more reliable information for overseeing these programs, we recommend that the Secretary of Transportation direct the FAA Administrator to follow the agency’s guidance for managing major acquisition systems by: * establishing, maintaining, and controlling an accurate, valid, and current performance measurement baseline, which would include negotiating all authorized, unpriced work within 3 months; * conducting an integrated baseline review of any major contract modifications within 6 months; and: * preparing a rigorous life-cycle cost estimate, including a risk assessment, in accordance with the Acquisition System Toolset’s guidance and identifying the level of uncertainty inherent in the estimate. Agency Comments: We requested comments on a draft of this report from the Secretary of Transportation or his designee. On January 23, 2003, FAA officials, including the Manager of Terminal Automation, provided us with the following oral comments on the draft. FAA agreed with the general tone and intent of the draft report as well as with our conclusions and recommendations. FAA noted that, in focusing its time and energy on meeting the technical performance and schedule requirements of STARS, it spent less time and energy on its business management of the program. However, the manager noted, the agency is now taking steps to reduce the program’s cost risks, such as defining and negotiating with the prime contractor all of the contract work planned for fiscal years 2003 through 2005. FAA also plans to conduct an integrated baseline review when it receives its appropriation for fiscal year 2003. According to FAA, our recommendations will further strengthen FAA’s commitment to improve the business management of the STARS program. FAA provided additional clarifying and technical information, which we incorporated as appropriate. Scope and Methodology: To conduct our work, we reviewed FAA’s 5-year Capital Investment Plan, which proposes funding for programs to modernize the national airspace system. We also analyzed data from cost performance reports that the STARS contractor developed for FAA. We interviewed FAA officials responsible for air traffic control modernization planning and budgeting. We also reviewed data from the FAA STARS program, contracting officials, and the Department of Transportation’s Office of the Inspector General. We did not independently verify the cost and performance data we received from FAA. We performed our work from October 2002 through January 2003 in accordance with generally accepted government auditing standards. As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 5 days after the date of this letter. At that time, we will send copies to interested Members of Congress, the Secretary of Transportation, and the FAA Administrator. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Should you or your staff have questions on matters discussed in this report, please contact Gerald L. Dillingham at (202) 512-2834 or Keith A. Rhodes at (202) 512-6412. We can also be reached by E-mail at dillinghamg@gao.gov and rhodesk@gao.gov, respectively. GAO contacts and key contributors to this report are listed in appendix I. Sincerely yours, Gerald L. Dillingham, Ph.D. Director, Physical Infrastructure Issues: Signed by Gerald L. Dillingham: Keith A. Rhodes Chief Technologist, Applied Research and Methods: Signed by Keith A. Rhodes [End of section] Appendixes: Appendix I: GAO Contacts and Staff Acknowledgments: GAO Contacts: Belva Martin (202) 512-2834 Madhav Panwar (202) 512-6228: Staff Acknowledgments: In addition to those individuals named above, Yvette Banks, Geraldine Beard, Jennifer Echard, Elizabeth Eisenstadt, Elizabeth Marchak, and Karen Richey made key contributions to this report.: FOOTNOTES [1] A previous version of STARS has been in use at smaller facilities since 1999. Philadelphia is the 14th-largest terminal facility in terms of handling operations under instrument flight rules. FAA’s data show that Philadelphia handled 686,000 operations between January and December 2000. [2] To support STARS operations at terminal facilities, FAA currently has four systems dedicated to maintenance. [3] The Department of Defense (DOD) is also procuring STARS for 153 of its facilities. [4] FAA currently plans to deploy STARS to a total of about 170 terminal and support facilities. [5] U.S. General Accounting Office, National Airspace System: Status of FAA’s Standard Terminal Automation Replacement System, GAO-02-1071 (Washington, D.C.: Sept. 17, 2002). [6] The report does not address DOD’s efforts to deploy the equipment. [7] Throughout this report, we use the term “deploy” to denote efforts by FAA to put STARS software, hardware, and other supporting equipment into a facility to test it and eventually use it to control traffic. [8] FAA noted that the $1.46 billion would be predominantly for labor costs that the agency would incur for STARS or a similar automation system. [9] Defense Contract Management Agency, Earned Value Management Implementation Guide (October 1997). [10] This account funds security activities related mainly to the security of FAA facilities and equipment. Funding for some airport security comes from another FAA account--Grants-in-Aid for Airports. [11] GAO-02-1071. [12] As previously noted, FAA officials said the $1.46 billion would be labor costs that the agency would incur for STARS or a similar automation system. [13] Earned Value Management Guide (October 1997). [14] The purpose of this review, conducted jointly by the agency and the contractor, is to understand and assess the adequacy, accuracy, and risks of a performance measurement baseline with respect to the contract’s work scope, schedule, technical requirements, and resource availability. [15] U.S. General Accounting Office, Air Traffic Control: Improved Cost Information Needed to Make Billion Dollar Modernization Investment Decisions, GAO/AIMD-97-20 (Washington, D.C.: Jan. 22, 1997). [16] The 90 percent confidence level is based on an analysis of the estimate’s uncertainty made by using a Monte Carlo model. GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: