This is the accessible text file for GAO report number GAO-10-843 entitled 'Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges' which was released on October 12, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: September 2010: Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges: GAO-10-843: GAO Highlights: Highlights of GAO-10-843, a report to congressional requesters. Why GAO Did This Study: The National Weather Service’s (NWS) weather products are a vital component of the Federal Aviation Administration’s (FAA) air traffic control system. In addition to providing aviation weather products developed at its own facilities, NWS also provides on-site staff at each of FAA’s en route centers—the facilities that control high- altitude flight outside the airport tower and terminal areas. NWS’s on- site staff is called a center weather service unit. For several years, NWS and FAA have been exploring options for improving the aviation weather services provided at en route centers. GAO agreed to (1) determine the status of the agencies’ efforts to restructure aviation weather services, (2) assess the agencies’ progress in establishing performance baselines in order to measure the effect of any changes, and (3) evaluate plans to address key challenges. To do so, GAO evaluated agency progress and plans and compared agency efforts with leading practices. What GAO Found: After developing and shelving four proposals for restructuring the center weather service units over the last 5 years, in July 2010, senior NWS and FAA officials agreed to continue the current center weather service units at each of the 21 en route centers through September 2011 and to take immediate steps to improve aviation weather services by (1) having the service units provide forecasts for 10 key FAA terminal radar facilities and (2) having nearby weather forecast offices support FAA’s en route centers when the service units are closed for the night. In addition, the agencies agreed to establish a joint team to baseline current capabilities and develop firm requirements for aviation weather services supporting air traffic flow management. While this agreement is important, the details have not been fully defined. Thus, it is not yet clear what will happen to the 21 service units after September 2011, when the immediate improvements in services will be in place, whether there are any costs associated with these steps, and who will pay for them. Until the two agencies further define their plans, the risk remains that the agencies will misjudge their responsibilities and not fulfill their agreements. FAA and NWS have made progress in identifying performance measures for the weather service units located at FAA en route centers, and NWS is beginning to track its service units’ performance. However, NWS has not yet tracked, established baselines for, and reported to FAA on all applicable performance measures. Specifically, of seven possible performance measures, NWS is tracking performance for three of the measures and partially tracking a fourth measure. Of these four measures, the agency has established a sound baseline and reported on two of these measures and has made partial progress on two others. The agency is not tracking performance, documenting baselines, or reporting on three of the measures because it has not yet determined how to track them. Without an understanding of the current level of performance of the identified measures, the agencies will be limited in their ability to evaluate what progress has been made. In addition, until NWS regularly reports on its performance, the agencies lack the information they need to determine what is working well and what needs to be improved. In September 2009, GAO identified three challenges in modifying NWS’s aviation weather services provided at FAA’s en route centers: achieving interagency collaboration, defining requirements, and aligning changes with the Next Generation Air Transportation System (NextGen)-—a long-term initiative to increase the efficiency of the national airspace system. The agencies have not yet fully addressed these challenges. Specifically, while senior agency officials recently agreed on how to proceed, work remains to be done to refine requirements, develop and execute an implementation plan, and to ensure that improvements are aligned with the long-term vision for NextGen. Until these fundamental challenges are addressed, the agencies are unlikely to achieve significant improvements in the aviation weather services provided at en route centers. What GAO Recommends: GAO recommends that the Departments of Commerce and Transportation define their agreements, refine performance management processes, and address key challenges. In commenting on a draft of this report, Commerce agreed with GAO’s recommendations and identified plans to address them; Transportation agreed to consider the recommendations. View [hyperlink, http://www.gao.gov/products/GAO-10-843] or key components. For more information, contact David A. Powner at (202) 512- 9286 or pownerd@gao.gov. [End of section] Contents: Letter: Background: NWS and FAA Officials Agreed to Improve Services, but This Agreement Is Not Fully Defined: NWS Has Made Progress in Measuring Center Weather Service Unit Performance but Has Not Yet Established Performance Baselines or Reported on Key Measures: NWS and FAA Have Not Fully Addressed Key Challenges: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Key FAA Concerns and NWS Steps to Address Them: Appendix III: Comments from the Department of Commerce: Appendix IV: GAO Contact and Staff Acknowledgments: Tables: Table 1: Key Weather Products Provided by the Aviation Weather Center: Table 2: Key Systems Used in the Center Weather Service Units: Table 3: Key Products and Services Provided by Center Weather Service Units: Table 4: Chronology of Efforts by FAA and NWS: Table 5: Performance Measures for Center Weather Service Units: Table 6: Key FAA Concerns and NWS Actions: Figures: Figure 1: FAA Facilities Involved in Air Traffic Control: Figure 2: Center Weather Service Unit Locations and Service Areas: Figure 3: Analysis of NWS Performance Measurement: Abbreviations: FAA: Federal Aviation Administration: NextGen: Next Generation Air Transportation System: NOAA: National Oceanic and Atmospheric Administration: NWS: National Weather Service: [End of section] United States Government Accountability Office: Washington, DC 20548: September 9, 2010: The Honorable Bart Gordon: Chairman: Committee on Science and Technology: House of Representatives: The Honorable Brad Miller: Chairman: Subcommittee on Investigations and Oversight: Committee on Science and Technology: House of Representatives: The National Weather Service (NWS) plays a significant role in providing weather services to the aviation community. NWS's weather products and data are vital components of the Federal Aviation Administration's (FAA) air traffic control system, providing weather information to local, regional, and national air traffic management, navigation, and surveillance systems. NWS aviation weather products include forecasts and warnings of meteorological conditions that could affect air traffic, including thunderstorms, air turbulence, and icing. In addition to providing aviation weather products that are developed at its own facilities, NWS also provides staff on site at each of FAA's en route centers--the facilities that control high- altitude flight outside the airport tower and terminal areas. This group of NWS meteorologists--called a center weather service unit-- provides air traffic staff with forecasts, advisories, and periodic weather briefings on regional conditions. Over the last few years, FAA and NWS have been exploring options for enhancing the efficiency of the aviation weather services provided at en route centers. In September 2005, FAA asked NWS to restructure its services to be more efficient. Since then, at FAA's request, NWS has submitted five different proposals for restructuring its services. Because of your interest in possible changes to NWS's aviation weather services, we agreed to (1) determine the status of the agencies' efforts to restructure aviation weather services and products, (2) assess the agencies' progress in establishing performance baselines in order to measure the effect of any changes, and (3) evaluate plans to address key challenges. To address our objectives, we analyzed agency documentation, including FAA's requirements for aviation weather services and NWS's proposals. We evaluated the agencies' progress in establishing a baseline of performance and compared NWS's performance measurement practices with government and industry guidance and best practices. To evaluate plans to address challenges, we compared agency efforts with leading practices in industry and government on interagency collaboration and system development. We also interviewed officials of both agencies. We conducted our work at NOAA and FAA facilities in the Washington, D.C., metropolitan area. We conducted this performance audit from October 2009 to September 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Additional details on our objectives, scope, and methodology are provided in appendix I. Background: FAA is responsible for ensuring safe, orderly, and efficient air travel in and around the United States. NWS supports FAA by providing aviation-related forecasts and warnings at air traffic facilities across the country. Among other support and services, NWS provides four meteorologists at each of FAA's 21 en route centers to provide on- site aviation weather services. This arrangement is defined and funded under an interagency agreement. FAA's Mission and Organizational Structure: In performing its primary mission to ensure safe air travel, FAA reported that air traffic in the national airspace system exceeded 43 million flights and 745 million passengers in 2008. In addition, at any one time, as many as 7,000 aircraft--both civilian and military-- could be aloft over the United States. In 2004, FAA's Air Traffic Organization was formed to, among other responsibilities, improve the provision of air traffic services. More than 34,000 employees within FAA's Air Traffic Organization support the operations that help move aircraft through the national airspace system. The agency's ability to fulfill its mission depends on the adequacy and reliability of its air traffic control systems, as well as weather forecasts made available by NWS and automated systems. These resources reside at, or are associated with, several types of facilities: air traffic control towers, terminal radar approach control facilities, air route traffic control centers (en route centers), and the Air Traffic Control System Command Center. The number and functions of these facilities are as follows: * 510 air traffic control towers manage and control the airspace within about 5 miles of an airport. They control departures and landings, as well as ground operations on airport taxiways and runways. * 163 terminal radar approach control facilities provide air traffic control services for airspace within approximately 40 miles of an airport and generally up to 10,000 feet above the airport, where en route centers' control begins. Terminal controllers establish and maintain the sequence and separation of aircraft. * 21 en route centers control planes over the United States--in transit and during approaches to some airports. Each center handles a different region of airspace. En route centers operate the computer suite that processes radar surveillance and flight planning data, reformats the data for presentation purposes, and sends it to display equipment used by controllers to track aircraft. The centers control the switching of voice communications between aircraft and the center, as well as between the center and other air traffic control facilities. Four of these en route centers also control air traffic over the oceans. * The Air Traffic Control System Command Center manages the flow of air traffic within the United States. This facility regulates air traffic when weather, equipment, runway closures, or other conditions place stress on the national airspace system. In these instances, traffic management specialists at the command center take action to modify traffic demands in order to keep traffic within system capacity. See figure 1 for a visual summary of the facilities that control and manage air traffic over the United States. Figure 1: FAA Facilities Involved in Air Traffic Control: [Refer to PDF for image: illustration] Air Traffic Control Tower: Preflight; Takeoff. Terminal Radar Approach Control: Departure. Air Route Traffic Control Center: Departure; En route; descent. Terminal Radar Approach Control: Descent; Approach. Air Traffic Control Tower: Landing. Source: GAO analysis of FAA data. [End of figure] NWS's Mission and Organizational Structure: The mission of NWS--an agency within the Department of Commerce's National Oceanic and Atmospheric Administration (NOAA)--is to provide weather, water, and climate forecasts and warnings for the United States, its territories, and its adjacent waters and oceans to protect life and property and to enhance the national economy. In addition, NWS is the official source of aviation-and marine-related weather forecasts and warnings, as well as warnings about life-threatening weather situations. The coordinated activities of weather facilities throughout the United States allow NWS to deliver a broad spectrum of climate, weather, water, and space weather services in support of its mission. These facilities include 122 weather forecast offices located across the country that provide a wide variety of weather, water, and climate services for their local county warning areas, including advisories, warnings, and forecasts; 9 national prediction centers[Footnote 1] that provide nationwide computer modeling to all NWS field offices; and 21 center weather service units that are located at FAA en route centers across the nation and provide meteorological support to air traffic controllers. NWS Provides Aviation Weather Services to FAA: As an official source of aviation weather forecasts and warnings, several NWS facilities provide aviation weather products and services to FAA and the aviation sector. These facilities include the Aviation Weather Center, weather forecast offices located across the country, and 21 center weather service units located at FAA en route centers across the country. Aviation Weather Center: The Aviation Weather Center located in Kansas City, Missouri, issues warnings, forecasts, and analyses of hazardous weather for aviation. Staffed by 55 personnel, the center develops warnings of hazardous weather for aircraft in flight and forecasts of weather conditions for the next 2 days that could affect both domestic and international aviation. The center also produces a Collaborative Convective Forecast Product, a graphical representation of expected thunderstorms or related conditions at 2, 4, and 6 hours. This is used by FAA to manage aviation traffic flow across the country. The Aviation Weather Center's key products are described in table 1. Table 1: Key Weather Products Provided by the Aviation Weather Center: Weather product: Significant Meteorological Information; Description: An advisory concerning the occurrence or expected occurrence of potentially hazardous weather conditions that may affect the safety of aircraft operations in the en route environment. Weather product: Convective Significant Meteorological Information; Description: A text product describing the occurrence or expected occurrence of thunderstorms and related weather conditions over the contiguous United States within 2 hours of issuance time. Weather product: Airman's Meteorological Information; Description: An advisory concerning the occurrence or expected occurrence of certain weather conditions that may affect the safety of aircraft in the en route environment, but at intensities that do not meet the criteria to develop a Significant Meteorological Information product. Weather product: Collaborative Convective Forecast Product; Description: A graphical convection forecast developed for strategic planning and management of en route air traffic. It is produced every 2 hours through collaboration--by way of an online chat room--among the Aviation Weather Center, the Meteorological Services of Canada, industry meteorologists, and the center weather service units. These collaborative forecasts are produced between March 1 and October 31 every year. Source: GAO analysis of NWS data. [End of table] Weather Forecast Offices: NWS's 122 weather forecast offices issue terminal area forecasts for approximately 632 locations every 6 hours or when conditions change, consisting of the expected weather conditions significant to a given airport or terminal area, and are primarily used by commercial and general aviation pilots. The terminal area forecasts are updated every 3 hours for 35 key airports and every 2 hours for the airports in New York, Atlanta, and Chicago. Center Weather Service Units: NWS's center weather service units are located at each of FAA's 21 en route centers and operate 16 hours a day, 7 days a week (see figure 2). Each center weather service unit usually consists of three meteorologists and a meteorologist-in-charge who provide strategic advice and aviation weather forecasts to FAA traffic management personnel. Governed by an interagency agreement, FAA currently reimburses NWS approximately $13 million annually for this support. Figure 2: Center Weather Service Unit Locations and Service Areas: [Refer to PDF for image: illustrated map] Center Weather Service Unit Locations: Albuquerque; Anchorage; Atlanta; Boston; Chicago; Cleveland; Denver; Fort Worth; Houston; Indianapolis; Jacksonville; Kansas City; Los Angeles; Memphis; Miami; Minneapolis; New York; Oakland; Salt Lake City; Seattle. Washington, DC. Sources: NWS (data); Map Resources (map). Note: Four center weather service units (Anchorage, Oakland, Miami, and New York) have oceanic responsibilities that are not pictured here. [End of figure] Center Weather Service Units: An Overview of Systems and Operations: The meteorologists at the center weather service units use a variety of systems to gather and analyze information compiled from NWS and FAA weather sensors. Key systems used to compile weather information include FAA's Weather and Radar Processor, FAA's Integrated Terminal Weather System, FAA's Corridor Integrated Weather System, and a remote display of NWS's Advanced Weather Interactive Processing System. Meteorologists at several center weather service units also use NWS's National Centers--Advanced Weather Interactive Processing System. Table 2 provides a description of key systems. Table 2: Key Systems Used in the Center Weather Service Units: System: Weather and Radar Processor; Description: This FAA system is used in en route centers and receives NWS products and data, information from automated weather sensors located at airports, and data from other sources such as weather satellites and radars. It compiles the information and provides current weather and forecasts to air traffic supervisors, traffic flow managers, and the center weather service unit meteorologists. System: Advanced Weather Interactive Processing System--Remote Display; Description: This NWS system integrates hydrometeorological data from a variety of sources and produces graphical displays at NWS weather forecast offices, river forecast centers, and national centers. This system aids forecaster analysis and decision making. Meteorologists at the en route centers have access to this system through a Remote Display system, which provides a dedicated connection to the supporting weather forecast office. The Remote Display is funded by FAA, and maintenance is provided by NWS. System: Integrated Terminal Weather System; Description: This FAA system furnishes air traffic controllers and meteorologists with full-color graphic displays of weather information concerning airport terminal airspace within a 60-mile radius. The system also projects movement of severe weather systems up to 1 hour in the future and has been installed at 39 airports. System: Corridor Integrated Weather System; Description: This FAA system is a prototype decision support tool that gathers weather information to help controllers select the most efficient routes for diverting traffic to avoid severe weather conditions. This system provides traffic flow managers with comprehensive convective weather data needed for tactical modifications occurring within 2 hours to the operational plan. These tactical modifications to the operational plan may include weather impacts on air traffic control capacity, a need to modify the mitigation plan, and the execution of a modified mitigation plan. System: National Centers--Advanced Weather Interactive Processing System; Description: This NWS system is a meteorological data visualization and integrated product generation system that provides a national scope of weather information. It is composed of software that ingests, analyzes, displays, and integrates various types of hydrometeorological data including numerical model, surface, upper- air, satellite, radar, and text data. This system is used in only a few center weather service units. Sources: GAO analysis of FAA and NWS data. [End of table] NWS meteorologists at the en route centers provide several products and services to the FAA staff, including meteorological impact statements, center weather advisories, periodic briefings, and on- demand consultations. These products and services are described in table 3. In addition, center weather service unit meteorologists receive and disseminate pilot reports, provide input every 2 hours to the Aviation Weather Center's creation of the Collaborative Convective Forecast Product, train FAA personnel on how to interpret weather information, and provide weather briefings to nearby terminal radar approach control facilities and air traffic control towers. Table 3: Key Products and Services Provided by Center Weather Service Units: Product or service: Meteorological impact statement; Description: An unscheduled forecast of weather conditions that are expected to adversely impact the flow of air traffic in the en route center's area of responsibility within 4 to 12 hours. Product or service: Center weather advisory; Description: A short-term, unscheduled warning of hazardous weather conditions used primarily by air crews to anticipate and avoid adverse weather conditions in the en route and terminal environments. It describes current weather conditions or adverse weather conditions-- such as moderate to severe icing or turbulence, thunderstorms, low- level wind shear, and low ceilings and visibility--beginning within the next 2 hours. Product or service: Briefings; Description: Short updates provided by NWS meteorologists to FAA supervisors twice a day: these briefings include current weather warnings and advisories, a summary of forecasted weather across the national airspace, terminal forecasts, and other pertinent meteorological information. Product or service: On-demand consultation; Description: Unscheduled verbal presentations regarding ongoing or expected weather conditions provided to FAA traffic control personnel, supervisors, and other FAA facilities. Sources: GAO analysis of FAA and NWS data. [End of table] FAA Seeks to Improve Aviation Weather Services Provided at En Route Centers: In recent years, FAA has undertaken multiple initiatives to assess and improve the performance of the center weather service units.[Footnote 2] Studies conducted in 2003 and 2006 highlighted concerns with the lack of standardization of products and services at NWS's center weather service units. To address these concerns, the agency sponsored studies that determined that weather data could be provided remotely using current technologies, and that private sector vendors could provide these services. In 2005, the agency requested that NWS restructure its aviation weather services by consolidating its center weather service units to a smaller number of sites, reducing personnel costs, and providing products and services 24 hours a day, 7 days a week. NWS subsequently submitted a proposal for restructuring its services, but FAA declined the proposal citing the need to refine its requirements. In December 2007, FAA issued revised requirements and asked NWS to respond with proposals defining the technical and cost implications of three operational concepts. The three concepts involved (1) on-site services provided within the existing configuration of offices located at the 21 en route centers, (2) remote services provided by a reduced number of regional facilities, and (3) remote services provided by a single centralized facility. NWS responded with three proposals, but FAA rejected these proposals in September 2008, noting that while elements of each proposal had merit, the proposed costs were too high. FAA requested that NWS revise its proposal to bring costs down while stating a preference to move toward a single center weather service unit with a backup site. As a separate initiative, NWS began a series of improvements in order to address FAA's key concerns. Specifically, in April 2008, the agency initiated a program to improve the consistency of the center weather service units' products and services. This program involved standardizing the technology, collaboration, and training for all 21 center weather service units and conducting site visits to evaluate and provide feedback to each unit. NWS reported that it completed these efforts in 2009. A summary of FAA's key concerns and NWS's efforts to address them is included in appendix II. After two requests for deadline extensions on a new proposal, NWS provided FAA with an updated proposal in June 2009 based on the two- site approach FAA had requested in September 2008. FAA responded to NWS's proposal by requesting more information and stated that the agencies would work together to resolve issues. From September through November 2009, the agencies completed a series of meetings to address issues from the proposal and agreed that NWS would resubmit its proposal in December 2009 to consolidate the service units. In December 2009, FAA revised its requirements to reflect the agencies' efforts aimed at improving center weather service operations. However, NWS did not submit its proposal in December 2009 to consolidate the center weather service units. According to NWS officials, they decided not to submit the proposal because (1) the NWS labor union and others raised concerns about consolidating offices, (2) NWS could implement technical improvements more quickly under the current organizational structure, and (3) the agency wanted to focus its efforts and resources on future weather system development rather than restructuring existing operations. Table 4 provides a chronology of the agencies' assessment and improvement efforts. Table 4: Chronology of Efforts by FAA and NWS: Time frame: November 2003; Activity: FAA performed a functional audit of center weather service units and found that the services provided at different en route locations were inconsistent, that products were not standardized, and there was little communication and collaboration between neighboring service units. Time frame: September 2005; Activity: FAA requested that NWS restructure its aviation weather services to provide improved services more efficiently. Specifically, FAA requested that NWS consolidate 20 of the center weather service units (excluding the unit in Alaska) to a smaller number of sites, reduce NWS personnel costs by 20 percent, and deliver forecast products and services 24 hours a day, 7 days a week. Time frame: January 2006; Activity: FAA initiated an analysis of the value of different activities performed by the center weather service units. Similar to the 2003 study, the results of this analysis noted the lack of standardization of products, services, tools, and procedures. In addition, the report found that quality assurance was provided on an informal basis, there was no formal feedback process for products and services, and meteorological training was not standardized. Time frame: August 2006; Activity: NWS conducted a prototype in which center weather service unit products and services were completed and delivered remotely from the closest weather forecast office. This prototype showed that remote operations were possible and effective, but that they would be difficult to implement because of the need for cultural change, technology upgrades, and communication stability. Time frame: September 2006; Activity: An FAA study reported that it was possible to deliver weather information, products, and services from one or many locations with currently available state-of-the-art technology platforms. Time frame: October 2006; Activity: FAA administered a market survey to determine whether the private sector could provide remote weather services at a lower cost than currently provided. Ten organizations, including private sector firms and government-funded laboratories, responded that they could provide the services that FAA wanted; Separately, NWS presented its first proposal for restructuring its aviation weather services to FAA. In this proposal, NWS suggested moving meteorologists from the en route centers to regional weather forecast offices and providing remote aviation weather services from the weather forecast offices. Time frame: April 2007; Activity: FAA declined NWS's proposal. Instead, FAA reported that it would redefine its requirements for the functions provided by the center weather service units. Time frame: December 2007; Activity: FAA transmitted its redefined requirements to NWS and requested a written response detailing three different operational concepts. Time frame: April 2008; Activity: NWS initiated a short-term improvement program for the center weather service units. The goal of this program was to standardize the technology and training for the units to improve the consistency of products and services. Time frame: May 2008; Activity: In response to the new requirements, NWS provided FAA with a second series of proposals, which identified three alternatives for restructuring the center weather service units. Time frame: September 2008; Activity: FAA rejected NWS's second series of proposals, stating that while elements of each alternative had merit, the agency could not accept them because the costs were too high. FAA requested that NWS deliver a revised proposal by December 2008, stating a preference to move toward a single center weather service unit with a back-up site. Time frame: December 2008; Activity: NWS requested and FAA approved a 60-day extension on the deadline for NWS's third proposal to allow time to address any public misperceptions regarding the proposed changes. Time frame: February 2009; Activity: NWS requested another 60-day extension on the deadline for NWS's proposal to allow the new NOAA Administrator time to work with the then-unnamed FAA Administrator on the consolidation. FAA approved a 30-day extension. Time frame: June 2009; Activity: NWS provided FAA with its third proposal, which was to consolidate 20 of the 21 center weather service units into two locations. Time frame: August 2009; Activity: FAA responded to NWS's proposal by requesting more information and stating that the agencies would work together to resolve issues. Time frame: November 2009; Activity: The agencies completed a series of meetings to resolve issues regarding the proposal and agreed that NWS would submit its fourth proposal to consolidate the center weather service units in December 2009. Time frame: December 2009; Activity: FAA revised its prior set of requirements to reflect the agencies' collaboration during meetings held earlier in the year; NWS did not provide its fourth proposal to consolidate center weather service units to FAA because (1) the NWS labor union and others raised concerns about consolidating offices, (2) NWS could implement technical improvements more quickly under the current organizational structure, and (3) the agency wanted to focus its efforts and resources on future weather system development rather than restructuring operations. NWS officials decided to consider other options. Sources: GAO analysis of NWS and FAA data. [End of table] Prior GAO Reports Identified Concerns with Center Weather Service Units; Recommended Steps to Improve Quality Assurance: In January 2008, we reported on concerns about inconsistencies in products and quality among center weather service units.[Footnote 3] We noted that while both NWS and FAA have responsibilities for assuring and controlling the quality of aviation weather observations, neither agency monitored the accuracy and quality of the aviation weather products provided at center weather service units, performed annual evaluations of aviation weather services provided at en route centers, and provided feedback to the center weather service units. We recommended they do so. The Department of Commerce agreed with our recommendations, and the Department of Transportation stated that FAA planned to revise its requirements and that these would establish performance measures and evaluation procedures. In September 2009, we reported that the agencies were considering plans to consolidate 20 of the 21 existing center weather service units to two locations, but it was not clear whether and how the changes would be implemented.[Footnote 4] Moreover, we reported that NWS and FAA faced challenges in their efforts to improve the aviation weather structure, including achieving interagency collaboration, defining FAA's requirements, and aligning any changes with the Next Generation Air Transportation System. We also identified three challenges the agencies would face in implementing their plans-- developing a feasible schedule that includes adequate time for stakeholder involvement, undertaking a comprehensive demonstration to ensure no services are degraded, and effectively reconfiguring the infrastructure and technologies. We recommended that the agencies address these challenges, and NOAA and the Department of Transportation agreed with our recommendations. NWS and FAA Officials Agreed to Improve Services, but This Agreement Is Not Fully Defined: After developing and shelving four different proposals for restructuring the center weather service units over the last 5 years, NWS and FAA have reached agreement on how to improve aviation weather services. In March 2010, NWS proposed maintaining the current 21 center weather service units collocated at en route centers, increasing staffing at the Aviation Weather Center in order to provide remote services during the service units' off-hours, and developing a new collaborative weather product. NWS estimated that these improvements would cost FAA about $3 million per year. This is in addition to the annual cost of maintaining the existing 21 centers. [Footnote 5] NWS also estimated that it would be able to implement the proposal within 21 months. FAA responded that it was not prepared to accept the proposal because of the increased costs. Subsequently, in July 2010, FAA and NWS reached an agreement on the steps the two agencies would take to improve aviation weather services. Specifically, FAA proposed and NWS agreed to continue the current center weather service units at each of the 21 en route centers through September 2011 and to take immediate steps to improve aviation weather services by (1) having the service units provide forecasts at 10 key FAA terminal radar approach control facilities [Footnote 6] and (2) providing around-the-clock coverage at all of the en route centers by having the local weather forecast office support the en route centers when the center weather service units are closed for the night--a practice that currently is used at selected en route centers. In addition, the agencies agreed to establish a joint team to baseline current capabilities and develop firm requirements for NWS products and services supporting FAA's air traffic flow management out through 2015. The agencies expect that the joint team will establish an implementation plan by November 2010 and then begin to implement it. However, the agencies' documentation of this agreement does not address the future locations of the center weather service units, or provide details and a schedule for the proposed improvements to services. As a result, it is not clear what will happen to the 21 service units after September 2011, when the immediate improvements in services will be in place, whether there are any costs associated with these steps, whether the benefits outweigh the costs, and who will pay for them. Until this agreement is further defined in writing and formalized between the two agencies, the risks remain that the agencies will misjudge their responsibilities and not fulfill their agreements. NWS Has Made Progress in Measuring Center Weather Service Unit Performance but Has Not Yet Established Performance Baselines or Reported on Key Measures: According to best practices in the federal government and in industry, organizations should measure performance in order to evaluate the success or failure of their activities and programs.[Footnote 7] Performance measurement involves identifying performance goals and measures, establishing performance baselines by tracking performance over time, identifying targets for improving performance, and measuring progress against those targets. In January 2008, we recommended that NWS and FAA develop performance measures and track metrics for the products and services provided by center weather service units and that they provide feedback to the center weather service units so that they could improve their performance.[Footnote 8] Further, in September 2009, we recommended that the agencies approve their draft performance measures and establish performance baselines so that they could understand the effects of any changes from restructuring aviation weather services.[Footnote 9] Over the past year, NWS has made progress in identifying performance measures, tracking performance on selected measures, and reporting on the selected measures; however, the agency is not yet tracking or reporting on all applicable performance measures. In December 2008, FAA provided NWS five performance measures of center weather service unit performance. Under the current interagency agreement, NWS is required to track and report to FAA on these measures. In addition, in its last two proposals, NWS proposed additional measures, two of which could be tracked under the current organizational structure and using current products.[Footnote 10] We previously recommended that NWS immediately identify the current level of performance of the proposed measures that could be identified under the current organizational structure, so that they will have a performance baseline to compare to should they decide to implement operational changes. The agency agreed with this recommendation. Table 5 describes the performance areas applicable to the current center weather service unit structure. Table 5: Performance Measures for Center Weather Service Units: Performance measure: Product participation; Description: 90 percent participation in the Collaborative Convective Forecast Product when convection is expected to occur in an en route center's domain; Source: Required by interagency agreement. Performance measure: Format consistency; Description: Consistency of two products (the meteorological impact statement and the center weather advisory)--including the products' formats, information content, and procedures for issuance across all en route centers; Source: Required by interagency agreement. Performance measure: Service provision (briefings); Description: Daily briefings delivered twice a day are to be provided 100 percent of the time; Source: Required by interagency agreement. Performance measure: Service provision (organizational); Description: Center weather service unit services are to be provided on-site or in backup mode, 16 hours a day, 7 days a week 100 percent of the time; Source: Required by interagency agreement. Performance measure: Forecast accuracy; Description: A determination of the accuracy of forecasts used in decisions for traffic management initiatives; Source: Required by interagency agreement. Performance measure: Customer satisfaction; Description: A measure of satisfaction with product quality, timeliness, accuracy, and customer service, as well as the number of complaints received; Source: Proposed by NWS. Performance measure: Training completion; Description: A measure of completion of standardized training; Source: Proposed by NWS. Sources: GAO analysis of FAA and NWS information. [End of table] NWS has started tracking performance for three of the seven measures and is partially tracking a fourth. Specifically, NWS has tracked data on each center weather service unit's (1) participation in the Collaborative Convective Forecast Product, (2) organizational service provision, and (3) customer satisfaction. Further, it has partially tracked data on format consistency, by collecting data on one of two required products. However, the agency has not tracked data on the other measures for a number of reasons. For example, the agency did not track the format consistency for the second of the two required products because, until recently, the briefing has not had a consistent format. Also, the agency is not tracking training completion because it has not yet determined what standardized training will be provided. For the forecast accuracy measure, agency officials stated that they do not currently have the means to track this measure, but that they are developing a tool to do so. Of the measures it is tracking, NWS has established baselines and reported its results on two measures and has partially done so for two other measures. Specifically, NWS has established baselines on each center weather service unit's participation in the Collaborative Convective Forecast Product and its organizational service provision. In addition, NWS has partially established a baseline on the format consistency measure in that it has historical data for one of the two required products. However, because it has not tracked the format consistency of the second product, NWS has not established a complete baseline for that measure. Further, while NWS has calculated customer satisfaction scores from its 2009 site evaluations, it does not yet have a reliable baseline because it has not yet matured its approach to documenting this measure. Specifically, NWS changed its approach during its 2010 site evaluations, which will make it harder to compare scores from year to year. Moreover, the agency mixed positive and negative findings to come up with its rating scores for some sites, thereby rendering the 2009 scores at selected sites ineffective at measuring a site's performance. Figure 3 identifies NWS's efforts to track data, develop baselines, and report on the performance measurement areas. Figure 3: Analysis of NWS Performance Measurement: [Refer to PDF for image: table] Performance measure: Product participation; Currently tracking? Yes; Baseline established? Yes; Results reported? Yes; Status and evaluation: NWS has collected data on this measure for each center weather service unit since 2009. The agency provided feedback to the service units and has reported its results to FAA. Performance measure: Format consistency; Currently tracking? Partial; Baseline established? Partial; Results reported? Partial; Status and evaluation: NWS has collected data on one of two products since 2009 and reported its results for the first three months of 2010 to FAA in its quarterly progress reports. NWS recently developed a standardized format for the second product, but has not yet begun tracking it. Performance measure: Service provision (briefings); Currently tracking? No; Baseline established? No; Results reported? No; Status and evaluation: NWS has not proactively monitored this measure. NWS currently relies on center weather service units to report when there is an issue with briefings not being delivered. As of June 2010, the agency did not have plans to track this measure because it had not identified how to gather this data. In late August 2010, while this report was in final processing, NWS proposed an approach for tracking this measure. Performance measure: Service provision (organizational); Currently tracking? Yes; Baseline established? Yes; Results reported? Yes; Status and evaluation: NWS collected data on the hours of operation of each center weather service unit since 2009, and has reported its results for the first three months of 2010 to FAA in its quarterly progress report. Performance measure: Forecast accuracy; Currently tracking? No; Baseline established? No; Results reported? No; Status and evaluation: NWS is developing a tool that will capture forecast accuracy and plans to implement it later this year. Performance measure: Customer satisfaction; Currently tracking? Yes; Baseline established? Partial; Results reported? Partial; Status and evaluation: NWS officials are measuring customer satisfaction using a rating score derived from annual site evaluations. The rating score is based on the number of negative findings at each site. NWS conducted site evaluations of each service unit in 2009 and reported its results to FAA. NWS recently began a new round of site evaluations in 2010. However, the agency’s approach is not yet mature enough to form a reliable baseline of performance. Specifically, NWS modified its approach to the site visits in 2010 to factor in lessons learned from the 2009 visits, which will make it harder to compare scores from year to year and identify performance trends. Moreover, the agency mixed positive and negative findings to come up with its rating scores for some sites, thereby rendering the 2009 scores at selected sites ineffective at measuring a site’s performance and not valuable as a feedback mechanism. Performance measure: Training completion; Currently tracking? No; Baseline established? No; Results reported? No; Status and evaluation: While NWS reported it tracked the service units’ completion of customer service training in 2008, the agency noted that it has not yet determined what standardized training it will require. Source: GAO analysis based on NWS information. It is important for NWS and FAA to track performance in the identified measures in order to understand the value currently provided and to assess the impact of any changes they make to operations. Reporting also helps improve performance. For example, after reporting on its performance in product participation and organizational service provision for 2009, NWS noted significant improvements in 2010. Until the agencies track and develop a performance baseline for all applicable measures, they will be limited in their ability to evaluate progress that has been made and whether or not they are achieving their goals. In addition, until NWS regularly reports on its performance, the agencies lack the information they need to determine what is working well and what needs to be improved. Moreover, as the agencies refine their approach to performance measurement, it will be important to revisit and refine the performance measures to ensure an appropriate mix of process-and outcome-oriented measures. For example, NWS could consider measuring the number of aircraft incidents attributed to inaccurate aviation weather forecasts or the number of weather-related delays as a percentage of all delays. NWS and FAA Have Not Fully Addressed Key Challenges: In September 2009, we identified three challenges that FAA and NWS faced in modifying the current aviation weather structure: (1) achieving interagency collaboration, (2) defining requirements, (3) aligning changes with the Next Generation Air Transportation System (NextGen)--a long-term initiative to increase the efficiency of the national airspace system. The agencies have taken initial steps to collaborate, refine requirements, and look for ways to align their plans with NextGen, but they have not yet fully addressed the challenges. Until these fundamental challenges are addressed, the agencies are unlikely to achieve significant improvements in the aviation weather services provided at en route centers. Achieving Interagency Collaboration: We have previously reported on key practices that can help enhance and sustain interagency collaboration.[Footnote 11] The practices generally consist of two or more agencies defining a common outcome, establishing joint strategies to achieve the outcome, agreeing upon agency roles and responsibilities, establishing compatible policies and procedures to operate across agency boundaries, and developing mechanisms to monitor, evaluate, and report the results of collaborative efforts. In September 2009, we reported that NWS and FAA had not defined a common outcome for modifying the aviation weather services provided at en route centers, established joint strategies, or agreed upon their respective responsibilities.[Footnote 12] We recommended that the agencies complete these activities. NOAA and the Department of Transportation agreed with our recommendation. Since September 2009, NWS and FAA have made progress in defining a common outcome, but have not yet established joint strategies to achieve the outcome or agreed upon agency responsibilities. Specifically, in July 2010, the two agencies defined a common outcome when they reached an agreement to continue the current center weather service unit configuration at each of the 21 en route centers and to take immediate steps to improve aviation weather services. The two agencies also plan to form a team that will develop an implementation plan by November 2010. However, the agreement does not provide the details needed to establish joint strategies and only provides general agency responsibilities. Until the agencies establish joint strategies and agree on respective agency responsibilities, it may prove difficult to move forward in efforts to improve aviation weather services. Defining Requirements: According to best practices of leading organizations, requirements describe the functionality needed to meet user needs and perform as intended in the operational environment.[Footnote 13] A disciplined process for developing and managing requirements can help reduce the risks associated with developing or acquiring a system or product. In September 2009, we reported that FAA's requirements were unstable and recommended that the agencies establish and finalize requirements for aviation weather services at en route centers.[Footnote 14] NOAA and the Department of Transportation agreed with our recommendation. FAA updated its requirements in December 2009 based on the work that the two agencies did in the fall of 2009. However, these changes were nullified by the more recent decision to continue with 21 center weather service units. In July 2010, the two agencies agreed to establish a joint team to develop firm requirements for NWS products and services supporting FAA's air traffic flow management out to 2015, including those provided by the center weather service units. While this is an important step, significant work remains to be done to revise these requirements. Until the requirements are in place, the agencies may find it difficult to move forward in efforts to improve aviation weather services. Alignment with the Next Generation Air Transportation System: In September 2009, we reported that neither FAA nor NWS had ensured that their restructuring plans fit with the national vision for NextGen--a long-term initiative to transition FAA from the current radar-based system to an aircraft-centered, satellite-based system. We recommended that the agencies ensure that any proposed organizational changes are aligned by seeking a review by the Joint Planning and Development Office,[Footnote 15] the office responsible for planning and coordinating NextGen. NOAA and the Department of Transportation agreed with our recommendation. Among other agreements in July 2010, the two agencies plan to work together to develop requirements and an implementation plan that extends through 2015--the NextGen Midterm Operating Capability date-- by November 2010. However, because this plan has not been developed or approved, it is not clear that future actions will be aligned with NextGen. As NWS and FAA discuss the current proposal and plan improvements to aviation weather services, it will be important for the agencies to continue to ensure alignment with the long-term goals of NextGen. Conclusions: After many years of proposals and counter proposals for improving the center weather service units, NWS and FAA recently agreed to continue the current center weather service unit configuration at each of the 21 en route centers through September 2011 and to take immediate steps to improve aviation weather services. However, many questions remain about what will happen, when, and at what cost. Given the long history of unresolved issues between FAA and NWS regarding the center weather service units, it is more important than ever that the two agencies be extremely clear on what their commitments entail. An important component of any effort to improve operations is a solid understanding of current performance. While NWS has made progress in measuring the performance of the center weather service units, is not adequately documenting performance baselines or reporting on several of its performance measures. Further, the agency has begun efforts to measure customer satisfaction, but the process is immature, and the results are unreliable. Specifically, NWS has changed its approach to the annual evaluations making it difficult to compare performance from year to year, and its scoring process mixes positive and negative findings for several sites. As a result, the scores may not accurately reflect each center's performance. Until NWS has a solid understanding of the current level of performance, it will be limited in its ability to evaluate what progress has been made and whether or not it is achieving its goals. As the agencies move forward with plans to make aviation weather services more efficient, they continue to face challenges, including a record of false starts on interagency collaboration, unstable requirements, and a lack of assurance that operational changes will align with the future vision of NextGen. Until these challenges are fully addressed, the agencies will likely find it difficult to make meaningful changes in aviation weather services. Recommendations for Executive Action: To improve the aviation weather products and services provided at FAA's en route centers, we are making three recommendations to the Secretaries of Commerce and Transportation. Specifically, we recommend that the Secretaries direct the NWS and FAA Administrators to: * define, document, and sign the agencies' recent agreements on (1) the locations of the center weather service units, (2) immediate improvements in aviation weather services and operating hours, and (3) the development of an implementation plan for improvements through 2015; * ensure that NWS regularly tracks progress, documents performance baselines, and reports on its format consistency, forecast accuracy, and training performance measures; and: * ensure that NWS develops a reliable customer satisfaction baseline by refining the questions used during annual evaluations, so that comparable information is collected from year to year, and revising the scoring process to ensure that scores accurately reflect each center's performance. * In addition, we are reiterating our prior recommendations to the two agencies to address key challenges in achieving interagency collaboration, defining requirements, and aligning any organizational changes with plans for NextGen. Agency Comments and Our Evaluation: We received written comments on a draft of this report from the Secretary of Commerce, who transmitted NOAA's comments (see appendix III). In its comments, NOAA stated that our report is generally representative of challenges facing NWS and FAA in the execution of aviation weather services provided by the center weather service units. The agency agreed with our recommendations and identified plans to implement selected parts of the recommendations. Specifically, NOAA reiterated its plan to form a joint NWS/FAA team to determine weather requirements for traffic flow management and to implement products and services through the year 2015. NOAA stated that this team's results will serve as additional documentation of the agreements. In addition, NOAA reported that it plans to begin measuring format consistency in September 2010 and forecast accuracy in December 2010. NOAA also noted that it was using 2009 site evaluations as the basis for its scoring and that the 2009 results would serve as a baseline for comparison to 2010 and subsequent results. However, as we discuss in the report, our analysis of the 2009 site evaluations and scoring process found that the results were not reliable because the process for collecting information on customer service was inconsistent, and the scores did not always accurately reflect the centers' performance. As a result, the 2009 scores are not useful as a baseline or as a feedback tool. Moving forward, as NOAA analyzes the results from its ongoing 2010 site evaluations, it will be important to ensure that the scores accurately reflect each center's performance. Further, in future years, it will be important to ensure that comparable information is collected from year to year so that a reliable performance baseline can be established. The Department of Transportation's Director of Audit Relations provided comments on a draft of this report via e-mail. In those comments, he noted that the department agreed to consider our recommendations. Both departments also provided technical comments that we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Commerce, the Secretary of Transportation, the Director of the Office of Management and Budget, and other interested parties. The report also will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff members have any questions about this report, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. Signed by: David A. Powner: Director, Information Technology: Management Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives were to (1) determine the status of the agencies' efforts to restructure aviation weather services and products, (2) assess the agencies' progress in establishing performance baselines in order to measure the effect of any changes, and (3) evaluate plans to address key challenges. To determine the status of the agencies' efforts to restructure aviation weather services and products, we analyzed Federal Aviation Administration (FAA) and National Weather Service (NWS) documentation, including FAA's requirements for center weather service units, the interagency agreement between FAA and NWS, and NWS's proposals to meet FAA needs for center weather service units. We also interviewed officials from both agencies to discuss their plans and status in reaching a decision on proposed changes. To assess the agencies' progress in establishing performance baselines, we identified the agencies' previous efforts to establish baselines and evaluated the extent to which they have made progress in doing so. We analyzed NWS's approach to measuring center weather service unit performance and compared its performance measurement practices with guidance and best practices in performance management identified by government and industry. Specifically, we assessed the agencies' actions taken to identify performance measures, track them, establish baselines of performance, and report on those baselines. We also assessed the reliability of the performance data that NWS reported. Specifically, for the customer satisfaction measurements, we analyzed supporting data and calculated customer satisfaction scores using NWS's guidance for developing scores. We then compared the scores we calculated with NWS's scores. In instances where our scores did not match NWS's, we interviewed agency officials in order to determine why NWS's scores did not match our own, focusing on four sites with the largest number of findings. We found that the agency's customer satisfaction data was not reliable. For the other reported measures, we evaluated supporting data and interviewed responsible agency officials to determine the agency's processes for validating the data. We found that the data reported for these performance measures was sufficient to meet our reporting purposes. To evaluate plans to address key challenges identified in our prior report, we reviewed agency documents including FAA requirements, an NWS proposal, plans for the Next Generation Air Transportation System (NextGen), and FAA's response to NWS's proposal. We compared agency efforts with leading practices in industry and government on interagency collaboration and system development. In addition, we interviewed the FAA contracting officer's technical representative for the center weather service units to discuss the challenges the agency would have in implementing NWS's proposal, as well as the agency's plans to ensure requirements were stabilized. We also interviewed NWS officials to discuss their plans for aligning their system development initiatives with NextGen. We also interviewed the co-chair of the weather working group of the Joint Planning and Development Office to determine whether the office had reviewed NWS's proposal and if the office had concerns about the proposal's impact on NextGen. We conducted our work at National Oceanic and Atmospheric Administration (NOAA) and FAA facilities in the Washington, D.C., metropolitan area. We conducted this performance audit from October 2009 to September 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Key FAA Concerns and NWS Steps to Address Them: Table 6 lists concerns that FAA identified in a series of studies between 2003 and 2006, as well as the steps that NWS has taken to address these concerns. Table 6: Key FAA Concerns and NWS Actions: Category: Products and services; FAA concern: Lack of standardized products and services; NWS actions taken: * Conducted a terminal forecast improvement program; * Established a Web presence for the center weather service units; * Inserted aviation products in NWS operational system; * Created enhanced terminal forecast monitoring tools; * Initiated performance measurement of consistency in products and services. Category: Products and services; FAA concern: Dissatisfaction with the Collaborative Convective Forecast Product (particularly with accuracy and the time it takes to produce the product); NWS actions taken: * Initiated performance measurement of participation in this product; * Supported FAA research and development of tools that enhance NWS convective forecasts. Category: Products and services; FAA concern: Confusion on priority of center weather service unit services; NWS actions taken: No actions taken. Category: Products and services; FAA concern: Need for a forecast for terminal radar approach control facilities; NWS actions taken: * Established these forecasts in certain locations. Category: Procedures; FAA concern: Lack of standardized procedures; NWS actions taken: * Established a Web presence for the center weather service units. Category: Procedures; FAA concern: Lack of collaboration with nearby NWS offices; NWS actions taken: * Conducted a terminal forecast improvement program; * Created tools for enhanced terminal forecast monitoring. Category: Procedures; FAA concern: Category: Inadequate oversight of the center weather service units; NWS actions taken: * Conducted center weather service unit site reviews. Category: Procedures; FAA concern: Lack of routine evaluation and feedback; NWS actions taken: * Initiated annual site evaluations; * Established a new quality assurance manager position. Category: Training; FAA concern: Inadequate training; NWS actions taken: * Conducted NWS meteorology training. Category: Training; FAA concern: Lack of knowledge of center weather service unit customers; NWS actions taken: * Conducted customer service training; * Sent meteorologists to FAA's air traffic training. Category: Organizational support; FAA concern: Need for services 24 hours a day, 7 days a week; NWS actions taken: No actions taken. Category: Organizational support; FAA concern: Need for a reduction in the cost of services; NWS actions taken: No actions taken. Category: Organizational support; FAA concern: Need for restructuring to a smaller number of sites; NWS actions taken: No actions taken. Category: Organizational support; FAA concern: Desire for on-site meteorologists[A]; NWS actions taken: No actions taken. Category: Organizational support; FAA concern: Need for performance-based service; NWS actions taken: * Initiated a performance measurement program (identified performance measures and began measuring performance). Sources: GAO analysis of NWS and FAA data. [A] An FAA official stated that the concerns expressed over the years have evolved and that the agency does not currently consider the desire for on-site meteorologists as a concern. [End of table] [End of section] Appendix III: Comments from the Department of Commerce: The Secretary Of Commerce: Washington, D.C. 20230: August 23, 2010: Mr. David A. Powner: Director: Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Powner: Thank you for the opportunity to review and comment on the Government Accountability Office's draft report entitled, "Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges" (GA0-10-843). On behalf of the Department of Commerce, I have enclosed the National Oceanic and Atmospheric Administration's programmatic comments to the draft report. Sincerely, Signed by: Gary Locke: Enclosure: [End of letter] Department of Commerce: National Oceanic and Atmospheric Administration: Comments to the Draft GAO Report Entitled "Aviation Weather: Agencies Need to Improve Performance Measurement and Fully Address Key Challenges" GAO-10-843 (September 2010): General Comments: The Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) appreciates the opportunity to review the Government Accountability Office's (GAO) report on aviation weather. The report is generally representative of challenges facing NOAA's National Weather Service (NWS) and the Department of Transportation's Federal Aviation Administration (FAA) in the execution of the aviation weather services provided by Center Weather Services Units (CWSU). The report contains three new recommendations; however, the text makes reference to "four recommendations," the fourth being a reiteration or reference to prior GAO recommendations to both NOAA and FAA. Since NOAA has already responded to GAO's prior recommendations, NOAA's response below addresses the three current recommendations only. NOAA Response to GAO Recommendations: The draft GAO report states, "To improve the aviation weather products and services provided at FAA's en route centers, we are making four recommendations to the Secretaries of Commerce and Transportation. Specifically, we are recommending that the Secretaries direct the NWS and FAA Administrators to..." Recommendation 1: "define, document and sign the agencies' recent agreements on (1) the locations of the center weather service units, (2) immediate improvements in aviation weather services and operating hours, and (3) the development of an implementation plan for improvements through 2015;" NOAA Response: NOAA agrees with this recommendation. The Assistant Administrator for Weather Services responded to the FAA in a letter dated July 29, 2010, confirming the NWS' commitment to establish an NWS-FAA team to work collaboratively to determine weather requirements for Traffic Flow Management and implementation of products and services through 2015. The results from the NWS/FAA Team will serve as additional documentation of the agreement between NWS and FAA on the path forward in providing aviation weather services to Traffic Flow Management. Recommendation 2: "ensure that NWS regularly tracks progress, documents performance baselines, and reports on its forecast consistency, forecast accuracy, and training performance measures;" NOAA Response: NOAA agrees with this recommendation. We continue to track performance of our CWSUs and will work with the FAA to improve our measures and move them into the FAA's Aviation Weather Quality Management System. Our tracking for forecast consistency will be in place for CWSU products in September 2010, and will be provided to the FAA in our Quarterly Reports. Tracking of forecast accuracy will begin in December 2010. Recommendation 3: "ensure that NWS develops a reliable customer satisfaction baseline by refining the questions used during annual evaluations so that comparable information is collected from year to year and revising the scoring process to ensure that scores accurately reflect each center's performance." NOAA Response: NOAA agrees with this recommendation. After conducting the initial 2009 Site Review program, NOAA established a baseline for customer service based upon findings from the reviews. Specifically, the Quality Verification Rating (QVR) scores calculated from the 2009 site reviews will serve as the baseline measure of customer satisfaction, and will be used to compare QVR scores from the 2010 and subsequent site reviews. [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: David A. Powner, (202) 512-9286, or pownerd@gao.gov: Staff Acknowledgments: In addition to the individual named above, Colleen Phillips, Assistant Director; Neil Doherty; Rebecca Eyler; Joshua Leiling; and Jessica Waselkow made key contributions to this report. [End of section] Footnotes: [1] These centers include the National Centers for Environmental Prediction Central Operations, Aviation Weather Center, Environmental Modeling Center, Hydrometeorological Prediction Center, Ocean Prediction Center, Storm Prediction Center, Tropical Prediction Center/National Hurricane Center, Climate Prediction Center, and the Space Weather Prediction Center. [2] FAA is also involved in a longer-term initiative to increase the efficiency of the national airspace system and to improve its overall safety. This initiative, called the Next Generation Air Transportation System, is a joint effort of the National Aeronautics and Space Administration, the White House Office of Science and Technology Policy, and the Departments of Transportation, Homeland Security, Defense, and Commerce. FAA anticipates that this initiative may lead to major changes in the aviation weather program that would supersede any short-term changes to aviation services. [3] GAO, Aviation Weather: FAA Is Restructuring Services at Key Centers; Both FAA and the National Weather Service Need to Better Ensure Product Quality, [hyperlink, http://www.gao.gov/products/GAO-08-258] (Washington, D.C.: Jan. 11, 2008). [4] GAO, Aviation Weather: FAA and the National Weather Service are Considering Plans to Consolidate Weather Service Offices, but Face Significant Challenges, [hyperlink, http://www.gao.gov/products/GAO-09-761] (Washington, D.C.: Sept. 9, 2009). [5] NWS estimated that the cost of maintaining its 21 center weather service units will grow over the next 5 years from roughly $13.2 to $14.8 million. [6] The 10 facilities are terminal radar approach control sites in Northern California, Southern California, New York, Potomac, Chicago, Dallas/Fort Worth, Houston, Atlanta, Miami, and Denver. [7] Department of the Navy, Office of the Chief Information Officer, Guide for Developing and Using Information Technology (IT) Performance Measurements (Washington, D.C.: October 2001); and General Services Administration, Office of Governmentwide Policy, Performance-Based Management: Eight Steps To Develop and Use Information Technology Performance Measures Effectively (Washington, D.C.: 1996). [8] [hyperlink, http://www.gao.gov/products/GAO-08-258]. [9] [hyperlink, http://www.gao.gov/products/GAO-09-761]. [10] The other six performance measures cannot be measured now because they involve new products or a consolidated number of office locations. [11] GAO, Results-Oriented Government: Practices That Can Help Enhance and Sustain Collaboration among Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: Oct. 21, 2005). [12] [hyperlink, http://www.gao.gov/products/GAO-09-761]. [13] Carnegie Mellon University Software Engineering Institute, Capability Maturity Model® Integration for Development, Version 1.2 (Pittsburgh, Pa.: August 2006). Capability Maturity Model® and Capability Maturity Modeling are registered in the U.S. Patent and Trademark Office. CMM is a service mark of Carnegie Mellon University. [14] [hyperlink, http://www.gao.gov/products/GAO-09-761]. [15] The Joint Planning and Development Office has multiple federal partners, including FAA; the Departments of Transportation, Commerce, Defense, and Homeland Security; the National Aeronautics and Space Administration; and the White House Office of Science and Technology Policy. As part of NextGen, the Joint Planning and Development Office envisions restructuring air traffic facilities, including en route centers, across the country as well as transitioning to new technologies. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: