NextGen Air Transportation System:
FAA's Metrics Can Be Used to Report on Status of Individual Programs, but Not of Overall NextGen Implementation or Outcomes
GAO-10-629, Jul 27, 2010
To prepare for forecasted air traffic growth, the Federal Aviation Administration (FAA), in partnership with other federal agencies and the aviation industry, is planning and implementing the Next Generation Air Transportation System (NextGen), a new satellite-based air traffic management system that will replace the current radar-based system and is expected to enhance the safety and capacity of the air transport system. GAO was asked to review FAA's metrics for (1) tracking the status of NextGen programs and the implementation of NextGen capabilities, the reliability of those metrics, and any limitations or gaps and (2) measuring the performance and outcomes of NextGen capabilities that are implemented and any limitations. GAO analyzed FAA program progress reports and associated metrics for monitoring. GAO also reviewed agency performance and accountability reports and discussed internal performance reporting methods with FAA officials.
FAA has metrics that allow it to monitor the progress of its programs for acquiring software and hardware. These metrics include Earned Value Management (EVM) measurements that show how well a program is meeting its planned cost and schedule targets for system development. Previous GAO reports have identified issues with FAA's implementation of EVM, which continue to affect the accuracy and reliability of some of FAA's program status reports. For example, for one acquisition program, FAA implemented EVM metrics only for the contractor's performance and not for the government's. As a result, the EVM data did not pick up delays that occurred after the contractor delivered the system and the EVM system did not provide early warnings of delays and potential cost overruns. In addition, GAO's previous work has shown that FAA is not able to report on how slippage in one program's schedule or budget will ultimately affect the implementation of other NextGen acquisition programs or operational capabilities whose progress depends on the completion of the first program. GAO has made recommendations to address these issues, which FAA and the Department of Transportation have begun to implement. FAA has also designated specific positions within the NextGen Integration and Implementation Office-known as solution set coordinators-to monitor and track progress toward implementing a portfolio of operational improvements into the national airspace system. However, the role of the coordinators and the process for resolving any disputes across FAA lines of business have not been clearly defined or delineated and it is uncertain whether the processes in place in this portfolio management structure will strengthen oversight and create a greater likelihood that required activities are completed on time. FAA has broad goals for NextGen as a whole, such as increasing capacity and reducing noise and emissions, but has not yet developed specific goals and outcome-based performance metrics to track the impact of and benefits realized from the entire NextGen endeavor. The agency has multiple efforts underway to develop such metrics: FAA's Air Traffic Organization (ATO), which manages the air traffic control system, has started to compile and review a set of metrics for measuring outcomes and performance associated with NextGen improvements. These metrics are likely to measure such things as the extent to which improvements increase throughput at airports, reduce emissions, and reduce flight times, but they are in the early stages of development. Recently, FAA also committed to developing performance metrics with industry, but it has no timeline or action plan for completing this effort. Separately, the Joint Planning and Development Office (JPDO), which is responsible for the long-term planning for NextGen and partnering with other federal agencies, has been working to develop a list of potential metrics, which range from fuel consumed per distance flown to curb-to-curb travel time. Without specific goals and metrics for the performance of NextGen as a whole, together with a timeline and action plan for implementation, it is not clear whether NextGen technologies, systems, and capabilities will achieve desired outcomes and be completed within the planned time frames. The FAA Administrator should clarify dispute resolution processes within FAA's portfolio management structure, and develop a timeline and action plan to agree with stakeholders on a list of specific goals and outcome-based performance metrics for NextGen. DOT agreed to consider GAO's recommendations and provided technical comments that GAO incorporated as appropriate.
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To ensure that FAA can effectively manage NextGen solution sets, the Secretary of Transportation should direct the FAA Administrator to develop written policies and procedures for dispute resolution across different FAA lines of business and outline the appropriate roles of the solution set managers, program managers, and the NextGen Management Board in managing these portfolios of improvements.
Agency Affected: Department of Transportation
Status: Closed - Implemented
Comments: In July 2010, we found that the Federal Aviation Administration (FAA) had instituted a portfolio approach to implementing the Next Generation Air Transportation System (NextGen) designed to help the agency assess and convey the implementation status of interrelated capabilities and operational improvements, but that this approach lacked written policies and procedures to guide resolution of any disputes between FAA lines of business and that roles and responsibilities for oversight and accountability were unclear. Because implementation of NextGen capabilities requires action across various lines of business with separate budgets within FAA, the potential exists for disputes and issues to arise that may hamper timely implementation. A lack of clear dispute resolution procedures and clear roles related to oversight and accountability raise questions about how quickly and effectively any such disputes or issues that arise will be resolved, and who is responsible and accountable for implementation. We therefore recommended that FAA develop written policies and procedures for dispute resolution across different FAA lines of business and outline the appropriate roles of portfolio managers, program managers, and the NextGen Management Board, which is responsible for overseeing NextGen implementation overall, in managing these portfolios of improvements. In response, in 2011, FAA initiated quarterly portfolio management reviews which include members from all FAA lines of business affected by the NextGen capabilities being implemented. As a written policy and procedure, any disputes or issues across these lines of business are to be resolved at these quarterly meetings, and if not resolved, are elevated to the NextGen Management Board - which includes the leadership of all FAA lines of business and has the authority to force timely resolution of emerging NextGen issues. In addition, in April 2013, an agencywide framework called "Ideas-to-In-Service" (i2i) was initiated and key components were incorporated into the agency's acquisition management system. The i2i framework defines the collaboration, structure, and coordination required of all FAA lines of business and staff offices to insure the successful implementation of NextGen. The i2i framework also clearly defines the role of portfolio managers, who are responsible for tracking performance, schedules, milestones and resources for their portfolio of NextGen capabilities, and are responsible for conducting the quarterly portfolio management reviews, in relation to the NextGen Management Board and individual project managers. These changes will improve oversight and accountability for implementation of NextGen capabilities, resulting in greater assurance that improvements to the air transportation system will be made and the benefits NextGen promises will be realized.
Recommendation: To ensure that the outcomes and performance expected from NextGen improvements are understood and can be monitored, the Secretary of Transportation should direct the FAA Administrator to develop a timeline and action plan to work with industry and federal partner agencies to develop an agreed-upon list of outcome-based performance metrics and goals for NextGen broadly and for specific NextGen portfolios, programs, and capabilities. The Administrator should then share this list with the appropriate congressional oversight committees. Furthermore, the Administrator should establish a clear timeline to align NextGen performance metrics with FAA's agencywide goals and performance plans.
Agency Affected: Department of Transportation
Status: Closed - Implemented
Comments: In July 2010, we found that, although the Federal Aviation Administration (FAA) has broad goals for the Next Generation Air Transportation System (NextGen), such as enhancing safety, reducing aviation's environmental impact, and increasing operations and efficiency, specific goals for NextGen as a whole have yet to be determined and FAA has not agreed on a set of overall performance metrics that it can use to measure progress. In addition, FAA's performance reports to the Congress and the public discuss only a few NextGen capabilities and programs that are expected to have an effect on existing agencywide metrics and do not include any performance information specific to ongoing NextGen capabilities that are being implemented. We therefore recommended that the FAA Administrator develop a timeline and action plan to work with NextGen stakeholders to develop an agreed-upon list of outcome-based performance metrics and goals for NextGen broadly and for specific NextGen portfolios, programs, and capabilities. Further, we recommended that NextGen performance metrics be aligned with FAA's agencywide goals and performance plans. In response, FAA set up an agency wide NextGen Performance Metrics Working Group, conducted interviews with all of the FAA representatives and developed a set of performance metrics that were used as a starting point with Industry collaboration. FAA has aligned these metrics under broad NextGen Goals that are now aligned directly with agencywide goals identified in FAA's most recent strategic plan. For example, one specific metric is to "improve throughput (at core airports) during adverse weather by 14 percent by 2018" which aligns with the NextGen goal to "increase efficient performance," which will lead to progress on FAA's strategic goal to "deliver access through innovation." In order to connect NextGen goals, actions, and results, FAA also developed the NextGen Dashboard which evolved into the NextGen Performance Snapshots (NPS). The NPS website was launched in March 2012. The purpose of the NPS is to ensure transparency and alignment of the FAA's goals by comparing key NextGen initiatives with performance outcomes at locations where NextGen technologies have been deployed. According to the FAA, the agency plans to continue to update the NPS with additional metrics as they mature. These actions will help ensure that stakeholders, interested parties, Congress, and the American people will have a clear picture of where implementation stands at any given time, and whether the technologies, capabilities, and operational improvements that are being implemented are resulting in positive outcomes and improved performance for operators and passengers.