Aviation Forecasting: FAA Should Implement Additional Risk-Management Practices in Forecasting Aviation Activity
Highlights
What GAO Found
Both of the Federal Aviation Administration's (FAA) annual activity forecasts—the National Aerospace Forecast (Aerospace) and airport-level Terminal Area Forecast (TAF)—have consistently overestimated aviation activity since 2004 and 2010, respectively, and have been less accurate the further out they forecast. For example, for Aerospace passenger enplanement forecasts made between fiscal year 2004 and 2014, the mean percentage error was less than 1 percent for 1-year-ahead forecasts, 15 percent for 5-year-ahead forecasts, and 31 percent for 10-year-ahead forecasts (see table). An important factor affecting forecast accuracy was the inaccuracy of the inputs used in the TAF and Aerospace models—such as gross domestic product and fuel prices—resulting from events such as the 2007–2009 recession and fuel price spikes.
Accuracy of the Federal Aviation Administration's (FAA) 1-Year, 5-Year, and 10-Year National Aerospace Forecasts Made between Fiscal Years 2004 and 2014
|
Mean percentage error |
|||
|
Aviation activity metric |
1-Year forecasta |
5-Year forecastb |
10-Year forecastc |
|
Passenger enplanements (number of passengers boarding planes) |
-0.2% |
14.7% |
31.3% |
|
Revenue passenger miles |
-0.6% |
14.0% |
26.9% |
|
Total operationsd |
1.9% |
25.5% |
54.7% |
|
Available seat miles |
0.1% |
18.6% |
40.5% |
Source: GAO analysis of FAA data. | GAO-16-210
aincludes 11 1-year Aerospace forecasts made from fiscal year 2004 through 2014.
bincludes 7 5-year forecasts made from fiscal year 2004 through 2010.
cincludes 2 10-year forecasts made from fiscal year 2004 through 2005.
dincludes air carrier, air taxi, general aviation, and military flights arriving at and departing from airports
In developing forecasts, FAA has implemented most of the practices associated with five key risk-management principles GAO selected as applicable to forecasting: (1) setting measurable goals; (2) using best available data; (3) identifying, analyzing, and documenting risk; (4) adopting strategies to respond to risks; and (5) monitoring and reviewing performance. However, FAA lacks some risk-management practices that could enhance FAA's ability to manage for risk and uncertainty. For example, FAA monitors forecast performance by reevaluating the Aerospace and TAF forecast models, but has not fully identified, analyzed, and documented risk. While FAA has identified risks in the Aerospace forecast, it has not reported on the likelihood of these risks. For the TAF, FAA has not systematically assessed the TAF's uncertainty, partly due to recent changes in forecasting methodology. Nor has FAA established error thresholds that would trigger more thorough review of the forecast methodology. Given FAA's reliance on forecasts for decision-making, managing and understanding the nature of uncertainty is important to good decision-making. While FAA has provided limited information about the Aerospace and TAF models, it has not documented the models and assumptions in a manner that would allow stakeholders outside FAA to understand how forecasts are developed or enable FAA to retain organizational knowledge.
Why GAO Did This Study
The FAA annually prepares forecasts of future aviation activity and uses these forecasts to help manage most of its $15 billion in annual spending. While forecasting is inherently uncertain, managing that uncertainty is essential to informed decisions.
GAO was asked to examine the accuracy of and FAA's use of two annual forecasts of aviation activity. This report discusses the accuracy of FAA's forecasts from 2004 through 2014 and strengths and limitations of FAA's consideration of risks in developing its forecasts. This report focuses on the use of the Aerospace and TAF forecasts to inform key operational and investment decisions. GAO compared these two forecasts to actual activity from 2004 through 2014 for the Aerospace forecasts and from 2010 through 2014 for the TAF forecasts and identified factors affecting that accuracy. GAO compared FAA's treatment of risk in developing forecasts to selected risk- management practices recommended by the Office of Management and Budget, GAO, and others.
Recommendations
GAO recommends that the Secretary of the Department of Transportation (DOT) require the FAA to: (1) report on uncertainty and set error response thresholds for both forecasts and (2) document FAA's methods and assumptions for the forecasting models. The DOT partially concurs with the first recommendation and fully concurs with the second. DOT agrees to report on uncertainty but not to set thresholds. GAO believes that thresholds ensure systematic review of forecast accuracy.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Transportation | To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to apply risk-management practices to analyze and report on uncertainty. Specifically, the FAA should, for both the Aerospace and TAF forecasts, analyze and report the forecast's uncertainty, establish forecast error thresholds, and develop an approach that will prompt forecast review when error thresholds are exceeded, and, for TAF forecasts, monitor and publish multi-year historical error performance, as FAA does for the Aerospace Forecast. | The Federal Aviation Administration (FAA) annually prepares forecasts of future aviation activity, including projections of the number of flights and passengers, 20 of more years into the future. These forecasts are used by FAA to develop major areas of its operational plans and manage its budget and by external stakeholders, such as airlines and airports, for future planning and investment. The two main forecasts that FAA develops are the National Aerospace Forecast (Aerospace), which forecasts aviation activity at the national level, and the Terminal Area Forecast (TAF), which forecasts aviation activity at the individual airport level. While forecasting is inherently uncertain,... managing that uncertainty is essential to making informed decisions. In 2016, GAO reported that it compared FAA's treatment of risk in creating aviation forecasts to select risk-management practices recommended by the Office of Management and Budget, GAO, and others, and found that FAA lacks some risk management practices that could enhance FAA's ability to manage for risk and uncertainty. FAA followed or partially followed most of the selected risk-management principles and practices in developing its aviation activity forecasts. However, FAA did not analyze the forecast uncertainty or establish forecast error thresholds for either the Aerospace or the TAF forecasts. Further, FAA did not monitor and publish multi-year historical error performance for the TAF forecast. By not taking these steps, FAA was limited in its ability to manage forecast uncertainty and users were limited in their understanding of the source and nature of forecast uncertainty. Therefore, GAO recommended that FAA should report on uncertainty and set error response thresholds for both forecasts. In 2020, FAA officials told us that they established error forecast thresholds for the Aerospace and TAF forecasts. In addition, FAA published information about forecast uncertainty in the Aerospace and TAF forecasts and published multi-year historical error performance for the TAF. These actions should improve FAA's ability to manage forecast uncertainty and increase users' understanding of the sources and nature of forecast uncertainty, which essentially meets the intent of GAO's recommendation.
View More |
| Department of Transportation | To help FAA better manage and understand the uncertainties of its forecasts, the Secretary of Transportation should direct the FAA to fully document its methods and assumptions in developing the Aerospace and TAF forecasting models to provide greater transparency to internal users and external stakeholders. | The Federal Aviation Administration (FAA) annually prepares forecasts of future aviation activity, including projections of the number of flights and passengers, 20 of more years into the future. These forecasts are used by FAA to develop major areas of its operational plans and manage its budget and by external stakeholders, such as airlines and airports, for future planning and investment. The two main forecasts that FAA develops are the National Aerospace Forecast (Aerospace), which forecasts aviation activity at the national level, and the Terminal Area Forecast (TAF), which forecasts aviation activity at the individual airport level. While forecasting is inherently uncertain,...
|