Federal Air Marshal Service: Actions Needed to Better Incorporate Risk in Deployment Strategy
Highlights
What GAO Found
The U.S. Federal Air Marshal Service (FAMS) deploys air marshals in part based on assessed risk, but could better incorporate risk in its deployment strategy. FAMS may deploy air marshals on flights with known risk—meaning certain higher risk flights where the Transportation Security Administration (TSA) or FAMS knows of an increase in the threat or consequence of a terrorist attack—as well as on other international and domestic flights of U.S. air carriers. However, GAO identified three ways FAMS could better incorporate risk into its deployment decisions, in accordance with Department of Homeland Security (DHS) and TSA risk management policy and guidance.
- FAMS considers its travel budget and number of personnel, but not risk when initially dividing its annual resources between domestic and international flights. With this approach, FAMS attempts to maximize the total number of flights it can cover, but because this model does not account for risk, FAMS cannot ensure it is devoting its resources to the highest risk flights overall.
- FAMS officials report that when FAMS revised its domestic deployment strategy in 2014, their choice of geographic focus areas and resource allocation levels were based on subject matter experts' professional judgment, not a risk assessment. FAMS officials stated this approach was appropriate because they were updating the strategy, not conducting a study. While providing perceptions of risk, the experts' input was not systematically collected and assigned risk values. Doing so would better position FAMS to ensure its resources are targeted using a risk-based approach.
- FAMS does not document rationales for its international deployment decisions because it has not identified a need to do so. Without documentation of the basis for these decisions, neither FAMS nor an external party can effectively oversee these decisions.
The performance measure FAMS has used to assess the extent to which it has met its flight coverage and resource use targets—the TSA coverage score—reflects some but not all attributes of successful performance measures. The TSA coverage score is a composite score for 11 performance categories, including air marshal coverage of certain domestic and international flights. The measure lacks clarity because the measure's name and definition are not consistent with the methodology used to calculate it. For example, the measure aggregates flight coverage and resource allocation information, which reduces the clarity of the score and makes it difficult to interpret. It also lacks objectivity because, as a composite measure, it does not show performance below or above desired levels in the 11 categories—information that would aid decision making. FAMS officials stated that they did not report scores for all categories to TSA or FAMS leadership because they were not asked to do so. Without clear and objective performance information, DHS, TSA, and FAMS leadership may be making decisions based on an inaccurate impression of FAMS's performance. This is a public version of a classified report GAO issued in February 2016. Information that DHS deemed classified or sensitive has been omitted.
Why GAO Did This Study
Following the September 11, 2001, hijacking of four U.S. airliners, individuals with terrorist ties have attempted attacks against the nation's civil aviation system. To help address such threats, FAMS, an office within TSA, is tasked with promoting confidence in the nation's civil aviation system through the deployment of air marshals to protect U.S. air carriers, airports, passengers, and crews. GAO was asked to review FAMS operations. This report examines the extent to which (1) FAMS deploys air marshals based on risk and (2) FAMS's performance measure reflects attributes of successful performance measures. GAO reviewed FAMS's strategy and performance measure documents, analyzed FAMS's fiscal years 2010 through 2014 flight coverage and performance measure data, and interviewed FAMS and TSA officials.
Recommendations
GAO recommends that FAMS (1) further incorporate risk into FAMS's method for dividing resources between international and domestic flights, (2) conduct a risk assessment to support certain domestic deployment decisions, (3) document the rationale for FAMS's selection of international deployment destinations, (4) adopt a consistent name and definition for the TSA coverage score, and (5) report performance results for all categories that comprise the score. DHS concurred with all of the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Office of Law Enforcement - Federal Air Marshal Service | To better ensure that FAMS uses its resources to cover the highest-risk flights, in addition to considering risk when determining how to divide FAMS's international flight coverage resources among international destinations, the Director of FAMS should incorporate risk into FAMS's method for initially setting its annual target numbers of average daily international and domestic flights to cover. |
In May 2016, we found that FAMS officials considered risk when selecting specific domestic and international flights to cover, but they did not consider risk when deciding how to initially divide their annual resources between domestic and international flights. Rather, each year FAMS considered two variables--travel budget and number of air marshals--to identify the most efficient way to divide the agency's resources between domestic and international flights. As a result, we recommended that FAMS incorporate risk into FAMS's method for initially setting its annual target numbers of average daily international and domestic flights to cover. In March 2018, FAMS revised its deployment methodology to no longer set an annual target number of average daily international and domestic flights to cover. Rather, FAMS prioritized deploying air marshals on as many flights as possible with passengers who have been identified as potentially higher risk because they match TSA's intelligence-based screening rules, among other risk-based priorities. In November 2020, FAMS officials explained that they were again evaluating their concept of operations and planned to make further adjustments to their deployment strategy over the coming year. Given the passage of time since GAO made the recommendation in 2016, and the uncertainty about when and how FAMS will address it, GAO is closing this recommendation as not implemented.
|
| Office of Law Enforcement - Federal Air Marshal Service | To better ensure that FAMS uses its resources to cover the highest-risk flights, the Director of FAMS should conduct and document a risk assessment--systematically collecting information on and assigning value to current risks--to further support FAMS's domestic resource allocation decisions, including the identification of high-priority geographic areas. |
In May 2016, we reported that the Federal Air Marshal Service's (FAMS) choice of domestic geographic focus areas and resource allocation levels were based on professional judgment, not risk assessment. With regard to the geographic focus areas, for example, FAMS officials explained that they did not conduct a risk assessment to inform this decision, but rather selected these areas in consultation with 30 subject matter experts from various offices within the Transportation Security Administration based on their intuitive, qualitative perceptions of threats, vulnerabilities, potential impacts, history, and the demographics of the areas. Without fully incorporating risk when determining such priorities, FAMS could not reasonably ensure it is targeting its resources to the highest-risk flights. As a result, we recommended that the Director of FAMS conduct and document a risk assessment to further support FAMS's domestic resource allocation decisions, including the identification of high-priority geographic areas. In March 2018, FAMS revised its deployment strategy to no longer select domestic flights based on geographic focus areas. Rather, FAMS now prioritizes deploying air marshals on as many flights as possible with passengers who have been identified as potentially higher risk because they match the Transportation Security Administration's intelligence-based screening rules, among other risk-based priorities. With its updated deployment strategy, FAMS is better able to demonstrate to internal and external audiences the rationale for why the agency deploys air marshals on certain domestic flights and not others. As a result, this recommendation is closed as implemented.
|
| Office of Law Enforcement - Federal Air Marshal Service | To better ensure that FAMS uses its resources to cover the highest-risk flights, the Director of FAMS should document the rationale for FAMS's selection of international destinations for air marshal deployment and the proportion of flights to cover at each destination. |
In May 2016, we reported that the U.S. Federal Air Marshal Service (FAMS) did not document rationales for its international deployment decisions. FAMS officials knowledgeable about FAMS's deployment strategy reported that they selected international destinations to cover and the proportion of flights to cover in each destination based upon intelligence and vulnerability information. These officials reported that they documented the results of these decisions--the number and percentage of U.S.-flagged air carrier flights air marshals will cover by country and specific airport--in a planning roster, but they did not document the rationales for these decisions because they had not identified a need to do so. DHS's Integrated Risk Management Framework establishes transparency and documentation as important characteristics of homeland security risk management. The framework also states that documentation enables critical analyses of the approach. Consequently, we recommended that the Director of FAMS document the rationale for FAMS's selection of international destinations for air marshal deployment and the proportion of flights to cover at each destination. FAMS officials report that in response to GAO's report, in the summer of 2016, FAMS began documenting on a monthly and quarterly basis the rationales for its selection of international destinations to cover and the proportion of flights to cover in each destination. A senior FAMS official reported that this tool has been helpful as it permits FAMS officials to analyze deployment changes over time. Further, documentation of the basis for these decisions also permits external parties with authorized oversight responsibilities to determine the extent to which the decisions are intelligence-driven and risk-based, and management's directives are being carried out. As a result, this recommendation is closed as implemented.
|
| Office of Law Enforcement - Federal Air Marshal Service | To improve the usefulness of the performance information DHS, TSA, and FAMS leadership use to oversee FAMS performance in achieving its mission priorities, the Director of FAMS should adopt a consistent name and definition for the performance measure referred to as the TSA coverage score that accurately reflects its calculation method and composite nature. |
In May 2016, we reported that the performance measure FAMS has used to assess the extent to which it has met its flight coverage and resource use targets--previously referred to as the TSA coverage score--reflected some but not all attributes of successful performance measures. The TSA coverage score was a composite score for 11 performance categories, including air marshal coverage of certain domestic and international flights, but the measure lacked clarity because the measure's name and definition were not consistent with the methodology used to calculate it. Further, we found that FAMS and TSA used different names for the score--FAMS referred to it internally as the TSA coverage score and TSA referred to it more formally as "the percentage of risk-based flight coverage by the FAMS." As a result, we recommended that FAMS adopt a consistent name and definition for this performance measure that accurately reflects its calculation method and composite nature. In March 2017, TSA officials reported that FAMS had changed the name of the measure to "Composite Index of Federal Air Marshal Service Risk-Based Flight Coverage Goals" and provided documentation showing that FAMS and TSA now referred to this measure with the same name. These changes improve the clarity of the measure to users, including DHS and TSA leadership, who rely on the measure to help them oversee and assess FAMS's performance. As a result, this recommendation is closed as implemented.
|
| Office of Law Enforcement - Federal Air Marshal Service | To improve the usefulness of the performance information DHS, TSA, and FAMS leadership use to oversee FAMS performance in achieving its mission priorities, the Director of FAMS should report the performance results for each of the subcategories that comprise the TSA coverage score to FAMS and TSA leadership. |
In May 2016, we reported that TSA and FAMS leadership did not have visibility into FAMS's performance in most of the 11 categories that comprise the TSA coverage score and TSA officials who reviewed FAMS's performance reported that this would be useful. In March 2017, TSA officials reported that FAMS now reports the performance results for each of the 11 subcategories of the TSA coverage score--now referred to as the Composite Index of Federal Air Marshal Service Risk-Based Flight Coverage Goals--to both FAMS and TSA leadership on a regular basis. Specifically, FAMS officials provided documentation showing that they now include results for the 11 categories in monthly reports to the Director of FAMS and in quarterly reports to the TSA Office of the Chief Financial Officer. With more complete information on FAMS performance, DHS, TSA, and FAMS leadership are now better positioned to make decisions based on a more accurate impression of FAMS's performance and identify possible corrective action where FAMS's performance is below targets. As a result, this recommendation is closed as implemented.
|