Aviation Security: Actions Needed to Systematically Evaluate Cost and Effectiveness Across Security Countermeasures
Highlights
What GAO Found
The Transportation Security Administration (TSA) has data on the effectiveness of some, but not all of its passenger aviation security countermeasures. Specifically, TSA has data on passenger prescreening, checkpoint and checked baggage screening, and explosives detection canines. Further, TSA is taking steps to improve the quality of this information. However, it does not have effectiveness data for its Behavior Detection and Analysis (BDA) program and the U.S. Federal Air Marshal Service (FAMS). For BDA—a program to identify potential threats by observing passengers for behaviors indicative of stress, fear, or deception—in July 2017, GAO reported that (1) TSA does not have valid evidence supporting most of its behavioral indicators, and (2) TSA should continue to limit future funding for its behavior detection activities until it can provide such evidence. For FAMS—a program that deploys armed law enforcement officers on certain flights at an annual cost of about $800 million for fiscal year 2015—officials reported that one of the primary security contributions is to deter attacks. However, TSA does not have information on its effectiveness in doing so, nor does it have data on the deterrent effect resulting from any of its other aviation security countermeasures. While officials stated that deterrence is difficult to measure, the Government Performance and Results Act of 1993, as updated, provides that agencies are to assess the effectiveness of their programs. Further, the Office of Management and Budget and GAO have suggested approaches for measuring deterrence. Developing such methods for TSA countermeasures, especially for an effort such as FAMS in which the primary goal is deterrence, would enable TSA to determine whether its substantial investment is yielding results.
TSA has a tool to compare the security effectiveness of some aviation security countermeasures, but has no efforts underway to systematically evaluate potential cost and effectiveness tradeoffs across all countermeasures. In 2014, the agency developed a tool to analyze the security effectiveness of alternate combinations of some countermeasures for the purpose of informing acquisition and deployment decisions, but does not have a tool to assess such tradeoffs across the entire system of countermeasures. TSA officials explained that the aviation security system is constantly evolving, and assessing a system in flux is challenging. However, DHS policy and TSA's strategic plan call for the systematic evaluation of costs and effectiveness of TSA's chosen mix of aviation security countermeasures. Without such an analysis, TSA is not well positioned to strike an appropriate balance of costs, effectiveness, and risk.
This is a public version of a classified report that GAO issued in August 2017. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.
Why GAO Did This Study
Since the attacks of September 11, 2001, TSA has spent billions of dollars on aviation security programs. However, recent attacks involving aircraft and airports in other countries underscore the continued threat to aviation and the need for an effective aviation security program.
GAO was asked to review TSA's passenger aviation security countermeasures. This report examines the extent to which TSA has (1) information on the effectiveness of selected passenger aviation security countermeasures and (2) systematically analyzed the cost and effectiveness tradeoffs among countermeasures.
GAO reviewed TSA documentation on the effectiveness of six passenger aviation security countermeasures in fiscal year 2015—the most recent year for which data were available. GAO selected these countermeasures because they involve direct interaction with passengers, their belongings, or their personal information, and are largely operated and funded by TSA. GAO also reviewed TSA documents and interviewed TSA officials regarding efforts to systematically analyze cost and effectiveness tradeoffs across countermeasures.
Recommendations
GAO recommends that TSA (1) explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures, with FAMS as a top priority to address, and (2) systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures. DHS concurred with these recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | The Administrator of TSA should explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures; such an effort should identify FAMS—a countermeasure with a focus on deterring threats—as a top priority to address. (Recommendation 1) |
In September 2017, we reported on the costs and effectiveness of Transportation Security Administration's (TSA) passenger aviation security countermeasures. We found that TSA had methods to measure its effectiveness in detecting and disrupting threats, but the agency has no such methods to measure its effectiveness in deterring attacks on the U.S. aviation system. For example, TSA officials explained that canine teams that patrol airports provide a deterrent presence, but officials did not have any data on these canines' deterrent effect. Further, with regard to air marshals, TSA officials explained that one of the primary security contributions and a key aspect of the air marshals' mission is to deter attacks. As a result, we recommended that TSA explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures. The Department of Homeland Security concurred and has since undertaken such an effort. Specifically, from 2018 through 2021, TSA Risk and Capabilities Analysis officials studied threat shifting--the response of an adversary due to their perception of countermeasures or procedures--as a means to understand and influence deterrence. These studies focused on airport checkpoint screening countermeasures, explosives trace detection for checked baggage, and federal air marshals. According to TSA documentation, the purpose of this effort was to understand how TSA could provoke advantageous threat shifting by changing countermeasure characteristics such as visibility, frequency, credibility, avoidability, and predictability. Further, in 2021, the department's Science and Technology Directorate studied whether the deterrent effect of passenger screening canine teams could be quantitatively and qualitatively measured. TSA officials explained that they have used the results of these studies to inform TSA leadership decisions about new or revised countermeasures. For example, TSA officials explained that they considered their findings on threat shifting when deliberating the possibility and potential parameters of a screening requirement for aviation workers. By pursuing these methods, TSA is better positioned to understand and improve its efforts to deter attacks on the U.S. aviation system. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should systematically evaluate the potential cost and effectiveness tradeoffs across countermeasures, as TSA improves the reliability and extent of its information on the effectiveness of aviation security countermeasures. (Recommendation 2) |
In September 2017, we reported on the costs and effectiveness of Transportation Security Administration's (TSA) passenger aviation security countermeasures-passenger prescreening, checkpoint screening, explosives detection canines, and federal air marshals, among others. During the course of our review, we found that TSA did not have efforts underway to systematically evaluate potential cost and effectiveness tradeoffs across these countermeasures. Such analysis would better position TSA to achieve its stated goal of optimizing resource allocation and striking an appropriate balance of costs, effectiveness, and risk. As a result, we recommended that TSA systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures. The Department of Homeland Security concurred and TSA officials reported in June 2020 that they have since expanded their use of an analytical process called the Transportation Security Capability Analysis Project. TSA officials explain that in November 2017, they expanded this process beyond its initial focus on technology to include 34 mission essential capability needs. In November 2017, TSA also added systematic consideration of the cost and effectiveness of options to mitigate TSA capability gaps. TSA officials report that this process supports better decision making by weighing the value of recommended courses of action taking into consideration cost and effectiveness information. As a result, this recommendation is closed as implemented.
|