TSA Should Limit Future Funding for Behavior Detection Activities
GAO-14-159: Published: Nov 8, 2013. Publicly Released: Nov 13, 2013.
What GAO Found
Available evidence does not support whether behavioral indicators, which are used in the Transportation Security Administration's (TSA) Screening of Passengers by Observation Techniques (SPOT) program, can be used to identify persons who may pose a risk to aviation security. GAO reviewed four meta-analyses (reviews that analyze other studies and synthesize their findings) that included over 400 studies from the past 60 years and found that the human ability to accurately identify deceptive behavior based on behavioral indicators is the same as or slightly better than chance. Further, the Department of Homeland Security's (DHS) April 2011 study conducted to validate SPOT's behavioral indicators did not demonstrate their effectiveness because of study limitations, including the use of unreliable data. Twenty-one of the 25 behavior detection officers (BDO) GAO interviewed at four airports said that some behavioral indicators are subjective. TSA officials agree, and said they are working to better define them. GAO analyzed data from fiscal years 2011 and 2012 on the rates at which BDOs referred passengers for additional screening based on behavioral indicators and found that BDOs' referral rates varied significantly across airports, raising questions about the use of behavioral indicators by BDOs. To help ensure consistency, TSA officials said they deployed teams nationally to verify compliance with SPOT procedures in August 2013. However, these teams are not designed to help ensure BDOs consistently interpret SPOT indicators.
TSA has limited information to evaluate SPOT's effectiveness, but plans to collect additional performance data. The April 2011 study found that SPOT was more likely to correctly identify outcomes representing a high-risk passenger--such as possession of a fraudulent document--than through a random selection process. However, the study results are inconclusive because of limitations in the design and data collection and cannot be used to demonstrate the effectiveness of SPOT. For example, TSA collected the study data unevenly. In December 2009, TSA began collecting data from 24 airports, added 1 airport after 3 months, and an additional 18 airports more than 7 months later when it determined that the airports were not collecting enough data to reach the study's required sample size. Since aviation activity and passenger demographics are not constant throughout the year, this uneven data collection may have conflated the effect of random versus SPOT selection methods. Further, BDOs knew if passengers they screened were selected using the random selection protocol or SPOT procedures, a fact that may have introduced bias into the study. TSA completed a performance metrics plan in November 2012 that details the performance measures required for TSA to determine whether its behavior detection activities are effective, as GAO recommended in May 2010. However, the plan notes that it will be 3 years before TSA can begin to report on the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective. This is a public version of a sensitive report that GAO issued in November 2013. Information that TSA deemed sensitive has been redacted.
Why GAO Did This Study
TSA began deploying the SPOT program in fiscal year 2007--and has since spent about $900 million--to identify persons who may pose a risk to aviation security through the observation of behavioral indicators. In May 2010, GAO concluded, among other things, that TSA deployed SPOT without validating its scientific basis and SPOT lacked performance measures. GAO was asked to update its assessment. This report addresses the extent to which (1) available evidence supports the use of behavioral indicators to identify aviation security threats and (2) TSA has the data necessary to assess the SPOT program's effectiveness. GAO analyzed fiscal year 2011 and 2012 SPOT program data. GAO visited four SPOT airports, chosen on the basis of size, among other things, and interviewed TSA officials and a nonprobability sample of 25 randomly selected BDOs. These results are not generalizable, but provided insights.
What GAO Recommends
For more information, contact Stephen M. Lord at (202) 512-4379 or firstname.lastname@example.org.
Matter for Congressional Consideration
Comments: The Homeland Security Appropriations Act, 2015, enacted in March 2015, imposes a funding restriction on TSA based in part on the findings in GAO's November 2013 report. Specifically, the Act provides that $25 million of TSA's appropriation shall be withheld from obligation for "Headquarters Administration" until TSA submits to the Appropriations Committees of the Senate and House of Representatives a report providing evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security and the plans that will be put into place to collect additional performance data. A bill that had been pending in the House of Representatives in the 113th Congress to fund the Department of Homeland Security for fiscal year 2015 included a similar report requirement and funding restriction. Although the bill was not passed before the 113th Congress concluded, its accompanying committee report explained the funding would be withheld to help ensure that security-related funding is directed to programs that have demonstrated their effectiveness. The report further stated that questions remain over the value of TSA's Behavior Detection Officer program, which has not been sufficiently validated and for which few measures have been developed to prove its intrinsic value to the aviation security environment. In the absence of scientifically validated evidence for using behavioral indicators to identify aviation security threats, there is no assurance that security-related funding is directed to programs that have demonstrated their effectiveness.
Matter: To help ensure that security-related funding is directed to programs that have demonstrated their effectiveness, Congress should consider the findings in this report regarding the absence of scientifically validated evidence for using behavioral indicators to identify aviation security threats when assessing the potential benefits of behavior detection activities relative to their cost when making future funding decisions related to aviation security.
Recommendation for Executive Action
Comments: No executive action taken. The Department of Homeland Security (DHS) did not concur with GAO's November 2013 recommendation to the TSA Administrator to limit future funding support for the agency's behavior detection activities until TSA can provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security. DHS identified two main areas where it disagreed with information presented in GAO's report: (1) the findings related to the Screening of Passengers by Observation Techniques (SPOT) validation study and (2) the findings related to the research literature. For example, DHS stated that research cited in the GAO report did not relate to the use of behavior detection in an airport security environment. However, as described in GAO's November 2013 report, GAO reviewed several documents on behavior detection research and suicide bomber indicators that DHS's Science and Technology Directorate (S&T) and TSA officials provided to GAO, including an unclassified and a classified literature review that S&T had commissioned. After GAO reviewed the research, the report conclusions, which were confirmed in discussions with subject matter experts and an independent review of studies, indicated that scientifically validated evidence did not support whether the use of behavioral indicators by unaided human observers could be used to identify passengers who may pose a threat to aviation security. In January 2015, TSA provided documentation describing its plans to enhance its behavioral-based screening program, including its efforts to compile articles and research involving suicide bombers that are to be a basis to revise the behavioral detection indicators to be used in new protocols. In February 2015, TSA officials told us they have revised the behavioral indicators and are in the process of pilot testing the use of these new protocols in the airport environment. In August 2015, TSA officials estimate that they will complete pilot testing at 3 to 5 airports by late 2015. At that time, TSA plans to make a determination about whether the new protocols are ready for further testing, including an operational test in up to 10 airports to determine the protocols' effectiveness. The operational test, which has yet to be developed, has an estimated completion date in the latter half of 2016. However, TSA officials stated that the operational test cannot begin until a behavior detection covert test methodology is developed. TSA officials stated that it also plans to conduct a study on the use of the new protocols at 50 airports to examine disparity questions regarding racial, ethnicity, and religious garb demographics. According to these officials, this study will begin at airports as the new protocols are implemented and will require 12 to 15 months of data collection. Further, this study is not expected to be completed until 2018. GAO plans to monitor the progress of TSA's efforts to improve its behavioral-based screening program, as well as results of the operational test and study of its new protocols that are intended to demonstrate the effective operational use of behavior detection in an airport security environment. As GAO previously reported in 2010 and 2013, TSA's efforts over the last 10 years to validate SPOT have lacked rigor and were unsuccessful in validating the program. Although TSA has revised the behavioral indicators as a result of its research and is pilot testing new protocols, based on its current schedule, it will not have the information to determine whether its changes to the behavioral-screening program are effective for another 3 years. In the absence of such evidence, the agency continues to fund activities that have not been determined to be effective.
Recommendation: To help ensure that security-related funding is directed to programs that have demonstrated their effectiveness, the Secretary of Homeland Security should direct the TSA Administrator to limit future funding support for the agency's behavior detection activities until TSA can provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security.
Agency Affected: Department of Homeland Security