Aviation Security: Improved Testing, Evaluation, and Performance Measurement Could Enhance Effectiveness

What GAO Found

Evaluation of new technologies, programs, and processes . GAO has found that TSA has not consistently evaluated the overall effectiveness of new technologies before adopting them. For example, in March 2014, GAO found that TSA testing of certain Advanced Imaging Technology (AIT) systems—also referred to as full-body scanners—used to screen passengers at airports did not account for all factors affecting the systems. GAO reported that the effectiveness of AIT systems equipped with automated target recognition software (AIT-ATR)—which display anomalies on a generic passenger outline—relied on both the technology's capability to identify potential threat items and its operators' ability to resolve them. However, GAO found that TSA did not include operators' ability in determining overall AIT-ATR system performance. GAO recommended that TSA, in considering procurement of the next generation of AIT systems (AIT-2), measure system effectiveness based on the performance of both the technology and the screening personnel. The Department of Homeland Security (DHS) concurred and, in June 2015, TSA provided documentation showing that, while conducting operational testing of the AIT-2 system, the agency considered screening officer performance and measured AIT-2 system effectiveness based on both the performance of the AIT-2 technology and the screening officers who operate it. This should help TSA assess whether this screening system will meet mission needs and perform as intended.

Establishment of performance measures. GAO has found that TSA has not consistently established performance measures that fully reflect program goals. For example, in September 2014, GAO found that TSA's performance measures for Secure Flight—a passenger prescreening program—did not allow TSA to fully assess its progress toward achieving all of its goals. For example, one program goal was to accurately identify passengers on various watch lists, but TSA did not have measures to assess the extent of system matching errors, such as the extent to which Secure Flight is missing passengers who are actual matches to these lists. GAO recommended that TSA develop such measures. DHS concurred, and, as of April 2015, TSA was evaluating its current Secure Flight performance goals and measures and determining what new performance measures should be established to fully measure progress against program goals.

Use of program data. GAO has also reported on findings related to program data that TSA collected but had not analyzed, missing opportunities to refine and further improve TSA programs. For example, in January 2013, GAO reported that TSA collected and used key program data in support of its National Explosives Detection Canine Team Program, but could better analyze these data to identify program trends. For example, GAO found that in reviewing the results of certain covert tests, TSA did not analyze the results beyond the pass and fail rates, missing an opportunity to identify corrective actions. GAO recommended that TSA regularly analyze available data to identify program trends and areas that are working well and those in need of corrective action to guide program resources and activities. TSA concurred with GAO's recommendation and has taken actions to address this, including requiring analysis of the reasons for certain failed assessments.

Why GAO Did This Study

Since the attacks of September 11, 2001 exposed vulnerabilities in the nation's aviation system, billions of dollars have been spent on a wide range of programs designed to enhance aviation security. Continuing fiscal pressure highlights the need for DHS's TSA, the primary federal agency responsible for aviation security, to allocate its finite resources for the greatest impact.

This testimony addresses the extent to which TSA has (1) evaluated the overall effectiveness of new technologies, programs, and processes using robust methods of testing and evaluation, (2) established performance measures that fully reflect program goals, and (3) used program data to identify opportunities for improvement. This statement is based on findings from GAO reports and testimonies issued from January 2013 through June 2015, with selected updates conducted from April 2015 through October 2015 to, among other things, determine progress made in implementing previous GAO recommendations. For prior work, GAO analyzed TSA policy documents and interviewed TSA officials, among other things. For the updates, GAO reviewed documents and followed up with TSA officials about actions to address GAO recommendations.