Aviation Security: Efforts to Validate TSA's Passenger Screening Behavior Detection Program Underway, but Opportunities Exist to Strengthen Validation and Address Operational Challenges
GAO-10-763
Published: May 20, 2010. Publicly Released: May 20, 2010.
Skip to Highlights
Highlights
To enhance aviation security, the Transportation Security Administration (TSA) began initial testing in October 2003 of its Screening of Passengers by Observation Techniques (SPOT) program. Behavior Detection Officers (BDO) carry out SPOT's mission to identify persons who pose a risk to aviation security by focusing on behavioral and appearance indicators. GAO was asked to review the SPOT program. GAO analyzed (1) the extent to which TSA validated the SPOT program before deployment, (2) implementation challenges, and (3) the extent to which TSA measures SPOT's effect on aviation security. GAO analyzed TSA documents, such as strategic plans and operating procedures; interviewed agency personnel and subject matter experts; and visited 15 SPOT airports, among other things. Although the results from these visits are not generalizable, they provided insights into SPOT operations.
Although the Department of Homeland Security (DHS) is in the process of validating some aspects of the SPOT program, TSA deployed SPOT nationwide without first validating the scientific basis for identifying suspicious passengers in an airport environment. A scientific consensus does not exist on whether behavior detection principles can be reliably used for counterterrorism purposes, according to the National Research Council of the National Academy of Sciences. According to TSA, no other large-scale security screening program based on behavioral indicators has ever been rigorously scientifically validated. DHS plans to review aspects of SPOT, such as whether the program is more effective at identifying threats than random screening. Nonetheless, DHS's current plan to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. For example, factors such as the length of time BDOs can observe passengers without becoming fatigued are not part of the plan and could provide additional information on the extent to which SPOT can be effectively implemented. Prior GAO work has found that independent expert review panels can provide comprehensive, objective reviews of complex issues. Use of such a panel to review DHS's methodology could help ensure a rigorous, scientific validation of SPOT, helping provide more assurance that SPOT is fulfilling its mission to strengthen aviation security. TSA is experiencing implementation challenges, including not fully utilizing the resources it has available to systematically collect and analyze the information obtained by BDOs on passengers who may pose a threat to the aviation system. TSA's Transportation System Operations Center has the resources to investigate aviation threats but generally does not check all law enforcement and intelligence databases available to it to identify persons referred by BDOs. Utilizing existing resources would enhance TSA's ability to quickly verify passenger identity and could help TSA to more reliably "connect the dots." Further, most BDOs lack a mechanism to input data on suspicious passengers into a database used by TSA analysts and also lack a means to obtain information from the Transportation System Operations Center on a timely basis. TSA states that it is in the process of providing input capabilities, but does not have a time frame for when this will occur at all SPOT airports. Providing BDOs, or other TSA personnel, with these capabilities could help TSA "connect the dots" to identify potential threats. Although TSA has some performance measures related to SPOT, it lacks outcome-oriented measures to evaluate the program's progress toward reaching its goals. Establishing a plan to develop these measures could better position TSA to determine if SPOT is contributing to TSA's strategic goals for aviation security. TSA is planning to enhance its evaluation capabilities in 2010 to more readily assess the program's effectiveness by conducting statistical analysis of data related to SPOT referrals to law enforcement and associated arrests.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To help ensure that SPOT is based on valid scientific principles that can be effectively applied in an airport environment, the Secretary of Homeland Security should convene an independent panel of experts to review the methodology of the DHS Science and Technology (S&T) Directorate study on the SPOT program to determine whether the study's methodology is sufficiently comprehensive to validate the SPOT program. This assessment should include appropriate input from other federal agencies with expertise in behavior detection and relevant subject matter experts. |
In May 2010, we found that the Transportation Security Administration (TSA) had deployed its behavior detection program (referred to as "SPOT") without first validating the scientific basis for identifying passengers who may pose a security risk in an airport environment. The National Academy of Sciences reported in 2008 that a scientific consensus did not exist on whether behavior detection principles could be reliably used for counterterrorism purposes. Specifically, the study noted that the scientific support for linkages between behavioral and physiological markers and mental state was nonexistent for highly complex states, such as when individuals hold terrorist intent and beliefs. As part of its initial validation efforts, the Department of Homeland Security (DHS) planned to review aspects of TSA's behavior detection program, such as whether the program was more effective at identifying threats than random screening. However, the DHS review was not designed to conclusively demonstrate whether behavior detection techniques could be used to reliably identify individuals posing security risks in the airport environment. Thus, we recommended that DHS convene an independent expert panel, with input from outside experts in behavior detection, to review the methodology of the DHS study to determine whether the study's methodology was sufficiently comprehensive to validate the program. DHS concurred and completed a April 2011 review with input from a broad range of federal agencies and relevant experts. DHS characterized the study as an initial validation step that was not designed to fully validate whether behavior detection can be used reliably to identify individuals in an airport environment who may pose a security risk. Among other recommendations, the study recommended that additional steps be taken to validate the program. In July 2011, a senior DHS official stated that the findings of its independent expert panel were used to inform the study's recommendations. In August 2011, TSA provided us with the independent panel's final report that included findings and recommendations related to the study's methodology and future work. For example, the report noted that a broader evaluation would be needed to answer additional questions regarding the program and provided areas where future research could be performed. The report also provided information on panel participants which included a variety of experts from other federal agencies outside DHS. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | If research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then the TSA Administrator should conduct a comprehensive risk assessment to include threat, vulnerability, and consequence of airports nationwide to determine the effective deployment of SPOT if TSA's ongoing Aviation Modal Risk Assessment lacks this information. |
On December 28, 2012, the Transportation Security Administration (TSA) completed a risk-based allocation analysis which considered threat, vulnerability, and consequence in a framework to determine where to place behavior detection resources nationally to maximize security. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | If research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then the TSA Administrator should perform a cost-benefit analysis of the SPOT program, including a comparison of the SPOT program with other security screening programs, such as random screening, or already existing security measures. |
On December 28, 2012, the Transportation Security Administration (TSA) completed a return-on-investment analysis, in part, to inform the future direction of its behavior detection activities, including the Screening of Passengers by Observation Techniques (SPOT) program. This analysis assessed the additional value that behavior detection officers (BDO) add to TSA's checkpoint screening process and concluded that BDOs provide an integral value to the checkpoint screening system. The report stated that future iterations of the return-on-investment analysis would consider whether alternative combinations of countermeasures could provide comparable security with improved cost-efficiency. Further, the report noted that a future cost-benefit analysis would determine whether such options are realistic alternatives as part of a larger effort to ensure that TSA is employing the most effective and efficient concepts of operations available. In August 2013, TSA officials stated that while they plan to conduct a cost-benefit analysis, they did not have any efforts underway to do so. Given that the TSA did not have immediate plans to conduct a cost-benefit analysis, this recommendation is closed, not implemented.
|
| Transportation Security Administration | If research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then the TSA Administrator should revise and implement the SPOT strategic plan by incorporating risk assessment information, identifying cost and resources, linking it to other related TSA strategic documents, describing how SPOT is integrated and implemented with TSA's other layers of aviation security, and providing guidance on how to effectively link the roles, responsibilities, and capabilities of federal, state, and local officials providing program support. |
On December 5, 2012, Transportation Security Administration (TSA) officials issued a strategic plan that outlines goals for behavior detection activities. These goals include incorporating threat detection and risk assessments, integrating TSA's risk-based security efforts, and strengthening partnerships with local law enforcement entities to share intelligence. Further, the plan outlines resources needed for each activity area for the first year of the plan. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | If research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then the TSA Administrator should study the feasibility of using airport checkpoint-surveillance video recordings of individuals transiting checkpoints who were later charged with or pleaded guilty to terrorism-related offenses to enhance understanding of terrorist behaviors in the airport checkpoint environment. |
In April 2012, the Transportation Security Administration completed a study of the feasibility of using airport-checkpoint video surveillance recordings of individuals transiting checkpoints who were later charged with or pleaded guilty to terrorism related offenses to enhance the agency's understanding of terrorist behavior in the checkpoint environment. The study found that most checkpoints do not have enough video cameras positioned in a manner that would allow behaviors to be clearly captured. In addition, the cameras that are available do not have the requisite picture definition (i.e., that is 30 frames per second) needed to capture video of sufficient quality to allow the study of behaviors. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to provide additional assurance that TSA utilizes available resources to support the goals of deterring, detecting, and preventing security threats to the aviation system, the TSA Administrator should provide guidance in the SPOT Standard Operating Procedures or other TSA directive to Behavior Detection Officers (BDOs), or other TSA personnel, on inputting data into the Transportation Information Sharing System and set milestones and a time frame for deploying Transportation Information Sharing System access to SPOT airports so that TSA and intelligence community entities have information from all SPOT Law Enforcement Officer (LEO) referrals readily available to assist in "connecting the dots" and identifying potential terror plots. |
In May 2010, we found that the Transportation Security Administration (TSA) could better "connect the dots" regarding possible threats to the aviation system by more systematically collecting data on passengers identified through its behavior detection program and linking this information to other law enforcement and intelligence information. We recommended that TSA provide guidance to behavior detection officers (BDO) on how to input data into the Transportation Information Sharing System (TISS), which is a database used to share information among law enforcement and other agencies on suspicious activities and possible threats to the aviation system. In April 2011, TSA implemented revised procedures including guidance for BDOs to input incident reports into TISS. According to TSA, all BDO airports now have access to TISS. TSA also provided BDOs with a one day instructor led training course to provide an overview of TISS. In August 2011, TSA provided us with copies of related training materials, which include BDO related examples. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to provide additional assurance that TSA utilizes available resources to support the goals of deterring, detecting, and preventing security threats to the aviation system, the TSA Administrator should implement the steps called for in the TSA Office of Security Operations Business plan to develop a standardized process for allowing BDOs or other designated airport officials to send information to TSA's Transportation Security Operations Center about passengers whose behavior indicates that they may pose a threat to security, and provide guidance on how designated TSA officials are to receive information back from the Transportation Security Operations Center. |
In May 2010, we reported on a survey we conducted of Federal Security Directors at Screening of Passengers by Observation Techniques (SPOT) airports which found a notable inconsistency in the rates at which behavior detection officers (BDOs) at different airports contacted the Transportation Security Operations Center (TSOC). We stated that developing additional guidance in the SPOT operating procedures could help improve consistency in the extent to which BDOs utilize TSOC's resources. We recommended that the Transportation Security Administration (TSA) implement steps to develop a standardized process for allowing BDOs or other designated airport officials to send information to the TSOC about passengers whose behavior indicates that they may pose a threat to security and provide guidance on how designated TSA officials are to receive information back from the TSOC. On April 2012, we met with TSA officials who explained and subsequently sent us a December 2010 revised operations directive that specifies an incident report form and process for behavior detection officers to use when reporting suspicious individuals or activities to TSOC. The directive also contained guidance on how TSOC is to report back to the coordination center or Federal Security Director about actions taken in response to the incident report. These actions demonstrate the implementation of this recommendation.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to provide additional assurance that TSA utilizes available resources to support the goals of deterring, detecting, and preventing security threats to the aviation system, the TSA Administrator should direct the TSA Transportation Security Operations Center to utilize all of the law enforcement and intelligence databases available to it when running passenger names, for passengers who have risen to the level of a LEO referral. |
In May 2010, we reported that the Transportation Security Operations Center (TSOC) was not using all the resources at its disposal to support behavior detection officers in verifying potential risks to the aviation system. We stated that for passengers who have risen to the level of a law enforcement officer (LEO) referral at an airport checkpoint, having the TSOC consistently check their names against all the databases available to it could potentially help the Transportation Security Administration (TSA) identify threats to the aviation system and aid in "connecting the dots." We recommended that the TSOC should utilize all of the law enforcement and intelligence databases available to it when running passenger names, for passengers who have risen to the level of a LEO referral. On April 17, 2012, we met with TSA officials who told us that they were running the names of LEO referrals against all of the databases available to them. These actions demonstrate implementation of the recommendation.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to better measure the effectiveness of the program and evaluate the performance of BDOs, the TSA Administrator should establish a plan that includes objectives, milestones, and time frames to develop outcome-oriented performance measures to help refine the current methods used by Behavior Detection Officers for identifying individuals who may pose a risk to the aviation system. |
In November 2012, the Transportation Security Administration issued a Behavior Detection and Analysis Division Performance Metrics Plan that identifies current gaps in performance metrics collection and proposed metrics solutions and resource requirements for the next three years. This action demonstrates the implementation of this recommendation.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to better measure the effectiveness of the program and evaluate the performance of BDOs, the TSA Administrator should establish controls to help ensure completeness, accuracy, authorization, and validity of data collected during SPOT screening. |
In May 2010, we identified weaknesses in the Transportation Security Administration's (TSA) process for maintaining data in its behavior detection database. As a result, we recommended that TSA establish controls for this data. In October 2010, TSA officials briefed us and provided a demonstration on improvements the agency made to the database. TSA also provided us with data from the improved database. These improvements address and are consistent with our recommendation. As a result, the recommendation is closed as implemented.
|
| Transportation Security Administration | Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, to ensure the program's effective implementation, and to help ensure that TSA provides BDOs with the knowledge and skills needed to perform their duties, the TSA Administrator should establish time frames and milestones for its plan to systematically conduct evaluations of the SPOT training program on a periodic basis. |
In May 2010, we found that the Transportation Security Administration (TSA) had not established time frames and milestones for evaluating the Screening of Passengers by Observation Techniques (SPOT) training program on a periodic basis. To help ensure that TSA provides behavior detection officers (BDOs) with the knowledge and skills needed to perform their duties, we recommended that the TSA establish time frames and milestones as part of a plan to systematically conduct evaluations of the SPOT training program on a periodic basis. In February 2012, the TSA signed a decision memo, which established an annual review of SPOT training materials to ensure accuracy and currency. This action demonstrates the implementation of this recommendation.
|
Full Report
Topics
Air transportationAirport securityAviation securityBehavioral sciences researchCost analysisCounterterrorismEmployee trainingLaw enforcementOperational testingPassenger screeningPerformance measuresProgram evaluationRisk assessmentRisk factorsRisk managementSecurity threatsStrategic planningSystems testingTraining utilizationProgram implementation