This is the accessible text file for GAO report number GAO-10-763 
entitled 'Aviation Security: Efforts to Validate TSA's Passenger 
Screening Behavior Detection Program Underway, but Opportunities Exist 
to Strengthen Validation and Address Operational Challenges' which was 
released on May 20, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Ranking Member, Committee on Transportation and 
Infrastructure, House of Representatives: 

United States Government Accountability Office: 
GAO: 

May 2010: 

Aviation Security: 

Efforts to Validate TSA's Passenger Screening Behavior Detection 
Program Underway, but Opportunities Exist to Strengthen Validation and 
Address Operational Challenges: 

GAO-10-763: 

GAO Highlights: 

Highlights of GAO-10-763, a report to the Ranking Member, Committee on 
Transportation and Infrastructure, House of Representatives. 

Why GAO Did This Study: 

To enhance aviation security, the Transportation Security 
Administration (TSA) began initial testing in October 2003 of its 
Screening of Passengers by Observation Techniques (SPOT) program. 
Behavior Detection Officers (BDO) carry out SPOT’s mission to identify 
persons who pose a risk to aviation security by focusing on behavioral 
and appearance indicators. GAO was asked to review the SPOT program. 
GAO analyzed (1) the extent to which TSA validated the SPOT program 
before deployment, (2) implementation challenges, and (3) the extent 
to which TSA measures SPOT’s effect on aviation security. GAO analyzed 
TSA documents, such as strategic plans and operating procedures; 
interviewed agency personnel and subject matter experts; and visited 
15 SPOT airports, among other things. Although the results from these 
visits are not generalizable, they provided insights into SPOT 
operations. 

What GAO Found: 

Although the Department of Homeland Security (DHS) is in the process 
of validating some aspects of the SPOT program, TSA deployed SPOT 
nationwide without first validating the scientific basis for 
identifying suspicious passengers in an airport environment. A 
scientific consensus does not exist on whether behavior detection 
principles can be reliably used for counterterrorism purposes, 
according to the National Research Council of the National Academy of 
Sciences. According to TSA, no other large-scale security screening 
program based on behavioral indicators has ever been rigorously 
scientifically validated. DHS plans to review aspects of SPOT, such as 
whether the program is more effective at identifying threats than 
random screening. Nonetheless, DHS’s current plan to assess SPOT is 
not designed to fully validate whether behavior detection can be used 
to reliably identify individuals in an airport environment who pose a 
security risk. For example, factors such as the length of time BDOs 
can observe passengers without becoming fatigued are not part of the 
plan and could provide additional information on the extent to which 
SPOT can be effectively implemented. Prior GAO work has found that 
independent expert review panels can provide comprehensive, objective 
reviews of complex issues. Use of such a panel to review DHS’s 
methodology could help ensure a rigorous, scientific validation of 
SPOT, helping provide more assurance that SPOT is fulfilling its 
mission to strengthen aviation security. 

TSA is experiencing implementation challenges, including not fully 
utilizing the resources it has available to systematically collect and 
analyze the information obtained by BDOs on passengers who may pose a 
threat to the aviation system. TSA’s Transportation System Operations 
Center has the resources to investigate aviation threats but generally 
does not check all law enforcement and intelligence databases 
available to it to identify persons referred by BDOs. Utilizing 
existing resources would enhance TSA’s ability to quickly verify 
passenger identity and could help TSA to more reliably “connect the 
dots.” Further, most BDOs lack a mechanism to input data on suspicious 
passengers into a database used by TSA analysts and also lack a means 
to obtain information from the Transportation System Operations Center 
on a timely basis. TSA states that it is in the process of providing 
input capabilities, but does not have a time frame for when this will 
occur at all SPOT airports. Providing BDOs, or other TSA personnel, 
with these capabilities could help TSA “connect the dots” to identify 
potential threats. 

Although TSA has some performance measures related to SPOT, it lacks 
outcome-oriented measures to evaluate the program’s progress toward 
reaching its goals. Establishing a plan to develop these measures 
could better position TSA to determine if SPOT is contributing to 
TSA’s strategic goals for aviation security. TSA is planning to 
enhance its evaluation capabilities in 2010 to more readily assess the 
program’s effectiveness by conducting statistical analysis of data 
related to SPOT referrals to law enforcement and associated arrests. 

What GAO Recommends: 

GAO recommends that TSA, among other things, use an independent panel 
of experts to assist in validating SPOT, enhance SPOT data collection 
and analysis, fully utilize TSA resources to identify possible 
threats, and establish a plan to develop more outcome-oriented 
measures for SPOT. DHS reviewed a draft of this report and generally 
concurred with our recommendations although its plans do not fully 
address one of our recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-10-763] or key 
components. For more information, contact Stephen M. Lord at (202) 512-
4379 or lords@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DHS Is Taking Action to Validate the Scientific Basis of TSA's SPOT 
Program but Opportunities Exist to Help Inform Future Program 
Decisions: 

More Fully and Consistently Utilizing Available Information Technology 
Could Enhance TSA's Ability to Identify Threats to the Aviation System: 

TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps 
to Improve Evaluation Capabilities: 

TSA Developed and Deployed SPOT Training but Further Action Could 
Enhance Its Effectiveness: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: DHS Comments: 

Appendix III: GAO Contacts and Staff Acknowledgments: 

GAO Contact: 

Staff Acknowledgments: 

Obtaining Copies of GAO Reports and Testimony: 

Tables: 

Table 1: Summary of Desirable Characteristics for Developing a 
Strategic Plan: 

Table 2: Reasons for Arrests from SPOT Referrals, May 29, 2004, 
through August 31, 2008: 

Table 3: SPOT Instructor Evaluation Ratings, 2006 to September 2008, 
and March 2009: 

Table 4: TSA Training Standards and Evaluation Branch Recommendations 
for Improving SPOT Training and TSA Actions on the Recommendations: 

Figures: 

Figure 1: TSA's Layers of Aviation Security: 

Figure 2: The First Step in the SPOT Process: BDOs Observing 
Passengers About to Go Through Checkpoint Magnetometer: 

Figure 3: Budget and Personnel Growth in the SPOT Program, Fiscal 
Years 2007 through 201027: 

Figure 4: Passenger Boardings at SPOT Airports, May 29, 2004, through 
August 31, 200844: 

Abbreviations: 

AMRA: Aviation Modal Risk Assessment: 

BDO: Behavior Detection Officer: 

CBP: U.S. Customs and Border Protection: 

DEA: Drug Enforcement Agency: 

DHS: Department of Homeland Security: 

FAMS: Federal Air Marshal Service: 

FBI: Federal Bureau of Investigation: 

ICE: U.S. Immigration and Customs Enforcement: 

LEO: Law Enforcement Officer: 

NCIC: National Crime Information Center: 

NIPP: National Infrastructure Protection Plan: 

OMB: Office of Management and Budget: 

SOP: Standard Operating Procedures: 

SPOT: Screening of Passengers by Observation Techniques: 

S&T: Science and Technology Directorate: 

TSA: Transportation Security Administration: 

TSO: Transportation Security Officer: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

May 20, 2010: 

The Honorable John L. Mica: 
Ranking Member: 
Committee on Transportation: 
and Infrastructure House of Representatives: 

Dear Mr. Mica: 

The terrorist attacks of September 11, 2001, highlighted the need to 
improve security within the nation's civil aviation system to deter 
persons seeking to repeat similar attacks on the nation's critical 
infrastructure. In October 2003, the Transportation Security 
Administration (TSA) of the Department of Homeland Security (DHS) 
conducted an operational test of the use of behavior detection 
techniques to screen passengers in an airport environment, and 
subsequently began training certain Transportation Security Officers 
(TSO)--TSA employees responsible for screening passengers and their 
property--in these techniques. These TSOs performed behavior 
observation as a collateral duty. Beginning in fiscal year 2007, TSA 
created separate Behavior Detection Officer (BDO) positions as part of 
the Screening of Passengers by Observation Techniques (SPOT) program. 
[Footnote 1] According to TSA, the SPOT program is a derivative of 
other behavioral analysis programs that have been successfully 
employed by law enforcement and security personnel both in the United 
States and around the world, particularly that of Israel's airline, El 
Al.[Footnote 2] 

TSA designed SPOT to provide BDOs with a means of identifying persons 
who may pose a potential security risk at TSA-regulated airports 
[Footnote 3] by focusing on behaviors and appearances that deviate 
from an established baseline, and that may be indicative of stress, 
fear, or deception. Passengers in an airport terminal, including those 
waiting in security checkpoint lines, are observed by the BDOs to 
determine if their behavioral and appearance indicators--which are 
assigned varying points by SPOT--have (in combination) exceeded a 
predetermined numerical threshold. In cases where the passenger 
exceeds the threshold, the passenger is referred for additional 
screening by BDOs and a TSO. During this referral screening, if the 
passenger exhibits behaviors that exceed another numerical threshold, 
they are to be referred to a law enforcement officer (LEO) for further 
investigation. In addition to observing passengers at airport 
checkpoints, BDOs may patrol throughout an airport terminal, and 
sometimes participate in other activities, such as TSA's Visible 
Intermodal Prevention and Response team operations. These teams are 
responsible for periodically augmenting security at air and ground 
transportation facilities around the country.[Footnote 4] 

As of March 2010, TSA deployed about 3,000 BDOs at an annual cost of 
about $212 million; this force increased almost fifteen-fold between 
March 2007 and July 2009. BDOs have been selectively deployed to 161 
of the 457 TSA-regulated airports in the United States at which 
passengers and their property are subject to TSA-mandated screening 
procedures.[Footnote 5] The conference report accompanying the fiscal 
year 2010 DHS appropriations act provided that $211.9 million of 
aviation security funding was for the SPOT program.[Footnote 6] The 
administration has requested $232 million for SPOT for fiscal year 
2011, a $20.2 million (9.5 percent) increase over the current funding 
level. This increase would support a workforce increase from about 
3,000 to 3,350 BDOs. If this funding request is approved and 
maintained, SPOT would cost about $1.2 billion over the next 5 years. 

You asked us to address SPOT's development and implementation. This 
report addresses the following questions: 

1. To what extent did TSA determine whether SPOT had a scientifically 
validated basis for identifying passengers before deploying it and 
utilize recognized best practices during SPOT's development? 

2. What management challenges, if any, have emerged during the 
implementation of SPOT at the nation's airports? 

3. To what extent has TSA measured SPOT's effect on aviation security? 

4. To what extent has TSA incorporated the attributes of an effective 
training program into the training for SPOT? 

This report is a public version of the restricted report (GAO-10-
157SU) that we provided to you on May 14, 2010. DHS and TSA deemed 
some of the information in the restricted report as sensitive security 
information, which must be protected from public disclosure. 
Therefore, this report omits this information. Although the 
information provided in this report is more limited in scope, it 
addresses the same questions as the restricted report. Also, the 
overall methodology used for both reports is the same. 

To determine the extent to which TSA determined whether SPOT had a 
scientifically validated basis for identifying passengers who may pose 
a risk to aviation security before deploying it, we reviewed 
literature on behavior analysis by subject matter experts, and 
analyzed relevant reports and books on the topic. These included a 
2008 study by the National Research Council of the National Academy of 
Sciences that included a discussion section on deception and 
behavioral surveillance, as well as other issues related to behavioral 
analysis.[Footnote 7] We interviewed seven recognized experts in the 
field, and an expert on emergency responses to terror attacks and 
mathematical models in operations management.[Footnote 8] Although the 
views of these experts cannot be generalized across all experts on 
behavior analysis, because we selected these individuals based on 
their publications on behavioral analysis or related topics, their 
recognized accomplishments and expertise, and, in some cases, TSA's 
use of their work or expertise to design and review the SPOT program's 
behaviors, they provided us with an understanding of the fundamentals 
of behavior analysis, and its use in airports. We also interviewed 
cognizant officials from other U.S. government agencies that utilize 
behavior analysis in their work, including U.S. Customs and Border 
Protection (CBP), the U.S. Secret Service, the Federal Air Marshall 
Service (FAMS), and the Federal Bureau of Investigation (FBI). 
[Footnote 9] To better understand how SPOT incorporated expertise on 
behavior analysis for aviation security, we also interviewed current 
and retired officials of Israel's El Al Airlines, whose security 
processes TSA cites as providing part of the basis of the SPOT 
program.[Footnote 10] 

To determine to what extent TSA utilized best practices during SPOT's 
development--including carrying out a comprehensive risk assessment, a 
cost-benefit analysis, and a strategic plan--we interviewed program 
officials and reviewed related program documentation, including 
briefings used in the course of developing and fielding SPOT, 
strategic plans, and standard operating procedures.[Footnote 11] We 
compared these documents to DHS's 2006 Cost Benefit Analysis 
Guidebook,[Footnote 12] Office of Management and Budget (OMB) 
guidance,[Footnote 13] and DHS's 2006 and 2009 National Infrastructure 
Protection Plans (NIPP), which set forth a risk management framework 
to guide security decision making and resource allocation decisions, 
and our previous work on the characteristics of an effective strategic 
plan. 

To identify any challenges that emerged during implementation of the 
SPOT program, we conducted field site visits to 15 TSA-regulated 
airports with SPOT that represent almost 10 percent of the 161 TSA- 
regulated airports with SPOT to observe operations and meet with key 
program personnel.[Footnote 14] We chose airports with high, medium, 
and low passenger volume; airports with BDOs who are TSA (i.e., 
government) employees and an airport with BDOs employed by contractors 
as part of the TSA Screening Partnership Program; and airports with 
LEOs who were identified by TSA as having received some form of 
behavior detection training and airports where they were not known to 
have received such training.[Footnote 15] We also selected airports on 
the basis of TSA's assessment of which ones are at highest risk of 
attack by terrorists, including the 2 that ranked the highest, as 
reported in TSA's Current Airport Threat Assessment.[Footnote 16] 
Since the airports we selected range broadly in terms of passenger 
volume, physical size and layout, geographic location, and potential 
value as a target for terrorism, among other things, the results from 
these visits are not generalizable to other airports. However, these 
visits provided helpful insights into the operation of SPOT at 
airports. In addition, to determine whether challenges emerged in 
implementing SPOT, we compared TSA's approach for implementing and 
managing SPOT to our Standards for Internal Control in the Federal 
Government[Footnote 17] and to risk management principles we had 
previously identified.[Footnote 18] In reviewing TSA's approach to 
developing and implementing SPOT, we considered relevant laws, 
regulations, and other materials, including those related to privacy, 
such as TSA's Privacy Impact Assessments. To obtain comparative data 
on how SPOT had been implemented at different airports across the 
nation, we conducted a survey of all Federal Security Directors 
responsible for security operations at TSA-regulated airports with 
SPOT.[Footnote 19] (This accounted for all 161 TSA-regulated airports 
with SPOT because a single Federal Security Director may be 
responsible for several airports.) We obtained a 100 percent response 
rate. This survey asked, among other things, about the relationship 
between LEOs and the airport authority and BDOs. In addition, to 
understand the interaction of BDOs and LEOs, as well as other SPOT 
implementation issues, at each of the 15 TSA-regulated airports we 
visited we spoke with BDO managers, Federal Security Directors, 
Assistant Federal Security Directors, 1 or 2 BDOs, and 1 or 2 LEOs. 

To determine the extent to which TSA has measured SPOT's effect on 
aviation security, we obtained and analyzed the TSA SPOT referral 
database,[Footnote 20] which aims to record all incidents in which 
passengers who have passed through the checkpoint are sent to SPOT 
referral screening for additional questioning and screening of 
property and person. The database also maintains records of instances 
where passengers were referred by a BDO to a LEO for questioning. We 
assessed the reliability of the SPOT referral data by (1) performing 
electronic testing of required data elements, (2) reviewing existing 
information about the data and the system that produced them, and (3) 
interviewing agency officials knowledgeable about the data. We found a 
number of problems related to how the data were collected and recorded 
that are discussed later in this report. As a result, we were unable 
to use the SPOT referral data to assess whether any behavior or 
combination of SPOT behaviors could be used to reliably predict the 
final outcome of an incident involving the use of SPOT. However, with 
the stated limitations in mind, and after resolving certain 
contradictions and anomalies in the database, we utilized the SPOT 
referral data to provide examples of information used by TSA to report 
on the program's performance, including a count of arrests and the 
reasons for those arrests. In addition, to determine if individuals 
who were later charged with or pleaded guilty to terrorism-related 
offenses had transited SPOT airports and whether TSA could obtain 
information from these transits to enhance its understanding of 
terrorist behaviors, we reviewed CBP and Department of Justice 
information to (1) identify individuals who were charged with or 
pleaded guilty to terrorism-related offenses and (2) determine if 
these individuals had, prior to being charged, transited airports 
where SPOT had been deployed. Further, we used our survey of Federal 
Security Directors at SPOT airports to determine the extent to which 
video surveillance cameras, which could make video recordings of 
terrorists transiting airports, are present at checkpoints. 

To assess the extent that SPOT training incorporates the attributes of 
an effective training program, we had TSA training experts complete a 
training assessment tool that we developed using guidance we prepared 
in our previous work for assessing training courses and curricula. 
[Footnote 21] To better understand how other entities train their 
employees in behavior detection, and what their curricula include, we 
conducted site visits to the Secret Service, CBP, FAMS, and the FBI, 
and also interviewed nongovernmental experts on aspects of behavior 
detection training. We interviewed BDOs and BDO managers about the 
SPOT training. In addition, we interviewed El Al officials with regard 
to how El Al trains and tests its personnel in behavior recognition 
and analysis. Appendix I contains additional details about our scope 
and methodology. 

We conducted this performance audit from May 2008 through May 2010 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Background: 

The Aviation and Transportation Security Act established TSA as the 
federal agency with primary responsibility for securing the nation's 
civil aviation system, which includes the screening of all passenger 
and property transported by commercial passenger aircraft.[Footnote 
22] TSA currently has direct responsibility for, or oversees the 
performance of, security operations at approximately 457 TSA-regulated 
airports in the United States implementing security requirements in 
accordance with TSA-approved security programs and other TSA 
direction.[Footnote 23] At TSA-regulated airports, prior to boarding 
an aircraft, all passengers, their accessible property, and their 
checked baggage are screened pursuant to TSA-established procedures, 
which include, for example, passengers passing through security 
checkpoints where they and their identification documents are checked 
by TSOs and Travel Document Checkers, or by Screening Partnership 
Program employees. 

TSA uses multiple layers of security to deter, detect, and disrupt 
persons posing a potential risk to aviation security. These layers 
include three principal types of screening employees at airport 
checkpoints--Travel Document Checkers, who examine tickets, passports, 
and other forms of identification; TSOs, who examine property, 
including checked baggage, and persons using x-ray equipment and 
magnetometers, as well as other devices; and BDOs, using SPOT to 
assess passenger behaviors and appearance.[Footnote 24] BDOs are the 
only type of TSA screening employees not deployed to all TSA-regulated 
airports and all checkpoints within the airports where it is deployed 
on a regular basis. TSA deployed SPOT as an added layer of security to 
help deter terrorists attempting to exploit TSA's focus on prohibited 
items and other potential security weaknesses. Other security layers 
cited by TSA include intelligence gathering and analysis; passenger 
prescreening; random canine team searches at airports; federal air 
marshals; reinforced cockpit doors; federal flight deck officers; the 
passengers themselves; as well as other measures both visible and 
invisible to the public. Figure 1 shows TSA's 20 aviation security 
layers. 

Figure 1: TSA's Layers of Aviation Security: 

[Refer to PDF for image: illustration] 

Terrorist paths: through the following layers: 

Intelligence: 
Customs and Border Protection: 
Joint Terrorism Task Force: 
No-fly List and Passenger Pre-screening[A]: 
Crew Vetting: 
VIPR: 
Canines: 
[Start gray bar, screening layers applied to passengers and their 
property: 
Behavior Detection Officers: 
Travel Document Checker: 
Checkpoint/Transportation Security Officers: 
Checked Baggage[B]; 
End gray bar, screening layers applied to passengers and their 
property] 
Transportation Security Inspectors: 
Random Employee Screening: 
Bomb Appraisal Officers: 
Federal Air Marshal Service: 
Federal Flight Deck Officers: 
Trained Flight Crew: 
Law Enforcement Officers: 
Hardened Cockpit Door: 
Passengers: 

Source: TSA. 

[A] The No-Fly List is used to identify individuals who should be 
prevented from boarding an aircraft; it contains applicable records 
from the FBI's Terrorist Screening Center consolidated database of 
known or suspected terrorists. 

[B] The four layers inside the gray bar are screening layers of 
security applied to passengers and their property. 

[End of figure] 

The gray area in figure 1 highlights four layers that apply to 
passengers and their property as they seek to board an aircraft. 
Airport LEOs, another layer of security cited by TSA, do not report to 
TSA and may not maintain a physical presence at smaller TSA-regulated 
airports. According to TSA, each one of these layers alone is capable 
of stopping a terrorist attack. In combination, TSA states that their 
security value is multiplied, creating a much stronger system, and 
that a terrorist who has to overcome multiple security layers in order 
to carry out an attack is more likely to be preempted, deterred, or to 
fail during the attempt. 

SPOT Uses Behavior Detection Techniques to Assess Passenger Behaviors 
and Appearances: 

The SPOT program utilizes behavior observation and analysis techniques 
to identify potentially high-risk passengers. Individuals who exhibit 
suspicious behaviors, including both physical and appearance 
indicators, may be required to undergo additional screening. Field 
agents and law enforcement officers of other federal agencies and 
entities--such as the FBI, the Secret Service, CBP, and FAMS--utilize 
elements of behavior detection analysis as a part of their work. In 
addition, some foreign entities, such as Israel's El Al airlines, use 
behavior detection and analysis techniques as part of their security 
efforts. However, TSA emphasized to us that the SPOT program is unique 
among these entities because it uses a point system to help identify 
suspicious persons on the basis of their behavior and appearance and 
because behavior detection and analysis are the central focus of SPOT. 
Officials from the other agencies stated that their field personnel 
incorporate behavior detection as one of many skills used in their 
work; in contrast, behavior detection is the primary element of the 
BDOs' work. 

SPOT trains BDOs to look for and recognize facial expressions, body 
language, and appearance that indicate the possibility that an 
individual is engaged in some form of deception and fears discovery. 
These behaviors and appearances are listed on a SPOT score sheet used 
in SPOT training. 

Passenger behavior and appearance are to be compared by the BDOs--who 
typically work in two-person teams. BDOs are expected to "walk the 
line"--that is, to initiate casual conversations with passengers 
waiting in line, particularly if their observations led them to 
question someone exhibiting behaviors or appearances on the SPOT 
checklist. As the BDOs walk the line, and the passenger with SPOT 
indicators is reached, a casual conversation is used to determine if 
there is a basis for observed behaviors or appearances on the 
checklist. In most instances, these conversations provide information 
to the BDOs that permits them to consider the issue resolved, and 
hence not a security concern. Figure 2 below illustrates the first 
step of the three-step SPOT process, the BDO-passenger interaction at 
a checkpoint prior to the passenger passing through a magnetometer. 

Figure 2: The First Step in the SPOT Process: BDOs Observing 
Passengers About to Go Through Checkpoint Magnetometer: 

[Refer to PDF for image: illustration] 

Step 1: 
1. BDOs scan the passengers in line and occasionally initiate casual 
conversation. 

2. BDOs identify person(s) who exhibit clusters of suspicious 
behaviors that meet a given threshold. 

3. BDOs identify passengers exhibiting behaviors that exceed SPOT 
numerical threshold for referral questioning. 

Sources: GAO (analysis), ArtExplosion (clip art), TSA (data). 

Note: Circle around passenger shows a person who is exhibiting a 
cluster of suspicious behaviors. 

[End of figure] 

As shown in figure 2, passenger behavior and appearance are observed 
by the BDOs as passengers wait in line for screening at a security 
checkpoint. Even if the checkpoint is busy, the BDOs must attempt to 
visually scan all the passengers waiting in line, as well as persons 
near the checkpoint, to determine if any are showing behaviors or 
appearances on the SPOT checklist. According to TSA, on average a BDO 
has approximately 30 seconds to assess each passenger while the 
passenger waits in line. For passengers exhibiting indicators above 
baseline conditions, the BDOs are to (mentally) add up the points 
assigned to each indicator they observe. Both BDO team members must 
agree that observed indicators have exceeded the predetermined 
numerical threshold, although they do not have to identify the same 
indicators the passenger exhibited. In instances when a passenger's 
SPOT indicators place them above the numerical threshold, and the 
passenger has placed their property on the conveyor belt for x-raying, 
and has walked through the magnetometer or equivalent screening device 
for passengers, he or she will be directed to the second step of SPOT, 
referral screening. This involves additional questioning and physical 
search of their person and property by BDOs and TSOs. This referral 
screening occurs in the checkpoint area. 

If the passenger's behavior escalates further--accumulating more 
points based on the SPOT checklist--the BDOs are to refer the 
passenger to a LEO. A referral to a LEO is a potential third step in 
the SPOT process. BDOs are not LEOs--they do not conduct criminal 
investigations, carry weapons, or make arrests. 

After a passenger has been referred by the BDOs to a LEO, the LEO is 
then expected to independently determine, through additional 
investigation, such as questioning the passenger and, if appropriate, 
by conducting an identity verification and background check through 
the FBI's National Crime Information Center (NCIC), whether sufficient 
grounds exist to take further action, such as detaining or arresting 
the passenger. TSA officials who are LEOs also have access to NCIC, 
such as an airport's Assistant Federal Security Director for Law 
Enforcement or federal air marshals. NCIC is the FBI's computerized 
index of criminal justice information (i.e., criminal record history 
information, fugitives, stolen properties, and missing persons), 
available to federal, state, and local law enforcement and other 
criminal justice agencies at all times.[Footnote 25] Similarly, other 
federal LEOs also have such access, including CBP, and Drug 
Enforcement Agency (DEA) personnel. However, since both local and 
federal LEOs have other responsibilities, and may not be present at 
each operating checkpoint, BDOs may have to seek them out to request 
an NCIC check. According to TSA, aside from requiring that an airport 
maintain a law enforcement presence,[Footnote 26] it exercises no 
jurisdiction over the law enforcement activities of non-TSA officers 
or entities at an airport; thus, it cannot require LEOs to conduct an 
NCIC check or to provide BDOs with information about the ultimate 
disposition of cases referred by them to LEOs. 

Once the LEO concludes his or her investigation and determines whether 
the passenger will be arrested or detained, TSA officials are to 
evaluate the security concerns to determine whether to allow the 
passenger to proceed to the boarding gate. (In some instances, a LEO 
might choose not to arrest or detain a passenger; TSA would then 
decide whether the infraction was sufficiently serious to necessitate 
barring the passenger from boarding.) After a referral incident has 
been resolved, BDOs are to enter information about the incident into 
TSA's SPOT referral database. The data entered are to include time, 
date, location of the incident, behaviors witnessed, prohibited items 
found (if any), and information on the LEO's response (if applicable), 
such as whether the LEO questioned the passenger, arrested the 
individual, or released the passenger. The SPOT referral database 
contains no personal identifying information about passengers. 

SPOT Has Been Deployed in Phases: 

The SPOT program began with pilot tests in 2003 and 2004 at several 
New England airports, in which TSA began using uniformed BDOs at 
airport checkpoints. After some initial pilot projects and test 
deployments, 644 BDOs were deployed to 42 airports in the first phase 
of the program from November 2006 through June 2007. As of March 2010, 
about 3,000 BDOs utilizing SPOT were deployed at 161 of 457 TSA-
regulated airports.[Footnote 27] 

BDO eligibility is restricted to TSOs with at least 12 months of TSO 
experience, or others with related security experience. Applicants 
must apply and be accepted into the BDO training program. The training 
includes 4 days of classroom courses, followed by 3 days of on-the-job 
training. BDOs must memorize all of the behaviors and appearances on 
the SPOT checklist, as well as the point value assigned to each, in 
order to be able to add these up to determine if a passenger should be 
sent to SPOT referral screening. BDO applicants must also pass a job 
knowledge test at the conclusion of the training. The test includes 
related multiple choice questions, true or false statements, and case- 
based scenarios. 

DHS Is Taking Action to Validate the Scientific Basis of TSA's SPOT 
Program but Opportunities Exist to Help Inform Future Program 
Decisions: 

Although DHS is in the process of validating the way in which the SPOT 
program utilizes the science of behavior detection in an airport 
environment, TSA deployed SPOT nationwide before first determining 
whether there was a scientifically valid basis for using behavior and 
appearance indicators as a means for reliably identifying passengers 
as potential threats in airports. TSA reported that it deployed SPOT 
before a scientific validation of the program was completed in 
response to the need to address potential threats to the aviation 
system that would not necessarily be detected by existing layers of 
aviation security. TSA stated that no other large-scale U.S. or 
international screening program incorporating behavior-and appearance-
based indicators has ever been rigorously scientifically validated. 
While TSA deployed SPOT on the basis of some risk-related factors, 
such as threat information and airport passenger volume, it did not 
use a comprehensive risk assessment to guide its strategy of 
selectively deploying SPOT to 161 of the nation's 457 TSA-regulated 
airports. TSA also expanded the SPOT program over the last 3 years 
without the benefit of a cost-benefit analysis of SPOT. Additionally, 
TSA's strategic plan for SPOT could be improved by the inclusion of 
desirable characteristics identified in our prior work, such as risk 
assessment information, cost and resources analysis, and a means for 
collaboration with other key entities. 

TSA Is in the Process of Validating the Scientific Basis Used to 
Identify Passengers with SPOT Behaviors: 

TSA proceeded with deploying SPOT on a nationwide basis before 
determining whether the list of passenger behaviors and appearances 
underpinning the SPOT program were scientifically validated, and 
whether these techniques could be applied for counterterrorism 
purposes in an airport environment. In 2008, a report issued by the 
National Research Council of the National Academy of Sciences noted 
that behavior and appearances monitoring might be able to play a 
useful role in counterterrorism efforts but stated that a scientific 
consensus does not exist regarding whether any behavioral surveillance 
or physiological monitoring techniques are ready for use in the 
counterterrorist context given the present state of the science. 
[Footnote 28] The report also stated that the scientific evidence for 
behavioral monitoring is preliminary in nature.[Footnote 29] According 
to the report, an information-based program, such as a behavior 
detection program, should first determine if a scientific foundation 
exists and use scientifically valid criteria to evaluate its 
effectiveness before going forward. The report added that programs 
should have a sound experimental basis and documentation on the 
program's effectiveness should be reviewed by an independent entity 
capable of evaluating the supporting scientific evidence. The report 
also stated that often scientists and other experts can help 
independently assess the scientific evidence on the effectiveness of a 
program. A contributor to the National Research Council report also 
stated that no conclusive research has been conducted to determine if 
behavior detection can be reliably used on a larger scale, such as in 
an airport setting, to identify persons intending to cause harm to the 
aviation system. 

While TSA and DHS's Science and Technology (S&T) Directorate officials 
agreed that SPOT was deployed before its scientific underpinnings were 
fully validated, they stated that no large-scale U.S. or international 
operational screening program incorporating behavior-and appearance- 
based indicators has been rigorously scientifically validated. These 
officials also questioned the findings of the National Research 
Council report and stated that the study lacked sufficient information 
for its conclusions because it did not consider recent findings from 
unpublished DHS, defense, and intelligence community studies.[Footnote 
30] However, National Research Council officials stated that an agency 
should be cautious about relying on the results of unpublished 
research that has not been peer reviewed, such as that generated by 
DHS and the defense and intelligence community, and using unpublished 
work as a basis for proceeding with a process, method, or program. 
[Footnote 31] Moreover, we have previously reported that peer review 
is widely accepted as an important quality control mechanism that 
helps prevent the dissemination of potentially erroneous information. 
[Footnote 32] 

In addition to the unpublished research, TSA told us that the SPOT 
program was based on operational best practices from law enforcement, 
defense, and the intelligence communities. According to TSA officials, 
the agency based its choice of SPOT behavior, appearance, and 
deception indicators on existing research and training programs. For 
example, TSA cited research on emotions and their behavior indicators 
by Dr. Paul Ekman,[Footnote 33] interviewing and interrogation by Stan 
Walters,[Footnote 34] and nonverbal indicators by Dr. David Givens 
[Footnote 35] and Dr. Mark Frank[Footnote 36] as support for the 
choice of several of the behavior indicators. According to TSA, its 
development of the SPOT program was based on related DHS research and 
information from the training curricula of other federal agencies, 
such as the Federal Transit Administration and the Bureau of Alcohol, 
Tobacco, Firearms, and Explosives.[Footnote 37] 

As with the SPOT behavior indicators, TSA told us that it sought input 
in creating the SPOT point scoring system from subject matter experts 
and from participants in TSA's SPOT working group, which consisted of 
law enforcement officials from agencies such as FBI, DEA, and local 
law enforcement officials.[Footnote 38] While TSA officials said that 
they coordinated with relevant subject matter experts, such as Dr. 
Ekman, and based the SPOT scoring system on existing research and 
training programs, no validation of the behavior, appearance, and 
deception indicators was conducted prior to the deployment of SPOT in 
November 2006. According to TSA officials, they used professional 
judgment in developing the SPOT point system and stated that the 
purpose of developing the scoring system was to increase the 
objectivity of the SPOT process. 

Dr. Ekman stated that, in his opinion, and after reviewing the scoring 
system and observing the program in operation, it was not clear 
whether the SPOT behaviors and appearances, and the related point 
system, could be used effectively in an airport environment because no 
credible validation research on this issue had been conducted. He 
noted, for example, that research is needed to identify how many BDOs 
are required to observe a given number of passengers moving at a given 
rate per day in an airport environment, or the length of time that 
such observation can be conducted before observation fatigue affects 
the effectiveness of the personnel. He commented that observation 
fatigue is a well-known phenomenon among workers whose work involves 
intense observation, and that it is essential to determine the 
duration of effective observation and to ensure consistency and 
reliability among the personnel carrying out the observations. 

DHS has recognized the need to conduct additional research to 
scientifically validate the use of the SPOT behavioral indicators in 
an airport environment. DHS's S&T Directorate began research in 2007 
to determine if there is a statistically significant correlation 
between the SPOT behaviors exhibited by airport passengers and finding 
airport passengers with prohibited items (such as weapons), false 
documents, and illegal drugs or who pose a potential risk to aviation 
security. According to S&T, this research is expected to be completed 
in fiscal year 2011 and is to include three key elements. First, the 
study's purpose is to assess the reliability of the SPOT program by 
analyzing TSA's SPOT database to determine patterns of BDO scoring to 
measure consistency across BDOs, teams, locations, and other 
variables. Second, the study aims to compare the current 
implementation of SPOT to random passenger screening. Specifically, 
according to S&T officials, 130,000 passengers are to be randomly 
selected for additional SPOT referral screening. The study's design 
states that data collected about these passengers will be compared to 
data for passengers screened through the normal SPOT process. S&T 
officials expect that the results of this element of the study will 
provide a better understanding of how SPOT compares to random 
selection, as well as providing a baseline of each indicator present 
in the traveling public. Third, the study also aims to utilize live 
and video data, as available, to measure SPOT score ratings by BDOs of 
behaviors exhibited by passengers against ratings of the same 
passengers by subject matter experts. This element of the study could 
help determine whether BDOs are using, or are continuing to use, the 
SPOT score sheet correctly as time passes after their initial 
training. According to S&T officials, the study is to form the basis 
for BDO performance and training requirements. 

The S&T Directorate reported some preliminary findings associated with 
this research in February 2008. The Directorate reported that although 
some of the existing literature supported the possibility of using 
behavioral and physiological cues, the results are not 
methodologically strong enough to support standardized applications in 
an operational setting.[Footnote 39] The preliminary findings also 
noted that it is not known whether behavioral and physiological cues 
linked to deception in planning a hostile action will be the same or 
different as those indicators linked to deception by an individual 
after they have already engaged in a hostile action. However, an S&T 
program director stated that although early literature can be 
characterized as methodologically weak, more recent unpublished 
research sponsored by DHS, the Department of Defense, and the 
intelligence community is promising in that it has demonstrated some 
linkages between behavioral and physiological indicators and 
deception.[Footnote 40] 

In March 2009, the Under Secretary (Acting) for DHS's S&T Directorate 
testified that the Directorate had performed an initial validation of 
the behavior indicators used by BDOs.[Footnote 41] The Under Secretary 
stated that this analysis provided statistically significant support 
that persons demonstrating select behavioral indicators are more 
likely to possess prohibited items and that behaviors can distinguish 
deceptive from nondeceptive individuals. According to S&T, this 
validation was the result of statistical analyses performed by S&T 
using operational data from the SPOT program database. However, we 
identified weaknesses in TSA's process for maintaining these data. For 
example, controls over the SPOT database to help ensure the 
completeness and accuracy of the data were missing. Specifically, the 
SPOT database did not have computerized edit checks built into the 
system to review the format, existence, and reasonableness of data. 
For example, we found that discrepancies existed between the number of 
passengers arrested by local law enforcement at the screening 
checkpoints and the number of screened passengers recorded as 
arrested. In another example, we found that the total number of LEO 
referrals differed from the number of passenger records with 
information on the reasons for LEO referral. Internal control 
standards state that controls should be installed at an application's 
interfaces with other systems to ensure that all inputs are received 
and are valid and that outputs are correct and properly distributed. 
[Footnote 42] TSA officials explained these issues as data anomalies 
and planned to change instructions to staff entering data to reduce 
these problems. Although TSA is taking steps to update the SPOT 
database, which are discussed later in this report, the data used by 
S&T to conduct its preliminary validation of related behaviors lacked 
such controls. In addition, BDOs could not input all behaviors 
observed in the SPOT database because the database limits entry to 
eight behaviors, six signs of deception, and four types of prohibited 
items per passenger referred for additional screening. Because of 
these data-related issues, meaningful analyses could not be conducted 
to determine if there is an association between certain behaviors and 
the likelihood that a person displaying certain behaviors would be 
referred to a LEO or whether any behavior or combination of behaviors 
could be used to distinguish deceptive from nondeceptive individuals. 
As a result, TSA lacks assurance that the SPOT data can be used 
effectively to determine that the person poses a risk to aviation 
security. S&T has recognized weaknesses in the procedures for 
collecting data on passengers screened by SPOT and plans to more 
systematically collect data during its study by, for example, 
requiring BDOs to record more complete and accurate information 
related to a passenger referral immediately following resolution. 

The S&T study is an important step to determine whether SPOT is more 
effective at identifying passengers who may be threats to the aviation 
system than random screening. However, S&T's current research plan is 
not designed to fully validate whether behavior detection and 
appearances can be effectively used to reliably identify individuals 
in an airport terminal environment who pose a risk to the aviation 
system. For example, research on other issues, such as determining the 
number of individuals needed to observe a given number of passengers 
moving at a given rate per day in an airport environment or the 
duration that such observation can be conducted by BDOs before 
observation fatigue affects effectiveness, could provide additional 
information on the extent to which SPOT can be effectively implemented 
in airports. In another example, Dr. Ekman told us that additional 
research could help determine the need for periodic refresher training 
since no research has yet determined whether behavior detection is 
easily forgotten or can be potentially degraded with time or lack of 
use. While S&T officials agree on the need to validate the science of 
behavior detection programs, they told us that some of these other 
issues could be examined in the future but are not part of the current 
plan due to time and budgetary constraints. According to S&T, some 
additional analysis is underway to inform the current BDO selection 
process. This analysis is intended to provide information on the 
knowledge, skills, abilities, and other characteristics of successful 
BDOs. Since the analysis is scheduled for completion in May 2010, it 
remains unclear to what extent the findings will help to validate the 
science related to SPOT. While we recognize the potential benefits of 
these efforts, we believe that an assessment by an independent panel 
of experts of the planned methodology of DHS's study could help DHS 
assess the costs and benefits associated with a more comprehensive 
methodology designed to fully validate the science related to SPOT. 
Our prior work has recommended the use of such independent panels for 
comprehensive, objective reviews of complex issues.[Footnote 43] In 
addition, according to the National Research Council, an independent 
panel could provide an objective assessment of the methodology and 
findings of DHS's study to better ensure that SPOT is based on 
validated science. Thus, an independent panel of experts could help 
DHS develop a comprehensive methodology to determine if the SPOT 
program is based on valid scientific principles that can be 
effectively applied in an airport environment for counterterrorism 
purposes. 

SPOT Was Deployed Nationwide on Basis of Threat, but Without a 
Comprehensive Risk Assessment: 

According to DHS's National Infrastructure Protection Plan (NIPP), 
risk assessments are to be documented, reproducible (so that others 
can verify the results), defensible (technically sound and free of 
significant errors), and complete. The NIPP states that comprehensive 
risk assessments are necessary for determining which assets or systems 
face the highest risk, for prioritizing risk mitigation efforts and 
the allocation of resources, and for effectively measuring how 
security programs reduce risks. For a risk assessment to be considered 
complete, the NIPP states that it must specifically assess threat, 
vulnerability, and consequence;[Footnote 44] after these three 
components have been assessed, they are to be combined to produce a 
risk estimate.[Footnote 45] 

According to TSA, SPOT was deployed to TSA-regulated airports on the 
basis of threat information in TSA's Current Airport Threat Assessment 
list.[Footnote 46] TSA deployed SPOT to 161 of 457 TSA-regulated 
airports. TSA officials told us that this selective deployment creates 
unpredictability for persons seeking to cause harm to the aviation 
system because they would not know which airports had BDO teams and 
because BDOs are occasionally sent out to the smaller airports that do 
not have BDOs on a permanent basis. Although TSA's selective 
deployment of SPOT was based on threat information, TSA did not 
conduct vulnerability and consequence assessments to inform the 
deployment of BDOs. As a result, it could not combine the results to 
conduct a comprehensive risk assessment to inform the deployment of 
BDOs to those airports with the highest risks. 

TSA officials told us that while they have not completed a 
comprehensive risk assessment for airport security, they have prepared 
and are currently reviewing a draft of a comprehensive, scenario-based 
Aviation Modal Risk Assessment--known as the AMRA--which is to serve 
as a comprehensive risk assessment for aviation security.[Footnote 47] 
According to TSA officials, the AMRA is to address all three elements 
of risk for domestic commercial aviation, general aviation, and air 
cargo.[Footnote 48] Although TSA planned to release the AMRA in 
February 2008, it now expects to finalize the AMRA in 2010. According 
to TSA, the AMRA may help provide information for the prioritization 
of BDO deployment within airports, but could not provide specifics on 
how it would do so. Further, TSA officials noted that information from 
AMRA would inform BDO deployment in conjunction with other TSA 
priorities not related to SPOT.[Footnote 49] Since the AMRA is not yet 
complete, it is not clear whether it will provide the risk analysis--
including assessments of vulnerability and consequence--needed to 
inform TSA's decisions and planning for any revisions or future 
deployment of SPOT. If AMRA lacks information relevant to the 
deployment of SPOT and further research determines that SPOT has a 
scientifically validated basis for using behavior detection for 
counterterrorism purposes in the airport environment, then conducting 
a comprehensive risk assessment of airports could strengthen TSA's 
ability to establish priorities and make cost-effective resource 
decisions regarding the deployment of BDOs to those airports deemed to 
have the highest priority risks. 

TSA Deployed SPOT Nationwide Without Conducting a Cost-Benefit 
Analysis but Such an Analysis Could Help Inform Program Decisions 
Moving Forward: 

DHS and other federal guidance recommend conducting a cost-benefit 
analysis before implementing new programs to avoid unnecessary costs 
and identify the best way to achieve goals at the lowest costs among 
potential alternatives. Our prior work has also supported the use of 
cost-benefit analyses during retrospective reviews to validate the 
agency's original assumptions regarding costs and benefits.[Footnote 
50] In addition, the DHS February 2006 Cost-Benefit Analysis Guidebook 
and OMB guidance both recommend the use of cost-benefit analysis, both 
in the planning stage for a program, and when significant milestones 
or financial options are to be assessed.[Footnote 51] The DHS 
Guidebook states that a cost-benefit analysis is designed to identify 
optimal financial solutions among competing alternatives. OMB guidance 
also identifies cost-benefit analysis as one of the key principles to 
be considered when making capital expenditures, that expected benefits 
of proposed actions should be explained, and that a baseline should be 
identified discussing costs and benefits in comparison with clearly 
defined alternatives. DHS's 2006 and 2009 NIPPs also state that 
priority is to be given to those protective measures that provide the 
greatest mitigation of risk for the resources that are available. The 
DHS NIPPs add that effective protective programs seek to use resources 
efficiently by focusing on actions that offer the greatest mitigation 
of risk for any given expenditure. In addition, measuring cost 
effectiveness of SPOT was a key TSA goal in an October 2005 version of 
the SPOT strategic plan. 

Although the DHS and OMB guidance recommend that a cost-benefit 
analysis be conducted prior to deploying a program nationwide--and 
potentially incurring substantial costs--TSA did not conduct such an 
analysis of SPOT to inform its pilot testing prior to full-scale 
nationwide deployment. In early 2003, TSA began conducting a pilot 
test of the SPOT program at Boston Logan airport to better understand 
the benefits of the program. According to Boston Logan's Federal 
Security Director, the primary purpose of this pilot test was to 
understand the potential of the program, not to validate its success. 
[Footnote 52] TSA officials stated that the program had several 
benefits, one of which was its "negligible cost." However, TSA did not 
analyze the pilot test results to determine if SPOT was more cost 
effective than other alternatives, such as random screening of 
passengers. In October 2004, TSA implemented additional pilot programs 
in Providence, Rhode Island and Portland, Maine with the goal of 
providing Federal Security Directors with an additional layer of 
security to identify high-risk passengers for additional screening 
using behavior detection techniques. TSA concluded that the pilot 
program was successful and cited several security benefits of these 
pilots. For example, TSA personnel in Providence identified two 
individuals in possession of illegal drugs, who were then arrested. 
Law enforcement also arrested another individual referred to them for 
providing a fraudulent passport. In another example, BDOs in Portland 
discovered a passenger with multiple passports and a hidden luggage 
compartment. The passenger was interviewed by LEOs and later released. 

TSA determined that these initial pilot tests at three airports were 
successful without comparing pilot test data to other possible 
security alternatives. For example, the results of random screening of 
passengers at the pilot airports could have provided TSA with 
objective baseline data. Specifically, these data could have been 
compared to data collected during the SPOT pilots to determine if SPOT 
was more effective than random screening in detecting passengers who 
pose a potential risk to aviation security. TSA concluded that the 
pilot tests were successful because pilot airports were able to easily 
incorporate SPOT into their security program, train personnel in SPOT, 
and implement procedures for an additional layer of security according 
to TSA. 

TSA conducted additional pilot tests at the Minneapolis-St. Paul, 
Minnesota and Bangor, Maine airports in October 2005. TSA also 
deployed the program to nine additional airports in response to TSA's 
holiday preparedness plan in December 2005 to further operationally 
test the program. Senior SPOT program officials explained that TSA did 
not conduct an analysis of the pilot testing because the program was 
in its infancy and officials were focused on deploying SPOT to 
additional airports. Since that time, TSA has not conducted a cost-
benefit analysis, which could help the agency establish the value of 
the program relative to other layers of aviation security. Moreover, a 
cost-benefit analysis could also be useful considering recent program 
growth. For example, from fiscal year 2007 through fiscal year 2009, 
TSA allotted about $383 million for SPOT. During this period, SPOT's 
share of TSA's total screening operations budget increased from 1 
percent to 5 percent.[Footnote 53] The conference report accompanying 
the fiscal year 2010 DHS appropriations act designates $212 million of 
the appropriated aviation security funding for the SPOT program. 
[Footnote 54] A cost-benefit analysis could have provided TSA 
management with analysis on whether this allocation was a prudent 
investment, as well as whether this level of investment in SPOT is 
appropriate. Figure 3 shows the growth in the budget and personnel 
numbers for SPOT from fiscal years 2007 through 2010. 

Figure 3: Budget and Personnel Growth in the SPOT Program, Fiscal 
Years 2007 through 2010: 

[Refer to PDF for image: 2 vertical bar graphs] 

Fiscal year: 2007; 
Total actual cost: $14 million; 
Actual BDO allocation: 589. 

Fiscal year: 2008; 
Total actual cost: $144 million; 
Actual BDO allocation: 2,011. 

Fiscal year: 2009; 
Total actual cost: $198 million; 
Actual BDO allocation: 2,860. 

Fiscal year: 2010; 
Total appropriated: $212 million; 
Total BDO allocation: 2,986. 

Source: GAO analysis of TSA data. 

Note: The actual BDO allocation for fiscal year 2009 is as of June 
2009. The appropriated amount for SPOT for fiscal year 2010 is the 
amount reflected in the conference report accompanying the fiscal year 
2010 DHS appropriations act. The appropriated amounts prior to fiscal 
year 2010 cannot be determined because funding was appropriated as a 
lump sum with funding for other screeners and the relevant conference 
reports did not allocate a specific amount for SPOT. BDO allocation 
figures are full-time equivalents. 

[End of figure] 

SPOT's Strategic Plan Could be Strengthened by Addressing Key 
Characteristics of a Successful Strategy: 

Our previous work,[Footnote 55] and the Government Performance and 
Results Act,[Footnote 56] set forth several key elements of a 
strategic plan. Such plans can guide agencies in planning and 
implementing an effective government program. Table 1 summarizes the 
desirable characteristics of an effective strategic plan, as 
identified in our prior work. In April 2009, we reported that these 
characteristics are the starting point for developing a strategic 
plan.[Footnote 57] 

Table 1: Summary of Desirable Characteristics for Developing a 
Strategic Plan: 

Desirable characteristic: Purpose, scope, and methodology; 
Description: Addresses why the plan was produced, the scope of its 
coverage, and the process by which it was developed. 

Desirable characteristic: Problem definition and risk assessment; 
Description: Addresses the particular problems and threats the plan is 
directed towards. 

Desirable characteristic: Goals, subordinate objectives, activities, 
and performance measures; 
Description: Addresses what the plan is trying to achieve, steps to 
achieve those results, as well as the priorities, milestones, and 
performance measures to gauge results. 

Desirable characteristic: Resources, investments, and risk management; 
Description: Addresses what the plan will cost, the sources and types 
of resources and investments needed, and where resources and 
investments should be targeted based on balancing risk reductions with 
cost. 

Desirable characteristic: Organizational roles, responsibilities, and 
coordination; 
Description: Addresses who will implement the plan, what their roles 
will be compared to others, and mechanisms for them to coordinate 
their efforts. 

Desirable characteristic: Integration and implementation; 
Description: Addresses how the plan relates to the agency's other 
goals, objectives, and activities, to other federal and nonfederal 
entities involved in implementation or coordination, and their plans 
to implement the strategic plan. 

Source: GAO analysis based on GAO-09-369 and GAO-04-408T. 

[End of table] 

TSA officials at Boston Logan airport told us that they completed the 
first strategic plan for SPOT in 2006. The strategic plan was last 
updated in March 2007. The March 2007 plan includes some of the 
desirable characteristics described above, such as an overall purpose. 
However, incorporating additional characteristics of an effective 
strategic plan could enhance the plan's usefulness in program 
management and resource allocation decisions to effectively manage the 
deployment of SPOT if TSA determines that the program has a 
scientifically valid basis. TSA officials stated that they believed 
the plan was sufficiently comprehensive to develop a national program, 
such as SPOT. However, these officials told us that the plan was not 
updated after TSA expanded the program in 2008 and 2009. They also 
stated that the program's focus remained on deploying SPOT to 
additional airports. Our assessment of the extent to which the SPOT 
strategic plan addresses these characteristics is presented below. 

Purpose, scope, and methodology: The SPOT strategic plan addresses why 
the plan was developed (i.e., purpose) and the scope of its coverage. 
Specifically, the plan describes a strategy to utilize behavior 
detection screening as an additional layer of security. The plan also 
notes that the primary focus is to expand SPOT in the aviation 
environment while also developing a capability to deploy BDOs to 
support security efforts in all modes of transportation. However, the 
plan does not discuss the process by which it was developed (i.e., 
methodology). According to TSA, officials responsible for developing 
the plan received input from relevant stakeholders at Boston Logan 
airport and TSA headquarters. We believe incorporating the methodology 
into the plan could make the document more useful to TSA and other 
organizations, such as local law enforcement, responsible for 
implementing the plan. 

Problem definition and risk assessment: The plan addresses the 
particular threat it is directed towards. Specifically, the plan 
describes the need to implement SPOT to counter terrorist activities, 
improve security, and incorporate additional layers of protection 
within aviation security. However, the plan does not incorporate risk 
assessment information to identify priorities or guide program 
implementation because TSA has not conducted a comprehensive risk 
assessment related to the deployment of SPOT.[Footnote 58] Using 
available risk assessment information to inform the development of a 
strategic plan would help ensure that clear priorities are established 
and focused on the areas of greatest need. Specifically, incorporating 
the results of a risk assessment in the program's strategic plan could 
help inform TSA's decisions such as whether to deploy SPOT to 
additional TSA-regulated airports, to shift SPOT teams from one 
airport to another, or to remove SPOT at airports where the benefit of 
addressing the risk does not outweigh the costs, as well as to 
identify and communicate the risks to aviation security if SPOT was 
not deployed to all TSA-regulated airports. 

Goals, subordinate objectives, activities, and performance measures: 
The plan outlines several goals, objectives, and activities for the 
SPOT program to achieve. For example, the plan outlines a goal to 
develop multimodal partnerships, including at the local level, to 
support SPOT. An associated objective for this goal includes 
identifying and fostering advocates within each mode of transportation 
by developing transportation, intelligence, and law enforcement 
working groups with relevant officials to share information and foster 
cooperation. The plan also includes a goal to develop and implement 
performance measures for SPOT. However, the plan did not include 
performance measures for SPOT. Incorporating performance measures into 
the plan could help TSA officials measure progress in implementing the 
plan's goals, objectives, and activities. 

Resources, investments, and risk management: The plan does not 
identify the costs and resources needed to achieve program objectives 
discussed in the plan. Incorporating information about cost and 
resources would facilitate TSA's ability to allocate resources across 
programs according to priorities and constraints, track costs and 
performance, and shift such investments and resources as appropriate. 

Organizational roles, responsibilities, and coordination: The SPOT 
program relies on a close partnership with law enforcement officers at 
airports. TSA provides briefings to law enforcement on the SPOT 
program, and TSA officials conduct outreach efforts to local law 
enforcement as needed. The SPOT SOP guidance and SPOT training include 
guidance about ensuring that LEOs receive complete and accurate 
information about each SPOT referral. However, while the strategic 
plan identifies TSA officials and offices as responsible parties for 
implementing the strategic plan, it does not provide guidance on how 
to effectively link the roles, responsibilities, and capabilities of 
federal, state, and local officials providing program support. 
Moreover, although SPOT SOP guidance discusses the need for BDOs to 
coordinate with other TSA personnel, such as TSOs and TDCs, TSA does 
not identify their roles and responsibilities in regards to the SPOT 
program in the program's strategic plan. Integrating these elements 
into the strategic plan could help to clarify the relationships 
between these various implementing parties, which would thereby 
increase accountability and improve the effectiveness of 
implementation. 

Integration and implementation: The SPOT strategic plan does not 
discuss how its scope complements, expands upon, or overlaps with 
other related strategic documents. For example, TSA's April 2008 
Office of Security Operations Organizational Business Plan for Fiscal 
Year 2010 describes how its goals--including those for SPOT--relate to 
DHS and TSA strategic goals.[Footnote 59] However, TSA does not link 
goals in the SPOT strategic plan with other related strategic 
documents, such as the Aviation Implementation Plan of DHS's 
Transportation Systems Sector-Specific Plan[Footnote 60] and the 
Passenger Checkpoint Screening Program Strategic Plan.[Footnote 61] By 
linking goals in its SPOT strategic plan to other TSA efforts, TSA 
could better ensure that the program's objectives are integrated with 
other TSA security programs and that resources are used effectively by 
minimizing any unnecessary duplication with these other actions. 

More Fully and Consistently Utilizing Available Information Technology 
Could Enhance TSA's Ability to Identify Threats to the Aviation System: 

Inconsistencies in the use of available information technology to aid 
in the collection and recording of data on passengers by BDOs during 
referrals to LEOs, lack of guidance on, or a mechanism for, BDOs to 
request the TSA's Transportation Security Operations Center to run the 
names of passengers exhibiting suspicious behaviors against law 
enforcement and intelligence databases, and the Center's not checking 
all of the databases available to it--have limited TSA's ability to 
identify potential terrorist threats to the aviation system.[Footnote 
62] Among other information, these databases include terrorism-related 
watch lists. 

Systematic Collection of Data on Passengers Identified Through the 
SPOT Program Could be Improved to Better Identify Activity Potentially 
Harmful to the Aviation System: 

TSA is not fully utilizing the resources it has available to 
systematically collect the information obtained by BDOs on passengers 
whose behaviors and appearances resulted in either SPOT referral 
screening, or in a referral to LEOs, and who thus may pose a risk to 
the aviation system. TSA's July 2008 Privacy Impact Assessment on the 
TSA Transportation Security Operations Center, and its August 2008 
Privacy Impact Assessment on SPOT, state that information may be 
obtained by BDOs to check an individual's identity against 
intelligence, terrorist, and law enforcement databases and to permit 
intelligence analysts to conduct trend analysis.[Footnote 63] 

The August 2008 SPOT Privacy Impact Assessment states that information 
about a passenger who has exceeded the SPOT behavior threshold, 
leading to LEO referral, may be collected and entered into DHS's 
Transportation Information Sharing System.[Footnote 64] According to 
the SPOT Privacy Impact Assessment, information collected may be 
submitted to the Transportation Information Sharing System database 
for analysis, and, through it to other linked intelligence databases 
and the intelligence analysts who study them, to detect, deter, and 
defeat a criminal or terrorist act in the transportation domain before 
it occurs. The SPOT Privacy Impact Assessment notes that terrorist 
acts that threaten transportation security are most vulnerable in the 
planning stages and that the timely passage of SPOT referral 
information may assist in identifying such efforts before they become 
operational. A June 2008 Transportation Information Sharing System 
Privacy Impact Assessment similarly states that one goal is to use the 
system data to find trends and patterns that may indicate 
preoperational terrorist or criminal activity--that is, to "connect 
the dots" about a planned terrorist attack or criminal enterprise. 
Information in TSA's Transportation Information Sharing System is 
primarily activity or behavioral information but may also contain 
personal information regarding the individuals identified by the BDO 
through SPOT. According to TSA, BDOs do not analyze the data obtained 
during referrals; if they have the appropriate training, they may 
enter the data by computer into the Transportation Information Sharing 
System, where they can be analyzed by intelligence analysts. Other 
appropriately trained and officially designated TSA officials, such as 
Federal Security Directors, may also enter data into the system. 

According to TSA, a 2008 pilot program it conducted that involved BDOs 
entering data into the Transportation Information Sharing System 
database was "invaluable," in part because over 40 referrals have 
since been passed on to other LEO organizations for further 
investigation, most of which came from BDO input. A February 2006 TSA 
memorandum describes the Transportation Information Sharing System as 
"a critical element in the success of SPOT" because it provides the 
necessary platform for the reporting of information obtained as a 
result of SPOT referrals. TSA noted that through the use of the 
Transportation Information Sharing System, two different BDO teams had 
separately identified and selected the "same extremist" for secondary 
questioning.[Footnote 65] TSA officials also told us about an incident 
in which an individual sought to board an aircraft with a handgun on 
two separate occasions, at two different airports. Although the 
handgun was detected both times, the individual was released after 
providing what seemed to be a credible explanation. After the second 
incident, however, intelligence analysts who reviewed the system 
information saw that this individual had tried twice in 2 weeks to 
bring a weapon onto an aircraft. A LEO was dispatched to the person's 
home, and an arrest was made. Without the data inputted into the 
system both times, no pattern would have been detected by the 
analysts, according to TSA. Although the pilot program illustrated the 
benefits of BDOs entering data into the system, access to the system 
was not expanded to all SPOT airports in 2008 or 2009. 

Internal control standards call for management to develop policies, 
procedures, and techniques to help enforce management directives. TSA 
does not provide official guidance on how or when BDOs or other TSA 
personnel should enter data into the Transportation Information 
Sharing System or which data should be entered. Official guidance on 
what data should be entered into the system on passengers could better 
position TSA personnel to be able to consistently collect information 
to facilitate synthesis and analysis in "connecting the dots" with 
regard to persons who may pose a threat to the aviation system. 

On March 18, 2010, TSA officials told us that TSA recognizes the value 
of recording SPOT incidents for the purposes of intelligence 
gathering. As a result, TSA decided that certain data would be entered 
into the Transportation Information Sharing System, and would, in 
turn, be analyzed as a way to potentially "connect the dots" with 
other transportation security incidents.[Footnote 66] 

TSA officials said that the Federal Security Director at each SPOT 
airport has been given the discretion to decide which personnel should 
have access to the Transportation Information Sharing System. However, 
TSA has not developed a plan detailing how many personnel would have 
access to the system, or when they would have access at SPOT airports. 
TSA officials said that training is currently being provided to 
personnel responsible for using the system at all SPOT airports 
although they did not provide information on the number being trained. 

Standard practices for defining, designing, and executing programs 
include developing a road map, or program plan, to establish an order 
for executing specific projects needed to obtain defined programmatic 
results within a specified time frame. However, TSA stated that it has 
not developed a schedule or milestones by which database access will 
be deployed to SPOT airports, or a date by which access at all SPOT 
airports will be completed. Setting milestones for expanding 
Transportation Information Sharing System access to all SPOT airports, 
and setting a date by which the expansion will be completed, could 
better position TSA to identify threats to the aviation system that 
may otherwise go undetected and help TSA track its progress in 
expanding Transportation Information Sharing System access as 
management intended. 

Guidance on and a Mechanism for Running Names of Referred Passengers 
Through the Databases Available to the Transportation Security 
Operations Center Could Help Improve SPOT Practices: 

Internal control standards state that policies, procedures, 
techniques, and other mechanisms are essential to help ensure that 
actions are taken to address program risks.[Footnote 67] The current 
process makes the BDOs dependent on the LEOs with regard to the 
timeliness that LEOs respond to BDO calls for service, as well as with 
regard to whether the LEOs choose to question the passengers referred 
to them or conduct a background check. Our analysis of the SPOT 
referral database found a wide variation in the percent of times that 
LEOs responded to calls for service at SPOT airports.[Footnote 68] 
Moreover, if a local LEO decides to run a background check on a 
passenger referred to them, they would be accessing the FBI's NCIC and 
not other intelligence and law enforcement databases. 

Although LEOs may not always respond to calls for service, question 
passengers, or check passenger names against databases available to 
TSA, TSA has not developed a mechanism allowing BDOs to send 
information to the Transportation Security Operations Center about 
passengers whose behavior indicates that they may be a possible threat 
to aviation security. According to TSA's July 2008 Transportation 
Security Operations Center Privacy Impact Assessment, passenger 
information may be submitted to the Transportation Security Operations 
Center to ascertain, as quickly as possible, the individual's 
identity, whether they are already the subject of a terrorist or 
criminal investigation, or to analyze suspicious behavior that may 
signal some form of preoperational surveillance or activity.[Footnote 
69] 

Our survey of Federal Security Directors at SPOT airports found a 
notable inconsistency in the rates at which BDOs at different airports 
contacted the Transportation Security Operations Center.[Footnote 70] 
Developing additional guidance in the SPOT operating procedures could 
help improve consistency in the extent to which BDOs utilize 
Transportation Security Operations Center resources. Given the range 
of responses we received from SPOT airports about whether the BDOs 
contact the Transportation Security Operations Center to verify 
passenger identities and run their names against terrorist and 
intelligence databases and the inconsistencies identified related to 
LEO responses to BDO requests for service, developing a standard 
mechanism and providing BDOs with additional guidance could help TSA 
achieve greater consistency in the SPOT process. Such a mechanism 
would provide designated TSA officials with a means of verifying 
passenger identities and help them determine whether a passenger was 
the subject of a terrorist or criminal investigation and thus posed a 
risk to aviation security. 

Standards for internal control state that effectively using available 
resources, including key information databases, is one element of 
functioning internal controls.[Footnote 71] In this connection, it is 
widely recognized among intelligence entities and police forces that a 
capability to "run" names against databases that contain criminal and 
other records is a potentially powerful tool to both identify those 
with outstanding warrants and to help discover an ongoing criminal or 
security-related incident. Additionally, TSA recommended in an April 
2008 Organizational Business plan for its Office of Security 
Operations that the SPOT program should establish a mechanism and 
policy for allowing real-time checks of federal records for 
individuals whose behavior indicates they may be a threat to security. 
[Footnote 72] The Office of Security Operations plan also states that 
BDOs should communicate the data to U.S. intelligence centers, with 
the purpose of permitting rapid communication of this information to 
local LEOs to take action. However, TSA officials told us that because 
of safety concerns, the Transportation Security Operations Center does 
not provide information from database checks directly to BDOs because 
BDOs are not LEOs, are unarmed, and do not have the training needed to 
deal with potentially violent persons.[Footnote 73] If the mechanism 
discussed in the Office of Security Operations business plan were 
implemented, it would allow the Transportation Security Operations 
Center to use BDO information to conduct real-time record checks of 
passengers and communicate the results to LEOs for action. Such a 
mechanism could increase the chances to detect ongoing criminal or 
terror plans. 

TSA's Transportation Security Operations Center Does Not Use All 
Database Resources When Contacted: 

The final report of the National Commission on Terrorist Attacks Upon 
the United States (the "9/11 Commission Report") recommends that in 
carrying out its goal of protecting aviation, TSA should utilize the 
larger set of information maintained by the federal government, that 
is, the entire Terrorist Screening Database--the U.S. government's 
consolidated watch list that contains information on known or 
suspected international and domestic terrorists--as well as other 
government databases, such as intelligence or law enforcement 
databases.[Footnote 74] However, the Transportation Security 
Operations Center is not using all the resources at its disposal to 
support BDOs in verifying potential risks to the aviation system. This 
reduces the opportunities to "connect the dots" that would increase 
the chances of detecting terrorist attacks in their planning stage, 
which the SPOT Privacy Impact Assessment states is when they are the 
most vulnerable. 

According to TSA, the Transportation Security Operations Center has 
access to multiple law enforcement and intelligence databases that can 
be used to verify the identity of airline passengers; these include 
among others:[Footnote 75] 

1. the Selectee list, which identifies persons who must undergo 
enhanced screening at the checkpoint prior to boarding; 

2. the No-Fly list,[Footnote 76] which lists persons prohibited from 
boarding aircraft; and: 

3. the Terrorist Identity Datamark Environment terrorist list. 
[Footnote 77] 

TSA stated that the Transportation Security Operations Center checks 
passenger names submitted to it against these three databases if the 
passenger has been referred by a BDO to a LEO, but has not been 
arrested. Of the three databases that the Transportation Security 
Operations Center is to check in the case of a referral, passengers 
would have already been screened against two--the Selectee and No-Fly 
lists--in accordance with TSA passenger prescreening procedures when 
purchasing a ticket. The third database checked--the Terrorist 
Identity Datamark Environment--tracks terrorists but not persons 
wanted for other crimes. The FBI's NCIC information system would 
contain names of such persons, but is not among the three databases 
checked for nonarrest referrals. If the passenger has been arrested, 
the Transportation Security Operations Center will run the passenger's 
name against the additional law enforcement and intelligence databases 
available to it. 

In addition, TSA told us that the Operations Center does not have 
direct electronic access to the Terrorist Screening Database and must 
call the FBI's Terrorist Screening Center to provide it with a name to 
verify. TSA stated that this is done if a passenger's identity could 
not be verified using the Operations Center databases. In effect, if a 
passenger has been referred to a LEO, but not arrested, the Operations 
Center is to check the three databases shown above to verify the 
passenger's identity. If a passenger has been arrested, but the three 
databases do not list the person, the Center can check the additional 
databases available to it. If none of these databases can verify the 
person's identity, the Operations Center can contact the Terrorist 
Screening Center by telephone to request further screening. 

For passengers who have risen to the level of a LEO referral at an 
airport checkpoint, having the Transportation Security Operations 
Center consistently check their names against all the databases 
available to it could potentially help TSA identify threats to the 
aviation system and aid in "connecting the dots." TSA indicated that 
there are no obstacles to rapidly checking all databases rather than 
the three listed. We did not analyze the extent to which the law 
enforcement and intelligence databases available to TSA may contain 
overlapping information. 

TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps 
to Improve Evaluation Capabilities: 

TSA has established some performance measures by tracking SPOT 
referral and arrest data, but lacks the measures needed to evaluate 
the effectiveness of the SPOT program and, as a result, has not been 
able to fully assess SPOT's contribution to improving aviation 
security. TSA emphasized the difficulty of developing performance 
measures for deterrence-based programs, but stated that it is 
developing additional measures to quantify the effectiveness of the 
program. The SPOT program uses teams to assess BDO proficiency, 
provide individual and team guidance, and address issues related to 
the interaction of BDOs with TSA checkpoint personnel. However, TSA 
does not systematically track the teams' recommendations or the 
frequency of the teams' airport visits. TSA states that it is working 
to address these issues and plans to do so by the end of fiscal year 
2010. 

TSA Has Taken Action to Collect Data for Some Performance Measures, 
but Work Remains to Assess Progress Towards Achieving Strategic Goals: 

TSA agreed that the SPOT program lacked sufficient performance 
measures in the past, but stated that it has some performance measures 
in place including tracking data on passengers referred for additional 
screening and the resolution of this screening, such as if prohibited 
items were found or if law enforcement arrested the passenger and the 
reason for the arrest. TSA is also working to improve its evaluation 
capabilities to better assess the effectiveness of the program. DHS's 
NIPP, internal controls standards, and our previous work on program 
assessment state that performance metrics and associated program 
evaluations are needed to determine if a program works and to identify 
adjustments that may improve its results.[Footnote 78] Moreover, 
standard practices in program management for defining, designing, and 
executing programs include developing a road map, or program plan, to 
establish an order for executing specific projects needed to obtain 
defined programmatic results within a specified time frame.[Footnote 
79] Congress also needs information on whether and in what respects a 
program is working well or poorly to support its oversight of agencies 
and their budgets; and agencies' stakeholders need performance 
information to accurately judge program effectiveness.[Footnote 80] 
For example, in the Senate Appropriations Committee report 
accompanying the fiscal year 2010 DHS appropriations bill,[Footnote 
81] the committee noted that while TSA has dramatically increased the 
size and scope of SPOT, resources were not tied to specific program 
goals and objectives. In addition, the conference report accompanying 
the fiscal year 2010 DHS appropriations act requires TSA to report to 
Congress, within 60 days of enactment, on the effectiveness of the 
program in meeting its goals and objectives, among other things. 
[Footnote 82] This report was completed on March 15, 2010. 

Although TSA tracks data related to SPOT activities including 
prohibited items, law enforcement arrests related to SPOT referrals, 
and reasons for the arrests (output measures), it has not yet 
developed measures to gauge SPOT's effectiveness in meeting TSA 
strategic goals (outcome measures), such as identifying individuals 
who may pose a threat to the transportation system.[Footnote 83] OMB 
encourages the use of outcome measures because they are more 
meaningful than output measures, which tend to be more process-
oriented or means to an end.[Footnote 84] For example, TSA's Assistant 
General Manager for the Office of Operation Process and Performance 
Metrics[Footnote 85] told us that SPOT staffing levels are currently 
used as one performance metric. The official said that since the SPOT 
is an added layer of security, additional SPOT staffing would add to 
security effectiveness. While staffing levels may help gauge how fast 
the program is growing, they do not measure the effectiveness in 
meeting strategic goals. 

Similarly, TSA also cited the number of prohibited items discovered by 
BDOs in SPOT metrics reports as a measure of program success.[Footnote 
86] However, TSA told us that possession of a prohibited item is often 
an oversight and not an intentional act; moreover, other checkpoint 
screening layers are intended to find such items, such as the TSOs and 
the property screening equipment.[Footnote 87] TSA also cited measures 
of BDO job performance as some of the existing measures of program 
effectiveness, but noted that these are "pass/fail" assessments of 
individual BDOs, rather than overall program measures. 

TSA notes that one purpose of the SPOT program is to deter terrorists, 
but that proving that it has succeeded at deterring terrorists is 
difficult because the lack of data has presented challenges for the 
SPOT program office when developing performance measures. We agree 
that developing performance measures, especially outcome measures, for 
programs with a deterrent focus is difficult. Nevertheless, such 
measures are an important tool to communicate what a program has 
accomplished and provide information for budget decisions. TSA uses 
proxy measures--indirect measures or indicators that approximate or 
represent the direct measure--to address deterrence, other security 
goals, or a combination of both. For example, TSA tracks the number of 
prohibited items found and individuals arrested as a result of SPOT 
referrals. According to OMB, proxy measures are to be correlated to an 
improved security outcome, and the program should be able to 
demonstrate--such as through the use of modeling--how the proxies tie 
to the eventual outcome.[Footnote 88]In using a variety of proxy 
measures, failure in any one of the identified measures could provide 
an indication on the overall risk to security. However, developing a 
plan that includes objectives, milestones, and time frames to develop 
outcome-based performance measures could better position TSA in 
assessing the effectiveness of the SPOT program. 

With regard to more readily quantifiable output performance measures, 
such as the number of referrals by BDOs, or the ratio of arrests to 
referrals, TSA was limited in its ability to analyze the data related 
to these measures. The SPOT database includes information on all 
passengers referred by BDOs for additional SPOT screening including 
the behaviors of the passengers that led to the additional screening, 
as well as the resolution of the screening process (e.g., no further 
action taken, law enforcement notification, law enforcement 
investigation, arrested, and reason for arrest). However, TSA reported 
that any analysis of the data had to be done manually.[Footnote 89] 

In March 2010, TSA migrated the SPOT referral data to its Performance 
Management Information System, allowing for more statistical and other 
analyses. According to TSA, migrating the SPOT referral database will 
enhance the SPOT program's analytic capabilities. For example, TSA 
stated that it would be able to conduct trend analyses, better 
segregate data, and create specific reports for certain data. This 
includes better tracking of performance data at specific airports, 
analyzing by categories of airports (threat or geographic location), 
and tracking the performance data of individual BDOs, such as number 
of referrals, number of arrests, arrest to referral ratios, and other 
analyses. However, since these changes to the database were not 
complete at the time of our audit, we could not assess whether the 
problems we identified with the database had been corrected. 

Over 4 Years, SPOT Resulted in About 1,100 Arrests Out of Almost 
14,000 Referrals to Law Enforcement: 

The SPOT referral database records the total number of SPOT referrals 
since May 29, 2004, how many were resolved, how many passengers BDOs 
referred to LEOs, the recorded reasons for the referral, and how many 
referrals led to arrests, among other things. As shown in figure 4, we 
analyzed the SPOT referral data for the period May 29, 2004, to August 
31, 2008. 

Figure 4: Passenger Boardings at SPOT Airports, May 29, 2004, through 
August 31, 2008: 

[Refer to PDF for image: illustration] 

* Approximately 2 billion passengers boarded aircraft at SPOT airports: 

* Approximately 152,000 SPOT secondary referrals: 

* Approximately 14,000 passengers referred to LEOs: 

* Approximately 1,100 SPOT related arrests: 

Note: Figure 4 is not drawn to scale. 

[End of figure] 

Figure 4 shows that approximately 2 billion passengers boarded 
aircraft at SPOT airports from May 29, 2004, through August 31, 2008. 
[Footnote 90] Of these, 151,943 (less than 1/100TH of 1 percent) were 
sent to SPOT referral screening, and of these, 14,104 (9.3 percent) 
were then referred to LEOs. These LEO referrals resulted in 1,083 
arrests, or 7.6 percent of those referred, and less than 1 percent of 
all SPOT referrals (0.7 percent of 151,943). 

We also analyzed the reasons for arrests resulting from SPOT 
referrals, for the May 29, 2004, through August 31, 2008, period. 
Table 2 shows, in descending order, the reasons for the arrests. 

Table 2: Reasons for Arrests from SPOT Referrals, May 29, 2004 through 
August 31, 2008: 

Reasons for arrest: Illegal alien; 
Number: 427; 
Percentage: 39%. 

Reasons for arrest: Outstanding warrants; 
Number: 209; 
Percentage: 19%. 

Reasons for arrest: Possession of fraudulent documents; 
Number: 166; 
Percentage: 15%. 

Reasons for arrest: Other; 
Number: 128; 
Percentage: 12%. 

Reasons for arrest: Possession of suspected drugs; 
Number: 125; 
Percentage: 12%. 

Reasons for arrest: No reason given; 
Number: 16; 
Percentage: 1%. 

Reasons for arrest: Undeclared currency; 
Number: 8; 
Percentage: 1%. 

Reasons for arrest: Suspect documents; 
Number: 4; 
Percentage: 0. 

Reasons for arrest: Total; 
Number: 1,083; 
Percentage: 99%[A]. 

Source: TSA, SPOT referral database from period of May 29, 2004, 
through August 31, 2008. 

[A] Total does not add to 100 percent due to rounding. 

[End of table] 

While SPOT personnel did not determine a specific reason for arrest 
for 128 cases categorized as "other" or 16 other cases categorized as 
"no reason given," our analysis of the SPOT database found that a 
specific reason for arrest could have been determined for these cases 
by using the LEO resolution notes included in the database. For 
example, we identified 43 additional arrests related to fraudulent 
documents, illegal aliens, and suspect documents, among others. The 
remaining 101 arrests originally characterized as "other" or "no 
reason given" included arrests for reasons such as intoxication, 
unruly behavior, theft, domestic violence, and possession of 
prohibited items. Many of the arrests resulting from BDO referrals 
would typically fall under the jurisdiction of various local, state, 
and federal agencies and are not directly related to threats to 
aviation security. For example, the 427 individuals arrested as 
illegal aliens, and the 166 arrested for possession of fraudulent 
documents, are subject to the enforcement responsibilities shared by 
U.S. Immigration and Customs Enforcement (ICE) and CBP. Although 
outstanding warrants and the possession of fraudulent or suspect 
documents could be associated with a terrorist threat, TSA officials 
did not identify any direct links to terrorism or any threat to the 
aviation system in any of these cases. 

According to TSA, anecdotal examples of BDO actions at airports show 
the value added by SPOT to securing the aviation system. However, 
because the SPOT program has not been scientifically validated, it 
cannot be determined if the anecdotal results cited by TSA were better 
than if passengers had been pulled aside at random, rather than as a 
consequence of being identified for further screening by BDOs. Some of 
the incidents cited by TSA include the following. 

* A BDO referred two passengers who were traveling together to 
referral screening due to suspicious behavior. During secondary 
screening, one passenger presented fraudulent travel documents. The 
other could not produce any documentation of his citizenship and it 
was determined he was in the United States illegally. ICE responded 
and interviewed both passengers. ICE stated one passenger was also in 
possession of $10,000 dollars which alarmed positive for narcotics 
when swept by a K-9 team. ICE arrested one passenger on a federal 
charge of possession of fraudulent identification documents and entry 
without inspection. ICE stated charges are still pending for the 
possession of $10,000. The second passenger was charged with a federal 
charge of entry without inspection. 

* A BDO referred a passenger to referral screening for exhibiting 
suspicious behavior. Port Authority of Portland (Oregon) Police 
responded and interviewed the passenger who did not give a statement. 
LEOs conducted an NCIC check which revealed that there was an 
outstanding warrant for the failure to appear for a theft charge. LEOs 
arrested the passenger on a state charge for an outstanding warrant 
for the failure to appear for theft. 

* A BDO referred a passenger for referral screening due to suspicious 
behavior. During the referral, the passenger admitted that he was 
unlawfully present in the United States. The Orlando (Florida) Police 
Department and CBP responded and interviewed the passenger who stated 
he had $100,000 in his checked baggage, which was confirmed by CBP. 
The passenger was arrested on a federal charge of illegal entry. 

Because these are anecdotal examples, they cannot be used to reliably 
generalize about the SPOT program's overall effectiveness or success 
rate. Our analysis of the SPOT referral database found that the 
referral data do not indicate if any of the passengers sent to 
referral screening, or those arrested by LEOs after being referred to 
them, intended to harm the aircraft, its passengers, or other 
components of the aviation system. Additionally, SPOT officials told 
us that it is not known if the SPOT program has ever resulted in the 
arrest of anyone who is a terrorist, or who was planning to engage in 
terrorist-related activity. 

Reviewing Airport Video Recordings of Individuals Later Arrested or 
Who Pleaded Guilty for Engaging in Terrorism-Related Activities Could 
Help TSA Better Identify Terrorist-Related Behaviors: 

Studying airport video recordings of the behaviors exhibited by 
persons waiting in line and moving through airport checkpoints and who 
were later charged with or pleaded guilty to terrorism-related 
offenses could provide insights about behaviors that may be common 
among terrorists or could demonstrate that terrorists do not generally 
display any identifying behaviors. TSA officials agreed that examining 
video recordings of individuals who were later charged with or pleaded 
guilty to terrorism-related offenses, as they used the aviation system 
to travel to overseas locations allegedly to receive terrorist 
training or to execute attacks, may help inform the SPOT program's 
identification of behavioral indicators. In addition, such images 
could help determine if BDOs are looking for the right behaviors or 
seeing the behaviors they have been trained to observe. 

Using CBP and Department of Justice information, we examined the 
travel of key individuals allegedly involved in six terrorist plots 
that have been uncovered by law enforcement agencies.[Footnote 91] We 
determined that at least 16 of the individuals allegedly involved in 
these plots moved through 8 different airports where the SPOT program 
had been implemented.[Footnote 92] Six of the 8 airports were among 
the 10 highest risk airports, as rated by TSA in its Current Airport 
Threat Assessment. In total, these individuals moved through SPOT 
airports on at least 23 different occasions. For example, according to 
Department of Justice documents, in December 2007 an individual who 
later pleaded guilty to providing material support to Somali 
terrorists boarded a plane at the Minneapolis-Saint Paul International 
Airport en route to Somalia to join terrorists there and engage in 
jihad. Similarly, in August 2008 an individual who later pleaded 
guilty to providing material support to Al-Qaeda boarded a plane at 
Newark Liberty International Airport en route to Pakistan to receive 
terrorist training to support his efforts to attack the New York 
subway system. 

Our survey of Federal Security Directors at 161 SPOT airports 
indicated most checkpoints at SPOT airports have surveillance cameras 
installed. As we previously reported, best practices for project 
management call for conducting feasibility studies to assess issues 
related to technical and economic feasibility, among other things. 
[Footnote 93] In addition, Standards for Internal Control state that 
effectively using available resources is one element of functioning 
internal controls.[Footnote 94] TSA may be able to utilize the 
installed video infrastructure at the nation's airports to study the 
behavior of persons who were later charged with or pleaded guilty to 
terrorism-related offenses, and determine whether BDOs saw the 
behaviors. The Director of Special Operations in TSA's Office of 
Inspection told us that video recordings could be used as a teaching 
tool to show the BDOs which behaviors or activities they did or did 
not observe. In addition, TSA indicated that although the airports may 
have cameras at the security screening checkpoints, the cameras are 
not owned by TSA, and in many cases, they are not accessible to TSA. 
However, TSA officials lack information on the scope of these 
potential limitations because prior to our work TSA did not have 
information on the number of checkpoints equipped with video 
surveillance. We obtained this information as part of our survey of 
Federal Security Directors at SPOT airports. While TSA officials noted 
several possible limitations of the use of the existing video 
surveillance equipment, these images provide TSA a means of acquiring 
information about terrorist behaviors in the checkpoint environment 
that is not available elsewhere. If current research determines that 
the SPOT program has a scientifically validated basis for using 
behavior detection for counterterrorism purposes in the airport 
environment, then conducting a study to determine the feasibility of 
using images captured by video cameras could better position TSA in 
identifying behaviors to observe. 

Standardization Teams Assess BDO Proficiency in SPOT Activities and 
Provide Guidance and Mentoring to BDOs: 

TSA sends standardization teams to SPOT airports on a periodic basis 
to conduct activities related to quality control. Teams observe SPOT 
operations at an airport for several days, working side by side with 
the BDOs, on multiple shifts, observing their performance, offering 
guidance, and providing training when required. According to TSA, the 
purpose of a standardization team visit is to provide operational 
support to the BDOs, which includes additional training, mentoring, 
and guidance to help maintain a successful SPOT program. 

The standardization teams are comprised of at least two G-Band, or 
Expert[Footnote 95] BDOs who have received an additional week of 
training on SPOT behaviors and mentoring skills. SPOT officials stated 
that the SPOT program uses its standardization teams to assess overall 
BDO proficiency by observing BDOs, reviewing SPOT score sheet data, 
and other relevant data. Standardization teams may also provide a 
Behavior Observation and Analysis review class to refresh BDOs if the 
team determines that such a class is needed. The SPOT program director 
also said that the standardization teams aim to monitor the airport's 
compliance with the SPOT program's Standard Operating Procedures. As 
part of this mentoring approach, the standardization teams provide 
individual and team guidance to the BDOs, offer assistance in program 
management, and cover issues related to the interaction of BDOs with 
other TSA checkpoint personnel. 

TSA reported to us that it does not systematically track the 
standardization teams' recommendations or the frequency of the teams' 
airport visits. Standards for Internal Control state that programs 
should have controls in place to assess the quality of performance 
over time and ensure that the findings of audits and other reviews are 
promptly resolved. Managers are to (1) promptly evaluate findings from 
audits and other reviews, including those showing deficiencies and 
recommendations reported by auditors and others who evaluate agencies' 
operations; (2) determine proper actions in response to findings and 
recommendations from audits and reviews; and (3) complete, within 
established time frames, all actions that correct or otherwise resolve 
the matters brought to management's attention.[Footnote 96] Although 
the standardization teams may provide an airport Federal Security 
Director with recommendations on how to improve SPOT operations, the 
SPOT program director stated that Federal Security Directors are not 
required to document whether they have implemented the team 
recommendations. TSA officials told us that standardization teams can 
follow up on recommendations made during previous visits. However, TSA 
did not track whether corrective actions were implemented or the 
frequency of the team's airport visits to ensure the implementation of 
the airport's SPOT program. TSA officials stated that they are 
currently examining ways to compile data to address this issue, and 
expect to have a system in place in fiscal year 2010. 

TSA Developed and Deployed SPOT Training but Further Action Could 
Enhance Its Effectiveness: 

Although TSA has taken steps to incorporate all four elements of an 
effective training program by planning, designing, implementing, and 
evaluating training for BDOs, further action could help enhance the 
training's effectiveness. TSA initially consulted outside experts for 
help in the training's development, which began as a half-day course 
and has grown to include classroom, on-the-job, and advanced training. 
TSA also has efforts underway to improve its training program, such as 
the deployment of SPOT recurrent training. However, TSA evaluations of 
SPOT program instructors found mixed quality among them, from 2006 
onwards. Additionally, TSA has ongoing plans to evaluate the SPOT 
training for effectiveness, but has not yet developed time frames and 
milestones for completing the evaluation. 

TSA Has Taken Actions to Develop and Deploy SPOT Training: 

TSA's SPOT Training Evolved Over Time: 

In 2003, TSA officials at Boston Logan International airport developed 
the initial half-day training course for SPOT based on an existing 
course developed for the Massachusetts State Police. Their goal was to 
take the behavior detection program designed for law enforcement and 
apply it to screeners at airport checkpoints. According to TSA 
officials at Boston Logan, after they recognized that the lecture-
style course they originally designed was not effective, they tasked 
an instructional system designer from TSA's Workplace Performance and 
Training (the former name of TSA's Operational and Technical Training 
Division)[Footnote 97] and an industrial psychologist from the Office 
of Human Capital to redesign and expand the course, which was piloted 
in 2005. The 2007 SPOT strategic plan included training objectives for 
the SPOT program as follows: 

* reviewing existing behavior observation training providers, 

* establishing and prioritizing multimodal training and assistance 
efforts based on threat assessments and critical infrastructure, 

* establishing a Center of Excellence for Behavior Detection Program 
training that would continually enhance the quantity and quality of 
training to selected candidates, and: 

* developing a recurrent training program designed to refresh and hone 
skills needed for an effective Behavior Detection Program. 

Since that time, the SPOT program implemented, or is in the process of 
implementing, some of these objectives. For example, in 2008, as part 
of its effort towards establishing a center for excellence in behavior 
detection training (third objective), the SPOT program participated in 
a meeting with behavior detection training officials from various DHS 
components facilitated by DHS's Screening Coordination Office to 
promote the sharing of information about behavior detection training 
and foster future collaboration. Additionally, the SPOT program worked 
with TSA's Operational and Technical Training Division to create a 
recurrent training component for BDOs (fourth objective). For example, 
in 2008, the SPOT program office added a course on detecting 
microfacial expressions called Additional Behavior Detection 
Techniques.[Footnote 98] This 3-day course builds on the behavior 
detection skills taught in basic training, by teaching BDOs how to 
detect microfacial expressions. After pilot testing, the course began 
implementation nationwide in January 2009. 

TSA Consulted with Some Experts on Developing SPOT Training: 

In developing an effective training program, we previously reported 
that consultation with subject matter experts and expert entities is a 
core characteristic of the strategic training and development process. 
[Footnote 99] TSA SPOT program staff told us that they consulted with 
experts on behavior detection and observed existing behavior detection 
courses before deploying the SPOT training program. According to SPOT 
program officials, a TSA staff member from Boston Logan International 
Airport attended other training programs offered by other federal 
agencies and private training organizations to inform the design of 
SPOT training.[Footnote 100] TSA officials told us that information 
from the training courses was used to help develop the list of 
behaviors or "stress elevators" for the program, and that the point 
system used to identify passengers for referral screening was based in 
part on consultations with several subject-matter experts. 

TSA documentation also notes that a SPOT working group created in 
February 2004 consulted with the FBI's Behavioral Science Unit. 
[Footnote 101] The Behavioral Science Unit specializes in developing 
and facilitating training, research, and consultation in the 
behavioral sciences for the FBI, law enforcement, intelligence, and 
military communities. While TSA officials from Boston Logan told us 
that the FBI was included in this initial SPOT working group, these 
officials agree that coordination with the FBI lapsed until June 2009 
when the SPOT Program Office reengaged with the Behavioral Science 
Unit, and held a meeting with the unit at the FBI Academy in Quantico, 
Virginia. Since that meeting, a subject matter expert from the SPOT 
Program Office has been invited to be a member of the Terrorism 
Research and Analysis Project, which is an ongoing working group 
sponsored by the unit. 

In July 2008, DHS's Screening Coordination Office facilitated a 
collaborative discussion on behavior detection that included TSA, CBP, 
and Secret Service officials to better ensure that components within 
DHS share information regarding their efforts in behavior detection 
and provide a forum for components to have an informed and 
collaborative discussion on current capabilities, best practices, and 
lessons learned. According to TSA, no further contact has occurred 
between the DHS Behavior Detection Working Group and the SPOT program. 
Thus, the extent to which the working group's expertise will be used 
to refine or augment SPOT training in the future is not yet clear. 

SPOT Program Office Recently Deployed Recurrent Training: 

Along with basic and remedial training required by the Aviation and 
Transportation Security Act, TSA policy requires its screening force 
to regularly complete recurrent (refresher) training. TSA recognized 
that ongoing training of screeners on a frequent basis and effective 
supervisory training are critical to maintaining and enhancing skills 
learned during basic training. According to agency officials, TSA is 
currently working with DHS S&T to determine the necessary frequency 
for refresher training for each training course within the SPOT 
program. Furthermore, TSA plans to place BDOs under TSA's Performance 
and Accountability Standards System (PASS) beginning in fiscal year 
2010. This will include a recertification module. 

In 2008, the SPOT program office began the process for developing 
recurrent SPOT training. Our internal control standards and training 
assessment guidance suggest that such refresher training should be 
considered integral to an effective training program from the start 
because work conditions and environments can be expected to change 
over time, and additional or updated training is essential to ensuring 
that the program mission continues to be accomplished.[Footnote 102] 
According to the SPOT program office, the recently deployed recurrent 
training will be semiannual. TSA's Operational and Technical Training 
Division initially planned to pilot test recurrent training in April 
2009 followed by full implementation of the course in approximately 
May 2009. Because the Operational and Technical Training Division 
focus was shifted to completing the revisions for the SPOT basic 
certification course, recurrent training was delayed until September 
2009 when they released the training on TSA's Online Learning Center. 

Instructor Evaluations Found Mixed Quality; Issues with Program 
Management Led to Instructor Retraining: 

Our previous work on elements of effective training states that 
instructors must be both knowledgeable about the subject matter and 
issues involved, as well as able to effectively transfer these skills 
and knowledge to others.[Footnote 103] Moreover, internal control 
standards state that all personnel need to possess and maintain a 
level of competence that allows them to accomplish their assigned 
duties.[Footnote 104] Management needs to identify appropriate 
knowledge and skills needed for various jobs and provide needed 
training, as well as to ensure that those teaching the skills are 
themselves competent. 

TSA conducted internal assessments of SPOT instructors episodically 
from 2006 through March 2008. These assessments involved a few 
instructors being rated at a time, and found a wide range of 
competency among the instructors. In January 2009, TSA's Office of 
Inspections and Investigations began an investigation of the SPOT 
training manager, who resigned shortly thereafter. TSA investigators 
determined that the training manager and other trainers had created a 
hostile training environment that intimidated some trainees. To 
address this problem, TSA stated that the program office reexamined 
the SPOT training program nationally. This included recertifying 47 of 
54 SPOT instructors in March 2009, which included evaluation by TSA's 
Office of Human Capital, Quality Assurance assessors. Additionally, in 
July 2009, TSA centralized SPOT training at five permanent, regional 
training facilities in Orlando, Florida; Houston, Texas; Phoenix, 
Arizona; Denver, Colorado; and Philadelphia, Pennsylvania.[Footnote 
105] According to the SPOT program director, this will allow the SPOT 
program office more oversight over training. Previously, training was 
provided at individual airports. 

After the March 2009 recertification training, ratings scores of SPOT 
instructors showed less variation than did previous ratings. We 
reviewed the quality assurance instructor evaluations of two SPOT 
instructors conducted by TSA's Office of Human Capital, Training 
Standards and Evaluation Branch, and the 167 SPOT program instructor 
evaluations of 54 SPOT instructors conducted by the SPOT program 
office and TSA's Operational and Technical Training Division since the 
program started in October 2006.[Footnote 106] After the 
recertification training, 93 percent of instructors were rated as 
exceeding expectations, compared to 30 percent in the 2006 to 
September 2008 ratings. Table 3 shows the ratings of instructors for 
March 2009 compared to the period of 2006 to September 2008.[Footnote 
107] 

Table 3: SPOT Instructor Evaluation Ratings, 2006 to September 2008, 
and March 2009: 

2006 - Sept 2008; 
Number of instructor evaluations: 73; 
Unsatisfactory (0-74%): Number: 3; 
Unsatisfactory (0-74%): Percent: 4%; 
Needs improvement (75-84%): Number: 5; 
Needs improvement (75-84%): Percent: 7%; 
Meets expectations (85-94%): Number: 36; 
Meets expectations (85-94%): Percent: 49%; 
Exceeds expectations (95-100%): Number: 22; 
Exceeds expectations (95-100%): Percent: 30%; 
No numeric score given: Number: 7; 
No numeric score given: Percent: 10%. 

March 2009; 
Number of instructor evaluations: 94; 
Unsatisfactory (0-74%): Number: 0; 
Unsatisfactory (0-74%): Percent: 0%; 
Needs improvement (75-84%): Number: 1; 
Needs improvement (75-84%): Percent: 1%; 
Meets expectations (85-94%): Number: 6; 
Meets expectations (85-94%): Percent: 6%; 
Exceeds expectations (95-100%): Number: 87; 
Exceeds expectations (95-100%): Percent: 93%; 
No numeric score given: Number: 0; 
No numeric score given: Percent: 0%. 

Source: GAO analysis of TSA Quality Assurance Instructor Evaluations 
for SPOT. 

[End of table] 

In addition to the variation in numeric scores and rating levels for 
the 2006 to September 2008 period, as shown in table 3, we found 
substantial variation in the comments about instructor competency for 
the same period. For example, in 32 out of 74 instructor evaluation 
forms that we reviewed where comments were made about the instructor 
prior to 2009, the comments ranged from superb to needs more 
experience as an instructor, as well as needs more time performing the 
job as a BDO to be able to teach others. In the comments on an 
instructor who was rated as "meets expectations," the instructor was 
described as having "limited experience within the SPOT program," that 
this was "a major concern," and it was recommended that the instructor 
spend as much time as possible functioning as a BDO. In other cases, 
however, SPOT instructors were described as competent, solid, and 
outstanding. For example, one instructor who received a rating of 
"exceeds expectations" was described as a superb instructor who "is a 
valued member of the National Training Team." As noted above, 
following the March 2009 recertification training, 93 percent of the 
instructors received a rating of "exceeds expectations" with only 1 
percent "needing improvement." Of the 94 instructor evaluations 
completed in March 2009, 82 contained written comments. Of these, 
multiple SPOT instructors were described as excellent, knowledgeable, 
and effective. For example, an instructor who received a rating of 
"exceeds expectations" was noted as demonstrating a high degree of 
material knowledge and great presentation skills. TSA attributed the 
increase in instructor ratings to two factors. The first is low 
turnover among SPOT instructors, which allows instructors to hone both 
their technical and instructor skills. The second factor cited by TSA 
is that TSA conducted a 2-day instructor refresher training 
immediately prior to the evaluations in March 2009. To ensure all 
instructors were reevaluated within a specific time frame, evaluations 
were scheduled and conducted in a controlled environment. Instructors 
knew in advance they were going to be evaluated and delivered modules 
of the BDO certification course to other BDO instructors. 

TSA Has Taken Some Action, but Has Not Evaluated the SPOT Training 
Program for Effectiveness: 

We previously reported that evaluation is an integral part of training 
and development efforts, and that agencies need to systematically plan 
for and evaluate the effectiveness of training and development. 
[Footnote 108] Employing systematic monitoring and feedback processes 
can help by catching potential problems at an early stage, thereby 
saving valuable time and resources that a major redesign of training 
would likely entail. Similarly, in 2006, TSA's Operational and 
Technical Training Division issued general evaluation standards for 
training programs, stating that training programs should be 
comprehensively evaluated on a periodic basis to identify program 
strengths and weaknesses.[Footnote 109] Moreover, standard practices 
in program management for defining, designing, and executing programs 
include developing a road map, or program plan, to establish an order 
for executing specific projects needed to obtain defined programmatic 
results within a specified time frame.[Footnote 110] 

The former SPOT training manager told us that the SPOT program 
internally evaluates the effectiveness of SPOT training through the 
job knowledge tests that BDO candidates must pass following the 
classroom portion of the training and the SPOT Proficiency/On-the-Job 
Training Checklist following the on-the-job portion of the training. 
Furthermore, the former training manager told us that TSA knows that 
the SPOT training is effective because BDOs are able to recognize 
behaviors at the checkpoint, and because of BDOs' demonstrated ability 
to identify criminals--such as drug couriers or people with 
outstanding arrest warrants--through the screening process. 

Although TSA has not conducted a comprehensive analysis of the 
effectiveness of the SPOT training program, TSA's Office of Human 
Capital, Training Standards and Evaluation Branch conducted training 
evaluations to assess how students use what they were taught in the 
SPOT basic training course. Specifically, from July through September 
2008, the Training Standards and Evaluation Branch conducted 
evaluations at 5 of the 161 airports where the SPOT program is 
currently operating. Based on BDO feedback at the 5 airports, the 
Training Standards and Evaluation Branch's final report contained a 
series of recommendations for improving the SPOT training program. 
These recommendations and TSA's actions to address them are summarized 
in table 4. 

Table 4: TSA Training Standards and Evaluation Branch Recommendations 
for Improving SPOT Training and TSA Actions on the Recommendations: 

Training Standards and Evaluation Branch recommendations: Ensure 
training instructors adhere to a set of professional guidelines; 
TSA action on recommendations: TSA sent 47 TSA Approved Instructors 
for the SPOT program to recertification training in March 2009. 

Training Standards and Evaluation Branch recommendations: Add local 
policies and procedure as an addendum to the (SPOT) Training; 
TSA action on recommendations: No action[A]. 

Training Standards and Evaluation Branch recommendations: Include more 
role-playing and scenarios in the classroom training so all trainees 
can practice casual conversation skills; 
TSA action on recommendations: TSA added more role-playing scenarios 
to their basic SPOT training. 

Training Standards and Evaluation Branch recommendations: Develop 
recurrent training that can be placed on the TSA Online Learning 
Center; 
TSA action on recommendations: TSA developed and deployed recurrent 
training on the TSA Online Learning Center in September 2009. 

Training Standards and Evaluation Branch recommendations: Develop 
templates for writing reports; 
TSA action on recommendations: TSA added an Incident Report Writing 
course to the TSA Online Learning Center. Additionally, TSA has 
developed templates for Incident Reports and After Action Reports. TSA 
has also developed Online Learning Center training for completing SPOT 
Referral Reports. 

Training Standards and Evaluation Branch recommendations: Provide more 
real world videos; 
TSA action on recommendations: TSA revised the SPOT training videos in 
late 2008. 

Training Standards and Evaluation Branch recommendations: Provide 
recurrent training of behaviors through online videos; 
TSA action on recommendations: The video scenarios for recurrent 
training will be available in the second quarter of fiscal year 2010. 

Training Standards and Evaluation Branch recommendations: Add parts of 
the Bomb Appraisal Officer task into the training; 
TSA action on recommendations: No action[A]. 

Training Standards and Evaluation Branch recommendations: Provide 
recurrent training outside of TSA (more Immigration and Customs 
Enforcement, DEA, and CBP training); 
TSA action on recommendations: No action[A]. 

Training Standards and Evaluation Branch recommendations: Have BDOs 
spend more time with an On-the-Job-Training mentor; 
TSA action on recommendations: No action[A]. 

Training Standards and Evaluation Branch recommendations: Validate the 
training for course content and On-the-Job-Training; 
TSA action on recommendations: In 2009, in coordination with DHS S&T, 
TSA began the scientific analysis of the BDO position to empirically 
derive and validate the knowledge, skills, and attributes that it 
requires. The analysis is projected to be completed in fiscal year 
2010. 

Training Standards and Evaluation Branch recommendations: Clarify 
SPOT's "Walk-the-Line" policy and communicate it to all BDO personnel; 
TSA action on recommendations: TSA issued revised SPOT Standard 
Operating Procedures to all BDOs in January 2009. 

Source: TSA, Training Standards and Evaluation Branch, Office of Human 
Capital, Memorandum For Operational and Technical Training, and 
Behavior Detection and Document Validation Branch, Office of Security 
Operations on Training Transfer (L3) of SPOT Training, October 30, 
2008. 

[A] According to TSA, the SPOT program office will determine if the 
recommended action is appropriate after the BDO job task analysis and 
training task analysis are completed. 

[End of table] 

Additionally, in conjunction with S&T, TSA conducted a training 
effectiveness evaluation on the Additional Behavior Detection 
Techniques course, which showed a statistically significant increase 
in knowledge and skills following completion of the course. 

S&T is currently conducting a BDO job task analysis, which may be used 
to evaluate and update the SPOT training curriculum. Following the 
completion of the job task analysis--anticipated in mid-May 2010--
TSA's Operational and Technical Training Division intends to conduct 
an in-depth training gap analysis,[Footnote 111] which will take 
approximately 2 months to complete. Following completion of the 
training gap analysis, the agency will develop project plans, 
including milestones for future development efforts, to address any 
training concerns. However, to date, the agency does not have an 
evaluation plan including time frames and milestones for completion. 
According to the Operational and Technical Training Division, TSA will 
conduct periodic evaluations as the BDO position evolves. By 
conducting a comprehensive evaluation of the effectiveness of its 
training program, TSA will be in a better position to determine if 
BDOs are being taught the knowledge and skills they need to perform 
their job. Furthermore, by developing milestones and time frames for 
conducting such evaluations systematically, as well as on a periodic 
basis, TSA could help ensure that the SPOT training program is 
evaluated in accordance with its directives to help ensure that the 
program continues to provide BDOs with the necessary tools required to 
carry out their responsibilities. 

Conclusions: 

TSA developed the SPOT program in the wake of September 11, 2001, in 
an effort to respond quickly to potential threats to aviation security 
by identifying individuals who may pose a threat to aviation security, 
including terrorists planning or executing an attack who were not 
likely to be identified by TSA's other screening security measures. 
Because TSA did not ensure that SPOT's underlying methodology and work 
methods were scientifically validated prior to its nationwide 
deployment, an independent panel of experts could help determine 
whether a scientific foundation exists for the way in which the SPOT 
program uses behavior detection analysis for counterterrorism purposes 
in the aviation environment. 

With approximately $5.2 billion devoted to screening passengers and 
their property in fiscal year 2009, it is important that TSA provides 
effective stewardship of taxpayer funds ensuring a return on 
investment for each layer of its security system. As one layer of 
aviation security, the SPOT program has an estimated projected cost of 
about $1.2 billion over the next 5 years if the administration's 
requested funding of $232 million for fiscal year 2011 remains at this 
level.[Footnote 112] The nation's constrained fiscal environment makes 
it imperative that careful choices be made regarding which investments 
to pursue and which to discontinue. If an independent expert panel 
determines that DHS's study is sufficiently comprehensive to determine 
whether the SPOT program is based on valid scientific principles that 
can be effectively applied in an airport environment for 
counterterrorism purposes, then conducting a comprehensive risk 
assessment including threat, vulnerability, and consequence could 
strengthen TSA's ability in making resource allocation decisions and 
prioritizing its risk mitigation efforts. Moreover, conducting a cost- 
benefit analysis could help TSA determine whether SPOT provides 
benefits greater than or equal to other security alternatives and 
whether its level of investment in the SPOT program is appropriate. 
Revising its strategic plan for SPOT to incorporate risk assessment 
information, cost and resource analysis, and other essential 
components could enhance the plan's usefulness to TSA in making 
program management and resource allocation decisions to effectively 
manage the deployment of SPOT. 

Providing guidance on how to use TSA's resources for running passenger 
names against intelligence and criminal databases available to the 
Transportation Security Operations Center and helping DHS to connect 
disparate pieces of information using the Transportation Information 
Sharing System and other related intelligence and crime database and 
data sources could better inform DHS and TSA regarding the identity 
and background of certain individuals and thereby enhance aviation 
security. In addition, implementing the steps called for in the TSA 
Office of Strategic Operations plan to provide BDOs with a real-time 
mechanism to verify passenger identities and backgrounds via TSA's 
Transportation Security Operations Center could strengthen their 
ability to rapidly verify the identity and background of passengers 
who have caused concern, and increase the likelihood of detecting and 
disrupting potential terrorists intending to cause harm to the 
aviation system. Additionally, developing outcome-oriented performance 
measures, making improvements to the SPOT database, and studying the 
feasibility of utilizing video recordings of individuals as they 
transited checkpoints and who were later charged with or pleaded 
guilty to terrorism-related offenses, could help TSA evaluate the SPOT 
program, identify potential vulnerabilities, and assess the 
effectiveness of its BDOs. Further, developing a plan for systematic 
and periodic evaluation of the training provided to BDOs along with 
time frames and milestones for its completion could help ensure that 
the SPOT training program is evaluated in accordance with its 
directives to help ensure that the program continues to provide BDOs 
with the necessary tools required to carry out their responsibilities. 

Recommendations for Executive Action: 

To help ensure that SPOT is based on valid scientific principles that 
can be effectively applied in an airport environment, we recommend 
that the Secretary of Homeland Security convene an independent panel 
of experts to review the methodology of the DHS S&T Directorate study 
on the SPOT program to determine whether the study's methodology is 
sufficiently comprehensive to validate the SPOT program. This 
assessment should include appropriate input from other federal 
agencies with expertise in behavior detection and relevant subject 
matter experts. 

If this research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism 
purposes in the airport environment, then we recommend that the TSA 
Administrator take the following four actions: 

* Conduct a comprehensive risk assessment to include threat, 
vulnerability, and consequence of airports nationwide to determine the 
effective deployment of SPOT if TSA's ongoing Aviation Modal Risk 
Assessment lacks this information. 

* Perform a cost-benefit analysis of the SPOT program, including a 
comparison of the SPOT program with other security screening programs, 
such as random screening, or already existing security measures. 

* Revise and implement the SPOT strategic plan by incorporating risk 
assessment information, identifying cost and resources, linking it to 
other related TSA strategic documents, describing how SPOT is 
integrated and implemented with TSA's other layers of aviation 
security, and providing guidance on how to effectively link the roles, 
responsibilities, and capabilities of federal, state, and local 
officials providing program support. 

* Study the feasibility of using airport checkpoint-surveillance video 
recordings of individuals transiting checkpoints who were later 
charged with or pleaded guilty to terrorism-related offenses to 
enhance understanding of terrorist behaviors in the airport checkpoint 
environment. 

Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of 
the S&T study, we recommend that the TSA Administrator take the 
following six actions to ensure the program's effective implementation: 

* To provide additional assurance that TSA utilizes available 
resources to support the goals of deterring, detecting, and preventing 
security threats to the aviation system, TSA should: 

- Provide guidance in the SPOT Standard Operating Procedures or other 
TSA directive to BDOs, or other TSA personnel, on inputting data into 
the Transportation Information Sharing System and set milestones and a 
time frame for deploying Transportation Information Sharing System 
access to SPOT airports so that TSA and intelligence community 
entities have information from all SPOT LEO referrals readily 
available to assist in "connecting the dots" and identifying potential 
terror plots. 

- Implement the steps called for in the TSA Office of Security 
Operations Business plan to develop a standardized process for 
allowing BDOs or other designated airport officials to send 
information to TSA's Transportation Security Operations Center about 
passengers whose behavior indicates that they may pose a threat to 
security, and provide guidance on how designated TSA officials are to 
receive information back from the Transportation Security Operations 
Center. 

- Direct the TSA Transportation Security Operations Center to utilize 
all of the law enforcement and intelligence databases available to it 
when running passenger names, for passengers who have risen to the 
level of a LEO referral. 

* To better measure the effectiveness of the program and evaluate the 
performance of BDOs, TSA should: 

- Establish a plan that includes objectives, milestones, and time 
frames to develop outcome-oriented performance measures to help refine 
the current methods used by Behavior Detection Officers for 
identifying individuals who may pose a risk to the aviation system. 

- Establish controls to help ensure completeness, accuracy, 
authorization, and validity of data collected during SPOT screening. 

* To help ensure that TSA provides BDOs with the knowledge and skills 
needed to perform their duties, TSA should: 

- Establish time frames and milestones for its plan to systematically 
conduct evaluations of the SPOT training program on a periodic basis. 

Agency Comments and Our Evaluation: 

We provided a draft of our report to DHS and TSA on March 19, 2010, 
for review and comment. On May 3, 2010, DHS provided written comments, 
which are reprinted in appendix II. In commenting on our report, DHS 
stated that it concurred with 10 of our recommendations and identified 
actions taken, planned, or under way to implement them. However, the 
actions DHS reported it plans to take and has underway do not fully 
address the intent of our first recommendation. DHS also concurred in 
principle with an eleventh recommendation stating that it had convened 
a working group to determine the feasibility of implementing it. DHS 
commented on the scientific basis underlying SPOT and on two 
statements in our report that it believed were inaccurate--
specifically, DHS disagreed with our reliance on a 2008 National 
Research Council report published under the auspices of the National 
Academy of Sciences on issues related to behavior detection, and 
second, on issues related to unpublished research they had cited as a 
partial validation of some aspects of the SPOT program.[Footnote 113] 
Finally, DHS commented on our conclusion regarding the use of the SPOT 
referral data. 

Regarding our first recommendation that DHS convene an independent 
panel of experts to review the methodology of DHS's Science and 
Technology Directorate (S&T) study on SPOT, and to include appropriate 
input from other federal agencies with relevant expertise, DHS 
concurred and stated the current process includes an independent 
review of the program that will include input from other federal 
agencies and relevant experts. Although DHS has contracted with the 
American Institutes for Research to conduct its study, it remains 
unclear who will oversee this review and whether they are sufficiently 
independent from the current research process. DHS's response also 
does not describe how the review currently planned is designed to 
determine whether the study's methodology is sufficiently 
comprehensive to validate the SPOT program. As we noted in our report, 
research on other issues, such as determining the number of 
individuals needed to observe a given number of passengers moving at a 
given rate per day in an airport environment or the duration that such 
observation can be conducted by BDOs before observation fatigue 
affects effectiveness, could provide additional information on the 
extent to which SPOT can be effectively implemented in airports. Dr. 
Paul Ekman, a leading research scientist in the field of behavior 
detection, told us that additional research could help determine the 
need for periodic refresher training since no research has yet 
determined whether behavior detection is easily forgotten or can be 
potentially degraded with time or lack of use. Thus, questions exist 
as to whether behavior detection principles can be reliably and 
effectively used for counterterrorism purposes in airport settings to 
identify individuals who may pose a risk to the aviation system. To 
help ensure an objective assessment of the study's methodology and 
findings, DHS could benefit from convening an independent panel of 
experts from outside DHS to determine whether the study's methodology 
is sufficiently comprehensive to validate the SPOT program. 

DHS also concurred with our second recommendation to conduct a 
comprehensive risk assessment to determine the effective deployment of 
SPOT. DHS stated that TSA's Aviation Modal Risk Assessment is designed 
to evaluate overall transportation security risk, not deployment 
strategies. However, DHS noted that TSA is in the process of 
conducting an initial risk analysis using its risk management analysis 
tool and plans to update this analysis in the future. However, it is 
not clear from DHS's comments how this analysis will incorporate an 
assessment of TSA's deployment strategy for SPOT. 

DHS also concurred with our third recommendation to perform a cost- 
benefit analysis of SPOT. DHS noted that TSA is developing an initial 
cost-benefit analysis and that the flexibility of behavior detection 
officers already suggests that behavior detection is cost-effective. 
However, it is not clear from DHS's comments whether its cost-benefit 
analysis will include a comparison of the SPOT program with other 
security screening programs, such as random screening, or already 
existing security measures as we recommended. Completing its cost- 
benefit analysis and comparing it to other screening programs should 
help establish whether the SPOT program is cost-effective compared to 
other layers of security. 

With regard to our fourth recommendation to revise and implement the 
SPOT strategic plan using risk assessment information, DHS concurred 
and noted that analysis facilitated by the risk management analysis 
tool will allow the program to revise the SPOT strategic plan to 
incorporate the elements identified in our recommendation. 

DHS also concurred with our fifth recommendation to study the 
feasibility of using airport checkpoint-surveillance video recordings 
to enhance its understanding of terrorist behaviors. DHS noted that 
TSA agrees this could be a useful tool and is working with DHS's S&T 
Directorate to utilize video case studies of terrorists, if possible. 
These cases studies could help TSA determine what behaviors had been 
demonstrated by these persons convicted of terrorist-related offenses 
who went through SPOT airports, and what could be learned from the 
observed behaviors. 

DHS concurred with our sixth recommendation that TSA provide guidance 
in the SPOT SOP or other directives to BDOs, or other TSA personnel, 
on how to input data into the Transportation Information Sharing 
System database. DHS stated that the SPOT SOP is undergoing revision, 
and that the revised version will provide guidance directing the input 
of BDO data into the Transportation Information Sharing System. DHS 
anticipates release of the updated SPOT SOP in fiscal year 2010. DHS 
also agreed that TSA should set milestones and a time frame for 
deploying Transportation Information Sharing System access to SPOT 
airports so that TSA and intelligence community entities have 
information from all SPOT LEO referrals readily available to assist in 
"connecting the dots" and identifying potential terror plots. DHS 
stated that TSA is currently drafting a plan to include milestones and 
a time frame for deploying System access to all SPOT airports. 

DHS concurred with our seventh recommendation to develop a 
standardized process to allow BDOs or other designated airport 
officials to send information to TSA's Transportation Security 
Operations Center about passengers whose behavior indicates they may 
pose a threat to security, and to provide guidance on how designated 
TSA officials are to receive information back from the Center. DHS 
stated that TSA has convened a working group to address this 
recommendation. Moreover, TSA is developing a system and procedure for 
sending and receiving information from the Center and stated that it 
anticipates having a system in place later in fiscal year 2010. 

DHS concurred in principle with regard to our eighth recommendation 
that the Transportation Security Operations Center utilize all of the 
databases available to it when conducting checks on passengers who 
rise to the level of a LEO referral against intelligence and criminal 
databases. DHS stated that TSA has convened a working group to address 
this recommendation. According to DHS, this group will conduct a study 
during fiscal year 2010 to determine the feasibility of fully 
implementing this recommendation. As such, the study is to review the 
various authorities, permissions, and limitations of each of the 
databases or systems cited in our report. DHS stated that access to 
some of the systems, requires more justification than a BDO referral. 
Further, according to DHS, because some of the databases or systems 
contain classified information, TSA will also need to adopt a 
communication strategy to transmit the passenger information between 
the BDO and Transportation Security Operations Center. DHS stated that 
TSA will work on a process to collect the passenger information, 
verify the passenger's identity, through checks of databases, and 
analyze that information to determine if the passenger is the subject 
of an investigation and may pose a risk to aviation security. 

With regard to our ninth recommendation to establish a plan with 
objectives, milestones, and time frames to develop outcome-oriented 
performance measures for BDOs, DHS concurred and stated that TSA 
intends to consult with experts to develop outcome-oriented 
performance measures. 

DHS also concurred with our tenth recommendation to establish controls 
for SPOT data. DHS noted that TSA established additional controls as 
part of the SPOT database migration to TSA's Performance Management 
Information System and is exploring an additional technology solution 
to reduce possible errors. As noted in our report, since these changes 
to the database were not complete at the time of our audit, we could 
not assess whether the problems we identified with the database had 
been corrected. 

Regarding our eleventh recommendation to establish time frames and 
milestones to systematically evaluate the SPOT training program on a 
periodic basis, DHS concurred and stated that TSA intends to develop 
such a plan following completion of DHS's S&T Directorate's BDO Job 
Task Analysis, and TSA's training gap analysis, which identifies gaps 
in the training curriculum. 

DHS also commented on the scientific basis underlying SPOT. 
Specifically, DHS stated that decades of scientific research has shown 
the SPOT behaviors to be "universal in their manifestation." However, 
according to DHS, its S&T Directorate is examining the extent to which 
behavior indicators are appropriate for screening purposes and lead to 
appropriate and correct security decisions. DHS also commented that 
the results of this work, which is currently underway, will establish 
a scientific basis of the extent to which the SPOT program instruments 
and methods are valid. Thus, DHS's comments suggest that additional 
research is needed to determine whether these behaviors can be used in 
an airport environment for screening passengers to identify threats to 
the aviation system. 

Moreover, DHS took issue with our use of a report from the National 
Research Council of the National Academy of Sciences stating that we 
improperly relied upon this report.[Footnote 114] We disagree. DHS 
questioned the findings of the National Research Council report and 
stated that it lacked sufficient information for its conclusions 
because it principally focused on privacy as it relates to data mining 
and behavioral surveillance and was not intended to represent an 
exhaustive or definitive review of the research or operational 
literature on behavioral screening, including recent unpublished DHS, 
defense, and intelligence community studies. DHS also stated that the 
National Research Council report did not study the SPOT program and 
that the researchers did not conduct interviews with SPOT personnel. 

As we noted in our report, although the National Research Council 
report addresses broader issues related to privacy and data mining, a 
senior Council official--and one of the authors of the study--stated 
that the committee included behavior detection as a focus because any 
behavior detection program could have privacy implications. This 
official added that the primary objective of the report was to develop 
a framework for sound decision making for programs, such as SPOT, and 
help ensure a sound scientific and legal basis. According to this 
official, the National Academy of Sciences' Committee on Technical and 
Privacy Dimensions of Information for Terrorism Prevention and Other 
National Goals--which had oversight of the report--was briefed on the 
SPOT program as part of the study. The Committee also conducted 
meetings with three experts in behavior detection as part of their 
research. During the course of our review, we interviewed three 
Committee members responsible for developing the report's findings, as 
well as four other behavior detection experts, including the three who 
participated in the National Research Council study. Our discussions 
with these experts corroborated the report's findings. Thus, we 
believe that our use of the Council report was an appropriate and a 
necessary part of our review. 

However, the National Research Council report was only one of many 
sources that we analyzed with regard to the science of behavioral and 
physiological screening, and its applicability to an airport 
environment. As we noted in the description of our methodology, our 
study included interviews with officials from DHS as well as several 
of its components and other U.S. government agencies--each of which 
use elements of behavior detection in their daily work. We also 
interviewed El Al airline officials, a former director of security at 
Israel's Ben-Gurion airport, and seven nationally recognized experts 
in behavior detection as part of our review. Moreover, as we explained 
in the discussion of our scope and methodology, we conducted a survey 
about the SPOT program of all 118 Federal Security Directors for all 
SPOT airports, and conducted site visits to 15 SPOT airports. In 
addition, we analyzed the SPOT referral database, to the extent the 
data permitted, covering a 4-year period and the results from 2 
billion passengers passing through SPOT airports. Moreover, we 
attended both the basic and advanced training courses in behavior 
detection provided by TSA to BDOs, in order to better understand how 
the program is carried out. Therefore, our analysis of the program was 
not derived from or based on a single study by the National Research 
Council as DHS suggested, but rather is based on all of the 
information we gathered and synthesized from multiple, diverse, expert 
sources, each of which provided different perspectives about the 
program, as well as about behavior detection in general. 

DHS also disagreed with the accuracy of a statement included in our 
report that noted DHS S&T could not provide us with specific contacts 
related to sources of information for certain research it cited as 
support for the SPOT program. In its comments, DHS stated that it had 
provided us with all requested documents that represent DHS's S&T 
Directorate-sponsored research. We agree. However, DHS did not provide 
us with contact information for the sources of unpublished studies by 
the Department of Defense and other intelligence community studies 
that DHS S&T had cited as support for the SPOT program. Without such 
information, we are unable to verify the contents of these unpublished 
studies. 

Finally, DHS stated that while we were unable to use the SPOT referral 
data to assess whether any behavior or combination of SPOT behaviors 
could be used to reliably predict the final outcome of an incident 
involving the use of SPOT, it was able to analyze the SPOT referral 
database successfully after working with TSA to verify scores assigned 
to different indicators. Our concern with the data did not involve the 
question of whether some behaviors were entered erroneously, nor 
whether errors in coding were excessive or non-random. Rather, we were 
concerned with whether the data on behaviors were complete. 
Specifically, it cannot be determined from the SPOT referral database 
whether all behaviors observed were included for each referred 
passenger by each BDO or whether only the behaviors that were 
sufficient for a LEO referral were recorded into the database. It is 
not possible to determine from the database if the number of observed 
behaviors entered for a given passenger was the total number of 
observed behaviors, or whether additional behaviors were observed. A 
rigorous analysis of the relative effects of the different behaviors 
on the outcomes of the use of SPOT would require each BDO to record, 
for each of the observable behaviors, whether it was or was not 
observed. 

TSA also provided technical comments that we incorporated as 
appropriate. 

We will send copies of this report to the Secretary of Homeland 
Security; the TSA Administrator (Acting); and interested congressional 
committees as appropriate. The report will also be available at no 
charge on the GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-4379 or lords@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report are 
acknowledged in appendix III. 

Sincerely yours, 

Signed by: 

Stephen M. Lord: 
Director, Homeland Security and Justice Issues: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine the extent to which the Transportation Security 
Administration (TSA) determined whether the Screening of Passengers By 
Observation Techniques (SPOT) program had a scientifically-validated 
basis for identifying passengers before deploying it, we reviewed 
literature on behavior analysis by subject matter experts, interviewed 
seven experts in behavior analysis, interviewed other federal agencies 
and entities about how they use behavior detection techniques, and 
analyzed relevant reports and books on the topic. These included a 
2008 study by the National Research Council of the National Academy of 
Sciences that has a discussion regarding deception and behavioral 
surveillance, as well as other issues related to behavioral 
analysis.[Footnote 115] We interviewed Dr. Herbert S. Lin, who was a 
primary author of the report, as well as Dr. Robert W. Levenson, and 
Dr. Stephen E. Fienberg, both members of the Academy committee that 
oversaw the report, about the report's findings with regard to 
behavior detection, and the extent to which behavior detection in a 
complex environment, such as an airport terminal, has been validated 
with regard to its effectiveness in identifying persons who may be a 
risk to aviation security. Other behavior detection experts we 
consulted were Dr. Paul Ekman;[Footnote 116] Dr. Mark Frank;[Footnote 
117] Dr. David Givens;[Footnote 118] Dr. David Matsumoto;[Footnote 
119] and Mr. Rafi Ron, former director of security at Israel's Ben-
Gurion Airport. Dr. Ekman, Dr. Frank, and Mr. Ron provided expert 
advice for the National Research Council study. Dr. Givens was 
identified by TSA as having been their principal source for the 
nonverbal behavior indicators used by the SPOT program. We also 
interviewed Dr. Lawrence M. Wein, an expert in emergency responses to 
terror attacks and mathematical models in operations management. 
[Footnote 120] In addition, we interviewed officials from the 
Department of Homeland Security's (DHS) Science and Technology (S&T) 
Directorate regarding their ongoing research into behavior detection. 
Although the views of these experts cannot be generalized across all 
experts in behavior analysis because we selected individuals based on 
their publications on behavioral analysis or related topics, their 
recognized accomplishments and expertise, and, in some cases, TSA's 
use of their work or expertise to design and review the SPOT program's 
behaviors, they provided us with an overall understanding of the 
fundamentals of behavior analysis, and how it could be applied. 

To determine the basis for TSA's strategy to develop and deploy SPOT 
and evaluate to what extent SPOT was informed by a cost-benefit 
analysis and a strategic plan, we reviewed program documentation, 
including briefings prepared by the SPOT program office during the 
course of developing and fielding SPOT, two versions of a strategic 
plan for SPOT, and the 2009 SPOT standard operating procedures 
guidance. We compared the plans and analyses used by TSA to develop 
and implement SPOT to criteria on how to develop and implement 
programs in DHS's 2006 Cost Benefit Analysis Guidebook,[Footnote 121] 
as well as to Office of Management and Budget guidance on the utility 
of cost-benefit analyses in program implementation.[Footnote 122] We 
also analyzed the development of SPOT in light of the standards and 
criteria cited in DHS's 2006 National Infrastructure Protection Plan. 
We met with relevant TSA officials to discuss these issues. To assess 
whether DHS developed an effective strategic plan for SPOT prior to 
implementing the program, we interviewed TSA officials involved in 
development of the SPOT strategic plan. We analyzed whether the SPOT 
plan incorporated the desirable characteristics of an effective 
strategic plan as identified by previous GAO work on what strategic 
plans should include to be considered effective, such as a risk 
assessment, cost and resources analysis, and a means for collaboration 
with other key entities.[Footnote 123] We also examined it in light of 
the requirements of the Government Performance and Results Act of 
1993, which specifies the elements of strategic plans for government 
programs.[Footnote 124] We assessed whether the SPOT strategic plan 
was followed by TSA. As part of our analysis of the planning for SPOT 
before it was implemented on a nationwide basis, we reviewed TSA 
documentation related to the development and pilot testing of SPOT, 
such as a TSA white paper on SPOT, and interviewed key program 
officials from both headquarters and field offices.[Footnote 125] 

We also interviewed cognizant officials from other U.S. government 
agencies and agency entities that utilize behavior detection in their 
work, including U.S. Customs and Border Protection (CBP), the U.S. 
Secret Service, the TSA's Federal Air Marshal Service (FAMS) 
component, and the Federal Bureau of Investigation (FBI). We sought 
their views on the utility of various behavior detection methods, 
their experience with practicing behavior detection, and asked them 
about the extent to which TSA had consulted with them in developing 
and implementing the SPOT program. 

To better understand how SPOT incorporated expertise about the use of 
behavior detection in an airport setting, we interviewed officials 
from Israel's El Al Airlines, which is cited by TSA as having provided 
part of the basis of the SPOT program. We asked about El Al's methods 
to ensure the security of its passenger aircraft, and also interviewed 
a former head of security at Israel's Ben-Gurion airport, who has 
advised TSA on security issues. We asked TSA and SPOT program 
officials about their consultations with El Al, and about the ways in 
which they had utilized El Al's expertise, as well as about any other 
entities whose expertise they may have adopted into SPOT. 

To determine the challenges, if any, that emerged during 
implementation of the SPOT program, we interviewed headquarters and 
field personnel about how the program has utilized the resources 
available to it to ensure that it is effective. These resources 
included the support of law enforcement officers (LEOs), to whom 
passengers are referred by Behavior Detection Officers (BDOs) for 
additional questioning. In addition, we interviewed SPOT program and 
TSA officials about the databases available to them at TSA's 
Transportation Security Operations Center to determine if a suspect 
passenger is being sought by other U.S. law enforcement or 
intelligence entities, and whether there is guidance for BDOs on when 
and how to contact the Transportation Security Operations Center. We 
also asked about whether there is guidance and training for BDOs on 
how to access TSA's Transportation Information Sharing System 
database, which is owned by FAMS, and is available through the 
Transportation Security Operations Center.[Footnote 126] To determine 
if any management challenges had emerged related to management 
controls in developing and implementing SPOT, we compared TSA's 
approach for implementing and managing the SPOT program with GAO's 
Standards for Internal Control in the Federal Government[Footnote 127] 
and with risk management principles we had previously identified. 
[Footnote 128] Our legal counsel office reviewed court decisions 
relevant to the SPOT program. In addition, we interviewed attorneys 
from the American Civil Liberties Union, and obtained and reviewed 
TSA's Privacy Impact Assessments for SPOT, the Transportation Security 
Operations Center, and the Transportation Information Sharing System. 
We also met with and discussed relevant privacy and legal issues with 
TSA's Offices of Privacy and Civil Rights/Civil Liberties. To obtain 
data about certain aspects of the SPOT program that the SPOT program 
office did not have, we conducted a survey of Federal Security 
Directors[Footnote 129]whose responsibilities included security at all 
161 SPOT airports at the time of our survey. (Some Federal Security 
Directors have responsibility for more than one airport.) We obtained 
a 100 percent response rate. This survey asked, among other things, 
about whether there were cameras at security checkpoints that record 
the interactions of Transportation Security Officers (TSO), BDOs, and 
passengers; if the airport authority had an agreement with TSA that 
specifies certain law enforcement actions during a SPOT referral; and 
if there was an agreement, or any other comparable guidance that 
specified a time limit for LEOs to come to checkpoints after being 
called for help by BDOs. 

To determine the extent to which TSA has measured SPOT's effect on 
aviation security, we obtained and analyzed the TSA SPOT referral 
database, which records all incidents in which BDOs refer passengers 
to secondary, more intensive questioning, and which also records all 
incidents in which BDOs chose to refer passengers to LEOs. We found 
that the SPOT database was sufficiently reliable to count the number 
of arrests resulting from referrals from BDOs to LEOs, for examining 
the reasons for each arrest, and for counting the percentage of times 
that LEOs responded to BDO calls for service, and the length of time 
required. Use of these data required us to resolve apparent 
contradictions and anomalies in the database to make the data usable. 
Because of data problems, we were unable to conduct analyses to assess 
whether any behavior or combination of behaviors could be used to 
predict the final outcome of an incident involving the use of SPOT. In 
addition, we reviewed relevant standardization team reports and 
observed a standardization team visit in operation. 

In addition, we spoke with BDO managers, Federal Security Directors, 
and Assistant Federal Security Directors to determine how BDOs are 
evaluated. To do so, we conducted site visits to 15 commercial 
airports at which BDOs and SPOT have been deployed, or almost 10 
percent of the 161 airports with SPOT. We chose these airports taking 
into account the following criteria, among others: (1) each airport 
had BDOs deployed, and at each, the SPOT program had been in effect 
for no less than 3 months; (2) airports were chosen to provide a 
variety of sizes, as measured in annual passenger volume; physical 
location within the country (northeast, southwest, central, Pacific 
Coast, rural, urban); and estimated risk of terrorist incident, using 
DHS's Current Airports Threat Assessment[Footnote 130] list (visiting 
6 that were in the top 10, and others much lower); (3) BDOs who are 
employed by contractors, rather than employed directly by TSA; and (4) 
airports with LEOs who were identified to us by TSA as having received 
some form of behavior detection training and airports where they were 
not known to have received such training. In addition, we took into 
account the location of the airports with regard their proximity to 
subject matter experts on behavior detection whom we wished to 
interview, as well as the time and cost required to reach certain 
airports. 

At each of the airports we visited, we interviewed cognizant 
officials, including the Federal Security Director or Assistant 
assigned to the airport, the BDO program manager, one or two BDOs, and 
one or two LEOs who have interacted with BDOs. Since each of these 
airports differs in terms of passenger volume, physical size and 
layout, geographic location, and potential value as a target for 
terrorism, among other things, the results from these visits are not 
generalizable to other airports. However, these visits provided 
helpful insight into the operation of SPOT at airports. 

In addition, to determine if individuals had transited SPOT airports 
who were later charged with or pleaded guilty to terrorism-related 
offenses, we reviewed information contained in (1) the Treasury 
Enforcement Communication System II database maintained by CBP; 
[Footnote 131] (2) Department of Justice information and court 
documents, including indictments and related documents; and (3) media 
accounts of individuals accused of terrorism-related activities. We 
compared information pertaining to these individuals' dates of transit 
to the dates when SPOT was deployed to the various airports identified 
in the Treasury Enforcement Communication System and Justice 
Department data to determine if SPOT had been deployed at a given 
airport when the transits occurred. Further, we used our survey of 
Federal Security Directors at SPOT airports to determine the extent to 
which video surveillance cameras are present at checkpoints. 

To assess the extent that SPOT training incorporates the attributes of 
an effective training program, we had training experts at TSA 
headquarters complete a training assessment tool that we developed 
using our prior work for assessing training courses and curricula. 
[Footnote 132] To address training-related issues, including to 
understand better how other entities train their employees in behavior 
detection, and what their curricula include, we conducted site visits 
to the Secret Service, FAMS, CBP, and the FBI, and also interviewed 
nongovernmental experts on behavior detection (our selection of these 
experts is discussed above). As part of our assessment of SPOT 
training, we attended the basic SPOT training course given to BDOs, as 
well as the advanced SPOT course on behavior detection. We interviewed 
BDOs and BDO managers about the SPOT training, as well as officials of 
El Al airlines, with regard to how El Al trains and tests its 
personnel who utilize behavior recognition and analysis as part of 
their assessment of El Al passengers. 

We conducted this performance audit from May 2008 through May 2010, in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: DHS Comments: 

U.S. Department of Homeland Security: 
Washington, DC 20528: 

May 3, 2010: 

Mr. Steve Lord: 
Director, Homeland Security & Justice Issues: 
U.S. Government Accountability Office (GAO): 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Lord: 

Thank you for the opportunity to review and comment on GAO-10-157SU, 
the draft report titled: Aviation Security: Efforts to Validate 
Aspects of TSA 's Screening of Passengers by Observation Techniques 
(SPOT) Program Underway, But Opportunities Exist to Strengthen 
Validation and Address Operational Changes. The Transportation 
Security Administration (TSA) appreciates the U.S. Government 
Accountability Office's work in planning and conducting its review and 
issuing this report. 

TSA deployed the SPOT program in an effort to mitigate the threat of 
individuals with potentially hostile intent from boarding a commercial 
airplane and causing harm. Congress has encouraged the use of behavior 
recognition to enhance aviation security and has provided resources to 
support its implementation and expansion. The SPOT program fulfills 
the mandate of Section 1611 of the Implementing Recommendations of the 
9/11 Commission Act, P.L. 11053, that "TSA shall provide advanced 
training to the transportation security officers for the development 
of specialized security skills, including behavior observation and 
analysis...in order to enhance the effectiveness of layered 
transportation security measures." 

Intelligence continues to show there is no specific terrorist profile. 
In a March 10, 2010, hearing before the Senate Homeland Security and 
Governmental Affairs Committee, TSA Acting Administrator Gale Rossides 
highlighted the challenge faced by TSA leaders in "balancing the 
requirement to screen all passengers and to actually focus our 
officers' attention on the right passengers." TSA designed SPOT to 
increase its ability to focus on the "right passengers" by identifying 
persons exhibiting behaviors and appearances that may indicate stress, 
fear, and deception, and distinguish them from other travelers. 

SPOT is Based on Scientific Research and Law Enforcement Practices: 

TSAs development and deployment of SPOT was a planned and deliberate 
process based on more than 3 years of operational test-bed assessment 
of SPOT at Boston's Logan International Airport from June 2003 until 
nationwide rollout began in fiscal year (FY) 2007. TSA carefully 
developed SPOT by using selective behaviors recognized within both the 
scientific and law enforcement communities as displaying stress, fear, 
and deception. A SPOT working group, made up of various TSA and U.S. 
Department of Homeland Security (DHS) components,[Footnote 133] was 
created in February 2004. Other organizations, such as the 
Massachusetts State Police, the Federal Bureau of Investigation (FBI) 
Behavioral Sciences Unit, and the Federal Law Enforcement Training 
Center, were also involved in SPOT development. Through these working 
groups, TSA has developed and finalized SPOT standard operating 
procedures (SOPs) for a common ability to assess behaviors indicating 
hostile intent for both aviation and mass transit modes of 
transportation. TSA continues to consult with its SPOT working group 
partners as it updates the procedures and science behind the program. 

Decades of scientific research have shown the behaviors to be 
universal in their manifestation. In fact, the DHS Science and 
Technology Directorate (S&T) completed a study on suicide bomber 
indicators in July 2009 that illustrates a very high degree of overlap 
between operationally reported suicide bomber indicators and TSA SPOT 
behaviors. This result further bolsters TSA's contention that the SPOT 
program draws from the best practices of many defense, intelligence, 
and law enforcement organizations. 

SPOT Scientific Validation is Ongoing: 

S&T began research in 2007 to examine the validity of the SPOT 
program. The series of studies involved in this research is designed 
to assess the validity of the SPOT scoring system, including the use 
of individual behavioral indicators to identify high-risk travelers. 
More specifically, S&T's research plan aims to examine the extent to 
which these behavioral indicators are appropriate for screening 
purposes and lead to appropriate and correct security decisions. When 
this study is complete, SPOT will be one of the most, if not the most, 
rigorously tested behavior-based security screening programs in 
existence. 

Results of this work will establish a scientific basis of the extent 
to which the SPOT program, including its instrument and methods, such 
as the SPOT Referral Report and SOPs, are valid. Although it is 
challenging to establish the validity of a deterrent program in which 
the outcomes of interest are extremely rare, critical elements of 
reliability and validity will be rigorously assessed. Of particular 
importance is the evaluation of criterion-related validity, or the 
extent to which travelers are correctly selected for screening based 
on the SPOT scoring system. Establishing this degree of classification 
accuracy justifies the use of the SPOT program to discriminate high-
risk travelers from low-risk travelers. Regardless of any other 
metrics, the extent to which the SPOT scores accurately identify high-
risk travelers is critically important to program validity. 

Following criterion-related validity, the next central element of 
validity is the consistency of implementation of the instrument and 
program. This will be examined in a variety of ways, including an 
investigation of the consistency in the operational use of SPOT 
behavioral indicators Behavior Detection Officers (BDOs) and across 
locations and time periods, all of which represents reliability 
assessment. Finally, construct-related validity, or the extent to 
which the SPOT program behaviors truly represent the expressions of 
high-risk travelers, will be examined by comparing the SPOT behaviors 
to similar instruments in use for the same purpose. S&T's July 2009 
study of suicide bomber indicators was the first step in evaluating 
construct-related validity. 

This research is expected to be completed in FY 2011. TSA understands 
that after this validation is complete, there will be other areas 
where further research should be conducted, and it is TSA's intention 
to complete this research. 

National Academy of Sciences (NAS) Report Does Not Represent an 
Exhaustive or Definitive Review of the Research or Operational 
Literature on Behavioral Screening: 

TSA would like to specifically address a few comments in the GAO-10-
157SU report that we believe are inaccurate. The report draws heavily 
from a National Academy of Sciences (NAS) report which is being 
improperly relied upon. As the sponsor of the NAS study, DHS S&T 
questioned its findings, stating that the study lacked sufficient 
information for its conclusions because the NAS study principally 
focused on privacy as it relates to behavioral surveillance—not on 
behavioral surveillance technology itself. The study was not intended 
to, and the results do not represent an exhaustive or definitive 
review of the research or operational literature on behavioral and 
physiological screening, including recent findings from unpublished 
DHS, defense, and intelligence community studies. Furthermore, it 
should be noted that the report did not study the SPOT program, nor 
did any of the researchers conduct interviews with SPOT program 
personnel. 

Additionally, GAO states that "DHS S&T could not provide us with 
specific contacts related to the sources of this research." This 
statement is not accurate. The record should reflect that DHS S&T 
provided all requested documents that represented S&T-sponsored 
research and for which S&T possessed the requisite release authority. 
DHS was not able to release specific documents related to research for 
which it was not the originator. 

The report further states that the audit team was unable to use the 
SPOT referral data to assess whether any behavior or combination of 
SPOT behaviors could be used to reliably predict the final outcome of 
an incident involving the use of SPOT. However, DHS S&T was able to 
successfully conduct some preliminary analysis of the SPOT referral 
database. Prior to analysis of the SPOT reports, S&T worked with TSA 
to verify the scores assigned to each indicator with the SPOT score 
sheets and to rescore the pertinent sections and total accordingly for 
nearly 100,000 operational reports from 2008. While random errors were 
noted, errors in large databases that require manual entry are not 
uncommon. Convention suggests that large databases like this typically 
include an error rate of 3 to 5 percent. As long as such errors are 
random, the analytical method is robust enough to account for random 
errors in this range. 

In conclusion, TSA strongly believes that behavior detection is a 
vital layer in its aviation security strategy, and will continue to 
strengthen as the program matures. Leaders within the community of 
behavior detection researchers agree. TSA appreciates GAO's work to 
identify opportunities to enhance the SPOT program, and we will 
continue to work diligently to address the issues identified by GAO. 
Our ongoing progress demonstrates our commitment to TSA's mission of 
securing our Nation's transportation systems. 

We also appreciate the opportunity to provide you with, in 
collaboration with OHS S&T, comments to GAO's audit recommendations. 

Recommendation 1; To help ensure that SPOT is based on valid 
scientific principles that can be effectively applied in an airport 
environment, we (GAO) recommend that the Secretary of Homeland 
Security convene an independent panel of experts to review the 
methodology of the S&T Directorate study on the SPOT program before 
the study is implemented to determine whether the study's methodology 
is sufficiently comprehensive to validate the SPOT program. This 
assessment should include appropriate input from other federal 
agencies with expertise in behavior detection and relevant subject 
matter experts. 

Concurs: The U.S. Department of Homeland Security (DHS) Science & 
Technology Directorate's (S&T) current validation process includes an 
independent and comprehensive review of the ongoing SPOT study to be 
conducted in support of and in collaboration with the TSA SPOT 
program. The assessment will include input from other Federal agencies 
with expertise in behavior detection and relevant subject matter 
experts. S&T will work with TSA to present the SPOT validation project 
to the panel, produce a report summarizing the panel's 
recommendations, and implement pertinent suggestions in FY 2010. 

GAO further recommends that if this research determines that the SPOT 
program has a scientifically validated basis for using behavior 
detection for counterterrorism purposes in the airport environment, 
then the TSA Administrator take the following actions: 

Recommendation 2: Conduct a comprehensive risk assessment to include 
threat, vulnerability, and consequence of airports nationwide to 
determine the effective deployment of SPOT If TSA's ongoing Aviation 
Modal Risk Assessment lacks this information. 

Concur: TSA's Aviation Modal Risk Assessment (AMRA) is designed to 
evaluate the transportation security risk landscape and compare it to 
other modes. However the AMRA does not evaluate risk effectiveness of 
countermeasures or optimal deployment strategies. For the Aviation 
mode, TSA uses the Risk Management Analysis Tool (RMAT), a risk 
simulation model based on laboratory and operational data that 
evaluates risk using threat inputs, vulnerability information, and 
consequence estimates. TSA is in the process of conducting an initial 
risk analysis on the SPOT program using RMAT. The risk analysis is 
based on the initial SPOT validation results and will be updated as 
the validation study results are finalized. 

Recommendation 3: Perform a cost-benefit analysis of the SPOT program 
including a comparison of the SPOT program with other security 
screening programs, such as random screening, or already existing 
security measures. 

Concur: The SPOT program will use RMAT to perform a cost-benefit 
analysis of Behavior Detection Officers (BD0s) as a countermeasure. 
The first step in the process is the initial risk assessment that is 
being conducted on the SPOT program using RMAT. For the cost-benefit 
analysis, costs will be defined as the 5-year total cost of the 
countermeasure across the aviation system. Benefit will be defined as 
risk-reduction across the aviation security system against a portfolio 
of scenarios. TSA is currently developing an initial cost-benefit 
analysis for a variety of passenger-screening countermeasures 
including BDOs using the RMAT tool as a basis for analysis. BDOs' 
flexibility across a variety of risk scenarios suggests that behavior 
detection is a cost-effective countermeasure. 

Recommendation 4. Revise and implement the SPOT strategic plan by 
incorporating risk assessment information, identifying cost and 
resources, linking it to other related TSA strategic documents, 
describing how SPOT is integrated and implemented with TSA's other 
layers of aviation security, and providing guidance on bow to 
effectively link the roles, responsibilities and capabilities of 
federal, state, and local officials providing program support. 

Concur: The RMAT risk analysis of the BDO program is assisting the 
SPOT program in identifying other countermeasure capabilities that are 
linked to the behavior detection capability. This analysis will allow 
the SPOT program to develop a revision to the SPOT strategic plan that 
will incorporate the elements identified in the recommendation. 

Recommendation 5: Study the feasibility of using airport checkpoint-
surveillance video recordings of individuals transiting checkpoints, 
and who were later charged with or pleaded guilty to terrorism-related 
offenses, to enhance its understanding of terrorist behaviors in the 
airport checkpoint environment. 

Concur: TSA will study the feasibility of using checkpoint 
surveillance video recordings of individuals transiting checkpoints, 
and who were later charged with or pleaded guilty to terrorism-related 
offenses. TSA agrees that this could be a useful tool in understanding 
terrorist behaviors in the checkpoint environment. 

Additionally, TSA is currently working with DHS S&T/Human Factors to 
conduct operational video validation of the SPOT program. TSA will use 
a variety of video case studies to validate the SPOT program 
including, if possible, reviewing video of terrorists transiting the 
TSA checkpoint. It is exceedingly rare, however, for video cameras to 
capture terrorists transiting TSA checkpoints. Unfortunately, this 
factor significantly reduces the feasibility of conducting these case 
studies. 

GAO also recommends that concurrent with the DIIS S&T Directorate 
study of SPOT, and an independent panel assessment of the soundness of 
the methodology of the S&T study, the TSA Administrator take the 
following actions: 

Recommendation 6: Provide guidance in the SPOT SOP or other TSA 
directive to BDOs, or other TSA personnel, on inputting data into the 
Transportation Information Sharing System (TISS) and set milestones 
and a timeframe for deploying Transportation Information Sharing 
System access to SPOT airports so that TSA and intelligence community 
entities have information from all SPOT Law Enforcement officer (LEO) 
referrals readily available to assist in "connecting the dots" and 
identifying potential terror plots. 

Concur: TSA is currently undergoing a revision of the SPOT Standard 
Operating Procedure (SOP). The SOP will provide guidance directing the 
input of BDO data into TISS. TSA anticipates release of the updated 
SPOT SOP in FY 2010. Additionally, TSA is currently drafting a formal 
plan to include milestones and a timeframe for deploying T1SS access 
to all SPOT airports. 

Recommendation 7: Implement the steps called for in the TSA Office of 
Security Operations Business plan to develop a standardized process 
for allowing BDOs or other designated airport officials to send 
information to TSA's Transportation Security Operations Center (TSOC) 
about passengers whose behavior indicates that they may pose a threat 
to security, and provide guidance on how designated TSA officials are 
to receive information back from the Transportation Security 
Operations Center. 

Concur: TSA has convened a working group made up of members of the 
Office of Security Operations, Office of Chief Counsel, Office of 
Intelligence, and the Office of Law Enforcement/Federal Air Marshal 
Service (FAMS) to address this recommendation. TSA is developing a 
system and procedure for sending and receiving information from the 
TSOC and anticipates having a system in place in FY 2010. It should be 
noted that information from BDO referrals has been transmitted to the 
TSOC previously; however, TSA agrees to institute a standardized 
process. 

Recommendation 8: Utilize all of the databases available to the 
Transportation Security Operations Center when running passengers who 
rise to the level of a LEO referral against intelligence and criminal 
databases. 

Concur in principle: TSA has convened a working group composed of 
members of the Office of Security Operations, Office of Chief Counsel, 
Office of Intelligence, and the Office of Law Enforcement/FAMS to 
address this recommendation. This group will conduct a feasibility 
study during FY 2010 to examine if this recommendation can be fully 
implemented. This study will look at the various authorities, 
permissions, and limitations of each of the databases or systems cited 
in this audit. Access to some of the systems, such as Criminal History 
Record Check (CHRC), requires more justification than a BDO referral. 
Because some of the databases or systems contain classified 
information, TSA will also need to adopt a communication strategy to 
transmit the passenger information back and forth between the BDO and 
TSOC. TSA will work on a process to collect the passenger information, 
verify the passenger's identity, through checks of databases, and 
analyze that information to determine if the passenger is the subject 
of an investigation and may pose a risk to aviation. 

Recommendation 9: Establish a plan that includes objectives, 
milestones, and timeframes to develop outcome-oriented performance 
measures to help refine the current methods used by Behavior Detection 
Officers for identifying individuals who may pose a risk to the 
aviation system. 

Concur: TSA understands the value of outcome-oriented performance 
measures. However, as noted by GAO, there is difficulty in 
establishing these measures for a deterrence-based program. 
Nonetheless, TSA will consult with industry experts to develop outcome-
oriented performance measures. TSA will establish a plan that includes 
objectives, milestones. and timeframes, with an end result of 
producing outcome-oriented performance measures to help refine the 
current methods used by BDOs for identifying individuals who may pose 
a risk to the aviation system. 

Recommendation 10: Establish controls to help ensure completeness, 
accuracy, authorization, and validity of data collected during SPOT 
screening. 

Concur: In March 2010, TSA migrated the SPOT database to TSA's 
Performance Management Information System. This migration greatly 
enhances the SPOT program's capabilities, as they relate to 
completeness, accuracy, authorization, and validity of data collected 
during SPOT screening. Additional controls have been put in place to 
address the shortcomings of the previous database which were 
highlighted by GAO. TSA is also examining a technology solution to 
allow one-time transcription of all SPOT referral data. This will 
reduce the possibility of errors due to incorrect transcription from 
one medium to another. 

Recommendation 11: Establish timeframes and milestones for its plan to 
systematically conduct evaluations of the SPOT training program on a 
periodic basis. 

Concur: DHS S&T, in conjunction with TSA has sponsored a BDO Job Task 
Analysis (JTA). Outputs of the JTA will include Knowledge, Skills, 
Abilities, and Other characteristics of BDOs and Training Learning 
Objectives. These two items will enable TSA to conduct an in-depth 
training gap analysis. This analysis will begin immediately following 
completion of the JTA and will take approximately three months to 
complete. Upon completion of the training gap analysis, TSA will 
develop detailed project plans with milestones and schedules based on 
the scope of the overall curriculum development/revision effort. 

Sincerely yours, 

Signed by: 

Jerald E. Levine: 
Director: 
DHS GAO/OIG Liaison Office: 

[End of section] 

Appendix III: GAO Contacts and Staff Acknowledgments: 

GAO Contact: 

Stephen M. Lord, (202) 512-4379, or lords@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, David M. Bruno, Assistant 
Director, and Jonathan R. Tumin, managed this assignment. Ryan 
Consaul, Jeff C. Jensen, Kevin Remondini, and Julie E. Silvers made 
significant contributions to the work. Arthur James, Jr., Amanda 
Miller, and Douglas Sloane assisted with design, methodology, and data 
analysis. Chris Dionis assisted with issues related to training. 
Katherine Davis and Debra Sebastian provided assistance in report 
preparation; Tracey King and Tom Lombardi provided legal support; and 
Pille Anvelt and Barbara Hills developed the report graphics. 

[End of section] 

Footnotes: 

[1] BDOs must have at least 12 months experience as a TSO, or related 
security work experience, and must pass a BDO training course. 

[2] TSA cautions that the applicability of El Al's security processes 
to those used by TSA is constrained by differences in the scale of El 
Al's worldwide operations and the flexibilities that El Al has in 
implementing security processes compared to constraints on TSA. For 
example, El Al security screeners are encouraged to spend as much time 
with passengers as needed, and are not concerned whether passengers 
experience delays in boarding an aircraft. 

[3] For the purposes of this report, the term "TSA-regulated airport" 
refers to a U.S. airport operating under a TSA-approved security 
program. 

[4] Visible Intermodal Prevention and Response teams are comprised of 
federal air marshals, surface transportation security inspectors, 
TSOs, BDOs, and canines. 

[5] TSA classifies its regulated airports in the United States into 
one of five categories--X, I, II, III, and IV. Generally, category X 
airports have the largest number of passenger boardings and category 
IV airports have the least. 

[6] See H.R. Rep. No. 111-298 at 77 (2009) (Conf. Rep.). The 
conference report directed TSA to report, no later than 60 days after 
enactment, on the scientific basis for using behavior pattern 
recognition for observing airline passengers for signs of hostile 
intent, the effectiveness of the SPOT program in meeting its goals and 
objectives, and the justification for expanding the program. The 
conference report also directed us to review this report and to 
provide our findings to the Committees no later than 120 days after 
the TSA report is submitted. TSA completed its report to Congress on 
March 15, 2010. 

[7] National Research Council, Protecting Individual Privacy in the 
Struggle Against Terrorists: A Framework for Assessment (Washington, 
D.C.: National Academies Press, 2008). We reviewed the approach used 
and the information provided in this study and found the study and its 
results to be reliable for the purposes for which we used it in this 
report. 

[8] See appendix I for additional information on the experts we 
interviewed. 

[9] For reasons of scope, we did not assess the scientific basis of 
the methods and processes used by these agencies in their application 
of behavioral detection. 

[10] Although SPOT is based in some respects on El Al's aviation 
security program, El Al's processes differ in substantive ways from 
those used by the SPOT program. In particular, El Al does not use a 
list of specific behaviors with numerical values for each, or a 
numerical threshold to determine whether or not to question a 
passenger; rather, El Al security officers utilize behavioral 
indicators as a basis for interviewing all passengers boarding El Al 
passenger aircraft, and accessing relevant intelligence databases, 
when deemed appropriate. In addition, El Al officials told us that 
they train all their personnel--not just security officers--in 
elements of behavior analysis, and conduct covert tests of their 
employees' attentiveness at frequent intervals. According to these 
officials, El Al also permits what is termed "profiling," in which 
passengers may be singled out for further questioning based on their 
nationality, ethnicity, religion, appearance, or other ascriptive 
characteristics, but these are not the only basis on which a passenger 
may be questioned. In addition, El Al security officers are empowered 
to bar any passenger from boarding an aircraft. The scale of El Al 
operations is considerably smaller than that of major airlines 
operating within the United States. As of 2008, El Al had a fleet of 
34 aircraft. In Israel, El Al operates out of one hub airport, Ben-
Gurion International, and also flies to Eilat, a city in southern 
Israel; in contrast, there are 457 TSA-regulated airports in the 
United States. In 2008, El Al had passenger boardings of about 3.6 
million; in contrast, Southwest Airlines alone flew about 102 million 
passengers in the same year. 

[11] Unless otherwise noted in the report, we refer to the SPOT 
strategic plan issued in March 2007. 

[12] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: Feb. 1, 
2006). 

[13] OMB, Circular No. A-94, Guidelines and Discount Rates for Benefit-
Cost Analysis of Federal Programs (Washington, D.C.: October 1992); 
Circular No. A-4, Regulatory Analysis (Washington, D.C.: Sept. 2003). 

[14] See appendix I for additional details on the airports we visited. 

[15] At airports participating in TSA's Screening Partnership Program, 
private-sector contractors perform screening activities, including 
SPOT, in accordance with TSA requirements and oversight. See 49 U.S.C. 
§ 44920. Unless otherwise specified, references to TSOs include 
private-sector contract screeners. For more information, see GAO, 
Aviation Security: Progress Made to Set Up Program Using Private-
Sector Airport Screeners, but More Work Remains, GAO-06-166 
(Washington, D.C.: Mar. 31, 2006). 

[16] The TSA Current Airport Threat Assessment is a threat estimate 
designed to provide a snapshot of the current terrorist threat to 
airports in the United States as well as for major international 
airports serving as last points of departure for U.S. airlines. 

[17] GAO, Standards for Internal Controls in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[18] See GAO, Risk Management: Further Refinements Needed to Assess 
Risks and Prioritize Protective Measures at Ports and Other Critical 
Infrastructure, [hyperlink, http://www.gao.gov/products/GAO-06-91] 
(Washington, D.C.: Dec. 15, 2005) and Transportation Security: 
Comprehensive Risk Assessments and Stronger Internal Controls Needed 
to Help Inform TSA Resource Allocation, [hyperlink, 
http://www.gao.gov/products/GAO-09-492] (Washington, D.C.: Mar. 27, 
2009). 

[19] Federal Security Directors are the highest ranking TSA officials 
responsible for security operations at TSA-regulated airports. See 49 
U.S.C. § 44933. They and their assistants coordinate with both federal 
and nonfederal entities present at their airports, including the FAMS, 
the Drug Enforcement Administration, and CBP. When appropriate, 
Federal Security Directors may bar an individual from boarding an 
aircraft. 

[20] The SPOT referral data we analyzed covered the period May 29, 
2004, through August 31, 2008. These were the data available at the 
time of our analysis. 

[21] GAO, Human Capital: A Guide for Assessing Strategic Training and 
Development Efforts in the Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1, 
2004). 

[22] See Pub. L. No. 107-71, 115 Stat. 597 (2001). For purposes of 
this report, "commercial aircraft" refers to a U.S. or foreign-based 
air carrier operating under TSA-approved security programs with 
regularly scheduled passenger operations to or from a U.S. airport. 

[23] See 49 C.F.R. pt. 1542. Some commercial airports with fewer than 
2,500 annual enplanements (passengers boarding an aircraft) do not 
have TSA-approved screening processes. Enplanements are the number of 
paying passengers on a scheduled or nonscheduled (charter) flight. 
Infants and airline personnel are not included. A stop at an airport 
is not considered an enplanement if the passenger does not transfer 
aircraft. 

[24] Private-sector screeners under contract to and overseen by TSA, 
and not TSOs, perform screening activities at airports participating 
in TSA's Screening Partnership Program. See 49 U.S.C. § 44920. 

[25] These requests would typically be made to the law enforcement 
entity employing the LEO, such as the airport authority police 
department. The department would have a computer that can access NCIC. 
According to the FBI's website, the NCIC database consists of 19 files 
or databases. Seven are property files which contain records for 
articles, boats, guns, license plates, securities, vehicles, and 
vehicle and boat parts. Twelve are person files are the Convicted 
Sexual Offender Registry, Foreign Fugitive, Identity Theft, 
Immigration Violator, Missing Person, Protection Order, Supervised 
Release, Unidentified Person, U.S. Secret Service Protective, Violent 
Gang and Terrorist Organization, and Wanted Person Files. The 
Interstate Identification Index, which contains automated criminal 
history record information, is also accessible through the same 
network as NCIC. The Violent Gang and Terrorist Organization file 
includes the names of known or suspected terrorists. 

[26] See 49 CFR §§ 1542.215, .217. 

[27] TSA-regulated airports have regular commercial passenger service 
and comply with TSA regulations for passengers and their property in 
order to operate. 

[28] National Research Council, Protecting Individual Privacy in the 
Struggle Against Terrorists: A Framework for Assessment (Washington, 
D.C.: National Academies Press, 2008). The report's preparation was 
overseen by the National Academy of Sciences Committee on Technical 
and Privacy Dimensions of Information for Terrorism Prevention and 
Other National Goals. Although the report addresses broader issues 
related to privacy and data mining, a senior National Research Council 
official stated that the committee included behavior detection as a 
focus because any behavior detection program could have privacy 
implications. 

[29] Specifically, the report states that the scientific support for 
linkages between behavioral and physiological markers and mental state 
is strongest for elementary states, such as simple emotions; weak for 
more complex states, such as deception; and nonexistent for highly 
complex states, such as when individuals hold terrorist intent and 
beliefs. 

[30] DHS's S&T Directorate could not provide us with specific contacts 
related to the sources of this research. 

[31] Peer review is the process of subjecting an author's scholarly 
work, research, or ideas to the scrutiny of others who are experts in 
the same field. Such review is considered a form of scientific 
validation. 

[32] For example, we reported that the National Institutes of Health 
did not post its researchers' final reports because the risks 
associated with posting results that have not been scrutinized and 
validated by peer review are too great. See GAO, University Research: 
Most Federal Agencies Need to Better Protect against Financial 
Conflicts of Interest, [hyperlink, 
http://www.gao.gov/products/GAO-04-31] (Washington, D.C.: November 
2003). 

[33] Dr. Ekman is professor emeritus of psychology at the University 
of California Medical School, San Francisco, and is considered one of 
the world's foremost experts on facial expressions. His books include: 
Emotions Revealed: Recognizing Faces and Feelings to Improve 
Communications and Emotional Life (New York: Holt and Company, 2003); 
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking 
the Face: A Guide to Recognizing Emotions from Facial Clues (Englewood 
Cliffs, N.J.: Prentice-Hall, 1975). Dr. Ekman has published more than 
100 articles. 

[34] Mr. Walters is the author of the Principles of Kinesic Interview 
and Interrogation: 2ND Edition as well as numerous training materials 
related to interviewing and interrogation techniques. 

[35] Dr. Givens is the director of the nonprofit Center for Nonverbal 
Studies, in Spokane, Washington. Dr. Givens is the author of Love 
Signals: A Practical Field Guide to the Body Language of Courtship 
(St. Martin's, New York, 2005) and Crime Signals: How to Spot a 
Criminal Before You Become a Victim (St. Martin's, 2008). The Center's 
Web site links to Dr. Givens' online reference tool, The Nonverbal 
Dictionary of Gestures, Signs and Body Language Cues. Dr. Givens said 
that he had did not know which nonverbal indicators had been selected 
by TSA for use in SPOT, that he had not been asked by TSA to review 
their choices from his list, or to review other aspects of the SPOT 
program. According to Dr. Givens, attempting to detect more than nine 
nonverbal indicators would be difficult for most individuals, even 
those trained in behavior detection. 

[36] Dr. Frank is Associate Professor, Department of Communication, 
College of Arts and Sciences, at the University at Buffalo, State 
University of New York. He is on the Advisory Board of the 
University's Center for Unified Biometrics and Sensors, and has 
conducted research supported by DHS, the Defense Advanced Research 
Projects Agency, and the National Science Foundation. Dr. Frank told 
us that he had observed SPOT at an airport and had some coordination 
with TSA. However, he said that he had not reviewed the SPOT training 
curriculum or the SPOT scoring system. Dr. Frank stated that no study 
has been performed to validate use of behavior detection in an airport 
setting. 

[37] According to DHS's S&T Directorate, it completed a study on 
suicide bomber indicators in July 2009. The program manager stated 
that they reviewed 157 documents and identified approximately 1,200 
suicide indicators, which were similar to SPOT suicide bomber 
indicators. S&T stated that the study provides preliminary support for 
the detection of suicide bomber indicators and that SPOT represents 
best practices from defense and intelligence organizations. 

[38] According to TSA, the FBI participated in discussions related to 
SPOT's development in 2006. 

[39] American Institutes for Research, Behavioral Indicators Related 
to Deception in Individuals with Hostile Intentions: Interim Results 
(Washington, D.C.: February 2008). According to S&T officials, this 
review included research conducted prior to 2005. 

[40] DHS could not provide us with specific contacts related to the 
sources of this research; we were therefore unable to determine the 
extent to which it has demonstrated linkages between behavioral and 
physiological indicators and deception. 

[41] Statement of the Under Secretary (Acting), DHS S&T Directorate, 
before the Subcommittee on Homeland Security, Committee on 
Appropriations, U.S. House of Representatives, March 26, 2009. 

[42] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[43] See GAO, Oil and Gas Royalties: The Federal System for Collecting 
Oil and Gas Revenues Needs Comprehensive Reassessment, [hyperlink, 
http://www.gao.gov/products/GAO-08-691] (Washington, D.C.: Sept. 3, 
2008). GAO, Combating Nuclear Smuggling: Additional Actions Needed to 
Ensure Adequate Testing of Next Generation Radiation Detection 
Equipment, [hyperlink, http://www.gao.gov/products/GAO-07-124T] 
(Washington, D.C.: Sept. 18, 2007). GAO, Space Operations: NASA 
Efforts to Develop and Deploy Advanced Spacecraft Computers, 
[hyperlink, http://www.gao.gov/products/GAO/IMTEC-89-17] (Washington, 
D.C.: Mar. 31, 1989). GAO, Quadrennial Defense Review: Future Reviews 
Could Benefit from Improved Department of Defense Analyses and Changes 
to Legislative Requirements, [hyperlink, 
http://www.gao.gov/products/GAO-07-709] (Washington, D.C.: Sept. 14, 
2007). GAO, Coast Guard: Challenges for Addressing Budget Constraints, 
[hyperlink, http://www.gao.gov/products/GAO/RCED-97-110] (Washington, 
D.C.: May 1997). 

[44] DHS's NIPP defines risk as a function of threat, vulnerability, 
and consequence. Threat is an indication of the likelihood that a 
specific type of attack will be initiated against a specific target or 
class of targets. Vulnerability is the probability that a particular 
attempted attack will succeed against a particular target or class of 
targets. Consequence is the effect of a successful attack. 

[45] As updated in 2009, the NIPP states that to be complete a risk 
assessment is to assess threat, vulnerability, and consequence for 
every defined risk scenario. However, because the original 2006 
version of the NIPP described risk assessments that included all three 
components as "credible," our previous reports use this term rather 
than "complete" (see GAO, Transportation Security: Comprehensive Risk 
Assessments and Stronger Internal Controls Needed to Help Inform TSA 
Resource Allocation, [hyperlink, 
http://www.gao.gov/products/GAO-09-492] (Washington, D.C.: Mar. 27, 
2009)). 

[46] We reported in March 2009 that TSA does not assign uncertainty or 
varying levels of confidence associated with the intelligence 
information used to identify threats to the transportation sector and 
guide TSA's planning and investment decisions. Since TSA's 
intelligence products have not assigned confidence levels to its 
analytic judgments, it is difficult for TSA to correctly prioritize 
its tactics and investments based on uncertain intelligence. In March 
2009, we recommended that TSA work with the Director of National 
Intelligence to determine the best approach for assigning uncertainty 
or confidence levels to analytic intelligence products and to apply 
this approach. TSA agreed with this recommendation and said it has 
begun taking action to address it. See [hyperlink, 
http://www.gao.gov/products/GAO-09-492]. 

[47] The AMRA is being developed by TSA pursuant to the National 
Strategy for Aviation Security, which was issued according to Homeland 
Security Presidential Directive-16. HSPD-16 directs the production of 
an overarching national strategy to optimize and integrate government- 
wide aviation security efforts. AMRA was previously known as the Air 
Domain Risk Assessment or ADRA. 

[48] Commercial aviation includes that sector of the nation's civil 
aviation system that provides for the transportation of individuals by 
scheduled or chartered operations for a fee, including air carriers 
and airports. General aviation includes all civil aviation other than 
commercial and military operations, including flight operations such 
as personal/family transportation, emergency services, wildlife and 
land surveys, traffic reporting, agricultural aviation, firefighting, 
and law enforcement. Air cargo is defined as cargo carried on 
passenger and all-cargo aircraft. 

[49] In addition, TSA states that its risk management analysis toolset 
may also be useful in prioritizing BDO deployments since the toolset 
analyzes various scenarios for which the use of BDOs may be applicable. 

[50] See GAO, Reexamining Regulations: Opportunities Exist to Improve 
Effectiveness and Transparency of Retrospective Reviews, [hyperlink, 
http://www.gao.gov/products/GAO-07-791] (Washington, D.C.: July 16, 
2007). 

[51] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February 
2006); OMB, Circular No. A-94, Guidelines and Discount Rates for 
Benefit-Cost Analysis of Federal Programs (Washington, D.C.: October 
1992); OMB, Circular No. A-4, Regulatory Analysis (Washington, D.C.: 
September 2003). 

[52] A pilot test is a preliminary test or study to try out procedures 
and discover problems before the main study begins. This enables 
researchers to make last minute corrections and adjustments. In a 
pilot, the entire study with all of its instruments and procedures is 
conducted in miniature. See W.P. Vogt, Dictionary of Statistics and 
Methodology: A Nontechnical Guide for the Social Sciences (Newbury 
Park: Sage Publications, 1993).  

[53] The increase rate for TSA's other screening operations combined 
was about 0.27 percent from fiscal year 2007 to fiscal year 2009 (from 
$3.727 billion to $3.737 billion, a $10 million increase). The 
screening operations account includes privatized screening; passenger 
and baggage screener performance, compensation, and benefits; screener 
training and other; human resource services; and checkpoint support. 

[54] See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.). 

[55] [hyperlink, http://www.gao.gov/products/GAO-04-408T]. 

[56] Pub. L. No. 103-62, 107 Stat. 285 (1993). 

[57] See GAO, National Preparedness: FEMA Has Made Progress, but Needs 
to Complete and Integrate Planning, Exercise, and Assessment Efforts, 
[hyperlink, http://www.gao.gov/products/GAO-09-369] (Washington, D.C.: 
Apr. 30, 2009). 

[58] TSA, Strategic Plan for Behavior Detection Program (Washington, 
D.C.: 2007). 

[59] TSA, Office of Security Operations, Strategy Deployment 
Organizational Business Plan for Fiscal Year 2010 (Washington, D.C.: 
Apr. 8, 2008). 

[60] Within the Transportation System Sector-Specific Plan, the 
aviation implementation plan outlines transportation security goals 
and key objectives with associated programs within the aviation 
community. The plan notes that SPOT is intended to identify suspicious 
activities within the aviation domain. 

[61] TSA issued its Passenger Checkpoint Screening Program Strategic 
Plan in August 2008 to outline its strategy and approach for 
implementing advanced security capabilities in airport checkpoints 
using a combination of people, processes, and technology at all 
airport checkpoints. The plan cites TSA's behavior detection 
capability as one of three strategic initiatives. 

[62] The Transportation Security Operations Center is the central 
operations and information-gathering point for TSA across the nation; 
it serves as a 24/7-point of contact for all transportation security 
concerns, providing access to multiple criminal justice and 
intelligence-related databases. 

[63] DHS, Privacy Impact Assessment for the TSA Operations Center 
Incident Management System (Washington, D.C.: July 8, 2008), and 
Privacy Impact Assessment for the Screening of Passengers by 
Observation Techniques (SPOT) Program (Washington, D.C.: Aug. 5, 2008). 

[64] DHS, Privacy Impact Assessment for the Transportation Information 
Sharing System (Washington, D.C.: June 2008). The Transportation 
Information Sharing System is a database owned by the TSA's FAMS 
component; the data entered into it may be shared with other federal, 
state, or local law enforcement and law enforcement support entities. 
Federal air marshals file reports related to the observation of 
suspicious activities and input this information, as well as incident 
reports submitted by airline employees and other individuals within 
the aviation domain, into the Transportation Information Sharing 
System. 

[65] Because the SPOT program has not been scientifically validated, 
it cannot be determined if these anecdotal results were better than if 
passengers had been pulled aside at random, rather than as a 
consequence of being identified for further screening by BDOs. 

[66] Some details about the process were deleted because TSA 
considered them to be Sensitive Security Information. 

[67] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[68] Some details from our analysis were deleted because TSA 
considered them to be Sensitive Security Information. 

[69] This information can be submitted about individuals whose 
suspicious activity resulted in BDO or LEO referral. See TSA's July 
2008 Transportation Security Operations Center Privacy Impact 
Assessment. 

[70] Some details of our survey results were deleted because TSA 
considered them to be Sensitive Security Information. 

[71] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 
For example, information should be recorded and communicated to 
management and others within the entity who need it and in a form and 
within a time frame that enables them to carry out their internal 
control and other responsibilities. Further, effective information 
technology management is critical to achieving useful, reliable, and 
continuous recording and communication of information. 

[72] TSA, Strategy Deployment, Organizational Business Plan, Office of 
Security Operations, Fiscal Year 2010 (Washington, D.C.: April 2008). 
According to TSA, the Office of Security Operations is the operational 
arm of TSA and employs the largest TSA workforce. It is responsible 
for airport checkpoint and baggage screening operations as well as 
other special programs designed to secure all assigned transportation 
modes. 

[73] In March 2010, TSA told us that over the next 18 months, it will 
expand access to information classified up to the "Secret" level to an 
additional 10,000 TSA personnel, including all BDOs, all SPOT 
Transportation Security Managers (who are responsible for the local 
operations of the SPOT program and supervision of the BDOs), and all 
Supervisory TSOs (who directly supervise TSOs and the screening 
process). 

[74] The Terrorist Screening Database is the central terrorist 
watchlist consolidated by the FBI's Terrorist Screening Center and 
used by multiple agencies to compile their specific watchlists and for 
screening. 

[75] The other databases available to TSA are omitted because TSA 
considered them to be Sensitive Security Information. 

[76] The No-Fly list is used to identify individuals who should be 
prevented from boarding an aircraft. The No-Fly and Selectee lists 
contain applicable records from the FBI's Terrorist Screening Center 
consolidated database of known or suspected terrorists. Pursuant to 
Homeland Security Presidential Directive 6, dated September 16, 2003, 
the Terrorist Screening Center--operational since December 2003 under 
the administration of the FBI--was established to develop and maintain 
the U.S. government's consolidated terrorist screening database (the 
watch list) and to provide for the use of watch-list records during 
security-related screening processes. See GAO-08-136T, Aviation 
Security: TSA Is Enhancing Its Oversight of Air Carrier Efforts to 
Screen Passengers against Terrorist Watch-List Records, but Expects 
Ultimate Solution to Be Implementation of Secure Flight (Washington, 
D.C.: Sept. 9, 2008). 

[77] According to DHS, the Terrorist Identity Datamark Environment is 
the database maintained by the National Counterterrorism Center--the 
primary organization in the U.S. government for integrating and 
analyzing intelligence pertaining to terrorism and counterterrorism-- 
to serve as a central repository for all information on known or 
suspected international terrorists with the exception of purely 
domestic terrorism information. See, DHS, Office of Inspector General, 
The DHS Process for Nominating Individuals to the Consolidated 
Terrorist Watchlist (Washington, D.C.: February 2008). 

[78] DHS, National Infrastructure Protection Plan: Partnering to 
Enhance Protection and Resiliency (Washington, D.C.: 2009); 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]; and GAO, 
Performance Measurement and Evaluation: Definitions and Relationships, 
[hyperlink, http://www.gao.gov/products/GAO-05-739SP] (Washington, 
D.C.: May 2005). 

[79] The Project Management Institute, The Standard for Program 
Management© (2006). 

[80] GAO, Executive Guide: Effectively Implementing the Government 
Performance and Results Act, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-96-118] (Washington, D.C.: June 
1996). 

[81] See S. Rep. No. 111-31, at 56-57 (2009); see also S. Rep. No. 110-
396, at 59 (2008). 

[82] See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.). The report 
further directs that GAO review the report submitted by TSA and 
provide its findings to the committees no later than 120 days after 
the SPOT report is submitted to the committees. 

[83] Output measures help determine the extent to which an activity 
was performed as planned. Outcome-related measures are more robust 
measures because they provide a more comprehensive assessment of the 
success of the agency's efforts, as stated in DHS's 2009 NIPP. 

[84] OMB and the Council for Excellence in Government, Performance 
Measurement Challenges and Strategies (Washington, D.C.: June 18, 
2003). 

[85] The Office's primary work involves metrics infrastructure; it 
assists TSA programs, if requested, in developing applications to 
track quantitative measures, such as surrendered items. It also tracks 
data for its Management Objectives Report related to three areas: 
employees, security effectiveness, and efficiency. 

[86] The types of prohibited items found have included knives, guns, 
gun ammunition, certain chemicals, strike-anywhere matches, and 
certain liquids/gels/aerosols; other illegal items discovered include 
narcotics and fraudulent identity documents. 

[87] According to TSA, TSOs focus on detecting high-risk threats which 
have the ability to cause catastrophic damage to an airplane in flight 
(e.g., explosives). 

[88] OMB and the Council for Excellence in Government, Performance 
Measurement Challenges and Strategies (Washington, D.C.: June 18, 
2003). 

[89] We also found that the SPOT referral database had a number of 
weaknesses. TSA designated our discussion of these weakness as 
sensitive security information. 

[90] Our estimate of the total number of passengers who went through 
checkpoints is based on Bureau of Transportation Statistics data that 
we obtained for the airports at which SPOT was deployed during this 
period. Some figures were rounded. 

[91] The analysis included only flights leaving the United States. 
Department of Justice data show that more than 400 individuals have 
been convicted in the United States for terrorism-related offenses 
since September 11, 2001. We did not examine the travel itineraries of 
all these individuals. 

[92] The events included the Mumbai, India attack of 2008; a plot to 
attack the Quantico, Virginia, Marine base in 2008; an effort by five 
Americans to receive training and fight in Pakistan in December 2009; 
a plot to attack infrastructure in New York City in 2009; an effort to 
provide men and support for terrorists in Somalia in 2008; and an 
attack on a U.S. base in Afghanistan by an American who received 
training in Pakistan. We were unable to confirm whether BDOs were 
stationed at the checkpoints used by these individuals at the time 
they traveled. 

[93] See, GAO, Supply Chain Security: Feasibility and Cost-Benefit 
Analysis Would Assist DHS and Congress in Assessing and Implementing 
the Requirement to Scan 100 Percent of U.S.-Bound Containers, 
[hyperlink, http://www.gao.gov/products/GAO-10-12] (Washington, D.C.: 
Oct. 30, 2009). The Project Management Institute, A Guide to the 
Project Management Body of Knowledge. 

[94] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[95] G-Band, or Expert BDOs, have advanced to a lead role, are able to 
provide technical expertise on the SPOT program, and are one band away 
from a supervisory role. 

[96] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[97] TSA's Operational and Technical Training Division, within the 
Office of Security Operations, provides assistance with development 
and implementation of technical training for screening, Behavior 
Detection Officers, Bomb Appraisal Officers, the Aviation Direct 
Access Screening Program, and technical management training. 

[98] In May 2009, the title of the course was changed to "Additional 
Behavior Detection Techniques" because ABDT is actually a supplemental 
tool for BDOs to use during the Casual Conversation phase of SPOT 
Referral Screening. The course was formerly titled "Advanced Behavior 
Detection Techniques." Microfacial expressions are very brief facial 
expressions that can last as little as 1/25 of a second. 

[99] GAO, Human Capital: A Guide for Assessing Strategic Training and 
Development Efforts in the Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1, 
2004). 

[100] The TSA staff member attended the following external training 
courses: John Reid and Associates' Reid Techniques of Interrogation 
and Advanced Reid Techniques of Interrogation; Massachusetts State 
Police Academy's Basic Investigations and Professional Development 
Program Interview Techniques; International Security Defense Systems' 
Verification Agent for Virgin Atlantic Security Systems; New Mexico 
Technology, Materials and Research Center's Prevention and Response to 
Suicide Bomber Indicators; Abraxis Corporation's Detecting Deception 
and Eliciting Response; Langevin Learning Services' Instructional 
Techniques for New Instructors; Ekman Group's Understanding Emotions 
and Detecting Truth; Chameleon Associates' Suspicious Behavior 
Detection; and Federal Transit Administration's Terrorist Awareness, 
Recognition, and Response. 

[101] The purpose of the SPOT working group was to help refine the 
list of SPOT behaviors and to develop standard operating procedures 
and a concept of operations for the program. 

[102] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] and 
[hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[103] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[104] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[105] The SPOT program retains the discretion to train BDOs at a site 
other than one of the five training facilities if it is more fiscally 
responsible to do so. For example, if there are 15 BDO candidates at a 
single airport, the SPOT program will train them at that airport 
rather than sending them to a training facility. 

[106] Some SPOT instructors have been evaluated multiple times. While 
the SPOT program office provided us with print or electronic copies of 
all SPOT instructor evaluations, some forms contained only numeric 
ratings and no written comments; others had no numeric scores. Because 
instructor names were redacted from the evaluations, the numbers may 
include duplicates. Additionally, the evaluations containing written 
comments were not always filled out using complete sentences, making 
it difficult to ascertain the rater's assessment of the instructor. 

[107] SPOT Instructors are evaluated using a Quality Assurance 
Instructor Evaluation, TSA Form 1909. Using this form, the evaluator 
assigns either 0 (zero) points, 0.5 points, or 1 point for each of 57 
ratable items depending on whether the instructor meets the standard 
as written, needs improvement to meet the standard, or does not meet 
the standard. The total points are then entered into a formula that 
generates a percentage. This percentage is used to determine the 
overall rating. Instructors receiving a score of 95 percent to 100 
percent are rated as exceeds expectations; 85 percent to 94 percent 
are rated as meets expectations; 75 percent to 84 percent are rated as 
needs improvement; and 0 percent to 74 percent are rated as 
unsatisfactory. 

[108] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[109] TSA, Operational and Technical Training Division, Training 
Standards (Sept. 28, 2006). 

[110] The Project Management Institute, The Standard for Program 
Management© (2006). 

[111] The training gap analysis identifies gaps in the training 
curriculum. 

[112] This estimate assumes that there would be no further increases 
for SPOT over the next 5 years above the requested $232 million level 
for fiscal year 2011. However, to stay even with inflation, the 
allocation would likely increase somewhat each year. 

[113] The National Research Council is a component of the National 
Academy of Sciences, a part of a private, nonprofit institution, the 
National Academies, which provide science, technology, and health 
policy advice under a congressional charter. 

[114] National Research Council, Protecting Individual Privacy in the 
Struggle Against Terrorists: A Framework for Assessment (Washington, 
D.C.: National Academies Press, 2008). The report's preparation was 
overseen by the National Academy of Science's 21-member Committee on 
Technical and Privacy Dimensions of Information for Terrorism 
Prevention and Other National Goals. We reviewed the approach used and 
the information provided in this study and found the study to be 
credible for our purposes. The contributors included recognized 
experts across a variety of fields, including William J. Perry, former 
Secretary of Defense, and Dr. Tara O'Toole, then-CEO and Director of 
the Center for Biosecurity of the University of Pittsburgh Medical 
Center, Professor of Medicine and of Public Health at the University 
of Pittsburgh. (Dr. O'Toole was subsequently nominated and confirmed 
as the Under Secretary of the DHS Science and Technology Directorate.) 

[115] National Research Council, Protecting Individual Privacy in the 
Struggle Against Terrorists: A Framework for Assessment (Washington, 
D.C.: National Academies Press, 2008). The report's preparation was 
overseen by the NAS's 21-member Committee on Technical and Privacy 
Dimensions of Information for Terrorism Prevention and Other National 
Goals. We reviewed the approach used and the information provided in 
this study and found the study to be credible for our purposes. The 
contributors included recognized experts across a variety of fields, 
including William J. Perry, former Secretary of Defense, and Dr. Tara 
O'Toole, then-CEO and Director of the Center for Biosecurity of the 
University of Pittsburgh Medical Center, Professor of Medicine and of 
Public Health at the University of Pittsburgh. (Dr. O'Toole was 
subsequently nominated and confirmed as the Under Secretary of DHS's 
Science and Technology Directorate. The National Research Council is a 
component of the National Academy of Sciences, a part of a private, 
nonprofit institution, the National Academies, which provide science, 
technology, and health policy advice under a congressional charter 

[116] Dr. Ekman is professor emeritus of psychology at the University 
of California Medical School, San Francisco, and is considered one of 
the world's foremost experts on facial expressions. His books include: 
Emotions Revealed: Recognizing Faces and Feelings to Improve 
Communications and Emotional Life (New York: Holt and Company, 2003); 
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking 
the Face: A guide to Recognizing Emotions from Facial Clues (Englewood 
Cliffs, N.J.: Prentice-Hall, 1975). Dr. Ekman has published more than 
100 articles. 

[117] Dr. Frank is Associate Professor, Department of Communication, 
College of Arts and Sciences, at the University at Buffalo, State 
University of New York. He is on the Advisory Board of the 
University's Center for Unified Biometrics and Sensors, and has 
conducted research supported by DHS, the Defense Advanced Research 
Projects Agency, and the National Science Foundation. 

[118] Dr. Givens is the director of the nonprofit Center for Nonverbal 
Studies, in Spokane, Washington. He is the author of Love Signals: A 
Practical Field Guide to the Body Language of Courtship (St. Martin's, 
New York, 2005) and Crime Signals: How to Spot a Criminal Before You 
Become a Victim (St. Martin's, 2008). The Center's Web site links to 
Dr. Givens' reference tool, The Nonverbal Dictionary of Gestures, 
Signs and Body Language Cues. 

[119] Dr. Matsumoto is a Professor, Department of Psychology at San 
Francisco State University, and is an associate of Dr. Ekman. 

[120] Dr. Wein is the Paul E. Holden Professor of Management Science 
at the Graduate School of Business, Stanford University. His homeland 
security-related work includes four papers in Proceedings of the 
National Academy of Sciences, on an emergency response to a smallpox 
attack, an emergency response to an anthrax attack, a biometric 
analysis of the US-VISIT Program, and an analysis of a bioterror 
attack on the milk supply. 

[121] DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February 
2006). 

[122] Office of Management and Budget (OMB), Circular No. A-94, 
Guidelines and Discount Rates for Benefit-Cost Analysis of Federal 
Programs (Washington, D.C.: October 1992); and Circular A-4, 
Regulatory Analysis (Washington, D.C.: September 2003). 

[123] [hyperlink, http://www.gao.gov/products/GAO-04-408T]. 

[124] Pub. L. No. 103-62, 107 Stat. 285 (1993). 

[125] TSA, Screening of Passengers by Observation Technique (SPOT) 
White Paper for the Department of Homeland Security (Washington, D.C.: 
Feb. 22, 2005). 

[126] The data from interviews of suspicious passengers by FAMS are 
inputted into the Transportation Information Sharing System, as are 
reports sent to FAMS from airline employees about suspicious 
passengers. 

[127] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[128] GAO, Transportation Security: Comprehensive Risk Assessments and 
Stronger Internal Controls Needed to Help Inform TSA Resource 
Allocation, [hyperlink, http://www.gao.gov/products/GAO-09-492] 
(Washington, D.C.: Mar. 27, 2009). 

[129] Federal Security Directors are the highest ranking TSA security 
officials at U.S. airports; Assistant Federal Security Directors are 
their assistants. Both are responsible for all aspects of security at 
airports, including coordination with federal and nonfederal law 
enforcement entities operating at airports, such as FAMS, the Drug 
Enforcement Administration, and CBP. 

[130] The Current Airports Threat Assessment is a threat estimate 
designed to provide a snapshot of the current terrorist threat to 
airports in the United States as well as for major international 
airports serving as last points of departure for U.S. airlines. 

[131] The Treasury Enforcement Communication System was designed to 
provide controlled access to a large database of information about 
suspects and to interface with a number of other law enforcement 
systems. These capabilities are provided to users through various 
applications, including the Inspection/Interagency Border Inspection 
System applications that facilitate passenger processing through the 
implementation of innovative border control technology. 

[132] GAO, Human Capital: A Guide for Assessing Strategic Training and 
Development Efforts in the Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1, 
2004). 

[133] Includes TSA's Office of Civil Rights, Office of Chief Counsel, 
and Privacy Office, and DHS's Policy Office and Transportation 
Security Laboratory. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: