This is the accessible text file for GAO report number GAO-07-623R entitled 'Aviation Security: TSA's Change to Its Prohibited Items List Has Not Resulted in Any Reported Security Incidents, but the Impact of the Change on Screening Operations Is Inconclusive' which was released on April 25, 2007. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. April 25, 2007: The Honorable Robert C. Byrd: Chairman: The Honorable Thad Cochran: Ranking Minority Member: Subcommittee on Homeland Security: Committee on Appropriations: United States Senate: The Honorable David E. Price: Chairman: The Honorable Harold Rogers: Ranking Minority Member: Subcommittee on Homeland Security: Committee on Appropriations: House of Representatives: The Honorable Hillary Rodham Clinton: United States Senate: Subject: Aviation Security: TSA's Change to Its Prohibited Items List Has Not Resulted in Any Reported Security Incidents, but the Impact of the Change on Screening Operations Is Inconclusive: The alleged August 2006 terrorist plot to detonate liquid explosives onboard multiple commercial aircraft bound for the United States from the United Kingdom has highlighted both the continued importance of securing the civil aviation system and the potential that improvised explosive devices (IED) may be smuggled onboard passenger aircraft. The Transportation Security Administration (TSA) has primary responsibility for ensuring the security of civil aviation, which includes the safety of passengers and flight crew.[Footnote 1] One measure TSA uses to protect the aviation system is prohibiting individuals from carrying items that it determines to be a threat to the aircraft and its passengers into an airport sterile area or onboard an aircraft either in their carry-on bag or on their person.[Footnote 2] To implement this measure, TSA maintains a prohibited items list that informs both the Transportation Security Officers (TSO) who conduct passenger screening and the traveling public of items that will not be allowed into an airport sterile area or onboard an aircraft. In December 2005, TSA revised its prohibited items list to allow passengers to carry: (1) metal scissors with pointed tips and a blade 4 inches or less in length as measured from the fulcrum; and (2) tools-- such as pliers, screwdrivers, and wrenches--7 inches or less in length (excluding crowbars, drills, hammers, and saws).[Footnote 3] TSA considers any incident that threatens the security or safety of an aircraft or its passengers and flight crew to be a security incident. These could include a range of activities onboard an aircraft such as disruptive passenger behavior, violence against a passenger or crew member, hijacking attempts, or the use of an improvised explosive device. By examining the security impacts of the change to the prohibited items list, this report considers the impacts that could result from a passenger attempting to use scissors or tools to hijack an aircraft or to commit other forms of violence onboard a flight. Such actions fall within TSA's statutory responsibility to ensure the safety and security of passengers and crew aboard aircraft. In accordance with Conference Report 109-699, which accompanied the fiscal year 2007 Department of Homeland Security (DHS) appropriations act,[Footnote 4] this report addresses the following questions: (1) What was TSA's basis for removing certain scissors and tools from the prohibited items list and what are stakeholder views on the change? (2) What have been the impacts, if any, of the removal of certain scissors and tools from the prohibited items list on the security of aircraft passengers and flight crew and on the effectiveness of checkpoint screening operations? To address these objectives, we analyzed TSA documentation and data, including TSA security incident reports, TSA written analyses related to the prohibited items list change, results of Threat Image Projection (TIP) testing,[Footnote 5] and data on training hours completed by TSOs. Although the TIP data we received had limitations, we believe that they are sufficiently reliable for the purposes of this report and that the data on training hours are sufficiently reliable as well. We also met with two Federal Security Directors (FSD) to obtain their views on the impact of the prohibited items list change on checkpoint screening operations.[Footnote 6] However, information obtained from our interviews with these FSDs cannot be generalized because we did not use random selection or representative sampling when determining which FSDs should be interviewed. In addition, we met with officials at the Federal Aviation Administration (FAA) and the Federal Air Marshals Service (FAMS)--a component of TSA--to obtain their views regarding the prohibited items list change. We also met with TSA officials to obtain information on their rationale behind the change. We spoke with 13 stakeholders within the aviation industry, including representatives of 4 domestic aviation associations, the largest association representing airline pilots in the United States, the largest association representing flight attendants in the United States, an association representing federal air marshals and other federal law enforcement officers, an international aviation association, and 5 aviation security experts. We also met with a major aircraft manufacturer to determine whether there are any major safety concerns related to the change to the prohibited items list. Finally, we incorporated aspects of a recently issued GAO report on passenger checkpoint screening procedures, which included a review of the factors TSA considered in modifying the prohibited items list and TSA's analysis supporting the December 2005 prohibited items list change.[Footnote 7] We conducted our work from November 2006 through March 2007 in accordance with generally accepted government auditing standards. More details about the scope and methodology of our work are presented in enclosure I. Results in Brief: TSA's stated purpose in removing certain scissors and tools from the prohibited items list was to shift TSO focus from items considered by TSA to pose a low threat (including certain scissors and tools) to items considered to pose a high threat, such as explosives. The change also was intended to better allocate TSA resources to implement other security measures that target explosives--a change supported by the majority of aviation industry stakeholders that we interviewed. TSA's decision to remove these items from the prohibited items list was based on the professional judgment of TSA officials that these items do not pose a significant threat to the security of the cockpit or to passengers and flight crew as well as internal studies that sought to examine, among other things, risks to flight security and considerations of customer concerns and screening efficiencies. As part of these internal studies, TSA collected data on the number and types of prohibited items surrendered at checkpoints and the time it takes for TSOs to conduct carry-on bag searches. In March 2007, we reported that TSA did not analyze these data to determine the extent to which TSO resources would actually be freed up to implement other security measures, nor did TSA analyze other relevant factors such as the amount of time taken to search for small scissors and tools and the number of TSOs conducting these searches. We recommended that TSA develop sound evaluation methods, when possible, that can help TSA determine whether proposed procedures would achieve their intended purpose.[Footnote 8] TSA concurred with the recommendation and stated that it plans to make better use of generally accepted research design principles and techniques when operationally testing proposed changes to screening procedures. Based on our analysis of TSA data for the third and fourth quarters of fiscal year 2005 (a 6-month period), we determined that TSOs spent, on average, less than 1 percent of their time--about 1 minute per day over the 6-month period--searching for the approximately 1.8 million sharp objects, other than knives and box cutters, that were found at passenger screening checkpoints between April 2005 and September 2005. Therefore, it may not have been accurate for TSA to assume that no longer requiring TSOs to search for small scissors and tools would significantly contribute to TSA's efforts to free up TSO resources that could be used to implement other security measures. TSA acknowledged that its data collection and analysis effort may not have been methodologically rigorous but stated that it did serve to provide insights regarding the type and quantity of items collected at the passenger checkpoint. TSA officials also stated that even if TSO resources were not freed up as intended, they continue to view their decision to allow small scissors and tools onboard aircraft as sound, particularly because their review of threat information determined that small scissors and tools do not pose a significant threat to aviation security. Additionally, 9 of the 13 aviation industry stakeholders whom we interviewed supported the removal of small scissors and tools from the prohibited items list because they believe small scissors and tools do not pose a risk to the security of the aircraft and stated that the change will increase TSA's focus on IEDs; the remainder disagreed, citing potential increased security risks. TSA officials acknowledged that small scissors and tools, as well as other items permitted onboard commercial aircraft, may potentially be used as weapons against passengers and flight crew, but stated that these items cannot be used to hijack an aircraft given the other layers of security in place, such as hardened cockpit doors that prevent unauthorized access to the flight deck. Based on our review of TSA security incident reports from the time period following the prohibited items list change (December 2005 through February 2007), there have been no reported security incidents onboard an aircraft involving the use of small scissors or tools. However, the impact of the prohibited items list change on security is uncertain because the absence of an event occurring involving the use of these items does not preclude the possibility that future occurrences could happen. In addition, with respect to the effectiveness of the change on checkpoint screening operations, it is not possible to determine this because the available data are inconclusive. As we reported in March 2007, TSA conducted informal studies 30, 60, and 90 days following the change and concluded that TSO time was freed up to focus on high-threat items, but our analysis of TSA data does not support this conclusion.[Footnote 9] TSA agrees that the agency could have conducted a more methodologically sound evaluation of the impact of the prohibited items list change, but continues to believe that the change nevertheless significantly contributed to the agency's efforts to free up TSO resources to focus on detecting high-threat items, such as explosives. It also is not clear whether the change had any impact on TSOs' ability to detect explosives--a key goal of the change. One way TSA measures the effectiveness of the passenger screening system in detecting threat items, such as explosives, is the results of threat image projection testing.[Footnote 10] However, TSA does not claim nor do the data definitively support that TSA's change to the prohibited items list had any impact on threat image projection results because TSA implemented other changes to checkpoint screening operations at or around the same time as the prohibited items list change. With regard to TSA's efforts to increase training for identifying explosives as part of its overall effort to become a more risk-based organization, TSA data between October 2004 and January 2007 show an increase in the average number of hours spent in training per TSO, but this trend began before the change to the prohibited items list and there are other factors that may have contributed to this increase. Background: In accordance with applicable laws and regulations, TSA prohibits weapons, explosives or incendiaries, and other items that TSA believes pose a significant threat to civil aviation security onboard commercial aircraft.[Footnote 11] TSA has divided these prohibited types of items into seven categories. Individuals are prohibited from carrying these items into an airport sterile area or onboard an aircraft either in their carry-on bag or on their person. Table 1 provides a description of the items included in the seven categories. Table 1: Categories and Descriptions of Prohibited Items. Category of prohibited item: Guns and firearms; Description of items included in the category: BB guns; compressed air guns; firearms; flare pistols; gun lighters; parts of guns and firearms; pellet guns; realistic replicas of firearms; spear guns; starter pistols; stun guns/cattle prods/ shocking devices. Category of prohibited item: Sharp objects; Description of items included in the category: Axes and hatchets; bows and arrows; ice axes/ice picks; knives of any length, except rounded- blade butter and plastic cutlery; meat cleavers; razor-type blades, such as box cutters, utility knives, and razor blades not in a cartridge, but excluding safety razors; sabers; scissors, metal with pointed tips and a blade length greater than 4 inches as measured from the fulcrum; swords; throwing stars (martial arts). Category of prohibited item: Club-like items; Description of items included in the category: Baseball bats; billy clubs; blackjacks; brass knuckles; cricket bats; golf clubs; hockey sticks; lacrosse sticks; martial arts weapons, including nunchucks, and kubatons; night sticks; pool cues; ski poles. Category of prohibited item: All explosives; Description of items included in the category: Ammunition; blasting caps; dynamite; fireworks; flares in any form; gunpowder; hand grenades; plastic explosives; realistic replicas of explosives. Category of prohibited item: Incendiaries; Description of items included in the category: Aerosol, any, except for personal care or toiletries in limited quantities; fuels, including cooking fuels and any flammable liquid fuel; gasoline; gas torches, including microtorches and torch lighters; lighter fluid; strike- anywhere matches; turpentine and paint thinner; realistic replicas of incendiaries; all lighters. Category of prohibited item: Disabling chemicals and other dangerous items; Description of items included in the category: Chlorine for pools and spas; compressed gas cylinders (including fire extinguishers); liquid bleach; mace; pepper spray; spillable batteries, except those in wheelchairs; spray paint; tear gas. Category of prohibited item: Tools; Description of items included in the category: Crowbars; drills and drill bits, including cordless portable power drills; hammers; saws and saw blades, including cordless portable power saws; other tools greater than 7 inches in length, including pliers, screwdrivers, and wrenches. Source: TSA. [End of table] Passenger screening is a process by which personnel authorized by TSA inspect individuals and property to deter and prevent the carriage of any unauthorized explosive, incendiary, weapon, or other items included on TSA's prohibited items list onboard an aircraft or into a sterile area.[Footnote 12] Passenger screening personnel--TSOs--must inspect individuals for prohibited items at designated screening locations.[Footnote 13] As shown in figure 1, the passenger screening functions are: * X-ray screening of property, * walk-through metal detector screening of individuals, * hand-wand or pat-down screening of individuals, * physical search of property and trace detection for explosives, and: * behavioral observation. Figure 1: Passenger Checkpoint Screening Functions: [See PDF for Image] Source: GAO and Nova Department Corporation. Notes: Explosive trace detection (ETD) works by detecting vapors and residues of explosives. Human operators collect samples by rubbing swabs along the interior and exterior of an object that TSOs determine to be suspicious, and place the swabs in the ETD machine, which then chemically analyzes the swab to identify any traces of explosive materials. Bomb Appraisal Officers (BAO) are available to respond to unresolved alarms at the checkpoint that involve possible explosive devices. The BAO may contact appropriate law enforcement or bomb squad officials if review indicates possible or imminent danger, in which case the BAO ensures that the security checkpoint is cleared. The BAO approves reopening of security lane(s) if no threat is posed. [A] BDOs are TSOs specially trained to detect suspicious behavior in individuals approaching the checkpoint. Should the BDO observe such behavior, he or she may refer the individual for individual screening or to a law enforcement officer. As of April 2007, BDOs are not present at all airport checkpoints. [B] The hand-wand or pat-down is conducted if a passenger is identified or randomly selected for additional screening because he or she met certain criteria or alarmed the walk-through metal detector. [C] Manual or ETD searches of accessible property occur if the passenger is identified or randomly selected for additional screening or if the screener identified a potential prohibited item on X-ray. [End of figure] Typically, passengers are only subjected to X-ray screening of their carry-on items and screening by the walk-through metal detector. Passengers whose carry-on baggage alarms the X-ray machine, who alarm the walk-through metal detector, or who are designated as selectees-- that is, passengers selected by the Computer-Assisted Passenger Prescreening System (CAPPS[Footnote 14]) or other TSA-approved processes to receive additional screening--are screened by hand-wand or pat-down and have their carry-on items screened for explosives traces or physically searched. In addition to passenger checkpoint screening, other layers of aviation security recognized by TSA include, among other things: * Hardened cockpit doors to prevent unauthorized access or forced entry to the flight deck. * Deployment of federal air marshals on certain flights to provide physical security should an incident occur. * Federal Flight Deck Officers (FFDO) Program to train pilots on commercial passenger and cargo aircraft on how to use lethal force against an intruder on the flight deck.[Footnote 15] * Security training for flight and cabin crews to handle potential threats onboard aircraft. Flight and cabin crews are expected to defend the flight deck in accordance with a TSA and FAA-developed guidance manual known as the Common Strategy.[Footnote 16] In addition, TSA considers the vigilance of able-bodied passengers to be an important layer of aviation security. Able-bodied passengers are those passengers who may engage in self-defense actions should an incident occur onboard commercial aircraft. TSA Changed the Prohibited Items List to Shift TSO Resources to Higher- Threat Priorities and Most Aviation Industry Stakeholders Interviewed Supported TSA's Change: TSA Conducted Various Studies to Determine Whether Changing the Prohibited Items List Would Free Up TSO Resources, but Some Efforts Lacked Methodological Rigor: As we reported in March 2007,[Footnote 17] TSA changed the prohibited items list in an effort to shift TSO resources to focus on higher threats, such as explosives, and based on its determination that small scissors and tools do not pose a risk to aviation security.[Footnote 18] TSA's decision was informed by the conclusions reached by an Explosives Detection Improvement Task Force established in October 2005 by the TSA Assistant Secretary to respond to the threat of IEDs being carried through the checkpoint. The goal of the task force was to apply a risk-based approach to screening passengers and their baggage in order to enhance TSA's ability to detect IEDs. As part of its analysis, the task force considered a number of factors including threat information, TSO effectiveness, and overall screening performance. According to TSA officials, the task force also considered the results of a Prohibited Items Working Group that was established in February 2005 by the then-TSA Assistant Secretary to develop recommendations for modifying the prohibited items list to better reflect the current aviation security environment. The Prohibited Items Working Group assessed each item on the prohibited items list using four criteria: (1) risks to flight security (i.e., can the item be used to take down an aircraft in flight); (2) legal restrictions (i.e., hazardous and other materials that are prohibited from the aircraft or from the flight cabin); (3) public concern and screener effectiveness (i.e., would permitting the item onboard an aircraft cause significant passenger and flight crew concern regarding their safety); and (4) international standards (i.e., international protocols recommend that the item be prohibited from the aircraft or the flight cabin). At the conclusion of its analysis, the working group recommended that scissors with pointed tips less than 6 centimeters (2.36 inches) and tools less than 7 inches be removed from the prohibited items list because these items were not considered to represent a risk to the aircraft or cockpit security. Although the working group based its size restriction for scissors on the size parameters recommended by the International Civil Aviation Organization (ICAO)--which is to provide for the safe, orderly, and efficient development of international civil aviation--the working group deviated from ICAO's recommendation to prohibit all pointed/edged scissors.[Footnote 19] A TSA representative from the working group stated that this change was recommended because the working group concluded that pointed/edged scissors could not be used to gain access to the cockpit to take down an aircraft in flight. In addition, the working group stated that concentrating on such items diminished TSA's efforts to focus on identifying objects that pose the greatest threat to aviation security, such as IEDs. Subsequent to the analysis of the working group, TSA's Explosives Detection Improvement Task Force collected information from several sources to test its assumption that a disproportionate amount of TSO resources was being spent searching for small scissors and tools. First, TSA reviewed data maintained in TSA's Performance Management Information System,[Footnote 20] which showed that during the third and fourth quarters of fiscal year 2005 (a 6-month period), TSOs collected a total of about 1.8 million sharp objects other than knives or box cutters and about 468,000 tools. The sharp objects constituted 19 percent of all prohibited items surrendered at the checkpoint during this period and tools constituted 5 percent of the items. Second, based on information provided by FSDs, TSOs, and other screening experts, TSA determined that scissors constituted a large majority of the total number of sharp objects found at passenger screening checkpoints. TSA also concluded that small screwdrivers, wrenches, and pliers make up a large majority of the tools found at checkpoints. Third, TSA headquarters officials searched through surrendered items bins at four airports and found that most of the scissors had blades less than 4 inches in length and a very large percentage of the tools that were surrendered were 7 inches or smaller. Based on these collective efforts, TSA's Explosive Detection Improvement Task Force concluded that a significant number of items found at the checkpoint were low-threat, easily identified items, such as small scissors and tools, and that a disproportionate amount of time was spent searching for these items--time that could have been spent searching for high-threat items, such as explosives. The task force also concluded that because TSOs can generally identify scissors and tools on X-ray images easily, if small scissors and tools were no longer on the prohibited items list, TSOs could avoid conducting time- consuming physical bag searches to locate and remove these items. TSA ultimately concurred with the recommendations provided by the Explosive Detection Improvement Task Force and decided to remove scissors less than 4 inches and certain tools less than 7 inches from the prohibited items list. Although TSA's rationale for its December 2005 change to the prohibited items list was to reduce focus on low-threat items in order to free up TSO time, attention, and resources to implement screening practices that better focus on high-threat items--such as Screening Passengers by Observation Technique (SPOT) and Unpredictable Screening Process (USP)[Footnote 21]--we reported in March 2007 that TSA had not conducted the necessary analysis of the data collected to determine the extent to which the removal of small scissors and tools from the prohibited items list could free up TSO resources. Specifically, we found that TSA had not analyzed the data on sharp objects surrendered at the checkpoint along with other relevant factors, such as the amount of time taken to search for scissors and the number of TSOs at the checkpoint conducting these searches. Based on our analysis of TSA's data for the third and fourth quarters of fiscal year 2005 (a 6-month period), where we considered these other relevant factors, we determined that TSOs spent, on average, less than 1 percent of their time--about 1 minute per day over the 6-month period--searching for the approximately 1.8 million sharp objects, other than knives and box cutters, that were found at passenger screening checkpoints between April 2005 and September 2005. If the average amount of time TSOs spent searching for sharp objects per day over a 6-month period was less than 1 minute per TSO, and sharp objects constituted just 19 percent of all prohibited items surrendered at checkpoints over this period, then it may not be accurate to assume that no longer requiring TSOs to search for small scissors and tools would significantly contribute to TSA's efforts to free up TSO resources that could be used to implement other security measures. TSA stated that the decision to remove small scissors and small tools from the prohibited items list was not only based on an analysis of data but was also firmly rooted in its assessment of risk, professional judgment, and experience. According to TSA, this included interviews with FSDs who unanimously indicated the change would free up TSO resources, as well as examinations of the prohibited items surrendered at several airports and a study to determine the amount of time taken to conduct bag searches. TSA acknowledged that this particular data collection and analysis effort may not have been methodologically rigorous, but stated that it did serve to provide insights regarding the type and quantity of items collected at the passenger checkpoint and the analysis effort generally supported the decision. Additionally, the TSA Assistant Secretary stated that even if TSA determined that the proposed prohibited items list modification would not free up existing TSO resources to conduct explosives detection procedures, he would have implemented the change anyway considering that such items no longer posed a significant security risk given the multiple layers of aviation security. In our March 2007 report, we recommended that TSA develop sound evaluation methods, when possible, that can help TSA determine whether proposed procedures that are operationally tested would achieve their intended purpose, such as enhancing TSA's ability to detect prohibited items and freeing up existing TSO resources that could be used to implement proposed procedures.[Footnote 22] TSA concurred with the recommendation and stated that it plans to make better use of generally accepted research design principles and techniques when operationally testing proposed changes to screening procedures. For example, TSA agreed to consider using random selection, representative sampling, and control groups in order to isolate the impact of proposed changes to screening procedures from the impact of other variables. Most Aviation Industry Stakeholders We Contacted Supported TSA's Changes to the Prohibited Items List: The majority (9 of 13) of the aviation industry stakeholders that we interviewed supported the removal of small scissors and tools from the prohibited items list. In general, these stakeholders said that they believe that the layers of aviation security reduce a passenger's ability to access the cockpit with low-threat items, and further noted that passengers may carry other items onboard an aircraft (such as glass bottles, pens, and sharpened credit cards) that may also be used as weapons. Stakeholders also stated that TSOs will be able to better focus on detecting IEDs if low-threat items such as small scissors and tools are removed from the prohibited items list. However, 4 out of 13 aviation industry stakeholders that we interviewed were opposed to the prohibited items list change, stating that permitting scissors increases the risk of violence against passengers and flight crew onboard an aircraft. Some of these stakeholders also stated that scissors also increase the risk that hijackings could be successfully implemented because scissors have bladed edges and pointed tips and therefore can be used as knives, and because terrorists can train with scissors to perfect their use as weapons. These stakeholders further stated that unlike other items that can be improvised to create a cutting surface (such as broken glass bottles), terrorists would not need to alter scissors onboard aircraft to use them as weapons. This could also allow a passenger to use the cutting edge and/or the sharpened tip of a scissor as a weapon without alerting other passengers or flight crew, as compared with the attention that could be drawn to a passenger that breaks a glass bottle. TSA acknowledges that scissors and tools may be used as weapons against passengers and flight crew. However, TSA stated that other items that are permitted onboard commercial aircraft, such as pens and glass bottles, may also be used as weapons against passengers and flight crew. TSA also maintained that its focus is on detecting explosives or items that can be used to breach the cockpit and potentially hijack the aircraft, which TSA and the majority of the aviation industry stakeholders that we spoke with view as a significant threat to aviation. TSA maintained that small scissors and tools cannot be used to hijack an aircraft, particularly given the other layers of security. Although stakeholders who both supported and disagreed with TSA's change stated that the layers of security implemented since September 11, 2001--particularly the hardened cockpit door--have decreased the likelihood of a successful hijacking, stakeholders generally stated that the risk of a hijacking is highest when the cockpit door is opened. In an attempt to mitigate this potential vulnerability, and in accordance with the air carrier's responsibility to ensure that no passenger can access the flight deck when the cockpit door needs to be opened during flight, air carriers will typically place a beverage cart between passengers and the cockpit with a flight attendant standing behind the cart. The beverage cart and the flight attendant serve as a "secondary barrier" between passengers and the cockpit door. However, two aviation stakeholders--a former law enforcement officer who provides self-defense training and a representative from the association of flight attendants--stated that this secondary barrier can be circumvented by a determined terrorist using a scissor to attack the flight attendant who is manning the beverage cart, which could allow the terrorist to negotiate around the beverage cart and then access the open cockpit door. A senior TSA official stated that flight crew protocols are sufficient to ensure passengers cannot breach the cockpit and that mechanisms are in place to ensure that cockpit doors are opened for brief periods of time. No Security Incidents against Passengers or Crew Using Scissors or Tools Have Been Reported to TSA Since the Change to the Prohibited Items List, but the Impact of the Change on Screening Operations Is Inconclusive: No Onboard Incidents Involving Small Scissors or Tools Reported to TSA Since Prohibited Items List Change and FAA Does Not Believe These Items Pose a Risk to the Integrity of an Aircraft: Based on our review of TSA security incident reports from the time period following the prohibited items list change (December 2005 through February 2007), there have been no reported security incidents onboard an aircraft involving the use of small scissors or tools.[Footnote 23] However, TSA and aviation security stakeholders we spoke with acknowledged that the absence of an onboard incident involving scissors or tools as weapons does not preclude the possibility of such an incident in the future. In addition, based on aircraft vulnerability and system safety and security analyses performed to date by government and industry, neither FAA nor a major aircraft manufacturer we interviewed perceive any meaningful increase in risk to the integrity of an aircraft associated with TSA's decision to permit small scissors and tools onboard aircraft.[Footnote 24] FAA officials also stated that aircraft are designed so that there are many layers of protection to prevent damage to the integrity of an aircraft from within (e.g., hardened cockpit doors and separate and redundant wiring for critical systems with few internal access points). The aircraft manufacturer stated that while it is possible that terrorists or others intending to do harm to the aviation system could use these items in ways not currently foreseeable, given current risk mitigation activities, the ability to inflict major damage to an aircraft with them is extremely remote. Impact of Prohibited Items List Change on Checkpoint Screening Operations Is Inconclusive: TSA conducted informal studies 30, 60, and 90 days after the prohibited items list changes went into effect to determine whether the change had resulted in reductions in the percentage of carry-on bags that were searched and overall screening time. However, in a prior report, we identified limitations in TSA's methodology for conducting these studies and concluded that it may not be accurate to assume that the prohibited items list change freed up resources.[Footnote 25] TSA agrees that the agency could have conducted a more methodologically sound evaluation of the impact of the prohibited items list change, but TSA continues to believe that the change did significantly contribute to the agency's efforts to free up TSO resources to focus on detection of high-threat items, such as explosives. TSA officials stated that that they have not conducted or planned any additional studies on the prohibited items list change to determine the impact of the change on the effectiveness of screening operations. Officials continue to view the change as sound based on their professional judgment and assessment of risk, and state that the change allowed the agency to shift focus from low risks to areas such as increased focus on explosive devices and increased training. In February 2007, a TSA official stated that some FSDs interviewed several TSOs after the prohibited items list change went into effect, and these TSOs reported that the change did save screening time. However, TSA could not identify how many TSOs were interviewed, at which airports the TSOs were located, and how the TSOs were selected for the interview; nor did TSA document the results of these interviews. As TSA did not use random selection or representative sampling when determining which TSOs should be interviewed, the interview results cannot be generalized. Most of the FSDs we interviewed in August 2006 as part of our passenger screening procedures review stated that the prohibited items list change, in addition to another change, did not collectively free up TSO resources to perform screening activities focused on threats considered to pose a high risk, such as explosives.[Footnote 26] Specifically, 13 of 19 FSDs we interviewed at airports that tested USP and SPOT said that TSO resources were not freed up as a result of the prohibited items list change and another change made by TSA during this time frame.[Footnote 27] In addition, 9 of the 19 FSDs said that in order to operationally test the procedures, TSOs had to work overtime, switch from other functions (such as checked baggage screening), or a screening lane had to be closed. Moreover, 13 of the 19 FSDs stated that TSOs did not experience more time to conduct explosives training.[Footnote 28] In addition to the lack of clarity about the impact of changes to the prohibited items list on TSO's available time, it also is not clear whether the change had any impact on TSOs' ability to detect IEDs--a key goal of the change. The results of threat image projection (TIP) testing are one way that TSA measures the effectiveness of the passenger screening system in detection of threat items, such as explosives. The results of TSA TIP testing are considered sensitive security information and thus could not be included in this report. Nevertheless, it is not clear whether TSA's change to the prohibited items list had any impact on TSOs' ability to identify IEDs during TIP testing because multiple factors could have accounted for the changes in TIP scores over time. For example, TSA implemented other changes to checkpoint screening operations at or around the same time as the prohibited items list change. These changes include both new and revised procedures, such as: revising the USP to include selected screening process elements like explosive trace detection of footwear and accessible property; screening 100 percent of passengers' footwear; banning liquids and gels; revising bulk-item pat downs to include the waistline down to the ankles; targeting threat area searches within baggage; revising the CAPPS rules; and implementing the new SPOT procedure. In fact, FSDs we interviewed at two category X airports[Footnote 29] in February 2007 as well as other TSA officials stated that at this time it is not possible to isolate the effect of the prohibited items list change from these additional changes in order to determine its impact on checkpoint screening operations and whether the prohibited items list change freed up TSO resources.[Footnote 30] With regard to TSA's efforts to increase training for identifying IEDs as part of its overall effort to become a more risk-based organization, TSA data between October 2004 and January 2007 show an increase in the average number of hours spent in training per TSO, but this trend began before the change to the prohibited items list and there are other factors that may have contributed to this increase. Our analysis of these data show an increase of an average of 1.68 hours per TSO in monthly IED training over the 29-month period, from an average of 0.42 hours per TSO in October 2004 to an average of 2.10 hours per TSO in February 2007. According to TSA's TSO training officials, there are two primary explanations for the increase: (1) in October 2005 TSA provided a 4-hour IED training course to 18,000 TSOs over a 3-week period, and, according to TSA, about 98 percent of the 48,236 TSOs on board had received classroom, checkpoint, or computer-based improvised explosive device recognition training as of February 6, 2007; and (2) in May 2006 TSA instituted a new monthly requirement of 4 hours of IED training per TSO. Thus, although a goal of the prohibited items list change was to increase TSO training hours for detecting IEDs, TSA program officials acknowledge, and we agree, that it is not clear whether the change to the prohibited items list had any impact on time spent in training. Concluding Observations: TSA is faced with the challenge of addressing numerous threats to commercial aviation security, as demonstrated by the alleged August 2006 terrorist plot to detonate liquid explosives onboard multiple commercial aircraft bound for the United States from the United Kingdom. TSA's December 2005 change to the prohibited items list is one of several efforts TSA has made to focus its resources on addressing the threat posed by explosives, which TSA considers to be the most significant threat to commercial aviation security. While TSA's consideration of threat information, the professional judgment of TSA personnel, data analysis, and international standards all constitute reasonable inputs to making informed decisions on how to best anticipate and address threats given its available resources, the impact of the prohibited items list change on security and screening effectiveness is inconclusive. Nevertheless, we are encouraged that TSA recognized the limitations in its analysis of data used to help inform the prohibited items list change and plans to improve the methodological rigor for evaluating proposed changes to passenger screening procedures in the future, as we recommended in our March 2007 report. This effort will be particularly important as additional changes to passenger screening procedures--including future revisions to the prohibited items list--are considered and implemented. Agency Comments: We provided a draft of the report to DHS for its review and comment. TSA provided technical comments, which we incorporated as appropriate. We will send copies of this report to the Secretary of the Department of Homeland Security; the Assistant Secretary, TSA; and interested congressional committees as appropriate. We will also make this report available at no charge on GAO's Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in enclosure II. Signed by: Robert Goldenkoff: Acting Director, Homeland Security and Justice Issues: Enclosures: Enclosure I: Scope and Methodology: To describe TSA's basis for removing certain scissors and tools from the prohibited items list and stakeholder views on the change, we obtained and analyzed TSA documentation of the proposed prohibited items list change considered by TSA's Explosives Detection Improvement Task Force, which was the deliberating body for proposed TSA procedural changes that were considered between October 2005 and December 2005. We also obtained and analyzed a draft TSA Prohibited Items Working Group analysis, as well as TSA public statements and testimonies regarding the rationale for the prohibited items change. We also met with TSA officials to obtain information on their rationale behind the change. In addition, we met with officials at the Federal Aviation Administration (FAA) and the Federal Air Marshals Service (FAMS)--a component of TSA--to obtain their views regarding the prohibited items list change. We spoke with organizations within the aviation community including four domestic aviation associations, one international aviation association, a major aircraft manufacturer, the largest association representing airline pilots in the United States, the largest association representing flight attendants in the United States, and an association representing federal air marshals and other federal law enforcement officers.[Footnote 31] In addition, we met with five aviation security experts to obtain their views on TSA's change to the prohibited items list.[Footnote 32] We selected these experts based on their depth of experience in the field of aviation, employment history, and their recognition in the aviation security community. We also met with a major aircraft manufacturer to determine whether there are any major safety concerns related to the change to the prohibited items list. Finally, we incorporated aspects from our recently issued report on passenger checkpoint screening procedures,[Footnote 33] which included a review of the factors TSA considered in modifying the prohibited items list and TSA's analysis supporting the December 2005 prohibited items list change. To determine the impacts, if any, that the removal of certain scissors and tools from the prohibited items list had on security and on the effectiveness of screening operations, we obtained and reviewed TSA documentation and data including the results of threat image projection (TIP) testing and data on training hours completed by Transportation Security Officers (TSO). We sent written questions about data quality control and reporting procedures to TSA officials responsible for collecting and analyzing these data, and received responses to these questions. The TIP data TSA provided contained limitations. First, the data contained only monthly averages for tests in which improvised explosive devices (IED) images had been successfully identified by TSOs, according to individual airports in each airport category; we did not receive the raw numbers of image presentations from which the percentages were derived. Therefore, to compute an average percentage of successful TIP tests across all airports, we computed an average of averages. Computing an average in this manner can provide a result that is slightly different than if raw data had been used. For example, we could not adjust our computations to account for differing numbers of image presentations or the rate of image presentations by airport. Second, there were missing values, or no test results, for some airports in certain months. Despite these limitations, we believe the TIP data were sufficiently reliable to provide an indication of TSOs' abilities to identify IED images. In addition, we interviewed Federal Security Directors (FSD) from Boston Logan Airport and Washington Dulles International Airport to obtain anecdotal information about their views on the impact of the prohibited items list change on checkpoint screening operations. However, the perspectives of these two FSDs cannot be considered to be representative of the views of FSDs nationwide or generalized because we did not use random selection or representative sampling when determining which FSDs should be interviewed. To determine whether the change to the list of prohibited items had any impact on TSO time spent in training, we also analyzed training data provided by TSA on the average number of hours spent in training per TSO for the period from October 2004 through February 2007. TSA uses a dynamic system to capture training data called the Online Learning Center, and TSA offers several reasons for the dynamic nature of this system. First, TSA employees and contractors are continuously allowed to update training history hours. As a result, data on training hours and attendance extracted from the database at two different points in time may vary as employees and contractors update their training history. Second, there can be a delay in updating training data due to manual entry of student results. TSA policy is that final reports are generated on the 10TH of each month in order to permit time to collect and consolidate airport data for manual data entry. The training hour data were sufficiently reliable for our purpose in showing a general increase in IED training over time. Our results are based on the data TSA provided to us on March 21, 2007. To determine if any security incidents onboard an aircraft involving the use of small scissors or tools were reported to TSA, we reviewed and analyzed TSA security incident reports from the time period following the prohibited items list change (December 22, 2005--the effective date of the prohibited items list change--through February 28, 2007). Because TSA is the agency with primary responsibility for aviation security and maintains records of aviation security incidents, TSA security incident reports were our primary source of information for identifying incidents involving small scissors or tools. We followed a two-step process to identify incidents appropriate to our review. During the first step, one analyst reviewed all incidents in each daily TSA security incident report to identify any incidents that he or she discerned involved small scissors or tools based on key words or phrases in the incident title or description. A log was created for each incident report reviewed. In the second step, a random sample of 10 percent of the incident reports was selected, and these reports and their accompanying logs were reviewed by a second analyst to verify the accuracy of the first analyst's judgments. We limited the scope of our TSA security incident report review to incidents that occurred on commercial passenger aircraft in-flight. We defined "in-flight" as the time between aircraft take-off and landing. Although it is possible that there were some incidents involving small scissors or tools that occurred during the time period of our review that were not reported to TSA, and thus not recorded in the incident reports, we found the incident reports sufficiently reliable for our purposes. We conducted our work from November 2006 through March 2007 in accordance with generally accepted government auditing standards. Enclosure II: GAO Contact and Staff Acknowledgments: GAO Contact: Robert Goldenkoff (202) 512-2757 or goldenkoffr@gao.gov. Staff Acknowledgments: Key contributors to this report were Maria Strudwick, Assistant Director; David Alexander; Christopher Backley; Amy Bernstein; Tony Cheesebrough; Adam Hoffman; Stanley Kostyla; Tom Lombardi; and Brian Sklar. (440557): FOOTNOTES [1] See 49 U.S.C. §§ 114(d), 44903(b). [2] Sterile areas are located within the terminal where passengers are provided access to boarding aircraft. [3] 70 Fed. Reg. 72,930 (Dec. 8, 2005). [4] See H.R. Conf. Rep. No. 109-699, at 139 (2006) (accompanying H.R. 5441, enacted into law as the Department of Homeland Security Appropriations Act, 2007, Pub. L. No. 109-295, 121 Stat 1355 (2006)). [5] The Threat Image Projection system is designed to test TSOs' detection capabilities by projecting threat images, including guns, knives, and explosives, onto carry-on bags as they are screened during actual operations. TSOs are responsible for identifying the threat image and calling for the bag to be searched. Once prompted, TIP identifies to the TSO whether the threat is real and then records the TSO's performance in a database that could be analyzed for performance trends. [6] FSDs are the ranking authorities responsible for leading and coordinating security activities at U.S. commercial airports at which TSA provides for or oversees the provision of screening activities. [7] GAO, Aviation Security: Risk, Experience, and Customer Concerns Drive Changes to Airline Passenger Screening Procedures, but Evaluation and Documentation of Proposed Changes Could Be Improved, GAO-07-57SU (Washington, D.C.: Mar. 7, 2007). The information in this report is considered sensitive security information in accordance with 49 C.F.R. part 1520 and is not available to the public. A public version of this report (GAO-07-634) is expected to be issued in May 2007. [8] Ibid. [9] Ibid. [10] The results of TSA TIP testing are considered sensitive security information and thus could not be included in this report. [11] See 49 U.S.C. § 44902; 49 C.F.R. §§ 1540.111, 1544.201(d). [12] Access to sterile areas is controlled by TSOs (or by nonfederal screeners at airports participating in the Screener Partnership Program) at checkpoints where they conduct physical screening of individuals and their carry-on baggage for weapons, explosives, and other prohibited items. [13] TSOs must deny passage beyond the screening location to any individual or property that has not been screened or inspected in accordance with passenger screening standard operating procedures. If an individual refuses to permit inspection of any item, that item must not be allowed into the sterile area or onboard an aircraft. [14] CAPPS is a computer-assisted system that, based on information obtained from airline reservation systems, identifies passengers that may pose a high risk to aviation security. These high-risk passengers, along with other individuals selected for secondary screening, and their carry-on baggage are subject to additional and more thorough screening. [15] Administered by TSA, the FFDO Program deputizes volunteer pilots of commercial passenger aircraft as armed federal law enforcement officers for the purpose of defending the flight deck "against acts of criminal violence or air piracy." Since the program was officially established on February 25, 2003, TSA has deputized thousands of eligible flight crew members as FFDOs. [16] The Common Strategy is a detailed guidance manual developed by TSA and FAA for pilots and other crewmembers to identify their responsibilities and the appropriate responses during in-flight security threats. In January 2005, TSA and FAA issued a revised version of the Common Strategy. The previous version, referred to as Common Strategy #1, was the strategy in effect on September 11, 2001. Common Strategy #1 was developed jointly by industry, FAA, and FBI, and presumed a hijacker whose motive might be ransom, escape from the law, political asylum, or publicity. According to the Common Strategy, the terrorist attacks of September 11, 2001, demonstrated that Common Strategy #1 was not effective in dealing with a new breed of hijacker whose motives are terrorism, mass murder, and suicide. [17] GAO-07-57SU. [18] The change to the prohibited items list was one of several other changes to TSA procedures intended to focus more TSA resources on higher threats, such as explosives. The Explosive Detection Improvement Task Force recommended seven proposed procedures that were ultimately implemented by TSA. These procedures were considered by senior TSA officials as especially important for enhancing the detection of explosives and for deterring terrorists from attempting to carry out an attack. According to TSA, some of the proposed procedures, such as the prohibited items list change, could also free up TSOs so that they could spend more time on procedures for detecting explosives and less time on procedures associated with low security risks, such as identifying small scissors in carry-on bags. The seven proposed procedures tested by the task force reflect both new procedures and modifications to existing procedures. The procedures are discussed in detail in GAO-07-57SU. [19] Nations that are members to ICAO agree to cooperate with other member states to meet standardized international aviation security measures. ICAO recommends that pointed/edged scissors of any size should be prohibited from the flight cabin, while rounded or blunt scissors less than 6 cm should be permitted in the flight cabin. The TSA Prohibited Items Working Group utilized the ICAO size parameters, but applied the parameters to both rounded/blunt scissors as well as pointed/edged scissors, thus deviating from the ICAO recommendation to ban all pointed/edged scissors. [20] TSA's Performance Management Information System is designed to collect, analyze, and report passenger and baggage screening performance data, such as wait times at selected airports, workload data, and the performance and utilization of passenger and baggage screening equipment. TSA headquarters uses the Performance Management Information System data to support external reporting on performance and internal decision-making processes. [21] Screening Passengers by Observation Technique involves specially trained TSOs observing the behavior of passengers and resolving any suspicious behavior through casual conversation with passengers and referring suspicious passengers to selectee screening. Unpredictable Screening Process entails random selection of passengers across two screening lanes to be subjected to a predetermined element of the selectee screening process. [22] GAO-07-57SU. [23] Because TSA is the primary agency responsible for aviation security and maintains records of aviation security incidents, TSA security incident reports were our primary source of information for identifying incidents involving small scissors or tools. These security incident reports summarize transportation security incidents-- including aviation--that are reported to TSA and include descriptions of the incident. We used the December 2005-February 2007 time period because it was after the effective date of the prohibited items list change. Pursuant to TSA-issued Security Directive 1544-04-15, all aircraft operators are required to immediately report all threats that could affect the security of commercial aircraft to TSA. [24] The Federal Aviation Administration has primary responsibility for ensuring the safety of civil aviation operations, including the operation of air traffic control and regulating the manufacture, operation, and maintenance of aircraft. See 49 U.S.C. § 44701. [25] The results of the informal follow-on studies, which were conducted at 6 to 9 airports, show that the percentage of carry-on bags searched increased by about 0.1 percentage point at the time of the 30- day study, then decreased by 2.3 and 0.7 percentage points at the time of the 60-day and 90-day studies, respectively. However, the results of these informal studies may not be reliable due to the limitations in the methodology TSA used to conduct the studies. Specifically, TSA did not use a methodology that would control for factors other than the prohibited items list change that may influence the percentage of carry- on bags searched by TSOs. To do this, TSA would have had to develop a formal, systematic methodology for randomly selecting various times of day, location of checkpoints, number of checkpoints, and so on for data collection. By not controlling for such factors, TSA may not know the extent to which a reduction in the percentage of carry-on bags searched is due to the prohibited items list changes. See GAO-07-57SU. [26] See GAO-07-57SU. An additional measure intended to free up TSO resources involved changes to CAPPS rules to reduce the number of passengers selected for secondary screening. TSA's assumption is that these changes could allow TSOs who were normally assigned to selectee screening duties to be reassigned to new procedures, which may require new screening positions. [27] Since we did not use random selection or representative sampling when determining which FSDs should be interviewed, the interview results cannot be generalized. [28] Of the remaining 6 FSDs, 5 said that TSO resources were freed up as a result of the prohibited items list and CAPPS rules changes, and 1 was uncertain whether TSO resources were actually freed up. [29] TSA classifies the over 400 commercial airports in the United States into one of five categories--X, I, II, III, and IV. Generally, category X airports have the largest number of passenger boardings and category IV airports have the smallest number. [30] The views of the two FSDs we interviewed may not be representative of the views of the population of FSDs. [31] Specifically, we met with the Air Transport Association (ATA), the National Air Carrier Association (NACA), the Regional Airline Association (RAA), the Air Carrier Association of America (ACAA), the International Air Transport Association (IATA), the Air Line Pilots Association (ALPA), the Association of Flight Attendants (AFA), and the Federal Law Enforcement Officers Association (FLEOA). [32] The views of these five experts may not necessarily represent the general view of other experts in the field of aviation security. [33] GAO-07-57SU. GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select "Subscribe to Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, D.C. 20548: Public Affairs: Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: