Aviation Security: TSA Improved Covert Testing but Needs to Conduct More Risk-Informed Tests and Address Vulnerabilities
Fast Facts
To test security screening at U.S. airports, TSA regularly tries to sneak guns and simulated bombs through checkpoints or in checked baggage. TSA changed its testing practices to better identify and address screening vulnerabilities.
We observed 26 covert tests and reviewed the test program and how results are used. We found that TSA’s ability to run covert tests has improved, but a new process intended to address vulnerabilities found in testing hasn't fully worked.
We made 9 recommendations, including that TSA establish timeframes for addressing the vulnerabilities it discovers.
TSA screening passengers at the checkpoint.
Highlights
What GAO Found
Two offices within the Transportation Security Administration (TSA) conduct covert tests at U.S. airports—Inspection and Security Operations. The Department of Homeland Security requires that agencies use risk information to make decisions, and TSA issues annual risk assessments of threats that its program offices should consult when making risk-based decisions, such as what covert tests to conduct. Of the two TSA offices that conduct covert tests, Inspection officials used TSA's risk assessment to guide their efforts. However, Security Operations officials relied largely on their professional judgment in making decisions about what scenarios to consider for covert testing. By not using a risk-informed approach, TSA has limited assurance that Security Operations is targeting the most likely threats.
Both Inspection and Security Operations have implemented processes to ensure that their covert tests produce quality results. However, GAO found that only Inspection has established a new process that has resulted in quality test results. Specifically, for the two reports Inspection completed for testing conducted in fiscal years 2016 and 2017 using its new process, GAO found that the results were generally consistent with quality analysis and reporting practices. On the other hand, Security Operations has not been able to ensure the quality of its covert test results, and GAO identified a number of factors that could be compromising the quality of these results. Unless TSA assesses the current practices used at airports to conduct tests, and identifies the factors that may be impacting the quality of covert testing conducted by TSA officials at airports, it will have limited assurance about the reliability of the test results it is using to address vulnerabilities.
In 2015, TSA established the Security Vulnerability Management Process to leverage agency-wide resources to address systemic vulnerabilities; however, this process has not yet resolved any identified security vulnerabilities. Since 2015, Inspection officials submitted nine security vulnerabilities identified through covert tests for mitigation, and as of September 2018, none had been formally resolved through this process. GAO found that in some cases, it took TSA officials overseeing the process up to 7 months to assign an office responsible to begin mitigation efforts. In part, this is because TSA has not established time frames and milestones for this process or established procedures to ensure milestones are met, in accordance with best practices for program management. Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities.
This is a public version of a classified report that GAO issued in January 2019. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.
Why GAO Did This Study
TSA uses covert testing to identify potential vulnerabilities in checkpoint and checked baggage screening systems at U.S. airports. In 2015, TSA identified deficiencies in its covert testing process, and in 2017, the Department of Homeland Security Office of Inspector General's covert testing identified deficiencies in screener performance. Since these findings, TSA has taken steps intended to improve its covert test processes and to use test results to better address vulnerabilities.
GAO was asked to review TSA's covert test programs, including how the results are used to address vulnerabilities. This report analyzes the extent to which (1) TSA covert tests are risk-informed, (2) TSA covert tests for fiscal years 2016 through March 2018 produced quality information, and (3) TSA uses covert test results to address any identified security vulnerabilities.
GAO observed 26 TSA covert tests, reviewed TSA guidance, analyzed test data for fiscal years 2016, 2017, and through March 2018, and interviewed TSA officials.
Recommendations
GAO is making nine recommendations, including that TSA use a risk-informed approach for selecting covert test scenarios, take steps to improve the quality of airport covert test results, and establish time frames and milestones for the key steps in its vulnerability management process. TSA concurred with all nine GAO recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | The Administrator of TSA should document its rationale for key decisions related to its risk-informed approach for selecting covert test scenarios, for both the Security Operations' and the Inspection's testing process. (Recommendation 1) |
In April 2019, we reported on the Transportation Security Administration's (TSA) Covert Test program. We reported that managers for TSA's covert test program are required to document their rationale for making risk-informed program decisions, and that fully documented risk-informed decisions address how threats, vulnerabilities, and consequences were considered. We found that the two offices within TSA that conduct covert testing, Inspection and Security Operations, were not fully documenting their respective risk-informed rationales for choosing test scenarios--i.e., the screening activities (e.g., pat downs, bag searches) and threat items (e.g., guns, explosive devices) involved in tests. In June 2019, TSA provided us documentation to show that its selection of scenarios for Security Operations' covert tests would be based on an intelligence and data-driven methodology that utilizes information from multiple TSA risk assessments to prioritize domestic aviation security system vulnerabilities. Further, in September 2019, TSA provided a memorandum concerning Inspection's risk-informed rationale for selecting scenarios that included a discussion of how the agency considers threat, vulnerability, and consequence when selecting scenarios. Fully documenting their risk-informed rationales will help TSA program managers ensure their scenario selection decisions are appropriately accounting for risk, as called for by DHS and TSA guidance. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should incorporate a more risk-informed approach into Security Operations' process for selecting the covert test scenarios that are used for tests conducted by TSA officials at airports. (Recommendation 2) |
In April 2019, we reported on the Transportation Security Administration's (TSA) Covert test program. We found that TSA officials relied largely on their professional judgment in making decisions about what scenarios to use for covert testing, but by not using a risk-informed approach, TSA had limited assurance that it was targeting the most likely threats. We recommended that TSA incorporate a more risk-informed approach into its process for selecting the scenarios that TSA officials at airports use for covert tests. In April 2019, TSA developed and implemented a new, risk-informed process that will better align its selection of scenarios for covert tests at U.S. airports with TSA risk assessments. For example, the number of test scenarios selected for the program will correspond with an assessment of the likelihood of such scenarios taking place. Given these actions, TSA is better positioned to ensure that its testing resources are targeting the most likely threats to checkpoint and checked baggage screening. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should assess the current covert testing process used by TSA officials at airports—including factors that may affect the covertness and consistency of the tests—to identify opportunities to improve the quality of test data, and make changes as appropriate. (Recommendation 3) |
The Transportation Security Administration (TSA) concurred with this recommendation. In response, it undertook a review of its covert test program, and found that test results collected by TSA headquarters officials and TSA officials at airports were not reliable for national-level analysis. Officials subsequently developed a new covert testing process known as Index Testing, to collect high-quality covert testing results from a selection of airports, and to conduct analysis of these results to identify detection rates for certain threat items over time, and to better understand the root causes of test failures. As part of this process, TSA developed new procedures and protocols for running tests, trained special teams of testers, and standardized all test items, among other measures to increase the covertness and consistency of tests. In March 2020, TSA implemented Index Testing and collected, analyzed, and validated data from the first quarter of testing. Given these actions TSA has met the intent of the recommendation, and the recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should assess Security Operations guidance for applying root causes for test failures, and identify opportunities to clarify how they should be applied. (Recommendation 4) |
In April 2019, we reported that managers for the Transportation Security Administration's (TSA) covert test program faced challenges with the quality of information that TSA staff at airports collect when Transportation Security Officers (TSO) at airports fail a covert test. Specifically, TSA requires airport staff running covert tests to identify and record information on the root cause of a test failure. We found that some testers' practices for determining root causes could result in inconsistent and potentially incorrect information, which could diminish the usefulness of the information for addressing problems with TSO performance. Therefore, we recommended that TSA assess its guidance for determining the root cause of covert test failures, and identify opportunities to clarify how TSA staff at airports who run tests should apply this guidance. In May 2020, TSA provided us revised guidance that updated and clarified definitions for the three categories TSA testers can identify as a root cause for failures. Further, in August 2020, TSA officials told us they completed work on a new system to record covert test results that allows testers to select the root cause of a test failure from among a preset list of options that are aligned with current screening procedures and relate to the procedure being tested. As a result of these actions, TSA has greater assurance that it is collecting reliable information on the root cause of test failures and is better positioned to use information on covert test failures to improve screener performance. Therefore, this recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should document the methodology for using the results of covert testing conducted by headquarters staff as a quality assurance process for covert testing conducted by TSA officials at airports. (Recommendation 5) |
TSA concurred with this recommendation. In response, it undertook a review of its quality assurance process for covert tests; found that the test results it collected were not reliable for national-level analysis; and ceased using tests conducted by headquarters staff as a quality assurance process for covert testing in December 2018. TSA subsequently developed a new covert testing process known as Index Testing in March 2020. In developing its Index testing program, TSA produced several documents to identify how the new process would produce quality test results. These documents include an extensive test guide, which provides detailed information on testing objects and protocols; a sampling methodology, which identifies how airport test locations were selected so that results can be applied to airports nationwide; and several memoranda detailing other key decisions, such as a memorandum on the time and day of the week testing is to occur. Taken together, these documents provide a coherent methodology for understanding TSA's actions to ensure the quality of Index testing results for national-level analysis. Given these actions TSA has met the intent of the recommendation, and the recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should establish timeframes and milestones for key steps in its Security Vulnerability Management Process that are appropriate for the level of effort required to mitigate identified vulnerabilities. (Recommendation 6) |
In April 2019, we reported on the Transportation Security Administration's (TSA) Covert test program. We found that TSA established the Security Vulnerability Management Process in 2015 to address any system-wide vulnerabilities it identifies through covert testing; however, this process has not yet resolved any security vulnerabilities identified through covert testing. In part, this is because TSA has not established time frames and milestones for this process. Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities that could result in potentially serious consequences for the traveling public. We recommended that TSA establish timeframes and milestones for the process that are appropriate for the level of effort required to mitigate identified vulnerabilities. In June 2019, TSA provided us with a revised version of the Security Vulnerability Management Process charter that established milestones and timeframes for completing key aspects of the process. For example, the revised charter requires that within 60 days of a vulnerability being submitted, the executive committee overseeing the process must identify a vulnerability owner (the entity within TSA that will lead efforts to address the vulnerability) and establish mitigation action plans, among other things. Given these actions, TSA is better positioned to make efficient and effective progress addressing security vulnerabilities identified through covert testing. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should revise existing guidance for the Security Vulnerability Management Process to establish procedures for monitoring vulnerability owners' progress against timeframes and milestones for vulnerability mitigation, including a defined process for escalating cases when milestones are not met. (Recommendation 7) |
In April 2019, we reported on the Transportation Security Administration's (TSA) Covert test program. We found that TSA established the Security Vulnerability Management Process in 2015 to address any system-wide vulnerabilities it identifies through covert testing; however, this process has not yet resolved any security vulnerabilities identified through covert testing. In part, this is because TSA has not established a process for monitoring agency progress against established timeframes and milestones for addressing vulnerabilities. Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities that could result in potentially serious consequences to the traveling public. We recommended that TSA revise existing guidance for the Security Vulnerability Management Process to establish procedures for monitoring vulnerability owners' progress against timeframes and milestones for vulnerability mitigation, including a defined process for escalating cases when milestones are not met. In September 2019, TSA revised and approved its charter for the Security Vulnerability Management Process. The revised charter establishes new procedures for monitoring; for example, to enhance accountability for monitoring, the revised charter requires that an executive-level sponsor be assigned to each vulnerability being addressed. Furthermore, according to the revised charter, TSA will review the status of each vulnerability on a quarterly basis and escalate any mitigation efforts that do not meet established milestones to an executive body composed of TSA leadership, including the TSA Administrator and Deputy Administrator. Given these actions, TSA is better positioned to make efficient and effective progress addressing security vulnerabilities identified through covert testing. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration |
Priority Rec.
The Administrator of TSA should develop processes for conducting and reporting to relevant stakeholders a comprehensive analysis of covert test results collected by TSA headquarters officials and TSA officials at airports to identify vulnerabilities in screener performance and common root causes contributing to screener test passes and failures. (Recommendation 8) |
In April 2019, we recommended that the Administrator of TSA develop processes for conducting and reporting to relevant stakeholders a comprehensive analysis of covert test results collected by TSA headquarters officials and TSA officials at airports to identify vulnerabilities in screener performance and common root causes contributing to screener test passes and failures. In response, TSA first undertook a review of its covert test program, and found that test results collected by TSA headquarters officials and TSA officials at airports were not reliable for national-level analysis. Officials subsequently developed a new covert testing process known as Index testing, to collect high-quality covert testing results from a selection of airports, and to conduct analysis of these results to identify detection rates for certain threat items over time and better understand the root causes of test failures. In March 2020, TSA collected and analyzed data from the first quarter of Index testing. In April 2020, TSA began reporting the results of Index testing to the Security Vulnerability Management Process, the agency-wide process for addressing security vulnerabilities, and to Federal Security Directors at airports. TSA officials stated that, moving forward, these reporting processes will be a routine part of Index testing. Given these actions TSA is able to conduct and report to stakeholders a more comprehensive analysis of covert test results to identify vulnerabilities in screener performance and the root causes of test passes and failures. Therefore, the recommendation is closed as implemented.
|
| Transportation Security Administration | The Administrator of TSA should develop a standard process for systematically documenting and disseminating to airport Federal Security Directors beneficial practices for conducting covert tests and using test results. (Recommendation 9) |
In April 2019, we recommended that the Administrator of TSA develop a standard process for systematically documenting and disseminating to airport Federal Security Directors beneficial practices for conducting covert tests and using test results. In response, TSA first conducted 20 site visits to the nation's largest airports from June to December 2019 to compile beneficial practices on local airport covert testing. Officials subsequently created and launched an internal website in February 2020 to ensure these and other beneficial practices are documented by Federal Security Directors and those with responsibility for managing local covert tests at airports, thereby providing the opportunity for airports to share methods that successfully aid in the execution of testing. As of March 2020, approximately 12 beneficial practices have been uploaded to the site. Given these actions, TSA is able to better document the methods or techniques determined to be effective for conducting covert tests at airports based on empirical evidence, as well as make this information available to relevant stakeholders responsible for managing testing programs. Therefore, the recommendation is closed as implemented
|