Aviation Security: TSA Should Ensure Screening Technologies Continue to Meet Detection Requirements after Deployment
Fast Facts
TSA tests its screening technologies before installing them at airports to ensure that they are detecting certain dangerous items as intended.
But screening technologies can become less effective over time, and we found that TSA does not continue to fully test them once they are installed. Some airport equipment that detects trace explosives or tests bottled liquids wasn’t performing as intended when the Department of Homeland Security evaluated it in 2015 and 2016.
We recommended that TSA ensure that its screening technologies continue to meet requirements after they are installed at airports.
Airport screening equipment
Highlights
What GAO Found
The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) operationalizes, or puts into effect, detection standards for its screening technologies by acquiring and deploying new technologies, which can take years. Detection standards specify the prohibited items (e.g., guns, explosives) that technologies are to detect, the minimum rate of detection, and the maximum rate at which technologies incorrectly flag an item. TSA operationalizes standards by adapting them as detection requirements, working with manufacturers to develop and test new technologies (software or hardware), and acquiring and deploying technologies to airports. For the standards GAO reviewed, this process took 2 to 7 years, based on manufacturers' technical abilities and other factors.
TSA's deployment decisions are generally based on logistical factors and it is unclear how risk is considered when determining where and in what order technologies are deployed because TSA did not document its decisions. TSA considers risks across the civil aviation system when making acquisition decisions. However, TSA did not document the extent risk played a role in deployment, and could not fully explain how risk analyses contributed to those decisions. Moving forward, increased transparency about TSA's decisions would better ensure that deployment of technologies matches potential risks.
Technology performance can degrade over time; however, TSA does not ensure that technologies continue to meet detection requirements after deployment to airports. TSA certifies technologies to ensure they meet requirements before deployment, and screeners are to regularly calibrate deployed technologies to demonstrate they are minimally operational. However, neither process ensures that technologies continue to meet requirements after deployment. In 2015 and 2016, DHS tested a sample of deployed explosives trace detection and bottled liquid scanner units and found that some no longer met detection requirements. Developing and implementing a process to ensure technologies continue to meet detection requirements after deployment would help ensure that TSA screening procedures are effective and enable TSA to take corrective action if needed.
Transportation Security Administration's (TSA) Process for Acquiring Screening Technologies to Meet Detection Standards
Why GAO Did This Study
TSA is responsible for overseeing security operations at roughly 440 TSA-regulated airports as part of its mission to protect the nation's civil aviation system. TSA uses technologies to screen passengers and their bags for prohibited items.
The TSA Modernization Act includes a provision for GAO to review TSA's deployment of screening technologies, and GAO was asked to review the detection standards of these screening technologies. This report addresses, among other things, (1) how TSA operationalizes detection standards, (2) the extent to which TSA considered risk when making deployment decisions, and (3) the extent to which TSA ensures technologies continue to meet detection requirements after deployment.
GAO reviewed DHS and TSA procedures and documents, including detection standards; visited DHS and TSA testing facilities; observed the use of screening technologies at seven airports, selected for varying geographic locations and other factors; and interviewed DHS and TSA headquarters and field officials.
Recommendations
GAO is making five recommendations, including that TSA document analysis of risk in deploying technologies, and implement a process to ensure technologies continue to meet detection requirements after deployment. DHS agreed with all five recommendations and said TSA either has taken or will take actions to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | The TSA Administrator should update TSA guidance for developing and approving screening technology explosives detection standards to reflect designated procedures, the roles and responsibilities of stakeholders, and changes in the agency's organizational structure. (Recommendation 1) |
In December 2019, we reported on the Transportation Security Administration's (TSA) processes for developing detection standards--which identify the characteristics of prohibited items, such as explosives, that passenger and checked baggage screening technologies are to detect--among other issues. During the course of our review, we found that TSA had not updated its 2015 guidance for developing new detection standards to reflect key changes in its procedures. Specifically, we found that, as of August 2019, the guidance did not accurately reflect (1) designated procedures for developing detection standards, (2) the roles and responsibilities of key stakeholders, and (3) TSA's organizational structure. For example, the 2015 guidance calls for an annual assessment of emerging threats, which a senior TSA official told us TSA no longer conducts because relevant emerging threats are now occurring more frequently and intelligence information is processed on an ongoing basis. Consequently, we recommended that TSA update the 2015 guidance for developing and approving screening technology explosives detection standards to reflect designated procedures, the roles and responsibilities of stakeholders, and changes in the agency's organizational structure. In November 2019, TSA provided the Requirements Engineering Integrated Process Manual (RE IPM), which presents designated procedures for developing detection standards and specifies the roles and responsibilities of key stakeholders, with references to offices in TSA's organizational structure. Updating its guidance for developing detection standards should provide TSA with better assurance that detection standards are developed in accordance with established policies and practices. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The TSA Administrator should require and ensure that TSA officials document key decisions, including testing and analysis decisions, used to support the development and consideration of new screening technology explosives detection standards. (Recommendation 2) |
In December 2019, we reported on the Transportation Security Administration's (TSA) development of detection standards for passenger and checked baggage screening technologies. During our review of TSA's steps to develop detection standards from fiscal years 2014 through 2018, we found that TSA and DHS's Science and Technology Directorate (S&T) did not document all key decisions--those that could potentially affect outcomes--regarding the testing and analyses (characterization) of explosive threat materials and the development of explosives detection standards. For example, we found that in five of the seven sets of testing and analyses of explosive materials--referred to as material threat assessments--we reviewed, TSA and S&T did not consistently document key steps, such as how selected samples were prepared for testing, and in three assessments officials did not cite standard operating procedures for at least one laboratory. Consequently, we recommended that TSA should require and ensure that TSA officials document key decisions, including testing and analysis decisions, used to support the development and consideration of new screening technology explosives detection standards. In November 2019, TSA provided the Requirements Engineering Integrated Process Manual (RE IPM), which details TSA's process for documenting key decisions, including testing and analysis decisions, used in the development of new screening technology explosives detection standards. In July 2022, TSA provided a new material threat assessment to demonstrate compliance with the 2019 guidance. This assessment clearly documents key testing and analysis decisions as well as how officials prepared selected samples, the methods they used to synthesize material samples for testing, and that standard operating procedures were followed. Documenting key decisions regarding the testing and analyses of explosive threat materials--especially those that could potentially affect outcomes--should help TSA better ensure that effective decisions are made and that organizational knowledge is retained regardless of changes in personnel. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The TSA Administrator should require and ensure that TSA officials document their assessments of risk and the rationale—including the assumptions, methodology, and uncertainty considered—behind decisions to deploy screening technologies. (Recommendation 3) |
In December 2019, we reported that TSA's process for incorporating risk into its plans for deploying screening technologies to specific airports lacks transparency. Consequently, we recommended that TSA should require and ensure that agency officials document their assessments of risk and the rationale behind decisions to deploy screening technologies. In September 2021, TSA provided the Requirements and Capabilities Analysis (RCA) Deployment Guiding Principles, which specifies that officials should determine the overall risk of airports and use that assessment to inform deployment strategies. While it is commendable that TSA has developed this deployment guidance, the guidance does not require officials to document risk assessments and the rationale behind deployment decisions. In May 2023, TSA stated that factors other than risk-such as training, logistics, site readiness, and long-term operational efficiency-are almost always the predominant factors influencing deployment. To fully address this recommendation, TSA should require agency officials to document their assessments of operational risk and the rationale-including the assumptions, methodology, and uncertainty considered-behind decisions to deploy screening technologies as part of screening technology deployment plans. As of August 2024, we are continuing to monitor TSA's implementation of this recommendation.
|
| Transportation Security Administration | The TSA Administrator should develop a process to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 4) |
In December 2019, we reported that TSA practices do not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. We recommended that TSA develop a process to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. In May 2020, TSA provided the Post Implementation and Periodic Review Policy (APM-20-031). According to the Policy, TSA will use Post Implementation Reviews (PIR) to explain how screening technology performance, including detection, is to be assessed over time, following deployment. Each technology system is to require its own PIR--or roadmap for reviewing component performance of the detection chain--because each technology has unique logistics data and detection chain. In December 2023, TSA updated the Policy to require comparable requirements for post-PIR (periodic) reviews. Version 1.1 of the Policy clarifies that periodic reviews will include an assessment of system performance requirements relative to effectiveness and suitability, and are to be planned for execution every 2 to 3 years. TSA program managers may seek to conduct periodic reviews more or less frequently with documented justification and approval from the TSA Component Acquisition Executive. These actions should help TSA assure that screening technologies continue to meet detection requirements after deployment and to be better positioned to take any necessary corrective actions, if needed. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The TSA Administrator should implement the process it develops to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 5) |
In December 2019, we reported that TSA practices do not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. In April 2020, TSA issued a policy for developing a review process to assess performance after the deployment of each technology, including detection over time. Since TSA cannot use live explosives or simulants to test screening technologies, the agency plans to measure, for each technology, the performance of system components within the detection chain instead of directly measuring detecting requirements. As of October 2023, TSA had conducted and provided reports on the results of several post-implementation and periodic reviews. Our analysis of these finalized and draft reports found that TSA has made notable progress. In particular, TSA leveraged the national standard for image quality testing of computed tomography systems to assess detection performance of the checked baggage screening systems. TSA is also working to develop a similar image quality test for advanced imaging technology that would evaluate degradation of detection performance. In May 2023, TSA reported that it had developed and tested a kit to measure the parameters for this image quality test. Generally, we continue to have concerns regarding the sufficiency of test data collected and reported during periodic reviews, and the absence of specific tests plans to measure the detection performance of the advanced imaging and computed tomography systems. In July 2024, TSA provided responses to our questions regarding the results of the most recent review reports, which we are reviewing. To fully address this recommendation, TSA needs to demonstrate that it has developed a review process for the testing of all TSA screening technologies deployed to the field. Since each technology has unique operational parameters and requirements, TSA's plans should address the specific technical approach the agency intends to use to ensure the screening technologies continue to meet detection requirements.
|