Aviation Security: DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges

In October 2009, we reported that the Transportation Security Administration (TSA) had completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan was not risk-based. Specifically, the strategic plan and its underlying strategy for the Passenger Screening Program (PSP) were developed using risk information, such as threat information, but were not based on the three elements of risk--threat, vulnerability, and consequence. We recommended that TSA conduct a complete risk assessment, including threat, vulnerability, and consequence assessments, which would apply to the PSP. TSA concurred and, in response, initiated approaches to identify risks at both the strategic and detailed level. At the strategic level, in March 2010, TSA updated its Aviation Modal Risk Assessment (AMRA), previously called the Aviation Domain Risk Assessment, which includes a comprehensive threat, vulnerability, and consequence assessment of the risk of a terrorist attack on the international and domestic aviation system. TSA has indicated that it will use AMRA to inform its PSP. At a more detailed level, TSA developed a simulation model that can be used to model specific risk scenarios and risk-reduction potential of specific technologies or other countermeasures. According to TSA, the tool will allow the agency to perform analysis at a level not previously available while not disrupting real world checkpoint operations. TSA indicated that the outputs of this tool are being used to inform the strategic planning and research and development processes. This recommendation is closed as implemented.

In October 2009, we reported that the Transportation Security Administration (TSA) had not completed a cost-benefit analysis to prioritize and fund the Passenger Screening Program's (PSP) priorities for investing in checkpoint technologies, as required by the National Infrastructure Protection Plan's (NIPP) risk management framework. At the time of our report, TSA had not developed life-cycle cost estimates of each screening technology the PSP is developing, procuring, or deploying, and could not provide us with information on their priorities for the research and development of checkpoint screening technologies or the processes they followed to develop these priorities. We recommended that TSA develop cost-benefit analyses to assist in prioritizing investments in new checkpoint screening technologies. TSA concurred and completed a life-cycle cost estimate for the PSP in November 2011. While completing a life-cycle cost estimate was an important first step in addressing our recommendation, it did not include cost-benefit analyses to assist in prioritizing investments in new checkpoint screening technologies, as required by the NIPP. In January 2014, TSA completed a cost-benefit analysis of a new checkpoint screening technology that compared the benefits of adding that technology to the passenger screening checkpoint environment with the overall cost to develop, deploy, and implement the technology. In doing so, TSA analyzed the technology's impact on overall checkpoint system security effectiveness, impact on the traveling public, staff utilization, and passenger throughput. While TSA does not require that cost-benefit analyses be conducted for all new technologies in the PSP, TSA's audit liaison told us in April 2014 that program officials said that they intend to conduct a cost-benefit analysis for all future technologies and for the process used for the January 2014 analysis to be repeatable. Weighing the costs and benefits of each checkpoint technology that TSA invests in relative to the costs and benefits of other solutions will help decision makers determine whether investments in technologies or in other security programs will provide the greatest mitigation of risk for the resources that are available. This recommendation is closed as implemented.

We reported in October 2009 that the Transportation Security Administration's (TSA) strategy for the Passenger Screening Program (PSP) did not have a mechanism - such as performance measures or other evaluation methods - to monitor, assess, or test the extent to which investments in new checkpoint technologies reduce or mitigate the risk of terrorist attacks. We recommended that TSA develop quantifiable performance measures to assess the extent to which investments in research, development, and deployment of checkpoint screening technologies achieve performance goals for enhancing security at airport passenger checkpoints. TSA concurred and, in March 2011, told us that all major technologies that are intended to be deployed to the checkpoint are required to undergo extensive testing in an operational environment during the Operational Testing and Evaluation (OT&E) phase. In addition, TSA said that it has gained an enhanced capability by opening the TSA Systems Integration Facility (TSIF), which allows TSA to mirror the constraints of an operational environment while reducing the impact on the flying public. Finally, according to TSA, baseline comparisons are performed, when relevant, for the previously installed technologies compared to the proposed upgrades to determine if the increase in abilities is acceptable. In addition to the testing performed prior to deployment, TSA uses several methods to evaluate technologies on an ongoing basis to ensure operational effectiveness. For example, TSA has implemented the Aviation Screening Assessment Program (ASAP) during which detection is measured and data are collected to identify trends. Also, TSA performs covert testing of the vulnerabilities of both the technologies and the processes used at the checkpoint. In addition, in January 2014, TSA completed a cost-benefit analysis of a new checkpoint technology that analyzed the technology's impact on overall checkpoint system security effectiveness, impact on the traveling public, staff utilization, and passenger throughput. The quantifiable measures were used to provide TSA decision makers with information to assess and prioritize their investment resources, and ensure they are investing in screening technologies to address the security needs of the passenger screening checkpoint with the highest priority. While TSA does not require that cost-benefit analyses be conducted for all new technologies in the PSP, TSA's audit liaison told us in April 2014 that program officials said that they intend to conduct a cost-benefit analysis for all future technologies and for the process used for the January 2014 analysis to be repeatable. This recommendation is closed as implemented.

We reported in October 2009 that the Transportation Security Administration (TSA) completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan was not risk-based. According to TSA officials, the strategic plan and its underlying strategy for the Passenger Screening Program (PSP) were developed using risk information, such as threat information. However, the strategic plan and its underlying strategy did not reflect some of the key risk management principles set forth in DHS's National Infrastructure Protection Plan (NIPP), such as conducting a risk assessment based on the three elements of risk--threat, vulnerability, and consequence--and developing a cost-benefit analysis and performance measures. Without adhering to all key risk management principles as required in the NIPP, TSA lacked assurance that its investments in screening technologies address the highest priority security needs at airport passenger checkpoints. We recommended that TSA, after conducting a complete risk assessment and completing cost-benefit analyses and quantifiable performance measures for the PSP, incorporate the results of these efforts into the PSP strategy as determined appropriate. TSA concurred and told us in January 2014 that the agency was updating its PSP strategic plan and expected to complete it by summer 2014. In June 2014 TSA informed GAO that TSA has moved away from developing strategic plans and has rolled their planning into the Acquisition Program Baseline (APB) which is updated annually. TSA officials stated the APB serves as an operational guide for program management, including laying out the steps to implementing a program and showing the requirements and timeframes needed in rolling out programs. TSA provided an approved APB in September 2014 which included the three elements of risk, and provided quantifiable performance measures for the PSP. As a result of TSA's actions this recommendation is now closed as implemented.

In October 2009, we reported that although the Transportation Security Administration (TSA) had tested earlier models of the explosives trace portal (ETP), the models ultimately chosen were not operationally tested before they were deployed to ensure they demonstrated effective performance in an operational environment. We recommended that TSA, to the extent feasible, ensure that technologies have completed operational tests and evaluations before they are deployed. TSA concurred with the recommendation and has taken actions to address it. For example, TSA implemented a test and evaluation process for all of its technology procurements in accordance with the Department of Homeland Security's policy. This includes developmental testing in the lab and operational testing in the field. Specifically, according to TSA, Operational Test and Evaluation (OT&E) focuses on validating operational effectiveness and suitability. A Test and Evaluation Master Plan (TEMP) is developed to describe the test and evaluation strategy for determining a system's operational effectiveness and suitability. An operational test plan is developed and approved, and an Operational Test Readiness Review is conducted prior to the start of testing. According to TSA, these elements ensure that operational tests and evaluations have been successfully completed before deploying checkpoint screening technologies to airport checkpoints. Subsequently, TSA provided a configuration management plan that, if implemented as required, should ensure that the specific configuration of a checkpoint screening technology that passes OT&E is the same one that is deployed to airports for day-to-day operational use. This recommendation is closed as implemented.

In October 2009, we reported that the Transportation Security Administration (TSA) had relied on technologies in day-to-day airport operations that had not been demonstrated to meet their functional requirements in an operational environment. For example, TSA substituted existing screening procedures with screening by the Whole Body Imager (later known as Advanced Imaging Technology) even though its performance had not yet been validated by testing in an operational environment. We recommended that TSA evaluate whether current passenger screening procedures should be revised to require the use of appropriate screening procedures until TSA determined that existing emerging technologies meet its functional requirements in an operational environment. TSA agreed with this recommendation. Subsequently, TSA provided us with documentation verifying that the agency had compared the effectiveness of its existing screening procedures and technologies that had not yet completed operational testing and were emerging at the time of our October 2009 report, and considering modifications to those procedures until operational testing had been completed. Therefore, this recommendation is closed as implemented.

In October 2009, we reported that the Transportation Security Administration (TSA) had relied on technologies in day-to-day airport operations that had not been demonstrated to meet their functional requirements in an operational environment. For example, TSA substituted existing screening procedures with screening by the Whole Body Imager (now known as Advanced Imaging Technology) even though its performance had not yet been validated by testing in an operational environment. We recommended that TSA should, in the future, prior to testing or using all checkpoint screening technologies at airports, determine whether TSA's passenger screening procedures should be revised to require the use of appropriate screening procedures until the performance of the technologies has been validated through successful testing and evaluation. TSA agreed with this recommendation. In response, TSA revised the Test and Evaluation Master Plan for its Passenger Screening Program to include consideration of how new technologies will integrate with existing screening procedures at the checkpoint and evaluation of the procedures during operational testing. TSA also provided documentation that such consideration had occurred as part of operational testing and evaluation of several new passenger screening technologies. Therefore, this recommendation is closed as implemented.

In October 2009, we reported that the Transportation Security Administration (TSA) procured and deployed explosives trace portal equipment even though TSA officials were aware that earlier tests did not demonstrate reliable performance in an airport environment. We recommended that TSA conduct an evaluation and determine whether it was cost effective to continue to use these machines. TSA concurred with this recommendation and later halted further deployment of these machines due to performance, maintenance, and installation problems. As of April 2011, TSA reported and provided documented evidence that it had removed from airports all 101 machines that it had deployed. This recommendation is closed as implemented.