TSA Acquisitions: Further Actions Needed to Improve Efficiency of Screening Technology Test and Evaluation
Highlights
What GAO Found
The Transportation Security Administration's (TSA) test and evaluation process has enabled TSA and Department of Homeland Security (DHS) officials to identify passenger and baggage screening technologies that will meet mission needs, but technology failures during testing have contributed to inefficiencies in the acquisition process. Consistent with departmental guidance and acquisition best practices, TSA's test and evaluation process provides information regarding the ability of technologies to meet mission needs before agency officials decide whether to begin full production, saving the agency from investing in potentially expensive yet ineffective equipment. From June 2010 to July 2015, half of the 22 systems that TSA tested successfully completed qualification and operational testing. TSA procured all but 1 of the 11 successful systems. Technologies that entered the test and evaluation process and were immature required significant modifications and retesting.
Number of Passenger and Baggage Screening Systems (Including Upgrades) Completing TSA Test and Evaluation Phases from June 2010 to July 2015
TSA has taken steps to improve its test and evaluation process by helping ensure technologies are mature before entering testing, but it is too soon to tell whether these actions will address all of the factors that contribute to acquisition inefficiencies. A key action TSA is taking involves developing a third party testing strategy, through which a third party will help ensure systems are mature prior to entering TSA's test and evaluation process. TSA plans to implement its approach in 2016, but it has yet to finalize key aspects of the strategy. For example, TSA has not identified whether there are a sufficient number of eligible third party testers or established a mechanism to oversee that testing. Without a finalized strategy, TSA risks unintended consequences, such as increasing acquisition costs. Further, TSA has not conducted or documented a comprehensive assessment of testing data and thus may be missing opportunities to identify additional areas for improvements to its acquisition process. An assessment of this data, such as costs incurred, could help TSA guide future reforms to the test and evaluation process to help ensure they address factors contributing to any acquisition inefficiencies.
Why GAO Did This Study
TSA, within the Department of Homeland Security, is responsible for securing the nation's civil aviation system while facilitating the movement of passengers and commerce at approximately 440 airports nationwide. TSA tests passenger and baggage screening technologies developed by industry to ensure they support TSA missions. In reviews from 2010 to 2014, GAO found that TSA encountered challenges in acquiring and deploying technologies. The Transportation Security Acquisition Reform Act contained a provision for GAO to assess TSA's test and evaluation activities for security-related technologies. This report assesses the extent to which (1) TSA's test and evaluation process helps it meet mission needs through the acquisition of passenger and baggage screening technologies, and (2) TSA has taken steps to improve the test and evaluation process.
GAO reviewed DHS and TSA acquisition and test policies, analyzed testing and acquisition documentation for technologies tested in the past five years, observed the testing process at DHS and TSA facilities, and spoke with DHS, TSA, and industry officials.
Recommendations
TSA should finalize its third party testing strategy before implementation and conduct and document a comprehensive assessment of testing data to identify key factors contributing to any acquisition inefficiencies. DHS concurred with GAO's recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | To help ensure that actions taken to improve the test and evaluation process address identified challenges, the Administrator of TSA should finalize all aspects of the third party testing strategy before implementing further third party testing requirements for vendors to enter testing. |
In our December 2015 report, we recommended that the Administrator of the Transportation Security Administration (TSA) finalize all aspects of the third party testing strategy before implementing further third party requirements for vendors to enter testing. This recommendation was made to mitigate the strategy's potential risks of unintended consequences. Certain aspects of the strategy-such as identifying how TSA would oversee third party testing as well as identifying the potential universe of third party testers needed to be finalized before proceeding further. In its response to this recommendation, the Department of Homeland Security concurred and identified initial planned actions to implement the finalized third party strategy. Subsequently, in the spring of 2016, the TSA Office of Security Capabilities finalized its third party tester application and approval process; established quality conformance standards for potential third party testers; and gathered and considered industry feedback on potential third party test strategy consequences among other actions. Collectively, TSA established and published program requirements and procedures for the third party test strategy. In late 2016, TSA formally delayed its planned implementation of the third-party testing program by a calendar year to instead be completed by December 31, 2017. TSA cited a need to conduct additional assessments, coordination challenges, and larger TSA security equipment related initiatives as the reasons for the delay. However, TSA had identified numerous potential third party testers. As part of its regular recommendation status reporting to GAO, TSA in the spring 2017, noted that it is was on track to meet the intent of the recommendation by the later revised date. TSA noted it had recently updated its qualification process by which qualified product lists will be populated and has already incorporated various aspects of third party testing for legacy security equipment qualification. In August 2017, TSA indicated that would produce two additional deliverables covering how TSA would implement the use of third party testing for vendors to enter testing. Specifically, TSA developed an Acquisition Qualification Policy and an Appendix to the revised TSA and Evaluation Guidebook. In a January 2018, TSA provided GAO with copies of these finalized and approved documents which, according to TSA, establish a consistent acquisition approach and allows TSA to ensure vendor provided information (such as third party test data) is sufficient to demonstrate system readiness to enter the TSA test and evaluation process. We believe the changes included in the updated documents meet the intent of this recommendation.
|
| Transportation Security Administration | To help ensure that the reforms TSA has underway are informed by existing information, the Administrator of TSA should conduct and document a comprehensive assessment of testing data available to date, such as timeframes for completing testing, costs incurred, and testing delays across all technology areas to identify key factors contributing to any acquisition inefficiencies and potential areas for reform. |
In its response to this recommendation, the Department of Homeland Security concurred and established a plan to complete such an assessment by June 30, 2016. In February 2016, TSA updated us on the status of this recommendation and noted it had completed its test and evaluation master tracking tool which would be essential to performing the comprehensive assessment. TSA also notified us that it planned to analyze the tracker's outputs and interview key stakeholders to further understand all testing and evaluation challenges. The tracker included information on technology testing activities since June 2010, which is consistent with our report review. In late July 2016, provided us with the results of the comprehensive assessment of available testing data it completed. The assessment incorporated analyses of requirements definition challenges, schedule delays, and cost impacts of technology failures. Based on this analysis, TSA identified key issues driving technology acquisition delays and cost overruns. These problems included system immaturity as well as internal agency coordination problems among other items. Based on the assessment's findings, TSA is continuing to implement key initiatives aimed at driving more efficient, technology acquisition outcomes.
|