Combating Nuclear Smuggling:

DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology

GAO-09-655, May 21, 2009

Additional Materials:

Contact:

Eugene E. Aloise
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is testing new advanced spectroscopic portal (ASP) radiation detection monitors. DNDO expects ASPs to reduce both the risk of missed threats and the rate of innocent alarms, which DNDO considers to be key limitations of radiation detection equipment currently used by Customs and Border Protection (CBP) at U.S. ports of entry. Congress has required that the Secretary of DHS certify that ASPs provide a significant increase in operational effectiveness before obligating funds for full-scale procurement. GAO was asked to review (1) the degree to which DHS's criteria for a significant increase in operational effectiveness address the limitations of existing radiation detection equipment, (2) the rigor of ASP testing and preliminary test results, and (3) the ASP test schedule. GAO reviewed the DHS criteria, analyzed test plans, and interviewed DHS officials.

The DHS criteria for a significant increase in operational effectiveness require a minimal improvement in the detection of threats and a large reduction in innocent alarms. Specifically, the criteria require a marginal improvement in the detection of certain weapons-usable nuclear materials, considered to be a key limitation of current-generation portal monitors. The criteria require improved performance over the current detection threshold, which for certain nuclear materials is based on the equipment's limited sensitivity to anything more than lightly shielded materials, but do not specify a level of shielding that smugglers could realistically use. In addition, DNDO has not completed efforts to improve current-generation portal monitors' performance. As a result, the criteria do not take the current equipment's full potential into account. With regard to innocent alarms, the other key limitation of current equipment, meeting the criteria could result in hundreds fewer innocent alarms per day, thereby reducing CBP's workload and delays to commerce. DHS increased the rigor of ASP testing in comparison with previous tests. For example, DNDO mitigated the potential for bias in performance testing (a concern GAO raised about prior testing) by stipulating that there would be no ASP contractor involvement in test execution. Such improvements added credibility to the test results. However, the testing still had limitations, such as a limited set of scenarios used in performance testing to conceal test objects from detection. Moreover, the preliminary results are mixed. The results show that the new portal monitors have a limited ability to detect certain nuclear materials at anything more than light shielding levels: ASPs performed better than current-generation portal monitors in detection of such materials concealed by light shielding approximating the threat guidance for setting detection thresholds, but differences in sensitivity were less notable when shielding was slightly below or above that level. Testing also uncovered multiple problems in ASPs meeting the requirements for successful integration into operations at ports of entry. CBP officials anticipate that, if ASPs are certified, new problems will appear during the first few years of deployment in the field. While DNDO's schedule underestimated the time needed for ASP testing, test delays have allowed more time for review and analysis of results. DNDO's original schedule anticipated completion in September 2008. Problems uncovered during testing of ASPs' readiness to be integrated into operations at U.S. ports of entry caused the greatest delays to this schedule. DHS's most recent schedule anticipated a decision on ASP certification as early as May 2009, but DHS recently suspended field validation due to ASP performance problems and has not updated its schedule for testing and certification. In any case, DNDO does not plan to complete computer simulations that could provide additional insight into ASP capabilities and limitations prior to certification even though delays have allowed more time to conduct the simulations. DNDO officials believe the other tests are sufficient for ASPs to demonstrate a significant increase in operational effectiveness.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: The Secretary of Homeland Security should direct the Director of DNDO to assess whether ASPs meet the criteria for a significant increase in operational effectiveness based on a valid comparison with PVTs' full performance potential, including the potential to further develop PVTs' use of energy windowing to provide greater sensitivity to threats to ensure a sound basis for a decision on ASP certification. Such a comparison could also be factored into an updated cost-benefit analysis to determine whether it would be more cost-effective to continue to use PVTs or deploy ASPs for primary screening at particular ports of entry.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: According to DHS's Domestic Nuclear Detection Office (DNDO), the current generation of radiation equipment, the polyvinyl toulene (PVT) portal monitors currently in use have limitations. Specifically, PVTs can detect radiation but cannot identify the source. To deal with this shortcoming, beginning in 2005, DNDO began developing and testing advanced spectroscopic portals (ASP), a new type of portal monitor designed to both detect radiation and identify the sources.However, concerned about the performance and cost of the new ASP monitors, Congress required the Secretary of Homeland Security to certify that the monitors will provide a "significant increase in operational effectiveness" before DNDO obligated funds for full-scale ASP procurement to replace many of the currently deployed PVTs. However, we found that DNDO had not completed efforts to fine tune PVTs' software and thereby improve sensitivity to nuclear materials. As a result, the criteria that compared ASPs to the current performance of PVTs did not take potential improvements into account, which affected any assessment of "significant" improvement over current technology. Accordingly, in May 2009, we recommended that DHS assess whether ASPs meet the criteria for significant increase in operatioal effectiveness based on a valid comparison of with PVTs' full performance potential, including the potential to furhter develop PVTs' use of energy windowing to provide greater sensitivity to threats. Energy windowing is one of the potential improvements that DNDO and CBP are investigating. DNDO has reported to Congress on several occasions on the PVT radiation portal monitor (RPM) improvement projects that are being pursued. Specifically, DNDO has spent $2.6 million in FY 2010, $1.8 million in FY 2012, and has requested $5 million for FY 2013 for research and development ot improve the capabilities of existing PVTs. Examples of projects include optimizing anomaly detection and background radiation. According to DNDO officials, a DHS Acquisition Review Board (ARB) was held in April 2011 to review the ASP program and an Acquisition Decision Memorandum (ADM) was issued recommending to the Secretary of Homeland Security that the department not seek certification of the ASP at that time. Shortly after the issuance of the ADM, the Secretary informed Congress of her decision to cancel the ASP program. In the letter to Congress, the Secretary stated that CBP would continue to rely on currently deployed radiation portal monitors and next-generation hand-held detection devises for secondary scanning for the foreseeable future. Further, according to DNDO offials, DNDO continues to work with CBP to investigate improvements that can be implemented in the currently deployed PVT RPM systems. According to DNDO officials, The John Hopkins University Applied Physics Laboratory is also in the process of conducting verification and validation (V&V) on PVT RPM improvement projects. Once the V&V is completed, DNDO will work with CBP to conduct a study to determine the path forward.

    Recommendation: The Secretary of Homeland Security should direct the Director of DNDO to revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies, to ensure a sound basis for a decision on ASP certification.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: The original schedule for ASP testing and certification was revised several times by DHS/DNDO officials and, ultimately, the program was cancelled by Secretary Napalitano in an October 3, 2011 letter to the Chairman of the House Committee on Homeland Security. When this recommendation was made in May 2008 DNDO/DHS officials were intent on completing the ASP testing and certification process by September 30, 2008. In the event, integration testing, injection studies, and field validation testing extended for years; it was the ultimate failure of ASPs to pass field validation testing after three rounds that led to the 2011 decision to terminate the program.

    Recommendation: If ASPs are certified, the Secretary of Homeland Security should direct the Director of DNDO to develop an initial deployment plan that allows CBP to uncover and resolve any additional problems not identified through testing before proceeding to full-scale deployment--for example, by initially deploying ASPs at a limited number of ports of entry.

    Agency Affected: Department of Homeland Security

    Status: Closed - Not Implemented

    Comments: ASPs were not certified. DNDO/DHS cancelled the program in 2011. This decision was first announced during Congressional testimony by the director of DNDO on July 26, 2011 and confirmed in an October 3, 2011 letter from Secretary Napolitano to the chairman of the House Homeland Security Committee.