Combating Nuclear Smuggling:

Additional Actions Needed to Ensure Adequate Testing of Next Generation Radiation Detection Equipment

GAO-07-1247T: Published: Sep 18, 2007. Publicly Released: Sep 18, 2007.

Additional Materials:

Contact:

Eugene E. Aloise
(202) 512-6870
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are key elements in our national defenses against such threats. DHS has sponsored testing to develop new monitors, known as advanced spectroscopic portal (ASP) monitors. In March 2006, GAO recommended that DNDO conduct a cost-benefit analysis to determine whether the new portal monitors were worth the additional cost. In June 2006, DNDO issued its analysis. In October 2006, GAO concluded that DNDO did not provide a sound analytical basis for its decision to purchase and deploy ASP technology and recommended further testing of ASPs. DNDO conducted this ASP testing at the Nevada Test Site (NTS) between February and March 2007. GAO's statement addresses the test methods DNDO used to demonstrate the performance capabilities of the ASPs and whether the NTS test results should be relied upon to make a full-scale production decision.

Based on our analysis of DNDO's test plan, the test results, and discussions with experts from four national laboratories, we are concerned that DNDO's tests were not an objective and rigorous assessment of the ASPs' capabilities. Our concerns with the DNDO's test methods include the following: (1) DNDO used biased test methods that enhanced the performance of the ASPs. Specifically, DNDO conducted numerous preliminary runs of almost all of the materials, and combinations of materials, that were used in the formal tests and then allowed ASP contractors to collect test data and adjust their systems to identify these materials. It is highly unlikely that such favorable circumstances would present themselves under real world conditions. (2) DNDO's NTS tests were not designed to test the limitations of the ASPs' detection capabilities--a critical oversight in DNDO's original test plan. DNDO did not use a sufficient amount of the type of materials that would mask or hide dangerous sources and that ASPs would likely encounter at ports of entry. DOE and national laboratory officials raised these concerns to DNDO in November 2006. However, DNDO officials rejected their suggestion of including additional and more challenging masking materials because, according to DNDO, there would not be sufficient time to obtain them based on the deadline imposed by obtaining Secretarial Certification by June 26. 2007. By not collaborating with DOE until late in the test planning process, DNDO missed an important opportunity to procure a broader, more representative set of well-vetted and characterized masking materials. (3) DNDO did not objectively test the performance of handheld detectors because they did not use a critical CBP standard operating procedure that is fundamental to this equipment's performance in the field. Because of concerns raised that DNDO did not sufficiently test the limitations of ASPs, DNDO is attempting to compensate for weaknesses in the original test plan by conducting additional studies--essentially computer simulations. While DNDO, CBP, and DOE have now reached an agreement to wait and see whether the results of these studies will provide useful data regarding the ASPs' capabilities, in our view and those of other experts, computer simulations are not as good as actual testing with nuclear and masking materials.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: The Secretary of Homeland Security should, if additional testing is needed, appoint an independent group within DHS, not aligned with the ASP acquisition process, to conduct objective, comprehensive, and transparent testing that realistically demonstrates the capabilities and limitations of the ASP system. This independent group would be separate from the recently appointed independent review panel.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: As GAO reported in May 2009 (GAO-09-655), the involvement of the DHS Science and Technology Directorate provided an independent check, within DHS, of the DHS Domestic Nuclear Detection Office's efforts to develop and test ASPs. The Science and Technology Directorate, which is responsible for developing and implementing the department's test and evaluation policies and standards, will have the lead role in the final phase of ASP testing. The Science and Technology Directorate identified two critical questions to be addressed through operational testing: (1) Will the ASP system improve operational effectiveness (i.e., detection and identification of threats) relative to the current-generation system, and (2) is the ASP system suitable for use in the operational environment at land and sea ports of entry?

    Recommendation: The Secretary of Homeland Security should, once the tests and studies have been completed, evaluated, and validated, determine in cooperation with CBP, DOE, and other stakeholders including independent reviewers, if additional testing is needed.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: The audit work for GAO's 2009 report on ASP testing (GAO-09-655) showed that the DHS Domestic Nuclear Detection Office (DNDO) has cooperated with CBP and DOE regarding the need for additional testing. For example, CBP added a final requirement to integration testing before proceeding to field validation to demonstrate ASPs' ability to operate for 40 hours without additional problems.

    Recommendation: The Secretary of Homeland Security should delay Secretarial Certification and full-scale production decisions of the ASPs until all relevant tests and studies have been completed and limitations to these tests and studies have been identified and addressed. Furthermore, results of these tests and studies should be validated and made fully transparent to the Department of Energy (DOE), the Bureau of Customs and Border Protection (CBP), and other relevant parties.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: DHS did in fact delay Secretarial Certification and, as discussed in GAO-09-655 (May 2009), designed and coordinated a new series of tests, originally scheduled to run from April 2008 through September 2008, to determine whether the new portal monitors meet the certification criteria. The audit work for the 2009 report also showed that DOE and CBP were participating in test design and analysis of test results. Delays to the 2008 round of testing have pushed back DHS' schedule for certification even further. In its May 2009 report, GAO issued an updated recommendation that DHS revise the schedule for ASP testing and certification to allow sufficient time for review and analysis of results from the final phases of testing and completion of all tests, including injection studies.

    Recommendation: The Secretary of Homeland Security should report the results of the tests and analyses to the appropriate congressional committees before large scale purchases of ASP's are made.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: On January 9, 2009 DHS/DNDO officials reported the full, classified results of the ASP testing to the staff of the appropriate committees of Congress. This briefing went into great detail about the capabilities of the ASP and PVT in detecting, among other things, certain nuclear materials. The briefing was based largely on the most recent, 2008 round of field testing, a round that GAO found in GAO-09-655 to be generally valid and much improved over previous rounds of testing. On of July 26, 2011 DNDO's Director, Warren Stern, in testimony before the House Homeland Security Subcommittee on Cyber security, Infrastructure Protection, and Security Technologies, announced that he was canceling the ASP program. As a result, there will be no procurement beyond the very small procurement of ASPs done for testing, evaluation, and research purposes. This briefing was accomplished before any large scale purchases of ASPs were made.

    Apr 7, 2014

    Mar 31, 2014

    Mar 28, 2014

    Mar 26, 2014

    Mar 12, 2014

    Mar 7, 2014

    Feb 27, 2014

    Feb 13, 2014

    Looking for more? Browse all our products here