This is the accessible text file for GAO report number GAO-09-655 
entitled 'Combating Nuclear Smuggling: DHS Improved Testing of Advanced 
Radiation Detection Portal Monitors, but Preliminary Results Show 
Limits of the New Technology' which was released on June 22, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

May 2009: 

Combating Nuclear Smuggling: 

DHS Improved Testing of Advanced Radiation Detection Portal Monitors, 
but Preliminary Results Show Limits of the New Technology: 

GAO-09-655: 

GAO Highlights: 

Highlights of GAO-09-655, a report to congressional requesters. 

Why GAO Did This Study: 

The Department of Homeland Security’s (DHS) Domestic Nuclear Detection 
Office (DNDO) is testing new advanced spectroscopic portal (ASP) 
radiation detection monitors. DNDO expects ASPs to reduce both the risk 
of missed threats and the rate of innocent alarms, which DNDO considers 
to be key limitations of radiation detection equipment currently used 
by Customs and Border Protection (CBP) at U.S. ports of entry. Congress 
has required that the Secretary of DHS certify that ASPs provide a 
significant increase in operational effectiveness before obligating 
funds for full-scale procurement. GAO was asked to review (1) the 
degree to which DHS’s criteria for a significant increase in 
operational effectiveness address the limitations of existing radiation 
detection equipment, (2) the rigor of ASP testing and preliminary test 
results, and (3) the ASP test schedule. GAO reviewed the DHS criteria, 
analyzed test plans, and interviewed DHS officials. 

What GAO Found: 

The DHS criteria for a significant increase in operational 
effectiveness require a minimal improvement in the detection of threats 
and a large reduction in innocent alarms. Specifically, the criteria 
require a marginal improvement in the detection of certain weapons-
usable nuclear materials, considered to be a key limitation of current-
generation portal monitors. The criteria require improved performance 
over the current detection threshold, which for certain nuclear 
materials is based on the equipment’s limited sensitivity to anything 
more than lightly shielded materials, but do not specify a level of 
shielding that smugglers could realistically use. In addition, DNDO has 
not completed efforts to improve current-generation portal monitors’ 
performance. As a result, the criteria do not take the current 
equipment’s full potential into account. With regard to innocent 
alarms, the other key limitation of current equipment, meeting the 
criteria could result in hundreds fewer innocent alarms per day, 
thereby reducing CBP’s workload and delays to commerce. 

DHS increased the rigor of ASP testing in comparison with previous 
tests. For example, DNDO mitigated the potential for bias in 
performance testing (a concern GAO raised about prior testing) by 
stipulating that there would be no ASP contractor involvement in test 
execution. Such improvements added credibility to the test results. 
However, the testing still had limitations, such as a limited set of 
scenarios used in performance testing to conceal test objects from 
detection. Moreover, the preliminary results are mixed. The results 
show that the new portal monitors have a limited ability to detect 
certain nuclear materials at anything more than light shielding levels: 
ASPs performed better than current-generation portal monitors in 
detection of such materials concealed by light shielding approximating 
the threat guidance for setting detection thresholds, but differences 
in sensitivity were less notable when shielding was slightly below or 
above that level. Testing also uncovered multiple problems in ASPs 
meeting the requirements for successful integration into operations at 
ports of entry. CBP officials anticipate that, if ASPs are certified, 
new problems will appear during the first few years of deployment in 
the field. 

While DNDO’s schedule underestimated the time needed for ASP testing, 
test delays have allowed more time for review and analysis of results. 
DNDO’s original schedule anticipated completion in September 2008. 
Problems uncovered during testing of ASPs’ readiness to be integrated 
into operations at U.S. ports of entry caused the greatest delays to 
this schedule. DHS’s most recent schedule anticipated a decision on ASP 
certification as early as May 2009, but DHS recently suspended field 
validation due to ASP performance problems and has not updated its 
schedule for testing and certification. In any case, DNDO does not plan 
to complete computer simulations that could provide additional insight 
into ASP capabilities and limitations prior to certification even 
though delays have allowed more time to conduct the simulations. DNDO 
officials believe the other tests are sufficient for ASPs to 
demonstrate a significant increase in operational effectiveness. 

What GAO Recommends: 

GAO recommends that DHS assess ASPs against the full potential of 
current equipment and revise the program schedule to allow time to 
conduct computer simulations of ASPs’ capabilities and to uncover and 
resolve problems with ASPs before full-scale deployment. DHS agreed to 
a phased deployment that should allow time to uncover ASP problems but 
disagreed with GAO’s other recommendations. GAO believes its 
recommendations remain valid. 

View [hyperlink, http://www.gao.gov/products/GAO-09-655] or key 
components. For more information, contact Gene Aloise at (202) 512-3841 
or aloisee@gao.gov. 

[End of section] 

Contents: 

Letter: 

DHS's Criteria for Significant Increase in Operational Effectiveness 
Require a Marginal Improvement in the Detection of Certain Nuclear 
Materials and a Large Reduction in Innocent Alarms: 

DHS Increased the Rigor of Advanced Portal Monitor Testing: 

Schedule Delays Have Allowed More Time for Analysis and Review of Test 
Results, but DNDO's Latest Schedule Does Not Include Computer 
Simulations to Provide Additional Insight into ASP Capabilities: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Comments from the Department of Homeland Security: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Figures: 

Figure 1: DHS Criteria for Demonstrating a Significant Increase in 
Operational Effectiveness: 

Figure 2: Test Sequence Leading Up to ASP Certification: 

Figure 3: Preliminary Results from 2008 Performance Testing for 
Detection of Certain Nuclear Materials: 

Abbreviations: 

ASP: advanced spectroscopic portal: 

CBP: Customs and Border Protection: 

DHS: Department of Homeland Security: 

DNDO: Domestic Nuclear Detection Office: 

DOE: Department of Energy: 

HEU: highly enriched uranium: 

PVT: polyvinyl toluene: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

May 21, 2009: 

Congressional Requesters: 

Preventing radioactive material from being smuggled into the United 
States is a key national security objective. In particular, terrorists 
could use special nuclear material such as highly enriched uranium 
(HEU) or plutonium in a nuclear weapon; other radioactive materials 
could be used in a radiological dispersal device (a "dirty bomb"). The 
national security mission of U.S. Customs and Border Protection (CBP), 
an agency within the Department of Homeland Security (DHS), includes 
screening for smuggled nuclear or radiological material while 
facilitating the flow of legitimate trade and travel. To screen cargo 
at ports of entry, CBP conducts primary inspections with radiation 
detection equipment called portal monitors--large stationary detectors 
through which cargo containers and vehicles pass as they enter the 
United States. When radiation is detected, CBP conducts secondary 
inspections using a second portal monitor to confirm the original alarm 
and a handheld radioactive isotope identification device to identify 
the radiation's source and determine whether it constitutes a threat. 
CBP officers must investigate each alarm until they are convinced that 
the vehicle, occupants, and any cargo pose no threat and can be allowed 
to enter the United States. 

According to DHS's Domestic Nuclear Detection Office (DNDO), the 
current generation of radiation detection equipment has limitations. 
[Footnote 1] Specifically, the polyvinyl toluene (PVT) portal monitors 
currently in use can detect radiation but cannot identify the source. 
As a result, the monitors' radiation alarms can be set off even by 
benign, naturally occurring radioactive material. One way to reduce the 
rate of such innocent alarms--and thereby minimize unnecessary 
secondary inspections and enhance the flow of commerce--is to adjust 
the operational thresholds for the level of radiation required for PVTs 
to alarm (i.e., operate the PVTs at a reduced level of sensitivity). 
However, reducing the sensitivity may make it more difficult to detect 
certain nuclear materials. 

Since 2005, DNDO has been developing and testing advanced spectroscopic 
portals (ASP), a new type of portal monitor designed to both detect 
radiation and identify the source. The new portal monitors use 
technology similar to that in handheld identification devices currently 
used for secondary screening. Key differences from handheld 
identification devices include a larger number of detectors, more 
sophisticated software, and a more extensive library of radiation 
signatures that may provide more consistent and rapid screening and may 
increase the likelihood of correct identification. DNDO hopes to use 
the new portal monitors to replace at least some PVTs currently used 
for primary screening, as well as PVTs and handheld identification 
devices currently used for secondary screening. However, the new portal 
monitors cost significantly more than PVTs. We estimated in September 
2008 that the lifecycle cost of each standard cargo version of the ASP 
(including deployment costs) is about $822,000, compared with about 
$308,000 for the PVT standard cargo portal, and that the total program 
cost for DNDO's latest plan for deploying radiation portal monitors-- 
which relies on a combination of ASPs and PVTs and does not deploy 
radiation portal monitors at all border crossings--would be about $2 
billion.[Footnote 2] Moreover, CBP officials expect operation and 
maintenance costs to be significantly higher for ASPs than for PVTs 
because of the greater complexity of ASP equipment. 

Concerned about the performance and cost of the new ASP monitors, 
Congress required the Secretary of Homeland Security to certify that 
the monitors will provide a "significant increase in operational 
effectiveness" before DNDO obligates funds for full-scale ASP 
procurement.[Footnote 3] The Secretary must submit separate 
certifications for primary and secondary inspection. In response, DNDO, 
CBP, and the DHS management directorate jointly issued criteria in July 
2008 for determining whether the new technology provides a significant 
increase in operational effectiveness--four criteria for primary 
screening and two for secondary screening (see figure 1). The primary 
screening criteria require that the new portal monitors detect 
potential threats as well as or better than PVTs, show improved 
performance in detection of HEU, and reduce innocent alarms. To meet 
the secondary screening criteria, the new portal monitors must reduce 
the probability of misidentifying special nuclear material (e.g., HEU 
and plutonium) and the average time to conduct secondary screenings. 

Figure 1: DHS Criteria for Demonstrating a Significant Increase in 
Operational Effectiveness: 

[Refer to PDF for image: illustration] 

Primary screening criteria: 

When special nuclear material is present in cargo without naturally 
occurring radioactive material, the ASP probability of a correct 
operational outcome must be equal to or greater than that of the PVT. 
(For HEU, ASPs must show improved performance compared to PVTs at
operational thresholds.) 

When special nuclear material is present in cargo with naturally 
occurring radioactive material, the ASP in primary must increase the 
probability of a correct operational outcome compared to the current 
end-to-end system. 

When licensable medical or industrial isotopes are present in cargo, 
the ASP probability of a correct operational outcome must be equal to 
or greater than that of the PVT. 

When the only radioactive source present in the cargo is naturally 
occurring radioactive material, the ASP must refer at least 80 percent 
fewer conveyances for further inspection than the PVT. 

Secondary screening criteria: 

When compared to the handheld radioactive isotope identification device,
ASP must reduce, by at least a factor of two, the probability that 
special nuclear material is misidentified as naturally occurring 
radioactive material, a medical/industrial radionuclide, unknown, or no 
source at all. 

When compared to the handheld radioactive isotope identification device,
the ASP must reduce the average time required to correctly release 
conveyances from secondary screening. 

Source: DHS. 

[End of figure] 

To demonstrate a significant increase in operational effectiveness for 
either primary or secondary screening, ASPs must satisfy all of the 
criteria for that deployment option, independent of satisfying the 
criteria for the other option. The criteria generally compare the new 
portal monitors to current-generation equipment as used under CBP's 
standard operating procedure. For example, the standard operating 
procedure for secondary screening calls for inconclusive readings to be 
sent for additional analysis to CBP's Laboratories and Scientific 
Services, which has access to additional software and trained experts. 

DNDO designed and coordinated a series of tests, originally scheduled 
to run from April 2008 through September 2008, to determine whether the 
new portal monitors meet the certification criteria for primary and 
secondary screening and are ready for deployment. Key phases of testing 
completed to date include verifying that ASPs meet DNDO's performance 
specification, which was followed by concurrent testing of the new and 
current equipments' ability to detect and identify threats and of ASPs' 
readiness to be integrated into operations for both primary and 
secondary screening at ports of entry. Two remaining phases not yet 
completed include field validation at four northern and southern border 
crossings and two seaports, as well as an independent evaluation, 
conducted by the DHS Science and Technology Directorate at one of the 
seaports, of the new portal monitors' effectiveness and suitability 
(see figure 2). Two ASP vendors have contracts with DNDO to develop the 
new portal monitors and are participating in the round of testing that 
began in 2008.[Footnote 4] DNDO designed the testing to allow each 
vendor's system to complete all test phases and be certified based on 
its own performance as providing a significant increase in operational 
effectiveness. 

Figure 2: Test Sequence Leading Up to ASP Certification: 

[Refer to PDF for image: illustration] 

Lead agency: DNDO; 
Phase: System qualification test: Verify technical achievement of
ASP performance specification. 

Lead agency: DNDO; 
Phase: Concurrent performance and integration testing: Performance 
testing: Evaluate ASP performance for detection and identification of 
threats; Integration testing: Demonstrate that ASPs are ready to be 
integrated into the interdiction systems at U.S. ports of entry and to 
advance to field validation. 

Lead agency: CBP; 
Phase: Field validation: Perform system installation procedures and 
process, train CBP officers in ASP use, and use ASPs to screen cargo. 

Lead agency: DHS Science and Technology Directorate; 
Phase: Operational test and evaluation: Conduct an independent 
evaluation of ASP effectiveness and suitability when operated by CBP. 

Source: GAO analysis of DNDO information. 

[End of figure] 

We have raised concerns since 2006 regarding DNDO's previous efforts to 
develop and test the new portal monitors. In October 2006, we found 
that DNDO's analysis of the benefits and costs of deploying the new 
portal monitors relied on assumptions of their anticipated performance 
level instead of actual test data.[Footnote 5] Among other things, we 
recommended that DNDO conduct further testing before spending 
additional funds to purchase the new equipment. In September 2007, we 
testified that DNDO's testing at the Department of Energy's (DOE) 
Nevada Test Site did not represent an objective or rigorous assessment 
because DNDO used biased test methods that enhanced the apparent 
performance of the ASPs and did not test the limitations of their 
detection capabilities.[Footnote 6] Most recently, we found in 
September 2008 that a DNDO report on testing conducted in 2007 did not 
accurately depict test results and could potentially be misleading. 
[Footnote 7] We concluded that the results could identify areas for 
improvement but should not be used as indicators of ASPs' overall 
performance. 

In this context, you asked us to review the 2008 round of testing 
leading up to the Secretary of Homeland Security's decision on ASP 
certification. We reviewed (1) the degree to which DHS's criteria for a 
significant increase in operational effectiveness address the 
limitations of the current generation of radiation detection equipment, 
(2) the rigor of the testing as a basis for determining ASPs' 
operational effectiveness and preliminary results of testing completed 
to date, and (3) the extent to which the test schedule allows time for 
DHS to review and analyze results. This report updates our September 
2008 testimony, which included preliminary observations on the DHS 
criteria for a significant increase in operational effectiveness and 
the 2008 round of testing.[Footnote 8] 

To conduct our review, we analyzed DHS's criteria for a significant 
increase in operational effectiveness and DNDO's written response to 
our detailed questions regarding the criteria. Because the criteria 
compare the new portal monitors to existing equipment, we analyzed the 
threat guidance used to set detection thresholds for PVTs and 
interviewed DOE and national laboratory officials responsible for the 
guidance. In addition, we analyzed the test plans for the 2008 round of 
testing, including the test schedule and reasons for any delays. We 
interviewed DNDO, CBP, and other DHS officials responsible for 
conducting and monitoring tests, and we observed 1 day each of 
performance testing at the Nevada Test Site and integration testing at 
DOE's Pacific Northwest National Laboratory. We analyzed preliminary or 
final results for the phases of testing completed during our review, 
and we interviewed DNDO and CBP officials regarding the results. 
(Appendix I presents a detailed discussion of the scope and methodology 
of our review.) 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives.[Footnote 9] 

DHS's Criteria for Significant Increase in Operational Effectiveness 
Require a Marginal Improvement in the Detection of Certain Nuclear 
Materials and a Large Reduction in Innocent Alarms: 

Although the DHS criteria for primary screening require an improved 
ability to detect certain nuclear materials at operational thresholds, 
ASPs could meet the criteria for improvement while still failing to 
detect anything more than lightly shielded material. DNDO officials 
acknowledge that passive radiation detection equipment, which includes 
both the new and current-generation portal monitors, is capable of 
detecting certain nuclear materials only when this material is 
unshielded or lightly shielded.[Footnote 10] For this reason, the DOE 
threat guidance used to set PVTs' detection threshold is based on the 
equipment's limited sensitivity to anything more than lightly shielded 
nuclear material rather than on the assumption that smugglers would 
take effective shielding measures. DOE developed the guidance in 2002 
and 2003 when CBP began deploying PVTs for primary screening. DOE and 
national laboratory officials responsible for the guidance told us the 
assumption of light shielding was based not on an analysis of the 
capabilities of potential smugglers to take effective shielding 
measures but rather on the limited sensitivity of PVTs to detect 
anything more than certain lightly shielded nuclear materials. In 
contrast, PVTs are more sensitive to the relatively strong radiation 
signature of other nuclear materials, and the threat guidance assumes a 
higher level of shielding for setting the operational threshold for 
detection of such materials. However, even for such materials, the DOE 
threat guidance assumes that shielding would not exceed a level 
provided by the contents of an average cargo container. 

Moreover, DNDO has not completed efforts to fine-tune PVTs' software 
and thereby improve sensitivity to nuclear materials. As a result, the 
criteria compare ASPs to the current performance of PVTs and do not 
take potential improvements into account, which affects any assessment 
of "significant" improvement over current technology. DNDO officials 
expect they can achieve small improvements to PVTs' performance through 
additional development of "energy windowing," a technique currently 
being used in PVTs to provide greater sensitivity than otherwise 
possible. Pacific Northwest National Laboratory officials responsible 
for developing the technique also told us small improvements may be 
possible, and CBP officials have repeatedly urged DNDO to investigate 
the potential of the technique. DNDO collected the data needed to 
further develop energy windowing during the 2008 performance testing at 
the Nevada Test Site but has not yet funded Pacific Northwest National 
Laboratory efforts to analyze the data and further develop the 
technique. 

Other aspects of the criteria for a significant increase in operational 
effectiveness require that ASPs either provide more than a marginal 
improvement in addressing other limitations of current-generation 
equipment or at least maintain the same level of performance in areas 
in which the current-generation equipment is considered adequate: 

* The primary screening requirement for an 80 percent reduction in the 
rate of innocent alarms could result in hundreds of fewer secondary 
screenings per day, thereby reducing CBP's workload and delays to 
commerce. The actual reduction in the volume of innocent alarms would 
vary and would be greatest at the nation's busiest ports of entry, such 
as Los Angeles/Long Beach, where CBP officials report that PVTs 
generate up to about 600 innocent alarms per day.[Footnote 11] A DNDO 
official said the requirement for an 80 percent reduction in innocent 
alarms was developed in conjunction with CBP and was based on a level 
that would provide meaningful workload relief. 

* The primary screening criteria requiring that ASPs provide at least 
the same level of sensitivity to plutonium and medical and industrial 
isotopes, but not specifying an improvement, were based on DNDO's 
assessment that PVTs adequately detect such materials, which have a 
stronger radiation signature than HEU.[Footnote 12] In addition, CBP 
officials said that including medical and industrial isotopes in the 
criteria addressed a CBP requirement for verifying that those 
transporting certain quantities of these materials into the United 
States are properly licensed.[Footnote 13] 

* The secondary screening requirement that ASPs reduce the probability 
of misidentifying special nuclear material by one-half addresses the 
inability of relatively small handheld devices to consistently locate 
and identify potential threats in large cargo containers. For example, 
a handheld device may fail to correctly identify special nuclear 
material if the material is well-shielded or the device is not placed 
close enough to a radiation source to obtain a recognizable 
measurement. According to CBP and DNDO, the requirement for a reduction 
in the average time to conduct secondary screenings is not more 
specific because the time varies significantly among ports of entry and 
types of cargo being screened. 

DHS Increased the Rigor of Advanced Portal Monitor Testing: 

Improvements to the 2008 round of testing addressed concerns we raised 
about earlier rounds of ASP testing. However, the testing still had 
limitations, and the preliminary results are mixed. 

Improvements to Testing Provided Credibility to Test Results: 

As we testified in September 2008, DHS's improvements to the 2008 round 
of ASP testing addressed concerns we raised about previous tests. A 
particular area of improvement was in the performance testing at the 
Nevada Test Site, where DNDO compared the capability of ASP and current-
generation equipment to detect and identify nuclear and radiological 
materials, including those that could be used in a nuclear weapon. The 
improvements addressed concerns we previously raised about the 
potential for bias and provided credibility to the results within the 
limited range of scenarios tested by DNDO. For example, we reported in 
2007 that DNDO had allowed ASP contractors to adjust their systems 
after preliminary runs using the same radiological materials that would 
be used in the formal tests. In contrast, the plan for the 2008 
performance test stipulated that there would be no system contractor 
involvement in test execution, and no ASP contractors were at the test 
location on the day we observed performance testing. Furthermore, DNDO 
officials told us, and we observed, that they did not conduct 
preliminary runs with threat objects used in the formal tests. In 2007, 
we reported that DNDO did not objectively test the handheld 
identification devices because it did not adhere to CBP's standard 
operating procedure for using the devices to conduct a secondary 
inspection, which is fundamental to the equipment's performance in the 
field. DNDO addressed this limitation in the 2008 round of performance 
testing: CBP officers operated the devices and adhered as closely to 
the standard operating procedure as test conditions allowed. While the 
test conditions did not allow CBP officers to obtain real-time 
technical support in interpreting the device's measurements, as they 
would in the field to increase the probability of correctly identifying 
a radiation source, DNDO officials said they addressed this limitation. 
For example, they treated a decision by a CBP officer to indicate the 
need for technical support as a correct outcome if the test scenario 
involved the use of a potential threat, such as HEU. 

Other aspects of testing, while not specifically addressing concerns we 
previously raised, also added credibility to the test results. Based on 
our analysis of the performance test plan, we concluded that the test 
design was sufficient to identify statistically significant differences 
between the new technology and current-generation systems when there 
were relatively large differences in performance. Specifically, DNDO 
conducted a sufficient number of runs of each scenario used in the 2008 
performance testing to identify such differences. 

With regard to the general conduct of the 2008 round of testing, two 
aspects, in particular, enhanced the overall rigor of the tests: (1) 
criteria for ensuring that ASPs met the requirements for each phase 
before advancing to the next, and (2) the participation of CBP and the 
DHS Science and Technology Directorate.[Footnote 14] The test and 
evaluation master plan established criteria requiring that the ASPs 
have no critical or severe issues rendering them completely unusable or 
impairing their function before starting or completing any test phase. 
In addition, the criteria established a cumulative limit of 10 issues 
requiring a work-around (e.g., a straightforward corrective step, such 
as a minor change in standard operating procedures) and 15 cosmetic 
issues not affecting proper functioning. DNDO and CBP adhered to the 
criteria even though doing so resulted in integration testing conducted 
at the Pacific Northwest National Laboratory taking longer than 
anticipated and delaying the start of field validation. For example, 
DNDO and CBP did not allow a vendor's ASP system to complete 
integration testing until all critical or severe issues had been 
resolved. 

The involvement of CBP and the DHS Science and Technology Directorate 
provided an independent check, within DHS, of DNDO's efforts to develop 
and test the new portal monitors. For example, the lead CBP official 
involved in ASP testing told us that DNDO provided an initial 
assessment of the severity of issues uncovered during testing, but CBP 
made the final decision on categorizing them as critical, severe, work- 
around, or cosmetic issues. CBP also added a final requirement to 
integration testing before proceeding to field validation to 
demonstrate ASPs' ability to operate for 40 hours without additional 
problems. According to CBP officials, their efforts to resolve issues 
prior to field validation reflect the importance CBP places on ensuring 
that ASPs are sufficiently stable and technically mature to operate 
effectively in a working port of entry and thereby provide for a 
productive field validation. 

The DHS Science and Technology Directorate, which is responsible for 
developing and implementing the department's test and evaluation 
policies and standards, will have the lead role in the final phase of 
ASP testing; the final phase, consisting of 21 days of continuous 
operation, is scheduled to begin at one seaport after the completion of 
field validation. The Science and Technology Directorate identified two 
critical questions to be addressed through operational testing: (1) 
Will the ASP system improve operational effectiveness (i.e., detection 
and identification of threats) relative to the current-generation 
system, and (2) is the ASP system suitable for use in the operational 
environment at land and sea ports of entry? The suitability of ASPs 
includes factors such as reliability, maintainability, and 
supportability. Because the operational testing conducted at one 
seaport is not sufficient to fully answer these questions--for example, 
because the testing will not allow threat objects to be inserted into 
cargo containers--the directorate plans to also conduct an independent 
analysis of the results from previous test phases, including 
performance testing. 

The 2008 testing still had limitations, which do not detract from the 
test results' credibility but do require that results be appropriately 
qualified. Limitations included the following: 

* The number of handheld identification device measurements collected 
during performance testing was sufficient to distinguish only 
particularly large differences from ASPs' identification ability. In 
particular, the standard operating procedure for conducting secondary 
inspections using ASPs, which requires less time than when using 
handheld devices, allowed DNDO to collect more than twice as many ASP 
measurements and to test ASPs' identification ability against more 
radiation sources than used to test handheld identification devices. 

* The performance test results cannot be generalized beyond the limited 
set of scenarios tested. For example, DNDO used a variety of masking 
and shielding scenarios designed to include cases where both systems 
had 100 percent detection, cases where both had zero percent detection, 
and several configurations in between so as to estimate the point where 
detection capability ceased.[Footnote 15] However, the scenarios did 
not represent the full range of possibilities for concealing smuggled 
nuclear or radiological material. For example, DNDO only tested 
shielding and masking scenarios separately, to differentiate between 
the impacts of shielding and masking on the probabilities of detection 
and identification. As a result, the performance test results cannot 
show how well each system would detect and identify nuclear or 
radiological material that is both shielded and masked, which might be 
expected in an actual smuggling incident. Similarly, DNDO used a 
limited number of threat objects to test ASPs' detection and 
identification performance, such as weapons-grade plutonium but not 
reactor-grade plutonium, which has a different isotopic composition. A 
report on special testing of ASPs conducted by Sandia National 
Laboratories in 2007 recommended that future tests use plutonium 
sources having alternative isotopic compositions. Sandia based its 
recommendations on results showing that the performance of ASP systems 
varied depending on the isotopic composition of plutonium. 

* The Science and Technology Directorate's operational testing is 
designed to demonstrate that the average time between equipment 
failures (the measure of ASPs' reliability) is not less than 1,000 
hours. Thus, the testing will not show how reliable the equipment will 
be over a longer term. DHS Science and Technology Directorate officials 
recognize this limitation and said they designed operational testing 
only to demonstrate compliance with the ASP performance specification. 
Furthermore, to the extent that the Science and Technology Directorate 
relies on performance test results to evaluate ASPs' ability to detect 
and identify threats, its analysis of ASPs' effectiveness will be 
subject to the same limitations as the original testing and analysis 
conducted by DNDO. 

Preliminary Test Results Are Mixed: 

The preliminary results presented to us by DNDO are mixed, particularly 
in the capability of ASPs used for primary screening to detect certain 
shielded nuclear materials. However, we did not obtain DNDO's final 
report on performance testing conducted at the Nevada Test Site until 
early April 2009, and thus we had limited opportunity to evaluate the 
report. In addition, we are not commenting on the degree to which the 
final report provides a fair representation of ASPs' performance. 
Preliminary results from performance testing show that the new portal 
monitors detected certain nuclear materials better than PVTs when 
shielding approximated DOE threat guidance, which is based on light 
shielding. In contrast, differences in system performance were less 
notable when shielding was slightly increased or decreased: Both the 
PVTs and ASPs were frequently able to detect certain nuclear materials 
when shielding was below threat guidance, and both systems had 
difficulty detecting such materials when shielding was somewhat greater 
than threat guidance. DNDO did not test ASPs or PVTs against moderate 
or greater shielding because such scenarios are beyond both systems' 
ability. (See figure 3 for a summary of performance test results for 
detection of certain nuclear materials.) 

Figure 3: Preliminary Results from 2008 Performance Testing for 
Detection of Certain Nuclear Materials: 

[Refer to PDF for image: table] 

Portal monitor system: ASP; 
At lowest shielding levels tested: Frequent; 
Light shielding: At about DOE threat guidance: Frequent; 
Light shielding: At more than DOE threat guidance: Difficult; 
Moderate to heavy shielding: Difficult. 

Portal monitor system: PVT; 
At lowest shielding levels tested: Difficult; 
Light shielding: At about DOE threat guidance: Difficult; 
Light shielding: At more than DOE threat guidance: Difficult; 
Moderate to heavy shielding: Difficult. 

Source: GAO analysis of DNDO information. 

Note: The specific amount and type of shielding assumed in DOE threat 
guidance is classified. 

[End of figure] 

With regard to secondary screening, ASPs performed better than handheld 
devices in identification of threats when masked by naturally occurring 
radioactive material. However, differences in the ability to identify 
certain shielded nuclear materials depended on the level of shielding, 
with increasing levels appearing to reduce any ASP advantages over the 
handheld identification devices--another indication of the fundamental 
limitation of passive radiation detection. 

Other phases of testing, particularly integration testing, uncovered 
multiple problems meeting requirements for successfully integrating the 
new technology into operations at ports of entry. Of the two ASP 
vendors participating in the 2008 round of testing, one has fallen 
several months behind in testing due to the severity of the problems it 
encountered during integration testing; the problems were so severe 
that it may have to redo previous test phases to be considered for 
certification. The other vendor's system completed integration testing, 
but CBP suspended field validation of the system after 2 weeks because 
of serious performance problems that may require software revisions. In 
particular, CBP found that the performance problems resulted in an 
overall increase in the number of referrals for secondary screening 
compared to the existing equipment. According to CBP, this problem will 
require significant corrective actions before testing can resume; such 
corrective actions could in turn change the ability of the ASP system 
to detect threats. The problem identified during field validation was 
in addition to ones identified during integration testing, which 
required multiple work-arounds and cosmetic changes before proceeding 
to the next test phase. For example, one problem requiring a work- 
around related to the amount of time it takes for the ASP to sound an 
alarm when a potential threat material has been detected. 
Specifications require that ASPs alarm within two seconds of a vehicle 
exiting the ASP. However, during testing, the vendor's ASP took longer 
to alarm when a particular isotope was detected. The work-around to be 
implemented during field validation requires that all vehicles be 
detained until cleared by the ASP; the effect on commerce must 
ultimately be ascertained during field validation. 

CBP officials anticipate that they will continue to uncover problems 
during the first few years of use if the new technology is deployed in 
the field. The officials do not necessarily regard such problems to be 
a sign that testing was not rigorous but rather a result of the 
complexity and newness of the technology and equipment. 

Schedule Delays Have Allowed More Time for Analysis and Review of Test 
Results, but DNDO's Latest Schedule Does Not Include Computer 
Simulations to Provide Additional Insight into ASP Capabilities: 

Delays to the schedule for the 2008 round of testing have allowed more 
time for analysis and review of results, particularly from performance 
testing conducted at the Nevada Test Site. The original schedule, which 
underestimated the time needed for testing, anticipated completion of 
testing in mid-September 2008 and the DHS Secretary's decision on ASP 
certification between September and November 2008. DHS officials 
acknowledged that scheduling a certification decision shortly after 
completion of testing would leave limited time to complete final test 
reports and said the DHS Secretary could rely instead on preliminary 
reports if the results were favorable to ASPs. DHS's most recent 
schedule anticipated a decision on ASP certification as early as May 
2009, but DHS has not updated its schedule for testing and 
certification since suspending field validation in February 2009 due to 
ASP performance problems. 

Problems uncovered during testing of ASPs' readiness to be integrated 
into operations at U.S. ports of entry have caused the greatest delays 
to date and have allowed more time for DNDO to analyze and review the 
results of performance testing. Integration testing was originally 
scheduled to conclude in late July 2008 for both ASP vendors. The one 
ASP system that successfully passed integration testing did not 
complete the test until late November 2008--approximately 4 months 
behind schedule. (The delays to integration testing were due in large 
part to the adherence of DNDO and CBP to the criteria discussed earlier 
for ensuring that ASPs met the requirements for each test phase.) In 
contrast, delays to performance testing, which was scheduled to run 
concurrently with integration testing, were relatively minor. Both ASP 
systems completed performance testing in August 2008, about a month 
later than DNDO originally planned. 

The schedule delays have allowed more time to conduct injection 
studies--computer simulations for testing the response of ASPs and PVTs 
to the radiation signatures of threat objects randomly "injected" 
(combined) into portal monitor records of actual cargo containers 
transported into the United States, including some containers with 
innocent sources of radiation. However, DNDO does not plan to complete 
the studies prior to the Secretary of Homeland Security's decision on 
certification even though DNDO and other officials have indicated that 
the studies could provide additional insight into the capabilities and 
limitations of advanced portal monitors. According to DNDO officials, 
injection studies address the inability of performance testing 
conducted at the Nevada Test Site to replicate the wide variety of 
cargo coming into the United States and the inability to bring special 
nuclear material and other threat objects to ports of entry and place 
them in cargo during field validation. Similarly, while they 
acknowledged that injection studies have limitations, DOE national 
laboratory officials said the studies can increase the statistical 
confidence in comparisons of ASPs' and PVTs' probability of detecting 
threats concealed in cargo because of the possibility of supporting 
larger sample sizes than feasible with actual testing. A February 2008 
DHS independent review team report on ASP testing also highlighted the 
benefits of injection studies, including the ability to explore ASP 
performance against a large number of threat scenarios at a practical 
cost and schedule and to permit an estimate of the minimum detectable 
amount for various threats.[Footnote 16] 

DNDO has the data needed to conduct the studies. It has supported 
efforts to collect data on the radiation signatures for a variety of 
threat objects, including special nuclear materials, as recorded by 
both ASP and PVT systems. It has also collected about 7,000 usable 
"stream-of-commerce" records from ASP and PVT systems installed at a 
seaport. Furthermore, DNDO had earlier indicated that injection studies 
could provide information comparing the performance of the two systems 
as part of the certification process for both primary and secondary 
screening. However, addressing deficiencies in the stream-of-commerce 
data delayed the studies, and DNDO subsequently decided that 
performance testing would provide sufficient information to support a 
decision on ASP certification. DNDO officials said they would instead 
use injection studies to support effective deployment of the new portal 
monitors. 

Conclusions: 

Given that radiation detection equipment is already being used at ports 
of entry to screen for smuggled nuclear or radiological materials, the 
decision whether to replace existing equipment requires that the 
benefits of the new portal monitors be weighed against the costs. DNDO 
acknowledges that ASPs are significantly more expensive than PVTs to 
deploy and maintain, and based on preliminary results from the 2008 
testing, it is not yet clear that the $2 billion cost of DNDO's 
deployment plan is justified. Even if ASPs are able to reduce the 
volume of innocent cargo referred for secondary screening, they are not 
expected to detect certain nuclear materials that are surrounded by a 
realistic level of shielding better than PVTs could. Preliminary 
results of DNDO's performance testing show that ASPs outperformed the 
PVTs in detection of such materials during runs with light shielding, 
but ASPs' performance rapidly deteriorated once shielding was slightly 
increased. Furthermore, DNDO and DOE officials acknowledged that the 
performance of both portal monitors in detecting such materials with a 
moderate amount of shielding would be similarly poor. This was one of 
the reasons that performance testing did not include runs with a 
moderate level of shielding. 

Two additional aspects of the 2008 round of testing call into question 
whether ASPs' ability to provide a marginal improvement in detection of 
nuclear materials and reduce innocent alarms warrants the cost of the 
new technology. First, the DHS criteria for a significant increase in 
operational effectiveness do not take into account recent efforts to 
improve the current-generation portal monitors' sensitivity to nuclear 
materials through the "energy windowing" technique, most likely at a 
much lower cost. Data on developing this technique were collected 
during the 2008 round of performance testing but have not been 
analyzed. Second, while DNDO made improvements to the 2008 round of ASP 
testing that provided credibility to the test results, its test 
schedule does not allow for completion of injection studies prior to 
certification even though the studies could provide additional insight 
into the performance of the new technology. Without results from 
injection studies, the Secretary of Homeland Security would have to 
make a decision on certification based on a limited number of test 
scenarios conducted at the Nevada Test Site. 

Assuming that the Secretary of Homeland Security certifies ASPs, CBP 
officials anticipate that they will discover problems with the 
equipment when they start using it in the field. Integration testing 
uncovered a number of such problems, which delayed testing and resulted 
in ASP vendors making multiple changes to their systems. Correcting 
such problems in the field could prove to be more costly and time 
consuming than correcting problems uncovered through testing, 
particularly if DNDO proceeds directly from certification to full-scale 
deployment, as allowed under the congressional certification 
requirement that ASPs provide a significant increase in operational 
effectiveness. 

Recommendations for Executive Action: 

We recommend that the Secretary of Homeland Security direct the 
Director of DNDO to take the following two actions to ensure a sound 
basis for a decision on ASP certification: 

* Assess whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a valid comparison with PVTs' full 
performance potential, including the potential to further develop PVTs' 
use of energy windowing to provide greater sensitivity to threats. Such 
a comparison could also be factored into an updated cost-benefit 
analysis to determine whether it would be more cost-effective to 
continue to use PVTs or deploy ASPs for primary screening at particular 
ports of entry. 

* Revise the schedule for ASP testing and certification to allow 
sufficient time for review and analysis of results from the final 
phases of testing and completion of all tests, including injection 
studies. 

If ASPs are certified, we further recommend that the Secretary of 
Homeland Security direct the Director of DNDO to develop an initial 
deployment plan that allows CBP to uncover and resolve any additional 
problems not identified through testing before proceeding to full-scale 
deployment--for example, by initially deploying ASPs at a limited 
number of ports of entry. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to DOE and DHS for their review and 
comment. DOE provided technical comments, which we have incorporated 
into our report as appropriate. DHS's written comments are reproduced 
in appendix II. 

DHS agreed in part with our recommendations. Specifically, DHS stated 
that it believes its plan to deploy ASPs in phases, starting at a small 
number of low-impact locations, is in accordance with our 
recommendation to develop an initial deployment plan that allows 
problems to be uncovered and resolved prior to full-scale deployment. 
We agree that this deployment plan would address our recommendation and 
note that DHS's comments are the first indication provided to us of the 
department's intention to pursue such a plan. 

In contrast, DHS did not concur with our recommendations to (1) assess 
whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a comparison with PVTs' full 
potential, including further developing PVTs' use of energy windowing; 
and (2) revise the ASP testing and certification schedule to allow 
sufficient time for completion of all tests, including injection 
studies. With regard to energy windowing, DHS stated that using current 
PVT performance as a baseline for comparison is a valid approach 
because the majority of increased PVT performance through energy 
windowing has already been achieved. While DHS may be correct, its 
assessment is based on expert judgment rather than the results of 
testing and analysis being considered by the department to optimize the 
use of energy windowing. Given the marginal increase in sensitivity 
required of ASPs, we stand by our recommendation to assess ASPs against 
PVTs' full potential. DHS can then factor PVTs' full potential into a 
cost-benefit analysis prior to acquiring ASPs. On this point, DHS 
commented that its current cost-benefit analysis is a reasonable basis 
to guide programmatic decisions. However, upon receiving DHS's 
comments, we contacted DNDO to obtain a copy of its cost-benefit 
analysis and were told the analysis is not yet complete. 

With regard to injection studies, DHS agreed that the schedule for ASP 
certification must allow sufficient time for review and analysis of 
test results but stated that DHS and DOE experts concluded injection 
studies were not required for certification. DHS instead stated that 
the series of ASP test campaigns would provide a technically defensible 
basis for assessing the new technology against the certification 
criteria. However, DHS did not rebut the reasons we cited for 
conducting injection studies prior to certification, including test 
delays that have allowed more time to conduct the studies and the 
ability to explore ASP performance against a large number of threat 
scenarios at a practical cost and schedule. On the contrary, DHS 
acknowledged the delays to testing and the usefulness of injection 
studies. Given that each phase of testing has revealed new information 
about the capabilities and limitations of ASPs, we believe conducting 
injection studies prior to certification would likely offer similar 
insights and would therefore be prudent prior to a certification 
decision. 

DHS provided additional comments regarding our assessment of the 
relative sensitivity of ASPs and PVTs and our characterization of the 
severity of the ASPs' software problems uncovered during field 
validation. With regard to sensitivity, DHS implied that our 
characterization of the relative ability of ASPs and PVTs is inaccurate 
and misleading because we did not provide a complete analysis of test 
results. We disagree. First, in meetings to discuss the preliminary 
results of performance testing conducted at the Nevada Test Site, DNDO 
officials agreed with our understanding of the ability of ASPs and PVTs 
deployed for primary screening to detect shielded nuclear materials. 
Furthermore, contrary to the assertion that a complete analysis 
requires a comparison of ASPs to handheld identification devices, our 
presentation is consistent with DHS's primary screening criterion for 
detection of shielded nuclear materials, which only requires that ASPs 
be compared with PVTs. Finally, while we agree that the performance 
test results require a more complete analysis, DNDO did not provide us 
with its final performance test report until early April 2009, after 
DHS provided its comments on our draft report. In the absence of the 
final report, which DNDO officials told us took longer than anticipated 
to complete, we summarized the preliminary results that DNDO presented 
to us during the course of our review as well as to congressional 
stakeholders. 

With regard to ASP software problems uncovered during field validation, 
we clarified our report in response to DHS's comment that the severity 
of the problems has not yet been determined. DHS stated that its 
preliminary analysis indicates the problems should be resolved by 
routine adjustments to threshold settings rather than presumably more 
significant software "revisions." However, given the history of lengthy 
delays during ASP testing, we believe that DHS's assessment of the 
severity of problems encountered during field validation may be overly 
optimistic. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the 
Secretaries of Homeland Security and Energy; the Administrator of NNSA; 
and interested congressional committees. The report will also be 
available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staffs have any questions about this report, please 
contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made key contributions to 
this report are listed in appendix III. 

Signed by: 

Gene Aloise: 
Director, Natural Resources and Environment: 

List of Requesters: 

The Honorable Joseph I. Lieberman:
Chairman:
Committee on Homeland Security and Governmental Affairs:
United States Senate: 

The Honorable Henry A. Waxman:
Chairman:
The Honorable John D. Dingell:
Chair Emeritus:
The Honorable Joe Barton:
Ranking Member:
Committee on Energy and Commerce:
House of Representatives: 

The Honorable Bennie G. Thompson:
Chairman:
The Honorable Peter T. King:
Ranking Member:
Committee on Homeland Security:
House of Representatives: 

The Honorable Edolphus Towns:
Chairman:
Committee on Oversight and Government Reform:
House of Representatives: 

The Honorable Bart Gordon:
Chairman:
Committee on Science and Technology:
House of Representatives: 

The Honorable Bart Stupak:
Chairman:
The Honorable Greg Walden:
Ranking Member:
Subcommittee on Oversight and Investigations:
Committee on Energy and Commerce:
House of Representatives: 

The Honorable Yvette D. Clarke:
Chairwoman:
The Honorable Daniel E. Lungren:
Ranking Member:
Subcommittee on Emerging Threats, Cybersecurity, and Science and 
Technology:
Committee on Homeland Security:
House of Representatives: 

The Honorable Charles E. Schumer:
United States Senate: 

The Honorable James R. Langevin:
House of Representatives: 

The Honorable Michael T. McCaul:
House of Representatives: 

[End of section] 

Appendix I: Scope and Methodology: 

To evaluate the degree to which Department of Homeland Security's (DHS) 
criteria for a significant increase in operational effectiveness 
address the limitations of the current generation of radiation 
detection equipment, we clarified the intent of the criteria through 
the Domestic Nuclear Detection Office's (DNDO) written answers to our 
questions and through interviews with U.S. Customs and Border 
Protection (CBP) officials. We also took steps to gain a fuller 
understanding of the strengths and limitations of the current- 
generation equipment, which the criteria use as a baseline for 
evaluating the effectiveness of advanced spectroscopic portals (ASP). 
In particular, we obtained copies of the Department of Energy (DOE) 
threat guidance and related documents used to set polyvinyl toluene 
(PVT) thresholds for detection of nuclear materials. We interviewed DOE 
and national laboratory officials responsible for the threat guidance 
about the process for developing it and the basis for its underlying 
assumptions, including shielding levels. We also interviewed DNDO and 
Pacific Northwest National Laboratory officials regarding the extent to 
which PVTs currently deployed at ports of entry meet the guidance and 
the development and use of energy windowing to enhance PVTs' 
sensitivity to nuclear materials. 

To evaluate the rigor of the 2008 round of testing as a basis for 
determining ASPs' operational effectiveness, we reviewed the test and 
evaluation master plan and plans for individual phases of testing, 
including system qualification testing conducted at vendors' 
facilities, performance testing conducted at the Nevada Test Site for 
evaluating ASP detection and identification capabilities, and 
integration testing conducted at Pacific Northwest National Laboratory 
for evaluating the readiness of ASPs to be used in an operational 
environment at ports of entry. We also reviewed draft plans for field 
validation conducted at CBP ports of entry and the DHS Science and 
Technology Directorate's independent operational test and evaluation. 
In reviewing these documents, we specifically evaluated the extent to 
which the performance test design was sufficient to identify 
statistically significant differences between the ASP and current- 
generation systems and whether DHS had addressed our concerns about 
previous rounds of ASP testing. We interviewed DNDO, CBP, and other DHS 
officials responsible for conducting and monitoring tests, and we 
observed, for one day each, performance testing at the Nevada Test Site 
and integration testing at DOE's Pacific Northwest National Laboratory. 
We also interviewed representatives of entities that supported testing, 
including DOE's National Nuclear Security Administration and Pacific 
Northwest National Laboratory, the National Institute of Standards and 
Technology, and the Johns Hopkins University Applied Physics 
Laboratory. We reviewed the DHS independent review team report of 
previous ASP testing conducted in 2007, and we interviewed the chair of 
the review team to clarify the report's findings. Finally, we examined 
preliminary or final results for the phases of testing completed during 
our review, and we interviewed DNDO and CBP officials regarding the 
results. 

To evaluate the test schedule, we analyzed the initial working schedule 
DNDO provided to us in May 2008 and the schedule presented in the 
August 2008 test and evaluation master plan, and we tracked changes to 
the schedule and the reasons for any delays. We interviewed DNDO and 
other officials with a role in testing to determine the amount of time 
allowed for analysis and review of results. We interviewed DNDO and 
Pacific Northwest National Laboratory officials regarding the injection 
studies, including reasons for delays in the studies and plans for 
including the results as part of the ASP certification process. 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Homeland Security: 

U.S. Department of Homeland Security: 
Washington, DC 20529: 

March 24, 2009: 

Mr. Gene Aloise: 
Director, Natural Resources and Environment: 
U.S Government Accountability Office: 
441 G Street NW: 
Washington, DC 20548: 

Dear Mr. Aloise: 

RE: Draft Report GAO-09-354SU, Combating Nuclear Terrorism: DHS 
Improved Testing of Advanced Radiation Detection Portal Monitors, But 
Preliminary Results Show Limits of the New Technology (360961): 

The Department of Homeland Security (DHS) appreciates the opportunity 
to comment on the above referenced draft report and clarify several 
points. DHS is committed to rigorous testing prior to the deployment of 
new technologies and is pleased that the Government Accountability 
Office (GAO) recognizes improvements that the Domestic Nuclear 
Detection Office (DNDO) has incorporated into its testing programs. 

The comments provided below in response to the subject report are 
restricted to points that are directly relevant to the three areas that 
GAO was asked to review: (1) the degree to which DHS's criteria for 
significant increase in operational effectiveness address the 
limitations of existing radiation detection equipment, (2) the rigor of 
Advanced Spectroscopic Portal (ASP) testing and preliminary test 
results, and (3) the ASP test schedule. 

Report Recommendation: "Assess whether ASPS meet the criteria for 
significant increase in operational effectiveness based on a valid 
comparison with PVTs' full performance potential, including the 
potential to further develop PVTs' use of energy windowing to provide 
greater sensitivity to threats. Such a comparison could also be 
factored in to an updated cost-benefit analysis to determine whether it 
would be more cost-effective to continue to use PVTs or deploy ASPs for 
primary screening at particular ports of entry." 

DHS does not concur with this recommendation. 

It is the judgment of experts that the majority of any increased 
Polyvinyl Toluene (PVT) performance from energy windowing has already 
been achieved in the existing systems; therefore we are confident that 
using the current understanding on PVT energy windowing as a baseline 
is a valid approach. DHS believes that Customs and Border Protection 
(CBP), the Office of the Undersecretary for Management, and DNDO have 
collaborated to establish a meaningful set of criteria to measure the 
relative operational effectiveness of the current ASP and PVT systems. 
Therefore, the current cost-benefit analysis is a reasonable basis to 
guide programmatic decisions. However, given that PVT systems will 
continue to be operated in the field, DNDO has initiated efforts to 
continuously improve the capability of the PVT systems. These efforts 
include an ongoing analysis of potential optimization of the PVT energy 
windowing capability that will proceed in parallel with other 
activities related to Certification. 

Report Recommendation. "Revise the schedule for ASP testing and 
certification to allow sufficient time for review and analysis of 
results from the final phases of testing and completion of all tests, 
including injection studies." 

DHS does not concur with aspects of this recommendation. 

We agree that there must be sufficient time to review the analyses and 
results. The current schedule for certification is predicated on 
completing all of the requisite testing and analysis of results 
required to satisfy the criteria for demonstrating significantly 
improved operational effectiveness. DHS has demonstrated that it is 
implementing a deliberate process to ensure all the pre-established 
exit criteria have been achieved prior to concluding the relevant phase 
of testing. 

However, a team of subject matter experts from the Department of Energy 
(DOE) and DHS concluded that injection studies were not required for 
Certification. The basis of this decision was a review of the series of 
test campaigns planned for ASP and the joint determination that these 
tests would provide a technically defensible basis for assessing the 
ASP technology against the Certification criteria. "Accordingly, 
Certification will be based on the whole body of experimental knowledge 
collected on ASP, including technical performance at Nevada Test Site 
(NTS), integration testing at 33I-G, field validation at ports of 
entry, reanalysis of selected raw data files through a validated replay 
tool, and independent Operational Test & Evaluation. Nevertheless, DHS 
believes that injection studies are useful and will continue to support 
and perform them as an aid for deployment decisions and for future 
development. 

Regarding the conclusions section in your report, we offer the 
following additional comments: 

* Conclusions regarding the relative effectiveness of PVT and ASP to 
detect Highly Enriched Uranium (HEU) require a more complete analysis 
than given in your report. For example, the effectiveness of a system 
should include the entire system (primary plus secondary inspection). 
Such an analysis would necessarily incorporate a comparison of ASP to a 
hand-held RIID. Without including the whole system, conclusions about 
overall detection effectiveness are inaccurate and misleading. 

* We agree with CBP that we will inevitably discover problems with this 
highly advanced and complex system as it transitions into field 
operation. There is simply no way to fully anticipate and replicate all 
real-world problems before deployment to the field: computer modeling 
and simulation does not provide the same level of experience or 
exposure. In anticipation of this fact, deployment will be accomplished 
in phases, starting with a small number of low impact locations. As the 
issues with the system are corrected, we will gradually build up to 
wider deployment. The process will be similar to what was used in the 
successful deployment of PVTs. Therefore, DHS believes ASP deployment 
will be implemented in accordance with your recommendation to the 
Secretary to discover and resolve problems through initial deployment 
to a limited number of locations. 

Finally, there were a few conclusions in the body of your report that 
we would like to comment on: 

* We agree that the NTS test did not "represent the full range of 
possibilities for concealing smuggled nuclear or radiological 
materials" because such a range is impractically large to create. 
However, the NTS test campaign included an extensive array of shielding 
and masking configurations in a plan designed jointly by DHS and DOE to 
cover a range relevant to a passive radiation scanning application. 

* Your report states that "CBP suspended field validation after one 
week because of serious performance problems requiring software 
revisions." Although it remains for analysis to determine the severity 
of the problems encountered, preliminary analysis indicates that the 
problems should be resolved by making adjustments to threshold 
settings. Such adjustments are part of any installation of an RPM (PVT 
or ASP) and are not software revisions. As mentioned above, it is 
impossible to anticipate all the problems that will occur in real-world 
operation, so it is not surprising to encounter problems, given that 
this is the first opportunity for the latest version of ASP to operate 
in the flow of real commerce. 

Thank you for the opportunity to review and provide comments on your 
report. We look forward to working with you on future homeland security 
issues. 

Sincerely, 

Signed by: 

Jerald E. Levine: 
Director: 
Departmental GAO/0IG Liaison Office: 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Gene Aloise, (202) 512-3841 or aloisee@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Ned Woodward, Assistant 
Director; Dr. Timothy Persons, Chief Scientist; James Ashley; Steve 
Caldwell; Joseph Cook; Omari Norman; Alison O'Neill; Rebecca Shea; 
Kevin Tarmann; and Eugene Wisnoski made key contributions to this 
report. 

[End of section] 

Footnotes: 

[1] DNDO was established within DHS in 2005; its mission includes 
developing, testing, acquiring, and supporting the deployment of 
radiation detection equipment at U.S. ports of entry. CBP began 
deploying portal monitors in 2002, prior to DNDO's creation, under the 
radiation portal monitor project. For additional information on DNDO's 
overall efforts to combat nuclear smuggling, see GAO, Nuclear 
Detection: Domestic Nuclear Detection Office Should Improve Planning to 
Better Address Gaps and Vulnerabilities, [hyperlink, 
http://www.gao.gov/products/GAO-09-257] (Washington, D.C.: Jan. 29, 
2009). 

[2] GAO, Combating Nuclear Smuggling: DHS's Program to Procure and 
Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed 
the Department's Previous Cost Estimates, [hyperlink, 
http://www.gao.gov/products/GAO-08-1108R] (Washington, D.C.: Sept. 22, 
2008). 

[3] Consolidated Appropriations Act, 2007, Pub. L. No. 110-161, 121 
Stat. 1844, 2069 (2007); Consolidated Security, Disaster Assistance, 
and Continuing Appropriations Act, 2009, Pub. L. No. 110-329, 121 Stat. 
3574, 3679 (2008). 

[4] DNDO had a contract with a third ASP vendor whose system uses a 
more expensive type of detector that must be cooled by liquid nitrogen. 
DNDO determined it was not in the best interests of the government to 
exercise the option on the contract and allowed it to expire in 
November 2008. The vendor's ASP did not participate in the 2008 round 
of testing. 

[5] GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to 
Support the Purchase of New Radiation Detection Portal Monitors Was Not 
Based on Available Performance Data and Did Not Fully Evaluate All the 
Monitors' Costs and Benefits, [hyperlink, 
http://www.gao.gov/products/GAO-07-133R] (Washington, D.C.: Oct. 17, 
2006). 

[6] GAO, Combating Nuclear Smuggling: Additional Actions Needed to 
Ensure Adequate Testing of Next Generation Radiation Detection 
Equipment, [hyperlink, http://www.gao.gov/products/GAO-07-1247T] 
(Washington, D.C.: Sept. 18, 2007). 

[7] GAO, Combating Nuclear Smuggling: DHS's Phase 3 Test Report on 
Advanced Portal Monitors Does Not Fully Disclose the Limitations of the 
Test Results, [hyperlink, http://www.gao.gov/products/GAO-08-979] 
(Washington, D.C.: Sept. 30, 2008). 

[8] GAO, Combating Nuclear Smuggling: DHS Needs to Consider the Full 
Costs and Complete All Tests Prior to Making a Decision on Whether to 
Purchase Advanced Portal Monitors, [hyperlink, 
http://www.gao.gov/products/GAO-08-1178T] (Washington, D.C.: Sept. 25, 
2008). 

[9] This report does not include certain details about the capabilities 
and limitations of PVTs and ASPs that DHS considers to be "for official 
use only." We have prepared a "for official use only" version of this 
report in which we include such details (GAO-09-354SU). 

[10] According to DNDO and CBP officials, active imaging techniques 
(e.g., radiography systems to provide images of the contents of cargo 
containers) and other measures complement radiation detection 
equipment. In particular, such measures provide the capability to spot 
smuggled nuclear materials that are too heavily shielded to be detected 
by PVTs or ASPs. 

[11] About 45 percent of all sea containers arriving in the United 
States come through Los Angeles/Long Beach. In fiscal year 2006, CBP 
cleared more than 5 million containers through the port. 

[12] The criteria require an improvement when the radiation emitted by 
naturally occurring radioactive material is used to mask smuggled 
special nuclear material, including both HEU and plutonium. 

[13] For additional information regarding the requirement to verify the 
legitimacy of radioactive material shipments, see GAO, Nuclear 
Security: NRC and DHS Need to Take Additional Steps to Better Track and 
Detect Radioactive Materials, [hyperlink, 
http://www.gao.gov/products/GAO-08-598] (Washington, D.C.: June 19, 
2008). 

[14] In the case of ASP testing, the Science and Technology Directorate 
serves as the independent operational test authority, which reports 
directly to the DHS Under Secretary for Management. 

[15] Masking is the use of naturally occurring radioactive material to 
make the radiation emitted by smuggled material appear to be caused by 
innocent cargo. In contrast, shielding blocks radiation from being 
emitted. 

[16] DHS Homeland Security Institute, Independent Review of the 
Department of Homeland Security Domestic Nuclear Detection Office 
Advanced Spectroscopic Portal: Final Report (Feb. 20, 2008). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: