Biosurveillance: DHS Should Not Pursue BioWatch Upgrades or Enhancements Until System Capabilities Are Established
Highlights
What GAO Found
The Department of Homeland Security (DHS) lacks reliable information about BioWatch Gen-2's technical capabilities to detect a biological attack and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system. DHS commissioned several tests of the technical performance characteristics of the current BioWatch Gen-2 system, but has not developed performance requirements that would enable it to interpret the test results and draw conclusions about the system's ability to detect attacks. Although DHS officials said that the system can detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, they have not specified the performance requirements necessary to reliably meet this operational objective. In the absence of performance requirements, DHS officials said computer modeling and simulation studies support their assertion. However, none of these studies were designed to incorporate test results from the Gen-2 system and comprehensively assess the system against the stated operational objective. Additionally, DHS has not prepared an analysis that combines the modeling and simulation studies with the specific Gen-2 test results to assess the system's capabilities to detect attacks. Finally, we found limitations and uncertainties in the four key tests of the Gen-2 system's performance. Because it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments, DHS relied on chamber testing and the use of simulated biothreat agents, which limit the applicability of the results. These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack a full understanding of the Gen-2 system's capability to detect attacks of defined types and sizes and cannot make informed decisions about the value of proposed upgrades.
The actions and decisions DHS made regarding the acquisition and testing of a proposed next generation of BioWatch (Gen-3) partially aligned with best practices GAO previously identified for developmental testing of threat detection systems. For example, best practices indicate that resilience testing, or testing for vulnerabilities, can help uncover problems early. While DHS took steps to help build resilience into the Gen-3 testing, future testing could be improved by using more rigorous methods to help predict performance in different operational environments. DHS canceled the Gen-3 acquisition in April 2014, but GAO identified lessons DHS could learn by applying these best practices to the proposed Gen-2 upgrades.
According to experts and practitioners, the polymerase chain reaction (PCR), which detects genetic signatures of biothreat agents, is the most mature technology to use for an autonomous detection system. DHS is considering autonomous detection as an upgrade to Gen-2, because according to DHS, it may provide benefits such as reduction in casualties or clean-up costs. But the extent of these benefits is uncertain because of several assumptions, such as the speed of response after a detection, that are largely outside of DHS's control. As a result, the effectiveness of the response—and the number of lives that could be saved—is uncertain. Further, an autonomous detection system must address several likely challenges, including minimizing possible false positive readings, meeting sensitivity requirements, and securing information technology networks.
Why GAO Did This Study
DHS's BioWatch program aims to provide early indication of an aerosolized biological weapon attack. Until April 2014, DHS pursued a next-generation autonomous detection technology (Gen-3), which aimed to enable collection and analysis of air samples in less than 6 hours, unlike the current system (Gen-2), which requires manual intervention and can take up to 36 hours to detect the presence of biological pathogens. DHS is taking steps to address the capability gap that resulted from the cancellation of Gen-3 by exploring other technology upgrades and improvements to the Gen-2 system.
GAO was asked to review (1) the technical capabilities of the currently deployed BioWatch system, (2) the Gen-3 testing effort, and (3) characteristics of autonomous detection as a possible option to replace the current BioWatch system. GAO analyzed key program documents, including test plans, test results, and modeling studies. GAO assessed Gen-3 testing against best practices, reviewed relevant literature, and discussed the BioWatch program and testing efforts with key agency officials and national laboratories staff.
Recommendations
GAO recommends DHS not pursue upgrades or enhancements for Gen-2 until it reliably establishes the system's current capabilities. GAO also recommends DHS incorporate best practices for testing in conducting any system upgrades. DHS generally concurred with GAO's recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To help ensure that biosurveillance-related funding is directed to programs that can demonstrate their intended capabilities, and to help ensure sufficient information is known about the current Gen-2 system to make informed cost-benefit decisions about possible upgrades and enhancements to the system, the Secretary of Homeland Security should direct the Assistant Secretary for Health Affairs and other relevant officials within the Department to not pursue upgrades or enhancements to the current BioWatch system until the Office of Health Affairs (OHA): (1) establishes technical performance requirements, including limits of detection, necessary for a biodetection system to meet a clearly defined operational objective for the BioWatch program by detecting attacks of defined types and sizes with specified probabilities; (2) assesses the Gen-2 system against these performance requirements to reliably establish its capabilities; and (3) produces a full accounting of statistical and other uncertainties and limitations in what is known about the system's capability to meet its operational objectives. |
In December 2015, DHS concurred with our recommendation, and over the years reported on steps the agency took in an attempt to address this multi-part recommendation designed to help decision-makers make informed cost-benefit decisions about upgrades or enhancements to the BioWatch program. For example, DHS sponsored workshops by the National Academy of Sciences and assessments by multiple national laboratories to assess the performance of the current BioWatch system. We have followed DHS's efforts to improve the BioWatch program, but we do not believe DHS has fully implemented or met the intent of our recommendation. We recognize the complexity in designing biodetection systems for use in the homeland, and that DHS efforts to improve biodetection in a cost conscious manner have improved. For example, in 2018, DHS terminated the Biodetection Technology Enhancements program to enhance BioWatch, because the technology at the time did not provide a cost-effective improvement to the capabilities of the existing BioWatch system. In May 2021, we reported that DHS's efforts to develop a new biodetection capability designed to replace BioWatch-Biological Detection for the 21st Century (BD21)-planned to address our 2015 recommendations as part of its acquisition program. Since issuing that work, DHS has pushed back decision points for the acquisition to allow more time for program acquisition strategy and detailed planning. Additionally, in January 2022, the Countering Weapons of Mass Destruction Office, which manages the BioWatch program and BD21 effort, issued new strategic guidance intended to guide its efforts over the department's environmental biodetection programs. This includes conducting a strategic review, in coordination with its interagency partners, of DHS's biodefense posture and role in the biodefense enterprise. While DHS's efforts to date do not fully respond to our 2015 recommendation, they continue to evolve their biodetection efforts which warrant continued oversight.
|
| Department of Homeland Security | To help reduce the risk of acquiring immature detection technologies, the Secretary of Homeland Security should direct the Assistant Secretary for Health Affairs, in coordination with the Under Secretary for Science and Technology, to use the best practices outlined in this report to inform test and evaluation actions for any future upgrades or changes to technology for BioWatch. |
Since the Department of Homeland Security's (DHS) testing of the BioWatch Generation 3 system in 2010 and 2011 and our evaluations of that testing in 2012 and 2015, DHS's acquisition guidance has evolved to help it improve its ability to manage acquisition programs. For example, DHS has developed additional guidance to increase the emphasis of test and evaluation earlier in the acquisition life cycle, which can help the department de-risk acquisition programs. Specifically, DHS moved the research and development driven technology development activity earlier in the acquisition lifecycle. Specific to the BioWatch program and DHS's efforts to improve its biodetection capabilities, DHS has taken several steps that address the intent of the recommendation. In addition to the updated guidance, DHS has cancelled one enhancement effort for the BioWatch program and is re-assessing its most current acquisition program given the state of the technology to meet mission needs. In 2018, DHS cancelled the Biodetection Technology Enhancements (BTE) program to enhance BioWatch because the department said BTE was neither technically feasible nor affordable. The department based this decision on the results of an alternatives analysis and a technology demonstration. The demonstration included an end-to-end test event to assess how all components worked together-from aerosol collection at the start to positive assay identification at the end-using simulants. In May 2021, we reported that the BD21 program was following DHS's acquisition guidance. In November 2021, DHS communicated that the BD21 program was pushing back decision points for the acquisition to allow more time for program acquisition strategy and detailed planning. DHS reported in January 2022 that the BD21 program would conduct an integrated system concept demonstration event to assess prototype system performance in an "operationally relevant environment," after which it will conduct additional technology readiness assessments to help in its decision-making. We believe these changes in DHS's acquisition policy along with the concrete examples of executing more realistic testing early in the acquisition lifecycle for two efforts intended to enhance or replace the BioWatch program align with the intent of our recommendation, but additional oversight may be needed to ensure sustainment.
|