Anthrax: DHS Faces Challenges in Validating Methods for Sample Collection and Analysis
Highlights
What GAO Found
A workgroup—led by the U.S. Department of Homeland Security (DHS) and made up of DHS and the Centers for Disease Control and Prevention (CDC), the Environmental Protection Agency (EPA), the Federal Bureau of Investigation (FBI), and the National Institute of Standards and Technology (NIST)—has attempted to address GAO’s recommendations to (1) validate environmental sampling methods for detecting Bacillus anthracis and (2) conduct studies to develop probability-based sampling approaches for indoor environments. This workgroup has taken some actions to validate environmental sampling methods (collection, transportation, preparation, analysis) and develop statistically based sampling approaches that will provide confidence statements when test results are negative. These activities were projected to be completed by fiscal year 2013, but delays are now expected.
While progress has been made in validating sampling methods for detecting Bacillus anthracis spores in indoor environments, their validation is not yet complete. Some studies have not begun. Although more is known about the methods’ performance characteristics—such as their limits of detection—other aspects of the methods are unknown, such as false negative rates. CDC has validated the preparation and analysis but not the collection methods for the swab and wipe. CDC states that field validation would be too difficult and laboratory validation of collection methods is not required. However, experts GAO talked to stated that collection methods could be validated in a laboratory.
Agencies that perform environmental sampling take the lead in validating the sampling methods. The FBI does not typically use CDC’s environmental sampling methods and validating its methods is outside the scope of the DHS-led workgroup. The FBI’s environmental sampling methods are not validated but the agency relies on DHS’s National Bioforensic Analysis Center (NBFAC) to validate its microbial forensic analytical methods. Thus, the FBI, through NBFAC, and CDC are attempting to validate analytical methods for Bacillus anthracis but neither is validating the collection methods. Nevertheless, improvements in sample collection procedures for the swab and wipe could be useful to the FBI in developing its sampling plans or in evaluating its sampling methods.
The workgroup must address several remaining challenges before the validation project can be completed: (1) clarifying the strategic plan’s scope—some agencies believe it is overly ambitious and differ on whether it includes linking sampling results to a risk-based decision process—and determining whether the workgroup is to continue; (2) reaching consensus on the range of sampling approaches that should be available to decision makers in different phases of a response; (3) establishing realistic estimates of the time for completing prioritized validation activities; (4) addressing scientific gaps, such as assessing risk in the absence of dose-response data; and (5) ensuring the availability of funds for critical tasks. While validating the methods provides information on performance characteristics, human health risks from any particular level of exposure remain uncertain. Since the workgroup has invested about $12 million and considerable resources over about 7 years, it would be prudent for it to complete prioritized tasks. Thus, the workgroup may wish to consider carefully what work is needed and think strategically in terms of its investments and their potential benefits.
Why GAO Did This Study
In 2005, assessing federal agencies’ activities for detecting Bacillus anthracis in postal facilities, GAO reported that the test results of their sampling were largely negative. GAO found that the agencies had not used validated sampling methods and approaches that would have given a defined level of confidence for negative results. Consequently, GAO recommended several actions. In this study, GAO was asked to identify the extent to which (1) DHS’s actions have addressed GAO’s recommendations regarding sampling, (2) the environmental sampling methods for B. anthracis spore detection in initial public health sampling and microbial forensic investigations have been validated, and (3) any challenges remain to completing validation. GAO analyzed agency documents and interviewed agency officials.
Recommendations
To ensure validated sampling methods and approaches are available for decision makers to respond to an indoor Bacillus anthracis release, DHS should (1) update the strategic plan and its roadmap with an agreed scope and timelines, and (2) complete the validation project. The Secretary of HHS and the Administrator of EPA should support DHSs goal of achieving validated sampling methods and a statistically based sampling approach. DHS agreed with our recommendations; EPA and HHS disagreed with our recommendation to them, stating that such an approach was not feasible or necessary. We continue to believe a validated statistical sampling approach will provide a broader range of options for decision makers responding to future incidents.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To ensure that federal agencies have validated sampling methods for detecting B. anthracis in indoor environments and--in the case of negative results--the option of using appropriate sampling approaches to make statistical confidence statements about the likelihood that a building is free of contamination when potentially there has been a low-level release, the Secretary of Homeland Security should take steps to complete the validation project. Statistically-based sampling designs for such purposes would encompass any sampling with a statistical basis, including a probabilistic only approach as well as one that combines judgmental and probabilistic sampling. Achieving a sufficiently rigorous validation of the sampling methods and ensuring that statistically rigorous and mutually acceptable sampling approaches are available will provide options that will better prepare decision makers to respond to a future bioterrorism incident. |
DHS has taken steps to complete the validation project in coordination with HHS and EPA. In August 2014, DHS indicated that it had funded work at the Pacific Northwest National Laboratory (PNNL) to develop algorithms that combine sampling information with other evidence related to building contamination. This information could be used to inform the development of confidence statements associated with such sampling. In addition, DHS is funding experiments at PNNL to develop guidance for the collection and analysis of composite samples on various types of surfaces. Both efforts are to allow confidence statements with the use of reduced numbers of samples in sampling designs. The PNNL's Visual Sample Plan software will incorporate related algorithms. Finally, a limited evaluation of the variability in wipe samples used by different individuals is to be conducted. According to DHS, tasks in the strategic plan's roadmap are to be completed by the spring or summer of 2015. This will also involve an external review of the output of the VSP workgroup by HSSAI in spring 2015.
|
| Department of Homeland Security | The Secretary of Homeland Security should update the strategic plan and its roadmap with an agreed-on scope and revised timelines. |
DHS has updated the strategic plan's roadmap with an agreed upon scope with CDC and EPA, with completion of all roadmap tasks projected by the summer of 2015. In July 2014, DHS stated that the scope of the VSP workgoup's focus was "the "cleanliness" of indoor environments that may have been contaminated with B. anthracis spores."
|
| Department of Homeland Security | The Secretary of Homeland Security should complete the validation project, including validating the collection methods in a laboratory setting in a manner that determines the potential sources of variation in collection method performance, including variation that could be introduced by individual samplers, and related ongoing studies. |
DHS has already taken a number of steps to complete the validation effort, including, in coordination with CDC and EPA, several studies evaluating and validating the sampling and analysis methods(swab, wipe, vacuum). Studies have also been undertaken to capture some of the variation in sample collection by different samplers using the wipe, the sampling method most likely to be used, according to DHS. Further, CDC and EPA are completing studies of sampling methods (e.g, vacuum) for porous surfaces and materials, which may not be feasonable. DHS projects completion of the project by the summer of 2015.
|
| Department of Health and Human Services | The Secretary of the Department of Health and Human Services and the Administrator of the Environmental Protection Agency should support DHS in its goal of achieving (1) validated sampling methods to understand the limitations of the data that would be provided to decision makers, and (2) a mutually acceptable statistically-based sampling approach that can be employed when decision makers--such as Incident Commanders and others--conclude that statistical confidence statements need to be made about the level of contamination in a particular indoor environment. |
In coordination with VSP Workgroup members DHS and EPA, HHS has assisted in updating the strategic plan's roadmap with the activities it will complete. According to DHS, all of the listed validation activities HHS is to conduct are funded, and are either under way or have been completed. These include of laboratory studies of the various sampling methods, including the swab, wipe, and vacuum methods for porous surfaces. In June 2014, HHS stated it would not be validating field sampling techniques in the manner that it validates a laboratory method but had standardized the collection procedures to reduce individual variation across collectors and surfaces and was completing studies of sampling methods for porous surfaces and materials. Regarding statistically based sampling approaches, CDC stated that it continues to use the current targeted sampling approach. However, CDC stated that should scenarios arise that warrant the use of a combined approach, that is, using targeted and statistical sampling, CDC has the ability to do so with the current VSP software. CDC stated that it had previously worked with the Pacific Northwest National Laboratory (PNNL) in its development of statistical sampling software that could be used in such a scenario.
|
| Environmental Protection Agency | The Secretary of the Department of Health and Human Services and the Administrator of the Environmental Protection Agency should support DHS in its goal of achieving (1) validated sampling methods to understand the limitations of the data that would be provided to decision makers, and (2) a mutually acceptable statistically-based sampling approach that can be employed when decision makers--such as Incident Commanders and others--conclude that statistical confidence statements need to be made about the level of contamination in a particular indoor environment. |
In coordination with VSP Workgroup members, DHS and HHS, EPA has assisted in updating the VSP Workgroup roadmap with the activities it will complete. All of the listed validation activities EPA is to conduct are funded, and are either under way or have been completed; these include completion of laboratory studies of some sampling methods, including the vacuum method in coordination with HHS. Also, EPA, in coordination with HHS, has provided input to DHS in its development of a sampling guidance document expected to be completed in mid-2015. Regarding statistically based sampling approaches, EPA stated that it does not agree that such approaches are needed. However, EPA also stated that should response leaders choose to implement a statistically based sampling approach, the already available PNNL-developed VSP software could be used.
|