Anthrax Detection: Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Results
GAO-05-251
Published: Mar 31, 2005. Publicly Released: Apr 05, 2005.
Skip to Highlights
Highlights
In September and October 2001, letters laced with Bacillus anthracis (anthrax) spores were sent through the mail to two U.S. senators and to members of the media. These letters led to the first U.S. cases of anthrax disease related to bioterrorism. In all, 22 individuals, in four states and Washington, D.C., contracted anthrax disease; 5 died. These cases prompted Congress to ask GAO to describe and assess federal agencies' activities to detect anthrax in postal facilities, assess the results of agencies' testing, and assess whether agencies' detection activities were validated.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should take a lead role in promoting and coordinating the activities of the various agencies that have the technical expertise related to environmental testing. |
Since we issued our report (GAO-05-251) in 2005, DHS has taken a lead role in promoting and coordinating the activities of the agencies with technical expertise related to environmental testing. DHS's Science & Technology (S&T) Directorate chairs and is the coordinator of the Validated Sampling Plan (VSP) workgroup, whose participants include CDC, EPA, DOD, NIST, the Pacific Northwest National Laboratory and Sandia National Laboratory, with occasional participation by the FBI. According to DHS, beginning in 2006, the VSP workgroup began meeting approximately every week. Its purpose is to develop a common understanding of the relevant terms and processes, outline study scope and guide deliberations, and to assemble relevant program needs to inform the development of a strategic plan. On March 27, 2007, prior to our March 29, 2007, testimony (GAO-07-687T), working with CDC, EPA, DOD, and the FBI, DHS developed a draft strategic plan and road map. The plan, entitled the Interagency Strategic Plan for Validation of Environmental Sampling Methods Used in Detection and Cleanup of B. anthracis Contamination in Facilities, was provided to us by DHS in July 2007. The plan outlined how individual agencies' efforts would lead to the validation of the overall anthrax sampling process. According to the plan, analysis of environmental samples in the event of the release of anthrax spores will be conducted by the CDC Laboratory Response Network (LRN), DOD, and limited private laboratories, and potentially EPA's Environmental Laboratory Response Network laboratories. Further, as work on anthrax (B. anthracis) is completed, extension of the framework will be applied to other families of threats. GAO considers that the steps that DHS has taken have addressed our recommendation.
|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that a definition of validation is developed and agreed on. |
According to DHS, a definition of validation for the purposes of anthrax (B, anthracis) sampling has been agreed upon. The VSP Work group, organized by DHS in 2006, agreed upon a definition of validation in response to GAO's recommendation. The workgroup adopted the International Standard ISO/IEC 17025 definition of validation (section 5.4 - Test and calibration methods and methods validation), which is a standard used by testing and calibration laboratories. Validation of methods is defined as "the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled." In addition, the standard states that the laboratory shall validate non-standard methods, laboratory-designed/developed methods, standard methods used outside their intended scope, and amplifications and modifications of standard methods to confirm that the methods are fit for the intended use. The validation shall be as extensive as is necessary to meet the needs of the given application or field of application. Section 5.4 of ISO/IEC 17025 also addresses other aspects of the process, such as where and how to take a sample, sample transportation and laboratory analysis of the sample and states that validation may include procedures for sampling, handling, and transportation. As a result, GAO considers that the adoption of the ISO/IEC 17025 definition of validation has addressed our recommendation.
|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should guarantee that the overall process of sampling activities, including methods, is validated so that performance characteristics, including limitations, are clearly understood and results can be correctly interpreted. |
While the DHS-led workgroup--the Validated Sampling Plan Working Group--has made progress since about 2007 in validating sampling methods for detecting Bacillus anthracis in indoor environments, validation is not yet complete. Validation activities were projected to be completed at the end of fiscal year 2013 but delays are now expected. Several challenges must be addressed before validation can be completed, such as clarifying the scope of the strategic plan, completing studies on the false negative rates for the methods for which CDC/LRN conducted validation studies, establishing realistic estimates of the time needed to complete validation, and ensuring the availability of funds. Further, according to DHS, it does not have specific oversight authority to guarantee that validation of the overall sampling process (by workgroup agencies) will be achieved. Thus we consider this recommendation not yet implemented. DHS has indicated that it plans to pursue completion but will need the continued support of the workgroup member agencies to achieve its goals.
|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should see that appropriate investments are made in empirical studies to develop probability-based sampling strategies that take into account the complexities of indoor environments. |
According to DHS, the 2007 interagency strategic plan incorporated two key activities toward this recommendation: the examination of tools to build actionable sampling plans that encompass probability-based sampling strategies and the development of a sampling strategy document. The DHS-led VSP workgroup defined a sampling strategy as, a set of operating precepts and diagnostic tools (including sample collection methods; packaging and shipping protocols; recovery, extraction, and analytic methods, and statistical analysis packages) that are combined to confidently answer specific hypotheses. Although the DHS-led workgroup has conducted studies to develop probability based sampling strategies that will provide confidence statements when test results are negative these studies did not always meet their objectives in this area. DHS had planned to conduct external validation of sampling software but funds are not currently available. Further, some key workgroup agencies have stated that it is unlikely that they will use such sampling strategies. The sampling strategy document is not yet completed. Thus, while the DHS workgroup has conducted work to address this issue, it still is not resolved. Therefore we consider this recommendation not yet implemented. Nevertheless, DHS has indicated that it will continue to address issues raised by the agencies and continues to believe in the development of quality data characterizing a potentially contaminated space.
|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that appropriate, prioritized investments are made for all biothreat agents. |
Prioritized investments for studies involving other biothreat agents have not yet begun. According to the 2007 strategic plan, as work on anthrax (B. anthracis) is completed, extension of the framework will be applied to other families of threats. In light of the fact that DHS is still working on Bacillus anthracis, it is not clear when it will be in a position to address other agents. Thus this recommendation has not been implemented.
|
| Department of Homeland Security | Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that agency policies, procedures, and guidelines reflect the results of such efforts. |
Although some agency guidelines, such as CDC?s, have been revised to include new information on the sampling methods, they cannot be fully revised to reflect such efforts since the validation project is not complete. In addition, VSP workgroup guidelines are still being revised and it is not clear when these will be published
|
Full Report
Topics
AnthraxBiological agentsBioterrorismDisease detection or diagnosisDiseasesHealth hazardsHomeland securityInfectious diseasesInteragency relationsLaboratoriesPostal facilitiesStrategic planningTerrorismTest facilitiesTestingEpidemicsDisease surveillance