Anthrax Detection:

Agencies Need to Validate Sampling Activities in Order to Increase Confidence in Negative Results

GAO-05-251: Published: Mar 31, 2005. Publicly Released: Apr 5, 2005.

Additional Materials:

Contact:

Nabajyoti Barkakati
(202) 512-6412
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

In September and October 2001, letters laced with Bacillus anthracis (anthrax) spores were sent through the mail to two U.S. senators and to members of the media. These letters led to the first U.S. cases of anthrax disease related to bioterrorism. In all, 22 individuals, in four states and Washington, D.C., contracted anthrax disease; 5 died. These cases prompted Congress to ask GAO to describe and assess federal agencies' activities to detect anthrax in postal facilities, assess the results of agencies' testing, and assess whether agencies' detection activities were validated.

The U.S. Postal Service, Centers for Disease Control and Prevention, and Environmental Protection Agency (EPA) conducted several interdependent activities, including sample collection and analytic methods, to detect anthrax in postal facilities in 2001. They developed a sampling strategy and collected, transported, extracted, and analyzed samples. They primarily collected samples from specific areas, such as mail processing areas, using their judgment about where anthrax would most likely be found--that is, targeted sampling. The agencies did not use probability sampling in their initial sampling strategy. Probability sampling would have allowed agencies to determine, with some defined level of confidence, when all results are negative, whether a building is contaminated. The results of the agencies' testing in 286 postal facilities were largely negative--no anthrax was detected. However, agencies did not use validated sample collection and analytical methods. According to the agencies, validated methods were not available in 2001. Thus, there can be little confidence in negative results. Validation is a formal, empirical process in which an authority determines and certifies the performance characteristics of a given method. Consequently, the lack of validation of agencies' activities, coupled with limitations associated with their targeted sampling strategy, means that negative results may not be reliable. In preparing for future incidents, the agencies have (1) made some changes based on what has been learned about some of the limitations of their sampling strategies, (2) made some revisions to their guidelines, (3) funded some new research, and (4) planned or conducted conferences addressing some of the issues GAO has identified. In addition, the Department of Homeland Security (DHS) has taken on the role of coordinating agencies' activities and has undertaken several new initiatives related to dealing with anthrax and other bio-threat agents. However, while the actions DHS and other agencies have taken are important, they do not address the issue of validating all activities related to sampling. Finally, the agencies have not made appropriate and prioritized investments to develop and validate all activities related to other bio-threat agents.

Recommendations for Executive Action

  1. Status: Closed - Not Implemented

    Comments: Prioritized investments for studies involving other biothreat agents have not yet begun. According to the 2007 strategic plan, as work on anthrax (B. anthracis) is completed, extension of the framework will be applied to other families of threats. In light of the fact that DHS is still working on Bacillus anthracis, it is not clear when it will be in a position to address other agents. Thus this recommendation has not been implemented.

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that appropriate, prioritized investments are made for all biothreat agents.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Not Implemented

    Comments: According to DHS, the 2007 interagency strategic plan incorporated two key activities toward this recommendation: the examination of tools to build actionable sampling plans that encompass probability-based sampling strategies and the development of a sampling strategy document. The DHS-led VSP workgroup defined a sampling strategy as, a set of operating precepts and diagnostic tools (including sample collection methods; packaging and shipping protocols; recovery, extraction, and analytic methods, and statistical analysis packages) that are combined to confidently answer specific hypotheses. Although the DHS-led workgroup has conducted studies to develop probability based sampling strategies that will provide confidence statements when test results are negative these studies did not always meet their objectives in this area. DHS had planned to conduct external validation of sampling software but funds are not currently available. Further, some key workgroup agencies have stated that it is unlikely that they will use such sampling strategies. The sampling strategy document is not yet completed. Thus, while the DHS workgroup has conducted work to address this issue, it still is not resolved. Therefore we consider this recommendation not yet implemented. Nevertheless, DHS has indicated that it will continue to address issues raised by the agencies and continues to believe in the development of quality data characterizing a potentially contaminated space.

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should see that appropriate investments are made in empirical studies to develop probability-based sampling strategies that take into account the complexities of indoor environments.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Not Implemented

    Comments: While the DHS-led workgroup--the Validated Sampling Plan Working Group--has made progress since about 2007 in validating sampling methods for detecting Bacillus anthracis in indoor environments, validation is not yet complete. Validation activities were projected to be completed at the end of fiscal year 2013 but delays are now expected. Several challenges must be addressed before validation can be completed, such as clarifying the scope of the strategic plan, completing studies on the false negative rates for the methods for which CDC/LRN conducted validation studies, establishing realistic estimates of the time needed to complete validation, and ensuring the availability of funds. Further, according to DHS, it does not have specific oversight authority to guarantee that validation of the overall sampling process (by workgroup agencies) will be achieved. Thus we consider this recommendation not yet implemented. DHS has indicated that it plans to pursue completion but will need the continued support of the workgroup member agencies to achieve its goals.

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should guarantee that the overall process of sampling activities, including methods, is validated so that performance characteristics, including limitations, are clearly understood and results can be correctly interpreted.

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: According to DHS, a definition of validation for the purposes of anthrax (B, anthracis) sampling has been agreed upon. The VSP Work group, organized by DHS in 2006, agreed upon a definition of validation in response to GAO's recommendation. The workgroup adopted the International Standard ISO/IEC 17025 definition of validation (section 5.4 - Test and calibration methods and methods validation), which is a standard used by testing and calibration laboratories. Validation of methods is defined as "the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled." In addition, the standard states that the laboratory shall validate non-standard methods, laboratory-designed/developed methods, standard methods used outside their intended scope, and amplifications and modifications of standard methods to confirm that the methods are fit for the intended use. The validation shall be as extensive as is necessary to meet the needs of the given application or field of application. Section 5.4 of ISO/IEC 17025 also addresses other aspects of the process, such as where and how to take a sample, sample transportation and laboratory analysis of the sample and states that validation may include procedures for sampling, handling, and transportation. As a result, GAO considers that the adoption of the ISO/IEC 17025 definition of validation has addressed our recommendation.

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that a definition of validation is developed and agreed on.

    Agency Affected: Department of Homeland Security

  5. Status: Closed - Implemented

    Comments: Since we issued our report (GAO-05-251) in 2005, DHS has taken a lead role in promoting and coordinating the activities of the agencies with technical expertise related to environmental testing. DHS's Science & Technology (S&T) Directorate chairs and is the coordinator of the Validated Sampling Plan (VSP) workgroup, whose participants include CDC, EPA, DOD, NIST, the Pacific Northwest National Laboratory and Sandia National Laboratory, with occasional participation by the FBI. According to DHS, beginning in 2006, the VSP workgroup began meeting approximately every week. Its purpose is to develop a common understanding of the relevant terms and processes, outline study scope and guide deliberations, and to assemble relevant program needs to inform the development of a strategic plan. On March 27, 2007, prior to our March 29, 2007, testimony (GAO-07-687T), working with CDC, EPA, DOD, and the FBI, DHS developed a draft strategic plan and road map. The plan, entitled the Interagency Strategic Plan for Validation of Environmental Sampling Methods Used in Detection and Cleanup of B. anthracis Contamination in Facilities, was provided to us by DHS in July 2007. The plan outlined how individual agencies' efforts would lead to the validation of the overall anthrax sampling process. According to the plan, analysis of environmental samples in the event of the release of anthrax spores will be conducted by the CDC Laboratory Response Network (LRN), DOD, and limited private laboratories, and potentially EPA's Environmental Laboratory Response Network laboratories. Further, as work on anthrax (B. anthracis) is completed, extension of the framework will be applied to other families of threats. GAO considers that the steps that DHS has taken have addressed our recommendation.

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should take a lead role in promoting and coordinating the activities of the various agencies that have the technical expertise related to environmental testing.

    Agency Affected: Department of Homeland Security

  6. Status: Closed - Not Implemented

    Comments: Although some agency guidelines, such as CDC?s, have been revised to include new information on the sampling methods, they cannot be fully revised to reflect such efforts since the validation project is not complete. In addition, VSP workgroup guidelines are still being revised and it is not clear when these will be published

    Recommendation: Given the lack of validated methods for detecting anthrax contamination in facilities, the Secretary of Homeland Security should develop a coordinated approach to (1) improve the overall process for detecting anthrax and (2) increase confidence in negative test results generated by that process. This approach would include working with agencies to ensure that appropriate validation studies of the overall process of sampling activities, including the methods, are conducted. Specifically, the Secretary should ensure that agency policies, procedures, and guidelines reflect the results of such efforts.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Dec 19, 2014

Dec 17, 2014

Nov 6, 2014

Oct 14, 2014

Sep 30, 2014

Sep 24, 2014

Sep 18, 2014

Looking for more? Browse all our products here