This is the accessible text file for GAO report number GAO-09-100 
entitled 'Health Information Technology: More Detailed Plans Needed for 
the Centers for Disease Control and Prevention's Redesigned BioSense 
Program' which was released on November 20, 2008.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 
GAO: 

November 2008: 

Health Information Technology: 

More Detailed Plans Needed for the Centers for Disease Control and 
Prevention's Redesigned BioSense Program: 

GAO-09-100: 

GAO Highlights: 

Highlights of GAO-09-100, a report to congressional committees. 

Why GAO Did This Study: 

In 2003, the Centers for Disease Control and Prevention (CDC), an 
agency within the Department of Health and Human Services (HHS), 
developed an electronic syndromic surveillance system called BioSense 
that uses health-related data to identify patterns of disease symptoms 
prior to specific diagnoses. In late 2007, CDC began to redesign the 
program to improve collaboration with stakeholders and address 
identified management weaknesses. Pursuant to House Report 110-231, GAO 
evaluated the BioSense program, focusing on the cost and timeline 
estimates and performance measures and benchmarks for implementing the 
program, among other objectives. To accomplish this, GAO analyzed 
relevant program documentation and interviewed CDC officials 
responsible for developing and implementing BioSense. 

What GAO Found: 

While CDC identified annual and long-term cost and timeline estimates 
and performance measures for the initial design of BioSense, these 
estimates and measures did not reflect the implementation of its 
redesigned program. CDC subsequently developed a draft plan for the 
redesigned program that described high-level cost and timeline 
estimates; however, the estimates are not reliable, and the plan did 
not include performance measures.
 
* According to best practices, cost estimates should be well-
documented, comprehensive and accurate, and must be credible before 
they can be considered to be reliable. However, CDC’s cost estimates 
for the redesigned program are not reliable because they are only 
partially documented, are not comprehensive and accurate, and therefore 
are not credible.
 
* Best practices for reliable timeline estimates include the 
identification of resources to complete each task, establishment of a 
critical path, and analysis of risks to the schedule. However, the 
agency has not implemented these practices, resulting in timelines for 
the redesigned program that are not reliable. 

* The Office of Management and Budget directs agencies to define 
outcome-based performance measures to gauge program results early 
enough for stakeholder review, and industry experts describe the need 
for stakeholder input in developing performance measures in order to 
monitor performance. While CDC established performance measures and 
benchmarks for the initial implementation of the BioSense program, it 
has not yet developed outcome-based performance measures to monitor the 
progress of the redesigned program and does not intend to complete 
their development until the end of 2009. 

Until program officials develop reliable cost and timeline estimates 
and outcome-based performance measures for the redesigned BioSense 
program, they will lack key components needed to effectively manage the 
program, increasing the risk that the agency will perpetuate weaknesses 
identified in its initial implementation of the program and related 
system. 

What GAO Recommends: 

GAO is recommending that CDC develop reliable cost and timeline 
estimates and outcome-based performance measures for implementing the 
redesigned BioSense program. In written comments on a draft of this 
report, HHS stated it welcomed the conclusions and recommendations and 
provided updated information about current efforts intended to address 
the recommendations. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-100]. For more 
information, contact Valerie C. Melvin at (202) 512-6304 or 
melvinv@gao.gov. 

[End of section] 

Contents: 

Letter: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Briefing Slides: 

Appendix II: Comments from the Department of Health and Human Services: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Abbreviations: 

CDC: Centers for Disease Control and Prevention: 

DHS: Department of Homeland Security: 

DOD: Department of Defense: 

ESSENCE: Electronic Surveillance System for the Early Notification of 
Community-based Epidemics: 

HHS: Department of Health and Human Services: 

NBIS: National Biosurveillance Integration System: 

NHIN: Nationwide Health Information Network: 

OMB: Office of Management and Budget: 

RODS: Real-time Outbreak and Disease Surveillance: 

VA: Department of Veterans Affairs: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

November 20, 2008: 

The Honorable Tom Harkin: 
Chairman: 
The Honorable Arlen Specter: 
Ranking Member: 
Subcommittee on Labor, Health and Human Services, Education, and 
Related Agencies: 
Committee on Appropriations: 
United States Senate: 

The Honorable Dave Obey: 
Chairman: 
The Honorable James T. Walsh: 
Ranking Member: 
Subcommittee on Labor, Health and Human Services, Education, and 
Related Agencies: 
Committee on Appropriations: 
House of Representatives: 

In 2002, President Bush signed the Public Health Security and 
Bioterrorism Preparedness and Response Act, which required specific 
activities aimed at improving the nation's preparedness for 
bioterrorism and other public health emergencies.[Footnote 1] In 
response to the passage of this act, the Centers for Disease Control 
and Prevention (CDC), an agency within the Department of Health and 
Human Services (HHS), developed, in 2003, an electronic syndromic 
surveillance information system called BioSense.[Footnote 2] In 2004, 
CDC established the BioSense program office to improve the nation's 
capabilities for conducting syndromic surveillance by providing 
federal, state, and local public health officials access to existing 
data from health care organizations across the country through use of 
the BioSense system. In late 2007, the agency initiated efforts to 
redesign the program to improve collaboration with public health 
stakeholders and to address management weaknesses identified in its 
initial implementation. CDC's National Center for Public Health 
Informatics is responsible for managing the program. 

Pursuant to House Report 110-231,[Footnote 3] we evaluated BioSense, 
focusing on (1) the costs and benefits of operating the BioSense system 
as compared to other state and local surveillance systems, (2) the 
usefulness of data provided by the system to hospitals and state and 
local public health officials, and (3) the cost and timeline estimates 
and performance measures and benchmarks for implementing the program. 
We provided subcommittee staff with preliminary results from our study 
on April 1, 2008, and subsequently agreed with your offices to continue 
our evaluation of CDC's plans for implementing BioSense, focusing on 
the annual and long-term cost and timeline estimates and performance 
indicators and benchmarks for implementing a redesigned program. On 
August 26, 2008, we provided your offices with briefing slides that 
outlined the final results of our study. The purpose of this report is 
to issue the published briefing slides to you and to officially 
transmit our recommendations to the Director of the Centers for Disease 
Control and Prevention. The slides, which discuss our scope and 
methodology and incorporate edits made since we initially provided the 
briefing, are included in appendix I. 

We conducted this performance audit from October 2007 to August 2008 at 
CDC's headquarters in Atlanta, Georgia, in accordance with generally 
accepted government auditing standards. Those standards require that we 
plan and perform the audit to obtain sufficient, appropriate evidence 
to provide a reasonable basis for our findings and conclusions based on 
our audit objectives. We believe that the evidence obtained provides a 
reasonable basis for our findings and conclusions based on our audit 
objectives. 

In summary, our study highlighted the following: 

* The costs and benefits of using data from the current BioSense system 
and other similar syndromic surveillance systems are difficult to 
measure and compare. Further, public health and hospital officials with 
whom we spoke said that because of the difficulty of doing so, they do 
not track the costs and benefits of using syndromic surveillance 
systems. 

* Federal, state, and local public health officials expressed mixed 
views on the usefulness of BioSense data as compared with the 
usefulness of data from similar systems. While stating that they find 
the data from the BioSense system useful to varying degrees, about half 
of the officials with whom we spoke said that they rely on other 
systems as their primary tool for syndromic surveillance. For example, 
officials with the Department of Defense (DOD) stated that, rather than 
BioSense, the department's public health personnel use DOD's Electronic 
Surveillance System for the Early Notification of Community-based 
Epidemics (ESSENCE). In contrast, Department of Veterans Affairs (VA) 
public health personnel do not regularly rely on syndromic surveillance 
system data; however, they use the civilian version of ESSENCE more 
often than BioSense. Both DOD and VA officials stated that ESSENCE 
allows more flexibility in evaluating and viewing data than BioSense. 
[Footnote 4] 

* Although CDC identified annual and long-term cost and timeline 
estimates and established performance measures and benchmarks for the 
initial implementation of the BioSense program and its related system, 
these estimates and measures did not reflect the implementation of its 
redesigned program. CDC subsequently developed a draft plan for the 
redesigned program that described high-level cost and timeline 
estimates; however, these estimates were not reliable, and the plan did 
not include performance measures. 

- According to best practices, cost estimates should be well- 
documented, comprehensive and accurate, and must be credible before 
they can be considered to be reliable. However, CDC's cost estimates 
for the redesigned program are not reliable because they are only 
partially documented, are not comprehensive and accurate, and therefore 
are not credible. 

- Best practices for reliable timeline estimates include the 
identification of resources to complete each task, establishment of a 
critical path, and analysis of risks to the schedule. However, the 
agency has not implemented these practices, resulting in timelines for 
the redesigned program that are not reliable. 

- The Office of Management and Budget directs agencies to define 
meaningful annual and long-term outcome-based performance measures to 
gauge the results of a program or activity early enough to allow time 
for stakeholder review, and industry experts describe the need for 
stakeholder input in developing performance measures in order to 
monitor project performance. CDC has not yet developed outcome-based 
performance measures to monitor the progress of the redesigned program 
and does not intend to complete their development until the end of 
2009. 

Until program officials develop reliable cost and timeline estimates 
and outcome-based performance measures for the redesigned BioSense 
program, they will lack key components needed to effectively manage the 
program. 

Conclusions: 

The costs and benefits of using data from the current implementation of 
the BioSense system and other similar systems are difficult to measure 
and compare, and public health and hospital officials with whom we 
spoke do not document or track the costs and benefits of using these 
systems. These officials also expressed mixed views on the usefulness 
of BioSense data as compared to data from other systems. 

While BioSense program officials have developed cost and timeline 
estimates for the initial implementation of the program, they have not 
yet reliably estimated costs and timelines for the redesigned program. 
Additionally, program officials do not expect to complete the 
development of outcome-based performance measures for the redesigned 
BioSense program before the end of 2009. Until CDC develops these key 
components of effective information technology program management, 
BioSense officials will lack the management and planning tools needed 
to gauge the success of the redesigned BioSense program in improving 
federal, state, and local partners' abilities to respond to public 
health events and will risk perpetuating weaknesses identified in the 
initial implementation of the program and its related system. 

Recommendations for Executive Action: 

To ensure that CDC defines reliable plans for effectively managing the 
development and implementation of the redesigned BioSense program and 
its related system, we are recommending that the Director of CDC 
instruct the Director of the National Center for Public Health 
Informatics to take the following three actions while in the planning 
phase of the redesigned BioSense program: 

* develop reliable cost estimates for the program that are well- 
documented, comprehensive and accurate, and credible; 

* develop reliable timeline estimates for implementing the program; 
and: 

* with stakeholder input, develop outcome-based performance measures 
that address all phases of the program and that focus the success of 
the program on CDC's federal, state, and local partners' use of the 
system for responding to public health events. 

Agency Comments and Our Evaluation: 

The Department of Health and Human Services' Assistant Secretary for 
Legislation provided written comments on a draft of this report. In the 
comments, HHS welcomed the conclusions and recommendations discussed in 
our report. 

In its comments, HHS provided information about steps the department 
has begun taking to address the conclusions and recommendations in our 
report. In particular, the department stated that it has launched a 
planning and implementation process to ensure that the program develops 
reliable cost estimates and timelines and identifies appropriate 
outcome-based performance measures. According to the department, among 
other activities, the program has initiated working groups of internal 
and external stakeholders to establish detailed tactical plans for 
projects, tasks, deliverables, and resources. The department stated 
that it intends to use the tactical plans to derive future cost and 
timeline estimates and involve stakeholders in establishing outcome- 
based performance measures. It added that a program plan for the 
redesigned Biosense is scheduled for completion by February 2009. If 
effectively implemented, HHS's stated actions toward developing 
reliable cost and timeline estimates and defining outcome-based 
performance measures should help improve the department's management of 
the redesigned BioSense program. 

Finally, with regard to the program, the department stressed the 
importance of noting that BioSense is not intended to replace local 
systems, but rather to add value by providing data, tools, and funding 
intended to improve surveillance efforts. HHS's written comments are 
reproduced in appendix II. 

DOD and VA also reviewed the draft report and provided technical 
comments, which we have incorporated as appropriate. 

We are sending copies of this report to interested congressional 
committees, the Director of the Office of Management and Budget, and 
the Secretaries of the Departments of Health and Human Services, 
Defense, and Veterans Affairs. In addition, the report will be 
available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

Should you or your staffs have any questions concerning this report, 
please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. Key contributors to this report 
are listed in appendix III. 

Signed by: 

Valerie C. Melvin: 
Director, Human Capital and Management Information Systems Issues: 

[End of section] 

Appendix I: Briefing Slides: 

Health Information Technology: More Detailed Plans Needed for The 
Centers for Disease Control and Prevention’s Redesigned BioSense 
Program: 

Briefing for Staff Members of the House and Senate Subcommittees on 
Labor, Health and Human Services, Education, and Related Agencies, 
Committees on Appropriations: 

August 26, 2008: 

Overview: 

Introduction: 

Objectives: 

Scope and Methodology: 

Results in Brief: 

Background: 

Results: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

[End of Overview] 

Introduction: 

A catastrophic health event could cause hundreds of thousands of 
casualties, weaken our economy, damage public morale and confidence, 
and threaten our national security. Threats of bioterrorism and high-
profile disease outbreaks, such as a pandemic influenza outbreak, have 
drawn attention to the need for surveillance systems for the early 
detection of and response to public health emergencies. 

In 2002, President Bush signed the Public Health Security and 
Bioterrorism Preparedness and Response Act, which required specific 
activities intended to improve the nation’s preparedness for 
bioterrorism and other public health emergencies. 

* The act required the Secretary of Health and Human Services to 
establish an integrated system of public health alert communication and 
surveillance networks among federal, state, and local public health 
officials, and laboratories, hospitals, and other health care 
facilities. 

In response to the passage of this act, in 2003 the Centers for Disease 
Control and Prevention (CDC) developed an electronic syndromic 
surveillance information system called BioSense. 

In 2004, the agency established the BioSense program office to improve 
the nation’s capabilities for conducting surveillance by providing 
federal, state, and local public health officials access to existing 
data from health care organizations across the country through the use 
of the BioSense system. In late 2007, CDC initiated efforts to redesign 
the program in order to improve collaboration with federal, state, and 
local public health stakeholders and to address management weaknesses 
identified in the initial implementation of BioSense.The program is 
managed by CDC’s National Center for Public Health Informatics. 

According to CDC, syndromic surveillance is a technique that uses 
health-related data to identify patterns of disease symptoms prior to 
specific disease diagnoses. Effective use of this technique can provide 
both situational awareness and early event detection. 

* Situational awareness is the knowledge of the size, location, and 
rate of spread of an outbreak. 

* Early event detection provides signals that a sufficient probability 
of a disease outbreak exists to warrant public health response and lead 
to the early identification of illnesses and other events of public 
health importance. 

In this regard, syndromic surveillance can facilitate preventing and 
controlling the spread of diseases. 

Pursuant to House Report 110-231,[Footnote 5] we evaluated BioSense, 
focusing on the costs and benefits of operating the BioSense system as 
compared to other state and local surveillance systems; the usefulness 
of data provided by the system to hospitals and state and local public 
health officials; and cost and timeline estimates and performance 
measures and benchmarks for implementing the program. As agreed with 
staff of the House Committee on Appropriations, Subcommittee on Labor, 
HHS, Education, and Related Agencies, we provided preliminary results 
from our study on April 1, 2008. Subsequently, we agreed with staff of 
the House and Senate Committees on Appropriations, Subcommittees on 
Labor,HHS, Education, and Related Agencies to continue our evaluation 
of CDC’s plans for implementing BioSense, focusing on the annual and 
long-term cost and timeline estimates, and performance indicators and 
benchmarks for implementing a redesigned program. 

The specific objectives addressed in our study were to: 

* Compare the cost and benefits of operating the BioSense system with 
the cost and benefits of operating state, local, and private 
surveillance systems. 

* Identify the views of hospitals and federal, state, and local public 
health officials on the usefulness of data provided by the BioSense 
system versus data provided by local surveillance systems for 
responding to emergencies. 

* Identify annual and long-term cost and timeline estimates and 
performance indicators and benchmarks for the initial implementation of 
BioSense, and evaluate CDC’s plans for implementing the redesigned 
BioSense program, focusing on annual and long-term cost and timeline 
estimates, and performance indicators and benchmarks. 

[End of Introduction] 

Scope and Methodology: 

To compare the cost and benefits of operating the BioSense system with 
the cost and benefits of operating other similar state, local, and 
private surveillance systems, we: 

* identified from CDC’s fiscal year 2009 budget and planning 
documentation the agency’s expected cost and projected benefits of 
operating the BioSense system; 

* reviewed cost and benefits information related to the use of the 
BioSense system and available information on cost and benefits reported 
for other surveillance systems used by hospitals and state and local 
public health organizations; and; 

* discussed with the BioSense users selected for our second objective 
the cost of and benefits realized from using data provided by the 
BioSense system. 

To determine the views of federal, state, and local public health 
officials and hospitals on the usefulness of BioSense data versus local 
surveillance data, we selected and interviewed officials of 
organizations that conduct syndromic surveillance using BioSense, other 
systems, or BioSense together with other systems. These organizations 
include: 

* the three federal agencies that have access to the BioSense 
system—CDC, the Department of Defense (DOD), and the Department of 
Veterans Affairs (VA); and; 

* three state and five local public health departments and two 
hospitals: 

- state public health departments in Arizona, Georgia, and North 
Carolina; 
- local public health departments in Dallas, El Paso, and Tarrant 
counties in Texas; Maricopa County in Arizona; and the East Metro 
Health District in Georgia; and; 
- Gwinnett Medical Center in Lawrenceville, Georgia, and a Tenet 
Healthcare hospital in El Paso, Texas. 

To select these organizations, we obtained from CDC the program 
office’s list of BioSense users from the most recent month at the time 
of our request—October 2007. 

* According to CDC, there are more than 500 users of the BioSense 
system, and they are located throughout 150 hospital, federal, state, 
and local facilities, including CDC. 

We selected organizations from different geographic areas that use 
BioSense and other syndromic surveillance systems. 

* We sorted users by state to identify a subset of organizations from a 
wide geographic dispersion. 

* From that subset, we selected states with varying numbers of BioSense 
users. 

* From those states, we selected a final set of states with high, 
midrange, and low numbers of BioSense users, based on the average 
number of users across the selected states. 

We then selected local health departments in municipalities with 
populations that varied from large to small. 

* From among the BioSense users we selected, we further identified 
organizations that also use BioSense and other commercial or locally-
developed systems based on literature research and our knowledge of the 
syndromic surveillance system user population. 

* We also included in our selection BioSense facilities that have not 
opted to use the data provided by the system. 

* We discussed with the selected users the usefulness of BioSense data 
as compared with data from other similar systems. 

To supplement our discussions with selected users, we met with 
officials from two public health associations to obtain their views on 
the usefulness of BioSense and of syndromic surveillance systems in 
general. We held discussions with officials from: 

* the Association for State and Territorial Health Officials, and; 

* the Council for State and Territorial Epidemiologists. 

We also interviewed public health officials from four other state and 
local jurisdictions at a meeting arranged for us by the National 
Association of City and County Health Officials. 

In addition, we asked the federal, state, local, and hospital 
representatives with whom we spoke to identify any lessons they had 
learned from developing and implementing other similar surveillance 
systems. 

Our selection of BioSense users does not provide a statistically-valid 
sample; therefore, the information provided by the organizations we 
selected cannot be generalized across the entire BioSense user 
population. 

To identify annual and long-term cost and timeline estimates and 
performance indicators and benchmarks for implementing the initial 
design of BioSense, we reviewed and assessed CDC’s relevant program 
management and budget justification documentation for the program. 

To evaluate CDC’s plans for implementing the redesigned BioSense 
program and related system, we obtained CDC’s draft plan for and other 
documentation relevant to implementing the redesigned BioSense program. 
We evaluated the costs, schedule, and performance components addressed 
in this documentation against criteria established by federal guidance 
to determine if CDC followed best practices to develop these components 
of its plan.[Footnote 6] 

* We analyzed CDC’s documented cost estimates for implementing there 
designed program, as well as documentation describing the practices 
that cost estimators followed to estimate costs of the program. 

* We assessed CDC’s estimated time frames for completing the strategic 
activities described in the plan. 

* We reviewed CDC’s documented approach for developing performance 
measures described in the plan and compared it to the Office of 
Management and Budget’s (OMB) guidance and industry practices to 
determine if the agency’s documented approach was consistent with 
federal and industry guidance.[Footnote 7] 

We supplemented our documentation reviews with interviews of officials 
from CDC’s BioSense program office to obtain additional information 
about the development of current and future plans for BioSense. We did 
not independently verify CDC’s cost or timeline estimates for the 
program. 

We conducted this performance audit from October 2007 to August 2008 at 
CDC’s headquarters in Atlanta, Georgia, in accordance with generally 
accepted government auditing standards. Those standards require that we 
plan and perform the audit to obtain sufficient, appropriate evidence 
to provide a reasonable basis for our findings and conclusions based on 
our audit objectives. We believe that the evidence obtained provides a 
reasonable basis for our findings and conclusions based on our audit 
objectives. 

[End of section] 

Results in Brief: 

The cost and benefits of using data from the current BioSense system 
and other similar syndromic surveillance systems are difficult to 
measure and compare. The state and local public health and hospital 
officials with whom we spoke stated that documenting the cost and 
benefits of using syndromic surveillance systems such as BioSense is 
difficult and that they do not develop or track cost or benefits of 
using these systems. 

Federal, state, and local public health and hospital officials 
expressed mixed views on the usefulness of BioSense data as compared 
with the usefulness of other similar systems’ data. While they stated 
that they find the data from the BioSense system useful in varying 
degrees, about half of them stated that they rely on other systems as 
their primary tool for syndromic surveillance. Officials from two 
public health associations differed in their views on the usefulness of 
nationwide surveillance overall and BioSense in particular. While an 
official from one of the associations noted that other methods of 
conducting syndromic surveillance were more useful than electronic 
syndromic surveillance systems, officials from the other association 
stated that establishing a national syndromic surveillance system is 
essential. Further, preliminary results of a study on the use of 
syndromic surveillance systems conducted for CDC reiterated public 
health and hospital officials’ views of the usefulness of BioSense 
data. 

Although CDC identified annual and long-term costs for the initial 
implementation of the BioSense program, these estimates did not reflect 
the cost of implementing the redesigned program and its related system. 
CDC subsequently developed a draft plan for the redesigned program with 
high-level cost estimates based on program officials’ professional 
judgment. Best practices require cost estimates to be well-documented, 
comprehensive and accurate, and credible before they can be considered 
to be reliable. However, CDC’s cost estimates for the program are not 
reliable because they are only partially documented, are not 
comprehensive and accurate, and are not credible. 

According to program officials, CDC is in the strategic planning phase 
of the redesigned program and is only able to provide high-level cost 
estimates during this phase. Program officials stated that they intend 
to reflect more accurate and refined estimates as the program plans 
evolve and the technical and performance requirements of the program 
are better defined. Until program officials are able to define the 
information needed to develop cost estimates that are comprehensive and 
accurate, CDC will not be positioned to provide reliable cost estimates 
for implementing the BioSense program. 

CDC also identified annual and long-term timeline estimates for the 
initial implementation of the BioSense program. However, the timelines 
were not relevant to the goals of the redesigned program. While CDC has 
developed timeline estimates for implementing the redesigned BioSense 
program, they are not reliable. Best practices for reliable timeline 
estimates include, among other things, the identification of resources 
to complete each task, establishment of a critical path, and analysis 
of risks to the schedule. However, the agency has not assigned 
resources to activities, established a critical path, or analyzed risks 
to the schedule to ensure that its estimates are reliable. 

According to BioSense program officials, the redesigned program is in 
the early stages of planning, and they have not yet completed tactical 
plans that would identify resources needed to complete tasks and a 
critical path for the schedule. Until program officials consider these 
components, CDC will not be positioned to provide reliable timeline 
estimates for its plans to fully implement the BioSense program. 

CDC established performance measures and benchmarks for the initial 
design of the BioSense program. However, the performance measures 
defined in the business case were not focused on intended results of 
the redesigned program. OMB directs agencies to define meaningful 
annual and long-term performance measures to gauge the intended results 
of carrying out a program or activity early in the program to allow 
time for stakeholder review. Additionally, industry experts describe 
the need for performance measures to be developed with 
stakeholders’input early in a project’s planning process to provide a 
central management and planning tool and to monitor the performance of 
the project against plans and stakeholders’needs. 

CDC’s plan for the redesigned BioSense program describes a need to 
develop outcome-based performance measures.However, program officials 
stated that they do not intend to complete development of performance 
measures until the end of 2009. Until BioSense program officials 
develop these measures, they will lack an effective management and 
planning tool for gauging the success of the program against plans and 
against federal, state, and local partners’ use of the BioSense system 
for responding to public health events. 

We are recommending that the Director of the Centers for Disease 
Control and Prevention instruct the Director of the National Center for 
Public Health Informatics to take the following three actions while in 
the planning phase of the redesigned BioSense program: 

* develop reliable cost estimates for the program that are well-
documented, comprehensive and accurate, and credible; 

* develop reliable timeline estimates for implementing the program; 
and; 

* develop performance measures that address all phases of the program 
and focus the success of the program on CDC’s federal and state 
partners’ use of the system for responding to public health events. 

We received oral comments on a draft of this briefing from BioSense 
program officials, including the Director of CDC’s National Center for 
Public Health Informatics. The officials generally agreed with the 
information included in the draft and provided additional information 
and technical comments which were incorporated into the briefing as 
appropriate. 

[End of Section] 

Background: 

Historically, disease outbreaks have been recognized based either on 
accumulated case reports of diseases by local public health departments 
or by clinicians and laboratorians who alert public health officials 
about potential disease outbreaks. 

Complete responsibility for disease surveillance is shared among health 
care providers, including public health officials from 59 state and 
territorial health departments; more than 3,000 county, city, and 
tribal health departments; multiple federal agencies, including CDC; 
and more than 180,000 public and private laboratories. 

* States take the lead in conducting disease surveillance and 
supporting local emergency response efforts through a public health 
infrastructure that is made up of state and county, city, and tribal 
health departments—i.e., local health departments. State health 
departments are responsible for collecting surveillance information 
from local health departments, coordinating investigations and response 
activities, and sharing disease surveillance data with CDC. 

Local health departments are responsible for conducting the initial 
investigations into reports of diseases. Initial response to a public 
health emergency is generally a local responsibility that could involve 
multiple local health departments in a geographical region, with state 
health departments providing additional support when needed. 

The federal government’s role in disease surveillance traditionally has 
been to collect data from state and local health departments and 
perform nationwide analyses of the aggregated state and local data 
using surveillance systems. CDC uses information and communications 
systems to share disease surveillance information with state and local 
health departments and provides funding and technical expertise to 
support surveillance at the state and local levels. 

For nearly 10 years, federal, state, and local public health 
organizations, private companies, and academic institutions have been 
developing systems for collecting and analyzing electronic surveillance 
data from sources such as hospital emergency departments, clinical 
laboratories, and pharmacies. These systems are intended to better 
support efforts to detect disease outbreaks through electronic 
syndromic surveillance and to more efficiently communicate information 
to public health officials. For example: 

* The University of Pittsburgh and Carnegie Mellon University developed 
the Real-time Outbreak and Disease Surveillance (RODS) system in 1999. 

* DOD developed the Electronic Surveillance System for the Early 
Notification of Community-based Epidemics (ESSENCE) in 1999. A civilian 
version of the system is available, at no charge, to any health 
department that wants to use the system locally. 

* According to CDC, about 20 states have developed electronic 
surveillance systems. 

CDC officials stated that, as tools to share information more quickly 
and efficiently become available, the roles of federal, state, and 
local public health agencies in conducting disease surveillance are 
likely to evolve. 

According to BioSense program officials, the initial vision for the 
program was to provide real-time health data to federal, state, and 
local public health organizations and participating hospitals in order 
to: 

* provide health situational awareness by enabling public health 
officials at the federal, state, and local levels to monitor the size, 
location, and rate of an outbreak; 

* support early event detection by enabling public health officials to 
detect a potential public health emergency at the earliest possible 
time; and; 

* assist public health officials in responding to a disease outbreak or 
other emergency. 

A key component of the program is the BioSense system—an electronic 
information system which CDC developed to provide data and analytical 
tools for public health and other health organizations to use in 
conducting syndromic surveillance. 

The system is made up of (1) the hardware and software that enable 
BioSense to collect relevant data from public health organizations and 
hospitals and (2) software that analyzes the collected data and 
provides the analyzed data and analytical tools via a Web-based 
application for use by public health and hospital officials who conduct 
syndromic surveillance activities. 

The current system collects site-specific data directly from (1) state 
and local public health departments that collect hospital data from 
within their jurisdictions and (2) hospital, DOD and VA facility, 
health department, and laboratory data sources. The collected data are 
stored in a central CDC data repository. 

To provide data to the system, facilities must electronically store and 
maintain the data, and they must establish data-sharing agreements with 
CDC. As part of these agreements, CDC provides to facilities, at 
minimal or no charge,the hardware and software needed to collect and 
translate the facility’s data into a standard format to enable 
transmitting the data to BioSense. 

According to CDC, the agency has agreements with and the system 
currently collects data from about 570 hospitals, 320 DOD military 
treatment facilities, 860 VA hospitals and outpatient clinics, and 
other data sources that provide public health officials with the 
information needed to conduct syndromic surveillance, such as patients’ 
reasons for seeking medical attention, over-the-counter drug sales, 
prescription drug information from clinicians, and clinical laboratory 
test requests that could indicate disease outbreaks. 

The BioSense data analysis software aggregates and analyzes the 
electronic data collected from these facilities to identify disease 
syndromes within state and local jurisdictions. The system then 
displays graphical views of the presence of syndromes across time and 
geographical locations in the form of charts, graphs, and maps via a 
Web-based application. BioSense users can access the application via 
the Internet at no charge. 

* The views displayed by the BioSense Web-based application can be 
accessed by public health and health care professionals, such as public 
health epidemiologists and hospital infectious disease clinicians, who 
use the system to facilitate their ability to track the outbreak, 
treatment, and spread of diseases within their own jurisdictions. 
[Footnote 8] 

* CDC reports that there are more than 500 users of the BioSense 
system, and that they are located throughout 150 hospital, federal, 
state, and local facilities, including CDC. 

The following graphic provides an overview of the current BioSense 
system. 

[Refer to PDF for image] 

This figure is an illustration of the information flow in the current 
BioSense system, as follows: 

Medical facilities: Hospital information systems: hospital data is 
transmitted to: 

State and local health departments: 
* Health departments data collection and BioSense data collection: 
State and local hospital data is transmitted to: 

CDC: 
BioSense data analysis and Web-based applications analyze data, 
transmit it to the BioSense data repository. 

Hospitals, DOD, and VA facilities, health departments, and commercial 
laboratories transmit data to BioSense data collection, then transmit 
it to: 

CDC: 
BioSense data analysis and Web-based applications analyze data, 
transmit it to the BioSense data repository. 

Source: GAO analysis of CDC data. 

[End of figure] 

According to program officials, the current BioSense system: 

* collects data directly from medical facilities or local public health 
departments, VA and DOD facilities, or from state public health 
departments that have already collected the local health department and 
hospital data within their jurisdictions; the collected data are 
analyzed and stored in a central CDC data repository; 

* is not integrated with other federal public health information 
technology initiatives, such as CDC’s electronic disease surveillance 
system; and; 

* was designed and developed with limited collaboration between CDC and 
state and local public health stakeholders. 

In June 2005, we reported on federal agencies’ progress in implementing 
public health information technology initiatives. We noted that public 
health officials said they did not find the BioSense system useful 
because of limitations in the data being collected.[Footnote 9] 

In 2006, an independent contractor evaluated the BioSense program and 
identified about 50 management weaknesses that increased program risks, 
including the lack of a clear, consistent vision and supporting plans 
to guide and constrain the scope of the program. 

During the third quarter of fiscal year 2007, a new director that CDC 
hired to lead the National Center for Public Health Informatics 
initiated efforts to redesign the BioSense program.[Footnote 10] 

* According to BioSense officials, the redesigned program includes a 
new strategy that emphasizes increased collaboration with state and 
regional public health officials by providing more opportunities for 
stakeholder training and holding round-table meetings with stakeholders 
to gain a better understanding of their needs. 

* BioSense officials stated that, as part of the program’s redesign, 
they intend to address all of the weaknesses identified by the 
contractor’s evaluation that are relevant to the new strategy. 

* The officials also told us that they plan to reengineer the BioSense 
system to reflect the redesigned program’s new mission to “provide a 
system that comprehensively monitors the health care system of the 
United States for evidence of acute health threats to the public.” 

Program officials intend for the redesigned BioSense system to: 

* collect and analyze already-aggregated data from states and health 
information exchanges, rather than collect site-specific data from 
individual data sources as the current system does; 

* store results of the analyzed data in a central data repository at 
CDC (the data collected from state and health information exchanges 
will remain with their original sources); 

* include enhanced, user-customizable data analysis tools to allow 
users to easily tailor the application to better detect disease 
outbreaks and emergencies within their own jurisdictions; 

* allow states to share data and views across jurisdictions by building 
regional data-sharing collaboratives; 

* incorporate case detection technology for use in catastrophic as well 
as routine public health capacities; and; 

* be integrated with other public health information technology 
components, such as CDC’s electronic notifiable disease surveillance 
systems and electronic laboratory reporting, the Department of Homeland 
Security’s (DHS) National Biosurveillance Integration System (NBIS), 
[Footnote 11] and with the planned Nationwide Health Information 
Network (NHIN).[Footnote 12] 

According to program officials, Web-based analytical tools and data 
views will still be available to BioSense users via the Internet. 

The following graphic provides an overview of CDC’s initial proposal 
for the redesigned BioSense system. 

[Refer to PDF for image] 

This figure is a schematic diagram of the CDC’s initial proposal for 
the redesigned BioSense system, as follows: 

1) Medical facilities: 
Hospital information system collects data; data is transmitted to: 

State and local health departments; Using Syndromic surveillance, 
health departments data collection is transmitted to BioSense data 
collection, then transmitted to and from: 

CDC, BioSense System, BioSense data repository, which feeds both: 
* Federal component of the NHIN; 
* DHS's NBIS. 

2) Regional collaboratives and health information exchanges information 
systems collect data for case detection, which is transmitted to: 

State and local health departments; Using Disease surveillance, health 
departments data collection is transmitted to BioSense data collection, 
then transmitted to and from and from: 

CDC, BioSense System, BioSense data repository, which feeds both: 
* Federal component of the NHIN; 
* DHS's NBIS. 

3) State and local health departments also interact with Hospital, DOD 
and VA facilities and commercial labs, which utilize their information 
systems to collect data which is transmitted to BioSense data 
collection, then transmitted to and from: 

CDC, BioSense System, BioSense data repository, which feeds both: 
* Federal component of the NHIN; 
* DHS's NBIS. 

Source: GAO analysis of CDC data. 

[End of figure] 

In May 2008, BioSense program officials developed a draft plan that 
outlined a vision for implementing the redesigned BioSense program 
within a four-year timeframe.[Footnote 13] The draft plan articulates 
four strategic goals for the redesigned program along with strategic-
level activities for accomplishing the goals. The four strategic goals 
are: 

* sponsor regional collaboration and health information exchanges; 

* pursue open collaborative development; 

* •transition to a federated data model,[Footnote 14] and; 

* expand case detection technology. 

BioSense program officials intend for the four goals to be accomplished 
by the end of 2012 and stated that they will continue to update the 
plan and incorporate more tactical and operational planning details 
throughout 2008 and 2009. 

The plan describes 2008 as a transitional year during which CDC will 
continue to support ongoing initiatives and work to expand 
collaboration with its stakeholders. 

* Program officials incorporated into the new strategy for BioSense 
ongoing initiatives, such as proof-of-concept efforts for evaluating 
technology solutions for a federated data model and case-detection, and 
linked the outcomes of these initiatives to the new program goals. 

BioSense program funding covers costs for the program office, including 
staff, equipment, and contractor support to develop, implement, 
maintain, and support the BioSense system. It also covers extramural 
activities, such as stakeholder round-table meetings, and research 
grants and cooperative agreements with universities and other public 
health researchers to evaluate the usefulness of the system for 
supporting early event detection, situational awareness, and emergency 
response. CDC reported spending about $184 million on the program from 
2003 through June 2008. 

The following graphic illustrates the evolution of the BioSense program 
and system since the initiation of the system’s development. 

[Refer to PDF for image] 

This figure is a timeline illustrating the evolution of the BioSense 
program as follows: 

January 2002: Public Health Security and Bioterrorism Preparedness and 
Response Act of 2002. 

March, 2003: BioSense application including DOD, VA, and laboratory 
data rolled out to state and local users. 

May, 2004: BioSense application rolled out to state and local users. 

May, 2004: CDC reorganization (creation of the National Center for 
Public Health Informatics). 

June, 2005: GAO reported limited use of BioSense system by state and 
local users. 

August, 2006: Independent evaluation of BioSense program. 

June, 2007: First roundtable meeting held with stakeholders. 

July, 2007: New Director of National Center for Public Health 
Informatics initiates redesign of BioSense. 

November, 2007: Second roundtable meeting held with stakeholders. 

February, 2008: Kickoff meeting held with health information exchanges. 

March, 2008: First three health information exchanges identified for 
redesign program. 

April, 2008: Third roundtable meeting held with stakeholders. 

May, 2008: Draft strategic plan provided to Subcommittee staff. 

Source: GAO based on CDC data. 

[End of figure] 

[End of section] 

Results: BioSense Costs and Benefits: 

Costs and benefits of using data from BioSense and other syndromic 
surveillance systems that public health officials rely on are difficult 
to measure and compare. The state and local public health and hospital 
officials with whom we spoke stated that documenting costs and benefits 
of using syndromic surveillance systems such as BioSense is difficult 
and that they do not develop or track the costs or benefits of using 
these systems. Factors limiting the ability to compare the costs and 
benefits of operating BioSense with other surveillance systems include: 

* Some of the systems in use, including BioSense, are provided to 
state, local, and hospital users at very low or no cost and require 
little investment by users.[Footnote 15] Therefore, any additional data 
provided by these systems are beneficial. Without a need to justify a 
substantial investment in the systems, these organizations have not 
tracked costs or benefits. 

* Costs of developing and implementing systems can vary widely 
depending on the approach used to acquire, maintain, and monitor 
surveillance data. 

* Implementation of syndromic surveillance systems varies widely in 
scope by the type and volume of data collected, size of the 
jurisdiction,and number of data providers. BioSense is the only 
national system that centrally supports federal, state, and local 
public health and hospital users and that evaluates nationwide data to 
provide a national view of the country’s health. In contrast, DOD and 
state and local public health departments’ systems are intended 
specifically for use within their jurisdictions, such as within 
military bases or state and local public health districts. 

* System costs and benefits are difficult to track within the wide 
range of response activities that are undertaken during outbreaks. 
Syndromic surveillance activities are conducted as a component of a 
wide range of emergency response activities, including activities such 
as identifying available hospital beds and identifying specific 
patients, and it is difficult to segregate the costs of specific tasks 
from the overall response effort. 

Consistent with other public health officials’ experiences in using 
syndromic surveillance systems, BioSense program officials have not 
widely reported actual quantifiable benefits of using the BioSense 
system. 

Views on the usefulness of data provided by the BioSense system as 
compared with the usefulness of other similar systems’ data are mixed. 
The federal agencies that use the BioSense system for syndromic 
surveillance reported mixed views on the usefulness of the data 
provided by the system. 

* BioSense program officials stated that CDC’s epidemiologists and 
infection control analysts find the data provided by the BioSense 
system useful for detecting and monitoring potential events that could 
have a nationwide impact on public health and for notifying local 
public health officials about potential outbreaks in their 
jurisdictions. Program officials reported that, in 2007, BioSense data 
were useful in identifying ten local public health events. 

- Among these events, one of them was detected by CDC personnel using 
data from BioSense before it was detected by the local health 
department. 

* DOD and VA officials stated that their departments’ public health 
personnel do not rely on data provided by the BioSense system to 
conduct syndromic surveillance activities. 

* DOD officials told us that they use ESSENCE because it provides the 
ability for users to monitor the local populations for which they are 
responsible—i.e., populations within military installations—whereas the 
current version of BioSense presents aggregated information at the 
state level. Therefore, the results of the BioSense system do not 
provide information that DOD can act on within its jurisdiction. 

- VA officials stated that their public health personnel do not 
regularly rely on data provided by BioSense or any other system to 
conduct syndromic surveillance activities. They told us that they use 
the civilian version of ESSENCE more often than BioSense, though both 
systems are occasionally monitored. 

* Both DOD and VA officials stated that ESSENCE allows them more 
flexibility in evaluating and viewing data than BioSense does. 

Epidemiologists and infectious disease personnel at state and local 
public health organizations and hospitals use data from the BioSense 
system, other similar systems, and manual techniques for conducting 
syndromic surveillance. Many organizations operate multiple syndromic 
surveillance systems simultaneously, allowing access to multiple 
sources of data that help them identify and respond to emergencies. 

* For example, one county public health department we visited used a 
“dashboard” that employs BioSense, ESSENCE, and RODS to conduct 
syndromic surveillance. 

Of the ten organizations and hospitals we contacted—five local public 
health departments, three state health departments, and two 
hospitals—eight reported that they use data provided by the BioSense 
system. 

All but one of the eight state and local public health departments and 
hospitals that use the BioSense system reported that they also use 
other electronic syndromic surveillance systems. The following table 
identifies these users and the systems. 

Local public health departments: 

Organization: Maricopa County, Arizona; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Check]; 
State-Developed system: [Empty]; 
Locally-developed system: 
Manual analysis: [Empty]. 

Organization: Gwinnett County, Georgia; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Empty]; 
State-Developed system: [Check]; 
Locally-developed system: [Empty]; 
Manual analysis: [Empty]. 

Organization: Tarrant County, Texas; 
BioSense: [Check]; 
ESSENSE[A]: [Check]; 
RODS: [Check]; 
State-Developed system: [Empty]; 
Locally-developed system: [Empty]; 
Manual analysis: [Empty]. 

Organization: El Paso, Texas; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Check]; 
State-Developed system: [Empty]; 
Locally-developed system: [Empty]; 
Manual analysis: [Empty]. 

State public health organizations: 

Organization: Arizona; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Check]; 
State-Developed system: [Check]; 
Locally-developed system: [Empty]; 
Manual analysis: [Check]. 

Organization: Georgia; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Empty]; [Check]; 
State-Developed system: [Check]; 
Locally-developed system: [Empty]; 
Manual analysis: [Empty]. 

Organization: North Carolina; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Check]; 
State-Developed system: [Check]; 
Locally-developed system: [Empty]; 
Manual analysis: [Empty]. 

Hospitals: 

Organization: Tenet Healthcare, El Paso, Texas; 
BioSense: [Check]; 
ESSENSE[A]: [Empty]; 
RODS: [Empty]; 
State-Developed system: [Empty]; 
Locally-developed system: [Empty]; 
Manual analysis: [Check]. 

Total: 

BioSense: 8; 
ESSENSE[A]: 1; 
RODS: 4; 
State-Developed system: 4; 
Locally-developed system: 1; 
Manual analysis: 2. 

Source: GAO analysis based on data provided by state and local public 
health organizations and hospitals. 

[A] The civilian version of ESSENCE is used by state and local health 
departments. 

[End of table] 

The users we interviewed at the public health departments and hospitals 
expressed mixed views about the usefulness of the data provided by the 
BioSense system. 

* Seven of the eight users we interviewed stated that they use the data 
for situational awareness and that the system was useful for this 
activity. Although some users stated that they use BioSense for early 
event detection, they found that the system was only somewhat useful 
for this activity because it does not provide sufficient health-related 
data for event detection within their jurisdiction. 

* Six of the eight BioSense users we interviewed stated that they use 
the data either weekly or daily. Three of these users stated that the 
BioSense system is their most important electronic syndromic 
surveillance tool; two of these users stated that BioSense is the only 
tool that they use for syndromic surveillance.[Footnote 16] 

* Three of the seven users who also use other electronic syndromic 
surveillance systems stated that the BioSense system is less important 
to their overall syndromic surveillance efforts because the other 
systems they use better enable them to evaluate and view data based on 
specific needs of their jurisdictions. 

The users we interviewed stated that deficiencies in the data provided 
by the BioSense system limited their ability to use the data to conduct 
syndromic surveillance in their jurisdictions: 

* Limited data about health within jurisdictions—Seven of the users we 
interviewed indicated that the BioSense system would be more effective 
if its analysis included data from more hospitals in the jurisdiction. 
For example, one local health department stated that the BioSense 
system analyzes and provides its users with data for only seven of the 
27 hospitals within its jurisdiction. 

* Limited flexibility to query data on multiple symptoms—Three of the 
users we interviewed explained that the BioSense system does not 
provide the ability to query and report on multiple symptoms or to 
tailor the system to detect emerging threats specific to their 
jurisdictions. Users stated that this type of system flexibility—which 
is available in other systems such as ESSENCE and RODS—enables public 
health officials to more accurately identify health conditions that 
have more than one symptom, such as influenza, and to detect and 
monitor new illnesses as they appear within their own jurisdictions. 

* Inability to access patient-level data—Five users we interviewed 
cited the inability to access patient-level data as a significant 
limitation of the BioSense system. According to these users, the 
BioSense system does not provide these data, while other systems, such 
as ESSENCE, can be enabled to allow access to data about specific 
patients. Without this ability, local health officials may be limited 
in their ability to quickly identify an infected patient in their local 
areas.[Footnote 17] 

While these users expressed mixed views about the usefulness of the 
data provided by the BioSense system and stated that the system does 
not yet fully address their needs, when asked if BioSense helped them 
to better respond to emergencies, six of them stated that the data 
provided by the BioSense system was useful for this purpose. According 
to four of these users, the system provides an additional data resource 
for confirming data provided by the other syndromic surveillance 
systems that they use. 

Of four additional local public health officials with whom we spoke at 
a meeting arranged by the National Association of County and City 
Health Officials, three stated that their organizations use the 
BioSense system on a very limited basis because their locally-developed 
systems provide access to more, better, or more detailed data, and 
allow more flexibility in using the data. The one other organization 
does not use any electronic system for syndromic surveillance, but 
instead relies on manual data analysis because the hospitals within its 
jurisdiction do not provide electronic data to be processed by any 
electronic surveillance system. 

Additionally, the public health experts from the two associations with 
whom we spoke offered differing views on the usefulness of the BioSense 
system and other similar electronic syndromic surveillance systems. 

* The Council for State and Territorial Epidemiologists official with 
whom we spoke stated that surveillance for disease and outbreak 
detection has been accomplished primarily by astute clinicians and 
physicians within a jurisdiction, and that communication support is 
more important for detecting outbreaks across jurisdictions than 
syndromic surveillance systems. Some capabilities developed through the 
BioSense program, such as the data-viewing tools and the collection of 
real-time hospital data, have been very useful, and the BioSense system 
has been shown to be helpful in monitoring cases of disease outbreaks, 
such as influenza. 

* On the other hand, the Association for State and Territorial Health 
Officials representatives with whom we spoke believe that establishing 
a national syndromic surveillance system is essential and there is no 
question about the need to have such a system. They pointed out that a 
national system such as BioSense should be made up of multiple systems 
instead of a single system, build on existing local and state 
capabilities, and include local and state health officials as integral 
partners in its design. While these officials had concerns with the 
initial design of BioSense, they stated that they are pleased with 
CDC’s new plans. 

In October 2006, CDC funded four institutions to conduct evaluations of 
syndromic surveillance systems. The results of the studies are intended 
to be used together to evaluate the usefulness of the BioSense system 
at the federal, state, and local levels of public health for supporting 
early event and situational awareness activities. 

Preliminary results for one of the studies were published in 2007. 
[Footnote 18] Initial case studies for this evaluation focused on a 
large salmonella outbreak and a tornado in Georgia, the 2006-2007 
influenza season, and a wildfire in south Georgia and north Florida. 
The study is to be completed in November 2008. The preliminary results 
of the evaluation reiterated the views of the users we contacted and 
found that: 

* Among the four events, the utility of syndromic surveillance was 
greatest for monitoring the influenza season. 

* In most instances, BioSense was not used because of the lack of local 
real-time hospital data from local hospitals and the inflexibility of 
the system. 

* Syndromic surveillance systems used first were those that were 
developed locally to support state and local public health and health 
care practices and relationships. 

In addition to BioSense system users, officials at BioSense facilities 
who do not use the system offered several explanations for not doing 
so. 

* An official we spoke with at one hospital said that they had never 
used the system because they use the state’s system; they stated that 
they rely on relationships with state and local public health 
departments for conducting surveillance and use the state-level system, 
which was designed to meet specific needs of their jurisdiction. 

* Officials at one local health department stated that they had 
accessed the BioSense system only once during the past year. The 
department uses ESSENCE, RODS, and manual analysis to conduct syndromic 
surveillance. These officials stated that they do not use BioSense 
because the system lacks adequate data for their jurisdiction and 
because BioSense does not provide the flexibility available in ESSENCE 
and RODS, such as allowing users to perform multiple queries on the 
data or to customize views of the data analyzed by the systems. 

State and local public health and hospital officials we interviewed 
identified lessons they had learned from their experiences in 
developing, implementing, and sustaining syndromic surveillance systems 
to ensure the systems are useful. They reported learning that: 

* Systems should be developed incrementally. Officials with experience 
in developing or implementing state and local systems emphasized that 
systems should be developed incrementally based on users’ needs and 
lessons learned from each increment. 

* Systems must be sustainable at the local level. Initiatives to 
develop and implement systems must build in support for IT 
infrastructure, installation and maintenance, and staff training. 

* Stakeholder communication and involvement in determining requirements 
is critical. The development and implementation of syndromic 
surveillance systems must involve frequent, effective communications 
with stakeholders while defining, documenting, and sharing project 
requirements. These experiences provide valuable lessons that BioSense 
program officials could apply to their approach for reengineering the 
system to help ensure that the data provided by the system are useful 
for BioSense stakeholders. 

Results: BioSense Plans: Cost Estimates: 

OMB has made developing accurate life-cycle cost estimates a priority 
for agencies in properly managing their portfolios of capital assets, 
including information technology, that have an estimated life of two 
years or more.[Footnote 19] In addition, the draft Cost Assessment 
Guide, which we developed based on best practices, defines 
characteristics of reliable cost estimates for federal 
agencies’information technology programs.[Footnote 20] The guide 
describes reliable cost estimates as those that are well-documented, 
comprehensive and accurate, and credible. 

* Well-documented cost estimates are those that include source data and 
their significance, clearly detailed calculations and results, and 
explanations of why particular methods and references were chosen. The 
data used to estimate the costs can be traced to their source 
documents. 

* Comprehensive and accurate cost estimates have enough detail to 
ensure that cost elements are neither omitted nor double counted, and 
all cost-influencing assumptions are detailed in the estimates’ 
documentation. 

* Cost estimates are credible when limitations of the analysis are 
discussed, assumptions made about the costs are varied, and outcomes 
are recomputed to determine how sensitive they are to changes in the 
assumptions. Risk analyses are also performed to determine the level of 
risk associated with the estimate. The estimate’s results are cross-
checked, and an independent cost estimate is developed to determine 
whether other estimating methods produce similar results. 

By addressing these characteristics, program officials should be better 
able to define reliable cost estimates that are comprehensive and 
accurate and that can be easily and clearly traced, replicated, and 
updated. Additionally, program officials should reliably estimate costs 
before proceeding too far into development or production to reduce the 
risk that the program will not be able to meet its established cost 
targets. 

CDC identified cost estimates for the initial BioSense in its fiscal 
year 2009 business case that was submitted to OMB in September 
2007.[Footnote 21] Program officials estimated the total cost for the 
program through 2013 to be $369million. However, these estimates did 
not reflect the cost of implementing the redesigned program and its 
related system. 

In May 2008, CDC included in its plans for the redesigned program cost 
estimates for completing the strategic activities that are intended to 
provide an operational program by 2012.[Footnote 22] Program officials 
report the total estimated cost of implementing the redesigned program 
from 2009 through 2012 to be about $314 million. The estimates cover 
costs of completing each of the strategic activities, such as 
federating existing data sources, developing case detection 
methodologies, and supporting research and development for real time 
surveillance. According to program officials, the estimates also cover 
the costs of existing BioSense operations including the program 
office’s staff, travel, transportation, and contractor support. 

CDC’s cost estimates for implementing the redesigned BioSense program 
are only partially documented. Specifically, the documentation for the 
estimates consisted only of explanations of the methods and references 
chosen to compute the estimates. According to an official with CDC’s 
Financial Management Office, the initial cost estimates for the 
redesigned BioSense program were developed based on program officials’ 
professional judgment.[Footnote 23] As professional judgment estimates, 
the official did not believe that the agency was required to develop or 
maintain supporting documentation. 

Additionally, the cost estimates for BioSense are not comprehensive and 
accurate because the estimates were developed without considering all 
cost elements. According to program officials, the estimates do not 
include the cost of technology required to support new methods for 
measuring performance of the system, such as tools for simulating 
emergency response scenarios to estimate the impact that use of the 
BioSense system has on the time required to respond to health threats. 

Finally, the cost estimates do not meet the criteria for credibility as 
defined in our cost estimation guide. The BioSense cost estimators did 
not vary major assumptions to determine whether and to what extent 
outcomes are sensitive to changes in the assumptions. Program officials 
did not conduct risk and uncertainty analyses to determine the level of 
risk and the extent to which the actual costs may vary from the initial 
estimates. Additionally, the cost estimates for BioSense have not been 
independently reviewed and verified to determine if other estimating 
methods would produce similar results. 

Program officials stated that the cost estimation activities were 
conducted in the early planning phases of the redesigned program and 
that they intend to refine cost estimates for a more accurate 
reflection of the cost to fully implement the BioSense program and its 
related system as the program requirements are defined in more 
detail.[Footnote 24] Until program officials develop well-documented, 
comprehensive and accurate, and credible cost data, CDC will not be 
able to reliably estimate the cost of fully implementing BioSense. 

Results: BioSense Plans: Timeline Estimates: 

Our draft Cost Assessment Guide identifies best practices for 
estimating project timelines and schedules, which include: 

* defining, sequencing, and estimating the duration for each activity; 

* assigning resources (e.g., labor, material, and overhead) to all 
activities; 

* identifying the critical path for all activities; 

* identifying float time—the amount of time a task can slip before 
affecting the critical path—between activities; and; 

* conducting a schedule risk analysis using statistical methods to 
determine the amount of time to reserve for contingencies. 

By following these practices, program officials should be better able 
to define reliable timeline estimates that can be used to identify when 
problems or changes may occur and the impact they may have on the 
success of the program. Further, according to OMB guidance,[Footnote 
25] reliable timeline estimates should be defined before program 
officials proceed too far into development or production to reduce the 
risk that the program will not be able to meet its time frames and to 
reduce the risk of undesirable outcomes. 

CDC’s fiscal year 2009 OMB business case for the initial BioSense 
program identified annual and long-term timelines and milestones for 
meeting program goals through fiscal year 2013. CDC also reported the 
goals and the status of the program’s efforts to meet the goals for 
fiscal years 2006 and 2007. For example, the business case identified a 
milestone to support research related to BioSense algorithms, 
bioterrorism, and pandemic flu in 2006 and reported that the program 
had met that milestone. 

In its draft plan for the redesigned program, CDC defined the timelines 
for completing the program’s strategic activities and goals within a 
four-year time frame, as shown in the following table.[Footnote 26] 
According to BioSense program officials, these timelines were defined 
based on experiences and knowledge gained from implementing prior and 
ongoing pilot projects.[Footnote 27] 

Strategic Goals: Sponsor regional collaboration and health information 
exchanges; 
Strategic Activities: Expand state and local partnerships with the CDC 
and develop regional collaboratives for real time surveillance; 
Timeline: 2008-2010. 

Strategic Goals: Sponsor regional collaboration and health information 
exchanges; 
Strategic Activities: Support linkages between public health and health 
information exchanges for real time surveillance; 
Timeline: 2009-2012. 

Strategic Goals: Pursue open collaborative development; 
Strategic Activities: Pursue open source collaborative development; 
Timeline: 2009-2012. 

Strategic Goals: Transition to a federated data model; 
Strategic Activities: Federate existing state and local real time 
surveillance data sources; 
Timeline: 2009-2010. 

Strategic Goals: Expand case detection technology; 
Strategic Activities: Refine and deploy BioSense case detection 
technologies; 
Timeline: 2009-2012. 

Strategic Goals: Expand case detection technology; 
Strategic Activities: Enhance incentives for clinical providers to 
transmit data to public health; 
Timeline: 2009-2011. 

Source: CDC’s National Center for Public Health Informatics. 

[End of table] 

While program officials have defined timelines for the program’s 
strategic-level activities, they have not completed the detailed plans 
needed to estimate reliable timelines for implementing the redesigned 
program. 

* While the strategic activities were defined and sequenced, resources 
have not yet been assigned to the activities, a critical path for the 
program’s schedule has not been identified, float time between 
activities has not been determined, and a schedule risk analysis has 
not been performed to plan for contingencies. 

BioSense program officials stated that the program is still in the 
early stages of its redesign and that they have recently begun to 
develop tactical-level plans that include the detailed information 
needed to define more specific timelines for completing the program’s 
strategic activities. In this regard, they drafted a document that 
reflects timelines for completing specific activities in fiscal years 
2008 and 2009 and stated that they intend to finalize a detailed 
tactical plan for the program by September 30, 2008. 

Until the program completes detailed plans that define the steps and 
resources needed to accomplish the plan’s strategic activities, 
identify the schedule’s critical path, and perform a schedule risk 
analysis, CDC will not be positioned to provide reliable timeline 
estimates that can be used to identify when problems or changes may 
occur and the impact they may have on CDC’s plans to fully implement 
BioSense by 2012. 

Results: BioSense Plans: Performance Measures: 

OMB directs agencies to define and select meaningful annual and long-
term outcome-based performance metrics that measure the intended result 
of carrying out a program or activity.[Footnote 28] Additionally, 
industry experts describe the need for performance measures to be 
developed with stakeholders’ input early in a project’s planning 
process to provide a central management and planning tool and to 
monitor the performance of the project against plans and stakeholders’ 
needs.[Footnote 29] According to the Software Engineering Institute, 
performance measures are effective mechanisms for providing credible 
evidence of a program’s progress.[Footnote 30] 

In the fiscal year 2009 business case for BioSense, program officials 
identified performance measures for the initial implementation of the 
program and reported the status of the program in meeting benchmarks 
for 2006 and 2007. For example, a decrease in the time required to 
update BioSense after receipt of data from data providers was 
identified as a performance measure with a benchmark to reduce the time 
from an average of 2.81 hours in 2006 to no more than 2 hours in 2007. 
However, the performance measures defined in the business case were not 
focused on intended results of the redesigned program. 

CDC’s draft plan for the redesigned BioSense program describes a need 
to develop outcome-based performance measures that focus the success of 
the program on federal, state, and local partners’ use of the BioSense 
system. They stated that new performance measures for the redesigned 
system are to be included in the program’s plan by the end of calendar 
year 2009.[Footnote 31] However, it is important to complete the 
development of performance measures early in the planning phase of the 
program to provide an essential planning tool that can be utilized 
throughout all phases of the development and implementation of 
BioSense. 

Unless BioSense program officials develop outcome-based performance 
measures based on stakeholder input during the planning phase of the 
program, they will lack an effective management and planning tool 
needed to monitor the performance of the program against plans and 
stakeholders’needs throughout all phases of the program. 

[End of section] 

Conclusions: 

The costs and benefits of using data from the current implementation of 
the BioSense system and other similar systems are difficult to measure 
and compare, and state and local public health and hospital officials 
with whom we spoke do not document or track costs and benefits of using 
these systems. Additionally, these officials expressed mixed views on 
the usefulness of BioSense data as compared to the usefulness of data 
from other systems. 

BioSense program officials have developed initial high-level cost and 
timeline estimates for completing the implementation of a redesigned 
BioSense program and related system that are intended to address 
weaknesses of the initial implementation of the program and improve the 
usefulness of the system for CDC’s stakeholders. However, program 
officials have not yet reliably estimated costs and timelines for fully 
implementing the redesigned BioSense and for managing the development 
and implementation of the program. Additionally, program officials do 
not expect to complete the development of performance measures before 
the end of calendar year 2009, which increases the risk that they will 
not be able to effectively monitor the performance of the program 
against plans and stakeholders’needs throughout all phases of 
development and implementation. 

Until BioSense officials develop reliable cost and timeline estimates 
and outcome-based performance measures, CDC’s plans for the redesigned 
program and its related system will lack key components needed to 
effectively manage the program throughout all phases of the program. 
Further, the agency will lack management and planning tools needed to 
effectively gauge the success of the BioSense system’s performance 
toward improving federal, state, and local partners’ abilities to 
respond to public health events. Moreover, the agency remains at risk 
that it will perpetuate weaknesses identified in the agency’s initial 
implementation of the program and related system and will continue to 
spend money on a program that does not provide intended results or meet 
stakeholders’ needs. 

[End of section] 

Recommendations for Executive Action: 

To ensure that CDC defines reliable plans for effectively managing the 
development and implementation of the redesigned BioSense program and 
its related system throughout all phases of the program, we are 
recommending that the Director of the Centers for Disease Control and 
Prevention instruct the Director of the National Center for Public 
Health Informatics to take the following three actions while in the 
planning phase of the program: 

* develop reliable cost estimates for fully implementing the program 
that are well-documented, comprehensive and accurate, and credible; 

* develop reliable timeline estimates for implementing the program; 
and; 

* with stakeholder input, develop outcome-based performance measures 
that address all phases of the program and that focus the success of 
the program on CDC’s federal, state, and local partners’ use of the 
system for responding to public health events. 

[End of section] 

Agency Comments and Our Evaluation: 

We received oral comments on a draft of this briefing from CDC’s 
BioSense program officials, including the Director of the National 
Center for Public Health Informatics. The officials generally agreed 
with the information included in our draft briefing and provided 
additional information and technical comments related to the program, 
which we have incorporated as appropriate. 

BioSense program officials acknowledged areas of deficiencies in the 
initial design of the BioSense system and stated that the current 
managers of the program, with extensive user engagement through 
BioSense roundtable meetings, had identified the shortcomings detailed 
in our briefing. They stated that they are consequently taking steps to 
reconfigure the system and to address these problems, as outlined in 
the draft BioSense Strategic Plan and as described in this briefing. 

[End of section] 

[End of slide presentation] 

Appendix II: Comments from the Department of Health and Human Services: 

Department Of Health & Human Services: 
Office Of The Secretary: 
Assistant Secretary for Legislation: 
Washington, DC 20201: 

November 10, 2000: 

Valerie C. Melvin: 
Director, Human Capital and Management Information Systems: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Melvin: 

Enclosed are the Departments comments on the U.S. Government 
Accountability Office's (GAO) draft report entitled: "Health 
Information Technology: More Detailed Plans Needed for the Centers for 
Disease Control and Prevention's Redesigned BioSense Program" (GAO 09-
100). 

The Department appreciates the opportunity to comment on this report 
before its publication. 

Sincerely, 

Signed by: 

Jennifer R. Luong, for: 

Vincent J. Ventimiglia, Jr. 
Assistant Secretary for Legislation: 

Attachment: 

General Comments Of The Department Of Health And Human Services (HHS) 
On The Government Accountability Office's (GAO) Draft Report Entitled: 
Health Information Technology: More Detailed Plans Needed For The 
Centers For Disease Control And Prevention's Redesigned Biosense 
Program (GAO-09-100): 

The Centers for Disease Control and Prevention (CDC) wishes to thank 
the Government Accountability Office (GAO) for the opportunity to 
review and comment on this Draft Report. We appreciate the effort and 
professionalism that went into the thorough evaluation of the BioSense 
program and welcome the recommendations and conclusions therein. In 
this regard we respectively submit the following general comments. 

In the Report's opening page, under "What GAO found" the authors note 
that "...cost estimates for the redesigned program are not reliable 
because they are only partially documented, are not comprehensive and 
accurate, and therefore are not credible", and "... the agency has not 
implemented these practices, resulting in timelines for the redesigned 
program that are not reliable." 

A planning and implementation process has been launched to ensure that 
the BioSense program develops reliable cost estimates and timelines, 
and identifies appropriate outcome-based performance measures. This 
process follows a phased approach that focuses on work products being 
vetted through internal and external stakeholders to ensure maximum 
input and impact to public health. 

This first phase of this process focused on identifying the steps 
necessary to implement the strategic objectives outlined in the 
redesigned BioSense strategic plan. The program initiated a series of 
working groups integrating stakeholders from the Association of State 
and Territorial Health Officials, National Association of County and 
City Health Officials, Council of State and Territorial 
Epidemiologists, and Association of Public Health Laboratories, as well 
as those internal to CDC. Each working group had representation by one 
or more of the partner organizations. The main deliverables were a 
series of work breakdown structures[Footnote 32] for each strategic 
objective. 

The second phase focused on synthesizing these work breakdown 
structures and refining the steps identified by the working groups to 
develop a tactical roadmap and define associated performance outcomes. 
The draft tactical plan and comprehensive breakdown structure 
deliverables are currently under review by CDC's National Center for 
Public Health Informatics (NCPHI) leadership to be delivered to working 
group members on 11/14/08. 

To determine comprehensive and accurate cost estimates and performance 
outcome measures, the BioSense program is preparing to implement two 
complementary efforts: Program Planning and Performance Measurement. 
The program planning effort expands the focus areas identified in the 
tactical plan into projects, detailed tasks, deliverables, and 
resources required. Cost estimates will be developed to implement these 
tasks, and activities will be prioritized accordingly. This activity is 
targeted for completion on 12/30/08. Once finalized, the program plan 
will include all projects and their related tasks and costs. The 
program plan is scheduled for completion in February 2009. 

The program plan will be managed to ensure reliable, timely and 
accurate activity and cost forecasting. 

A performance measurement working group will engage stakeholders, state 
and local representatives and subject matter experts such as members of 
ASTHO, NACCHO, CSTE, APHL and the NCPHI Board of Scientific Counselors 
to: 

* Review and adjust the proposed performance outcomes; 

* Identify baseline and target performance measures; 

* Develop a tracking and reporting process. 

CDC has preliminarily identified 24 milestones with accompanying target 
completion dates. Of these 24 milestones, six have been completed, four 
are in progress and fourteen are planned with estimated completion 
dates ranging from two weeks to 12 months. 

With regards to the statement made several times that BioSense is seen 
by many state personnel as not as useful as the state systems (ESSENCE, 
RODS, etc.), it is important to note that BioSense is not intended to 
replace local systems. Rather, BioSense adds value to local systems by 
1) aggregating data from local systems for regional or national view, 
2) making national datasets available, 3) making software tools 
available, and 4) funding regional collaboratives, health information 
exchanges (HIEs) and research to improve the field overall. [Footnote 
33] 

In closing, CDC appreciates GAO's review and recommendations. CDC is 
committed to continuing and enhancing its efforts to develop reliable 
cost and timeline estimates and engaging stakeholders in developing 
outcome-based performance measures for the redesigned BioSense program. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, key contributions to this 
report were made by Teresa Tucker (Assistant Director), Heather A. 
Collins, Neil J. Doherty, Amanda C. Gill, Nancy E. Glover, Franklin D. 
Jackson, Mohammad S. Khan, and Lee A. McCracken. 

[End of section] 

Footnotes: 

[1] 42 U.S.C. § 247d-4(b). 

[2] According to CDC, syndromic surveillance is a technique that uses 
health-related data to identify patterns of disease symptoms prior to 
specific diagnoses. Effective use of this technique can provide both 
situational awareness--knowledge of the size, location, and rate of 
spread of an outbreak--and early event detection, signaling a 
probability of an outbreak sufficient to warrant a public health 
response and leading to early identification of illnesses and other 
events of public health importance. 

[3] H.R. Rep. No. 110-231, at 122 (2007). 

[4] DOD developed the ESSENCE system in 1999. A civilian version of the 
system is available, at no charge, to any health department that wants 
to use the system locally. 

[5] H.R Rep. No. 110-231, at 122 (2007). 

[6] Executive Office of the President, Office of Management and 
Budget,Planning, Budgeting, Acquisition, and Management of Capital 
Assets,Circular No. A-11, Part 7 (Washington, D.C.: June 2008). See 
also, U.S. Government Accountability Office, Cost Assessment Guide: 
Best Practices for Estimating and Managing Program Costs, Exposure 
Draft, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] 
(Washington, D.C.: July 2007). 

[7] Executive Office of the President, Office of Management and Budget, 
Guide to the Program Assessment Rating Tool, (Washington, D.C.: January 
2008); Thomas Wettstein and Peter Kueng, "A Maturity Model for 
Performance Measurement Systems," Management Information Systems 2002--
Incorporating GIS and Remote Sensing(Southampton: WIT Press, 2002),113-
122; and Karen J. Richter, Ph.D., Institute for Defense Analyses, CMMI® 
for Acquisition (CMMI-ACQ) Primer, Version 1.2 (Software Engineering 
Institute, May 2008). 

[8] Users can access data for neighboring jurisdictions or other 
locations only with additional approval from the other jurisdictions 
involved and from CDC. 

[9] GAO, Information Technology: Federal Agencies Face Challenges in 
Implementing Initiatives to Improve Public Health Infrastructure, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-308] (Washington, 
D.C.: June 2005). 

[10] The BioSense program is managed by the National Center for Public 
Health Informatics within CDC. 

[11] The National BioSurveillance Integration System is a federal 
effort to combine multiple data streams from sector-specific 
agencies—those with medical, environmental, agricultural, and 
intelligence data—to give DHS situational awareness that is intended to 
allow earlier detection of events and to assist response activities. 

[12] The Nationwide Health Information Network is a critical portion of 
the HHS Office of the National Coordinator for Health Information 
Technology’s agenda intended to provide a secure, nationwide, 
interoperable health information infrastructure that will connect 
providers, consumers, and others involved in supporting health and 
health care. It is planned to be a “network of networks” built out of 
state and regional health information exchanges and other networks to 
support the exchange of health information by connecting these 
networks. 

[13] Centers for Disease Control and Prevention National Center for 
Public Health Informatics, BioSense Strategic Plan, FY 2008–2012, 
Version 6 (Draft); (Atlanta, Ga.). 

[14] A federated data model supports data sharing across a large 
community despite having data stored in different geographic locations. 
Use of this model would be expected to enable the public health 
community to store data locally, in state and local public health 
departments, while enabling CDC and other organizations to access and 
analyze that data. 

[15] Three organizations in our review reported that the only cost for 
using surveillance systems was staff time for using the systems, which 
is performed in combination with other activities. Most state and local 
users in our review used federal or state grant funding to cover system 
costs. 

[16] One of the two users stated that, although other tools were used 
at the facility, they were not used specifically for syndromic 
surveillance. 

[17] In their comments on a draft of this briefing, BioSense program 
officials stated that individual patient-level data are available from 
application-level views provided by the BioSense system. 

[18] James W. Buehler, M.D., Alexander P. Isakov, M.D, M.P.H., Michael 
J. Prietula, Ph.D.,Donna J. Smith, M.A., Ellen A. Whitney, M.P.H., 
Preliminary Findings from the BioSense Evaluation Project(Rollins 
School of Public Health, School of Medicine, and Goizueta Business 
School, Emory University, Atlanta, Ga.; 2007). 

[19] OMB Circular A-11, Part 7. 

[20] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP]. 

[21] OMB requires agencies to submit business cases through OMB Exhibit 
300’s for major information technology programs to establish cost, 
schedule, and measurable performance goals. 

[22] According to program officials, CDC’s cost estimates reflect the 
agency’s plan to complete the development and implementation of the 
redesigned BioSense program and its related system by 2012 rather than 
2013, as its previous plans reflected. 

[23] CDC defines its professional judgment estimates as informal 
estimates provided to Congress. According to CDC officials, the agency 
provides professional judgment estimates without regard to the 
competing priorities that the agency, the President, and their advisors 
must consider as budget submissions to the Congress are developed. 

[24] BioSense program officials stated that they intend to use the 
refined estimates for the preparation of future congressional budget 
submissions. 

[25] Executive Office of the President, Office of Management and 
Budget, Memorandum on Conducting Acquisition Assessments with 
Guidelines for Assessing the Acquisition Function (Washington, D.C.: 
May 21, 2008). 

[26] According to program officials, CDC’s timeline estimates were 
developed to support completion of the development and implementation 
of the redesigned BioSense program and its related system by 2012 
rather than 2013, as its previous plans reflected. 

[27] CDC officials described an additional activity, support for 
research to develop innovative and promising technologies for real time 
surveillance, as a cross-cutting activity that supports each goal from 
2009 through 2012. 

[28] OMB, Guide to the Performance Assessment Rating Tool. 

[29] Thomas Wettstein and Peter Kueng, “A Maturity Model for 
Performance Measurement Systems,” and Karen J. Richter, Ph.D., 
Institute for Defense Analyses, CMMI®for Acquisition (CMMI-ACQ) Primer, 
Version 1.2. 

[30] GAO, Veterans Benefits Administration: Progress Made in Long-Term 
Effort to Replace Benefits Payment System, but Challenges Persist, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-614] (Washington, 
D.C.: April 2007). 

[31] In commenting on a draft of this briefing, BioSense program 
officials stated they intend for federal, state, and local partners to 
be engaged in the development of outcome-based performance measures 
early in the detailed planning process, beginning in November 2008. 

[32] A work breakdown structure or WBS is a tree structure that permits 
summing of subordinate costs for tasks, materials, etc., into their 
successively higher level "parent" tasks, materials, etc. It is a 
fundamental tool commonly used in project management and systems 
engineering. 

[33] With regards to progress on establishing regional collaboratives, 
grants have been awarded to four sites: National Capital Region, 
Missouri-Kansas, Pennsylvania-Ohio, Southeast Region (Alabama, Florida, 
Georgia, Mississippi, North Carolina, South Carolina, Tennessee). In 
addition, three funded HIEs have already begun work: Indiana, New York, 
Eastern Washington/Western Idaho. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: