This is the accessible text file for GAO report number GAO-05-86 
entitled 'Data Quality: Census Bureau Needs to Accelerate Efforts to 
Develop and Implement Data Quality Review Standards' which was released 
on December 17, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters: 

November 2004: 

DATA QUALITY: 

Census Bureau Needs to Accelerate Efforts to Develop and Implement Data 
Quality Review Standards: 

GAO-05-86: 

GAO Highlights: 

Highlights of GAO-05-86, a report to congressional requesters

Why GAO Did This Study: 

Data from the decennial census are used to apportion and redistrict 
seats in the House of Representatives, distribute billions of dollars 
of federal funds, and guide the planning and investment decisions of 
the public and private sectors. Given the importance of these data, it 
is essential that they meet high quality standards before they are 
distributed to the public. After questions arose about the quality of 
certain data from the 2000 Census, the requesters asked GAO to review 
U.S. Census Bureau (Bureau) standards on the quality of data 
disseminated to the public.

What GAO Found: 

The Bureau did not have detailed agencywide standards for the review of 
data from the 2000 Census to determine if the data were of sufficient 
quality for public dissemination. Instead, analysts and managers in 
different parts of the Bureau primarily used their own judgment and 
unwritten, program specific guidance to decide when and whether data 
should be released and what supporting information should accompany the 
data. The lack of sufficient data quality review standards led to a 
variety of problems, including missed opportunities for correcting data 
before release, inconsistent decisions on disseminating data with 
similar quality issues, and inadequate communication to users about the 
reasons for dissemination decisions. As a result, some users of data 
from the 2000 Census lost confidence in the quality of the data and in 
the Bureau’s review procedures.

In the 4 years since the 2000 Census, the Bureau has publicly issued 
general information quality guidelines, including eight performance 
principles, and one new standard that allows individuals to request 
correction of certain errors in data disseminated by the Bureau. Both 
of these documents resulted from the enactment of the Information 
Quality Act in 2000 and the subsequent guidelines issued by the Office 
of Management and Budget in 2002. However, except for the one standard, 
the Bureau did not provide any specific guidelines or procedures on the 
implementation of the general guidelines. The Bureau also began work on 
other standards, including one on minimal information that must be 
provided with data and another on discussion of errors in data released 
to the public. Neither has been issued in final form. In response to 
our earlier recommendations, the Bureau created an interdirectorate 
working group charged with developing and publicly issuing Bureau-wide 
standards for quality in data releases. The working group has taken 
some steps, but the Bureau has not provided information on the scope or 
the time frame for its efforts to develop these standards.

The standards that the Bureau has under development and the activities 
of the working group are encouraging. However, it will be important for 
the Bureau to proceed with greater urgency to ensure that fully tested 
standards are in place for the 2010 Census. Until spring 2004, no 
additional resources were provided to support the working group, and 
over a year after it began, it has not issued any new standards or said 
when it will be ready to do so.

A comprehensive, Bureau-wide data quality framework, with interrelated 
standards, and specific implementing procedures could help ensure 
consistent decisions about the quality of the data from the next 
decennial census and conditions under which the data will be 
disseminated. Moreover, the benefits the Bureau can achieve by 
developing and effectively implementing comprehensive data quality 
standards would not be limited to the decennial census. Because they 
would apply to all data disseminated by the Bureau, it will be 
important for any new standards to be developed promptly, implemented 
across the Bureau, and released to the public.

What GAO Recommends: 

GAO recommends that the Bureau
* accelerate its effort to establish a comprehensive set of data 
quality review standards by developing and making public a detailed 
plan, including interim milestones for developing such standards and 
procedures, and 
* include the implementation of data quality review standards in the 
Bureau’s plans for the 2010 Census, and test new draft guidelines on 
data quality review using the annual American Community Survey or other 
surveys.

The Bureau [insert response]Commerce agreed with our second 
recommendation but not the first. However, because the Bureau has yet 
to approve and make public data quality review standards, we continue 
to believe that it needs to accelerate its effort. 

www.gao.gov/cgi-bin/getrpt?GAO-05-86.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Patricia A. Dalton at 
(202) 512-6806 or daltonp@gao.gov.

[End of section]

Contents: 

Letter: 

Results in Brief: 

Background: 

Scope and Methodology: 

Professional Judgment Drove Data Dissemination Decisions: 

The Bureau Has Made Limited Progress in Publicly Issuing New Standards 
on the Quality of Data Disseminated to the Public since the 2000 
Census: 

Greater Commitment to New Standards for Public Dissemination of Data 
Could Help Bureau Avoid Problems in Disseminating 2010 Census and Other 
Data: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Comments from the Department of Commerce: 

Letter November 17, 2004: 

The Honorable Wm. Lacy Clay: 
Ranking Minority Member: 
Subcommittee on Technology, Information Policy, Intergovernmental 
Relations and the Census: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Danny K. Davis: 
The Honorable Carolyn B. Maloney: 
The Honorable Charles A. Gonzalez:
House of Representatives: 

As one of the nation's principal statistical agencies, the U.S. Census 
Bureau (Bureau) collects and disseminates data that are used to 
apportion and redistrict seats in the House of Representatives, 
distribute billions of dollars of federal funds, and guide the planning 
and investment decisions of the public and private sectors. Given the 
importance of Bureau data to our economy and system of governance, 
census information, like other federal statistics, must be of high 
quality before it is released to the public. Specifically, the data 
must be accurate, timely, accessible, relevant, and objective. Failure 
to meet this threshold could impair decision making and erode public 
confidence in the information and the Bureau's credibility.

Producing high-quality data is a continuing challenge, in part because 
the methods used to collect and process census data are complex and 
subject to some degree of error. Consequently, the Bureau must decide 
if and when the quality of each set of data is high enough for it to be 
released and what caveats, if any, are needed to inform users of any 
shortcomings that could affect whether and how the data are used. The 
development and use of comprehensive data quality review standards--if 
they are well documented, transparent, clearly defined, and 
consistently applied--help statistical agencies make such decisions and 
communicate the results of the decisions to the public.

After the reliability of certain publicly released data from the 2000 
Census was called into question, concerns were raised about the 
adequacy of the Bureau's data quality review standards. Chief among 
these concerns was that the Bureau did not routinely and consistently 
include an adequate discussion of limitations to the data it 
disseminates or provide information on how it reaches its dissemination 
decisions.

At your request, we reviewed the Bureau's data quality review 
standards.[Footnote 1] Specifically, as discussed with your offices, we 
(1) examined the review standards that the Bureau had in place to guide 
decisions to disseminate 2000 Census data, (2) determined if the Bureau 
has subsequently developed additional review standards to guide 
decisions about data quality, and (3) assessed whether any such 
standards are likely to address for the 2010 Census the data quality 
review concerns raised after the release of certain data from the 2000 
Census.

To meet these objectives, we interviewed Bureau officials, reviewed 
relevant documents prepared both before and after the enactment of the 
Information Quality Act of 2000, and examined the guidelines other 
statistical agencies and organizations have developed governing the 
public dissemination of data. We did our audit work in Washington, 
D.C., and at the Bureau's headquarters in Suitland, Maryland, from 
August 2003 through October 2004, in accordance with generally accepted 
government auditing standards.

Results in Brief: 

The Bureau did not have detailed agencywide standards for reviewing 
data from the 2000 Census to determine if the data were of sufficient 
quality for public dissemination. Instead, analysts and managers within 
the different parts of the Bureau primarily used their own judgment and 
unwritten, program-specific practices to decide when and whether data 
should be released and what supporting information about data 
limitations, if any, should accompany them. This led to (1) the 
dissemination of data with uncorrected and undisclosed quality 
problems, (2) inconsistent decisions on disseminating data with similar 
quality problems, and (3) inadequate communication to users about the 
reasons for dissemination decisions. As a result, some users of data 
from the 2000 Census lost confidence in the quality of the data and in 
the Bureau's quality review procedures.

In the 4 years since the 2000 Census, the Bureau has publicly issued 
information quality guidelines that contain general quality goals and 
principles and one new standard that allows individuals to request 
correction of errors in data disseminated by the Bureau. Both of these 
initiatives came as a result of the enactment of the Information 
Quality Act in 2000[Footnote 2] and the subsequent guidelines issued by 
the Office of Management and Budget (OMB) in 2002.[Footnote 3] However, 
except for the one standard, the Bureau did not provide specific 
guidelines or procedures on the implementation of the general 
principles articulated in the information quality guidelines.

Since the 2000 Census, the Bureau has also initiated work on several 
other standards and guidelines on the quality of data released to the 
public. Some have been approved for internal use but have not yet been 
made publicly available. For example, one such standard specifies 
minimal information that must accompany any report of Bureau data. 
Additionally, the Bureau has identified several other initiatives on 
data quality review standards, which are in earlier stages of 
development. For example, the Bureau is working on a Bureau-wide 
standard for discussion and presentation of errors in data disseminated 
to the public that will be based on an existing working paper on the 
subject. Bureau officials said that the Bureau plans to make completed 
standards publicly available on its Internet site by the end of 2004.

In response to the recommendations contained in our 2003 reports on 
census counts of Hispanic subgroups[Footnote 4] and the 
homeless,[Footnote 5] the Bureau established an interdirectorate 
working group charged with developing Bureau-wide standards for quality 
in data releases. According to Bureau officials, the working group has 
taken some steps to address the tasks laid out in its charter. However, 
the Bureau has not provided information on the scope or the time frame 
for developing these standards.

The standards that the Bureau has under development and activities of 
the working group are steps in the right direction. However, the Bureau 
needs to accelerate its efforts to develop and implement quality 
standards for data it disseminates. Until spring 2004, no additional 
resources were provided to support the work of the group, and over a 
year after it began, the group has not issued any new standards or 
guidelines, nor indicated when it will be ready to do so. Although 
Bureau officials said that 2010 Census dissemination decisions would 
adhere to new Bureau dissemination guidelines, the actions the Bureau 
has taken to date are not enough to ensure that it will avoid in 2010 
the types of problems encountered in disseminating data from the 2000 
Census. Also, because the Bureau is distributing data from the American 
Community Survey (ACS),[Footnote 6] development of needed standards 
should not wait until 2010.

The development and implementation of a comprehensive, Bureau-wide data 
quality framework, with interrelated standards, and specific procedures 
will help ensure (1) the consistency of decisions about the quality of 
data from the next decennial census, the ACS, and other surveys and (2) 
the conditions under which the data will be disseminated. Thus, the 
benefits the Bureau can achieve by implementing comprehensive data 
quality review standards will not be limited to the decennial census. 
Because the standards will apply to all of the data publicly 
disseminated by the Bureau, the standards should be developed promptly 
and implemented across the Bureau.

Therefore, we recommend that the Secretary of Commerce direct the 
Director of the U.S. Census Bureau to (1) accelerate the Bureau's 
effort to establish comprehensive data quality standards and (2) 
include the implementation of data quality review standards in the 
Bureau's plans for the 2010 Census.

The Secretary of Commerce provided written comments on a draft of this 
report (see app. I). Commerce agreed with our recommendation that the 
Bureau include the implementation of data quality review standards in 
its plans for the 2010 Census, and said that the quality review 
standards will be used for the 2010 Census and for all applicable 
Bureau programs, including the ACS. However, Commerce did not agree 
with our recommendation that the Bureau accelerate its effort to 
establish comprehensive data quality standards. Commerce maintained 
that the Bureau has already completed much of the work of establishing 
comprehensive data quality standards and will continue to develop new 
standards where needed. While these are important steps, most of these 
standards are not available to the public, and the Bureau still lacks 
well-documented, transparent, clearly defined quality review 
guidelines and standards. Thus, we stand by our recommendation and urge 
the Bureau to accelerate its pace in completing the development of 
these standards and effectively implementing them.

Background: 

The Bureau is best known for counting the nation's population every 
10 years. In the future, the Bureau intends to collect much of the data 
that have traditionally been collected during the decennial census from 
the long-form questionnaire with the annual ACS. Beyond the decennial 
census, the Bureau also conducts numerous other surveys and censuses 
that measure changing individual and household demographics and the 
economic condition of the nation. Lawmakers and agency officials at the 
federal, state, and local levels rely on these data when they make 
decisions in a wide range of policy areas. Private-sector decision 
makers also use census data to guide their business plans.

Because of the critical and varied uses of census information, it is 
important that the Bureau's published data meet minimum quality 
standards. In addition, when the data are made public, it is equally 
important for the Bureau to disclose what has been done to ensure the 
quality of the data and identify any limitations so that potential 
consumers can decide whether the data are appropriate for a particular 
use.

Some degree of error in the census (and in virtually any survey) is 
inevitable because of limitations in enumeration, processing, and 
dissemination methods and errors in responses and imputation of data 
for nonresponses. Given the size and diversity of the U.S. population, 
the effort to count the entire population and provide detailed 
demographic characteristics every 10 years is one of the most complex 
of all government operations. The Bureau devotes significant resources 
to minimizing error and improving the quality of the decennial census.

Data quality standards and standardized quality control procedures can 
provide a consistent basis for making data dissemination decisions and 
informing the public about the quality of the data made available to 
it. In 2000, Congress passed what is now known as the Information 
Quality Act. This legislation directed OMB to issue governmentwide 
guidelines that "provide policy and procedural guidance to Federal 
agencies for ensuring and maximizing the quality, objectivity, utility, 
and integrity of information (including statistical information) 
disseminated by Federal agencies." The legislation also required each 
agency to issue its own implementing guidelines that include 
administrative mechanisms allowing affected persons to correct 
information maintained and disseminated by the agency.

The OMB guidelines,[Footnote 7] issued in final form in February 2002, 
define quality as encompassing utility, objectivity, and integrity. The 
guidelines require agencies to issue their own implementing guidelines 
by October 1, 2002. Additionally, they mandate that agencies adopt a 
standard of quality as a performance goal and act to incorporate data 
quality criteria into their data dissemination practices. The 
guidelines also require agencies to develop processes for reviewing the 
quality of data before they are disseminated.

Although OMB had some general guidance for survey processes prior to 
the enactment of the Information Quality Act, other than requirements 
for the evaluation of selected monthly and quarterly economic 
indicators,[Footnote 8] there were no governmentwide requirements 
relating to the quality of data disseminated by the federal agencies. 
Some statistical agencies within the United States developed their own 
extensive guidelines and standards that apply to data disseminated to 
the public. In July 2001, OMB identified the statistical agencies 
within the Departments of Education and Energy, the National Center for 
Education Statistics (NCES) and the Energy Information Administration, 
as good examples of agencies that have developed specific guidelines to 
implement their broad principles and diverse professional standards.
[Footnote 9]

Statistical agencies in other countries have also developed good 
examples of comprehensive guidelines for ensuring the quality of data 
disseminated to the public. Since 1985, Statistics Canada, the central 
statistical agency of the Canadian government, has published quality 
guidelines for its statistical activities. Subsequently, it added 
guidelines on quality assurance processes and management context and 
developed a policy and standards on informing users about data quality. 
More recently, the European Union recognized the importance of 
comprehensive, well-documented guidelines and standards to support its 
task of developing high-quality, comparable statistics from member 
countries. All members of the European Statistical System 
(ESS)[Footnote 10] have signed a quality declaration and approved 22 
recommendations for quality for future work within the system.[Footnote 
11]

Scope and Methodology: 

To address our first question on the standards that the Bureau had in 
place to guide its data dissemination decisions, we interviewed census 
officials, reviewed relevant agency documents, talked to data users, 
and reviewed various complaints about the quality of 2000 Census data. 
We built on our prior reports about the quality of data from the 2000 
Census on Hispanic subgroups[Footnote 12] and the homeless[Footnote 13] 
and the Bureau's decision-making processes for its decisions on whether 
to release those data. We also reviewed other GAO reports addressing 
aspects of the Bureau's procedures for assessing the quality of 
disseminated data.[Footnote 14] From these reports, we identified 
examples of several types of problems the Bureau encountered with 2000 
Census data, which might have been alleviated if the Bureau had 
implemented data quality standards and procedures. Our examples of data 
quality problems are not comprehensive, but illustrative.

To determine whether the Bureau has since developed Bureau-wide data 
quality standards, and, if so, whether they would likely address for 
the 2010 Census the data quality problems raised after the 2000 Census, 
we interviewed census officials responsible for developing agencywide 
standards, examined documents related to the development of new 
standards on data quality review, and reviewed the agency's Internet 
site for information on data quality review standards available to the 
public. We also reviewed OMB guidelines on the quality of data 
disseminated by federal agencies as well as the action taken by the 
Department of Commerce and the Bureau in response to the guidelines. We 
attended meetings of the Secretary of Commerce's Decennial Census 
Advisory Committee, the National Academy of Science Panel on Research 
on Future Census Methods, and the Washington Statistical Society's 
conference on Quality Assurance in the Government, all of which 
examined issues related to the quality of the data disseminated by the 
Bureau. We also discussed information quality standards and guidelines 
with officials in Eurostat, the statistical directorate of the European 
Union.

Additionally, we considered how the Bureau's actions in developing 
dissemination guidelines could improve the quality of data disseminated 
after the 2010 Census and for other Bureau data collection programs, 
such as the ACS that among other things, is intended to replace the 
long-form census questionnaire. To benchmark the Bureau's progress in 
developing data quality review standards with that of other statistical 
agencies, we also reviewed documents from entities that have developed 
standards for the quality of data disseminated to the public, including 
NCES; Statistics Canada, the central statistical agency of Canada; and 
ESS. However, we did not evaluate the implementation or effectiveness 
of these guidelines and standards or their specific applicability to 
the Bureau.

Our work addressed only standards and guidelines on data quality 
review. Although OMB's information quality guidelines and the Bureau's 
guidelines and performance principals cover all the key steps in data 
collection, analysis, and dissemination, we did not look at the 
Bureau's guidelines or standards for ensuring quality during the 
planning and data collection stages. Instead, as requested, we looked 
at Bureau guidance on steps taken after data collection, that is, 
guidance related to processing data, assessing their quality, and 
making them available to the public. We looked for documents spelling 
out standards, guidelines, procedures, and other criteria to guide 
decisions about identifying and correcting errors, determining if and 
when to release data, and revising data after release.

Our audit work was conducted in Washington, D.C., and at the Bureau's 
headquarters in Suitland, Maryland, from August 2003 through October 
2004. Our work was done in accordance with generally accepted 
government auditing standards.

We requested comments on a draft of this report from the Secretary of 
Commerce. On September 27, 2004, the Secretary provided written 
comments on the draft. The comments are reprinted in appendix I.

Professional Judgment Drove Data Dissemination Decisions: 

The Bureau had no agencywide standards or guidelines in place to guide 
decisions about disseminating data from the 2000 Census. Instead of 
agencywide, written guidance, professionals within the different parts 
of the Bureau primarily used their judgment and program-specific 
practices to decide when and whether data should be released and what 
supporting information, if any, should accompany them. This led to 
instances when (1) data were released with uncorrected and undisclosed 
quality problems, (2) inconsistent decisions were made on whether to 
release data sets with similar quality problems, and (3) the reasons 
for certain data dissemination decisions were inadequately 
communicated.

The Bureau Lacked Agencywide, Written Standards and Guidelines on the 
Quality of Census Data Disseminated to the Public: 

At the time the Bureau was making decisions about disseminating data 
from the 2000 Census, it did not have written, agencywide guidelines or 
standards to help inform its decisions on whether the data were of 
sufficient quality to be released. Although Bureau officials emphasized 
that the Bureau has a long tradition of high standards and procedures 
that yield quality data, they acknowledged that these practices were 
primarily part of the agency's institutional knowledge. According to 
one official, key individuals in each program area, relying primarily 
on professional judgment, determined whether the quality of the data 
was acceptable for release to the public. The official explained that 
the program areas develop their own guidance and procedures for 
ensuring data quality. Sometimes their guidance and procedures were 
written, but more often they were not. Further, the Bureau had no 
central inventory or repository of the guidance and practices of the 
different divisions.

Lack of Data Quality Review Guidelines Led to Inadequate Analysis of 
Potential Errors and Release of Data without Adequate Disclosure: 

As noted earlier, decennial census data are used to apportion and 
redistrict Congress. As release of data for each of these purposes is 
required by statute, they are known collectively as "public law" data. 
The Bureau had a number of quality assurance programs and procedures 
for assessing the accuracy of, and correcting errors in, public law and 
other data prior to their release. However, the lack of standard 
procedures and guidelines for dealing with quality problems contributed 
to lost opportunities to correct errors in the count of the population 
identified before the data were disseminated.

One such quality assurance program we reviewed was known as Demographic 
Full Count Review, in which analysts were to identify, investigate, and 
document suspected data discrepancies or "issues" in order to clear 
census data files and products for subsequent processing or public 
release.[Footnote 15] The Bureau contracted out some of the analysts' 
work because it lacked sufficient staff to conduct the Full Count 
Review on its own. Bureau reviewers were to determine whether and how 
to correct the data by weighing quality improvements against time and 
budget constraints. Analysts identified 4,809 possible discrepancies, 
such as instances when the location, population count, demographic 
characteristics, or a combination of these for housing units and group 
living facilities differed from what analysts expected. According to 
Bureau officials, only 5 of the 4,809 issues were investigated and 
corrected prior to the release of the public law data. All five 
involved group living facilities the Bureau calls "group quarters" for 
which the Bureau had the correct population counts, but placed the 
living facilities in the wrong places. The Bureau did not investigate 
most of the remaining issues prior to the release of the data in large 
part because they were insufficiently documented and the Bureau lacked 
the time and people to further investigate these issues. Subsequently, 
according to Bureau officials, the remaining issues that contained 
sufficient documentation were investigated as a part of the Count 
Question Resolution program, which ended in September 2003.

As we noted in our July 2002 report, the fact that public law data were 
released with over 4,800 unresolved data issues of unknown validity, 
magnitude, and impact is cause for concern. To the extent these 
unresolved discrepancies were in fact true errors in the population 
count or geography, they could have affected the drawing of 
congressional districts as well as other purposes for which census data 
are used.

The existence of data quality review guidelines could have helped the 
Bureau in this situation. For example, we found that the Bureau's lack 
of clearly defined requirements for documenting data issues resulted in 
a significant number of cases with inadequate documentation that the 
Bureau could not use to resolve the issues. Additionally, the Bureau 
had no mechanism for setting priorities for resolving these potential 
data errors. A sufficient set of guidelines could have helped the 
Bureau to ensure that the documentation of potential errors was 
adequate for decision making and to maximize the use of scarce 
resources in addressing the various data issues, giving top priority to 
investigating discrepancies likely to have the most adverse affect on 
the data.

The quality of certain data from the census long-form questionnaire 
have been called into question as well. In its 2004 comprehensive 
review of the 2000 Census, a panel of the National Research Council of 
the National Academy of Sciences assessed the quality of the long-form 
data using various benchmarks, and found that the overall quality of 
the information was less than that of the short-form questionnaire and 
had deteriorated since the 1990 Census.[Footnote 16] For example, at 
least 32 percent of the respondents failed to provide information on 
their property taxes, and 30 percent did not respond to all or some of 
the questions relating to income (compared with 12 and 13 percent, 
respectively, in 1990). Additionally, the panel noted that the Bureau 
did not measure and report the impact of some of the steps it took to 
address problems with missing data and recommended that the Bureau 
develop such measures and inform users about the need for caution in 
analyzing and interpreting these data.

Even more significant quality problems plagued the data for residents 
of group quarters. The panel found these data to be poor in comparison 
with the data for household residents, and also in comparison with data 
for group quarters from the 1990 Census. In 2000, missing data rates 
for some items were over 25 percent--one item was over 50 percent--for 
all residents of group quarters, and as high as 75 percent for prison 
inmates. Given the prevalence of missing data from residents of group 
quarters, the panel questioned whether the Bureau should have published 
these data at all for some or all types of group quarters.

The Bureau Made Conflicting Dissemination Decisions on Data with 
Similar Quality Problems: 

Our earlier work on Hispanic subgroups and the homeless showed that the 
Bureau's approach to data quality review led to inconsistent decision 
making. Faced with similar quality problems in data from the 2000 
Census, Bureau officials made different decisions about disseminating 
data and did not explain the reasons for their decisions.

For example, in an effort to improve the count of Hispanics and 
simplify the questionnaire, the Bureau redesigned its 2000 Census 
question on Hispanic origin and dropped a list of examples of Hispanic 
subgroups included in the 1990 Census. In May 2001, the Bureau released 
data on Hispanics and Hispanic subgroups as part of its first release 
summarizing the results of the 2000 Census. The Bureau also published 
The Hispanic Population, a 2000 Census brief that provided an overview 
of the size and distribution of the Hispanic population in 2000 and 
highlighted changes in the population since the 1990 Census. For the 
first time, the Bureau released data on Hispanic subgroups as a part of 
its release of the Full Count Review data even though it had not fully 
tested the impact of questionnaire changes on the subgroup data and 
provided little discussion of the potential limitations of the data.

Shortly after the Hispanic and Hispanic subgroup data from the 2000 
Census were released to the public, questions were raised about the 
counts for specific Hispanic subgroups. For example, the reported count 
of Dominican Hispanics was significantly lower than counts reported in 
other Bureau surveys. Representatives of affected Hispanic subgroups 
asked for an investigation and explanation of why the Bureau reported 
data that these subgroups considered to be of questionable quality. We 
found that a key factor behind the Bureau's release of apparently less-
than-accurate Hispanic subgroup data appeared to be a lack of adequate 
guidelines governing decisions on quality considerations that should be 
addressed before making data publicly available.[Footnote 17]

In contrast, the Bureau, citing quality problems, decided not to 
separately report certain information on people without conventional 
housing, including those commonly referred to as "homeless." 
Enumerating this segment of the population has been an ongoing 
challenge for the Bureau. To help locate and count these people in 
2000, the Bureau partnered with organizations providing services to the 
homeless and with local governments, some of which put considerable 
resources into the effort. When the Bureau decided not to separately 
report the number of people in transitional and emergency shelters as 
originally planned because of data quality problems, some of the 
organizations and local governments, which had expected to use the data 
for directing services to the homeless, questioned the Bureau's process 
for making that decision. Additionally, we found that the decision 
about when and whether to release data on people in emergency and 
transitional shelters changed several times. Decisions about the 
release of data with identified quality problems were not well 
documented and communicated with some Bureau partners and other 
stakeholders.[Footnote 18]

As a result, outside parties interested in both the Hispanic and 
homeless data from the 2000 Census questioned the quality of the data, 
the procedures the Bureau used to determine what data to release, and 
the value of their own participation in helping the Bureau prepare for 
the 2000 Census. Because the Bureau's reasons for data release 
decisions were not obvious, and it had no guidelines or standards that 
spelled out criteria for decisions, the Bureau left itself open to 
questions about the objectivity of its decisions and risked loss of 
public confidence.

In our reports on Hispanic and homeless Census 2000 data, we 
recommended that the Bureau (1) develop agencywide guidelines for its 
decisions on the level of quality needed to release data to the public, 
how to characterize any limitations in the data, and when it is 
acceptable not to release data and (2) ensure that these guidelines are 
documented, transparent, clearly defined, and consistently applied. We 
also recommended that the Bureau ensure that its plans for releasing 
data are clearly and consistently communicated to the public. The 
Bureau agreed with each of our recommendations and asked its 
Methodology and Standards Council[Footnote 19] to review existing 
statistical and quality guidelines, bring them together in one place, 
and develop data quality standards. We discuss the Bureau's actions 
later in this report.

The Bureau Has Made Limited Progress in Publicly Issuing New Standards 
on the Quality of Data Disseminated to the Public since the 2000 
Census: 

Since the first results of the 2000 Census were released, the Bureau 
has publicly issued a set of information quality guidelines and one new 
standard on the quality of data disseminated to the public. As required 
by the Information Quality Act and the OMB guidelines, the Department 
of Commerce[Footnote 20] and the Bureau published Information Quality 
Guidelines, but the guidelines contain only general quality goals and 
principles and do not provide any specific guidelines or procedures on 
the implementation of the general principles. Also as required by the 
Information Quality Act and the OMB guidelines, the Bureau published a 
standard that described a procedure allowing individuals to seek 
correction of certain errors in data disseminated by the Bureau. 
Additionally, the Bureau has begun developing several other standards 
on the quality of data disseminated to the public, but none have been 
publicly released in final form.

In March 2003, in response to our recommendations, the Bureau 
established an interdirectorate working group charged with the broad 
mandate of developing Bureau-wide standards for quality in data 
releases. The working group has taken some steps to address the tasks 
laid out in its charter. However, the Bureau has not provided 
information on the scope or the time frame for developing these 
standards.

The Bureau Has Taken Steps to Expand Its Guidance on Data Quality 
Review: 

Recognizing the paucity of Bureau-wide written standards on the quality 
of data disseminated to the public, the Bureau established a Quality 
Program in 1999 to develop consistent processes for producing quality 
products across the Bureau. The Bureau's Associate Director for 
Methodology and Standards with input from chiefs in a number of 
divisions compiled an inventory of data quality review documents used 
in different divisions[Footnote 21] and developed a Bureau-wide quality 
framework. The resulting quality framework was adopted to serve as a 
vehicle through which "the demographic, economic, and decennial areas 
can share and support common principles, standards, and guidelines." 
This framework provides the organization for documents in the intranet 
portal known as the Quality Management Repository (QMR). Additionally, 
the Bureau's description of the quality framework spells out the 
process for developing, reviewing, and approving quality framework 
documents. The document describing the quality framework and most of 
the documents in the QMR are internal documents not available to the 
public through the agency's Internet site. However, Bureau officials 
indicated that they intend to make some of the standards available 
through the Internet later in calendar year 2004.

The Bureau has publicly issued two data quality review documents and 
made them available through the Internet. In October 2002, in response 
to the requirements of the OMB guidelines, the Bureau published a set 
of information quality guidelines in eight performance areas, including 
the establishment of review procedures. The Bureau's guidelines 
identify broad quality goals and principles, but do not provide 
specific guidance to ensure consistent decisions. For example, the 
guideline on predissemination review of data says that "all documents 
released by the Census Bureau undergo extensive review that encompasses 
the content, statistical and survey methodology, and policy 
implications of the document," and that this review "ensures that the 
data and text of the document meet Census Bureau standards for quality" 
or the Bureau reserves the right to withhold the data from the public. 
However, the guideline does not indicate what the Bureau "standards for 
quality" are, how the Bureau will know if the data meet the standards, 
or who within the Bureau is responsible for the review.

The second document issued and made available on the Bureau's Web site 
is Census Bureau Standard: Correcting Information That Does Not Comply 
with Census Bureau Section 515 Information Quality Guidelines in March 
2002. This standard was also issued in response to the specific 
requirements of the Information Quality Act and the OMB guidelines that 
agencies provide procedures for correcting certain errors identified in 
data they disseminated and post these guidelines on their Web sites. 
The standard established procedures that allow individuals to request a 
correction of information they believe is erroneous and the Bureau to 
review the evidence and determine whether a correction is warranted.

The Bureau has also approved several additional Bureau-wide data 
quality review documents for implementation and internal distribution 
through the QMR on its intranet. On March 18, 2003, the Bureau issued 
Census Bureau Standard: Minimal Information to Accompany Any Report of 
Census Bureau Data for a 6-month trial period. The standard identifies 
13 specific items that the Bureau should report for every survey or 
census and specifies who is responsible for ensuring adherence to the 
standard. An accompanying memorandum from the Associate Director for 
Methodology and Standards to program associate directors said that 
implementation issues would be documented during the trial period and 
appropriate changes made prior to the final release of the standard. 
Even though the trial period is over, the Bureau has not made such 
changes or publicly issued the standard in final form. However, the 
standard is still in effect on a trial basis, according to one Bureau 
official.

The Bureau also released its Census Bureau Guideline: Quality Profiles 
on March 9, 2004, through the QMR. The document outlines a standardized 
quality profile, recommended for all recurring surveys and certain 
other programs, which is intended to present a consistent set of 
information on the quality of each program. As a guideline rather than 
a standard, this guidance is recommended rather than mandatory.

In addition, the Bureau has also initiated work on several proposals 
for additional standards. For example, a standard for discussion and 
presentation of errors in data disseminated to the public is under 
development. This standard is based on a technical paper that was 
issued in 1974 and revised in 1987. The Bureau said it would be issued 
in the near future, but has not provided a specific date.

Bureau Working Group Has Begun Developing Additional Standards on Data 
Quality Review, but None Have Been Issued: 

In response to our recommendations from reports on both homeless and 
Hispanic subgroup data from the 2000 Census, the Bureau established an 
interdirectorate working group on March 3, 2003, with the broad mandate 
to develop Bureau-wide standards for quality in data releases. However, 
the working group has not yet issued any draft or final standards or 
developed a time frame for doing so.

The working group is composed primarily of assistant division chiefs 
from the program areas--decennial, demographic, and economic. An 
assistant division chief from the Demographic Statistical Methods 
Division chairs the group.

According to the working group's charter, its mission is to: 

* "Document current Census Bureau data review procedures,

* "Benchmark Census Bureau review procedures with that of other 
agencies,

* "Document Census Bureau situations where review of data indicates 
data does not meet "quality requirements" and the outcome of those 
situations,

* "Propose standards for quality in Census Bureau data products,

* "Benchmark quality requirements for data release with other agencies,

* "Develop Census Bureau Standard: Quality in Census Bureau Data 
Releases." 

Bureau officials told us that the working group has reviewed the 
published detailed guidelines from NCES and the Canadian national 
statistical office. Benchmarking discussions have taken place with the 
Bureau of Labor Statistics and the National Center for Health 
Statistics. Additionally, the working group met with an official from 
the New Zealand national statistical office to discuss its standards. 
The group is also planning meetings with additional federal agencies. 
These organizations have published detailed guidance on how broad 
principles on data quality are to be put into practice, notably the 
organizational responsibilities and internal control mechanisms for 
applying them.

For example, Statistics Canada, the central statistical agency of the 
Canadian government, has developed an extensive and detailed set of 
quality guidelines that covers the quality of data disseminated to the 
public and the quality control processes that are supposed to be 
applied to ensure the quality of the data.[Footnote 22] In March 2000, 
Statistics Canada published its Policy on Informing Users of Data 
Quality and Methodology, which specifies the organization's 
responsibilities to inform users about the concepts and methodology for 
collecting, processing, and analyzing its data; the accuracy of these 
data; and any other features that affect their quality or fitness for 
use. By detailing mandatory documentation standards, guidelines for 
additional documentation, and examples of mandatory standardized 
summary documentation, the policy enhances the likelihood of consistent 
decision making throughout the organization. Additionally, making this 
information public ensures that any data user can determine what has 
been done to ensure the quality of the data and Statistics Canada's 
reasons for its decisions about release.

NCES has developed detailed standards designed to implement its broader 
policies on dissemination of statistical data. An NCES standard 
includes a section entitled "Establishment of Review Procedures," which 
includes a table showing the required reviews for each type of product 
and an illustration of the key steps in the review and adjudication 
process. As with the Statistics Canada policy, the NCES standard 
provides information on the quality assessments and reviews that data 
must undergo before being released to the public.[Footnote 23]

According to the Bureau's Associate Director for Methodology and 
Standards, the working group is making progress in conducting the work 
laid out in its charter. She said that the working group has reviewed 
different practices in divisions across the Bureau and benchmarked 
these practices against appropriate organizations. It has moved on to 
the task of identifying quality problems that have resulted from data 
quality review practices in different parts of the Bureau and assessing 
what could have been done differently. However, the Bureau did not 
provide any time frame for the working group's activities, information 
on how the Bureau intends to use the benchmarking exercises, or the 
intended scope and content of the Bureau-wide standard on quality in 
Bureau data releases.

The working group's charter indicates that its schedule should reflect 
an expeditious effort to complete its tasks. The Associate Director for 
Methodology and Standards, to whom the working group reports, 
emphasized that setting standards is a long-term process and pointed 
out that the Bureau has never issued a standard in less than a year. 
She noted that participation in the working group is added to the other 
responsibilities of its members and that initially the working group 
had no dedicated staff.[Footnote 24] Additionally, she said that the 
working group does not have a time frame for completing these 
activities.

Greater Commitment to New Standards for Public Dissemination of Data 
Could Help Bureau Avoid Problems in Disseminating 2010 Census and Other 
Data: 

The standards that the Bureau has under development and activities of 
the working group are steps in the right direction. However, the Bureau 
has provided limited indication that developing and implementing 
standards on the quality of data it disseminates is a priority. It has 
no official plans for such an initiative, and these issues are not 
included in the Bureau's plan for the 2010 Census. Until spring 2004, 
no additional resources were provided to support the working group, and 
a year and a half after it began, the group has not developed any new 
standards or guidelines or indicated when it will be ready to do so. 
Although Bureau officials said that 2010 Census dissemination decisions 
would adhere to Bureau dissemination guidelines, the actions the Bureau 
has taken to date are not enough to ensure that it will avoid in 2010 
the types of problems encountered in disseminating data from the 2000 
Census. A publicly issued, comprehensive, Bureau-wide data quality 
framework, with interrelated standards, and specific procedures 
(as evident in NCES, ESS, and Statistics Canada) could help ensure 
consistency of decisions about the quality of data from the next 
decennial census and the conditions under which the data will be 
disseminated. The benefits the Bureau can achieve by implementing data 
quality review standards should not be limited to the decennial census. 
Because the standards could apply to all of the data publicly 
distributed by the Bureau, the sooner they are developed and 
implemented across the Bureau, the sooner the Bureau will begin to reap 
their benefits.

Developing and Implementing Bureau-Wide Data Quality Review Standards 
Are Not Part of Official Bureau Plans: 

As noted above, the Bureau has not provided specific plans for further 
developing Bureau-wide data quality review standards or for 
implementing the broad data quality principles and guidelines outlined 
in its response to the OMB guidelines. It has not spelled out what 
needs to be done, how long it will take, what resources will be 
required, or how performance will be measured.

The Bureau's evolving plans for the 2010 Census devote little attention 
to data quality review issues. As it has for past decennial censuses, 
the Bureau focuses its plans for the 2010 Census on ensuring the 
quality of information collected during the data collection phase, 
rather than on how it will address potential quality problems that 
might be identified before the data are released. Bureau officials told 
us that whatever standards are developed will be applied to 
disseminating data from the 2010 Census. However, they said that the 
next decennial census is still a number of years away, and 
disseminating data from the 2010 Census is still farther in the future.

Data Quality Review Standards Could Also Aid Other Data Programs before 
2010: 

The 2010 Census is to differ significantly from is predecessor. The 
2010 Census, if implemented as planned, will ask the entire population 
to provide only basic information on the short form necessary for 
congressional apportionment. It will no longer collect more extensive 
information on a longer questionnaire from a sample of the population. 
Instead, the Bureau has developed the ACS that among other things, is 
intended to replace the long-form census questionnaire. The detailed 
data on social and economic conditions that were previously collected 
as a part of the decennial census will in the future be collected 
annually in the ACS. In fact, the ACS is a key component of the 
Bureau's plan for a reengineered 2010 Census. The ACS data are being 
collected and released annually for larger geographic areas, and data 
quality review standards could help improve the quality of these data 
immediately.[Footnote 25]

The Bureau has developed several measures of quality for the 
information included in the ACS and began reporting these measures on 
its Web site in December 2003. These reported measures are important 
steps in the right direction for the Bureau, but these program-specific 
measures have not been adopted as Bureau-wide standards for similar 
collections. A Bureau official said that these measures meet the 
requirements for minimum information on data quality of the Bureau's 
standard, which is being piloted. The measures developed for the ACS 
program are being reviewed for possible implementation in other 
household surveys.

Conclusions: 

Fully documented, transparent, clearly defined, and consistently 
applied standards on the quality of data disseminated to the public can 
help ensure that the Bureau makes consistent decisions about how it 
addresses data quality problems. Additionally, such standards can help 
the public understand the Bureau's reasons for its dissemination 
decisions, and can help protect the Bureau from allegations that it was 
inappropriately releasing or suppressing data. Because the cooperation 
and trust of the public is essential to a successful census, the Bureau 
must work to avert any loss of public confidence in the quality of data 
and in the integrity and objectivity of the Bureau.

Taken together, the quality problems that affected certain data from 
the 2000 Census underscore the importance of comprehensive data quality 
review guidelines for ensuring the Bureau makes more uniform decisions 
on data quality review and informs the public of limitations that could 
affect whether and how the data are employed.

The Bureau still has a long way to go in developing standards for the 
release of data to the public that will help avoid in the 2010 Census 
(and the ACS) the types of problems experienced in 2000. Additionally, 
since the standards would apply to all Bureau data collections, delay 
in their development and implementation means the Bureau is missing an 
opportunity for improving the quality of the other data it collects and 
disseminates. To avoid the problems it had with the dissemination of 
2000 Census data the Bureau should place greater emphasis on developing 
and implementing data quality standards.

Although the Bureau has established a program for addressing standards 
development, we identified the following causes of concern.

* In the absence of more detailed information about the activities and 
schedule of the working group, it is difficult to assess the Bureau's 
progress in developing these standards. Over a year and a half after 
establishing the working group, the Bureau has publicly issued no new 
standards and has not publicly released plans that provide information 
on its schedule and agenda for developing the standards. Also the 
Bureau has not publicly sought comments on the working group's 
initiatives through its advisory committees.

* Plans for the 2010 Census do not address procedures for dealing with 
data quality problems that are identified during the data quality 
review phase.

* The Bureau has not publicly announced any comprehensive plans for 
developing and implementing written, Bureau-wide quality standards and 
quality control processes.

A number of statistical agencies in the United States and elsewhere 
have developed comprehensive data quality review standards and quality 
control procedures that could serve as models for the Bureau. A Bureau-
wide set of quality standards on data disseminated to the public 
covering both the quality of the data and quality control procedures 
would apply not only to the decennial census, but also to all other 
data collected by the Bureau and released to the public. Such standards 
could help the Bureau avoid some of the problems it experienced in 
disseminating data from the 2000 Census. Much of the data that were 
previously collected during the decennial census are now being 
collected under the ACS. Because these data are collected and released 
annually, the ACS, or other annual household surveys, could serve as a 
test for proposed standards.

Recommendations for Executive Action: 

To ensure that the 2010 Census, the ACS, and other Census data products 
will provide public data users with more complete, accurate, and useful 
information, we recommend that the Secretary of Commerce direct the 
Director of the U.S. Census Bureau to take the following two actions: 

1. Accelerate the Bureau's effort to establish comprehensive data 
quality standards by developing and making public a detailed plan, 
including interim milestones, for developing such standards and 
procedures.

2. Include the implementation of the data quality review standards in 
the Bureau's plans for the 2010 Census, and test new draft guidelines 
on data quality review using the annual ACS test program and other 
surveys.

Agency Comments and Our Evaluation: 

The Secretary of Commerce provided us with written comments on a draft 
of this report on September 27, 2004, which are reprinted in appendix 
I. Commerce agreed with one of our two recommendations-namely, to 
establish data quality review standards as part of its plans for the 
2010 Census, and as indicated in the Secretary's letter, the Bureau is 
taking steps to implement it. However, Commerce disagreed with our 
first recommendation that the Bureau accelerate its effort to establish 
comprehensive data quality standards. Commerce also identified some 
specific issues and suggested changes to provide additional context and 
clarification and in some cases technical corrections. We made these 
changes and corrections to the text as appropriate, but believe our 
first recommendation still applies.

Commerce took exception to our characterization of the amount of work 
that the Bureau has completed in developing comprehensive data quality 
review standards and in developing a specific standard for decisions on 
data release. However, the activities and documents Commerce cited to 
demonstrate the Bureau's progress were mentioned in our draft report. 
For example, Commerce noted that the Bureau had developed a quality 
framework for Bureau documents, inventoried quality guidance used in 
specific program areas, and created an in-house repository of such 
documents. Commerce also pointed to the quality principles the Bureau 
developed and included as a part of its Information Quality Guidelines 
issued in response to OMB requirements. Our draft report credited the 
Bureau with all of these activities, although not always at the same 
level of detail as Commerce described in its comments.

Moreover, while these are important steps, most of this work is not 
available to the public. As we observed in our draft report, the only 
documents that have been made public on the agency's Internet site are 
the documents required by the Information Quality Act and the related 
OMB guidelines: (1) the Bureau's Information Quality Guidelines and (2) 
the standard allowing individuals to seek correction of certain errors 
in data disseminated by the Bureau.

Indeed, our primary concern is not with how much work has been done but 
whether the Bureau has well-documented, transparent, clearly defined 
quality review guidelines and standards, and whether the pace of its 
efforts is sufficient. As yet, the Bureau has not produced such 
guidelines nor has it documented plans for completing this work. Bureau 
officials said they will make existing standards available to the 
public on the agency's Internet site by the end of 2004, but have not 
indicated which standards will be included.

Therefore, we reaffirm our recommendation that the Bureau should 
accelerate its efforts to establish such data quality review standards 
by making public a detailed plan, including interim milestones, for 
developing such standards and procedures. Such a plan can assist the 
Bureau in prioritizing its work and addressing the resource constraints 
that will inevitably be present. If, as Commerce maintained, much of 
the work has already been completed, implementing the recommendation 
should not be unduly burdensome or time consuming. While we commend the 
Bureau for agreeing with our recommendation to implement data review 
guidelines and standards for the 2010 Census and the ACS, we believe it 
needs to accelerate its efforts to complete, make public, and fully 
implement these data review standards. The more time that elapses, the 
greater the risk of releasing data with quality problems.

As agreed with your offices, unless you release its contents earlier, 
we plan no further distribution of this report until 30 days from its 
issue date. At that time, we will send copies of this report to the 
Chairman of the House Committee on Government Reform, the Secretary of 
Commerce, and the Director of the U.S. Census Bureau. Copies will be 
made available to others on request. This report will also be available 
at no charge on GAO's home page at [Hyperlink, http://www.gao.gov].

Please contact me on (202) 512-6806 or by e-mail at 
[Hyperlink, daltonp@gao.gov] if you have any questions. Other key 
contributors to this report were Robert Goldenkoff, Elizabeth Powell, 
Robert Parker, Michael Volpe, and Andrea Levine.

Signed by: 

Patricia A. Dalton: 
Director, Strategic Issues: 

[End of section]

Appendixes: 

Appendix I: Comments from the Department of Commerce: 

THE SECRETARY OF COMMERCE: 
Washington, D.C. 20230:

September 27, 2004:

Ms. Patricia A. Dalton: 
Director, Strategic Issues: 
U.S. Government Accountability Office: 
Washington, DC 20548:

Dear Ms. Dalton:

The U.S. Department of Commerce appreciates the opportunity to comment 
on the U.S. Government Accountability Office draft report entitled Data 
Quality: Census Bureau Needs to Accelerate Efforts to Develop and 
Implement Data Quality Review Standards. The Department's comments on 
this report are enclosed.

Sincerely,

Signed by: 

Donald L. Evans: 

Enclosure:

Comments from the U.S. Department of Commerce, 
Regarding the U.S. Government Accountability Office Draft Report 
Entitled Data Quality: Census Bureau Needs to Accelerate Efforts to 
Develop and Implement Data Quality Review Standards:

The U.S. Department of Commerce thanks the Government Accountability 
Office (GAO) for the opportunity to review the draft report, Data 
Quality: Census Bureau Needs to Accelerate Efforts to Develop and 
Implement Data Quality Review Standards (GAO-04-469). This report 
discusses an important issue of concern to the Census Bureau-ensuring 
the quality of Census Bureau data releases.

The focus of the GAO report alternates between discussing what the 
Census Bureau is doing to establish comprehensive data quality 
standards for its products, develop data quality review standards, and 
develop guidelines for decisions on the level of quality needed to 
release data to the public. This report considerably expands the scope 
of two previous GAO reports on the quality of the Census' Bureau data 
products that were issued in January 2003. These reports are Methods 
for Collecting and Reporting Data on Homeless and Others Without 
Conventional Housing Need Refinement and Methods for Collecting and 
Reporting Hispanic Subgroup Data Need Refinement. In both of these 
reports, the GAO recommendation in this area was directed at the more 
limited area of data quality review. The reports recommended that:

"The Secretary of Commerce should direct the Bureau to ... (2) develop 
guidelines for decisions on the level of quality needed to release data 
to the public, how to characterize any limitations, and when it is 
acceptable to suppress data; . . ."

The GAO approach in these three reports points to the multifaceted 
aspect of producing and releasing high-quality data to the public. 
Developing comprehensive standards on overall data quality is an 
important aspect of responding to this issue.

The Census Bureau has made substantial progress in developing 
comprehensive standards and developing a specific standard for 
decisions on data release. The significant amount of work that has been 
completed in this area is not reflected in the GAO report. Both of 
these efforts have the same objectives: to gain consensus within the 
Census Bureau on a specific standard, to document the standard, to 
implement the standard consistently across Census Bureau products, and 
to inform Census Bureau data users of the quality procedures that are 
incorporated into Census Bureau products.

The Census Bureau established a Quality Program in 1999 to develop 
consistent processes for producing quality products across the Census 
Bureau. This program encompassed building quality into Census Bureau 
processes, developing training on quality procedures for survey 
operations, documenting current practices that lead to quality 
products, developing best practices for survey procedures, and 
promoting communication on quality procedures. An initial effort 
produced a 30-year inventory of quality guidance issued by Census 
Bureau directorates.

The Census Bureau developed a Quality Framework for Census Bureau 
documents-Principles, Standards, Guidelines-modeled after the 
Statistics Canada Quality Guidelines. A portal was created as an in-
house repository for the documents to provide accessibility to staff 
within the Census Bureau. Working groups were chartered to develop 
documents on an as-needed basis. In the spring of 2002, the Census 
Bureau developed a set of nine Quality Principles as a part of its 
Section 515 Information Quality Guidelines. These Quality Principles 
reference the Census Bureau Quality Framework and are consistent with 
those established by the federal statistical agencies in their Federal 
Register Notice (67 FR 38267-38470 - June 4, 2002), stating that the 
issuing agencies all had existing principles ensuring the quality of 
information disseminated to the general public. The Census Bureau's 
Section 515 Information Quality Guidelines provides a comprehensive 
statement of its approach to producing quality data. To date, the 
Census Bureau has had no information quality complaints.

These same nine principles are part of the Census Bureau Quality 
Framework. Beginning in 2001, documents were issued into the Census 
Bureau Quality Framework. Now there are nine Principles, seven 
Standards, eight Guidelines, and over one hundred Current Practices. 
The documents in the Quality Framework represent either updated 
documentation of a standard or guideline that was previously issued by 
some Census Bureau organization so that it relates to the entire Census 
Bureau (e.g. Census Bureau Standard: Pretesting Questionnaires and 
Related Materials for Surveys and Censuses; Census Bureau Guideline: 
Quality Profiles), or a standard or guideline for anew area of concern 
(e.g. Census Bureau Standard: Minimal Information to Accompany any 
Report of Census Bureau Data; Census Bureau Guideline: Language 
Translation of Data Collection Instruments and Supporting Materials). 
The current totality of Census Bureau principles, standards, and 
guidelines addresses many, but not all, procedures that might be put in 
place to ensure quality in its products and processes. Components of 
these quality procedures relate to data review. Implementation of these 
Quality Framework documents will prevent many of the specific quality 
issues noted concerning Census 2000 data.

The Census Bureau directed its Methodology and Standards Council to 
address agency-wide issues of quality. Each Quality Framework: standard 
has been developed by working groups chartered by the Methodology and 
Standards Council and composed of individuals within the organization 
who have expertise in the specific topic. All the affected operating 
units in the Census Bureau review the standard and comment on it, 
helping to assure both an appropriate and comprehensive standard and 
acceptance by the operating units. Once the Census Bureau Methodology 
and Standards Council approves the standard, the program associate 
directors are asked to concur. The program associate directors then 
take responsibility for implementing and enforcing the standard. The 
Quality Program Staff assists the program areas in implementing the 
standard through training and later evaluates the effectiveness of the 
implementation of the standard. The Census Bureau plans to make its 
existing standards available on its Internet site by the end of 2004.

A working group was chartered by the Methodology and Standards Council 
in March 2003 to address concerns raised in the two previous GAO 
reports relating to Census 2000 data releases. The charter directed the 
group to develop an agency-wide documented standard for quality in 
Census Bureau data releases as recommended in the earlier reports and 
is not limited exclusively to decennial data. The group has not been 
asked to develop comprehensive quality standards and guidelines as 
these are being developed within the Quality Framework described above. 
Rather, it is addressing the issues of the quality needed to release 
data, how limitations in data quality should be described, and when it 
is advisable to suppress data.

The group has made substantial progress thus far. It has drafted a 
report documenting current Census Bureau data review procedures; it has 
examined 12 situations where there was a concern with the quality of 
data planned for release and prepared a draft paper documenting those 
situations; it has benchmarked with two external agencies-Bureau of 
Labor Statistics and the National Center for Health Statistics-on data 
review procedures incidental to making data-release decisions. This is 
all preliminary and necessary work to both fulfill the charter of the 
working group and to propose standards to the Census Bureau Methodology 
and Standards Council for implementation in the Census Bureau. The 
working group is meeting weekly and making progress on the tasks in its 
charter. The ultimate task of establishing standards for data release 
decisions is extremely difficult and complex. The group, at this time, 
is actively developing specific standards for Census Bureau approval. 
This work will be completed for incorporation in the 2010 census.

The Census Bureau does not agree with Recommendation No. 1 to 
accelerate the effort to establish comprehensive data quality 
standards. Much of the work required has already been completed in the 
development of the existing principles, standards, and guidelines. The 
Census Bureau Methodology and Standards Council has now addressed all 
the issues identified in the 30-year inventory that it prepared` in 
2000,` by either issuing an updated document in the Quality Framework 
or determining that the original document was no longer relevant. The 
Council will continue to identify new areas where a standard or 
guideline is needed and charter working groups to address those 
concerns, prioritizing the needs that come to their attention. There 
will always be resource constraints for developing a standard in a 
specific area, as there are limits to the number and availability of 
individuals with given expertise.

We do, however, agree with Recommendation No. 2 to establish data 
quality review standards as part of our plans for the 2010 census. The 
quality standards developed within the Quality Framework discussed 
above will be used as' the quality review standards for the 2010 
census; the results of the standards working group on decisions on 
quality for data release will become a part of that Quality Framework. 
To that end, the working group chartered will complete its efforts to 
develop a standard. Once that standard has been approved and issued, 
the Census Bureau Program Quality Staff will assist the program 
directorates in implementing the standard. The objective is that the 
standard will be applicable to all Census Bureau programs, including 
the 2010 census and the American Community Survey (ACS).

Thank you for your support of our efforts to ensure quality in our 
Census Bureau data products. If you have additional questions or 
concerns, please contact Alan Tupek, Acting Chair of the Methodology 
and Standards Council, at 301/763-4287.

Specific Issues:

The Executive Summary, paragraph 2, and page 4, paragraph l, states 
that the Census Bureau did not provide any specific guidelines or 
procedures on the implementation of its Section 515 Information Quality 
Guidelines. This is not correct. The Census Bureau developed a specific 
standard to address Section 515 information quality complaints-Census 
Bureau Standard: Correcting Information that does not Comply with 
Census Bureau Section 51 5 Information Quality Guidelines (issued 5/16/
02).

Page 2, paragraph 2, The Census Bureau now has a standard that requires 
a discussion of limitations to the data it disseminates-Census Bureau 
Standard: Minimal Information to Accompany any Report of Census Bureau 
Data.

Page 5, paragraph 2 and page 27, paragraph 2. The report states that 
the Census Bureau has not provided additional resources to support the 
working group on data release. This is not correct. The Census Bureau 
established a Quality Program Staff (currently three individuals) in 
the spring of 2004. This staff supports all of the working groups that 
the Methodology and Standards Council charters to develop principles, 
standards, and guidelines; assists with the implementation of those 
documents; and works with program staff to incorporate quality into 
Census Bureau products and processes.

Page 15, paragraph 1, "The Bureau did not investigate most of the 
remaining issues in large part because they were insufficiently 
documented and the Bureau lacked the time and people to further 
investigate these issues."

Comment: The Census Bureau investigated the remaining issues containing 
sufficient documentation as part of the Count Question Resolution 
program, which was implemented between June 30, 2001 and September 30, 
2003.

Page 16, paragraph 2, "In 2000, missing data rates reached as high as 
50 percent for all group quarters residents, and as high as 75 percent 
for prison inmates."

Comment: The allocation rates for sample characteristics presented in 
the National Academy of Sciences report of the total group quarters 
population range from 18 percent (marital status) to 50 percent (wages 
last year).

Five of the 17 sample characteristics have allocation rates below 25 
percent; 11 rates are between 25 percent and 49.9 percent, and 1 rate 
is over 50 percent.

The highest rates are found in the correctional category, where the 
sample characteristics have allocation rates ranging from 31 percent 
(marital status) to 75 percent (occupation last year).

Four of the 17 allocation rates are under 25 percent, two rates are 
between 25 percent and 49.9 percent, 10 rates are between 50 percent 
and 74.9 percent, and there is one allocation rate above 75 percent 
(occupation last year).

Page 18, paragraph 2, "Additionally, we found that the decision about 
when and whether to release data on people in emergency and 
transitional shelters changed several times. Decisions about the 
release of data with identified quality problems appeared to be 
judgment calls made by individuals in parts of the Bureau different 
from those involved with Bureau partners and other stakeholders."

Comment: The Census Bureau worked closely with the National Coalition 
for the Homeless and other advocacy groups in developing plans and 
procedures for counting and producing tabulations for the population 
experiencing homelessness.

The Census Bureau considered several options for tabulating data in 
Summary File 1 (SF-1) from the Service-Based Enumeration (SBE) 
operation. Originally, the Census Bureau planned to separately publish 
the total number of people enumerated at emergency and transitional 
shelters.

People tabulated at the other service locations, such as soup kitchens, 
regularly scheduled mobile food vans, and targeted non-sheltered 
outdoor locations, would be included in the "Other noninstitutional 
group quarters" category.

After examining the 1998 Dress Rehearsal results, as well as the 
preliminary Census 2000 data. The Census Bureau decided in January 2001 
to include all people tabulated at the service locations during the SBE 
operation in one category, "Other noninstitutional group quarters," as 
part of its Summary File 1 (SF-1) release. This change was based on the 
Census Bureau's increasing concerns that the census tabulations of 
people enumerated at emergency and transitional shelters, without the 
provision of appropriate qualifiers and other limitations, would be 
misinterpreted. As a result, the Census Bureau decided to issue a 
special report on the results of the enumeration of emergency and 
transitional shelters with the appropriate caveats."

Page 21, The inventory and quality framework described was compiled by 
the Associate Director for Methodology and Standards, with input from 
the chiefs of the Computer-Assisted Survey Research Office, Decennial 
Statistics Studies Division, Demographic Statistical Methods Division 
(DMSD), Economic Statistical Methods and Programming Division, 
Planning, Research, and Evaluation Division, and Statistical Research 
Division.

Page 24, middle of page. The composition of the group stated in the 
text is incorrect. The group is composed of assistant division chiefs 
from the program areas-decennial, demographic, and economic. It is 
chaired by an assistant division chief in DSMD.

Page 25, The information concerning the working group on data release 
standards is incorrect. A correct statement follows:

The working group has reviewed the published detailed guidelines from 
the National Center for Education Statistics and from Statistics 
Canada. Benchmarking discussions have taken place with the Bureau of 
Labor Statistics and the National Center for Health Statistics. 
Additionally, the working group met with an official from Statistics:

New Zealand to discuss its standards. The group is planning meetings 
with additional federal agencies.

Page 29, The measures developed for the ACS program are being reviewed 
for possible implementation in other household surveys.

(450206): 

FOOTNOTES

[1] The terms "standards" and "guidelines" are often used without clear 
definition and sometimes interchangeably. The Bureau defines standards 
as methodological procedures that are required for all Bureau program 
areas and guidelines as procedures that are recommended for all Bureau 
program areas. We follow that distinction in discussing Bureau 
guidance. However, the term "guidelines" is also used, particularly in 
reference to governmentwide requirements, to refer to a broad set of 
related standards, guidelines, or a combination of these, and we follow 
that usage where appropriate. 

[2] Consolidated Appropriations, 2001, Pub. L. No. 106-554 (2000) 
(enacting H.R. 5658, §515) referred to by the Office of Management and 
Budget as the Information Quality Act.

[3] Office of Management and Budget, Guidelines for Ensuring and 
Maximizing the Quality, Objectivity, Utility, and Integrity of 
Information Disseminated by the Federal Government; Republication, 67 
Fed. Reg. 8452 (Feb. 22, 2002). 

[4] GAO, Decennial Census: Methods for Collecting and Reporting 
Hispanic Subgroup Data Need Refinement, GAO-03-228 (Washington, D.C.: 
Jan. 17, 2003).

[5] GAO, Decennial Census: Methods for Collecting and Reporting Data on 
the Homeless and Others without Conventional Housing Need Refinement, 
GAO-03-227 (Washington, D.C.: Jan. 17, 2003).

[6] The ACS is designed to replace the long-form census questionnaire 
and provide annual data for areas with populations of 65,000 or more 
and multiyear averages for smaller geographic areas using population 
and housing counts from the Intercensal Population Estimates. See GAO, 
The American Community Survey: Accuracy and Timeliness Issues, GAO-02-
956R (Washington, D.C.: Sept. 30, 2002), and ACS: Key Unresolved 
Issues, GAO-05-82 (Washington, D.C.: Oct. 8, 2004).

[7] 67 Fed. Reg. 8452.

[8] Office of Management and Budget, Statistical Policy Directive 
Number 3: Compilation, Release and Evaluation of Principal Federal 
Economic Indicators (Washington, D.C.: July 1985).

[9] Office of Management and Budget, Statistical Policy Working Paper 
31: Measuring and Reporting Sources of Error in Surveys (Washington, 
D.C.: June 2001).

[10] ESS includes Eurostat, the statistical directorate of the European 
Union; national statistical institutions of member countries; and a 
variety of academic and other statistical institutes. 

[11] European Statistical System, Quality Declaration of the European 
Statistical System (Brussels: September 2001).

[12] GAO-03-228.

[13] GAO-03-227.

[14] See GAO, 2000 Census: Refinements to Full Count Review Program 
Could Improve Future Data Quality, GAO-02-562 (Washington, D.C.: July 
3, 2002) and 2000 Census: Coverage Measurement Programs' Results, 
Costs, and Lessons Learned, GAO-03-287 (Washington, D.C.: Jan. 29, 
2003).

[15] GAO, 2000 Census: Refinements to Full Count Review Program Could 
Improve Future Data Quality, GAO-02-562 (Washington, D.C.: July 3, 
2002).

[16] National Research Council, The 2000 Census: Counting Under 
Adversity (Washington, D.C.: 2004).

[17] GAO-03-228.

[18] GAO-03-227.

[19] The Bureau's Methodology and Standards Council sets standards for 
the Bureau's surveys and censuses. It is chaired by the Associate 
Director for Methodology and Standards and includes division chiefs 
from across the Bureau. 

[20] Department of Commerce, Guidelines for Ensuring and Maximizing the 
Quality, Objectivity, Utility, and Integrity of Disseminated 
Information, 67 Fed. Reg. 62685 (Oct. 8, 2002). The Department of 
Commerce took a distributed approach, requiring its operating units 
(including the Bureau) to document and make available to the public 
their own information quality standards.

[21] Some Bureau programs have their own data quality guidance and 
report extensively on the quality and limitations of the data. For 
example, for the 2003 Annual Social and Economic Supplement of the 
Current Population Survey, the Bureau provides information on the 
limitations, including a recommendation on using cells with a small 
number of respondents. See U.S. Census Bureau, Current Population 
Survey, 2003 Public Use File Technical Documentation (Washington, D.C.: 
2003), G-7.

[22] Statistics Canada, Statistics Canada Quality Guidelines, 3RD ed. 
(Ottawa: October 1998), and Statistics Canada's Quality Assurance 
Framework (Ottawa: 2002).

[23] As noted earlier in this report, we did not evaluate the 
implementation or effectiveness of these guidelines and standards or 
their specific applicability to the Bureau.

[24] Following our inquiries about staff support, the Bureau 
established a Quality Program Staff of three individuals in the spring 
of 2004 to support all of the working groups chartered by the 
Methodology and Standards Council.

[25] See GAO-02-956R and GAO-05-82. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548: 

To order by Phone: 

 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: