This is the accessible text file for GAO report number GAO-10-324 entitled '2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness' which was released on April 23, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: April 2010: 2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness: GAO-10-324: GAO Highlights: Highlights of GAO-10-324, a report to congressional requesters. Why GAO Did This Study: Assessing the accuracy of the census is essential given that census data are used to apportion seats in Congress, to redraw congressional districts, and for many other public and private purposes. The U.S. Census Bureau's (Bureau) Census Coverage Measurement program (CCM) is to assess the accuracy of the 2010 Census and improve the design of operations for the 2020 Census. In April 2008, GAO recommended that the Bureau identify how it would relate CCM results—where the 2010 Census was accurate and inaccurate—to census operations to improve future censuses. Knowing where the 2010 Census was inaccurate can help inform research to improve the 2020 Census. GAO was asked to examine (1) the status of CCM planning and (2) the effects of design decisions since GAO issued its April 2008 report. GAO reviewed Bureau documents related to CCM design and National Academy of Sciences reports, and interviewed responsible Bureau officials. What GAO Found: Since GAO’s April 2008 report, the Bureau has finalized plans for 2010 CCM goals, the timing of operations, and the types of results to be produced. Planning continues in other areas, such as developing estimation methods, evaluating the CCM program, and implementing its Master Trace Project, which would enable the Bureau to link its datasets and systems to support a broad range of research. The deadlines for some of these plans have not yet passed, but the Bureau already has default plans in place in case further changes do not occur. In mid-December, the Director decided to make some additional changes to the CCM program to improve the quality of CCM results. GAO found that additional actions on Bureau decisions may make CCM more useful in informing Bureau decisions on future census and coverage measurement efforts: * The Bureau’s 2020 planning efforts are described in a series of decision memoranda issued in the summer of 2009. However, the Bureau has not yet taken steps to integrate CCM results with early 2020 planning to prepare for a census test in 2014. By describing, for example, what the Bureau might learn from CCM or how the results might feed into 2020 Census planning, the Bureau will better ensure that there are no gaps or overlaps in the use of CCM for early 2020 planning. * In September 2009, the Bureau began its Master Trace Project, which is intended to ensure that its datasets and systems can be used together to support detailed research into the causes of census coverage problems and facilitate research on the possible interactions of future operations. At the time of this review, the Bureau had not yet completed an inventory of the census databases that might be of potential interest for future research, identified which archived versions might be most useful, or mapped out how they might be archived and linked. Doing this quickly will be important as the census is already underway and it will be difficult to make changes to database structures or archival and data storage plans if the Bureau’s assessments determine that changes are necessary. * The Bureau reviewed its previous decision to start CCM’s Person Interviewing operation later than it did in 2000, and decided in June 2009 not to change it. However, the Bureau does not have a plan to assess the trade-offs in error between earlier and later start dates. Additional research on the trade-offs of different start dates could help the Bureau more fully understand the implications of CCM timing decisions on the resulting estimates of coverage error and better determine the optimal timing of Person Interviewing in future censuses. What GAO Recommends: GAO is making recommendations to improve the Bureau's use of CCM in planning for future operations, with which the Department of Commerce generally agreed. Commerce stated that it is taking action to ensure data preservation. Further, Commerce stated that although it considers a 2000 contamination study comprehensive, a new recall bias study is planned for 2010. View [hyperlink, http://www.gao.gov/products/GAO-10-324] or key components. For more information, contact Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. [End of section] Contents: Letter: Background: The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008 Report: The Bureau Needs to Take Additional Actions to Improve the Usefulness of CCM: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Comments from the Department of Commerce: Table: Table 1: Status of the Bureau's Decisions in Key Areas of the Census Coverage Measurement Program: [End of section] United States Government Accountability Office: Washington, DC 20548: April 23, 2010: The Honorable Darryl E. Issa: Ranking Member: Committee on Oversight and Government Reform: House of Representatives: The Honorable Patrick T. McHenry: Ranking Member: Subcommittee on Information Policy, Census, and National Archives: Committee on Oversight and Government Reform: House of Representatives: Data from the decennial census--a constitutionally mandated count of the national population--are used to apportion seats in Congress, redraw congressional districts, allocate billions of dollars in federal assistance to state and local governments each year, and inform the planning and investment decisions of numerous public and private sector entities. The census aims to locate and count people-- only once--in the right place, and collect complete and correct information about them. Because census data are central to so many critical functions, it is essential to assess census accuracy and improve the process when needed. In April 2008, we reported that the U.S. Census Bureau (Bureau) needed to finalize plans for its Census Coverage Measurement (CCM) program-- the effort intended to assess the accuracy of the 2010 Census and improve the design of operations for the 2020 Census--and made related recommendations to the Bureau's parent agency, the Department of Commerce.[Footnote 1] In particular, we noted that the Bureau should produce plans that include (1) a description of when it will provide CCM results, (2) how it plans to report its CCM results in relation to census operations, and (3) key decision points and plans for evaluating aspects of the CCM. Commerce has taken steps to implement actions for each recommendation. Since we issued our report, the National Academy of Sciences panel on Correlation Bias and Coverage Measurement in the 2010 Decennial Census released its final report on the Bureau's plans for CCM,[Footnote 2] recommending numerous steps to enhance the value of the program. You asked us to examine the current plan for the CCM program and to provide (1) an update on the status of the planning since our April 2008 report, and (2) the potential effects of major CCM decisions on the quality and usefulness of CCM data. To meet these objectives, we reviewed Bureau documents related to CCM design and National Academy of Sciences (NAS) reports, and we interviewed Bureau officials responsible for CCM. We focused primarily on the Bureau's decisions in the following areas: 2010 CCM goals, timing of operations and reporting results, estimation methods, results to be produced, program evaluation, and implementing its Master Trace Project. Specifically, to update the status of CCM, we reviewed scheduling documents and decision memorandums. To identify the potential effects we reviewed decision memorandums and related justifications, prior Bureau and NAS research and our reports related to CCM and evaluation. We conducted our work from June 2009 to February 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: The Bureau puts forth tremendous effort to conduct a complete and accurate count of the nation's population; nonetheless, some degree of coverage error is inevitable because of the inherent complexity of counting the nation's large and diverse population and limitations in census-taking methods. These census coverage errors can take a variety of forms, including a person missed (an undercount), a person counted more than once (an overcount), or a person who should not have been counted, such as a child born after Census Day (another type of overcount). To further understand and to inform users about the quality of the census, the Bureau has been evaluating coverage measurement for more than 50 years. While initial evaluations relied solely on demographic analysis--population estimates based on birth and death rates as well as immigration estimates--modern coverage measurement began with the 1980 Census when the Bureau began also comparing census counts to survey results from an independent coverage measurement sample of the population. Using statistical methods, the Bureau generated detailed measures of the differences among undercounts of particular ethnic, racial, and other groups, which have been referred to as "differential undercounts." These measures were also generated for the 1990 and 2000 censuses. Although the Bureau considered doing so in earlier decades, it has never used its estimates of coverage error to adjust census data. In 1980, the Director of the Census Bureau decided that potential adjustments would be flawed due to missing and inaccurate data. In 1990, the Bureau recommended statistically adjusting census data; however, the Secretary of Commerce determined that the evidence to support an adjustment was inconclusive and decided not to adjust. For the 2000 Census, a 1999 Supreme Court ruling held that the Census Act prohibited the use of statistical sampling to generate population data for apportioning the House of Representatives.[Footnote 3] The Bureau had planned to produce apportionment numbers using traditional census- taking methods, and provide statistically adjusted numbers for non- apportionment uses of the data such as congressional redistricting and allocating federal funds. The Bureau later determined that its statistical estimates did not provide a reliable measure of census accuracy and could not be used to adjust the non-apportionment census data. The Bureau is not planning to use CCM to adjust the 2010 Census. Instead, CCM will be used to evaluate coverage error to improve the 2020 and future censuses, and will focus on estimating various components of census coverage in addition to net coverage errors--the net effect on coverage after undercounts and overcounts are considered. These components of coverage include correct enumerations, erroneous enumerations (people or housing units that were counted but should not have been), and omissions (people or housing units that were not counted but should have been). The Bureau also plans to include imputations (counts of people and their characteristics that are provided for nonresponding households, usually based on responses from others under similar circumstances, such as from surrounding households). Statistical measurements of census coverage are obtained by comparing and matching the housing units and people counted by the independent coverage measurement sample to those counted by the census in and around the sample areas. The Bureau has developed separate address lists--one for the entire nation of over 134 million housing units that it will use to conduct the census and one for coverage measurement sample areas--and will collect each set of data through independent operations. For the 2010 Census, census operations began collecting population data from households in January 2010 and will continue through the end of July, while CCM operations will collect data by visiting each of the housing units in the coverage measurement sample during an operation called Person Interviewing from August through October. The statistical methodology the Bureau uses to estimate net coverage errors relies on an assumption that the chance that a person is counted by the census is not affected by whether he or she is counted in the independent coverage measurement sample, or vice versa. Because violating this "independence" assumption can bias coverage estimates, the Bureau takes special measures to maintain CCM's separation from the census, such as developing a separate address list for the coverage measurement sample discussed above. The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008 Report: Since our April 2008 report, the Bureau has finalized its plans in key areas of the CCM program including CCM's goals, the timing of operations, and the timing and types of results to be produced. Planning continues in other areas, such as developing estimation methods, evaluating the CCM program, and implementing its Master Trace Project. Continued progress and adherence to schedule will be important to ensure that the Bureau carries out CCM in order to meet its goal of improving the 2020 Census. For example, in our 2008 report, we recommended that the Bureau provide decision points and plans for evaluating CCM. In September 2009, the Bureau finalized its list of 22 planned evaluations for the 2010 Census, which included five that address specific methodological or procedural topics within the CCM program. However, all study plans are not due to be completed until April 2010. In addition, while the deadlines for finalizing CCM estimation methods have not yet passed, the Bureau has many of its default plans already in place. Default plans allow the Bureau to move forward on schedule even if new plans have not been developed. Table 1 shows the status of the Bureau's plans for the design of CCM in each of these areas. Table 1: Status of the Bureau's Decisions in Key Areas of the Census Coverage Measurement Program: CCM plan area: Goals; Decision status: Finalized; Decision date: Sept. 2005; Comment: The goals have not substantially changed and are subject to refinement as research on how to meet them progresses. CCM plan area: Timing: Of Person Interviewing; Decision status: Finalized; Decision date: June 2009; Comment: In 2009, the Bureau reconsidered and stayed with its earlier decision on the timing of this operation. CCM plan area: Timing: Of releasing results; Decision status: Finalized; Decision date: June 2009; Comment: The Bureau decided to move reporting forward by about 3 months from initial baseline schedule. CCM plan area: Estimation Methods; Decision status: In progress; Decision date: est. April 2010; Comment: Default plans are largely in place, but the Bureau is researching additional technical improvements. CCM plan area: Results to Produce: By level of geography; Decision status: In progress; Decision date: Spring 2010; Comment: The Bureau will publicize the levels of estimates planned for below the state level. CCM plan area: Results to Produce: By demographic groups; Decision status: Finalized; Decision date: June 2009; Comment: The Bureau plans public dissemination of the planned reporting groups in early 2010. CCM plan area: Evaluations: Topics; Decision status: Finalized; Decision date: Sept. 2009; Comment: Additional studies are expected outside the formal evaluation program. The Bureau also plans a series of technical memorandums documenting CCM process and results, similar to its approach for the 2000 Census. CCM plan area: Evaluations: Study plans; Decision status: In progress; Decision date: est. April 2010; Comment: Subject area experts will complete study plans on a rolling basis until the deadline. CCM plan area: Master Trace Project; Decision status: In progress; Decision date: est. Sept. 2010; Comment: The Director of the Census Bureau recently initiated the Master Trace Project. Source: GAO analysis of Bureau documentation and schedule. [End of table] Recent Changes to CCM Data Collection Plan Could Improve Quality of CCM Data: In September 2009, shortly after taking office, the Director of the Census Bureau asked the staff responsible for CCM to review its CCM design and propose specific changes that would reduce the introduction of nonsampling error--such as human errors made when recording data during interviews--into CCM and its resulting estimates. The staff proposed numerous changes intended to reduce error in collected data. They also proposed an additional research study. The Director approved all of these proposals in mid-December 2009. Key changes included: * increasing the reinterview rates for CCM field work to improve quality assurance; * increasing training time for short-term workers hired to conduct door-to-door visits during the Person Interviewing operation to improve interview techniques for local or other special situations due to current economic conditions (such as people who became homeless or have had to move frequently during the housing crisis); * increasing supervisor-to-employee field staffing ratios to improve quality and monitoring of field work at each level; and: * adding a telephone-based study to collect information about how well respondents recall information about their residence and possible movement since Census Day. In addition, the decision authorized a nearly 45 percent reduction in the CCM sample size that the Bureau believes would generate the cost savings to pay for the other changes. Our understanding of the issues suggests that these are reasonable efforts to improve survey quality. The Bureau's reduction in sample size will reduce precision of the estimates, yet the proposed changes should reduce nonsampling errors and thus provide users with more reliable estimates. For example, the Bureau expects short-term CCM workers to make fewer mistakes in identifying temporary or unconventional housing units when they have received additional training specific to their local circumstances, such as in areas with large numbers of seasonal or displaced workers. The Bureau Needs to Take Additional Actions to Improve the Usefulness of CCM: The Bureau's actions to finalize some areas of CCM program planning are important steps in the right direction. Still, in some cases, it will be important for the Bureau to take additional actions to help ensure the results of CCM are as useful as they could be to inform Bureau decisions on improving future censuses and coverage measurement efforts. For example, the Bureau could better document how CCM results will be used as part of the planning process for the 2020 Census. Indeed, the Bureau has already begun laying the foundation for its 2020 planning efforts. These early planning efforts are described in a series of decision memorandums issued in the summer of 2009, and include milestones leading up to a census test in April 2014, descriptions of planning phases, and a list of the various organizational components that conduct the census. Although these planning documents explicitly state the importance of relying on the 2010 Census Evaluation and Testing program--an ongoing assessment effort separate from CCM that, like CCM, is designed to improve future operations--the Bureau has not yet taken similar steps to integrate the CCM program with 2020 planning. In addition, the Bureau does not have specific plans in its CCM program goals to do anything beyond producing CCM results. Bureau officials have maintained that until it produces CCM results, it is difficult to determine how to use CCM data to improve the design of future decennials. While we agree with the Bureau that the results will determine the specifics of any potential design improvements, it is not premature to consider how the earliest results from CCM-- scheduled for early 2012--could help inform early planning and decisions. Importantly, by creating a "roadmap" that describes, for example, what the Bureau might learn from CCM or how the results might feed into early 2020 Census planning, the Bureau will better ensure that there are no gaps or overlaps in the use of CCM in early 2020 planning. The Bureau's Master Trace Project is another area where additional efforts are needed to ensure useful CCM results. The Bureau initiated the Master Trace Project in September 2009, to facilitate the use of census and CCM data for future research. Currently, Bureau data are collected and archived in different types of datasets and systems. The Master Trace Project is intended to ensure that these datasets and systems can be used together, or linked, to support detailed research into the causes of census coverage problems and facilitate research on the possible interactions of future operations. For example, a researcher might want to see if there is a relationship between the Bureau's employment practices and the magnitude of an undercount in a particular area. In so doing, the researcher may want to compare census payroll, overtime, and other human capital data to the data from that region collected and processed by census and CCM. Such datasets would not ordinarily be linked during the census. The Bureau has not yet taken the steps needed to ensure that such research across different data systems would be possible. The Bureau held a meeting in December 2009 with staff responsible for many major decennial systems and obtained agreement about the importance of data retention for this project; however, the Bureau has not yet resolved how it would make the project happen. In particular, the Bureau has not yet completed an inventory of the census databases that might be of potential interest for future research, identified which archived versions might be most useful, or mapped out how they might be archived and linked. Until this is done, it is unclear that Bureau or other researchers will have access to census operational data that they need to fully analyze the census coverage errors that CCM may uncover. Moving forward, it will be important for the Bureau to perform the initial assessment of its data systems, identify gaps in data collection, and identify any other related steps to ensure that key data can be linked. Doing this quickly will also be important as Census 2010 is underway and it could become increasingly difficult to make changes to database structures or archival and data storage plans if the Bureau's assessments determine that changes are necessary. A third area where the Bureau needs to do additional work is in assessing how the timing of CCM data collection might adversely affect CCM findings. When planning CCM, the Bureau faced the challenge of determining the optimal time to launch the CCM data collection operation, known as Person Interviewing (PI). If the Bureau starts PI too early, it increases the chance that it overlaps with census data collection, possibly compromising the independence of the two different operations and introducing a "contamination bias" error into CCM data. If the Bureau starts PI too late, it increases the chance that respondents will not accurately remember household information from Census Day, April 1, introducing error (known as "recall bias") in the CCM count. Both types of errors--contamination bias and recall bias--could affect the Bureau's conclusions about the accuracy of the census. An understanding of the trade-offs between these two types of biases would be important in future decisions regarding the optimal timing of PI. In early 2009, based on concerns by the National Academy of Sciences (NAS) and other stakeholders about the relative lateness in the start date of PI and its possible impact on the quality of CCM findings, the Bureau considered whether to start PI 6 weeks earlier than planned. In June 2009, the Bureau decided to keep the originally scheduled start on August 14, 2010. Bureau memorandums and officials justified the decision largely because of concern that it was too late in the planning process to make a change in the complex CCM schedule. The memorandums cited gaps in knowledge about the impact of timing on recall bias, presented research with differing conclusions about the extent of contamination in prior census tests, and justified the recommendation to not change the start date by the operational challenges faced to make the change. Bureau officials have also explained that the goal of using coverage measurement in 2000 to possibly adjust the census-created time pressures in 2000 that forced an early PI, and because such time pressures do not exist for PI in 2010, it is scheduled to begin more than 4 months after Census Day. By comparison, during the 2000 Census, the Bureau launched PI in April 2000 and had completed about 99 percent of its data collection by the end of the first week of August 2000, a week earlier than the scheduled 2010 PI start date. An extensive 2000 Census evaluation found no evidence of contamination bias caused by the earlier start of PI in 2000. Related Bureau research since then has also found no significant evidence of contamination bias during census tests, although one test found that census results could be affected. Yet Bureau officials remained concerned about the possibility, since the CCM questions are similar to follow-up questions used in one of the 2010 census follow-up operations. Furthermore, parts of this census operation are new in 2010, and end later than similar operations did in 2000. Moving forward, additional research on the trade-offs between recall bias and contamination errors could help the Bureau more fully understand the implications of choosing various start times for PI on the resulting estimates of coverage error and better determine the optimal timing of PI in future censuses. Currently, the Bureau has a telephone-based study planned in order to measure recall errors, which could provide additional information about when recall errors are more likely to occur. However, this study is limited to certain types of recall error, and the Bureau does not have an evaluation planned to measure possible contamination between the new, much later, parts of census follow-up and CCM data collection or to assess the trade-offs between the biases from starting earlier compared to starting later. Such additional study after the 2010 Census could provide the Bureau better information about the trade-offs in data quality from potential contamination and recall biases and provide a better basis for determining the optimal scheduling of coverage measurement operations. Conclusions: Assessing the accuracy of the census is an essential step in improving current and future censuses. The Bureau has made progress on designing and planning for its CCM program and continues activity to complete the plan. Additional actions in three CCM planning areas may further improve CCM or its usefulness to the 2020 Census. Specifically, the Bureau has stated the importance of using 2010 evaluation data such as CCM's for 2020 Census design, but has not yet taken steps to link CCM data to an improved 2020 design. If the Bureau is to best achieve its goal of using CCM to help improve the 2020 Census, it will need to integrate planning for any follow-up work on CCM results or data with the other early planning already underway for Census 2020. Second, the Bureau has many different processes that come together in the conduct of a decennial census, and archived data on those processes could provide useful information to researchers trying to figure out what worked well and what did not. The Master Trace Project can help researchers link CCM results and data to potential design changes for Census 2020. Determining which data need to be linked or archived to enable future linkage within the project can help prevent gaps in 2010 data that might hinder the project's ability to help identify improvements for the 2020 Census. Third, the timing of CCM's primary data collection operation--Person Interviewing--involves trade-offs between reducing contamination bias and reducing recall error that the Bureau did not have conclusive information on. Since 2010 Person Interviewing is starting 1 week after a similar operation ended in 2000, the chance of introducing recall bias errors into CCM data is higher in 2010 than it was in 2000. Although the Bureau has a study planned to measure some recall errors, there is no study planned to measure contamination between the new parts of census follow-up--which use questions similar to those asked by CCM and finish much later than follow-up did in 2000--and CCM or to assess the trade-offs between the two types of biases in timing decisions. Targeted research after the 2010 Census on the relationship between the timing of data collection and the trade-offs between these types of errors before the 2020 Census and its coverage measurement efforts could help the Bureau better determine the optimal timing of future data collection operations. Recommendations for Executive Action: We recommend that the Secretary of Commerce require the Director of the U.S. Census Bureau to take the following three actions to improve the usefulness of CCM for 2020: To help the Bureau achieve its goal of using CCM to improve the 2020 Census, better document links between the 2010 CCM program and 2020 Census planning, integrating the goal of using the CCM program to improve Census 2020, such as with CCM results and data, into those broader plans for 2020. To ensure that Bureau datasets from the 2010 Census can be used with other Bureau datasets to support research that could improve the census and CCM, complete the Master Trace Project's assessment of how key census and CCM data systems are, or can be, linked to each other; identify any potential data gaps; and identify other related steps for future action. To help the Bureau better determine the optimal timing of future coverage measurement data collection, fully assess the trade-offs between starting the data collection earlier, with the possibility of introducing contamination errors, and starting later, with the possibility of introducing recall errors. Agency Comments and Our Evaluation: The Secretary of Commerce provided written comments on a draft of this report on April 5, 2010. The comments are reprinted in appendix I. Commerce generally agreed with the overall findings and recommendations and appreciated our efforts in helping the Census Bureau develop a successful evaluation plan for the 2020 Census. Commerce also provided additional information and comments on certain statements and conclusions in the report. With respect to our second recommendation to complete the Master Trace Project's assessment of linking key census and CCM data systems, to identify any potential data gaps, and to identify other related steps for future action, Commerce maintained that it would be taking action to preserve adequate documentation and maximize the amount of data retained from each major decennial system. We commend the Bureau for committing to these steps and encourage its follow-through on them and its identification of remaining data gaps and additional steps needed. With respect to our third recommendation to fully assess the trade- offs between two types of error related to starting CCM data collection either earlier or later relative to Census Day, Commerce responded that (1) it is too late to create a new study for 2010 Census; (2) it considers a Bureau contamination study from 2000 to be definitive; and (3) it has recently developed a study on recall bias to try to measure some of the effects of scheduling CCM data collection at various periods of time following the census enumeration. We agree that it is too late to attempt any additional unplanned data collection during the 2010 Census, and we revised our discussion to clarify our intent that the recommended research be conducted after the 2010 Census. We also recognize the thoroughness of the 2000 contamination study the Bureau cites, commend the Bureau on undertaking additional study of recall bias, and look forward to reviewing its study plans when they are available. However, we recommended research comparing trade-offs between the two types of errors at a variety of start dates for CCM data collection--something the 2000 study did not discuss and something it is unclear that a study of only recall bias will achieve. Furthermore as we discussed in our draft report, the Bureau expressed concerns over possible contamination between CCM and new parts of census follow-up in 2010-- parts that were introduced after the 2000 study and that were not included in the scope of the 2000 study. We clarified our discussion of this in the report to better focus on the need for research that relates the trade-offs between the two types of error at different timing of data collection. Commerce provided additional information that in response to advice from various advisory panels and after additional research, it would soon make public its proposed geographic levels for CCM estimates. We reflected this decision in table 1 of our report. Finally, Commerce provided additional information about its plans to produce highly technical documentation of the results of CCM estimation including modeling, missing data, and errors in the estimates in a series of memorandums as it did for Census 2000. We reflected this decision in table 1 of this report. We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Jeff Dawson, Dewi Djunaidy, Ron Fecso (Chief Statistician), Andrea Levine, Ty Mitchell, Melanie Papasian, and Tamara F. Stenzel. Signed by: Robert Goldenkoff: Director: Strategic Issues: [End of section] Appendix I: Comments from the Department of Commerce: United States Department Of Commerce: The Secretary of Commerce: Washington, D.C. 20230: April 5, 2010: Mr. Robert Goldenkoff: Director: Strategic Issues: U.S. Government Accountability Office: Washington, DC 20548: Dear Mr. Goldenkoff: The Department of Commerce (Department) appreciates the opportunity to comment on the U.S. Government Accountability Office's draft report entitled "2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness" (GA040-324). The Department's comments on this report are enclosed. Sincerely, Signed by: Gary Locke: Enclosure: [End of letter] U.S. Department of Commerce Comments on the United States Government Accountability Office Draft Report Entitled "2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness," GAO 10-324. March 2010: The U.S. Census Bureau would like to thank the Government Accountability Office (GAO) for its efforts in examining the 2010 Census Coverage Measurement (CCM) program plans for assessing the accuracy of the 2010 Census, and for the opportunity to comment on the observations and recommendations in this report. We have no fundamental disagreements with the overall findings or the recommendations regarding improvements in our evaluation processes for the 2020 Census cycle. We do, however, offer the following comments on certain statements and conclusions in this report. * Page 6, second bullet: "The Bureau will not decide the level of geography for which to release estimates before knowing the quality of the results." Census Bureau response: In response to advice from various advisory panels and after additional research in the spring of 2010, we will make public the proposed geographic levels of the estimates below the state level (already planned). * Page 6, third bullet: "Additional studies are expected outside the formal evaluation program." Census Bureau response: We have recently developed a study on recall bias to try to measure some of the effects of scheduling Person Interviewing at various periods of time following the census enumeration. This study is not yet part of the formal evaluation program. In addition, to document the results from CCM estimation— including modeling, missing data, and errors in the estimates—we will produce a series of memorandums. The results will not come from experiments or evaluations but will be based on actual data and estimates from the CCM. This highly technical documentation is similar to our approach for Census 2000. * Page 9: "The Bureau's Master Trace Project is another area where additional efforts are needed to ensure useful CCM results." Census Bureau response: The Census Bureau is currently taking actions in accordance with GAO's recommendation on retaining and linking 2010 Census data (including CCM data) for purposes of analysis and 2020 census planning. In addition to preserving adequate documentation, the Census Bureau's immediate focus is to maximize the amount of data retained from each major decennial system. As the Census Bureau can strategically leverage the time of key staff members during the implementation of the 2010 Census, the polishing of data documentation and the development of guidelines on how to link data from different decennial systems will take final form. As part of this process, we plan to save all data collection and processing files from the CCM. * Page 11: "Yet Bureau officials remained concerned about the possibility [of contamination], since the CCM questions are similar to follow-up questions used in one of the 2010 follow-up operations." Census Bureau response: Parts of this census operation are new in 2010 and end later than similar operations did in 2000. Their timing also pushed back the start of Person Interviewing. * Page 13: "To help the Bureau ...fully assess the tradeoffs between starting the data collection earlier ...and starting later...." Census Bureau response: We ran a careful, and what we consider definitive, contamination study in 2000. We cannot create a new study for the 2010 Census now, as its too late,. We would have had to create a separate Person interviewing panel to undergo an early interview and measure its contamination. For reasons of cost and response burden, we do not want to repeat studies unless we expect to learn more from them. Furthermore, we would not have had an instrument ready to conduct a contamination test. In conclusion, we acknowledge the GAO's extensive work in reviewing these activities, and we appreciate its ongoing efforts to help us develop a successful evaluation plan for the 2020 Census. [End of section] Footnotes: [1] [hyperlink, http://www.gao.gov/products/GAO-08-414], 2010 Census: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results (April 25, 2008). [2] Robert M. Bell and Michael L. Cohen, eds., Coverage Measurement in the 2010 Census (Washington, D.C.: The National Academies Press, 2009). [3] Dep't of Commerce v. U.S. House of Representatives, 525 U.S. 316 (1999) (citing 13 U.S.C. §195. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: