This is the accessible text file for GAO report number GAO-09-413T entitled 'Information Technology: Census Bureau Needs to Strengthen Testing of 2010 Decennial Systems' which was released on March 6, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security, Committee on Homeland Security and Governmental Affairs, U.S. Senate: United States Government Accountability Office: GAO: For Release on Delivery: Expected at 2:30 p.m. EST: Thursday, March 5, 2009: Information Technology: Census Bureau Needs to Strengthen Testing of 2010 Decennial Systems: Statement of David A. Powner: Director, Information Technology Management Issues: Robert Goldenkoff: Director, Strategic Issues: GAO-09-413T: GAO Highlights: Highlights of GAO-09-413T, a testimony before the Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security, Committee on Homeland Security and Governmental Affairs, U.S. Senate. Why GAO Did This Study: The Decennial Census is mandated by the U.S. Constitution and provides vital data that are used, among other things, to reapportion and redistrict congressional seats and allocate federal financial assistance. In March 2008, GAO designated the 2010 Decennial Census a high-risk area, citing a number of long-standing and emerging challenges, including weaknesses in the U.S. Census Bureau’s (Bureau) management of its information technology (IT) systems and operations. In conducting the 2010 census, the Bureau is relying on both the acquisition of new IT systems and the enhancement of existing systems. Thoroughly testing these systems before their actual use is critical to the success of the census. GAO was asked to testify on its report, being released today, on the status and plans of testing of key 2010 decennial IT systems. What GAO Found: Although the Bureau has made progress in testing key decennial systems, critical testing activities remain to be performed before systems will be ready to support the 2010 census. Bureau program offices have completed some testing of individual systems, but significant work still remains to be done, and many plans have not yet been developed (see table below). In its testing of system integration, the Bureau has not completed critical activities; it also lacks a master list of interfaces between systems and has not developed testing plans and schedules. Although the Bureau had originally planned what it refers to as a Dress Rehearsal, starting in 2006, to serve as a comprehensive end- to-end test of key operations and systems, significant problems were identified during testing. As a result, several key operations were removed from the Dress Rehearsal and did not undergo end-to-end testing. The Bureau has neither developed testing plans for these key operations, nor has it determined when such plans will be completed. Weaknesses in the Bureau’s testing progress and plans can be attributed in part to a lack of sufficient executive-level oversight and guidance. Bureau management does provide oversight of system testing activities, but the oversight activities are not sufficient. For example, Bureau reports do not provide comprehensive status information on progress in testing key systems and interfaces, and assessments of the overall status of testing for key operations are not based on quantitative metrics. Further, although the Bureau has issued general testing guidance, it is neither mandatory nor specific enough to ensure consistency in conducting system testing. Without adequate oversight and more comprehensive guidance, the Bureau cannot ensure that it is thoroughly testing its systems and properly prioritizing testing activities before the 2010 Decennial Census, posing the risk that these systems may not perform as planned. Table: Status and Plans of 2010 System Testing: System: Headquarters processing; Testing status: In progress; Testing plan completed: Partial; Testing schedule completed: Partial. System: Master address and geographic information; Testing status: In progress; Testing plan completed: Partial; Testing schedule completed: Partial. System: Decennial response integration; Testing status: In progress; Testing plan completed: Partial; Testing schedule completed: Partial. System: Field data collection automation; Testing status: In progress; Testing plan completed: Partial; Testing schedule completed: Partial. System: Paper-based operations; Testing status: In progress; Testing plan completed: No; Testing schedule completed: Partial. System: Data access and dissemination; Testing status: In progress; Testing plan completed: Partial; Testing schedule completed: Partial. Source: GAO analysis of Bureau data. [End of table] What GAO Recommends: In its report, GAO recommended that the Secretary of Commerce direct the Bureau to complete key system testing activities, develop and maintain plans and schedules for integration testing, and improve the oversight of and guidance for systems testing. In comments on a draft of this report, the department agreed with GAO’s recommendations. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-413T]. For more information, contact David A. Powner at (202) 512-9286 or pownerd@gao.gov or Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. [End of section] Mr. Chairman and Members of the Subcommittee: Thank you for the opportunity to participate in today's hearing on the 2010 census. The U.S. Census Bureau (Bureau) is relying on both the acquisition of new systems and the enhancement of existing legacy systems for conducting operations for the 2010 Decennial Census. As you know, the census is mandated by the U.S. Constitution and provides data that are vital to the nation. These data are used, for example, to reapportion and redistrict the seats of the U.S. House of Representatives, realign the boundaries of the legislative districts of each state, and allocate federal financial assistance. Carrying out the census is the responsibility of the Department of Commerce's Census Bureau, which is relying on automation and technology to improve the coverage, accuracy, and efficiency of the 2010 census. Because the accuracy of the 2010 census depends in part on the proper functioning of these systems, both individually and when integrated, thorough testing of these systems before their actual use is critical to the success of the census. As you know, in March 2008, we designated the 2010 Decennial Census as a high-risk area, citing a number of long-standing and emerging challenges,[Footnote 1] including weaknesses in the Bureau's management of its information technology (IT) systems and operations. The 2010 Decennial Census remained as one of our high-risk areas in our recent high-risk update issued in January 2009.[Footnote 2] This statement summarizes the findings in our report, being released by the subcommittee today, on the status and plans of testing of key 2010 decennial IT systems.[Footnote 3] Our work for this report was performed in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Background: The Bureau's mission is to provide comprehensive data about the nation's people and economy. The 2010 census enumerates the number and location of people on Census Day, which is April 1, 2010. However, census operations begin long before Census Day and continue afterward. For example, address canvassing for the 2010 census will begin in April 2009, while the Secretary of Commerce must report tabulated census data to the President by December 31, 2010, and to state governors and legislatures by March 31, 2011. * The decennial census is a major undertaking for the Bureau that includes the following major activities: * Establishing where to count. This includes identifying and correcting addresses for all known living quarters in the United States (address canvassing) and validating addresses identified as potential group quarters, such as college residence halls and group homes (group quarters validation). * Collecting and integrating respondent information. This includes delivering questionnaires to housing units by mail and other methods, [Footnote 4] processing the returned questionnaires, and following up with nonrespondents through personal interviews (nonresponse follow- up). It also includes enumerating residents of group quarters (group quarters enumeration) and occupied transitional living quarters (enumeration of transitory locations), such as recreational vehicle parks, campgrounds, and hotels. It also includes a final check of housing unit status (field verification) where Bureau workers verify potential duplicate housing units identified during response processing. * Providing census results. This includes tabulating and summarizing census data and disseminating the results to the public. Role of IT in the Decennial Census: Automation and IT are to play a critical role in the success of the 2010 census by supporting data collection, analysis, and dissemination. Several systems will play a key role in the 2010 census. For example, enumeration "universes," which serve as the basis for enumeration operations and response data collection, are organized by the Universe Control and Management (UC&M) system, and response data are received and edited to help eliminate duplicate responses using the Response Processing System (RPS). Both UC&M and RPS are legacy systems that are collectively called the Headquarters Processing System. Geographic information and support to aid the Bureau in establishing where to count U.S. citizens are provided by the Master Address File/ Topologically Integrated Geographic Encoding and Referencing (MAF/ TIGER) system. The Decennial Response Integration System (DRIS) is to provide a system for collecting and integrating census responses from all sources, including forms and telephone interviews. The Field Data Collection Automation (FDCA) program includes the development of handheld computers for the address canvassing operation and the systems, equipment, and infrastructure that field staff will use to collect data. Paper-Based Operations (PBO) was established in August 2008 primarily to handle certain operations that were originally part of FDCA. PBO includes IT systems and infrastructure needed to support the use of paper forms for operations such as group quarters enumeration activities, nonresponse follow-up activities, enumeration at transitory locations activities, and field verification activities. These activities were originally to be conducted using IT systems and infrastructure developed by the FDCA program. Finally, the Data Access and Dissemination System II (DADS II) is to replace legacy systems for tabulating and publicly disseminating data. Comprehensive Testing Improves Chances of a Successful Decennial Census: As stated in our testing guide and the Institute of Electrical and Electronics Engineers (IEEE) standards,[Footnote 5] complete and thorough testing is essential for providing reasonable assurance that new or modified IT systems will perform as intended. To be effective, testing should be planned and conducted in a structured and disciplined fashion that includes processes to control each incremental level of testing, including testing of individual systems, the integration of those systems, and testing to address all interrelated systems and functionality in an operational environment. Further, this testing should be planned and scheduled in a structured and disciplined fashion. Comprehensive testing that is effectively planned and scheduled can provide the basis for identifying key tasks and requirements and better ensure that a system meets these specified requirements and functions as intended in an operational environment. Dress Rehearsal Includes Testing of Certain Systems and Operations: In preparation for the 2010 census, the Bureau planned what it refers to as the Dress Rehearsal. The Dress Rehearsal includes systems and integration testing,[Footnote 6] as well as end-to-end testing of key operations in a census-like environment. During the Dress Rehearsal period, running from February 2006 through June 2009, the Bureau is developing and testing systems and operations, and it held a mock Census Day on May 1, 2008. The Dress Rehearsal activities, which are still under way, are a subset of the activities planned for the actual 2010 census and include testing of both IT and non-IT related functions, such as opening offices and hiring staff. The Dress Rehearsal identified significant technical problems during the address canvassing and group quarters validation operations. For example, during the Dress Rehearsal address canvassing operation, the Bureau encountered problems with the handheld computers, including slow and inconsistent data transmissions, the devices freezing up, and difficulties collecting mapping coordinates. As a result of the problems observed during the Dress Rehearsal, cost overruns and schedule slippage in the FDCA program, and other issues, the Bureau removed the planned testing of several key operations from the Dress Rehearsal and switched key operations, such as nonresponse follow-up, to paper-based processes instead of using the handheld computers as originally planned. Bureau Is Making Progress in Key System Testing, but Lacks Plans and Schedules: Through the Dress Rehearsal and other testing activities, the Bureau has completed key system tests, but significant testing has yet to be done, and planning for this is not complete. Table 1 summarizes the status and plans for system testing. Table 1: Status of System Testing and Plans: System: Headquarters Processing--UC&M and RPS; Dress Rehearsal system testing: In progress; 2010 system testing: Testing status: In progress; 2010 system testing: Testing plan completed: Partial; 2010 system testing: Testing schedule completed: Partial. System: MAF/TIGER; Dress Rehearsal system testing: Completed; 2010 system testing: Testing status: In progress; 2010 system testing: Testing plan completed: Partial; 2010 system testing: Testing schedule completed: Partial. System: DRIS; Dress Rehearsal system testing: Completed; 2010 system testing: Testing status: In progress; 2010 system testing: Testing plan completed: Partial[A]; 2010 system testing: Testing schedule completed: Partial[A]. System: FDCA; Dress Rehearsal system testing: Partially completed[B]; 2010 system testing: Testing status: In progress; 2010 system testing: Testing plan completed: Partial; 2010 system testing: Testing schedule completed: Partial. System: PBO; Dress Rehearsal system testing: N/A[C]; 2010 system testing: Testing status: In progress; 2010 system testing: Testing plan completed: No; 2010 system testing: Testing schedule completed: Partial. System: DADS; Dress Rehearsal system testing: DADS[D] in progress; 2010 system testing: Testing status: DADS II in progress; 2010 system testing: Testing plan completed: Partial; 2010 system testing: Testing schedule completed: Partial. Source: GAO analysis of Bureau data. [A] Program officials stated that DRIS's test plan and schedule were completed but will be modified to reflect changes resulting from the switch to paper-based operations. [B] System testing related to operations removed from the Dress Rehearsal was not completed. These operations were later moved to PBO. [C] The office to support PBO was created in August 2008. [D] DADS system is being used for Dress Rehearsal system testing, but the replacement system, DADS II, is being developed and tested for 2010 operations. [End of section] Bureau Has Conducted Limited Integration Testing, but Has Not Developed 2010 Test Plans and Schedules for Integration Testing: Effective integration testing ensures that external interfaces work correctly and that the integrated systems meet specified requirements. This testing should be planned and scheduled in a disciplined fashion according to defined priorities. For the 2010 census, each program office is responsible for and has made progress in defining system interfaces and conducting integration testing, which includes testing of these interfaces. However, significant activities remain to be completed. For example, for systems such as PBO, interfaces have not been fully defined, and other interfaces have been defined but have not been tested. In addition, the Bureau has not established a master list of interfaces between key systems, or plans and schedules for integration testing of these interfaces. A master list of system interfaces is an important tool for ensuring that all interfaces are tested appropriately and that the priorities for testing are set correctly. As of October 2008, the Bureau had begun efforts to update a master list it had developed in 2007, but it has not provided a date when this list will be completed. Without a completed master list, the Bureau cannot develop comprehensive plans and schedules for conducting systems integration testing that indicate how the testing of these interfaces will be prioritized. With the limited amount of time remaining before systems are needed for 2010 operations, the lack of comprehensive plans and schedules increases the risk that the Bureau may not be able to adequately test system interfaces, and that interfaced systems may not work together as intended. Bureau Has Conducted Limited End-to-End Testing as Part of the Dress Rehearsal, but Has Not Developed Testing Plans for Critical Operations: Although several critical operations underwent end-to-end testing in the Dress Rehearsal, others did not. As of December 2008, the Bureau had not established testing plans or schedules for end-to-end testing of the key operations that were removed from the Dress Rehearsal, nor has it determined when these plans will be completed. These operations include: * update/leave, * nonresponse follow-up, * enumeration of transitory locations, * group quarters enumeration, and: * field verification. The decreasing time available for completing end-to-end testing increases the risk that testing of key operations will not take place before the required deadline. Bureau officials have acknowledged this risk in briefings to the Office of Management and Budget. However, as of January 2009, the Bureau had not completed mitigation plans for this risk. According to the Bureau, the plans are still being reviewed by senior management. Without plans to mitigate the risks associated with limited end-to-end testing, the Bureau may not be able to respond effectively if systems do not perform as intended. Bureau Lacks Sufficient Executive-Level Oversight and Guidance for Testing: As stated in our testing guide and IEEE standards, oversight of testing activities includes both planning and ongoing monitoring of testing activities. Ongoing monitoring entails collecting and assessing status and progress reports to determine, for example, whether specific test activities are on schedule. In addition, comprehensive guidance should describe each level of testing and the types of test products expected. In response to prior recommendations, the Bureau took initial steps to enhance its programwide oversight; however, these steps have not been sufficient. For example, in June 2008, the Bureau established an inventory of all testing activities specific to all key decennial operations. However, the inventory has not been updated since May 2008, and officials have no plans for further updates. In another effort to improve executive-level oversight, the Decennial Management Division began producing (as of July 2008) a weekly executive alert report and has established (as of October 2008) a dashboard and monthly reporting indicators. However, these products do not provide comprehensive status information on the progress of testing key systems and interfaces. Further, the assessment of testing progress has not been based on quantitative and specific metrics. The lack of quantitative and specific metrics to track progress limits the Bureau's ability to accurately assess the status and progress of testing activities. In commenting on our draft report, the Bureau provided selected examples where they had begun to use more detailed metrics to track the progress of end-to-end testing activities. The Bureau also has weaknesses in its testing guidance. According to the Associate Director for the 2010 census, the Bureau did establish a policy strongly encouraging offices responsible for decennial systems to use best practices in software development and testing, as specified in level 2 of Carnegie Mellon's Capability Maturity Model® Integration. [Footnote 7] However, beyond this general guidance, there is no mandatory or specific guidance on key testing activities such as criteria for each level or the type of test products expected. The lack of guidance has led to an ad hoc--and, at times--less than desirable approach to testing. Implementation of Recommendations Could Help Ensure Key Testing Activities are Completed: In our report, we are making ten recommendations for improvements to the Bureau's testing activities. Our recommendations include finalizing system requirements and completing development of test plans and schedules, establishing a master list of system interfaces, prioritizing and developing plans to test these interfaces, and establishing plans to test operations removed from the Dress Rehearsal. In addition, we are recommending that the Bureau improve its monitoring of testing progress and improve executive-level oversight of testing activities. In written comments on the report, the department had no significant disagreements with our recommendations. The department stated that its focus is on testing new software and systems, not legacy systems and operations used in previous censuses. However, the systems in place to conduct these operations have changed substantially and have not yet been fully tested in a census-like environment. Consistent with our recommendations, finalizing test plans and schedules and testing all systems as thoroughly as possible will help to ensure that decennial systems will work as intended. In summary, while the Bureau's program offices have made progress in testing key decennial systems, much work remains to ensure that systems operate as intended for conducting an accurate and timely 2010 census. This work includes system, integration, and end-to-end testing activities. Given the rapidly approaching deadlines of the 2010 census, completing testing and establishing stronger executive-level oversight are critical to ensuring that systems perform as intended when they are needed. Mr. Chairman and members of the subcommittee, this concludes our statement. We would be pleased to respond to any questions that you or other members of the subcommittee may have at this time. Contacts and Staff Acknowledgements: If you have any questions about matters discussed in this testimony, please contact David A. Powner at (202) 512-9286 or pownerd@gao.gov or Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. Other key contributors to this testimony include Sher`rie Bacon, Barbara Collier, Neil Doherty, Vijay D'Souza, Elizabeth Fan, Nancy Glover, Signora May, Lee McCracken, Ty Mitchell, Lisa Pearson, Crystal Robinson, Melissa Schermerhorn, Cynthia Scott, Karl Seifert, Jonathan Ticehurst, Timothy Wexler, and Katherine Wulff. [End of section] Footnotes: [1] GAO, Information Technology: Significant Problems of Critical Automation Program Contribute to Risks Facing 2010 Census, [hyperlink, http://www.gao.gov/products/GAO-08-550T] (Washington, D.C.: Mar. 5, 2008). [2] GAO, High-Risk Series: An Update, [hyperlink, http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: Jan. 22, 2009). [3] GAO, Information Technology: Census Bureau Testing of 2010 Decennial Systems Can Be Strengthened, [hyperlink, http://www.gao.gov/products/GAO-09-262] (Washington, D.C.: Mar. 5, 2009). [4] For example, in the "update/leave" operation, after enumerators update addresses, they leave questionnaires at housing units; this occurs mainly in rural areas lacking street names, house numbers, or both. [5] GAO, Year 2000 Computing Crisis: A Testing Guide, [hyperlink, http://www.gao.gov/products/GAO/AIMD-10.1.21] (Washington, D.C.: Nov. 1, 1998) and IEEE Std. 12207-2008, Systems and Software Engineering-- Software Lifecycle Processes (Piscataway, N.J.: 2008). [6] Individual program offices manage individual system testing for the Dress Rehearsal, and integration testing is managed by the pairs of program offices whose interfaces are being tested. [7] Capability Maturity Model® Integration is intended to provide guidance for improving an organization's processes and the ability to manage the development, acquisition, and maintenance of products and services. The model uses capability levels to assess process maturity. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: