This is the accessible text file for GAO report number GAO-08-555T entitled 'Hospital Quality Data: Issues and Challenges Related to How Hospitals Submit Data and How CMS Ensures Data Reliability' which was released on March 6, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Committee on Finance, U.S. Senate: United States Government Accountability Office: GAO: For Release on Delivery: Expected at 2:00 p.m. EST: Thursday, March 6, 2008: Hospital Quality Data: Issues and Challenges Related to How Hospitals Submit Data and How CMS Ensures Data Reliability: Statement for the Record: Linda T. Kohn, Acting Director: Health Care: GAO-08-555T: GAO Highlights: Highlights of GAO-08-555T, a statement for the record for the Committee on Finance, U.S. Senate. Why GAO Did This Study: Hospitals submit data on a series of quality measures to the Centers for Medicare & Medicaid Services (CMS) and receive scores on their performance. CMS instituted the Reporting Hospital Quality Data for Annual Payment Update Program (APU program) to collect the quality data from hospitals and report their rates on the measures on its Hospital Compare Web site. For hospital quality data to be useful to patients and other users, they need to be reliable, that is, accurate and complete. The Deficit Reduction Act of 2005 directed CMS to implement a value- based purchasing program for Medicare that beginning in fiscal year 2009 would adjust payments to hospitals based on factors related to the quality of care they provide. This statement provides information on (1) how hospitals collect and submit quality data to CMS and (2) how CMS works to ensure the reliability of the quality data submitted. This statement is based primarily on Hospital Quality Data: HHS Should Specify Steps and Time Frame for Using Information Technology to Collect and Submit Data (GAO- 07-320, Apr. 25, 2007) and Hospital Quality Data: CMS Needs More Rigorous Methods to Ensure Reliability of Publicly Released Data (GAO- 06-54, Jan. 31, 2006). In preparing these reports, GAO conducted case studies of eight hospitals, and reviewed documents of, and interviewed officials at CMS. What GAO Found: GAO reported in April 2007 that the eight case study hospitals visited used six steps to collect and submit quality data, two of which (steps 2 and 3) involved complex abstraction—the process of reviewing and assessing all relevant pieces of information in a patient’s medical record to determine the appropriate value for each data element. The six steps were (1) identify patients for whom the quality data should be submitted, (2) locate needed information in the medical records, (3) determine the appropriate value for each data element, (4) transmit the data to CMS, (5) review reports to ensure acceptance of the data by CMS, and (6) supply copies of selected medical records to CMS for data validation. Several factors account for the complexity of the abstraction process (steps 2 and 3), including the content and organization of the medical record, the scope of information and clinical judgment required for certain data elements, and frequent changes by CMS in its data specifications. GAO’s case studies also showed that existing information technology (IT) systems help hospitals gather some quality data but are far from enabling hospitals to automate the abstraction process. GAO reported in January 2006 that CMS had processes for ensuring the accuracy of the quality data submitted by hospitals but had no ongoing process for ensuring completeness of these data. To check accuracy, one CMS contractor electronically checks the data as they are submitted to the clinical warehouse. Another contractor conducts an independent audit by comparing the quality data submitted by a hospital from the medical records for a sample of five patients per quarter for each hospital to the quality data that the contractor reabstracts from the same medical records. The data are deemed to be accurate if there is 80 percent or greater agreement between these two sets of results. However, GAO also reported that CMS’s determination as to whether hospitals met the accuracy standard was statistically uncertain for some hospitals because of the small number of records examined—five cases per quarter per hospital regardless of the hospital’s size. In 2006 GAO also reported that CMS did not have an ongoing process for assessing the completeness of quality data submitted by hospitals and recommended that CMS take steps to improve its processes for ensuring the accuracy and completeness of the hospital quality data. CMS agreed the process needed improvement. For fiscal year 2008 and subsequent years, CMS required that hospitals attest each quarter to the completeness and accuracy of their data. Further, in a 2007 report to Congress that lays out a plan to implement a value-based purchasing program, CMS recognized the need to redesign the data infrastructure and validation process to support a value-based purchasing program by, for example, increasing the number of patient medical records sampled from selected hospitals. To view the full product, including the scope and methodology, click on [Hyperlink, http://www.GAO-08-555T]. For more information, contact Linda T. Kohn at (202) 512-7114 or kohnl@gao.gov. [End of section] Mr. Chairman and Members of the Committee: I am pleased to have the opportunity to comment as requested on topics related to the Centers for Medicare & Medicaid Services's (CMS) Value- based Purchasing Program Implementation Plan. On November 21, 2007, CMS issued a report to Congress that lays out its plan to implement this program. The plan builds on the foundation of CMS's Annual Payment Update (APU) program that requires participating hospitals to submit data--referred to here as quality data--that are used to calculate hospital performance on measures of the quality of care provided in order to avoid a reduction in their full Medicare payment update each fiscal year.[Footnote 1] The vast majority of acute care hospitals treating Medicare patients choose to submit quality data each quarter to CMS, rather than accept a reduced annual payment update. In the APU program, each quality measure consists of a set of standardized data elements, which define the specific data that hospitals need to submit to CMS. Hospitals determine a value for each data element of a measure for patients--Medicare and non-Medicare--who have a medical condition covered by the APU program, that is, heart attack, heart failure, pneumonia, or surgery. The values for the data elements consist of numerical data and other administrative and clinical information that are obtained from the medical records of the patients. Hospitals submit their quality data electronically, over the Internet, to a clinical data warehouse operated by a CMS contractor. In order to inform the public about hospital quality, CMS posts on a public Web site--Hospital Compare--the performance scores that hospitals receive on the quality measures derived from the data they submit. For hospital quality data to be useful to patients and other users, the data need to be reliable--that is, both accurate and complete. If a hospital submits complete data, that is, data on all the cases that meet the specific inclusion criteria for eligible patients, but the data are not collected, or abstracted, from the patients' medical records accurately, the data will not be reliable. Similarly, if a hospital submits accurate data, but those data are incomplete because the hospital leaves out eligible cases, the data will not be reliable. Although the APU program was originally set to expire in 2007, the Deficit Reduction Act of 2005[Footnote 2] (DRA) made the APU program permanent. The act also raised the Medicare payment reduction[Footnote 3] and required the Secretary of Health and Human Services (HHS) to increase the number of measures for which hospitals participating in the APU program would have to provide data in order to receive their full Medicare payment update. Furthermore, DRA directed the Secretary to develop a plan to implement a value-based purchasing program for Medicare that beginning in fiscal year 2009 would adjust payments to hospitals based on factors related to the quality of care they provide. My statement today provides information on (1) how hospitals collect and submit quality data to CMS and (2) how CMS works to ensure the reliability of the quality data submitted by hospitals. My statement is based primarily on findings from our two reports on hospital quality data.[Footnote 4] In April 2007, we reported on case studies that we conducted at eight individual acute care hospitals, which were participating in the APU program, in order to obtain information about the processes they used to collect and submit quality data to CMS. As we noted in our report, because our evidence was limited to the eight case studies, we cannot generalize to acute care hospitals across the country. In January 2006, we reported on the reliability of publicly reported information on hospital quality obtained through the APU program that included a review of CMS documents and interviews with CMS officials. We also reviewed CMS's November 21, 2007, report to Congress which discusses options to implement a value-based purchasing program.[Footnote 5] All the work for our two reports on hospital quality data was done in accordance with generally accepted government auditing standards. We conducted this performance audit from February to March 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives. In summary, in April 2007, we reported that the eight case study hospitals we visited used six steps to collect and submit quality data, two of which involved complex abstraction--the process of reviewing and assessing all relevant pieces of information in a patient's medical record to determine the appropriate value for each data element. Several factors account for the complexity of the abstraction process, including the content and organization of the medical record, the scope of information and clinical judgment required for certain data elements, and frequent changes by CMS in its data specifications. Our case studies also showed that existing IT systems can help hospitals gather some quality data but are far from enabling hospitals to automate the abstraction process. In January 2006 we reported that CMS had a process in place to assess the accuracy of the APU program data submitted by hospitals, but had no ongoing process to assess the completeness of those data. Hospitals Use Six Steps to Collect and Submit Quality Data and IT Systems Can Help: In our April 2007 report,[Footnote 6] we found that whether patient information was recorded electronically, on paper, or as a mix of both, all eight of the case study hospitals collected and submitted their quality data by carrying out six sequential steps: (1) identify patients for whom the quality data should be submitted, (2) locate needed information in the medical records, (3) determine the appropriate value for each data element, (4) transmit the data to CMS, (5) review reports to ensure acceptance of the data by CMS, and (6) supply copies of selected medical records to CMS for data validation. The description by hospital officials of the processes they used to collect and submit quality data indicated that steps 2 and 3 (locating the relevant clinical information and determining appropriate values for the data elements), which involve the process of abstraction, were the most complex steps of the six identified, due to several factors. The first complicating factor was that the information abstractors [Footnote 7] needed to determine the correct data element values for a given patient was generally located in many different sections of the patient's medical record. Much of the clinical information needed was found in the sections of the medical record prepared by clinicians. Often the information in question, such as contraindications for aspirin or beta blockers, could be found in any of a number of places in the medical record where clinicians made entries. As a result, abstractors frequently had to read through multiple parts of the record--sometimes the entire record--to find the information needed to determine the correct value for just one data element. The second factor was related to the scope of the information required for certain data elements. Some of the data elements that the abstractors had to fill in represented a composite of related data and clinical judgment applied by the abstractor, not just a single discrete piece of information. Such composite data elements typically were governed by complicated rules for determining the clinical appropriateness of a specific treatment for a given patient. The third factor was the necessity abstractors at the case study hospitals faced to adjust to frequent changes in the data specifications set by CMS. For example, from fall 2004 through summer 2006, roughly every 3 months hospital abstractors had to stop and take note of what had changed in the data specifications and revamp their quality data collection procedures accordingly. Some of these changes reflected modifications in the quality measures themselves. CMS changed its schedule for issuing revisions to its data specifications from every 3 months to every 6 months. All the case study hospitals found that, over time, they had to increase the amount of staff resources devoted to abstracting quality data for the CMS quality measures, most notably as the number of measures on which they were submitting data expanded. Officials at the case study hospitals generally reported that the amount of staff time required for abstraction increased proportionately with the number of conditions for which they reported quality data. For example, as the hospitals began to report on the surgical quality measures, they found that the staff hours needed for this new set of quality measures were directly related to the number of patient medical records to be abstracted and the number of data elements collected. In other words, they found no "economies of scale" as they expanded the scope of quality data abstraction. Officials at the case study hospitals estimated that the amount of staff resources devoted to abstracting data for the CMS quality measures ranged from 0.7 to 2.5 full-time equivalents (FTE),[Footnote 8] typically registered nurses. On the other hand, officials at the case study hospitals reported that the demands that quality data collection and submission placed on their clinical staff resources were offset by the benefits that they derived from the resulting information on their clinical performance. Each one had a process for tracking changes in their performance over time and providing feedback to individual clinicians and reports to hospital administrators and trustees. We found that the existing IT systems in the case study hospitals could facilitate the collection of quality data, but that there were limits on the advantages that the systems could provide. IT systems, and the electronic records they support, offered hospitals two key benefits: (1) improving accessibility to and legibility of the medical record, and (2) facilitating the incorporation of CMS's required data elements into the medical record. On the other hand, the limitations that hospital officials reported in using existing IT systems to collect quality data stemmed from having a mix of paper and electronic systems; the prevalence of data recorded in IT systems as unstructured paragraphs of narrative or text, as opposed to discrete data fields reserved for specific pieces of information; and the inability of some IT systems to access related data stored on another IT system in the same hospital. All the case study hospitals were working to expand the scope and functionality of their IT systems, but most officials at the case study hospitals viewed full-scale automation of quality data collection and submission through implementation of IT systems as, at best, a long-term prospect. CMS Has Processes for Ensuring Accuracy but Has No Ongoing Process for Ensuring Completeness of Quality Data: We reported in January 2006[Footnote 9] that CMS had processes for assessing the accuracy of the quality data submitted by hospitals for the APU program, but had no ongoing process in place to assess the completeness of those data. To check accuracy, one CMS contractor electronically checks the data as they are submitted to the clinical warehouse. Another contractor conducts an independent audit by comparing the quality data submitted by a hospital from the medical records for a sample of five patients per quarter for each hospital to the quality data that the contractor reabstracts from the same medical records. The data are deemed to be accurate if there is 80 percent or greater agreement between these two sets of results, which allows the hospital to receive the full payment update from Medicare. However, we also reported that CMS's determination as to whether hospitals met the accuracy standard was statistically uncertain for some hospitals because of the small number of records examined--five per quarter per hospital, regardless of the hospital's size. Further, CMS did not have an ongoing process for assessing the completeness of quality data submitted by hospitals. Because of the purposes for which these data may be used, there could be an incentive for hospitals to selectively report data on cases that score well on the quality measures. In our 2006 report we recommended that CMS take steps to improve its processes for ensuring the accuracy and completeness of the hospital quality data and CMS agreed the process needed to be improved. For fiscal year 2008 and subsequent years it required that hospitals attest each quarter to the completeness and accuracy of their data, including the volume of data, submitted to the clinical warehouse.[Footnote 10] Further, in its 2007 report to Congress that lays out a plan to implement a value-based purchasing program, CMS recognized the need to redesign the data infrastructure and validation process to support a value-based purchasing program, by, for example, increasing the number of patient medical records sampled from selected hospitals. For more information regarding this statement, please contact Linda T. Kohn at (202) 512-7114 or kohnl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Krister Friday, Shannon Slawter Legeer, and Eric Peterson made key contributions to this statement. [End of section] Footnotes: [1] The Medicare Prescription Drug, Improvement, and Modernization Act of 2003 created a financial incentive for hospitals, and CMS established the Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU) Program (the "APU program") to implement that incentive. See Pub. L. No. 108-173, § 501(b), 117 Stat. 2066, 2289-90. Most acute care hospitals (i.e., those paid under the Medicare inpatient prospective payment system) receive an annual payment update that increases the standardized payment amount that Medicare pays them per patient, based on projected increases in hospital operating expenses. [2] See Pub. L. No. 109-171, § 5001(a), 120 Stat. 4, 28-29. [3] The magnitude of the reduction in the annual payment update for hospitals not submitting the quality data rose from 0.4 percentage points to 2 percentage points, starting in fiscal year 2007. [4] See GAO, Hospital Quality Data: HHS Should Specify Steps and Time Frame for Using Information Technology to Collect and Submit Data, GAO-07-320 (Washington, D.C.: Apr. 25, 2007) and Hospital Quality Data: CMS Needs More Rigorous Methods to Ensure Reliability of Publicly Released Data, GAO-06-54 (Washington, D.C.: Jan. 31, 2006). [5] Centers for Medicare & Medicaid Services, Report to Congress: Plan to Implement a Medicare Hospital Value-Based Purchasing Program (Nov. 21, 2007). [6] See GAO-07-320. [7] Throughout this statement, we use the term "abstractor" to indicate hospital staff who are trained to follow a detailed protocol in order to extract specified information in a consistent fashion from the medical records of multiple patients. [8] These represent the FTEs devoted specifically to quality data collection and submission. Hospital officials noted that additional FTEs were involved in analyzing the hospital's performance on the quality measures and achieving improvements through changes in clinical process and educational efforts with the hospital's clinicians. [9] See GAO-06-54. [10] See 72 Fed. Reg. 47130, 47364 (Aug. 22, 2007). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "Subscribe to Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: