Hospital Quality Data:

CMS Needs More Rigorous Methods to Ensure Reliability of Publicly Released Data

GAO-06-54: Published: Jan 31, 2006. Publicly Released: Feb 24, 2006.

Additional Materials:

Contact:

Cynthia A. Bascetta
(202) 512-7207
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Medicare Modernization Act of 2003 directed that hospitals lose 0.4 percent of their Medicare payment update if they do not submit clinical data for both Medicare and non-Medicare patients needed to calculate hospital performance on 10 quality measures. The Centers for Medicare & Medicaid Services (CMS) instituted the Annual Payment Update (APU) program to collect these data from hospitals and report their rates on the measures on its Hospital Compare Web site. For hospital quality data to be useful to patients and other users, they need to be reliable, that is, accurate and complete. GAO was asked to (1) describe the processes CMS uses to ensure the accuracy and completeness of data submitted for the APU program, (2) analyze the results of CMS's audit of the accuracy of data from the program's first two calendar quarters, and (3) describe processes used by seven other organizations that assess the accuracy and completeness of clinical performance data.

CMS has contracted with an independent medical auditing firm to assess the accuracy of the APU program data submitted by hospitals, but has no ongoing process in place to assess the completeness of those data. CMS's independent audit checks accuracy by comparing the quality data submitted by hospitals from the medical records for a sample of five patients per calendar quarter for each hospital to the quality data that the contractor has reabstracted from the same records. The data are deemed to be accurate if there is 80 percent or greater agreement between these two sets of results. CMS has established no ongoing process to check data completeness. For the payment updates for fiscal years 2005 and 2006, CMS compared the number of cases submitted by a hospital to the number of Medicare claims that hospital submitted. However, these analyses did not address non-Medicare patient records, and the approach that CMS took in these analyses was not capable of detecting incomplete data for all hospitals. Although GAO found a high overall baseline level of accuracy when it examined CMS's assessment of the data submitted for the first two quarters of the APU program, the results are statistically uncertain for up to one-third of hospitals, and a baseline level of data completeness cannot be determined. The median accuracy score of 90 to 94 percent--depending on the calendar quarter and measures used--was well above the 80 percent accuracy threshold set by CMS, and about 90 percent of hospitals met or exceeded that threshold for both the first and the second calendar quarters of 2004. However, for approximately one-fourth to one-third of all the hospitals that CMS assessed for accuracy, the statistical margin of error for their accuracy score included both passing and failing accuracy levels. Consequently, for these hospitals, the small number of cases that CMS examined was not sufficient to establish with statistical certainty whether they met the accuracy threshold set by CMS. With respect to completeness of data, CMS did not assess the extent to which all hospitals submitted data on all eligible patients, or a representative sample thereof, for the two baseline quarters. As a result, there were no data from which to derive an assessment of the baseline level of completeness of the quality data that hospitals submitted for the APU program. Other reporting systems that collect clinical performance data have adopted a range of activities to ensure data accuracy and completeness, which include some methods employed by all, such as checking the data electronically to identify missing data. Officials from some of the other reporting systems and an expert in the field stressed the importance of including an independent audit in the methods used by organizations to check data accuracy and completeness. Most of the other reporting systems incorporate three methods into their process that CMS does not use in its independent audit. Specifically, most include an on-site visit in their independent audit, focus their audits on a selected number of facilities, and review a minimum of 50 patient medical records during the audit.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: In order for CMS to help ensure the reliability of the quality data it uses to produce information on hospital performance, the CMS Administrator should, focusing on the subset of hospitals for which it is statistically uncertain if they met CMS's accuracy threshold in one or more previous quarters, increase the number of patient records reabstracted by the Clinical Data Abstraction Center in a subsequent quarter so that the proportion of hospitals with statistically uncertain results is reduced.

    Agency Affected: Department of Health and Human Services: Centers for Medicare and Medicaid Services

    Status: Closed - Implemented

    Comments: In its August 2009 final rule to implement FY 2010 changes in the Medicare Inpatient Prospective Payment System, CMS alerted hospitals that starting in 2010 it will change its procedure for checking the accuracy of quality data that hospitals submit quarterly in order to receive their full annual payment update. Rather than reviewing 5 patient records per quarter from every hospital, CMS will henceforth review 12 patient records from a randomly selected sample of 800 hospitals. We had recommended that CMS increase the number of patient records it reviewed for hospitals with statistically uncertain results. Although the procedure that CMS adopted differs somewhat from the approach that we proposed, it addresses the problem we identified through an alternative mechanism that accomplishes the same result -- reducing an excessive degree of statistical uncertainty that arose from insufficiently large samples by focusing the data validation effort on a selected subset of hospitals in any given year. CMS noted that this increase in sample size would provide more reliable estimates of the accuracy of the hospital submitted data.

    Recommendation: In order for CMS to help ensure the reliability of the quality data it uses to produce information on hospital performance, the CMS Administrator should require hospitals to certify that they took steps to ensure that they submitted data on all eligible patients, or a representative sample thereof.

    Agency Affected: Department of Health and Human Services: Centers for Medicare and Medicaid Services

    Status: Closed - Implemented

    Comments: In its August 2009 final rule to implement FY 2010 changes in the Medicare Inpatient Prospective Payment System, CMS alerted hospitals that starting in 2010 they would be required to electronically acknowledge once a year the completeness and accuracy of the quality data that they submit quarterly in order to receive their full annual payment update. The rationale CMS provided for this requirement included a reference to our recommendation that CMS require hospitals to certify that the quality data they submitted were complete. CMS stated that this requirement would help to ensure that hospitals implemented the steps needed to guarantee the completeness and accuracy of the quality data that they submitted.

    Recommendation: In order for CMS to help ensure the reliability of the quality data it uses to produce information on hospital performance, the CMS Administrator should assess the level of incomplete data submitted by hospitals for the APU program to determine the magnitude of underreporting, if any, in order to refine how completeness assessments may be done in future reporting efforts.

    Agency Affected: Department of Health and Human Services: Centers for Medicare and Medicaid Services

    Status: Closed - Not Implemented

    Comments: A CMS official reported in a an email sent 7/26/2007 that CMS has addressed our recommendation on assessing the level of incomplete data through analyses that compared hospital quality data submissions with each hospital's Medicare claims submissions. In an email on 1/5/2009, the same official reported that CMS plans to conduct similar analyses in 2009. The official acknowledged that Medicare claims do not include all the patients for whom hospitals are required to submit data. CMS therefore plans to ask hospitals to self report their total number of Medicare and non-Medicare cases.

    Jul 23, 2014

    Jul 16, 2014

    Jul 15, 2014

    Jul 10, 2014

    Jun 30, 2014

    Jun 25, 2014

    Jun 24, 2014

    Jun 23, 2014

    Jun 18, 2014

    Looking for more? Browse all our products here