GAO-11-720, Medicare Physician Feedback Program: CMS Faces Challenges with Methodology and Distribution of Physician Reports


This is the accessible text file for GAO report number GAO-11-720 
entitled 'Medicare Physician Feedback Program: CMS Faces Challenges 
with Methodology and Distribution of Physician Reports' which was 
released on August 12, 2011. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

August 2011: 

Medicare Physician Feedback Program: 

CMS Faces Challenges with Methodology and Distribution of Physician 
Reports: 

GAO-11-720: 

GAO Highlights: 

Highlights of GAO-11-720, a report to congressional committees. 

Why GAO Did This Study: 

The Medicare Improvements for Patients and Providers Act of 2008 
(MIPPA) directed the Department of Health and Human Services (HHS) to 
develop a program to give physicians confidential feedback on the 
resources used to provide care to Medicare beneficiaries. In response, 
HHS’s Centers for Medicare & Medicaid Services (CMS) has established 
and implemented the Physician Feedback Program by distributing 
feedback reports to an increasing number of physicians that provided 
data on resources used and the quality of care. MIPPA mandated that 
GAO conduct a study of this program. To address this mandate, GAO 
identified (1) methodological challenges CMS faces in developing 
feedback reports and approaches CMS has tested to address them and (2) 
challenges CMS faces in distributing feedback reports and CMS’s plans 
to address them. GAO interviewed CMS officials and representatives 
from the program contractor and reviewed relevant documentation. 

What GAO Found: 

CMS faces challenges incorporating resource use and quality measures 
for physician feedback reports that are meaningful, actionable, and 
reliable. CMS had difficulty measuring the resources used by 
physicians to treat specific episodes of an illness, such as a stroke 
or a hip fracture, and the quality measures it used in the program’s 
most recent phase applied to a limited number of physicians. CMS must 
also make decisions to address several other methodological challenges 
with developing feedback reports: how to account for differences in 
beneficiary health status, how to attribute beneficiaries to 
physicians, how to determine the minimum number of beneficiaries a 
physician needs to treat to receive a report, and how to select 
physicians’ peer groups for comparison. These decisions involve trade-
offs; for example, a higher minimum case size requirement increases 
the reliability of the information in the reports, but it decreases 
the number of physicians eligible to receive one. While CMS has tested 
different approaches to measuring and comparing physician performance, 
methodological difficulties remain in developing feedback reports. 

CMS also faced challenges distributing feedback reports to physicians 
that its plans for improvement may not entirely address. In the most 
recent phase of the program, about 82 percent of physicians in CMS’s 
sample were not eligible to receive a report after CMS’s 
methodological decisions were applied (see figure). CMS plans to make 
a number of methodological changes in the next phase, but 
significantly increasing eligibility will continue to be challenging. 
The electronic distribution of feedback reports also presented 
multiple challenges that resulted in few physicians accessing their 
electronic reports in the most recent phase. Factors that may have 
contributed to this low access rate include CMS’s difficulty in 
obtaining accurate contact information, burdensome methods for 
electronic distribution, and lack of a strong incentive for physicians 
to review their reports. CMS conducted limited follow-up with 
physicians for whom feedback reports were produced. CMS plans to use a 
new distribution method in a four-state region in the next reporting 
phase. 

Figure: Number of Physicians Excluded from Receiving Feedback Reports, 
2010: 

[Refer to PDF for image: illustration] 

All sampled physicians: 
CMS began with a sample of 9,189 individual physicians affiliated with 
the 36 physician groups. 

Methodological requirements for a feedback report: 

At least 30 attributed beneficiaries for the resource use measures – 
only 2,205 physicians met this requirement; 

At least 11 attributed beneficiaries for at least one of the quality
measures – only 2,661 physicians met this requirement; 

Only 1,733 physicians met both requirements; 

At least 30 individual physicians practicing in the same medical 
specialty and geographic area for a peer group – only 1,645 physicians
met this requirement. 

Source: GAO analysis of CMS and contractor data. 

[End of figure] 

What GAO Recommends: 

GAO is recommending that CMS use methodological approaches that 
increase physician eligibility for reports, statistically analyze the 
impact of its methodological decisions on report reliability, identify 
and address factors that may have prevented physicians from reading 
reports, and obtain input from a sample of physicians on the 
usefulness and credibility of reports. CMS concurred with these 
recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-11-720] or key 
components. For more information, contact James C. Cosgrove at (202) 
512-7114 or cosgrovej@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

CMS Tested Various Approaches but Still Faces Several Methodological 
Challenges in Developing Physician Feedback Reports: 

CMS's Plans for Improvement May Not Fully Address Challenges in 
Distributing Reports to Physicians: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Comments from the Department of Health and Human Services: 

Appendix II: GAO Contact and Staff Acknowledgments: 

Table: 

Table 1: Percentage of Individual Physicians and Physician Groups 
Eligible for Select Resource Use and Quality Performance Measures on 
Phase II Feedback Reports, 2010: 

Figure: 

Figure 1: Number of Individual Physicians Excluded from Phase II 
Feedback Reports Based on CMS's Methodological Criteria, 2010: 

Abbreviations: 

AHRQ: Agency for Healthcare Research and Quality: 

BCSSI: Buccaneer Computer Systems and Services, Inc. 

CBO: Congressional Budget Office: 

CMS: Centers for Medicare & Medicaid Services: 

CTS: Community Tracking Study: 

E&M: evaluation and management: 

GEM: Generating Medicare Physician Quality Performance Measurement 
Results: 

HCC: Hierarchical Condition Categories: 

HEDIS®: Healthcare Effectiveness Data and Information Set: 

HHS: Department of Health and Human Services: 

IACS: Individuals Authorized Access to CMS Computer Services: 

MAC: Medicare Administrative Contractor: 

MedPAC: Medicare Payment Advisory Commission: 

MIPPA: Medicare Improvements for Patients and Providers Act of 2008: 

NCQA: National Committee for Quality Assurance: 

NPPES: National Plan and Provider Enumeration System: 

NQF: National Quality Forum: 

PECOS: Provider Enrollment, Chain, and Ownership System: 

PPACA: Patient Protection and Affordable Care Act: 

PQRS: Physician Quality Reporting System: 

UPIN: Unique Physician Identification Number: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

August 12, 2011: 

The Honorable Max Baucus: 
Chairman: 
The Honorable Orrin G. Hatch: 
Ranking Member: 
Committee on Finance: 
United States Senate: 

The Honorable Fred Upton: 
Chairman: 
The Honorable Henry A. Waxman: 
Ranking Member: 
Committee on Energy and Commerce: 
House of Representatives: 

The Honorable Dave Camp: 
Chairman: 
The Honorable Sander M. Levin: 
Ranking Member: 
Committee on Ways and Means: 
House of Representatives: 

In recent years, we and other federal fiscal experts--including the 
Congressional Budget Office (CBO) and the Medicare Trustees--have 
noted the rise in Medicare spending and the serious long-term 
financial challenges the program faces.[Footnote 1] Physicians play a 
central role in the generation of Medicare expenditures both through 
the services they provide and the services they order, including 
hospital admissions, diagnostic tests, and referrals to other 
physicians. There is evidence that not all of these services may be 
necessary or appropriate, and that greater spending does not 
necessarily result in better health outcomes. As a result, 
policymakers have been exploring methods to reduce costs and encourage 
physicians to practice efficiently--that is, to provide and order only 
those services that are necessary, sufficient, and appropriate to meet 
a beneficiary's health care needs. 

Efficiency may be encouraged by physician profiling, which measures 
and compares a physician's performance to a benchmark, such as the 
performance of his or her peers. Certain public and private health 
care purchasers routinely profile physicians in their networks and use 
the results for a number of purposes, including developing physician 
"report cards" or feedback reports and placing physicians in tiered 
networks that can be used to steer patients toward the most efficient 
providers. We and others have recommended that the Centers for 
Medicare & Medicaid Services (CMS), the agency within the Department 
of Health and Human Services (HHS) that administers the Medicare 
program, profile physicians and provide them with feedback on their 
use of health care resources to help identify and reduce overuse of 
Medicare services.[Footnote 2] In addition to profiling physicians on 
the resources used to provide care to Medicare beneficiaries, they can 
also be profiled on the quality of that care. Some specialty societies 
have called for the inclusion of quality measures in feedback reports 
and cautioned that focusing solely on costs could create a 
disincentive to providing appropriate, high-quality care. 

The Medicare Improvements for Patients and Providers Act of 2008 
(MIPPA) required HHS to establish and begin implementing by January 1, 
2009, a Physician Feedback Program that would include distribution of 
confidential feedback reports to physicians on the resources used to 
provide care to Medicare beneficiaries.[Footnote 3] MIPPA gave HHS the 
flexibility to apply the program to certain types of physicians, such 
as those who treat conditions that have high costs, and also provided 
flexibility on whether to provide reports to physician groups and 
whether to include information on quality. Because developing feedback 
reports requires a number of methodological decisions, such as 
selecting performance measures that accurately reflect physicians' 
resource use and quality of care, CMS has implemented the program in 
phases by testing different approaches for developing feedback reports 
and distributing reports to a small number of physicians and physician 
groups.[Footnote 4] 

The Patient Protection and Affordable Care Act (PPACA), which was 
enacted in 2010, directed HHS to adjust Medicare payments to 
physicians based on the quality of care provided compared to the cost 
using a "value-based payment modifier."[Footnote 5] HHS is directed to 
begin paying a limited group of physicians and physician groups 
differentially using the payment modifier on January 1, 2015, and all 
physicians and physician groups by January 1, 2017.[Footnote 6] The 
law also states that HHS is to coordinate the Physician Feedback 
Program with the value-based payment modifier.[Footnote 7] CMS has 
said that it intends to use the quality and cost measures from the 
Physician Feedback Program to develop the payment modifier and plans 
to distribute at least one feedback report to physicians before paying 
them differentially based on their performance. 

MIPPA mandated that GAO conduct a study of the Physician Feedback 
Program and report on our findings no later than March 1, 2011. 
[Footnote 8] To respond to this requirement, we conducted a series of 
briefings for congressional staff on our preliminary findings 
beginning in February 2011. This report contains information we 
provided during those briefings as well as additional information. 
Specifically, we (1) identified methodological challenges CMS faces in 
developing physician feedback reports and the approaches CMS has 
tested to address them and (2) identified challenges CMS faces in 
distributing physician feedback reports and CMS's plans to address 
them. 

To address these objectives, we interviewed relevant CMS officials and 
representatives from Mathematica Policy Research, Inc. (Mathematica), 
the contractor that assisted with the development and testing of 
different methodologies and distribution methods for the Physician 
Feedback Program.[Footnote 9] We reviewed internal agency reports and 
relevant studies, including reports by CMS contractors and the 
Medicare Payment Advisory Commission (MedPAC), summaries of comments 
provided by physicians who received feedback reports from CMS, and 
public comments submitted by medical specialty societies and other 
stakeholders in response to the portion of CMS's 2011 proposed 
physician fee schedule rule related to the Physician Feedback Program. 
[Footnote 10] In addition, we attended a CMS listening session on the 
Physician Feedback Program, at which representatives of medical 
specialty societies and other stakeholders commented on the 
methodological approaches CMS is considering in developing feedback 
reports. We limited our study to challenges with feedback report 
methodology and distribution as our initial audit work indicated that 
these were the primary challenges faced by the agency in its 
implementation of the Physician Feedback Program. Our work is based on 
the most current information available as of June 7, 2011. 

We conducted this performance audit from June 2010 through August 2011 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Background: 

Physicians can be profiled on the health care they provide to Medicare 
beneficiaries using measures in two performance dimensions: the 
resources used to provide care to beneficiaries and the quality of 
that care. CMS has established goals and made progress in developing 
its Physician Feedback Program. 

Resource Use Measures: 

Resource use can be measured using two methods: the per capita method 
and the per episode method. The per capita method measures the 
resources used by a physician to treat his or her Medicare 
beneficiaries over a fixed period of time. By definition, it is a 
comprehensive measure of a physician's practice patterns because it 
includes all health care resources used and is generally considered 
more straightforward than the per episode method to measure and 
understand. 

The per episode method measures the resource use associated with 
treating a specific episode of an illness in a beneficiary--for 
example, a stroke or a hip fracture. An episode of care may refer to 
all services related to a health condition with a given diagnosis from 
a patient's first encounter with a health care provider through the 
completion of the last encounter related to that condition, including 
postacute services such as home health, skilled nursing, and 
rehabilitation.[Footnote 11] Since this method provides condition- 
specific results, it may provide more useful, or "actionable," 
feedback to physicians. Per episode costs are generally considered 
more difficult to measure than per capita costs since it can be 
challenging to determine whether a particular health care service 
should be grouped to one episode of care or another. Per episode costs 
may be determined using "episode groupers," which are software 
programs that use diagnosis codes to assign claims to clinically 
distinct episodes of care.[Footnote 12] 

Using both the per capita and per episode methods may more fully 
capture differences in resource use among physicians. For example, in 
a 2006 report, MedPAC found that beneficiaries in Miami had 
significantly lower per episode costs for coronary artery disease than 
beneficiaries in Minneapolis, suggesting that Miami physicians were 
providing more efficient care for coronary artery disease. However, 
MedPAC noted that because the beneficiaries in Miami had more episodes 
of care for this disease, physicians in Miami actually used more 
health care resources in total to treat their coronary artery disease 
beneficiaries than physicians treating similar patients in 
Minneapolis. In this case, the per capita method and the per episode 
method together would provide a more complete picture of physicians' 
resource use than either method by itself. 

Quality Measures: 

Health care quality measures can be used to evaluate how well health 
care is delivered, and information obtained from such measures can 
promote accountability among physicians. Quality measures can be 
classified as process or outcome measures.[Footnote 13] Process 
measures assess whether appropriate clinical practices, such as 
screening and diagnosis, were followed. An example of a process 
measure is whether a patient with high blood pressure received 
appropriate medication. Outcome measures assess a patient's health 
status after receiving health care services. An example of an outcome 
measure is tracking the percentage of patients who were diagnosed with 
high blood pressure and whose blood pressure was adequately controlled 
during the measurement year. 

Efforts are under way by a range of organizations, including CMS and 
the National Committee for Quality Assurance (NCQA), to develop 
measures of physician quality, and by the National Quality Forum (NQF) 
to endorse the quality measures developed by others.[Footnote 14] For 
example, NCQA created the Healthcare Effectiveness Data and 
Information Set (HEDIS)®, which is a tool used by over 90 percent of 
health plans in the nation and includes measures of both health plan 
and physician performance.[Footnote 15] CMS has developed the 
Physician Quality Reporting System (PQRS), which is a quality 
reporting program that provides an incentive payment to professionals 
who satisfactorily report data on quality measures for covered 
professional services furnished during a specified reporting period. 
CMS also contracted with Masspro, a quality improvement organization 
for Massachusetts, to calculate performance rates for the Generating 
Medicare Physician Quality Performance Measurement Results (GEM) 
project. The GEM project used 2006 and 2007 Medicare administrative 
claims data to generate performance rates for 12 process measures that 
were drawn from HEDIS®, such as persistence of beta blocker treatment 
after a heart attack. 

CMS Physician Feedback Program Goals and Development: 

CMS established the Physician Feedback Program in 2008 with the goal 
of encouraging higher-quality and more efficient medical practice and 
creating a transparent process for developing meaningful, actionable, 
and fair physician performance indicators that could later be used in 
CMS's value-based purchasing initiative. Feedback reports can help 
ensure quality health care and control costs in three ways. First, the 
feedback reports are intended to be educational by providing useful 
information to physicians on how their resource use and quality of 
care compare to their peers'. Second, the reports are intended to be 
actionable by helping physicians identify and develop strategies for 
improving quality and reducing costs in their practices. Third, the 
reports are intended to help physicians become familiar with the 
resource use and quality measures that the agency plans to use to 
adjust their Medicare reimbursement under the value-based payment 
program. CMS intends to distribute at least one feedback report to 
physicians before paying them differentially under the value-based 
modifier. 

CMS has implemented the program in phases by distributing feedback 
reports to an increasing number of physicians in selected metropolitan 
areas. In each phase, CMS conducted pretesting to obtain physicians' 
reactions to the methodology and format of mock feedback reports, 
distributed feedback reports populated with actual performance data, 
and followed up with a sample of the profiled physicians to obtain 
their input on the reports. In Phase I, CMS distributed feedback 
reports to 239 physicians who practiced in one of 12 metropolitan 
areas.[Footnote 16] These reports were distributed in April and August 
2009, and included information about physicians' resource use but not 
their quality of care. In Phase II, CMS expanded the program to 
distribute feedback reports to 36 physician groups and to 1,641 
individual physicians who practiced within these groups from the same 
12 metropolitan areas used in Phase I.[Footnote 17] Phase II reports 
were produced in November 2010, and included resource use measures and 
selected quality measures as well as information on beneficiaries' 
hospital admissions. In addition, the reports to physician groups 
included hospitalization rates for ambulatory care sensitive 
conditions--acute conditions for which effective outpatient care could 
have prevented complications or more severe disease. The reports also 
contained the average per capita costs of treating Medicare 
beneficiaries, as well as per capita costs by specific categories of 
service, such as laboratory tests and imaging services. In addition, 
the reports provided summary information about the average annual cost 
of treating a subset of Medicare beneficiaries with selected common 
chronic conditions: congestive heart failure, chronic obstructive 
pulmonary disease, coronary artery disease, diabetes, and prostate 
cancer. 

CMS plans to continue to develop feedback reports and distribute them 
to an increasing number of physicians and physician groups. It plans 
to distribute Phase III reports to about 20,000 physicians in late 
2011, and intends to provide feedback reports to all physicians and 
physician groups by 2017. 

CMS Tested Various Approaches but Still Faces Several Methodological 
Challenges in Developing Physician Feedback Reports: 

CMS faces challenges in selecting resource use and quality performance 
measures that make feedback reports meaningful, reliable, and 
actionable. In addition, the agency faces trade-offs in making other 
key methodological decisions concerning risk adjustment, attribution 
of beneficiaries to physicians, minimum case size, and peer group 
selection. While CMS has tested different approaches to developing 
feedback reports, challenges remain in making methodological decisions 
that will enable CMS to accomplish its program goals. 

CMS Tested Approaches to Measuring Physician Resource Use and Quality 
but Still Faces Challenges: 

Measuring resource use. CMS intends to use both per capita and per 
episode methods to measure physicians' resource use, but it faces 
particular challenges in determining per episode costs for the 
Medicare population. In Phase I, CMS tested two commercially available 
episode groupers, but found that these groupers had the following 
shortcomings when used with Medicare claims data: 

* Because of the prevalence of comorbidities in the Medicare 
population, a beneficiary can be treated for several different 
conditions concurrently, and it was difficult for the groupers to 
determine which services belonged with a given episode. 

* Because diagnosis coding used for different Medicare claim types was 
inconsistent, claims from different sources were not always linked to 
the same episode of care, even when they appeared to be clinically 
related. For example, hospital, physician, and skilled nursing 
facility claims have slightly different diagnostic information. 

* Because it was difficult to identify the appropriate beginning and 
end of an episode involving a chronic condition, the commercial 
groupers did not work well to create episodes of care for the Medicare 
population, since a significant portion of Medicare beneficiaries have 
chronic conditions. 

CMS concluded that per episode measurements included in Phase I 
reports were inaccurate, and discontinued use of the commercial 
groupers. Some medical specialty societies and other stakeholders 
commended this decision. 

CMS intends for shortcomings to be addressed by the Medicare-specific 
episode grouper under development. In September 2010, CMS awarded four 
contracts to develop a Medicare-specific episode grouper.[Footnote 18] 
CMS plans to select a grouper developed under one of these four 
contracts for future feedback reports.[Footnote 19] However, it is not 
clear that all the problems identified with the commercial groupers 
can be solved by a Medicare-specific grouper and the timeline for its 
development is challenging. 

In Phase II feedback reports, CMS elected to provide information on 
resource use for beneficiaries with five high-cost, high-volume 
chronic conditions. Because episode measures were not available, it 
used per capita measures as proxies for per episode costs for patients 
with diabetes, congestive heart failure, coronary artery disease, 
chronic obstructive pulmonary disease, and prostate cancer. These 
proxies included all the resources used to treat beneficiaries with 
these select chronic conditions, regardless of whether the resource 
use was related to that specific condition. CMS officials stated that 
these proxies were adequate substitutes for episode-based cost 
measures for these chronic conditions. In Phase III reports, CMS plans 
to provide per capita information on subgroups with the same chronic 
conditions as in Phase II with the exception of prostate cancer. 
[Footnote 20] 

Measuring quality. CMS faces the challenge of incorporating into its 
feedback reports quality measures that are available, apply to 
specialists, and provide information on patient outcomes.[Footnote 21] 
Phase I reports did not contain quality measures. In Phase II, CMS 
included 12 GEM measures in feedback reports.[Footnote 22] These 
measures have the advantage of being readily available because they 
are based on claims data.[Footnote 23] For Phase III, CMS is 
considering 28 claims-based quality measures, which are endorsed by 
NQF. These 28 measures, most of which are HEDIS® measures, were vetted 
by an interagency committee composed of medical officers and other 
internal experts who reviewed the specifications of each measure, 
including whether the measure was an appropriate reflection of 
physician care and whether it was evidence based. 

While a number of quality measures available to CMS for use in 
feedback reports are applicable to primary care physicians, there are 
fewer measures for specialists. For example, the GEM measures used in 
Phase II reports are only applicable to primary care physicians and a 
limited number of specialists, such as cardiologists. In addition, the 
28 measures CMS is considering for Phase III reports are, as a whole, 
mostly applicable to primary care physicians, although individual 
measures apply to certain specialists. Some stakeholders have 
encouraged CMS to work with specialty societies to develop adequate 
quality measures. CMS officials stated that while the agency is 
willing to work with these specialty societies to ensure that selected 
measures accurately reflect physicians' practices, CMS prefers to use 
NQF-endorsed quality measures and many of the measures that specialty 
societies have created have not yet achieved NQF endorsement. In 
addition, CMS anticipates using PQRS measures that are applicable to 
specialists, but according to CMS officials, it has not done so yet 
because of limitations with the PQRS program, such as low physician 
participation rates.[Footnote 24] CMS officials said that PQRS has 
measures that are applicable to every type of physician, and the 
agency is working to increase physician participation in PQRS, which 
is currently voluntary. They expect program participation rates to 
increase when, in 2013, CMS plans to begin penalizing physicians who 
fail to report PQRS measures.[Footnote 25] 

In addition, the GEM measures CMS used for Phase II reports are 
process measures, which show whether a physician followed generally 
accepted recommendations for clinical practice but may not reflect the 
impact of the health care services on the health status of a 
beneficiary. CMS officials have stated that although there is a need 
to evaluate physician quality of care based on outcome measures, there 
are currently few suitable measures. NQF has also stated that there is 
a need to develop additional outcome quality measures, and funding of 
$75 million is authorized for this in each of fiscal years 2010 
through 2014. In addition, CMS officials stated that PQRS contains a 
number of clinical outcome measures, and it is likely that moving 
forward physician feedback reports will include these PQRS outcome 
measures. 

Other Key Methodological Decisions Involve Trade-offs, and CMS Has 
Tested Different Approaches to Inform These Decisions: 

Determining risk adjustment factors. CMS faces trade-offs in deciding 
which factors to use for risk adjustment, which accounts for 
differences outside the physician's control, such as beneficiary 
health status. Because sicker beneficiaries are expected to use more 
health care resources than healthier beneficiaries, the health status 
of physicians' beneficiaries must be taken into account to make 
meaningful comparisons among physicians. Without risk adjusting 
resource use, physicians who treat sicker beneficiaries could appear 
to use resources less efficiently than their peers in their feedback 
reports. CMS used the Hierarchical Condition Categories (HCC) model to 
risk adjust per capita resource use in the Phase I and Phase II 
feedback reports. This model was originally developed for risk 
adjustment in Medicare managed care. The HCC model used in Phase II 
feedback reports is a method of adjusting for the expected resource 
use of Medicare beneficiaries based on the health conditions they 
experienced during the previous year and other factors, such as gender 
and age. 

There are trade-offs involved in determining whether to use a 
prospective or concurrent risk adjustment model. A prospective model 
uses risk factors from a previous period to predict physicians' 
spending for a future period. A prospective model works well for some 
health conditions, such as chronic conditions, which are accurate 
predictors of health spending not only in the current year, but also 
in future years. Conversely, a concurrent model uses factors from the 
current period to adjust health spending for that period. The 
concurrent model may risk adjust health care costs incurred in the 
current year more fully by including acute conditions, such as a 
broken leg, as well as acute exacerbations of chronic illnesses, such 
as hospitalizations resulting from uncontrolled diabetes.[Footnote 26] 
However, it may be appropriate to categorize the expenditures 
associated with some complications as part of the physician's 
performance, as opposed to factors outside of the physician's control 
that require risk adjustment. For example, if a beneficiary needed to 
be hospitalized because of poorly managed diabetes, it could be 
appropriate to hold the physician accountable for those costs. CMS 
used a prospective model in Phase II feedback reports. 

CMS must also decide which factors, if any, should be added to the HCC 
model. Although CMS officials believe the HCC risk adjustment model 
adequately risk adjusted per capita costs, some stakeholders have 
questioned CMS's use of the HCC model and have urged CMS to adjust for 
additional factors that affect costs that CMS did not include. These 
factors include some socioeconomic indicators, patient noncompliance, 
and care setting. Some medical specialty societies and other 
stakeholders have stated that if CMS does not risk adjust physician 
resource use adequately, physicians could be discouraged from treating 
atypical or disadvantaged populations that may be more costly to 
treat. Although risk adjusting for additional factors could help 
address these concerns, there may be a case for not including them. 
For example, noncompliance with physicians' instructions may suggest 
that physicians have not adequately educated their patients on the 
importance of compliance. CMS officials also explained that they do 
not want to adjust for factors that can provide meaningful information 
about differences in practice patterns. CMS officials said that they 
plan to continue using the HCC model to risk adjust per capita costs 
in Phase III. 

Selecting an attribution method. CMS faces trade-offs in determining 
how to assign responsibility, or "attribute" beneficiaries' care to 
physicians, in a way that promotes program goals. Program goals 
include maximizing the number of physicians eligible for feedback 
reports and encouraging care coordination, while also ensuring that 
physicians are not held accountable for care they did not provide or 
influence. Medicare fee-for-service beneficiaries may seek care from 
any Medicare provider and often receive care from several physicians 
and other providers. This makes it difficult to attribute 
responsibility for all of the health care provided. Attributing care 
to the physician who directly provided it may appear to be 
straightforward, but it may not adequately reflect relative 
responsibility for that care. For example, individual physicians may 
have control over some costs directly incurred by another physician by 
referring beneficiaries to specialists. Physicians may also indirectly 
affect other health care costs by exercising their judgment regarding 
hospital and postacute care decisions. As a result, determining to 
whom a beneficiary's care should be attributed is an important 
methodological decision. 

In Phase I, CMS tested two attribution methods--a single and a 
multiple provider attribution method. 

* A single provider attribution method holds one physician responsible 
for all of a beneficiary's care. This method is designed to identify 
the principal "decision maker," such as the beneficiary's primary care 
physician, and holds this physician responsible for all care provided, 
including referrals and services provided by other physicians. The 
single provider method CMS tested attributed a beneficiary's entire 
cost of care to the single physician who provided the most evaluation 
and management (E&M) services that the beneficiary received. 

* A multiple provider attribution method holds more than one physician 
responsible for the care provided to a beneficiary. This method 
assumes that any one physician is unlikely to have complete 
responsibility for all of that care. The multiple provider method CMS 
tested held all physicians who billed for at least 10 percent of a 
beneficiary's E&M costs partially responsible for that beneficiary's 
care by attributing resource use in proportion to the amount of care 
provided by a given physician. 

In Phase II, CMS officials used a single provider attribution method. 
The agency generally prefers single provider attribution, believing 
that it encourages physicians to coordinate care. However, CMS has not 
provided evidence that using a single provider attribution method 
would lead to increased coordination, and physicians may not accept 
this method as a credible way to attribute costs. According to 
Mathematica officials, physicians profiled in Phase I generally 
preferred the multiple provider attribution method. These physicians' 
comments reflected concerns that it was unfair to attribute other 
providers' resource use to them. Furthermore, most of the physicians 
and other stakeholders who provided comments to Mathematica during 
pretesting in Phase II thought it inappropriate to be held 
accountable, even partially, for care provided by other physicians. 
Both specialists and primary care physicians told Mathematica that 
they did not have control over how another provider treated a 
beneficiary. Specialists noted that they treated beneficiaries for 
certain conditions and would not have knowledge of or be responsible 
for care unrelated to those conditions. Similarly, Mathematica 
reported that primary care physicians felt they had little control 
over the care provided by the specialists to whom they referred 
beneficiaries. 

Despite physicians' concerns about being held responsible for care 
they did not directly provide, they do have indirect control over some 
costs incurred by other providers, such as referrals to specialists 
and decisions about hospitalizations. Given that there is no 
definitive way to determine which costs a physician was indirectly or 
directly responsible for, a multiple provider attribution method may 
be the more reasonable way to attribute costs. For example, the 
multiple provider method CMS tested in Phase I held physicians 
accountable for a proportion of the total care provided to a 
beneficiary. Under this method, a physician who billed for 70 percent 
of a beneficiary's total E&M services was assigned 70 percent of the 
total Medicare resources used by that beneficiary--including office 
visits, hospitalizations, skilled nursing facility stays, and 
diagnostic tests and procedures. 

A multiple provider attribution approach also increases the number of 
physicians potentially eligible to receive feedback reports. Because 
multiple provider attribution holds more than one physician 
accountable for a beneficiary's care, more physicians will have 
patients attributed to them, thus increasing the number of physicians 
eligible for feedback reports. CMS officials recognize that using a 
single attribution method will not allow all physicians to be eligible 
to receive a report, and noted that it is likely that some Phase III 
reports will use a multiple provider attribution method to assign 
resource use to physicians.[Footnote 27] 

CMS set a threshold for the minimum amount of care that a physician or 
physician group needed to provide in order to be assigned 
responsibility for all or part of that beneficiary's care. For 
example, in Phase II, individual physicians needed to bill for at 
least 20 percent of a beneficiary's total E&M costs, and physician 
groups needed to bill for at least 30 percent of the total E&M costs 
in order to be assigned responsibility for that beneficiary's care. 
The minimum threshold was intended to reduce the likelihood that 
physicians and groups would be assigned responsibility for 
beneficiaries for whom they provided only minimal care. CMS is 
considering setting a lower threshold in Phase III to increase the 
number of physicians eligible to receive reports. 

Determining minimum case size. CMS faces a challenging trade-off in 
determining the minimum number of Medicare beneficiaries or episodes 
of care a physician must have to produce reliable information without 
excluding a large number of physicians--those without enough 
beneficiaries or episodes--from receiving a report. A higher minimum 
increases the reliability of the information, but decreases the number 
of physicians eligible to receive a report. In contrast, decreasing 
the minimum case size increases the number of physicians receiving 
reports but reduces reliability. 

In Phase I, CMS conducted a statistical reliability test to determine 
the minimum number of episodes a physician needed to be eligible for a 
feedback report. Reliability indicates how confidently one can 
classify a physician's performance relative to that of his or her 
peers. Estimates for this test range from zero to one, with an 
estimate above 0.8 generally considered a strong indicator of 
reliability. CMS used an estimate of 0.5--which is considered a 
moderate level of reliability--to help ensure that enough physicians 
would be eligible for Phase I reports.[Footnote 28] However, few 
physicians met the minimum case size requirements for certain 
episodes, such as acute myocardial infarctions, even when using this 
moderate level of reliability. CMS did not conduct a reliability test 
to determine the minimum number of beneficiaries a physician must 
treat for per capita cost measurement. 

In Phase II, CMS provided feedback reports to physicians with at least 
30 Medicare beneficiaries attributed to them. CMS did not conduct 
reliability tests for this estimate, stating that a minimum case size 
of 30 is generally accepted in the research community. However, as 
some stakeholders have noted, the appropriate minimum case size may 
vary by condition, suggesting that CMS should instead use a measure of 
reliability or precision to establish the appropriate case size. 
[Footnote 29] For example, Phase II reports contained resource use 
information for five high-cost, high-volume chronic conditions, and it 
is likely that different minimum case sizes were needed to generate 
reliable information for different conditions, such as diabetes and 
coronary artery disease. CMS officials noted that minimum case size is 
a major factor in excluding physicians from receiving feedback 
reports. CMS officials have considered reducing the minimum case size 
from 30 to 20 beneficiaries for Phase III reports. Officials analyzed 
the potential effect of this change on individual physicians' per 
capita resource use rankings, and found that nearly all physicians 
were ranked in the same quartile when the case size was lowered from 
30 to 20. According to CMS officials, this change would increase the 
number of physicians eligible to receive Phase III reports by about 10 
percent. 

Selecting peer groups for comparisons. CMS faces trade-offs in 
balancing stakeholders' preferences that feedback reports compare 
physicians only to those most like themselves--that is, peer groups 
representing narrow subspecialties or limited geographic areas--with 
the need to establish a minimum peer group size that is large enough 
to make statistically significant comparisons. 

Individual-level feedback reports distributed in Phases I and II 
contained two peer group comparisons: (1) physicians in the same 
specialty in the same metropolitan area and (2) physicians in the same 
specialty across all 12 metropolitan areas, which was meant to serve 
as a proxy for a nationwide comparison.[Footnote 30] Some stakeholders 
have encouraged CMS to compare physicians within a limited geographic 
area. However, if a large number of physicians in a limited geographic 
area were practicing inefficiently, a nationwide sample might be 
needed to identify the inefficiencies. 

In addition, some medical specialty societies and other stakeholders 
urged CMS to compare physicians only within narrow subspecialties. For 
example, the American Urological Association noted that surgeons with 
active surgery practices are substantially different from those who 
engage primarily in medical management of urological conditions, and 
comparisons that do not differentiate between these distinct types of 
physicians are not meaningful to physicians and do not promote 
learning and improvement. 

However, if CMS were to identify and compare physicians in smaller 
subspecialties, it would face the challenge of ensuring that the peer 
group size was large enough to make meaningful comparisons across 
physicians. In Phase I, CMS did not impose a minimum peer group size, 
but in Phase II it imposed a minimum peer group size of 30 physicians. 
However, because not all individual physicians had peer groups 
consisting of 30 physicians practicing in the same geographic area and 
in the same specialty, some physicians received a report that did not 
contain information on all performance measures. CMS officials said 
they may use a minimum peer group size of 15 for Phase III feedback 
reports. 

CMS's Plans for Improvement May Not Fully Address Challenges in 
Distributing Reports to Physicians: 

The majority of sampled physicians were not eligible to receive a 
Phase II report after CMS's methodological decisions were applied. CMS 
officials plan to revise their methodology to increase eligibility for 
Phase III reports, but significantly increasing the number of 
physicians who are eligible will be challenging. Further, CMS faced 
multiple challenges with the electronic distribution of feedback 
reports to eligible physicians, and as a result, few physicians 
accessed their reports. CMS officials plan to use a new distribution 
method for Phase III reports. 

Few Sampled Physicians Were Eligible to Receive a Feedback Report; 
Significantly Increasing Eligibility Will Continue to Be Challenging: 

Over 80 percent of CMS's initial sample of 9,189 physicians were 
ineligible to receive a Phase II feedback report after CMS's 
methodological decisions, such as minimum case size requirements, were 
applied. To identify physicians for the Phase II reports, CMS began 
with a sample of 9,189 individual physicians affiliated with 36 
physician groups.[Footnote 31] To be eligible for a Phase II report, 
individual physicians needed to meet CMS's criteria by having the 
following: 

* At least 30 Medicare beneficiaries attributed to them to meet the 
minimum case size requirement for per capita resource use measures. Of 
the 9,189 physicians in the original sample, 2,205 (24 percent) had at 
least 30 beneficiaries attributed to them. 

* At least 11 Medicare beneficiaries attributed to them who were 
eligible for 1 or more of the 12 GEM quality measures. Of the 9,189 
physicians in the original sample, 2,661 physicians (29 percent) had 
at least 11 beneficiaries attributed to them who were eligible for at 
least 1 of the 12 GEM quality measures. 

* A sufficient number of attributed beneficiaries for both the per 
capita resource use and GEM quality measures. Of the 9,189 physicians 
in the original sample, 1,733 physicians (19 percent) had a sufficient 
number of beneficiaries attributed to them for the per capita resource 
use and GEM quality measures. 

* At least 30 individual physicians in the same medical specialty and 
geographic area for a peer group.[Footnote 32] Of the remaining 1,733 
individual physicians, 1,645 physicians had a peer group of at least 
30 individual physicians.[Footnote 33] 

Figure 1 shows the number of physicians excluded by each criterion. 

Figure 1: Number of Individual Physicians Excluded from Phase II 
Feedback Reports Based on CMS's Methodological Criteria, 2010: 

[Refer to PDF for image: illustration] 

All sampled physicians: 
CMS began with a sample of 9,189 individual physicians affiliated with 
the 36 physician groups. 

Case size and peer group requirements: 
To be eligible for a report in the second phase of the program, 
sampled physicians needed to meet CMS criteria by having the following: 

At least 30 attributed beneficiaries for the resource use measures – 
only 2,205 physicians met this requirement; 

At least 11 attributed beneficiaries for at least one of the quality
measures – only 2,661 physicians met this requirement; 

Only 1,733 physicians met both requirements; 

At least 30 individual physicians practicing in the same medical 
specialty and geographic area for a peer group – only 1,645 physicians
met this requirement. 

Source: GAO analysis of CMS and contractor data. 

[End of figure] 

CMS's methodological criteria also excluded many specialists from 
receiving feedback reports. Over 90 percent of Phase II reports were 
created for generalists, such as internal medicine or family practice 
physicians. The single provider attribution method used by CMS--which 
assigned a beneficiary to the single physician who billed for the 
greatest number of E&M services for the beneficiary--limited the 
number of specialists eligible for a report, since specialists often 
provide fewer but more expensive E&M services to beneficiaries than 
generalists. Physicians also needed to have at least one GEM quality 
measure to receive a Phase II report, but the GEM measures were only 
applicable to a limited number of specialists, such as cardiologists 
and nephrologists. 

In addition, many of the 1,641 physicians eligible to receive a Phase 
II feedback report did not meet the methodological criteria needed to 
receive information on all performance measures, such as resource use 
for the five chronic condition subgroups or the 12 GEM quality 
measures. For example, only 5 percent of the 1,641 physicians eligible 
for Phase II reports were eligible to receive resource use information 
for their beneficiaries with chronic obstructive pulmonary disease, 
and none were eligible to receive this information for their 
beneficiaries with prostate cancer. Similarly, none of the 1,641 
physicians eligible for Phase II reports were eligible to receive 
information for 3 of the 12 GEM quality measures. By contrast, the 
majority of the 36 physician groups profiled received information on 
all performance measures (see table 1). 

Table 1: Percentage of Individual Physicians and Physician Groups 
Eligible for Select Resource Use and Quality Performance Measures on 
Phase II Feedback Reports, 2010: 

Resource use for chronic condition subgroups: 

Congestive heart failure: 
Individual physicians: 14; 
Physician groups: 100. 

Chronic obstructive pulmonary disease: 
Individual physicians: 5; 
Physician groups: 100. 

Diabetes: 
Individual physicians: 37; 
Physician groups: 100. 

Coronary artery disease: 
Individual physicians: 39; 
Physician groups: 100. 

Prostate cancer: 
Individual physicians: 0; 
Physician groups: 100. 

GEM quality measures: 

LDL screening for beneficiaries up to 75 years of age with diabetes: 
Individual physicians: 72; 
Physician groups: 100. 

Eye exam (retinal) for beneficiaries up to 75 years of age with 
diabetes: 
Individual physicians: 71; 
Physician groups: 100. 

HbA1c testing for beneficiaries up to 75 years of age with diabetes: 
Individual physicians: 71; 
Physician groups: 100. 

Medical attention for nephropathy for diabetics up to 75 years of age: 
Individual physicians: 35; 
Physician groups: 100. 

LDL-C screening for beneficiaries up to 75 years of age with 
cardiovascular conditions: 
Individual physicians: 38; 
Physician groups: 100. 

Beta blocker treatment after heart attack: 
Individual physicians: 0; 
Physician groups: 83. 

Persistence of beta blocker treatment after heart attack: 
Individual physicians: 0; 
Physician groups: 83. 

Colorectal cancer screening for beneficiaries up to 80 years of age: 
Individual physicians: 99; 
Physician groups: 100. 

Breast cancer screening for women up to 69 years of age: 
Individual physicians: 71; 
Physician groups: 100. 

Annual monitoring for beneficiaries on persistent medications 
(angiotensin-converting enzyme inhibitors or angiotensin receptor 
blockers, digoxin, diuretics, and anticonvulsants): 
Individual physicians: 95; 
Physician groups: 100. 

Antidepressant medication management (acute phase): 
Individual physicians: 0; 
Physician groups: 86. 

Disease-modifying antirheumatic drug therapy in rheumatoid arthritis: 
Individual physicians: 3; 
Physician groups: 100. 

Source: GAO analysis of CMS and contractor data. 

[End of table] 

As we stated earlier in this report, CMS is considering a number of 
methodological changes in Phase III, such as using a multiple provider 
attribution rule and lowering the minimum case size and peer group 
requirements. While such changes could lead to a modest increase in 
physician eligibility for Phase III reports, significantly increasing 
eligibility--particularly for individual physicians with small case 
sizes--will continue to be challenging. 

Multiple Challenges with Distribution Resulted in Few Physicians 
Accessing Their Electronic Feedback Reports: 

CMS faced multiple challenges distributing Phase II feedback reports, 
and as a result of these challenges, few physicians accessed their 
reports. In November 2010, CMS mailed letters to 36 physician groups 
and 1,641 individual physicians affiliated with those groups to notify 
them that electronic feedback reports were available for their review. 
However, as of March 2011--approximately 4 months later--less than 60 
percent of physician groups and less than 10 percent of individual 
physicians had accessed their reports electronically.[Footnote 34] 

Major challenges with Phase II distribution were CMS's difficulty 
obtaining physicians' contact information, methods of electronic 
distribution that were burdensome for physicians, and lack of a strong 
incentive for physicians to review the reports. 

Contact information. The lack of a comprehensive database with 
accurate names and addresses for physicians and physician groups made 
it difficult for CMS to notify physicians and physician groups about 
the availability of their feedback reports. Although reports in Phase 
II were produced in electronic form, CMS mailed hard copy notification 
letters to tell individual physicians and physician groups that an 
electronic feedback report was available and to provide instructions 
for accessing it. Because available databases contained incomplete or 
conflicting contact information, CMS had to use multiple sources, 
including Internet searches, to compile names and addresses--a process 
that took approximately 5 months.[Footnote 35] 

Despite CMS's efforts to obtain accurate contact information, some 
individual physicians and physician groups did not receive a 
notification letter and therefore did not know that a feedback report 
was available to them. In follow-up phone calls, CMS found that 27 of 
the 32 physician groups reached reported that they had not seen the 
notification letter and could not verify whether it had been received. 
Many of these physician groups reported that the notification letter 
was not addressed to the most appropriate person within the group 
practice, such as the director of quality assurance. CMS also called a 
sample of 10 individual physicians to ask whether they had received 
the notification letter. Of these physicians, 1 was retired, 1 
reported not receiving the letter, and the remaining 8 had no memory 
of receiving the letter. In addition, nearly 10 percent of the 
notification letters mailed to individual physicians were marked 
undeliverable and returned to CMS. 

Distribution method. CMS's electronic distribution method for Phase II 
reports was burdensome for some profiled physicians and physician 
groups. CMS transitioned from hard copy distribution of feedback 
reports in Phase I to electronic distribution in Phase II based on 
physicians' complaints that the reports distributed in Phase I were 
too long and cumbersome to manage in hard copy. According to CMS, 
electronic distribution was meant to help physicians navigate the 
reports. CMS used two methods to electronically distribute feedback 
reports in Phase II--one for individual physicians and one for 
physician groups. 

Individual physicians were instructed in the notification letter to 
contact their Medicare Administrative Contractor (MAC) to request a 
copy of their feedback report.[Footnote 36] In a report to CMS, 
Mathematica reported that finding contact information for the correct 
MAC may not have been a straightforward process for physicians. For 
example, the notification letter directed physicians to a directory 
with toll-free phone numbers listed by state for all MAC contact 
centers, requiring physicians to choose from several possible 
numbers.[Footnote 37] Mathematica also reported that MAC customer 
service representatives were not always aware of the feedback reports 
or the process for distributing them to physicians. According to CMS's 
estimate, the majority of individual physicians did not contact their 
MACs to request their reports. In February 2011, CMS mailed hard 
copies of the 1,596 feedback reports that had not yet been 
electronically accessed by individual physicians.[Footnote 38] 

In theory, the electronic distribution method for physician groups 
should have been more straightforward since groups were instructed to 
download their feedback reports from the Individuals Authorized Access 
to CMS Computer Services (IACS) system, which is the same system used 
to distribute PQRS reports. However, 8 of the 32 physician groups CMS 
reached in its follow-up calls reported difficulty downloading their 
reports from the IACS system. For example, some groups did not know 
that they needed to register for an IACS account--a process that takes 
approximately 10 business days to complete--while others reported not 
being able to download their feedback reports even after logging onto 
the IACS system. CMS subsequently e-mailed feedback reports directly 
to those physician groups that had trouble downloading their reports 
through IACS. 

CMS officials recognized the limitations with the distribution method 
for Phase II reports, and they plan to use a new distribution method 
for Phase III reports. CMS currently plans to distribute reports to 
20,000 individual physicians in one four-state region--Nebraska, 
Missouri, Iowa, and Kansas. According to CMS officials, the MAC 
serving this region has e-mail addresses for most physicians in the 
area. CMS plans to e-mail Phase III reports directly to physicians in 
this region, thereby avoiding the need to mail hard copy notification 
letters. In addition to distributing reports to individual physicians 
in the four-state region, CMS also plans to distribute Phase III 
reports to 35 physician groups that have participated in the PQRS 
group practice reporting option. CMS intends to e-mail feedback 
reports to these 35 physician groups. 

Incentive to access reports. Physicians did not have a strong 
incentive to access their Phase II feedback reports. The notification 
letter sent by CMS said that these reports were "for informational 
purposes only" and that they would not affect physicians' 
participation in the Medicare program or their Medicare payments. In 
pretesting for Phase II, many physicians noted that they would be 
unlikely to review a feedback report closely unless they had an 
incentive to do so. CMS officials said that they did not want to 
emphasize that the types of cost and quality measures contained in the 
feedback reports could affect physicians' payments in the future 
because they did not want the reports to sound threatening. Several 
physician groups suggested that CMS send feedback reports for those 
physicians affiliated with a group practice to the group's 
administrator, noting that individual physicians generally contact 
their administrators for guidance on such reports. In a report to CMS, 
Mathematica also noted that medical directors or others with quality 
oversight responsibilities in larger group practices would be more 
receptive to feedback reports than individual physicians. They added 
that these individuals are more familiar with the data used to create 
feedback reports, and have more experience analyzing quality and cost 
information for practice improvement. 

CMS made follow-up calls to representatives of 15 of the 36 profiled 
physician groups to obtain their input on the feedback reports, but it 
conducted minimal follow-up with individual physicians. At the time 
CMS attempted to follow up with individual physicians, only 4 had 
contacted their MACs to request a feedback report. Three of these 
physicians were unwilling to participate in a follow-up call about the 
report, and 1 physician was unable to download the feedback report 
that had been sent via e-mail. Similarly, CMS called a sample of 10 
physicians who had not requested their feedback reports to ask why 
they had not done so, and 8 of these physicians expressed no interest 
in their reports. 

Conclusions: 

In light of concerns about the long-term fiscal challenges facing the 
Medicare program, the Physician Feedback Program is an important 
effort that could encourage more efficient medical practice as well as 
higher-quality care. CMS has worked under challenging timelines to 
test different approaches to feedback report methodology and 
distribution. Initial phases of the program indicate that significant 
changes will need to be made for the program to meet its goal of 
producing reports with meaningful, actionable, and fair performance 
measures that apply to the majority of Medicare physicians. CMS will 
need to do more to solicit input and reactions from physicians and 
physician groups on the methodology and distribution of reports while 
the stakes are still relatively low--that is, before CMS begins paying 
physicians based on their performance on the resource use and quality 
measures included in the feedback reports beginning January 1, 2015. 

In the first two phases, CMS tested different methodological 
approaches to developing feedback reports; however, the majority of 
physicians in the most recent phase were ineligible for a feedback 
report once CMS's methodological criteria were applied. For example, 
CMS used a single provider attribution method in the most recent 
phase, believing that it may improve care coordination--but this 
method limited physician eligibility, and there is limited evidence to 
suggest that using this method would increase coordination. And while 
we also agree with CMS's decision to include quality measures in 
feedback reports, some physicians who would have been eligible to 
receive information on their resource use were disqualified from 
receiving a Phase II report because they were not eligible for at 
least 1 of the 12 GEM quality measures. Further, none of the 
individual physicians who were eligible for a Phase II report had 
enough beneficiaries attributed to them to receive performance data on 
all 12 quality measures. CMS did not face such sample size issues in 
the feedback reports it developed for physician groups. 

CMS has not conducted the rigorous statistical analysis it needs to 
fully understand the impact of its methodological decisions on 
reliability. For example, CMS used a minimum case size of 30 
beneficiaries for Phase II reports, but did not conduct reliability 
testing to determine this number. The results of such testing can and 
should influence how CMS ultimately uses the information. Lower levels 
of reliability may be acceptable if feedback reports remain 
confidential and are used solely for educational purposes. However, 
since CMS ultimately intends to pay physicians based on their 
performance as measured in the feedback reports, it must be reasonably 
confident that these measures reflect real differences in medical 
practice. It will also be difficult for CMS to obtain physician and 
stakeholder buy-in if it does not clearly demonstrate that its 
performance measures are reliable and robust. 

Furthermore, CMS faces challenges distributing feedback reports to 
physicians and physician groups that are eligible to receive them. CMS 
transitioned to electronic distribution based on physicians' 
complaints that hard copy reports were too long and cumbersome, yet 
few physicians accessed their Phase II reports electronically. 
Moreover, CMS conducted limited follow-up with profiled physicians to 
obtain their input on the feedback reports. As a result, the agency 
missed an important opportunity to increase physician engagement in 
the program and to ensure that their concerns are addressed while the 
program is still in its infancy. 

Recommendations for Executive Action: 

In order to develop feedback reports that are more reliable, credible, 
accessible, and applicable to a greater number of Medicare physicians, 
we recommend that the Administrator of CMS take the following four 
actions: 

* Use methodological approaches that increase the number of physicians 
eligible to receive a report, such as: 

- multiple provider attribution methods, which could also enhance 
credibility of the reports with physicians, and: 

- distributing feedback reports that include only resource use 
information, if quality information is unavailable. 

* Conduct statistical analyses of the impact of key methodological 
decisions on reliability. 

* Identify factors that may have prevented physicians from accessing 
their reports and, as applicable, develop strategies to improve the 
process for distributing reports and facilitating physicians' access 
to them. 

* Obtain input from a sample of physicians who received feedback 
reports on the usefulness and credibility of the performance measures 
contained in the reports and consider using this information to revise 
future reports. 

Agency Comments and Our Evaluation: 

We received written comments on a draft of this report from CMS, which 
are reprinted in appendix I. CMS concurred with our recommendations 
and identified actions agency officials are taking to implement them. 
These actions include refining the attribution methodology to increase 
the number of physicians receiving feedback reports in Phase III, 
analyzing the number of cases required to reliably measure quality and 
make credible comparisons, developing new strategies for distributing 
feedback reports, and obtaining input from individual physicians and 
physician groups about the information contained in the feedback 
reports. If these actions are implemented in accordance with our 
recommendations, CMS will be better positioned to meet its goals and 
objectives for the Physician Feedback Program. CMS also provided 
technical comments, which we incorporated as appropriate. 

We are sending copies of this report to the Administrator of CMS and 
relevant congressional committees. The report also will be available 
at no charge on the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions about this report, please 
contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff members who made major 
contributions to this report are listed in appendix II. 

Signed by: 

James C. Cosgrove: 
Director, Health Care: 

[End of section] 

Appendix I: Comments from the Department of Health and Human Services: 

Department Of Health & Human Services: 
Office Of The Secretary: 
Assistant Secretary for Legislation: 
Washington, DC 20201: 

July 29, 2011: 

James Cosgrove:
Director, Health Care: 
U.S. Government Accountability Office: 
441 G Street N.W. 
Washington, DC 20548: 

Dear Mr. Cosgrove: 

Attached are comments on the U.S. Government Accountability Office's 
(GAO) draft report entitled: "Medicare Physician Feedback Program: CMS 
Faces Challenges with Methodology and Distribution of Physician 
Reports" (GAO 11-720). 

The Department appreciates the opportunity to review this draft report 
prior to publication. 

Sincerely, 

Signed by: 

Jim R. Esquea: 
Assistant Secretary for Legislation: 

Attachment: 

[End of letter] 

General Comments Of The Department Of Health And Human Services (HHS) 
On The Government Accountability Office's (GAO) Draft Report Entitled, 
"Medicare Physician Feedback Program: CMS Faces Challenges With 
Methodology And Distribution Of Physician Reports (GAO-11-720): 

The Department appreciates the opportunity to review and comment on 
this draft report, which is a factual synopsis of Phase I and Phase II 
of the Physician Feedback Program, formerly called the Physician 
Resource Use Measurement and Reporting Program. 

This program was established as a confidential feedback program as 
required by the Medicare Improvements for Patient and Providers Act of 
2008 (MIPPA). The program was further modified as a result of the 
Affordable Care Act in 2010, which also requires a value-based payment 
modifier that provides for differential payments to specified 
physicians based on the quality of care furnished compared to cost. 
The value-based payment modifier would be applied to specified 
physicians beginning in 2015 and all physicians starting in 2017. 
Hence, we expect that this program, which started small with 
flexibility in the number and content of reports disseminated, will 
quickly evolve to affect physician payment nationwide. 

GAO Recommendations: 

In order to develop feedback reports that are more reliable, credible, 
accessible, and applicable to a greater number of Medicare physicians, 
we recommend that the Administrator of the Centers for Medicare and 
Medicaid Services (CMS): 

* use methodological approaches that increase the number of physicians 
eligible to receive a report, such as: 

- multiple provider attribution methods, which could also enhance 
credibility of the reports with physicians, and; 

- distributing feedback reports that include only resource use 
information, if quality information is unavailable; 

CMS Response: 

We concur with this recommendation. As we discussed with the GAO, we 
are planning in Phase III to refine the attribution models we used in 
Phase II, and to test new ones. As a result, the number of physicians 
receiving a report in Phase III will increase. This attribution 
methodology will permit us to distribute feedback reports that include 
only resource use information if quality information is unavailable. 

* conduct statistical analyses of the impact of key methodological 
decisions on reliability; 

CMS Response: 

We concur with this recommendation. We will be analyzing the results 
in these reports for reliability as well as analyzing the number of 
cases required to reliably measure quality and make credible 
comparisons. 

* identify factors that may have prevented physicians from accessing 
their reports and as applicable, develop strategies to improve the 
process for distributing reports and facilitating physicians' access 
to them. 

CMS Response: 

We concur with this recommendation. We are working to develop new 
strategies for distributing reports that will improve physician 
access. To this end, for Phase III, we are working with the 
Jurisdiction 5 (J5) Medicare Administrative Contractor (MAC) which 
serves the four-State region of Nebraska, Kansas, Missouri, and Iowa. 
This MAC has an advanced communications portal which allows robust 
communications between the MAC and physicians in these States. In 
particular, the MAC has e-mail information on a large number of the 
physicians it serves and we expect to use this list to provide 
individual physicians with their feedback reports. We are also working 
to develop an enterprise-wide solution that could be used to reach all 
physicians nationwide to provide feedback reports. 

* obtain input from a sample of physicians who received feedback 
reports on the usefulness and credibility of the performance measures 
contained in the reports and consider using this information to revise 
future reports. 

CMS Response: 

We concur with this recommendation. We will be working closely across 
CMS, with J5 MAC, and with State stakeholders to inform physicians in 
these four States that they will be receiving the feedback reports. 
Through these mechanisms we will reach out to these physicians about 
the information contained within the feedback reports, how the reports 
can help them understand the quality of care their Medicare patients 
receive and the resources used to provide this care. We will also 
obtain feedback from the recipients of the group reports. With both 
individual and group report recipients, we will discuss the importance 
of these reports as beginning to provide the information building 
blocks that could be used to calculate their value modifier. We 
anticipate that this outreach will help spur physicians to participate 
in focus groups following report dissemination to discuss these issues. 

[End of section] 

Appendix II: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

James C. Cosgrove, (202) 512-7114 or cosgrovej@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Phyllis Thorburn, Assistant 
Director; William A. Crafton; Cathleen Hamann; Julian Klazkin; Amanda 
Pusey; Jessica C. Smith; and Rachael Wojnowicz made key contributions 
to this report. 

[End of section] 

Footnotes: 

[1] Medicare is the federally financed health insurance program for 
persons aged 65 and over, certain individuals with disabilities, and 
individuals with end-stage renal disease. Medicare Part A covers 
hospital and other inpatient stays. Medicare Part B covers physician, 
outpatient hospital, home health, and other services. Medicare Parts A 
and B are known as original Medicare or Medicare fee-for-service. 

[2] Resource use can be defined as the costs to the Medicare program, 
including those contributions by Medicare beneficiaries, such as co- 
payments and deductibles. See GAO, Medicare: Focus on Physician 
Practice Patterns Can Lead to Greater Program Efficiency, [hyperlink, 
http://www.gao.gov/products/GAO-07-307] (Washington, D.C.: Apr. 30, 
2007); GAO, Medicare: Per Capita Method Can Be Used to Profile 
Physicians and Provide Feedback on Resource Use, [hyperlink, 
http://www.gao.gov/products/GAO-09-802] (Washington, D.C.: Sept. 25, 
2009); Medicare Payment Advisory Commission, Report to the Congress: 
Medicare Payment Policy (Washington, D.C.: Mar. 2005), 142; and CBO, 
Medicare's Payments to Physicians: Options for Changing the 
Sustainable Growth Rate (Washington, D.C.: Mar. 1, 2007), 16-17. 

[3] Pub. L. No. 110-275, § 131(c), 122 Stat. 2494, 2526. 

[4] CMS has distributed feedback reports to various health care 
providers--primarily physicians--as well as nurse practitioners and 
physician's assistants. For this report, we refer to providers as 
physicians and provider groups as physician groups. 

[5] Pub. L. No. 111-148, § 3007, 124 Stat. 119, 373-376 (codified at 
42 U.S.C. § 1395w-4(p)). 

[6] 42 U.S.C. § 1395w-4(p)(4)(B)(iii). 

[7] 42 U.S.C. § 1395w-4(p)(9). 

[8] Pub. L. No. 110-275, § 131(c)(2), 121 Stat. 2494, 2527. 

[9] Throughout this report, we generally attribute the analysis and 
actions taken by Mathematica to CMS. 

[10] See Medicare Program; Payment Policies Under the Physician Fee 
Schedule and Other Revisions to Part B for CY2011; Proposed Rule, 75 
Fed. Reg. 40040 (July 13, 2010). We identified 42 letters related to 
the Physician Feedback Program that were submitted in response to the 
proposed rule via the website [hyperlink, http://www.regulations.gov]. 
Sixteen of the letters we reviewed were from medical specialty 
societies; 25 of the letters were from other stakeholders, such as the 
National Business Group on Health; and 1 letter was from a medical 
specialty society and one other organization that commented jointly. 

[11] For chronic conditions, which do not have clearly defined start 
or end dates, episodes of care may be measured over a specified time 
period, such as on a 12-month basis. 

[12] PPACA requires that CMS develop a Medicare-specific episode 
grouper by January 1, 2012. Pub. L. No. 111-148, § 3003(a)(4), 124 
Stat. 119, 366-8 (codified at 42 U.S.C. § 1395w-4(n)(9)(A)). 

[13] Other types of measures can also be used to evaluate the quality 
of care, such as tracking a patient's experience with health care 
services. 

[14] NQF is a nonprofit organization that fosters agreement on 
national standards for measuring and public reporting of health care 
performance data. NCQA is a national nonprofit organization that 
develops health care quality and performance standards and accredits 
health plans, physicians, and other health care providers. 

[15] HEDIS® is a group of standardized measures used to measure 
clinical performance in areas such as medication use, control of high 
blood pressure, breast cancer screening, immunization, and 
comprehensive diabetes care. 

[16] The areas were those included in an ongoing Community Tracking 
Study (CTS) being conducted by a research organization, the Center for 
Studying Health System Change. The CTS sites were designated because 
they provide a random sample of communities that represent different 
geographic areas, populations, physician and health care market 
structures, patterns of Medicare spending, and experience with public- 
or private-sector performance measurement. They were Boston, 
Massachusetts; Cleveland, Ohio; Greenville, South Carolina; 
Indianapolis, Indiana; Lansing, Michigan; Little Rock, Arkansas; 
Miami, Florida; Northern New Jersey; Orange County, California; 
Phoenix, Arizona; Seattle, Washington; and Syracuse, New York. 

[17] Physician groups were selected based on the following criteria: 
that they have at least 5,000 Medicare beneficiaries in 2007 and at 
least one physician who participated in the PQRS program since it 
began in 2007. 

[18] PPACA requires CMS to seek endorsement of the episode grouper by 
the consensus-based entity that has a contract for performance 
measurement under the Medicare program. Currently, that contract is 
with NQF. See Pub. L. No. 111-148, § 3003(a)(4), 124 Stat. 119, 366-7 
(codified at 42 U.S.C. § 1395w-4(n)(9)(A)(iv)); 42 U.S.C. § 
1395aaa(a)(1). 

[19] Two contracts were awarded to make the existing commercially 
available software more usable for the Medicare population, and two 
contracts were awarded to have a new episode grouper constructed. 

[20] CMS officials explained that they found that prostate cancer was 
rarely reported in their sample. 

[21] Including quality measures in feedback reports is optional. 42 
U.S.C. § 1395w-4(n)(1)(A)(iii). 

[22] Phase II feedback reports also provided a link to CMS's Hospital 
Compare, Nursing Home Compare, and Home Health Compare websites to 
provide information on the quality of the hospitals used by the 
physician's beneficiaries and nursing homes and home health agencies 
in the physician's metropolitan area. In addition, physician group 
feedback reports contained six ambulatory care sensitive conditions, 
which are medical conditions for which timely and coordinated 
outpatient care could have prevented the need for hospitalization. 
These include congestive heart failure and dehydration. 

[23] A recent Agency for Healthcare Research and Quality (AHRQ) report 
also noted that other beneficial aspects of administrative data, 
including claims data, are that they are relatively inexpensive to 
acquire in electronic formats, coded by health information 
professionals using accepted coding systems, and drawn from large 
populations and therefore more representative of the populations of 
interest. However, the report states that administrative data are 
limited in that because most administrative data are intended for 
financial management rather than quality assessment, they contain 
varying degrees of clinical detail and are often limited in content, 
completeness, timeliness, and accuracy. Patrick Romano, Peter Hussey, 
and Dominique Ritley, Selecting Quality and Resource Use Measures: A 
Decision Guide for Community Quality Collaboratives, Final Contract 
Report (prepared by the University of California and RAND Corporation, 
under contract No. 08003967), AHRQ Publication No. 09(10)-0073 
(Rockville, Md.: AHRQ, May 2010). 

[24] CMS considered, but decided not to include, measures from the 
PQRS program in Phase II feedback reports because of current 
limitations, such as physicians' low participation rate in the program 
and because physicians had flexibility to choose which measures to 
report under PQRS. CMS officials stated that as a result of these 
limitations, it would have been difficult to make meaningful 
comparisons using PQRS measures. 

[25] CMS officials stated that they also plan to include measures from 
the Health Information Technology for Economic and Clinical Health Act 
in future feedback reports. 

[26] CMS has used a concurrent risk adjustment model for its Physician 
Group Practice Demonstration. 

[27] CMS officials explained that one way to increase physician 
eligibility is to use more than one attribution rule in future 
physician feedback reports; for example, the attribution rule CMS uses 
could vary by physician specialty. 

[28] The minimum number of episodes required varied by physician 
specialty and condition. 

[29] A measure of reliability or precision could include, for example, 
a confidence interval. 

[30] In Phase II, physician groups were compared to other physician 
groups in 12 metropolitan areas. 

[31] CMS's initial sample consisted of individual physicians who were 
affiliated with 1 of the 36 physician groups in 2007, the year of 
Medicare claims data from which the performance measures in the 
reports were derived; practiced in 1 of the 12 metropolitan areas 
selected for Phase II report distribution; were considered eligible 
for beneficiary attribution based on select criteria; and had a valid 
Unique Physician Identification Number (UPIN) in 2007. The UPIN has 
been changed to the National Provider Identifier. 

[32] For example, a cardiologist practicing in Miami, Florida, had to 
have a peer group of at least 30 other cardiologists in Miami with at 
least 30 attributed beneficiaries for the resource use measure and at 
least 11 attributed beneficiaries for at least one GEM quality measure 
relevant to cardiologists, such as the percentage of patients 
receiving beta blocker treatment after a heart attack. 

[33] Four of the 1,645 physicians were disqualified because CMS could 
not identify their National Provider Identifier or could not locate a 
verifiable address; as a result, CMS created Phase II feedback reports 
for 1,641 individual physicians. 

[34] As of March 2011, 20 of 36 profiled physician groups had logged 
onto the Individuals Authorized Access to CMS Computer Services (IACS) 
system. Because this database is used for a number of purposes, CMS 
was unable to determine how many of these groups actually downloaded 
their Phase II feedback reports. 

[35] These sources include the Provider Enrollment, Chain, and 
Ownership System (PECOS) database; the National Plan and Provider 
Enumeration System (NPPES) database; and the IACS system. CMS intended 
to use PECOS as the primary source of contact information but found 
multiple mailing addresses listed for individual physicians and 
physician groups. PECOS also did not clearly identify the most 
appropriate contact person within a physician group, such as the 
director of quality assurance. As a result, CMS used other sources in 
order to obtain contact information, such as NPPES and IACS, but in 
some cases, the names and addresses listed in the other sources did 
not match any of the information listed in PECOS. CMS ultimately 
developed decision rules to select contact information for individual 
physicians and physician groups from competing sources, and in some 
cases, relied on Internet searches. 

[36] Once contacted, the MAC verified the identity of the requesting 
physician and then forwarded the request to Buccaneer Computer Systems 
and Services, Inc. (BCSSI), a CMS contractor. BCSSI then e-mailed the 
feedback report to the physician. 

[37] According to CMS officials, the MAC directory lists several 
telephone numbers for the convenience of their customers. CMS 
officials reported that physicians can sort the directory to find the 
correct MAC contact number in their state. 

[38] CMS mailed hard copy feedback reports to the 1,596 physicians who 
had not accessed their electronic reports by January 2011. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: