This is the accessible text file for GAO report number GAO-07-373 
entitled 'Nursing Homes: Federal Actions Needed to Improve Targeting 
and Evaluation of Assistance by Quality Improvement Organizations' 
which was released on June 29, 2007.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Ranking Member, Committee on Finance, U.S. Senate:

United States Government Accountability Office:

GAO:

May 2007:

Nursing Homes:

Federal Actions Needed to Improve Targeting and Evaluation of 
Assistance by Quality Improvement Organizations:

Quality Improvement Organizations:

GAO-07-373:

GAO Highlights:

Highlights of GAO-07-373, a report to the Ranking Member, Committee on 
Finance, U.S. Senate. 

Why GAO Did This Study:

In 2002, CMS contracted with Quality Improvement Organizations (QIO) to 
help nursing homes address quality problems such as pressure ulcers, a 
deficiency frequently identified during routine inspections conducted 
by state survey agencies. CMS awarded $117 million over a 3-year period 
to the QIOs to assist all homes and to work intensively with a subset 
of homes in each state. Homes’ participation was voluntary. To evaluate 
QIO performance, CMS relied largely on changes in homes’ quality 
measures (QM), data based on resident assessments routinely conducted 
by homes. GAO assessed QIO activities during the 3-year contract 
starting in 2002, focusing on (1) characteristics of homes assisted 
intensively, (2) types of assistance provided, and (3) effect of 
assistance on the quality of nursing home care. GAO conducted a Web-
based survey of all 51 QIOs, visited QIOs and homes in five states, and 
interviewed experts on using QMs to evaluate QIOs.

What GAO Found:

Although more homes volunteered to work with the QIOs than CMS expected 
them to assist intensively, QIOs typically did not target their 
assistance to the low-performing homes that volunteered. Most QIOs’ 
primary consideration in selecting homes was their commitment to 
working with the QIO. CMS did not specify selection criteria for 
intensive participants but contracted with a QIO that developed 
guidelines encouraging QIOs to select committed homes and exclude those 
with many survey deficiencies or QM scores that were too good to 
improve significantly. Consistent with the guidelines, few QIOs 
targeted homes with a high level of survey deficiencies, and eight QIOs 
explicitly excluded these homes. GAO’s analysis of state survey data 
confirmed that selected homes were less likely than other homes to be 
low-performing in terms of identified deficiencies. Most state survey 
and nursing home trade association officials interviewed by GAO 
believed QIO resources should be targeted to low-performing homes.

QIOs were provided flexibility both in the QMs on which they focused 
their work with nursing homes and in the interventions they used. Most 
QIOs chose to work on chronic pain and pressure ulcers, and most used 
the same interventions?conferences and distribution of educational 
materials?to assist homes statewide. The interventions used to assist 
individual homes intensively varied and included on-site visits, 
conferences, and small group meetings. Just over half the QIOs reported 
that they relied most on on-site visits to assist intensive 
participants. Sixty-three percent said such visits were their most 
effective intervention. Of the 15 QIOs that would have changed the 
interventions used, most would make on-site visits their primary 
intervention. Homes indicated that they were less satisfied with the 
program when their QIO experienced high staff turnover or when their 
QIO contact possessed insufficient expertise.

Shortcomings in the QMs as measures of nursing home quality and other 
factors make it difficult to measure the overall impact of the QIOs on 
nursing home quality, although staff at most of the nursing homes GAO 
contacted attributed some improvements in the quality of resident care 
to their work with the QIOs. The extent to which changes in homes’ QM 
scores reflect improvements in the quality of care is questionable, 
given the concerns raised by GAO and others about the validity of the 
QMs and the reliability of the resident assessment data used to 
calculate them. In addition, quality improvements cannot be attributed 
solely to the QIOs, in part because the homes that volunteered and were 
selected for intensive assistance may have differed from other homes in 
ways that would affect their scores; these homes may also have 
participated in other quality improvement initiatives. Ongoing CMS 
evaluation of QIO activities for the contract that began in August 2005 
is being hampered by a 2005 Department of Health and Human Services 
decision that QIO program regulations prohibit QIOs from providing to 
CMS the identities of homes being assisted intensively.

What GAO Recommends:

GAO recommends that the CMS Administrator (1) further increase the 
number of low-performing homes that QIOs work with intensively, (2) 
improve monitoring and evaluation of QIO activities, and (3) require 
QIOs to share with CMS the identity of homes assisted intensively in 
order to facilitate evaluation. CMS agreed with the first two 
recommendations, but did not specifically indicate if it agreed with 
the third. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-373].

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Kathryn G. Allen, (202) 
512-7118, allenk@gao.gov.

[End of section]

Contents:

Letter:

Results in Brief:

Background:

QIOs Generally Had a Choice among Homes That Volunteered but Did Not 
Target Assistance to Low-Performing Homes:

QIO Contract Flexibility Resulted in Variation in Assistance Provided 
to Intensive Participants:

QIOs' Impact on Quality Is Not Clear, but Staff at Homes We Contacted 
Attributed Some Improvements to QIOs:

Conclusions:

Recommendations for Executive Action:

Agency Comments and Our Evaluation:

Appendix I: Scope and Methodology:

Appendix II: Publicly Reported Quality Measures:

Appendix III: Comments from the Centers for Medicare & Medicaid 
Services:

Appendix IV: GAO Contact and Staff Acknowledgments:

Related GAO Products:

Tables:

Table 1: Quality Measures on Which QIOs Could Focus Their Quality 
Improvement Efforts in the 7th SOW:

Table 2: Examples of Other Categories of Homes Stakeholders Suggested 
QIOs Should Include as Intensive Participants:

Table 3: Examples of Resident Care Improvements Made by Homes as a 
Result of Intensive Assistance Provided by QIOs, 7th SOW:

Table 4: QMs as of November 2002 and as of February 2007:

Figures:

Figure 1: Timeline for 7th SOW Contract and Concurrent Special Studies 
by QIOs to Improve the Quality of Nursing Home Care:

Figure 2: Levels of QIO Assistance and Nursing Home Participation in 
the 7th SOW:

Figure 3: QIO Contract Evaluation Scoring Methodology for the 7th SOW:

Figure 4: QIOs' Considerations in Choosing among Homes That Volunteered 
for Intensive Assistance in the 7th SOW:

Figure 5: Comparison of Nonintensive and Intensive Participants' 
Performance on State Surveys:

Figure 6: QMs Selected by QIOs for Statewide Interventions and QMs 
Selected by Nursing Homes for Intensive Assistance, 7th SOW:

Figure 7: Statewide Interventions Most Relied on by QIOs, 7th SOW:

Figure 8: Intensive Interventions Most Relied on by QIOs and Frequency 
of Interventions (Range and Median Number) during the 7th SOW:

Abbreviations:

CMS: Centers for Medicare & Medicaid Services: 
FTE: full-time-equivalent: 
HHS: Department of Health and Human Services: 
IOM: Institute of Medicine: 
MDS: minimum data set: 
NQF: National Quality Forum: 
OSCAR: On-Line Survey, Certification, and Reporting system: 
PARTner: Program Activity Reporting Tool: 
PRO: Peer Review Organization: 
QIO: Quality Improvement Organization: 
QM: quality measure: 
SOW: statement of work:

United States Government Accountability Office:

Washington, DC 20548:

May 29, 2007:

The Honorable Charles E. Grassley: 
Ranking Member: 
Committee on Finance: 
United States Senate:

Dear Senator Grassley:

The federal government plays a major role in the financing and 
oversight of nursing home care for individuals who are aged or 
disabled. Medicare and Medicaid payments for nursing home services 
totaled $67 billion in 2004, including a $46 billion federal 
share.[Footnote 1] The Centers for Medicare & Medicaid Services (CMS) 
defines quality standards that the nation's approximately 16,400 
nursing homes must meet to participate in the Medicare and Medicaid 
programs and contracts with state survey agencies to assess homes' 
compliance through routine inspections, known as standard surveys, and 
through complaint investigations. Under 3-year contracts beginning in 
August 2002 and referred to as the 7th statement of work (SOW), CMS 
directed Medicare Quality Improvement Organizations (QIO) to work with 
nursing homes to improve the quality of care provided to residents in 
the 50 states, the District of Columbia, and the territories.[Footnote 
2]

As a condition of their contracts, QIOs were required to provide (1) 
information to all Medicare-or Medicaid-certified nursing homes in each 
state about systems-based approaches to improving resident care and 
clinical outcomes and (2) intensive assistance to a subset of each 
state's homes, typically 10 to 15 percent, that were selected by the 
QIOs from among those homes that volunteered for assistance.

In a series of congressionally requested studies undertaken since 1998, 
we have reported on the unacceptably high proportion of nursing homes 
providing poor care to residents.[Footnote 3] Based in part on our 
recommendations, CMS has undertaken a number of enforcement initiatives 
to encourage nursing home compliance with federal quality standards, 
including improved oversight by both state survey agencies and CMS, and 
tougher enforcement measures to ensure that homes correct deficiencies 
and maintain compliance with federal standards. For example, CMS 
expanded its Special Focus Facility program in which state agencies 
survey selected homes more frequently and terminate those that fail to 
improve significantly within 18 months.

CMS's decision to offer direct assistance to nursing homes that 
volunteer to work with QIOs represents a new strategy in the effort to 
help address long-standing quality problems in nursing homes. To 
evaluate QIO performance in improving nursing home care, CMS relied 
primarily on changes in nursing homes' quality measures (QM) during the 
contract period. QMs are numeric measures derived from resident 
assessments--known as the minimum data set (MDS)--that nursing homes 
routinely conduct and submit to CMS.[Footnote 4] The QMs were developed 
to permit comparisons across nursing homes of the quality of care 
provided to residents and have been publicly reported on CMS's Nursing 
Home Compare Web site since 2002.[Footnote 5]

In 2005, CMS renewed the QIO contracts, including the nursing home 
component, for another 3-year period, with a budget of $96 million to 
assist nursing homes.[Footnote 6] Given the decision to continue the 
program, you asked us to assess QIOs' work with nursing homes for the 
7th SOW, covering the period August 2002 through January 2006. For this 
report, we assessed (1) characteristics of nursing homes the QIOs 
assisted intensively, (2) the assistance the QIOs provided to nursing 
homes, and (3) the effect of QIOs' assistance on the quality of nursing 
home care.

To assess the characteristics of nursing homes that the QIOs selected 
to assist intensively from among the homes that volunteered, we 
analyzed CMS data on deficiencies cited in standard surveys of nursing 
homes and compared the results for homes assisted intensively by the 
QIOs with homes that were not assisted intensively.[Footnote 7] To 
gather information about the QIOs' criteria for selecting homes for 
intensive assistance, we fielded a Web-based survey to the 37 
organizations that held the 51 QIO contracts in the states and the 
District of Columbia, achieving a 100 percent response rate.[Footnote 
8] To determine the type of quality improvement assistance QIOs 
provided to nursing homes, our Web-based survey collected data on the 
types, frequency, and perceived effectiveness of specific interventions 
used to assist homes both statewide and in the group assisted 
intensively; interventions included activities such as on-site visits, 
mailings, and conferences. To gather more detailed information about 
QIOs' work with nursing homes, we conducted site visits to five states-
-Colorado, Florida, Iowa, Maine, and New York--where we interviewed QIO 
personnel, staff from nursing homes that had received intensive 
assistance, and key stakeholders.[Footnote 9] The five states accounted 
for 15 percent of nursing home beds nationwide in 2002 and represented 
a range in terms of such characteristics as number of nursing home 
beds, region of the country, and QIOs' performance on the nursing home 
component in the 7th SOW. In the five states, we interviewed staff from 
28 nursing homesæ4 to 8 per state; in addition, we interviewed staff 
from 4 homes in four other states for a total of 32 homes. We sought to 
select a group of homes that represented a range in terms of state 
survey deficiencies, improvement in QM scores during the 7th SOW, 
distance from the QIO, and urban versus rural location. However, the 
experiences of the 32 homes in our sample cannot be generalized to all 
homes that received intensive assistance from the QIOs nationwide. To 
assess the effect of QIOs' assistance on nursing home quality, we 
reviewed performance requirements in the QIO contracts for both the 7th 
and the 8th SOWs; reports on QIOs' work with nursing homes, including 
the 2006 report on the QIO program by the Institute of Medicine 
(IOM);[Footnote 10] and other documents. We also conducted interviews 
with nursing homes, CMS officials, officials from state quality 
assurance programs and state MDS accuracy review programs, and experts 
on the nursing home QMs and the MDS data on which they are based. We 
conducted our review from October 2005 through May 2007 in accordance 
with generally accepted government auditing standards. (For a more 
detailed description of our scope and methodology, see app. I.)

Results in Brief:

Although QIOs generally had a choice of homes to select for intensive 
assistance because more homes volunteered than CMS expected QIOs to 
assist, QIOs typically did not target the low-performing homes that 
volunteered. Most QIOs reported in our Web-based survey that their 
primary consideration in selecting homes was their commitment to 
working with the QIO. CMS did not specify selection criteria for 
intensive participants but contracted with a QIO to develop guidelines, 
which encouraged QIOs to select homes that appeared committed to 
quality improvement and to exclude homes with a high number of survey 
deficiencies, high management turnover, or QM scores that were too good 
to improve significantly. Consistent with the guidelines, only 2 
percent of the QIOs that responded to our survey cited a high level of 
survey deficiencies among their top three considerations in choosing 
among homes that volunteered for assistance, and eight QIOs explicitly 
excluded such homes. QIOs reasoned that these homes might be more 
focused on improving their survey results than on committing time and 
resources to quality improvement projects that might target other care 
areas. Our analysis of state survey data showed that, nationwide, 
intensive participants were less likely to be low-performing than other 
homes in their state in terms of the number, scope, and severity of 
deficiencies for which they were cited in standard surveys from 1999 
through 2002. This result may reflect the nature of the homes that 
volunteered for assistance, the QIOs' selection criteria, or a 
combination of the two. Most of the stakeholders we interviewed who 
expressed an opinion said that QIOs' resources should be targeted to 
low-performing homes. CMS has directed a small share of QIO resources 
to low-performing homes in the current 8th SOW. Specifically, each QIO 
is required to provide intensive assistance to up to three 
"persistently poor-performing homes" identified in consultation with 
the state survey agency.

The 7th SOW contracts allowed QIOs flexibility in the QMs they focused 
on and the interventions they used. While the majority of QIOs selected 
the same QMs and most used the same interventions to assist homes 
statewide, the interventions used to assist intensive participants and 
staffing to accomplish program goals varied. Of eight possible QMs, 
most QIOs and intensive participants worked on chronic pain and 
pressure ulcers.[Footnote 11] While intensive participants were 
supposed to have a choice of QMs to focus on, some intensive 
participants told us that the QIO made the selection and that chronic 
pain and pressure ulcers were not necessarily their greatest quality- 
of-care challenges. The interventions QIOs relied on most for homes 
statewide were conferences and the distribution of educational 
materials; for intensive participants, they relied most on on-site 
visits, conferences, and small group meetings.[Footnote 12] Although 
the interventions QIOs used with intensive participants varied, most 
QIOs (63 percent) considered on-site visits the most effective, and 
some would make on-site visits their primary intervention if they had 
the opportunity to change the interventions they used during the 7th 
SOW. Insufficient experience or expertise and high turnover among QIO 
personnel negatively affected homes' satisfaction with the program and 
the extent of their quality improvements. Turnover was particularly 
high at 24 of the 51 QIOs, where one-quarter or more of the QIO 
personnel who assisted nursing homes worked less than half of the 36- 
month contract. One intensive participant home had four QIO principal 
contacts over the course of the 3-year contract.

The impact of QIOs on the quality of nursing home care cannot be 
determined from available data, but at most nursing homes we contacted, 
staff attributed some improvements in the quality of resident care to 
their work with QIOs. Nursing homes' QM scores generally improved 
enough for all of the QIOs to meet--and some to surpass widelyæthe 
modest targets set by CMS for improvement among homes both statewide 
and in the group assisted intensively. However, the overall impact of 
the QIOs on the quality of nursing home care cannot be determined from 
these data because of the shortcomings of the QMs as measures of 
nursing home quality and because confounding factorsæincluding homes' 
participation in other quality improvement efforts and any preexisting 
differences between homes that volunteered and were selected for 
intensive assistance and other homesæmake it difficult to attribute 
quality improvements solely to the QIOs. Multiple long-term care 
professionals we interviewed stated that QMs should not be used in 
isolation to measure quality improvement, but combined with other 
indicators, such as state survey data. In addition, the effectiveness 
of the individual interventions QIOs used to assist homes cannot be 
evaluated with the limited data CMS collected from the QIOs. CMS 
planned to enhance evaluation of the program during the 8th SOW, but a 
determination by the Department of Health and Human Services (HHS) 
Office of General Counsel that the QIO program regulations prohibit 
QIOs from providing to CMS the identities of the homes they are 
assisting has hampered the agency's efforts to collect the necessary 
data. Although we cannot determine the overall impact of the QIOs on 
the quality of nursing home care, over two-thirds of the 32 nursing 
homes we interviewed attributed some improvements in care to their work 
with the QIOs.

We are recommending that the CMS Administrator increase the extent to 
which QIOs target intensive assistance to low-performing homes and also 
direct QIOs to focus intensive assistance on the quality-of-care areas 
on which homes most need improvement. We are also recommending that the 
CMS Administrator improve monitoring and evaluation of the QIO program 
by revising program regulations to require QIOs to provide to CMS the 
identities of the nursing homes they are assisting, collecting more 
complete and detailed data on QIO interventions, and identifying a 
broader spectrum of measures than QMs to evaluate changes in nursing 
home quality. In commenting on a draft of this report, CMS concurred 
with but did not indicate how it would implement our recommendations to 
increase the number of homes that QIOs assist intensively and collect 
more complete and detailed data on the interventions QIOs use to assist 
homes. CMS did not specifically indicate if it agreed with our 
recommendation to revise program regulations to allow QIOs to reveal to 
CMS the identity of the nursing homes they are assisting, but did 
indicate that it continues to explore options which would allow access 
to such data in order to facilitate evaluation. CMS did not comment on 
the remaining two recommendations.

Background:

Beginning in the late 1990s, CMS took steps to broaden the mechanisms 
in place intended to help ensure that nursing home residents receive 
quality care. To augment the periodic assessment of homes' compliance 
with federal quality requirements, CMS contracted for the development 
of QMs and tasked QIOs with providing assistance to homes to improve 
quality. CMS used QMs both to provide the public with information on 
nursing home quality of care and to help evaluate QIO efforts to 
address quality-of-care issues, such as pressure ulcers. During the 7th 
SOW, organizations other than QIOs were also working with nursing homes 
to improve quality.

Indicators of Nursing Home Quality:

Two indicators used by CMS to assess the quality of care that nursing 
homes provide to residents are (1) deficiencies identified during 
standard surveys and complaint investigations and (2) QMs. Both 
indicators are publicly reported on CMS's Nursing Home Compare Web site.

Survey Deficiencies:

Under contract with CMS, state agencies conduct standard surveys to 
determine whether the care and services provided by nursing homes meet 
the assessed needs of residents and whether nursing homes are in 
compliance with federal quality standards.[Footnote 13] These standards 
include preventing avoidable pressure ulcers; avoiding unnecessary 
restraints, either physical or chemical; and averting a decline in a 
resident's ability to perform activities of daily living, such as 
toileting or walking.[Footnote 14] During a standard survey, a team 
that includes registered nurses spends several days at a home reviewing 
the quality of care provided to a sample of residents. States are also 
required to investigate complaints filed against nursing homes by 
residents, families, and others. Complaint investigations are less 
comprehensive than standard surveys because they generally target 
specific allegations raised by the complainants.

Any deficiencies identified during standard surveys or complaint 
investigations are classified according to the number of residents 
potentially or actually affected (isolated, pattern, or widespread) and 
their severity (potential for minimal harm, potential for more than 
minimal harm, actual harm, or immediate jeopardy). Deficiencies cited 
at the actual harm and immediate jeopardy level are considered serious 
and could trigger enforcement actions such as civil money penalties. We 
have previously reported on the considerable interstate variation in 
the proportion of homes cited for serious care problems, which ranged 
during fiscal year 2005 from 4 percent of Florida's 691 homes to 44 
percent of Connecticut's 247 homes.[Footnote 15] We reported that such 
variability suggests inconsistency in states' interpretation and 
application of federal regulations; in addition, both we and CMS have 
found that state surveyors do not identify all serious 
deficiencies.[Footnote 16]

Quality Measures:

QMs are relatively new indicators of nursing home quality. Although 
survey deficiencies have been publicly reported since 1998, CMS did not 
begin posting QMs on its Nursing Home Compare Web site until November 
2002. QMs are derived from resident assessments known as the MDS that 
nursing homes routinely collect on all residents at specified 
intervals.[Footnote 17] Conducted by nursing home staff, MDS 
assessments cover 17 areas, such as skin conditions, pain, and physical 
functioning.

In developing QMs, CMS recognized that any publicly reported indicators 
must pass a rigorous standard for validity and reliability. In October 
2002, we reported that national implementation of QMs was premature 
because of validity and reliability concerns.[Footnote 18] Valid QMs 
would distinguish between good and poor care provided by nursing homes; 
reliable QMs would do so consistently. One of our main concerns about 
publicly reporting QMs was that the QM scores might be influenced by 
other factors, such as residents' health status. As a result, the 
specification of appropriate risk adjustment was a key requirement for 
the validity of any QMs. Risk adjustment is important because it 
provides consumers with an "apples-to-apples" comparison of nursing 
homes by taking into consideration the characteristics of individual 
residents and adjusting the QM scores accordingly. For example, a home 
with a disproportionate number of residents who are bedfast or who 
present a challenge for maintaining an adequate level of nutrition-- 
factors that contribute to the development of pressure ulcers--may have 
a higher pressure ulcer score. Adjusting a home's QM score to fairly 
represent to what extent a home does or does not admit such residents 
is important for consumers who wish to compare one home to another. 
Appendix II lists the 10 QMs initially adopted and publicly reported by 
CMS--6 applicable to residents with chronic care problems (long-stay 
residents) and 4 applicable to residents with post-acute-care needs 
(short-stay residents).

MDS data are self-reported by nursing homes, and ensuring their 
accuracy is critical for establishing resident care plans, setting 
nursing home payments, and publicly reporting QM scores. In February 
2002, we concluded that CMS efforts to ensure the accuracy of MDS data, 
which are used to calculate the QMs, were inadequate because the agency 
relied too much on off-site review activities by its contractor and 
planned to conduct on-site reviews in only 10 percent of its data 
accuracy assessments, representing fewer than 200 of the nation's then 
approximately 17,000 nursing homes.[Footnote 19] Although we 
recommended that CMS reorient its review program to complement ongoing 
state MDS accuracy efforts as a more effective and efficient way to 
ensure MDS data accuracy, CMS disagreed and continued to emphasize off- 
site reviews.[Footnote 20]

Evolution of the QIO Program and the Quality Improvement Process:

Over the past 24 years, the QIO program has evolved from a focus on 
quality assurance in the acute care setting to quality improvement in a 
broader mix of settings, including physician offices, home health 
agencies, and nursing homes. Established by the Peer Review Improvement 
Act of 1982[Footnote 21] and originally known as Peer Review 
Organizations (PRO), QIOs initially focused on ensuring minimum 
standards by conducting retrospective hospital-based utilization 
reviews that looked for inappropriate or unnecessary Medicare services. 
According to the 2006 IOM report, as it became apparent that standards 
of care themselves required attention, QIOs gradually shifted from 
retrospective case reviews to collaboration with providers to improve 
the overall delivery of care--a shift consistent with transformational 
goals set by CMS's Office of Clinical Standards and Quality, which 
oversees the QIO program.[Footnote 22]

In contrast to enforcing standards, quality improvement tries to ensure 
that organizations have effective processes for continually measuring 
and improving quality. The goal of quality improvement is to close the 
gap between an organization's current performance and its ideal 
performance, which is defined by either evidence-based research or best 
practices demonstrated in high-performing organizations. According to 
the quality improvement literature, successful quality improvement 
requires a commitment on the part of an organization's leadership and 
active involvement of the staff. The 2006 IOM report notes that QIOs 
rely on various mechanisms to promote quality improvement, including 
one-on-one consulting and collaboratives.[Footnote 23] While the former 
provides direct and specialized attention, the latter relies on 
workshops or meetings that offer opportunities for providers to share 
experiences and best practices. Quality improvement often relies on the 
involvement of early adopters of best practices--providers who are 
highly regarded as leaders and can help convince others to change--for 
the diffusion of best practices. Key tools for quality improvement 
include (1) root cause analysis, a technique used to identify the 
conditions that lead to an undesired outcome; (2) instruction on how to 
collect, aggregate, and interpret data; and (3) guidance on bringing 
about, sustaining, and diffusing internal system redesign and process 
changes, particularly those related to use of information technology 
for quality improvement. Quality improvement experts also emphasize the 
importance of protecting the confidentiality of provider information, 
not only to protect the privacy of personal health information but also 
to encourage providers to evaluate their peers honestly and to prevent 
the damage to providers' reputations that might occur through the 
release of erroneous information.

Section 1160 of the Social Security Act provides that information 
collected by QIOs during the performance of their contract with CMS 
must be kept confidential and may not be disclosed except in specific 
instances; it provides the Secretary of HHS with some discretion to 
determine instances under which QIO information may be disclosed. The 
regulations implementing the statute limit the circumstances under 
which confidential information obtained during QIO quality review 
studies, including the identities of the participants of those studies, 
may be disclosed by the QIO. During the 7th SOW, QIOs submitted a list 
of nursing home participants to CMS as a contract deliverable.

CMS Contract Funding and Requirements:

During the 7th SOW, CMS awarded a total of $117 million to QIOs to 
improve the quality of care in nursing homes in all 50 states, the 
District of Columbia, and the territories. The performance-based 
contracts for QIO assistance to nursing homes delineated broad 
expectations regarding QIO assistance to nursing homes, provided 
deadlines for completing four contract deliverables, and laid out 
criteria for evaluating QIO performance.[Footnote 24] For contracting 
purposes, the QIOs were divided into three groups with staggered 
contract cycles. The four contract deliverables, however, were all due 
on the same dates, irrespective of the different contract cycles. The 
contracts also required QIOs to work with a QIO support contractor 
tasked to provide guidelines for recruiting and selecting nursing homes 
as intensive participants, train QIOs in standard models of quality 
improvement assistance, and provide tools and educational materials, as 
well as individualized consultation if needed, to help QIOs meet 
contractual requirements.[Footnote 25] QIOs and nursing homes were also 
involved in other quality improvement special studies with budgets 
separate from the QIO contracts for the 7th SOW. These studies varied 
greatly in terms of length, the clinical issue(s) covered, the number 
of QIOs involved, and the characteristics of the nursing homes that 
participated. Figure 1 shows the 7th SOW contract cycles, deliverables 
for the nursing home component, and the duration of the special studies.

Figure 1: Timeline for 7th SOW Contract and Concurrent Special Studies 
by QIOs to Improve the Quality of Nursing Home Care:

[See PDF for image]

Source: GAO analysis of the 7th SOW and CMS descriptions of special 
studies.

[A] In the 7th SOW, QIOs were divided into three groups with staggered 
contract cycles. The four contract deliverables, however, were all due 
on the same dates, irrespective of contract cycle.

[B] The term states includes the 50 United States and the District of 
Columbia.

[C] QIOs could add--but not delete or change--QMs for their intensive 
participants through September 2003.

[End of figure]

Contract funding. The $117 million awarded to QIOs to improve the 
quality of care in nursing homes during the 7th SOW included (1) $106 
million awarded to provide statewide and intensive assistance to 
homes,[Footnote 26] (2) $5.6 million awarded to selected QIOs to 
conduct eight special studies focused on nursing home care, and (3) 
$5.3 million awarded to the QIO that served as the support contractor 
for the nursing home component.[Footnote 27] CMS allocated a specific 
amount for each component of the contracts, but allowed QIOs to move 
funds among certain components.[Footnote 28] Just over half of the 51 
QIOs did not spend all of the funds allocated to the nursing home 
component, but on average the QIOs overspent the budget for the nursing 
home work by 3 percent.

Contract requirements for quality improvement activities. Per the 
contracts for the 7th SOW, QIOs were required to provide (1) all 
Medicare-and Medicaid-certified homes with information about systems- 
based approaches to improving patient care and clinical outcomes, and 
(2) intensive assistance to a subset of homes in each state. The 
contracts directed QIOs working in states with 100 or more nursing 
homes to target 10 to 15 percent of the homes for intensive 
assistance.[Footnote 29] Figure 2 illustrates that QIOs provided two 
levels of assistance--statewide and intensive--and that homes' 
participation was either nonintensive or intensive. Intensive 
participants received both statewide and intensive assistance. 
Selection of intensive participants from among the nursing homes that 
volunteered was at the discretion of each QIO, but the SOW required the 
QIO support contractor (the Rhode Island QIO) to provide guidelines and 
criteria for QIOs to use in determining which homes to select. 
Participation in the program was voluntary, and QIOs were prohibited 
from releasing the names of participating nursing homes except as 
permitted by statute and regulation.[Footnote 30]

Figure 2: Levels of QIO Assistance and Nursing Home Participation in 
the 7th SOW:

[See PDF for image]

Source: GAO analysis of the 7th SOW.

[A] Nursing homes on the official list of intensive participants 
submitted to CMS by the QIOs by February 3, 2003.

[End of figure]

Under the contracts, the quality improvement assistance provided by 
QIOs focused on areas related to eight chronic care and post-acute-care 
QMs publicly reported on the CMS Nursing Home Compare Web site. QIOs 
were required to consult with relevant stakeholders and select from 
three to five of the eight QMs on which QIOs' quality improvement 
efforts would be evaluated (see table 1).[Footnote 31] Intensive 
participant homes were also required to select one or more QMs on which 
to work with the QIO. Although they could select one QM, they were 
encouraged to select more than one.

Table 1: Quality Measures on Which QIOs Could Focus Their Quality 
Improvement Efforts in the 7th SOW:

Chronic care QMs: Decline in activities of daily living; Post-acute- 
care QMs: Failure to improve and manage delirium.

Chronic care QMs: Pressure ulcers; Post-acute-care QMs: Inadequate pain 
management.

Chronic care QMs: Inadequate pain management; Post-acute-care QMs: 
Improvement in walking.

Chronic care QMs: Physical restraints; Post-acute-care QMs: [Empty].

Chronic care QMs: Infections; Post-acute-care QMs: [Empty].

Source: CMS.

Note: Although CMS adopted 10 QMs, the QIOs were evaluated only on the 
8 listed here (see app. II).

[End of table]

To improve QM scores, QIOs were expected to develop and implement 
quality improvement projects focused on care processes known to improve 
patient outcomes in a manner that utilized resources efficiently and 
reduced burdens on providers. The QIO support contractor developed a 
model for QIOs to facilitate systems change in nursing homes. This 
model emphasized the importance of QIOs' statewide activities to form 
and maintain partnerships, conduct workshops and seminars, and 
disseminate information on interventions to improve quality. For 
intensive participants, the model emphasized conducting one-on-one 
quality improvement assistance as well as conferences and small group 
meetings. According to contract language, QIOs were expected to 
coordinate their projects with other stakeholders that were working on 
similar improvement efforts or were interested in teaming with the QIO. 
But ultimately, each QIO determined for itself the type, level, 
duration, and intensity of support it would offer to nursing homes.

Evaluation of QIO contract performance. CMS evaluated QIOs' performance 
on the nursing home component of the contract using nursing home 
provider satisfaction with the QIO, QM improvement among intensive 
participants, and QM improvement statewide (see fig. 3).[Footnote 32] 
Nursing home provider satisfaction was assessed by surveying all 
intensive participants and a sample of nonintensive participants around 
the 28th month of each 36-month contract. CMS expected at least 80 
percent of respondents to report that they were either satisfied or 
very satisfied.

Figure 3: QIO Contract Evaluation Scoring Methodology for the 7th SOW:

[See PDF for image]

Note: QM improvement was calculated using the following formula: 
(baseline rate minus remeasurement rate) / baseline rate. For example, 
if a nursing home had a baseline rate of 20 percent for the pain 
management QM (e.g., 20 percent of the home's residents had severe or 
moderate pain), a 10 percent improvement would mean that 18 percent of 
residents had moderate or severe pain at remeasurement [(20 percent - 
18 percent) / 20 percent].

[A] All intensive participants and a sample of nonintensive 
participants were surveyed to assess their satisfaction with the QIO.

[B] The weight (percentage of total score) given to this element 
depended on the proportion of the state's homes that were included in 
the intensive participant group; the weight ranged from 44 percent, if 
10 percent of the homes were included, to 66 percent, if at least 15 
percent of the homes were included.

[C] The weight (percentage of total score) given to this element was 
the difference between 80 percent and the weight given to the intensive 
participant element and ranged from 14 to 36 percent.

[End of figure]

QIOs were also expected to achieve an 8 percent improvement in QM 
scores among both intensive participants and homes statewide. The term 
improvement was defined mathematically to mean the relative change in 
the QM score from when it was measured at baseline to when it was 
remeasured. The statewide improvement score included the QM improvement 
scores for intensive participants averaged with those of nonintensive 
participants.

CMS established two scoring thresholds for the contracts that 
encompassed scores from all components of the SOW. If a QIO scored 
above the first threshold it was eligible for a noncompetitive contract 
renewal; if it scored below that threshold, it was eligible for a 
competitive renewal only upon providing information pertinent to its 
performance to a CMS-wide panel that decided whether to allow the QIO 
to bid again for another QIO contract.[Footnote 33]

CMS contract monitoring. CMS formally evaluated each QIO at months 9 
and 18 of the 7th SOW. If CMS found that a QIO failed to meet contract 
deliverables or appeared to be in danger of failing to meet contract 
goals, it could require the QIO to make a performance improvement plan 
or corrective plan of action to address any barriers to the QIOs 
successfully fulfilling contract requirements. In addition, CMS 
reviewed materials such as QIOs internal quality control plans, which 
were intended to help QIOs monitor their own progress and to document 
any project changes made to improve their performance.

Other Nursing Home Quality Improvement and Assurance Initiatives:

The QIO program operated in the context of other quality improvement 
initiatives sponsored by federal and state governments and nursing home 
trade associations. As stated earlier, CMS funded a number of special 
nursing home studies involving subsets of the QIOs and nursing homes, 
which addressed a variety of clinical quality-of-care issues and which 
are summarized in figure 1. Under CMS's Special Focus Facility program, 
state survey agencies were required to conduct enhanced monitoring of 
nursing homes with histories of providing poor care. During the 7th 
SOW, CMS revised the method for selecting homes for the Special Focus 
Facility program to ensure that the homes performing most poorly were 
included; increased the minimum number of homes that must be included, 
from a minimum of two per state to a minimum of up to six, depending on 
the number of homes in the state; and strengthened enforcement for 
those nursing homes with an ongoing pattern of substandard 
care.[Footnote 34] In addition, concurrent with the 7th SOW, at least 
eight states had programs that provided quality assurance and technical 
assistance to nursing homes in their states.[Footnote 35] These 
programs varied in terms of whether they were voluntary or mandatory, 
which homes received assistance, the focus and frequency of the 
assistance provided, and the number and type of staff employed.

In addition to government-operated quality improvement initiatives, 
three long-term care professional associations joined together in July 
2002 to implement the Quality First Initiative.[Footnote 36] This 
initiative was based on a publicly articulated pledge on the part of 
the long-term care profession to establish an environment of continuous 
quality improvement, openness, and leadership in participating homes.

QIOs Generally Had a Choice among Homes That Volunteered but Did Not 
Target Assistance to Low-Performing Homes:

Although QIOs generally had a choice of homes to select for intensive 
assistance because more homes volunteered than CMS expected QIOs to 
assist, QIOs typically did not target the low-performing homes that 
volunteered. Most QIOs reported in our Web-based survey that they did 
not have difficulty recruiting homes, and their primary consideration 
in selecting homes from the pool of volunteers was that the homes be 
committed to working with the QIOs. In the 7th SOW, CMS did not specify 
recruitment and selection criteria for intensive participants, leaving 
the development of guidelines to the QIO support contractor, which 
encouraged QIOs to select homes that seemed committed to quality 
improvement and to exclude homes with a high number of survey 
deficiencies, high management turnover, or QM scores that were too good 
to improve significantly.[Footnote 37] Our analysis of state survey 
data showed that, nationwide, intensive participants were less likely 
to be low-performing than other homes in their state in terms of the 
number, scope, and severity of deficiencies for which they were cited 
in standard surveys from 1999 through 2002. This result may reflect the 
nature of the homes that volunteered for assistance, the QIOs' 
selection criteria, or a combination of the two. The stakeholders we 
interviewed--including officials of state survey agencies and nursing 
home trade associations--generally believed QIOs' resources should be 
targeted to low-performing homes.

QIOs Generally Had a Choice of Which Nursing Homes to Assist 
Intensively:

Most QIOs had a choice of which nursing homes to assist intensively, as 
more homes volunteered than the QIOs could receive credit for serving 
under the terms of their contracts.[Footnote 38] Of the 38 QIOs in 
states with 100 or more homes, which were expected to work intensively 
with 10 to 15 percent of the homes, 30 reported in our Web-based survey 
that more than 15 percent of homes expressed interest in intensive 
assistance, and 8 reported that more than 30 percent of homes expressed 
interest.[Footnote 39] Most QIOs selected about as many intensive 
participants as needed to get the maximum weight for the intensive 
participant element of their contract evaluation score. Nationwide, the 
intensive participant group included just under 15 percent (2,471) of 
the 16,552 homes identified by CMS at the beginning of the 7th 
SOW.[Footnote 40]

Most QIOs--82 percent of the 51 that responded to our survey--reported 
that it was not difficult to recruit the target number of homes for 
intensive assistance; the remainder reported that it was difficult (12 
percent) or very difficult (4 percent) to recruit enough 
volunteers.[Footnote 41] Among the QIOs we interviewed, personnel at 
two that reported difficulties recruiting homes cited homes' lack of 
familiarity with QIOs as a barrier. Personnel at one of these two QIOs 
commented that the QIO's first task was to build trust among homes and 
address confusion about its role, as some homes thought the QIO was a 
regulatory authority charged with investigating complaints and citing 
homes for deficiencies.

Commitment to Working with QIOs Was QIOs' Primary Consideration in 
Selecting Homes from among Those That Volunteered:

QIOs that responded to our Web-based survey almost uniformly cited 
homes' commitment to working with them as a key consideration in 
choosing among the homes that volunteered to be intensive participants. 
QIOs had wide latitude in choosing among homes because CMS did not 
specify the characteristics of the homes they should recruit or select, 
leaving it to the QIO support contractor to provide voluntary 
guidelines. The QIO support contractor developed guidelines based on 
input from a variety of sources, including QIOs that worked with 
nursing homes during the 6TH SOW. Issued at the beginning of the 7th 
SOW, the guidelines emphasized the important role the selected homes 
would play in the QIOs' contract performance and encouraged QIOs to 
select homes that demonstrated a willingness and ability to commit time 
and resources to quality improvement. The QIO support contractor also 
encouraged QIOs to exclude homes with a high number of survey 
deficiencies, high management turnover, and QM scores that were too 
good to improve significantly. With respect to homes' survey histories, 
the QIO support contractor reasoned that homes with a high number of 
deficiencies might be more focused on improving their survey results 
than on committing time and resources to quality improvement projects. 
For example, the care areas in which a home was cited for deficiencies 
might not correspond with any of the eight QMs to which CMS limited the 
QIOs' quality improvement activities (see table 1). In fact, the 
quality of care area in which homes were most frequently cited for 
serious deficiencies in surveys in 2006 was the provision of 
supervision and devices to prevent accidents, which does not have a 
corresponding QM.[Footnote 42]

Consistent with the guidelines, 76 percent of the 41 QIOs that reported 
in our Web-based survey their considerations in selecting homes for the 
intensive participant group ranked homes' commitment as their primary 
consideration. Nearly all QIOs ranked commitment among their top three 
considerations (see fig. 4).[Footnote 43]

Figure 4: QIOs' Considerations in Choosing among Homes That Volunteered 
for Intensive Assistance in the 7th SOW:

[See PDF for image]

Note: Forty-one QIOs reported their considerations in choosing among 
homes that volunteered for intensive assistance.

[End of figure]

Homes' QM scores were also an important consideration for QIOs. QIOs 
were particularly interested in including homes that had poor QM scores 
in areas where the QIO planned to focus or in assembling a group of 
homes that represented a mix of QM scores. With respect to homes' 
overall QM scores, the QIOs that responded to our survey were more 
likely to seek homes with moderate overall scores than homes with poor 
or good overall scores. Similarly, personnel at most QIOs we contacted 
gave serious consideration to homes' QM scores, looking for homes that 
appeared to need help and could demonstrate improvement. For example, 
personnel at one QIO said that they tended to select homes whose QM 
scores were worse than the statewide average; personnel at another QIO 
said that this QIO selected homes with scores it thought could be 
improved, eliminating homes with either very high or very low scores. 
Personnel at one QIO acknowledged that some QIOs might "cherry pick" 
homes in this way in order to satisfy CMS contract requirements but 
argued that it was not possible for QIOs to predict which homes would 
improve the most.

QIOs generally gave less consideration to the number of deficiencies 
homes had on state surveys than to their QM scores. However, the 17 
QIOs that ranked survey deficiencies among their top three 
considerations in our survey were more likely to seek homes with 
deficiencies in areas where they planned to focus or homes with an 
overall low level (number and severity) of survey deficiencies than 
homes with an overall high level. Moreover, of the 33 QIOs that 
reported in our survey systematically excluding some of the homes that 
volunteered from the intensive participant group, nearly one-quarter 
(8) excluded homes with a high number of survey deficiencies. None 
excluded homes with a low number of survey deficiencies.[Footnote 44]

Personnel at the QIOs we interviewed offered several reasons for 
excluding homes with a high number of survey deficiencies from the 
intensive participant group. Personnel at several QIOs concurred with 
the QIO support contractor that such homes were likely to be too 
consumed with correcting survey issues to focus on quality improvement. 
Personnel at one QIO suggested that the kind of assistance very poor- 
performing homes needæhelp improving the basic underlying structures of 
operationæwas not the kind the QIO offered. Personnel at some QIOs said 
they considered not just the level of deficiencies for which homes were 
cited on recent surveys but the level over multiple years or the 
specific categories of deficiencies. For example, personnel at one QIO 
said that although the QIO excluded homes with long-standing histories 
of poor performance, it actively recruited homes that had performed 
poorly only on recent surveys. Personnel at another QIO stated that 
their concern was to avoid homes with competing priorities. This QIO 
sought to include homes with deficiencies in the areas it planned to 
address but to exclude homes with deficiencies in other areas on the 
assumption that these homes would not benefit from the assistance it 
planned to offer. Personnel we interviewed at two QIOs said that they 
worked with some extremely poor-performing homes but did not include 
them on the official list of intensive participants submitted to CMS; 
personnel at one of these QIOs explained that they did not want to be 
held responsible if these homes were unable to improve.

QIOs Did Not Target Intensive Assistance to Low-Performing Homes:

Our analysis of homes' state survey histories from 1999 through 2002 
indicates that QIOs did not target intensive assistance to homes that 
had performed poorly in state surveys. Nationwide, the homes in the 
intensive participant group were less likely than other homes in their 
state to be low-performing in terms of the number, scope, and severity 
of deficiencies for which they were cited in surveys during that time 
frame. As illustrated in figure 5, the intensive participant group 
included proportionately more homes in the middle of the performance 
spectrum and proportionately fewer at either end. Although our analysis 
focused on survey deficiencies rather than QMs, this result is 
generally consistent with the results of our Web-based survey 
concerning QIOs' use of QM scores as selection criteria, which showed 
that QIOs were more likely to select homes with moderate overall scores 
than homes with poor or good overall scores and to seek a mix of 
performance levels among homes in the group. However, not knowing the 
composition of the pool of homes that volunteered for assistance, we 
cannot determine whether the composition of the intensive participant 
groupæin particular, the disproportionately low number of low- 
performing homes in the groupæwas a function of which homes 
volunteered, which homes the QIOs selected from among the volunteers, 
or a combination of both factors.

Figure 5: Comparison of Nonintensive and Intensive Participants' 
Performance on State Surveys:

[See PDF for image]

Note: Homes are categorized as low-, moderately, or high-performing on 
the basis of the number, scope, and severity of deficiencies for which 
they were cited, relative to other homes in their state, in three 
standard state surveys from 1999 through 2002. All differences are 
statistically significant at p-value < 0.05 level.

[End of figure]

On a state-by-state basis, none of the QIOs targeted assistance to low- 
performing homes by including proportionately more such homes in the 
intensive participant group. Most QIOs (33 of 51) worked intensively 
with homes that were generally representative of the range of homes in 
their state in terms of performance on state surveys from 1999 through 
2002. In these states, there was no significant difference in the 
proportion of high-, moderately, or low-performing homes among 
intensive participants compared with nonintensive participants. 
However, 18 QIOs worked intensively with a group that differed 
significantly from other homes in the state: 8 of these QIOs worked 
with proportionately fewer low-performing homes, 5 worked with 
proportionately more moderately performing homes, and 9 worked with 
proportionately fewer high-performing homes.[Footnote 45]

Stakeholders Often Stated QIOs Should Target Intensive Assistance to 
Low-Performing Homes:

Stakeholders we interviewed who expressed an opinion about the homes 
QIOs should target for intensive assistance--11 of the 16 we 
interviewed--almost uniformly said that the QIOs should concentrate on 
low-performing homes.[Footnote 46] Survey officials in one state 
suggested that QIOs should use state survey data to assess homes' need 
for assistance because these data are often more current than QM data. 
In their emphasis on low-performing homes, stakeholders echoed the 
views expressed in the 2006 IOM report, which recommended that QIOs 
give priority for assistance to providers, including nursing homes, 
that most need improvement. Other stakeholder suggestions regarding the 
homes QIOs should target are listed in table 2. Because the QIOs were 
required to protect the confidentiality of QIO information about 
nursing homes that agreed to work with them, stakeholders were 
generally not informed which homes were receiving intensive assistance. 
One exception was in Iowa, where the QIO obtained consent from the 
selected homes to reveal their identities.

Table 2: Examples of Other Categories of Homes Stakeholders Suggested 
QIOs Should Include as Intensive Participants:

Category of home: Special focus facilities; Explanation: One state 
survey official suggested that CMS mandate that QIOs work with the low- 
performing homes selected by state survey agencies for the Special 
Focus Facility program.[A].

Category of home: Homes lacking resources for quality improvement; 
Explanation: Stakeholders suggested targeting small rural facilities, 
"stand-alone" facilities that lack the resources of corporate chains, 
or facilities that are struggling financially..

Category of home: High-performing homes; Explanation: Several 
stakeholders advocated including some high-performing homes. One 
stakeholder group suggested that such homes could serve as models and 
share their approaches with homes that were struggling. Another 
suggested that QIOs may include homes at varying performance levels to 
avoid stigmatizing the intensive participants as "bad homes.".

Source: GAO analysis.

Note: Eleven of the 16 stakeholders we interviewed expressed an opinion 
about which homes the QIOs should include as intensive participants.

[A] Seventeen (13 percent) of the 129 facilities in the Special Focus 
Facility program as of January 2005 were also among the QIOs' 2,471 
intensive participants in the 7th SOW.

[End of table]

Several stakeholders said that low-performing homes can improve with 
assistance. However, one suggested that QIOs might have to adapt their 
approachæfor example, by streamlining their trainingæto avoid 
overburdening homes that are struggling with competing demands. Another 
agreed that low-performing homes can benefit from working with a QIO 
but added that real improvements in the quality of care in these homes 
would require attention to staffing, turnover, pay, and recognition for 
staff. The results of one special study funded by CMS during the time 
frame of the 7th SOW supported stakeholders' contention that low- 
performing homes can improve, although the improvements documented in 
these homes cannot be definitively attributed to the QIOs.[Footnote 47] 
In this study, known as the Collaborative Focus Facility project, 17 
QIOs worked intensively with one to five low-performing homes 
identified in consultation with the state survey agency.[Footnote 48] 
According to a QIO assessment of the project, the participating homes 
showed improvement in areas related to the assistance provided by the 
QIO in terms of both the number of serious state survey deficiencies 
for which they were cited and their QM scores.[Footnote 49] CMS 
officials pointed out that these improvements were hard-won: one-third 
of the homes that were asked to participate in the Collaborative Focus 
Facility project refused, and those that did participate required more 
effort and resources from the QIOs to improve than did other homes 
assisted by the QIOs.

Overall, CMS has specifically directed only a small share of QIO 
resources to low-performing homes. In the current contracts (the 8th 
SOW), CMS required QIOs to provide intensive assistance to some 
"persistently poor-performing homes" identified in consultation with 
each state survey agency. However, the number of such homes that the 
QIOs must serve is smallæranging from one to three, depending on the 
number of nursing homes in the stateæand accounts for less than 10 
percent of the homes the QIOs are expected to assist intensively. Less 
than 17 percent of the 144 persistently poor-performing homes the QIOs 
selected in consultation with state survey agencies to assist in the 
8th SOW were also special focus facilities in 2005 or 2006.

QIOs and stakeholders tended to disagree about whether participation in 
the program should remain voluntary for all homes. QIO personnel we 
interviewed who expressed an opinion generally supported voluntary 
participation on the theory that homes that were forced to participate 
would probably be less engaged and put forth only minimal effort. 
Personnel at some QIOs that opposed mandatory participation suggested 
that creating incentives for homes to improve their quality of careæfor 
example, through pay for performanceæwould increase homes' interest in 
working with the QIO. In contrast, most of the state survey agency and 
trade association officials we interviewed who expressed an opinion 
about the voluntary nature of the QIO program said that some homes 
should be required to work with the QIO. Officials at one state survey 
agency pointed out that the low-performing homes that really need 
assistance rarely seek it; these officials believed that working with 
the QIO should be mandatory for low-performing homes and voluntary for 
moderately to high-performing homes. Another state survey agency 
official recommended that 25 to 40 percent of the homes assisted 
intensively be chosen from among the lower-performing homes in the 
state and required to work with the QIO.

QIO Contract Flexibility Resulted in Variation in Assistance Provided 
to Intensive Participants:

The 7th SOW contracts allowed QIOs flexibility in the QMs they focused 
on and the interventions they used, and while the majority of QIOs 
selected the same QMs and most used the same interventions to assist 
homes statewide, the interventions for intensive participants and 
staffing to accomplish program goals varied. Most QIOs and intensive 
participants worked on the chronic pain and pressure ulcer QMs, but 
these were not the QMs that some intensive participants believed 
matched their greatest quality-of-care challenges. To assist all homes 
statewide, QIOs generally relied on conferences and the distribution of 
educational materials. The top three interventions for intensive 
participants included on-site visits (87 percent), followed by 
conferences (57 percent), and small group meetings (48 percent). 
According to nursing home staff we interviewed, turnover and experience 
levels of the QIO personnel that provided them assistance affected 
their satisfaction with the program and the extent of their quality 
improvements.

Most Quality Improvement Efforts Focused on Chronic Pain and Pressure 
Ulcers:

Under the terms of the contracts, both QIOs and intensive participants 
could select QMs to focus on, but most chose to work on two of the same 
QMs.[Footnote 50] While nearly all QIOs chose to work statewide on 
chronic pain and pressure ulcers, they differed on their selection of 
additional QMs (see fig. 6). QIO personnel we interviewed told us they 
based the choice of QMs for their statewide work on input from 
stakeholders and nursing homes or QM data. For example, some 
stakeholders told us that specific QMs selected addressed existing long-
term care challenges and were ones on which homes in the state ranked 
below the national average. Personnel from two QIOs said they selected 
QMs based on input from homes in their state about which QMs the homes 
were interested in working on, and personnel from several QIOs stated 
that they selected QMs on which their homes could improve. Personnel 
from one QIO specifically mentioned that they selected QMs related to 
the quality of life for nursing home residents.

Figure 6: QMs Selected by QIOs for Statewide Interventions and QMs 
Selected by Nursing Homes for Intensive Assistance, 7th SOW:

[See PDF for image]

Source: GAO analysis of QIO support contractor data.

[End of figure]

Most intensive participants worked on a subset of the QMs selected by 
their QIO--chronic pain and pressure ulcers (see fig. 6). The degree to 
which intensive participants knew they had a choice of QMs was unclear. 
Of the 14 intensive participants we interviewed that commented on 
whether they had a choice, 9 said that they did. Staff from these homes 
generally reported having selected QMs related to clinical issues on 
which they could improve. However, the remaining 5 homes indicated that 
their QIO selected the QMs on which they received assistance. Most of 
these 5 homes' staff reported that they would have preferred to work on 
different QMs from the list of eight that are publicly reported on the 
CMS Nursing Home Compare Web site or other clinical issues that reflect 
their greatest quality-of-care challenges.

Statewide Interventions Less Variable Than Those for Intensive 
Participants:

The terms of the QIO contract with CMS allowed QIOs to determine the 
kinds of quality improvement interventions they offered to homes, and 
those selected by QIOs were consistent with an approach recommended by 
the QIO support contractor: QIOs generally relied most on conferences 
and the distribution of educational materials to assist homes statewide 
and on on-site visits to assist intensive participants. However, there 
was a greater variety of interventions frequently relied on to assist 
intensive participants. In general, QIOs reported that the 
interventions they relied on most were also the most effective for 
improving the quality of resident care.

Statewide Assistance:

Almost three-quarters of the QIOs included conferences among the two 
interventions they relied on most to provide quality improvement 
assistance to homes statewide (see fig. 7). These QIOs held an average 
of nine conferences over the course of the 7th SOW, typically in 
various cities throughout the state to accommodate homes from different 
regions. Sixty-eight percent of these QIOs reported that more than half 
the homes in their state sent staff to least one conference, and 16 
percent of QIOs reported that all or nearly all homes did so. QIO 
personnel reported holding conferences to educate homes on quality 
improvement, discuss the relationship between MDS assessments and the 
QMs, and provide QM-specific clinical information or best practices. 
Some conferences included presentations by state or national experts.

Figure 7: Statewide Interventions Most Relied on by QIOs, 7th SOW:

[See PDF for image]

Source: GAO survey of QIOs. 

[End of figure]

Almost three-quarters of QIOs also ranked the distribution of 
educational materials by mail, fax, or e-mail among their top two 
statewide interventions. Thirty-two percent of these QIOs sent 
materials four or fewer times per year, whereas 27 percent sent 
materials 12 or more times per year to all or nearly all homes in the 
state. For the QIOs we interviewed, these materials included 
newsletters, QM-specific tools or clinical information related to the 
QMs, and QM data progress reports for the home or state, overall.

Almost one-third of the QIOs (31 percent) reported that the type or 
intensity of interventions they used to assist homes statewide changed 
over the course of the 7th SOW.[Footnote 51] For example, two QIOs 
reported that they concentrated much of their statewide efforts into 
the first half of the 3-year period; one QIO specifically reported 
doing so in the interest of ensuring that any improvements in QMs were 
reflected in its evaluation scores, which, as specified by the 
contract, were calculated near the mid-point of the contract 
cycle.[Footnote 52] In contrast, five other QIOs reported that they 
increased the intensity of their statewide work over time, in some 
cases concentrating on homes whose performance was lagging.

For the 8th SOW, CMS has focused resources on assistance to intensive 
participants by eliminating expectations for improvements in QMs 
statewide. However, the contracts still contain statewide elements, 
including a requirement to promote QM target-setting.

Intensive Assistance:

Fifty-one percent of QIOs ranked on-site visits as their most relied on 
intervention with intensive participants and 87 percent ranked it among 
their top three interventions (see fig. 8).[Footnote 53] Both the 
number of visits and the time spent at sites varied considerably. The 
median number of visits was 5 but ranged from 1 to 20.[Footnote 54] 
Sixty-eight percent of QIOs that included on-site visits among their 
top three interventions spent an average of 1 to 2 hours at sites each 
time they visited, while 20 percent spent 3 to 4 hours. QIOs that 
ranked on-site visits as their number one intervention made more and 
longer visits to intensive participants than did QIOs that ranked them 
lower. When surveyed about a typical on-site visit, the majority of QIO 
respondents reported that they generally reviewed the homes' QM data, 
provided education or best practices, or both. Approximately one-third 
of QIOs that conducted site visits indicated that they had discussions 
with the home about their systems or processes for care, homework 
assignments, or quality improvement activities.[Footnote 55] Some QIOs 
(26 percent) reported that they conducted team-building exercises with 
the staff when on site.

Figure 8: Intensive Interventions Most Relied on by QIOs and Frequency 
of Interventions (Range and Median Number) during the 7th SOW:

[See PDF for image]

Source: GAO survey of QIOs.

[A] The median number of times an intervention was provided is the 
midpoint of all the times that an intervention was provided in the 7th 
SOW, as reported by QIOs.

[End of figure]

QIOs varied in the interventions they used in addition to on-site 
visits, with conferences, small group meetings emphasizing peer-to-peer 
learning, and telephone calls being the three others most commonly 
used. QIOs that included conferences among their three most relied on 
interventions typically held between 3 and 10 during the 7th SOW, but 
as with site visits, some variation existed. After conferences, QIOs 
were most likely to rely on small group meetings and telephone calls 
with individual homes. Nearly half of the QIOs ranked these two 
interventions among their three most relied on, but few ranked them 
highest. The number of homes that attended small group meetings varied. 
An average of 6 to 10 homes was most common, but one-fifth of QIOs 
reported having an average of 20 or more homes represented at each 
meeting. As for telephone calls, the vast majority of QIOs (92 percent) 
that ranked these calls among their three most relied on interventions 
called all or nearly all of their intensive participants, typically on 
a monthly basis.

Our interviews with QIOs and intensive participant homes suggested that 
the small group meetings they held generally followed a similar format, 
while telephone calls were used for a variety of purposes. For example, 
personnel from several QIOs and intensive participant homes told us 
that their small group meetings generally included a formal 
presentation on the QMs or related best practices, as well as a time 
for less formal information sharing and peer-to-peer learning among the 
attendees. Participants shared stories about their successes and 
challenges conducting quality improvement. Personnel from a number of 
QIOs told us they used telephone calls to follow up after visits or 
meetings, discuss the homes' progress on quality improvement, and to 
decide on next steps.

Almost two-thirds of QIOs indicated that the type or intensity of 
interventions for intensive participants varied over time. Of these 
QIOs, 36 percent reduced the intensity of their interventions 
(substituting small group meetings or telephone calls for on-site 
visits), while 33 percent did the reverse (in some cases increasing the 
frequency of on-site visits or substituting small group meetings for 
conferences to increase participation). For example, personnel from a 
few QIOs told us that while they initially relied on on-site visits to 
begin the quality improvement process, they came to rely more on 
telephone calls or on small group meetings where intensive participants 
could share their success stories or ways to overcome barriers to 
quality improvement. Seventy-nine percent of QIOs surveyed varied their 
interventions based on the needs of intensive participants. Thus, 
personnel from three QIOs told us they realized that some homes did not 
need frequent on-site visits, while others needed more. The two 
specific needs that QIOs cited most as having precipitated changes in 
their interventions were nursing home staffing changes and turnover (23 
percent) and poorer performance by some homes relative to others (15 
percent). A few QIOs also noted that interventions varied by the 
preferences or levels of readiness and participation of the homes with 
which they were working.

QIO and Nursing Home Perspectives on the Interventions:

Most QIOs we surveyed deemed conferences the most effective statewide 
intervention and on-site visits the most effective intensive 
intervention; intensive participant homes we interviewed also found 
these interventions valuable. For homes statewide, most QIOs (54 
percent) reported that conferences were their most effective 
intervention, followed by distribution of educational materials and on- 
site visits. Of the one-quarter of QIOs that reported they would change 
their statewide approach, the largest proportion (46 percent) would 
make conferences their primary intervention. Staff from several nursing 
homes we interviewed tended to concur that conferences were valuable 
aspects of the program because conferences included expert presenters, 
energized or motivated attendees, and were free.

For intensive participants, most QIOs (63 percent) deemed on-site 
visits their most effective intervention, followed by conferences and 
small group meetings. Of the 15 QIOs that said they would change their 
approach with these homes, most (60 percent) would make on-site visits 
their primary intervention, while fewer would rely on small group 
meetings, conferences, and other interventions. One QIO began 
conducting on-site visits and small group meetings when it became 
apparent that telephone calls were less productive than had been 
anticipated because of the difficulty of getting the right staff on the 
telephone at the right time, the lack of speaker phones at many homes, 
and the lack of staff engagement on the phone. Staff from a number of 
nursing homes we interviewed agreed that visits by QIO personnel were 
helpful. Some homes indicated that having someone from the QIO visit 
the home maximized the number of staff that could take advantage of the 
quality improvement training offered. Furthermore, the on-site visits 
were motivating and kept staff on track with quality improvement 
efforts. Regarding small group meetings, staff we interviewed from a 
few homes stated that meeting with staff from other homes helped 
validate their own efforts or facilitated the sharing of materials and 
experiences. Staff from one nursing home specifically reported that 
they were disappointed not to have formally participated in small group 
meetings with other facilities in the state.

Homes also found particular types of assistance less helpful. Some 
homes' staff reported that they did not feel they had the time or the 
staff necessary to complete some of the homework assignments expected 
of them, such as conducting chart reviews. Staff at some homes stated 
that the QIO provided quality improvement information with which they 
were already familiar.

QIO Staffing and Turnover Influenced Intensive Participants' 
Satisfaction with Program:

Our interviews with nursing home staff who worked intensively with the 
QIOs indicated that homes' satisfaction with the program was influenced 
by the training and experience of the primary QIO personnel who served 
as their principal contact with the QIOs, as well as by turnover among 
these individuals during the course of the 7th SOW.[Footnote 56]

When a home's principal contact with the QIO was a nurse or someone 
with long-term care or quality improvement experience, nursing home 
staff tended to report that this person possessed the knowledge and 
skills necessary to help them improve the quality of care in their 
home. Interviewees also spoke appreciatively of QIO personnel who were 
knowledgeable, motivating, and kept them on track with their efforts. 
However, when the QIO principal contact lacked these qualifications or 
characteristics, he or she was perceived as unable to effectively 
address clinical topics with staff. Staff at one home said explicitly 
that working with an experienced nurse, instead of a social worker who 
seemed to lack knowledge of long-term care, would have led to greater 
improvement in clinical quality.

The extent to which QIO primary personnel had the training or 
experience that homes considered important varied. More than half (58 
percent) of the primary QIO personnel who worked with nursing homes 
during the 7th SOW were trained in nursing, and 42 percent held an 
advanced degree. Nationwide, 27 percent of the primary personnel who 
worked with nursing homes had less than 1 year of long-term care 
experience, while 30 percent had more than 10 years of such 
experience.[Footnote 57] Just over half of primary QIO personnel (54 
percent) working with nursing homes had 4 or fewer years of quality 
improvement experience. Nine percent of QIO personnel had more than 10 
years' experience in both long-term care and quality improvement. Few 
of the personnel working with nursing homes during the 7th SOW gained 
any of their experience working for the QIO during the 6TH SOW because 
there was little overlap in personnel across the two periods.

Our interviews with intensive participants suggested that turnover 
among primary QIO personnel lowered nursing homes' satisfaction with 
the program. Our survey revealed that turnover was particularly high at 
some QIOs. At 24 QIOs, 25 percent or more of primary personnel who 
worked with nursing homes did so for less than half of the 36-month 
contract, and at 6 QIOs, the proportion was 50 percent or more. When a 
nursing home's principal contact with a QIO changed frequently, nursing 
home staff we interviewed reported that they received inconsistent 
assistance that was disruptive to their efforts to improve quality of 
care. For example, one nursing home we visited had four different 
principal contacts over the course of the 7th SOW and found this to be 
frustrating because, just as they were establishing a relationship with 
a contact, the contact would leave. Staff at another home complained 
that their interaction with QIO primary personnel turned out not to be 
the learning experience that the staff thought it would be.

Staffing levels for the nursing home component also varied among QIOs. 
As would be expected, given the wide variation in the number of nursing 
homes per state, the number of full-time-equivalent (FTE) staff working 
with nursing homes varied across the QIOs, ranging from 0.50 to 12. 
However, the ratio of QIO staff FTEs to intensive participant homes 
also showed significant variation. On average, the ratio was about 1 to 
14; but for at least 9 QIOs, the ratio of staff FTEs to homes was 1 to 
10 or fewer, and for at least 8 QIOs, the ratio was 1 to 18 or more.

QIOs' Impact on Quality Is Not Clear, but Staff at Homes We Contacted 
Attributed Some Improvements to QIOs:

Although the QIOs' impact on the quality of nursing home care cannot be 
determined from available data, staff we interviewed at most nursing 
homes attributed some improvements in the quality of resident care to 
their work with the QIOs. Nursing homes' QM scores generally improved 
enough for the QIOs to surpass by a wide margin the modest contract 
performance targets set by CMS; however, the overall impact of the QIOs 
on the quality of nursing home care cannot be determined from these 
data because of the shortcomings of the QMs as measures of nursing home 
quality and because confounding factors make it difficult to attribute 
quality improvements solely to the QIOs. Multiple long-term care 
professionals we interviewed indicated that QMs should not be used in 
isolation to measure quality improvement, but combined with other 
indicators, such as state survey data. Moreover, the effectiveness of 
the individual interventions QIOs used to assist homes also cannot be 
evaluated with the available data. CMS planned to enhance evaluation of 
the program during the 8th SOW, but a 2005 determination by HHS's 
Office of General Counsel that the QIO program regulations prohibit 
QIOs from providing to CMS the identities of the homes they are 
assisting has hampered the agency's efforts to collect the necessary 
data. Although the impact of the QIOs on the quality of nursing home 
care is not known, over two-thirds of the 32 nursing homes we 
interviewed attributed some improvements in care to their work with the 
QIOs.

All QIOs Met Modest Targets for QM Improvement, but the Impact of the 
QIOs on the Quality of Nursing Home Care Cannot Be Determined:

Although all of the QIOs met the modest targets CMS set for QM 
improvement among homes both statewide and in the intensive participant 
group, the impact of the QIOs on the quality of nursing home care 
cannot be determined because of the limitations of the QMs and because 
improvements cannot be definitively attributed to the QIOs. The 
effectiveness of the specific interventions used by the QIOs to assist 
homes also cannot be evaluated with the available data.

All QIOs Met CMS's Modest Targets for Improvement in Nursing Home QMs:

All QIOs met the CMS performance targets for the nursing home component 
of the 7th SOW. In addition to receiving an overall passing score for 
this component, nearly all QIOs surpassed expectations for each of the 
three elements that contributed to the overall score: provider 
satisfaction, improvement in QM scores among intensive participants, 
and improvement in QM scores among homes statewide. In fact, about two- 
thirds of the QIOs achieved at least five times the expected 8 percent 
improvement among intensive participants, and nearly half achieved at 
least twice the expected 8 percent improvement statewide.[Footnote 58]

CMS officials stated that the targets set for the nursing home 
component of the contract were purposely modest. Because the 7th SOW 
marked the first time all QIOs were required to work with nursing homes 
on quality improvement, and little data existed to predict how much 
improvement could be expected, CMS deliberately designed performance 
criteria to limit QIOs' chances of failing. For example, expectations 
for improvements in QM scores were set no higher for intensive 
participants than for homes statewide. In addition, CMS modified the 
evaluation plan so that if an intensive participant worked on more than 
one QM, the QM that improved least was dropped before the home's 
average improvement was calculated. CMS officials told us that, based 
in large part on QIOs' performance in the 7th SOW, the agency raised 
its expectations for the 8th SOW. For example, QIOs are required to 
work with most intensive participants on four specified QMs and to 
achieve an improvement rate of 15 to 60 percent, depending on the QM 
and the homes' baseline scores. In addition, CMS will no longer drop 
the QM that improved least when calculating homes' average 
improvement.[Footnote 59]

CMS's Use of QMs to Evaluate QIO Performance Is Problematic:

Long-term care experts we interviewed generally agreed that CMS's use 
of QMs to evaluate nursing home quality--and by extension, QIOs' 
performanceæis problematic because of unresolved issues related to the 
QMs and the MDS data used to calculate them.

QMs. As we reported in 2002, the validity of the QMs CMS proposed to 
publicly report in November 2002 was unclear.[Footnote 60] Although the 
validation study commissioned by CMS found that most of the publicly 
reported QMs were valid and reflected the quality of care delivered by 
facilities, long-term care experts have criticized the study on several 
grounds. For example, a 2005 report concluded that (1) the statistical 
criteria for the validity assessments were not stringent and (2) the 
researchers did not attempt to determine whether QMs were associated 
with quality of care at the resident level.[Footnote 61] As a result, 
it is not clear whether a resident who triggers a QM (e.g., is assessed 
as having his or her pain managed inadequately) is actually receiving 
poor care.[Footnote 62] The lack of correlation among the QMsæa home 
may score well on some QMs and poorly on othersæalso calls into 
question their validity as measures of overall quality. Since 2002, CMS 
has removed or replaced 5 of the original 10 QMsæincluding some of 
those on which the QIOs were evaluated during the 7th SOW--to address 
limitations in the QMs, such as reliability and measurement problems. 
(See app. II for a list of the QMs as of November 2002 and February 
2007).

Risk adjustment also impacts the validity of QMs. There is general 
recognition that some QMs should be adjusted to account for the 
characteristics of residents. However, there is disagreement about 
which QMs to adjust, what risk factors should be used, or how the 
adjustment should be made. For example, one expert we interviewed 
suggested that in many cases pressure ulcers start in hospitals; the 
pressure ulcer QM does not account for the origin of ulcers. Another 
expert highlighted the difficulty of making an appropriate adjustment-
-noting, for example, that improperly risk-adjusting the pressure ulcer 
QM could mask poor care that contributed to the development of ulcers.

MDS. We have also previously reported concerns about MDS 
reliabilityæthat is, the consistency with which homes conduct and code 
the assessments used to calculate the QMs.[Footnote 63] CMS awarded a 
contract for an MDS accuracy review program in 2001 but revamped the 
program in 2005, near the end of the QIOs' 7th SOW, acknowledging 
weaknesses--mainly its reliance on off-site, rather than on-site, 
accuracy and verification reviews--that we had previously 
identified.[Footnote 64] Some states that sponsor on-site MDS accuracy 
reviews continue to report troubling rates of errors in the data. For 
example, officials of Iowa's program reported an average MDS error rate 
of approximately 24 percent in 2005.

Our interviews with long-term care experts and nursing home staff 
suggested that the chronic pain QMæwhich was selected as a focus of 
quality improvement work by many QIOs and intensive participant nursing 
homesæmay be particularly vulnerable to error in the underlying MDS 
data. Possible sources of error are systematic differences in the 
extent to which facilities identify and assess residents in pain and 
misunderstandings about how to accurately code MDS questions specific 
to pain. For example, staff from two nursing homes told us that their 
pain management QM scores improved after staff realized that they had 
been mistakenly coding residents as having pain even though their pain 
was successfully managed. Moreover, experts we interviewed noted that 
higher-quality homes may have worse pain QM scores because they do a 
better job of identifying and reporting pain in residents.

The use of MDS data to measure the quality of care in nursing homes is 
also problematic because the MDS was not designed as a quality 
measurement tool and does not reflect advances in clinical practice. 
CMS is updating the MDS now to address these limitations. For example, 
instead of asking homes to classify the severity of a pressure ulcer on 
the basis of a four-stage system, the draft MDS now under review 
includes a measurement tool intended to more accurately classify the 
severity of a pressure ulcer.[Footnote 65] In addition, facilities are 
asked to indicate whether the pressure ulcer developed at the facility 
or during a hospitalization. CMS does not yet have an official release 
date for the revised MDS but anticipates that all validation and 
reliability testing will be completed by December 2007.

Other Measures of Quality. Multiple long-term care professionals we 
interviewed, including stakeholders and experts on quality measurement, 
recommended both that the QMs undergo continued refinement and that 
they not be used in isolation to assess the quality of care in nursing 
homes. They suggested a number of other sources of information as 
alternatives or complements to QMs for measuring quality. For example, 
a representative of the National Quality Forum (NQF), a group with 
which CMS contracted to provide recommendations on quality measures for 
public reporting, stated that experts do not consider the QMs 
sufficient in themselves to rate homes and that the other quality 
markersæsuch as perceptions of care by family members, residents, and 
staff; state survey data; and resident complaintsæalso provide useful 
information about quality of care. Other long-term care professionals 
we interviewed suggested these and other measures, including nursing 
home staffing levels and staff turnover and retention rates.

Influence of Other Factors on Nursing Home Quality Makes It Difficult 
to Evaluate QIOs' Impact:

Factors such as the existence of other quality improvement efforts make 
it difficult to evaluate QIOs' work with nursing homes and attribute 
quality improvement solely to QIOs. In an assessment of the QIO program 
during the 7th SOW, CMS and QIO officials acknowledged this difficulty. 
The assessment found that intensive participants improved more than 
nonintensive participants on all five QMs studied, and for each QM, 
intensive participants that worked on the QM improved more than 
intensive participants that did not.[Footnote 66] However, the authors 
noted that these results could not be definitively attributed to the 
efforts of the QIOs because improvements may have been influenced by a 
variety of factors, including preexisting differences between intensive 
participants and nonintensive participants;[Footnote 67] public 
reporting of the QMs, which may have focused homes' attention on 
improving these measures; and other quality improvement efforts to 
which homes may have been exposed. As noted earlier in this report, 
these other efforts included, but were not necessarily limited to, 
initiatives sponsored by state governments, nursing home trade 
associations, and CMS. While these other efforts varied considerably in 
the intensity of technical assistance offeredæranging from a trade 
association-sponsored program that homes characterized as essentially 
signing a quality improvement pledge, to state-sponsored programs that 
involved on-site visits by experienced long-term care nurses who 
provided best-practice guidelines, educational materials, and clinical 
toolsæthe fact that the efforts were present made it impossible to 
attribute quality improvements solely to the QIOs.

In its 2006 report on all aspects of the QIO program, IOM highlighted 
similar shortcomings in previous studies of the QIO program and called 
for more systematic and rigorous evaluations. IOM concluded that 
although the QIOs may have contributed to improvements in the quality 
of care, the existing evidence was inadequate to determine the extent 
of their contribution. In its response to the IOM study, CMS 
acknowledged the need to strengthen its methods of evaluating the 
program and outlined plans to convene an evaluation expert advisory 
panel to make recommendations on the framework for the next contracts 
(the 9TH SOW, which will begin in 2008). CMS also stated that it will 
collect information during the 8th SOW that will allow it to control 
for differences in motivation between intensive and nonintensive 
participants but did not specify the nature of this 
information.[Footnote 68] Subsequently, HHS's Office of General Counsel 
determined that QIO program regulations prohibited QIOs from providing 
to CMS the identities of intensive participants.[Footnote 69] CMS 
officials acknowledged that this prohibition posed a considerable 
challenge to their evaluation plans and said that as a short-term 
solution the agency might contract with one of the QIOs to evaluate the 
program, with the possible stipulation that the findings be verified by 
an independent auditor.

CMS Data Are Too Limited to Evaluate Effectiveness of Specific QIO 
Interventions:

CMS collected little information about the specific interventions QIOs 
used to assist nursing homes and acknowledged that the information it 
did have was not sufficiently comprehensive or consistent to be used to 
evaluate the interventions' effectiveness. In general, CMS's oversight 
of QIOs' work on the nursing home component consisted of ensuring that 
the QIOs produced the reports and deliverables specified in the 
contracts and appeared on track to meet performance targets.

CMS's primary source of data about QIOs' interventions was the monthly 
activity reports the QIOs were required to submit through the Program 
Activity Reporting Tool (PARTner). In these reports, QIOs were to 
document the specific interventions they provided to each home, using 
such activity codes as "on-site support" and "stand-alone workshops on 
quality improvement." However, with only seven activity codes for QIOs 
to choose from, the level of detail in these reports was low. For 
example, the same code would be used for a full-day visit as for an 
hour visit. Moreover, because QIOs were not expected to enter any code 
more than once per month for a home, a code for on-site support could 
indicate a single visit or multiple visits. The system also captured no 
information about the content of visits or other interventions. From 
the perspective of the QIOs, the system was of limited use: More than 
half of the 52 QIOs surveyed by IOM rated PARTner fair or poor in terms 
of both value and ease of use. Staff at one QIO we interviewed reported 
using tracking systems they developed themselves, rather than PARTner, 
to monitor their work.

CMS regional offices and the nursing home satisfaction survey gathered 
some additional information about the interventions used by QIOs. The 
CMS regional offices gathered information through telephone calls and 
visits to the QIOs and by participating in quarterly conference calls 
during which QIOs and CMS regional and central offices discussed issues 
related to the nursing home component of the contract. The regional 
office staff also reviewed information entered into the PARTner data 
system by QIOs, but they focused their evaluations on QIO contract 
compliance and not on the effectiveness of specific interventions 
because--as some regional staff emphasized--the contracts were 
performance-based, and therefore it was not their place to 
"micromanage" the QIOs or to advocate for or against specific 
interventions. Feedback from nursing homes was gathered through the 
nursing home satisfaction survey, conducted after the midpoint of the 
contract cycle by a contractor for CMS.[Footnote 70] The survey 
collected information about the frequency of, and homes' satisfaction 
with, a range of interventions, including on-site visits, training 
workshops, one-on-one telephone calls, conference calls, one-to-one e- 
mails, and broadcast e-mails. However, the survey collected no 
information about the content of these interventions or the aspects 
that contributed to providers' satisfaction or dissatisfaction.

In its 2006 report on the QIO program, IOM emphasized the need for CMS 
to gather more information about specific interventions and noted that 
CMS was uniquely positioned to determine which interventions lead to 
high levels of quality improvement. The agency responded that it will 
collect information during the 8th SOW to better explore the 
relationship between the intensity of assistance provided by the QIO 
and the level of improvement, but did not specify the type of 
information it will collect. As of March 2007, CMS had not yet 
implemented a revamped PARTner system. In addition, the agency 
cancelled its plans to conduct an initial survey of nursing homes early 
in the contract period and now plans to conduct only one, later in the 
contract period. CMS officials explained that the delay and 
cancellation were due at least in part to the determination by HHS's 
Office of General Counsel that QIOs could not provide to CMS the 
identities of intensive participants to CMS.

Homes That Received Intensive Assistance Generally Attributed Some 
Improvements in Quality of Care to Work with QIOs:

Although the impact of the QIOs on the overall quality of nursing home 
care cannot be determined, staff we interviewed at over two-thirds of 
the 32 nursing homes stated that they improved the care delivered to 
residents as a result of working intensively with the QIOs. Staff at 23 
of the 32 homes told us that they implemented new, or made changes to 
existing, policies and procedures related to pain or pressure ulcers. 
Of the 23 nursing homes, staff from 21 stated that they changed the way 
they addressed resident pain. In general, these changes involved 
implementing pain scales or new assessment forms. Staff at some 
facilities noted that working with the QIO heightened staff awareness 
of resident pain, including awareness of cultural differences in the 
expression of pain. Staff at 8 of the 23 nursing homes stated that they 
changed the way they addressed pressure ulcers. In general, these 8 
homes implemented new assessment tools, changed assessment plans, or 
revised facility policies using materials provided by the QIO. (Table 3 
provides examples of resident care improvements related to pain 
assessment and treatment and pressure ulcers.) Staff at 13 of the 32 
nursing homes stated that the changes they made as a result of working 
with the QIOs were sustainable, but staff from some nursing homes noted 
that staffing turnover at their facilities could affect sustainability.

Table 3: Examples of Resident Care Improvements Made by Homes as a 
Result of Intensive Assistance Provided by QIOs, 7th SOW:

Care area: Pain; Example: Had nurses evaluate acute pain management at 
end of each shift with nurse aide involvement.

Care area: Pain; Example: Used interventions other than medications, 
such as massage, compresses, and repositioning.

Care area: Pain; Example: Recorded signs of pain when providing care 
for wounds such as pressure ulcers.

Care area: Pain; Example: Began using or resumed using pain scales to 
assess resident pain.

Care area: Pain; Example: Implemented pain policy that addresses both 
cognitively intact residents and residents who have dementia or are 
nonverbal.

Care area: Pressure ulcers; Example: Increased skin assessments to four 
times a week and had nurse aides document changes on a daily basis.

Care area: Pressure ulcers; Example: Established a wound care team.

Care area: Pressure ulcers; Example: Used a tracking tool to measure 
depth and width of pressure ulcers.

Care area: Pressure ulcers; Example: Conducted skin checks when a 
resident returned to the facility, such as after a hospitalization.

Source: GAO interviews with staff from nursing homes assisted 
intensively by the QIOs.

[End of table]

Of the 32 nursing homes we contacted, staff from 4 specifically stated 
that working with the QIO did not change their quality of care. For 
example, staff from one home stated that the QIO did not offer their 
facility any new or helpful information and did not offer feedback on 
how the facility's processes could improve. Staff from another home 
reported that the information provided by the QIO was on techniques 
their facility had already implemented. Staff at a third home noted 
that while the QIO was a good resource, the home could have done as 
much on its own, without assistance from the QIO. Staff at three 
facilities, none of which reported making any policy or procedural 
changes, said the facilities experienced worse survey results while 
working with their QIO; staff from two of the three reported being 
cited for quality deficiencies in the specific areas they had been 
addressing with the QIO. Staff at one of these facilities believed they 
were cited because their work with the QIO had made the surveyor more 
aware of the facility's problems in this area.

Conclusions:

Although it is difficult to evaluate the impact of QIO assistance, the 
QIO program does have the potential to help improve the quality of 
nursing home care. CMS program improvements for the 8th SOW, such as 
the agency's decision to focus resources on intensive rather than 
statewide assistance and its plans to improve evaluation, are positive 
steps that could result in more effective use of available funds and 
provide more insight into the program's impact. Our evaluation of 
assistance provided during the 7th SOW, however, raised two major 
questions about the future focus, oversight, and evaluation of the QIO 
program, which we address below.

Given the available resources, which homes and quality-of-care areas 
should CMS direct QIOs to target for intensive assistance? We found 
that QIOs generally did not target intensive assistance to homes that 
performed poorly in state surveys, partly because of concerns about the 
willingness and ability of such homes to simultaneously focus on 
quality improvement and cooperate with the QIOs. However, the 
Collaborative Focus Facility project during the 7th SOW demonstrated 
that low-performing homes could improve their survey results and QM 
scores; subsequently, CMS required that during the 8th SOW each QIO 
work with up to three such homes--about 10 percent of the total number 
that QIOs are expected to assist intensively. Stakeholders we 
interviewed believed that even more emphasis should be placed on 
assisting low-performing homes. We found that there was little overlap 
between homes that participated in the QIO Collaborative Focus Facility 
project and in CMS's Special Focus Facility program, which is a program 
involving about 130 nursing homes nationwide that, on the basis of 
their survey results, receive increased scrutiny and enforcement by 
state survey agencies. The limited overlap suggests that each state has 
more than three low-performing facilities that could benefit from QIO 
assistance.

Targeting assistance to low-performing homes could pose challenges 
given the voluntary nature of the program--homes must agree to work 
with a QIO. QIOs maintain that voluntary participation is critical to 
ensuring homes' commitment to the program. However, the risk in this 
approach is that some of the homes that need help most will not get it. 
Indeed, in the Collaborative Focus Facility project, some of the low- 
performing homes that were asked to participate refused QIO assistance. 
In addition, QIOs expended more resources working to improve these low- 
performing homes than were required to assist better-performing homes. 
Thus, increasing the number of low-performing homes QIOs are required 
to assist above the small number mandated for the 8th SOW might 
necessitate decreasing the total number of homes assisted. However, 
existing resources might be maximized if QIOs worked with each home 
only on the quality-of-care areas that pose particular challenges for 
that home.

Could interim steps be taken to improve oversight and evaluation of 
QIOs' work with nursing homes before the contracting cycle that begins 
in August 2008? Currently, CMS collects data primarily on QIO outcomes-
-specifically, changes in QM scores--and costs. CMS needs more detailed 
data, particularly about the type and intensity of interventions used 
to assist nursing homes, to improve its oversight and evaluation of the 
QIO program. Without such data, CMS cannot hold QIOs fully accountable 
for their performance under their contract with CMS. Some evaluation 
activities are now scaled back or on hold because HHS determined early 
in the 8th SOW that program regulations prohibited the QIOs from 
providing to CMS the identities of the intensive participants. Such a 
firewall presents a major impediment to improved oversight and 
evaluation of the QIO program and prevented CMS from implementing 
interim changes it planned to make. For example, for the 7th SOW, CMS 
contracted for one nursing home satisfaction survey to be conducted 
near the end of the contract period--too late to be of use in interim 
monitoring of the QIOs' performance. For the 8th SOW, CMS had planned 
to contract for two surveys but was forced to cancel the one planned 
for early in the contract period because it was unable to provide the 
names of intensive participants to its survey contractor. Moreover, the 
lack of these data would preclude CMS from independently verifying QIO 
compliance with such contract requirements as the geographic dispersion 
of intensive participants in each state.

CMS evaluated QIOs' work with nursing homes primarily on the basis of 
changes in QM scores; given the weaknesses of QM data, the current 
reliance on these data appears unwarranted. While CMS actions to 
improve the MDS instrument as a quality measurement tool are important, 
the agency has not yet established an implementation date. Although 
multiple long-term care professionals believe that multiple indicators 
of quality, including deficiencies on homes' standard and complaint 
surveys and residents' and family members' satisfaction with care, 
should be used to measure quality improvement, CMS is not currently 
drawing on these data sources to evaluate QIOs' efforts. Recognized 
shortcomings in these other data sources--such as the understatement of 
survey deficiencies by state surveyors--underscore the importance of 
using multiple data sources to evaluate QIO outcomes.

Recommendations for Executive Action:

To ensure that available resources are better targeted to the nursing 
homes and quality-of-care areas most in need of improvement, we 
recommend that the Administrator of CMS take the following two actions:

* Further increase the number of low-performing homes that QIOs assist 
intensively.

* Direct QIOs to focus intensive assistance on those quality-of-care 
areas on which homes most need improvement.

To improve monitoring of QIO assistance to nursing homes and to 
overcome limitations of the QMs as an evaluation tool, we recommend 
that the Administrator of CMS take the following three actions:

* Revise the QIO program regulations to require QIOs to provide to CMS 
the identities of the nursing homes they are assisting in order to 
facilitate evaluation.

* Collect more complete and detailed data on the interventions QIOs are 
using to assist homes.

* Identify a broader spectrum of measures than QMs to evaluate changes 
in nursing home quality.

Agency Comments and Our Evaluation:

We obtained written comments from CMS on our draft report. CMS 
addressed three of our five recommendations. It concurred with two of 
the three recommendations but did not specify how it would implement 
them, and it continues to explore options for implementing the third 
recommendation. Our evaluation of CMS's comments follows the order we 
presented each recommendation in the report. CMS's comments are 
included in app. III.

Further increase the number of low-performing homes that QIOs assist 
intensively. CMS agreed with this recommendation but did not specify a 
time frame for addressing it or indicate how many low-performing homes 
it will expect QIOs to assist in the future. Although our report 
focused on the most recently completed contract period (the 7th SOW), 
we acknowledged that in the current contract period, CMS required QIOs 
to provide intensive assistance to some "persistently poor-performing" 
homes identified in consultation with each state survey agency. 
However, we pointed out that the number of these homes the QIOs were 
required to serve was small, accounting for less than 10 percent of the 
homes they were expected to assist intensively. CMS commented that 
preliminary estimates from a special study conducted during the 7th SOW 
indicated that assisting chronically poor-performing homes cost the 
QIOs 5 to 10 times as much as assisting the "usual" home.[Footnote 71] 
Our report acknowledged that additional resources were required for 
QIOs to assist low-performing homes but suggested that CMS could 
decrease the total number of homes assisted in order to increase the 
number of low-performing homes beyond the small number mandated for the 
8th SOW.

Direct QIOs to focus intensive assistance on those quality-of-care 
areas on which homes most need improvement. CMS did not directly 
respond to this recommendation, but did point out that about one-third 
of QIOs were working primarily with homes on QMs on which the homes 
scored worse than the national average during the 8th SOW. Our 
recommendation was to direct all QIOs to focus intensive assistance on 
QMs that reflect homes' greatest quality-of-care challenges. We had 
reported that some nursing homes assisted intensively by QIOs did not 
have a choice of QMs on which to work. We concluded that having QIOs 
work intensively with homes only on the quality-of-care issues that 
posed particular challenges to them would maximize program resources.

Revise QIO program regulations to require QIOs to provide CMS with the 
identities of the homes assisted in order to facilitate evaluation. CMS 
did not specifically indicate whether it agreed with this 
recommendation, but did indicate that it continues to explore options 
which would allow access to data on the homes assisted intensively in 
order to facilitate evaluation. However, CMS expressed concern that 
providing this access could potentially subject the information to laws 
that could afford third parties similar access. We believe that CMS 
should continue to evaluate how best to maintain an appropriate balance 
between disclosure and confidentiality. If CMS's evaluation indicates 
that it is unable to incorporate adequate confidentiality safeguards to 
promote voluntary participation in QIOs' quality improvement 
initiatives, the agency could seek legislation that would provide such 
safeguards.

Collect more complete and detailed data on the interventions QIOs use 
to assist homes. CMS responded to this recommendation, although it 
labeled it "improve the monitoring of QIO activities," and agreed with 
our recommendation. CMS noted that, in concert with HHS, it is 
reviewing recommendations from the IOM's 2006 report on QIOs, which may 
result in redesigning the program, including systems for evaluating QIO 
activities in different care settings, such as nursing homes. CMS did 
not discuss how it planned to collect additional data on QIO nursing 
home interventions. Further, it stated that it may incorporate data- 
handling and -reporting features of the nursing home subtask into 
overall program improvements. We have reservations about this plan 
because we found that CMS collected little information about specific 
QIO interventions with nursing homes during the 7th SOW, the 
information collected was not sufficiently comprehensive or consistent 
to be used to evaluate the interventions' effectiveness, and QIOs 
themselves reported that the data collection system was of limited use 
to them.

Identify a broader spectrum of measures than QMs to evaluate changes in 
nursing home quality. CMS did not directly address this recommendation. 
However, the agency took issue with our judgment that the use of QMs to 
evaluate nursing home quality--and by extension, QIOs' performance--is 
problematic. CMS commented that the QMs have passed through rigorous 
development, testing, deployment, and national consensus processes. We 
reported that the study commissioned by CMS to validate the QMs has 
been criticized by experts on several grounds, including a lack of 
statistical rigor. We also noted that CMS has revised or is currently 
revising both the QMs and the MDS data used to calculate them to 
address limitations, such as reliability and measurement problems. For 
example, CMS has removed or replaced 5 of the original 10 QMs since 
2002, including some of those on which the QIOs were evaluated during 
the 7th SOW. In addition, CMS is currently updating the MDS to reflect 
advances in clinical practice and to improve its utility as a quality 
measure tool. While we expect that these efforts will improve the QMs 
as measures of nursing home quality, we believe that the QMs' current 
limitations argue for the use of a broader spectrum of measures to 
evaluate changes in nursing home quality. Multiple long-term care 
professionals we interviewed recommended that the QMs not be used in 
isolation to assess the quality of care in nursing homes; these 
professionals suggested a range of measures that could be used to 
supplement the QMs, including perceptions of care by family members, 
residents, and staff; state survey data; and nursing home staffing 
levels.

As arranged with your office, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 30 days 
after its issue date. At that time, we will send copies to the 
Administrator of the Centers for Medicare & Medicaid Services and 
appropriate congressional committees. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please 
contact me at (202) 512-7118 or allenk@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made major contributions to 
this report are listed in appendix IV.

Sincerely yours,

Signed by:

Kathryn G. Allen: 
Director, Health Care:

[End of section]

Appendix I: Scope and Methodology:

Our analysis of QIOs' work with nursing homes had three major 
components: (1) site visits to five QIOs, (2) analysis of state survey 
data to compare homes that were assisted intensively with homes that 
were not, and (3) a Web-based survey of 51 QIOs.

Site Visits:

We visited a QIO in each of five states to gather detailed information 
about QIOs' work with nursing homes from the perspective of the QIOs, 
nursing homes in the intensive participant group, and stakeholders; we 
used this information to address all three objectives.[Footnote 72] We 
selected the states-æand by extension, the QIOs that worked in those 
states--on the basis of six criteria described in the following 
section. After selecting the QIOs, we identified nursing homes that 
received intensive assistance and stakeholders to contact for 
interviews. We conducted most of our site visit interviews in March and 
April 2006.

Selection of QIOs:

We based our selection of QIOs on the following criteria:

* Number of nursing home beds in the state. We divided the states into 
three groups of 17 states each based on the number of nursing home beds 
at the beginning of the 7th SOW (2002). We over-sampled states with 
high numbers of nursing home beds by selecting one state with a low 
number of beds, one state with a medium number, and three states with a 
high number.

* Evaluation score for the nursing home component of the 7th SOW 
relative to scores of other QIOs. We divided the states into three 
groups of 17 based on the QIOs' evaluation scores for the 7th SOW. To 
help us identify the possible determinants of scores, we selected more 
states at each end of the spectrum than in the middle: two states with 
scores in the bottom third, one state with a score in the middle third, 
and two states with scores in the top third.

* State survey performance of homes selected for intensive assistance 
relative to homes not selected. We also considered the extent to which 
the homes selected for intensive assistance by a given QIO at the 
beginning of the 7th SOW differed from the homes that were not 
selected, in terms of serious deficiencies cited on state surveys (both 
the proportion of homes cited in each group and the average number of 
serious deficiencies per home). We chose one QIO that selected worse 
homes, three QIOs that selected homes that were neither better nor 
worse, and one QIO that selected better homes.

* Presence of a state-sponsored nursing home quality improvement 
program. At the time we selected QIOs for site visits, we were aware of 
four states that had state-sponsored quality improvement initiatives in 
place during the 7th SOW.[Footnote 73] To learn more about these 
efforts and how they interacted with and compared with efforts by the 
QIOs, we included one state (Florida) with its own quality improvement 
initiative.[Footnote 74] After we made our selection, we learned that 
another state we had selected (Maine) had a state-sponsored quality 
improvement program.

* QIO participation in the Collaborative Focus Facilities project. CMS 
has funded QIOs to conduct several special studies with nursing homes, 
including one in which the 17 participating QIOs each worked 
intensively with up to five nursing homes identified by their state 
survey agencies as having significant quality problems. To learn more 
about the challenges involved in working with low-performing homes, we 
selected two states whose QIOs participated in this project.

* Census region. We selected states from four different regions of the 
country: Northeast, Midwest, South, and West.

Using these criteria, we selected the following five states: Colorado, 
Florida, Iowa, Maine, and New York. Together these states represented 
15 percent of nursing home beds nationwide at the beginning of the 7th 
SOW (2002).

Selection of Nursing Homes:

Overall, we interviewed staff from 32 nursing homes in nine states. To 
assist in the development of our site visit protocols, we interviewed 
staff from 4 homes in four states. During the site visits to five 
states, we interviewed staff from 28 nursing homes. In each state, we 
interviewed staff from 4 to 8 nursing homes that received intensive 
assistance from the QIO, for a total of 28 homes in these five states. 
The number of homes we selected in each of the five states visited 
varied depending on the number of homes the QIO was expected to select 
for intensive assistance, an expectation based on the number of homes 
in the state. Specifically, we selected either four homes or 7 percent 
of the maximum number of homes that each of the five QIOs was expected 
to assist intensively, whichever was greater.[Footnote 75]

We chose homes on the basis of four characteristics: number of serious 
deficiencies in the standard state survey at the beginning of the 7th 
SOW (2002), improvement in QM scores during the 7th SOW, distance from 
the QIO (in order to include homes that were more difficult for QIOs to 
visit), and urban versus rural location. Specifically, we sought to 
include (1) at least one home that had one or more serious deficiencies 
and that finished in the top third of the intensively assisted homes in 
their state in terms of improvement on QM scores, and (2) at least one 
home that had one or more serious deficiencies and that finished in the 
bottom third of the intensively assisted homes in their state in terms 
of improvement on QM scores. For the remaining homes, we sought a group 
whose state survey deficiency levels and QM improvement scores were 
representative of the range among intensive participants in their 
state. However, the experiences of this sample of 32 homes cannot be 
generalized to the entire group of homes that received intensive 
assistance from the QIOs nationwide.

Selection of Stakeholders:

In each state we also interviewed officials from three stakeholder 
groups: (1) the state survey agency; (2) the local affiliate of the 
American Health Care Association, which generally represents for-profit 
homes; and (3) the local affiliate of the American Association of Homes 
and Services for the Aging, which represents not-for-profit homes.

Analysis of State Survey Data:

To assess the characteristics of the nursing homes that were selected 
by the QIOs for intensive assistance from among the homes that 
volunteered, we analyzed 3 years of standard state survey data on 
deficiencies cited at nursing homes and compared the results for homes 
that were assisted intensively with results for homes that were not; we 
used this information to address our first objective.[Footnote 76] The 
analysis involved three steps:

1. identifying nursing homes that had three standard state surveys from 
1999 through 2002;

2. ranking nursing homes in each state in each year, based on the 
number of serious and other deficiencies, and then classifying homes as 
consistently low-, moderately, or high-performing; and:

3. identifying on a nationwide and state-by-state basis any 
statistically significant differences between homes selected and not 
selected by the QIO, in terms of the proportion of low-, moderately, or 
high-performing nursing homes.

Identifying Homes with Three Standard Surveys:

To identify homes whose performance was consistently lower or higher 
than other homes in their state prior to the selection of homes by the 
QIOs, we included in our analysis only homes for which we were able to 
identify three standard surveys from January 1, 1999, through November 
1, 2002. Using the state survey calendar year summary files for 1999 
through 2002 for the 50 states and the District of Columbia, we 
obtained 3 years of deficiency data from standard surveys for 16,303 
homes.[Footnote 77]

Classifying Homes as Low-, Moderately, or High-Performing:

CMS classifies deficiencies according to their scope and severity. For 
each of the three surveys, we ranked all of the nursing homes in each 
state based on the number of deficiencies in two categories: (1) actual 
harm or immediate jeopardy and (2) potential for more than minimal 
harm.[Footnote 78] Deficiencies in the first category are considered 
serious deficiencies. We gave more weight to the serious deficiencies 
by sorting the homes first on the number of deficiencies in the first 
category and then on the number of deficiencies in the second category. 
Homes with the same number of deficiencies in each category were 
assigned the same rank. Based on these rankings, we identified homes in 
the bottom and top quartile in each state in each survey.[Footnote 79]

We classified homes as low-performing if they ranked in the bottom 
quartile in the most recent of the three surveys and in at least one of 
the two preceding surveys. We classified homes as high-performing if 
they ranked in the top quartile in the most recent of the three surveys 
and in at least one of the two preceding surveys. We classified homes 
as moderately performing if they did not meet the criteria for 
inclusion in either the low-or high-performing group. Of the 16,303 
homes with three standard state surveys during the period we specified, 
we classified 15 percent as low-performing, 65 percent as moderately 
performing, and 20 percent as high-performing.

To assess the stability of our categorization of homes as low-(or high- 
) performing, we ran a logistic regression model to predict the 
probability of a home being categorized as low-(or high-) performing in 
the most recent of the three surveys given its categorization in the 
two prior surveys. The regression results showed that homes that were 
categorized as low-(or high-) performing in one survey were 
significantly more likely to be categorized as low-(or high-) 
performing in the other surveys as well.

Determining Statistically Significant Differences between Homes 
Assisted Intensively and Homes Not Assisted Intensively by the QIOs:

Our final step was to determine, on both a nationwide and state-by- 
state basis, whether there was a statistically significant difference 
in the proportion of (1) low-performing homes, (2) moderately 
performing homes, and (3) high-performing homes in the group assisted 
intensively by the QIOs compared with the group not assisted 
intensively.[Footnote 80]

Web-Based Survey of QIOs:

To gather information about the characteristics of the QIOs, including 
their process for selecting homes for intensive assistance from the 
pool of volunteers and the interventions they used, on July 19, 2006, 
we launched a two-part Web-based survey of QIOs in all 50 states and 
the District of Columbia; we used this information to address 
objectives one and two.[Footnote 81] We achieved a 100 percent response 
rate. The first part of the survey gathered information about the 
primary personnel who worked with nursing homes during the 7th SOW, 
including information about their employment with the QIO, and their 
relevant credentials and experience.[Footnote 82] The second part of 
the survey gathered information on a range of other topics, including 
information about stakeholder involvement with the QIO, recruitment and 
selection of nursing homes for intensive assistance, interventions used 
with intensive participants, interventions used with homes statewide, 
and QIOs' communication with CMS. We specifically inquired about QIOs' 
use of six interventions: (1) mailings, faxes, and e-mails; (2) 
conferences; (3) small group meetings; (4) conference calls and video 
or Web conferences with multiple homes; (5) telephone conversations 
with individual homes; and (6) on-site visits.[Footnote 83] We asked 
QIOs to rank and provide information on the two interventions they 
relied on most to assist homes statewide and on the three interventions 
they relied on most to assist homes in the intensive participant 
group.[Footnote 84] We also asked QIOs to rank the effectiveness of the 
interventions they used and to identify the interventions they would 
use if they could do the 7th SOW over again.

[End of section]

Appendix II: Publicly Reported Quality Measures:

In November 2002, CMS began a national Nursing Home Quality Initiative 
that included the development of QMs that would be publicly reported on 
the CMS Web site called Nursing Home Compare. CMS has continued to 
refine the QMs and, as shown in table 4, has dropped some QMs and added 
others.

Table 4: QMs as of November 2002 and as of February 2007:

QM: Chronic care QM; QM as of November 2002: [Empty]; QM as of February 
2007: [Empty].

QM: Chronic care QM; Decline in activities of daily living; QM as of 
November 2002: Yes; QM as of February 2007: Yes.

QM: Pressure ulcers; QM as of November 2002: Yes; QM as of February 
2007: [Empty].

QM: Pressure ulcers[A]; QM as of November 2002: Yes; QM as of February 
2007: [Empty].

QM: Pressure ulcers in high-risk residents; QM as of November 2002: 
[Empty]; QM as of February 2007: Yes.

QM: Pressure ulcers in low-risk residents; QM as of November 2002: 
[Empty]; QM as of February 2007: Yes.

QM: Inadequate pain management; QM as of November 2002: Yes; QM as of 
February 2007: Yes.

QM: Physical restraints; QM as of November 2002: Yes; QM as of February 
2007: Yes.

QM: Infections; QM as of November 2002: Yes; QM as of February 2007: 
[Empty].

QM: Weight loss; QM as of November 2002: [Empty]; QM as of February 
2007: Yes.

QM: Urinary tract infection; QM as of November 2002: [Empty]; QM as of 
February 2007: Yes.

QM: Catheter insertion; QM as of November 2002: [Empty]; QM as of 
February 2007: Yes.

QM: Depression; QM as of November 2002: [Empty]; QM as of February 
2007: Yes.

QM: Bowel or bladder control in low-risk residents; QM as of November 
2002: [Empty]; QM as of February 2007: Yes.

QM: Bedfast; QM as of November 2002: [Empty]; QM as of February 2007: 
Yes.

QM: Worsening ability to move about room; QM as of November 2002: 
[Empty]; QM as of February 2007: Yes.

QM: Administration of influenza vaccination during flu season; QM as of 
November 2002: [Empty]; QM as of February 2007: Yes.

QM: Assessment for and administration of pneumococcal vaccination; QM 
as of November 2002: [Empty]; QM as of February 2007: Yes.

QM: Post-acute-care QM; QM as of November 2002: [Empty]; QM as of 
February 2007: [Empty].

QM: Failure to improve and manage delirium; QM as of November 2002: 
Yes; QM as of February 2007: Yes.

QM: Failure to improve and manage delirium (facility-adjusted)[A]; QM 
as of November 2002: Yes; QM as of February 2007: [Empty].

QM: Inadequate pain management; QM as of November 2002: Yes; QM as of 
February 2007: Yes.

QM: Improvement in walking; QM as of November 2002: Yes; QM as of 
February 2007: [Empty].

QM: Pressure ulcers; QM as of November 2002: [Empty]; QM as of February 
2007: Yes.

QM: Administration of influenza vaccination during flu season; QM as of 
November 2002: [Empty]; QM as of February 2007: Yes.

QM: Assessment for and administration of pneumococcal vaccination; QM 
as of November 2002: [Empty]; QM as of February 2007: Yes.

Source: CMS.

[A] Facility-level risk adjustment was intended to take into account 
the fact that some homes may admit frailer, sicker residents, or may 
specialize in a particular area of care that may account for a larger 
proportion of residents for a particular measure. CMS reported the 
delirium measure both with and without facility adjustment.

[End of table]

[End of section]

Appendix III: Comments from the Centers for Medicare & Medicaid 
Services:

[See PDF for image]

[End of figure]

Department Of Health & Human Services: Centers for Medicare & Medicaid 
Services: 

Administrator: 
Washington, DC 20201: 

Date: May 11, 2007: 

To: Kathryn G. Allen: 
Director, Health Care: 
Government Accountability Office: 

From: Leslie V. Norwalk, Esq. 
Acting Administrator: 

Subject: Government Accountability Office's (GAO) Draft Report: 
"Federal Actions Needed to Improve Targeting and Evaluation of 
Assistance By Quality Improvement Organizations" (GAO-07-373):

Background:

The Centers for Medicare & Medicaid Services (CMS) launched the 
National Nursing Home Quality Initiative (NHQI) in November 2002, 
marking the beginning of the first organized work with the Nation's 
nursing homes on national quality improvement activities outside of the 
Survey & Certification process. This coincided with the beginning of 
the Quality Improvement Organizations' (QIOs) 7th Statement of Work 
(SOW) contract cycle and marked the first time the QIOs were tasked 
with "core" work in the nursing home setting in all 53 contract regions.

Since, up until this time, the entire CMS interface for nursing home 
quality was through the Survey & Certification program, Senator 
Grassley requested that the GAO collect data about the nursing home 
task in the QIO contract ("Task 1A"). Specifically, Senator Grassley 
requested that the GAO collect information about the nursing homes 
which worked with the QIOs and the quality improvement outcome measures 
associated with this work, especially in the 7th SOW (2002-2005).

The GAO conducted the study over an 18 month period from October 2005 
to April 2007. During the course of the study, the GAO conducted a web- 
based survey of 51 QIOs, conducted on-site reviews of QIOs and nursing 
homes in five States (CO, FL, IA, ME, and NY), and interviewed experts 
on "using quality measures to evaluate QIOs." The three initial 
research questions were as follows:

1. What assistance are QIOs providing to nursing homes to help them 
improve resident care?
2. Have nursing homes that have worked with QIOs improved their quality 
of care?
3. How effective is CMS oversight of QIO assistance to nursing homes?

Page 2-Kathryn G. Allen:

GAO Recommendations:

The GAO made the following three recommendations for future QIO work in 
the nursing home setting:

1. The CMS should further increase the number of low-performing homes 
that QIOs work with intensively.
2. The CMS should improve the monitoring and evaluation of QIO 
activities.
3. The CMS should require QIOs to share with CMS the identity of homes 
assisted intensively in order to facilitate evaluation.

CMS Response: Executive Summary:

Recommendation 1: CMS agrees with the recommendation and points out 
additional facts about the nursing home work of the QIO program. CMS 
has already identified this as an issue from its own review, and has 
taken steps to address it. For example, in the 8th SOW, 34 percent of 
QIOs are working with an average of 56 percent of the nursing homes in 
their states. However, there are significant cost implications for this 
recommendation. A study conducted during the 7th SOW showed that 
chronically poor performing nursing homes cost between five and tenfold 
more to work with than nursing homes that are closer to average 
performance.

Recommendation 2: CMS agrees with the recommendation, and has already 
made progress toward re-designing the management of the Q10 program.

Recommendation 3: CMS continues to explore options which would allow 
access to this data for evaluation of the QIOs individually and the 
program as a whole, while maintaining appropriate safeguards necessary 
to promote voluntary participation in quality improvement initiatives. 
CMS will continue to seek a balance between disclosure and 
confidentiality.

CMS Response to GAO Recommendations:

1. The CMS should further increase the number of low-performing homes 
that QI0s work with intensively.

The CMS agrees with this recommendation. However, CMS notes that the 
draft report does not draw enough attention to two important aspects of 
this recommendation. 'the first is that CMS, in its own review of the 
7'H SOW, identified this need and has already taken steps to address 
it. CMS enhanced coordination between the Survey & Certification and 
the Q10 Programs. As part of this effort, a CMS Long-Term Care 
Coordinating Task Force was created to establish formal communication 
lines within the Agency. CMS recently released the Task Force's second 
annual report, "2007 Action Plan for (Further Improvement of) Nursing 
Home Quality." This leadership activity translated into significant 
exploration, both in Task IA core contract activities and with the Q10 
Support Contractor, to find innovative ways for the new nursing home 
teams in:

Page 3-Kathryn G. Allen:

QIOs and the State Survey Agencies to work collaboratively to assist 
poorly performing nursing homes.

Secondly, the draft report identifies the special study awarded during 
the 7th SOW to pilot this new approach (the "Collaborative Focus 
Facility" Special Study), but draws insufficient attention to the fact 
that preliminary analysis of cost data in that special study indicates 
that it is very costly for QIOs to work intensively with chronically 
poor performing nursing homes with multiple persistent survey 
deficiencies. Preliminary estimates from the pilot are that additional 
costs for this type of work run five to ten times the cost of helping 
the "usual" nursing home. There is also significant controversy over 
the best way to define a "low-performing" nursing home.

The draft report also fails to note where, for a large number of 
states, the QIO is working intensively with either the great majority 
of nursing homes in the state or a significant percentage of the homes. 
In the 8th SOW, the QIO program operates in 53 contract regions. In 
eight of these contract regions, the QIOs work intensively with 68 
percent or more (up to 100 percent, for an average of 81 percent) of 
the nursing homes in that state or territory. In ten additional 
contract regions, the QIOs work intensively with more than one quarter 
of the nursing homes in the state (up to 45 percent, for an average of 
35 percent). Overall, for these 18 contract regions (34 percent of the 
program) the QIOs work intensively with an average of 56 percent of the 
nursing homes in the state. The overwhelming majority of these nursing 
homes score worse (in most cases much worse) than the national average 
on the quality measures they work to improve.

Finally, the draft report strongly suggests that use of the publicly 
reported quality measures as the main metric for quality interventions 
is "problematic". CMS disagrees with this opinion, as the publicly 
reported quality measures in the nursing home setting have passed 
through rigorous development, testing, deployment, and national 
consensus processes involving the highest levels of technical expertise 
on health care quality measurement in the country.

The CMS uses the Minimum Data Set (MDS), which nursing homes report 
periodically for each resident, to generate publicly reported quality 
measures (currently 14 for long-term care residents and five for 
patients in short-stay skilled nursing facilities). Since 2002, the 
Nursing Home Compare Web site has provided facility-level quality 
measures to the public, along with the service array and other basic 
descriptors of facilities, sorted by geography and other 
characteristics. The nursing home industry regularly employs the 
quality measures for quality management, alongside other quality 
management tools such as Survey & Certification and internal quality 
assessment and improvement. State Survey Agencies also regularly use 
the quality measures to assist in their work. The quality measures 
serve as a basis for improvement activities and research, and CMS 
continues to update and improve the measures and their reporting 
overtime. This ongoing process for revising measures has become a major 
engine for generating tools and insights that substantially advance the 
measurement and improvement of quality in Medicare.

'Nursing Home Compare at www.medicare.gov/nhcompare/ home.asp:

Page 4-Kathryn G. Allen:

2. The CMS should improve the monitoring and evaluation of Q10 
activities.

The CMS agrees with this recommendation. The Institute of Medicine of 
the National Academies, in its report released last year (Medicare's 
Quality Improvement Organization Program: Maximizing Potential), 
outlined I9 specific recommendations to CMS regarding the management, 
cost accounting, and evaluation systems for the QIO contracts. Right 
now, CMS, in concert with the Department of Health and Human Services, 
is evaluating the recommendations from the IOM report and considering 
ways to redesign the QIO Program. The IOM recommendations apply equally 
to all aspects of the QIO Program, and all settings in which QIOs are 
currently deployed (hospitals, physician offices, and home health 
agencies, as well as nursing homes). 'thus, there are no 
recommendations specific to the work QIOs do with nursing homes. 
However, many of the aspects of the QIO nursing home work (especially 
its data handling and reporting features) may be incorporated into the 
redesigned QIO contract. In addition, CMS and the Department are 
looking at a completely new approach to QIO evaluation for future 
contract cycles.

3. The CMS should require QIOs to share with CMS the identity of homes 
assisted intensively in order to facilitate evaluation.

The CMS is very much aware of the regulatory restrictions imposed upon 
disclosure of the identities of the identified participant groups 
(IPGs) and other identifiable data connected to the QIOs' quality 
review study activity as defined in Federal regulations at 42 CFR 
480.101. Given that the nature of the work with practitioners, 
providers and institutions such as nursing homes is voluntary and most 
often addresses poor performance in health care delivery, the 
regulations implementing section 1 160 of the Social Security Act are 
very restrictive in protecting highly sensitive identifying information 
from disclosure. The GAO correctly notes that under the regulatory 
provision at 42 CFR 480.140, CMS can only view the IPGs (which GAO 
calls "intensive participants") on site at a QIO. The regulatory 
provision allows for the on-site evaluation of the work of the QIO and 
prevents the QIO from transferring this data to CMS where it would then 
fall under potentially less restrictive disclosure rules (Health 
Insurance Portability and Accountability Act and Privacy Act) and be 
subject to the Freedom of Information Act. However, CMS is committed to 
conducting effective and efficient oversight of program activities and, 
therefore, continues to explore options which would allow access to 
this data for evaluation of the QIOs individually and the program as a 
whole, while maintaining appropriate safeguards necessary to promote 
voluntary participation in quality improvement initiatives. The CMS 
greatly appreciates the GAO's recommendation in this area and will 
continue to seek a balance between disclosure and confidentiality.

Conclusion:

The CMS appreciates the GAO's efforts to study the QIOs' work with 
nursing homes and will consider the GAO's recommendations in defining 
future QIO work in this area and others.

[End of section]

Appendix IV: GAO Contact and Staff Acknowledgments:

GAO Contact:

Kathryn G. Allen, (202) 512-7118 or allenk@gao.gov:

Acknowledgments:

In addition to the contact named above, Walter Ochinko, Assistant 
Director; Nancy Fasciano; Sara Imhof; Elizabeth T. Morrison; Colbie 
Porter; and Andrea Richardson made key contributions to this report.

[End of section]

Related GAO Products:

Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not 
Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241. 
Washington, D.C.: March 26, 2007.

Nursing Homes: Despite Increased Oversight, Challenges Remain in 
Ensuring High-Quality Care and Resident Safety. GAO-06-117. Washington, 
D.C.: December 28, 2005.

Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses in 
State and Federal Oversight of Quality of Care. GAO-05-78. Washington, 
D.C.: November 12, 2004.

Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in Federal 
Standards and Oversight. GAO-04-660. Washington D.C.: July 16, 2004.

Nursing Home Quality: Prevalence of Serious Problems, While Declining, 
Reinforces Importance of Enhanced Oversight. GAO-03-561. Washington, 
D.C.: July 15, 2003.

Nursing Homes: Public Reporting of Quality Indicators Has Merit, but 
National Implementation Is Premature. GAO-03-187. Washington, D.C.: 
October 31, 2002.

Nursing Homes: Quality of Care More Related to Staffing than Spending. 
GAO-02-431R. Washington, D.C.: June 13, 2002.

Nursing Homes: More Can Be Done to Protect Residents from Abuse. GAO- 
02-312. Washington, D.C.: March 1, 2002.

Nursing Homes: Federal Efforts to Monitor Resident Assessment Data 
Should Complement State Activities. GAO-02-279. Washington, D.C.: 
February 15, 2002.

Nursing Homes: Sustained Efforts Are Essential to Realize Potential of 
the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.: September 
28, 2000.

Nursing Home Care: Enhanced HCFA Oversight of State Programs Would 
Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.: November 4, 
1999.

Nursing Home Oversight: Industry Examples Do Not Demonstrate That 
Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington, 
D.C.: August 13, 1999.

Nursing Homes: Proposal to Enhance Oversight of Poorly Performing Homes 
Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999.

Nursing Homes: Complaint Investigation Processes Often Inadequate to 
Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999.

Nursing Homes: Additional Steps Needed to Strengthen Enforcement of 
Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.: March 18, 
1999.

California Nursing Homes: Care Problems Persist Despite Federal and 
State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.

FOOTNOTES

[1] Medicare is the federal health care program for elderly and certain 
disabled individuals. Medicare may cover up to 100 days of skilled 
nursing home care following a hospital stay. Medicaid is the joint 
federal-state health care financing program for certain categories of 
low-income individuals. Medicaid also pays for long-term care services, 
including nursing home care.

[2] QIOs take a variety of forms. They can be for-or not-for-profit 
organizations and can be either sponsored by a significant number of 
actively practicing area physicians or have available to them a 
sufficient number of these physicians to assure adequate peer review. 
In general, QIOs cannot be health care facilities. Prior to 1999, QIOs 
focused on quality improvement in the hospital setting. Beginning in 
1999, CMS required QIOs to also work in an alternative setting; about 
two-thirds selected nursing homes. The QIOs currently also work with 
physician offices, home health agencies, rural or underserved 
populations, and Medicare Advantage organizations to improve Medicare 
beneficiaries' quality of care. For the 7th SOW, the 53 QIO contracts, 
one for each state, the District of Columbia, and 2 territories (Puerto 
Rico and the Virgin Islands) were held by 37 organizations. We excluded 
the 2 territories from our study because of substantial differences in 
health care financing between the territories and the states. 

[3] See Related GAO Products at the end of this report.

[4] The minimum data set (MDS) consists of data that are periodically 
collected to assess the care needs of residents in order to develop an 
appropriate plan of care. State surveyors use MDS data to help assess 
the quality of resident care, and Medicare and some state Medicaid 
programs also use MDS data to adjust nursing home payments.

[5] The Web site can be accessed at www.Medicare.gov/NHCompare/ 
home.asp.

[6] The QIO contract is divided into tasks and subtasks; the nursing 
home component is subtask 1a. The amount budgeted for this component in 
the 8th SOW (the QIO contract covering the period from 2005 through 
2008) was approximately $10 million less than was budgeted in the 7th 
SOW.

[7] We ranked nursing homes as high-, moderately, or low-performing on 
the basis of the number, scope, and severity of the deficiencies for 
which they were cited (relative to other homes in their state) in three 
standard state surveys from 1999 through 2002. We based our 
classification of homes on their performance level relative to other 
homes in the state to take into account the inconsistency in how states 
conduct surveys, a problem we have reported on since 1998. A limitation 
of our analysis is that we did not have information about all of the 
homes that volunteered for intensive assistance, only those that were 
selected by the QIOs, and therefore did not know the extent to which 
low-performing homes volunteered for intensive assistance.

[8] Because a QIO is responsible for quality improvement activities in 
each state and the District of Columbia, we refer to the 51 QIOs 
throughout this report. 

[9] To assist in the development of our site visit interview protocols, 
we also interviewed personnel from three other QIOs. On each of our 
five site visits, we interviewed officials from three stakeholder 
groups: (1) the state survey agency; (2) the local affiliate for the 
American Health Care Association, which generally represents for-profit 
homes; and (3) the local affiliate for the American Association of 
Homes and Services for the Aging, which represents not-for-profit 
homes. 

[10] IOM of The National Academies, Committee on Redesigning Health 
Insurance Performance Measures, Payment, and Performance Improvement 
Programs, Board on Health Care Services, Medicare's Quality Improvement 
Organization Program: Maximizing Potential (Washington, D.C.: The 
National Academies Press, 2006). The Medicare Prescription Drug, 
Improvement, and Modernization Act of 2003, Pub. L. No. 108-173, 
§109(d), 117 Stat. 2066, 2173-74, directed the Secretary of Health and 
Human Services to ask the IOM to conduct an evaluation of the QIO 
program administered by CMS. In 2006, the IOM issued a report that 
examined performance within the entire QIO program, including the 
nursing home component, during the 7th SOW. 

[11] A pressure ulcer is an area of damaged skin and tissue that 
results from constant pressure due to an individual's impaired 
mobility. The pressure results in reduced blood flow and eventually 
causes cell death, skin breakdown, and the development of an open 
wound. Pressure ulcers can occur in individuals who are bed-or 
wheelchair-bound, sometimes after only a few hours. 

[12] In our survey of the QIOs, we asked them to identify the 
interventions they relied on most and the interventions that were most 
effective in improving the quality of nursing home care; we allowed the 
QIOs to define these terms. 

[13] CMS's Survey and Certification Group is responsible for oversight 
of state survey agency activities. 

[14] Surveys must be conducted at each home on average once every 12 
months but no less than once every 15 months.

[15] This analysis excluded 13 states because fewer than 100 homes were 
surveyed, and even a small increase or decrease in the number of homes 
with serious deficiencies in such states could produce a relatively 
large percentage-point change. In fiscal year 2005, about 17 percent of 
the 16,337 homes surveyed had serious deficiencies. See GAO, Nursing 
Homes: Efforts to Strengthen Federal Enforcement Have Not Deterred Some 
Homes from Repeatedly Harming Residents, GAO-07-241 (Washington, D.C.: 
Mar. 26, 2007).

[16] CMS is evaluating a new survey methodology to help ensure that 
surveyors do not miss serious care problems. National implementation 
will depend on the outcome of the evaluation. 

[17] MDS assessments are conducted for all nursing home residents 
within 14 days of admission and at quarterly and yearly intervals 
unless there is a significant change in condition. In addition, 
Medicare beneficiaries in a Medicare-covered stay are assessed through 
MDS on or before the 5TH, 14TH, 30TH, 60TH, and 90TH day of their stays 
to determine if their Medicare coverage should continue.

[18] GAO, Nursing Homes: Public Reporting of Quality Indicators Has 
Merit, but National Implementation Is Premature, GAO-03-187 
(Washington, D.C.: Oct. 31, 2002).

[19] See GAO-02-279. 

[20] Some states that adjust nursing home payments to account for 
variation in resident care needs have their own separate MDS review 
programs.

[21] Pub. L. No. 97-248, §141-50, 96 Stat. 381-95. PROs were renamed 
QIOs in 2002. Under the provisions of the Peer Review Improvement Act 
of 1982 and implementing regulations, a QIO can be either a physician- 
sponsored entity or a physician-access entity. See 42 C.F.R. §475.101 
(2005). QIOs are allowed to be either for-or not-for-profit entities 
and are required to include at least one consumer representative on the 
QIO governing board. Funding for QIO activities comes from the Medicare 
Trust Funds.

[22] IOM, Medicare's Quality Improvement Organization Program: 
Maximizing Potential.

[23] IOM defines collaboratives as interventions designed to bring 
together stakeholders working toward quality improvement for the same 
clinical topic. Participants usually follow the same processes to reach 
goals and interact on a regular basis to share knowledge, experiences, 
and best practices. 

[24] According to a CMS official, all QIO contracts prior to the 6th 
SOW, which began in 2000, were considered "cost plus fixed fee" and 
there were no deliverables, or set targets, that QIOs had to meet in 
order to obtain payment. In the late 1990s, however, the Office of 
Management and Budget instructed CMS to make QIO contracts performance- 
based with deliverables and objectives that QIOs had to meet during the 
contract cycle. In response, CMS changed the QIO contract so that part 
of QIOs' fee was based on their performance. 

[25] The Rhode Island QIO was awarded the support contract for nursing 
homes for the 7th SOW. The contract defined roles for the QIO support 
contractor, including (1) providing QIOs with information on clinical 
topics and management systems' approaches and techniques for quality 
improvement; (2) facilitating coordination and communication between 
QIOs; (3) maintaining a nursing home informational clearinghouse Web 
site with best practices, tools, and interventions; and (4) being 
available for ongoing technical assistance.

[26] The $106 million represented 13 percent of the total amount ($809 
million) awarded to QIOs for their base contracts. CMS did not budget 
separately for statewide and intensive assistance.

[27] The QIO support contractor subcontracted with another QIO to 
provide data analysis.

[28] For example, QIOs could move funds between the nursing home 
component and the other components under task 1, which covered clinical 
quality improvement efforts with home health agencies, hospitals, 
physician offices, underserved and rural beneficiaries, and Medicare 
Advantage organizations.

[29] QIOs working in the 13 states with fewer than 100 nursing homes 
were expected to target at least 10 homes. 

[30] See Social Security Act §1160; 42 C.F.R. §480.140 (2005).

[31] Stakeholders may include representatives of nursing homes, trade 
associations, ombudsmen, state survey agencies, medical directors, 
directors of nursing, geriatric nursing assistants, other licensed 
professionals, academicians, and consumers.

[32] Under the 8th SOW contract, QIOs will not be held accountable for 
QM improvement statewide.

[33] QIOs could fail to meet contract expectations for up to 2 of the 
12 components and still remain eligible for noncompetitive renewal of 
their contracts.

[34] Initiated in January 1999, the Special Focus Facility program was 
expanded by CMS in December 2004. Expansion strengthened enforcement 
authority so that if homes in the program fail to significantly improve 
performance from one survey to the next, immediate sanctions must be 
imposed; if homes show no significant improvement in 18 months and 
three surveys, they must be terminated from participation in the 
Medicare and Medicaid programs.

[35] The eight states are Florida, Maryland, Texas, Washington, Maine, 
Michigan, Missouri, and North Carolina. We identified some of these 
states by reviewing reports and asking officials in states that we knew 
had quality assurance programs to identify other states with similar 
programs. We did not attempt to determine if additional states had 
similar programs.

[36] The organizations included the American Health Care Association, 
the Alliance for Quality Nursing Home Care, and the American 
Association of Homes and Services for the Aging, which are three of the 
largest long-term care organizations and together represent the 
majority of the approximately 16,400 nursing facilities in the United 
States.

[37] In the 8th SOW contracts, CMS specified more selection parameters, 
requiring QIOs to work with two groups of intensive participants, 
including some "persistently poor-performing" homes identified in 
consultation with state survey agencies; increasing the overall number 
of intensive participants; and requiring geographic distribution of 
these homes.

[38] QIOs could select more than 15 percent of the homes in their state 
for intensive assistance. However, the weight given to this component 
in a QIO's contract evaluation score could not exceed 66 percent-- 
generally, the weight given if the intensive participant group 
comprised 15 percent of homes in the state.

[39] The 13 QIOs in states with fewer than 100 homes were expected to 
work intensively with at least 10 homes.

[40] The 38 QIOs that were expected to work intensively with 10 to 15 
percent of the homes in their state worked with an average of 15 
percent. The other 13 QIOs worked with an average of 15 homes.

[41] The largest proportion of QIOs (27 percent) reported that their 
most effective recruiting tactic was hosting statewide or regional 
conferences for homes; however, 20 percent did not use this tactic at 
all. The vast majority of QIOs (84 to 98 percent) also sent materials 
to homes, contacted homes by telephone, and asked nursing home trade 
associations or other groups to inform homes of the opportunity to 
participate.

[42] Deficiencies are deemed serious if they constitute either actual 
harm to residents or actual or potential for death/serious injury. 

[43] Although many QIOs excluded some interested homes from the 
official list of intensive participants submitted as a contract 
deliverable, most QIOs (75 percent) reported that they gave these homes 
more assistance than they did other homes in the state, and 37 percent 
reported that they gave these homes as much assistance as they gave 
intensive participants. 

[44] Some QIOs also considered financial status and management 
stability in making their selections. Among the 51 QIOs surveyed, 8 
excluded homes that were struggling financially and 5 excluded homes 
with recent management turnover. Personnel at one of the QIOs we 
interviewed explained that the QIO excluded homes with known leadership 
instability in order to avoid having to perform a great deal of 
training and retraining as administrators came and went. 

[45] These numbers do not sum to 18 because 4 of the 5 QIOs that 
selected proportionately more moderately performing homes also selected 
proportionately fewer low-or high-performing homes.

[46] Stakeholders included officials of state survey agencies and state 
nursing home trade associations.

[47] One reason that improvements cannot be definitively attributed to 
the QIOs is that homes may have benefited from other quality 
improvement efforts as well.

[48] In most cases, the state survey agencies and QIOs issued joint 
letters of invitation to the homes, and those that agreed to work with 
the QIOs signed a participation agreement that addressed issues of 
confidentiality and information sharing. The state survey agencies' 
role was generally limited to identifying and helping recruit homes for 
the project. As with homes in the intensive participant group, there 
was little overlap between homes in the Collaborative Focus Facility 
project and homes selected by state survey agencies for the Special 
Focus Facility program. Although the Puerto Rico QIO participated in 
the Collaborative Focus Facility project, our analysis focused on QIOs 
in the 50 states and the District of Columbia. 

[49] Over a 1-year period, the average number of survey deficiencies 
the homes received in five areas (comprehensive assessment, 
comprehensive care plan, pressure sore prevention/treatment, quality of 
care, and physical restraints) changed little, going from 2.59 to 2.60, 
but the average number of serious deficiencies they received in these 
areas declined from 0.93 to 0.71. The homes' QM scores for physical 
restraints and high-and low-risk pressure ulcers improved an average of 
31 percent (or 38 percent when the score with the lowest improvement 
was dropped from the average). 

[50] For their statewide assistance, three-quarters of the QIOs 
selected three QMs, the minimum number contractually allowed; the 
remainder selected four QMs. No QIOs selected the maximum of five.

[51] The intensity of interventions varies by type of intervention (for 
example, on-site versus telephone calls) and with the frequency of use.

[52] In its 2006 report on QIOs, the IOM recommended that Congress 
permit extension of the contract from 3 to 5 years to allow for 
measurement, refinement, and evaluation of technical assistance efforts.

[53] Because the largest component of the QIOs' contract evaluation 
related to the intensive participants, we asked QIOs to rank and 
provide detailed information on a greater number of interventions for 
intensive participants than for statewide participants.

[54] The median number of times an intervention was provided is the 
midpoint of all the times that an intervention was provided, as 
reported by QIOs. Half the QIOs reported a number above the median and 
half reported a number below. 

[55] Nearly all QIOs (94 percent) also reported asking intensive 
participants to complete homework assignments on their own. These 
assignments most frequently involved conducting self audits, comparing 
existing policies and procedures with checklists provided by the QIO, 
and developing new practice protocols related to selected QMs. For 
example, two homes told us they were given cause-and-effect analysis 
exercises to complete to identify possible causes of and solutions to a 
problem. Staff from another home told us that QIO personnel asked them 
to conduct a mock survey to prepare for their next standard survey.

[56] We defined primary personnel as individuals who devoted more than 
20 percent of a full-time work week to the nursing home component of 
the contract. Some primary QIO personnel served as the principal 
contacts, providing quality improvement assistance to homes. According 
to our survey, 78 percent of QIOs also used outside experts 
(consultants or subcontractors) for their quality improvement efforts. 
The majority of QIOs reported using these experts to provide 
presentations or training at conferences, participate in conference 
calls, and develop or review materials. QIOs personnel we interviewed 
told us they also used outside experts to train their primary personnel 
or to provide technical assistance to intensive participant homes. 

[57] Among individual QIOs, the extent of long-term care experience 
spanned a wide spectrum. At five QIOs, 75 percent or more of the 
primary personnel who worked with nursing homes had less than 1 year of 
long-term care experience, while at two QIOs, all of the primary 
personnel who worked with nursing homes had more than 10 years' 
experience. 

[58] The improvement, or relative change, in a home's QM scores is 
calculated by subtracting its score at remeasurement from its score at 
baseline and dividing by its score at baseline. For example, if the 
number of residents with chronic pain in a 100-bed home decreased from 
20 to 12æwhich translates to a change in scores from 0.20 to 0.12æthe 
improvement in the home's pain QM would be 40 percent ([0.20-0.12]/ 
0.20). 

[59] The four QMs specified in the contract are pressure ulcers among 
high-risk patients, restraints, depression management, and chronic pain 
management. With most intensive participants, QIOs are expected to work 
on all four QMs and achieve a relative improvement rate of 15 to 60 
percent. With the small group of persistently poor-performing homes 
QIOs are now required to assist, they are expected to work on two QMs 
(pressure ulcers among high-risk patients and restraints) and achieve 
an improvement rate of 10 percent.

[60] GAO-03-187. 

[61] Greg Arling and others, "Future Development of Nursing Home 
Quality Indicators," The Gerontologist, vol. 45, no. 2 (2005).

[62] A resident who triggers a QM is included in both the numerator and 
denominator when a facility's QM score is calculated.

[63] GAO-02-279 and GAO-03-187. 

[64] In April 2005, CMS ended work under its data assessment and 
verification contract but signed a new contract in September 2005 that 
focused on on-site reviews of MDS accuracy. 

[65] Stages of pressure ulcer formation are I--skin of involved area is 
reddened, II--upper layer of skin is involved and blistered or abraded, 
III--skin has an open sore and involves all layers of skin down to 
underlying connective tissue, IV--tissue surrounding the sore has died, 
exposing muscle and bone.

[66] William Rollow and others, "Assessment of the Medicare Quality 
Improvement Organization Program," Annals of Internal Medicine, vol. 
145, no. 5 (2006).

[67] Because homes must volunteer and be selected by the QIOs to 
receive intensive assistance, intensive participants may differ from 
nonintensive participants in ways that affect their capacity to improve 
their QM scores, such as differences in motivation and commitment, 
available resources, and competing priorities. 

[68] At a meeting on October 31, 2006, of the Technical Expert Panel 
convened by the contractor tasked to design an evaluation of the QIO 
program for the Office of the Assistant Secretary for Planning and 
Evaluation of HHS, panel members underscored the difficulty of 
controlling for a subjective condition such as motivation to improve 
the quality of care and noted the potential for biased assessments of 
the impact of the QIOs if differences in motivation are not accounted 
for appropriately.

[69] According to CMS guidance, the names of participants in 
collaborative quality improvement projects constitute quality review 
study information. See QIO Manual, §16005 (Rev. 07-11-03). Federal 
regulations specify that quality review study information revealing the 
identities of practitioners and institutions must be disclosed to CMS 
"on site" or at the QIOs' place of operation. See 42 C.F.R. §480.140 
(2005). That restriction does not apply to disclosures to certain other 
federal agencies, such as HHS Office of Inspector General or GAO. See 
42 C.F.R. §480.140(b)(2005).

[70] For the survey conducted during the 7th SOW, the response rate for 
nursing homes was 95 percent.

[71] CMS did not provide this cost estimate during the course of our 
work.

[72] To assist in the development of our site visit interview 
protocols, we interviewed personnel from three additional QIOs 
(Massachusetts, Rhode Island, and Washington) and staff from one 
nursing home in each of four other states (Maryland, Massachusetts, New 
Hampshire, and Virginia).

[73] The four states were Florida, Maryland, Texas, and Washington. We 
subsequently learned that four other states, (Maine, Michigan, 
Missouri, and North Carolina) also had state-sponsored quality 
improvement programs. 

[74] We contacted officials of programs in six states: Florida, 
Maryland, Michigan, North Carolina, Texas, and Washington.

[75] QIOs working in states with at least 100 nursing homes were 
expected to target 10 to 15 percent of all homes in the state for 
intensive assistance. In the state we selected that had the highest 
number of homes (Florida), 7 percent of the homes in the state equaled 
approximately 8 homes. 

[76] This analysis drew on data from the On-line Survey, Certification, 
and Reporting system (OSCAR), a database maintained by CMS that 
compiles the results of every state survey conducted at Medicare-and 
Medicaid-certified facilities nationwide. 

[77] We eliminated from the analysis 1,946 homes that had a standard 
survey in the year prior to November 1, 2002, but for which we were 
unable to identify two additional surveys during the period we 
specified. The homes that we eliminated represented a larger proportion 
of the group of homes not selected by the QIOs (11.8 percent) than of 
the group of homes that were selected by the QIOs (3.4 percent).

[78] CMS defines immediate jeopardy as actual or potential for death/ 
serious injury.

[79] Because homes with the same number of deficiencies were assigned 
the same rank, in some cases the top and bottom quartiles included more 
than 25 percent of the homes in the state. We based our classification 
of homes on their performance level relative to other homes in the 
state to take into account the inconsistency in how states conduct 
surveys, a problem we have reported on since 1998. An alternative 
approach, which would not take into account the inconsistency in how 
states conduct surveys, would be to classify homes based on the 
absolute number of deficiencies they had receivedæfor example, to 
classify all homes with five or more serious deficiencies as low- 
performing homes. For data on inconsistencies, see GAO-06-117 and GAO-
07-241.

[80] We used the Satterthwaite t-test because it does not require the 
variances of the two groups to be equal. We rejected the null 
hypothesis that the proportions of two groups were equal when the p- 
value from the Satterthwaite t-test was less than 0.05. 

[81] We asked the QIOs to complete a separate survey for each state in 
which they worked during the 7th SOW.

[82] We defined primary personnel as employees, subcontractors, or 
consultants who worked with nursing homes or provided direct oversight 
of those individuals, excluding administrative support staff and 
individuals who worked less than 20 percent of a full-time work week on 
the nursing home component. 

[83] QIOs were also given the option of specifying other interventions 
they used.

[84] Because QM improvement among intensive participants constituted 
the largest part of the QIOs' contract evaluation score, we asked QIOs 
to rank and provide detailed information on a greater number of 
interventions for intensive participants than for statewide 
participants.

GAO's Mission:

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548:

To order by Phone: 

Voice: (202) 512-6000:

TDD: (202) 512-2537:

Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: