This is the accessible text file for GAO report number GAO-10-737 
entitled 'Health Care Quality Measurement: The National Quality Forum 
Has Begun a 4-Year Contract with HHS' which was released on July 14, 
2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 
GAO: 

July 2010: 

Health Care Quality Measurement: 

The National Quality Forum Has Begun a 4-Year Contract with HHS: 

GAO-10-737: 

GAO Highlights: 

Highlights of GAO-10-737, a report to congressional committees. 

Why GAO Did This Study: 

The Medicare Improvements for Patients and Providers Act of 2008 
(MIPPA) directed the Department of Health and Human Services (HHS) to 
enter into a 4-year contract with an entity to perform five duties 
related to health care quality measurement and authorized $40 million 
from the Medicare Trust Funds for the contract. In January 2009, HHS 
awarded a contract to the National Quality Forum (NQF), under which 
HHS will reimburse NQF for its costs and pay additional fixed fees. 
Established in 1999, NQF is a nonprofit member organization that 
fosters agreement on national standards for measuring and public 
reporting of health care performance data. 

This is the first of two reports MIPPA requires GAO to submit on NQF’s 
contract with HHS. In this report, which covers the first contract 
year—January 14, 2009, to January 13, 2010—GAO describes (1) the 
status of NQF’s work on the five duties under MIPPA; (2) the costs and 
fixed fees NQF has reported; and (3) what NQF and HHS do in order to 
help ensure that NQF’s reported costs are proper. 

GAO reviewed relevant MIPPA provisions and reviewed HHS and NQF 
documents, such as HHS’s contract with NQF, monthly progress reports 
and invoices for the first contract year, and policies and other 
documents that describe how HHS and NQF review invoices. GAO also 
interviewed NQF and HHS officials responsible for implementing and 
overseeing the contract. 

What GAO Found: 

NQF has begun work for each of five duties required by MIPPA related 
to quality measures: (1) make recommendations on a national strategy 
and priorities; (2) endorse quality measures, which involves a process 
for determining which ones should be recognized as national standards; 
(3) maintain—that is, update or retire—endorsed quality measures; (4) 
promote electronic health records; and (5) report annually to Congress 
and the Secretary of HHS. As of January 13, 2010—the end of the first 
contract year—NQF’s work for four MIPPA duties was in progress and it 
had completed its first annual report for the fifth duty. For example, 
NQF had begun the duties related to endorsement and maintenance by 
initiating the endorsement process for three projects HHS selected and 
by starting maintenance reviews for a set of measures of interest to 
or used by HHS. While NQF began work for each of the duties in the 
first contract year, HHS determines on an annual basis the work NQF 
will be expected to perform under the five duties each contract year. 

NQF reported costs and fixed fees totaling approximately $6.5 million 
for the first contract year, including direct and indirect costs as 
well as fixed fees. Specifically, NQF reported about $3.2 million in 
direct costs, or 49 percent of the total. These were costs 
specifically incurred for the NQF contract, such as direct labor for 
NQF employees. NQF also reported about $2.9 million in indirect costs, 
which cover additional items such as employee benefits and overhead. 
Finally, NQF reported about $360,000 in fixed fees for the first 
contract year. Over $5 million of the reported costs and fixed fees 
were incurred in the second half of the contract year. 

NQF and HHS rely on reviews of NQF invoices in order to help ensure 
that NQF’s reported costs are proper. At NQF, officials told us that 
they review the invoices prior to submitting them to HHS and carry out 
other activities, such as using an electronic system to track labor 
hours, in order to help ensure that the costs they report in the 
invoices are proper. Like NQF, HHS relies on reviews of NQF invoices 
in order to help ensure NQF’s reported costs are proper. These reviews 
are governed by HHS policies and procedures and by requirements 
applicable to federal contracts generally. 

While NQF has begun work under the MIPPA contract, it is too early for 
GAO to assess whether, or to what extent, NQF will be successful in 
carrying out the five MIPPA duties. This report describes NQF’s work 
for the first of 4 contract years. In the remaining 3 years of the 
contract, HHS will determine on an annual basis specific work for NQF 
to complete under each of the five MIPPA duties. Therefore, it is not 
yet known exactly what work NQF will be expected to complete during 
the remainder of the contract period. GAO’s second report, which is 
due in January 2012, will provide another opportunity to review NQF’s 
performance and costs. HHS and NQF reviewed a draft of this report and 
provided technical comments, which GAO incorporated as appropriate. 

View [hyperlink, http://www.gao.gov/products/GAO-10-737] or key 
components. For more information, contact Linda T. Kohn at (202) 512-
7114 or kohnl@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

NQF Has Begun Work for Each of the Five Duties Required by MIPPA 
Related to Quality Measures: 

NQF Reported about $6.5 Million in Costs and Fixed Fees for the First 
Contract Year: 

NQF and HHS Rely on Reviews of NQF Invoices in Order to Help Ensure 
That NQF's Reported Costs Are Proper: 

Concluding Observations: 

Agency and Other External Comments: 

Appendix I: National Quality Forum's Endorsement Process and Example 
Project: 

Appendix II: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Description of Quality Measurement Duties as Specified in the 
Medicare Improvements for Patients and Providers Act of 2008 (MIPPA): 

Table 2: Procedures Required under HHS Policy When Reviewing Invoices 
and Implementation of These Procedures for the NQF Contract: 

Table 3: National Quality Forum's (NQF) Endorsement Process and 
Example Project: 

Figure: 

Figure 1: National Quality Forum's (NQF) Costs and Fixed Fees Reported 
for the First Year of the Contract with HHS: 

Abbreviations: 

APU program: Reporting Hospital Quality Data for Annual Payment Update 
Program: 

ASPE: Assistant Secretary for Planning and Evaluation: 

CMS: Centers for Medicare & Medicaid Services: 

CSAC: Consensus Standards Approval Committee: 

FAR: Federal Acquisition Regulation: 

HHS: Department of Health and Human Services: 

MIPPA: Medicare Improvements for Patients and Providers Act of 2008: 

NQF: National Quality Forum: 

PPACA: Patient Protection and Affordable Care Act of 2010: 

QDS: Quality Data Set: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

July 14, 2010: 

Congressional Committees: 

Health care quality measures are used to evaluate how health care is 
delivered, and information obtained from such measures can promote 
accountability among health care providers and help consumers make 
informed choices about their care. The Department of Health and Human 
Services (HHS) encourages use of quality measures through programs 
that provide financial incentives to health care providers who 
voluntarily collect and report information on certain quality 
measures, which HHS then makes publicly available.[Footnote 1] For 
example, as part of one program, HHS reported that in fiscal year 
2009, almost all--96 percent--of eligible hospitals participating in 
Medicare reported on their performance against certain quality 
measures. Recent legislation requires HHS to implement additional 
programs that will rely on quality measures, such as a pay-for-
performance program under which HHS will pay incentives to hospitals 
based on their performance on selected quality measures.[Footnote 2] 

The Medicare Improvements for Patients and Providers Act of 2008 
(MIPPA) directed HHS to enter into a 4-year contract with an entity to 
perform five duties related to health care quality measurement: (1) 
make recommendations on a national strategy and priorities, (2) 
endorse quality measures, (3) maintain endorsed quality measures, (4) 
promote electronic health records, and (5) report annually to Congress 
and the Secretary of HHS.[Footnote 3] MIPPA authorized $10 million per 
year--$40 million in total--from the Medicare Trust Funds for this 4-
year contract, which covers the period from January 14, 2009, through 
January 13, 2013. In addition, the Patient Protection and Affordable 
Care Act (PPACA), which was enacted in March 2010, established 
additional duties for the entity. 

On January 14, 2009, HHS awarded the 4-year contract required by 
MIPPA, after issuing a solicitation seeking competitive proposals, 
[Footnote 4] to the National Quality Forum (NQF), a nonprofit 
organization established in 1999 that fosters agreement on national 
standards for measurement and public reporting of health care 
performance data. NQF uses a process recognized under the National 
Technology Transfer and Advancement Act of 1995 that grants quality 
measures and other standards endorsed by consensus-based entities, 
such as NQF, standing as national voluntary consensus standards. 
[Footnote 5] NQF uses its process to evaluate available quality 
measures to determine which ones are qualified to be endorsed--that 
is, recognized--as national standards. NQF-endorsed quality measures 
have been used by HHS in its quality measurement programs.[Footnote 6] 
In 2008, prior to receiving the contract award, NQF's revenue from all 
sources was approximately $10 million. NQF staff told us that while 
NQF has previously received funding from HHS for some of its work 
related to quality measures, the $10 million per year authorized by 
MIPPA for the contract is larger than previous funding. 

HHS's 4-year contract with NQF is a cost-plus-fixed-fee contract, 
under which HHS will pay NQF for its costs and additional fixed fees 
for its services.[Footnote 7] The Federal Acquisition Regulation (FAR) 
[Footnote 8] provides that cost-plus-fixed-fee and other types of cost-
reimbursement contracts may only be used when the contractor's 
accounting system is adequate for determining costs under the contract 
and appropriate government surveillance during performance will 
provide reasonable assurance that efficient methods and effective cost 
controls are used. For the purposes of this report, we refer to costs 
that are allowable under the contract as "proper."[Footnote 9] 

MIPPA required GAO to study the performance of and costs incurred by 
NQF under its 4-year contract with HHS and submit a first report by 
July 14, 2010, and a second report by January 14, 2012.[Footnote 10] 
This first report covers the first contract year that began January 
14, 2009, and ended January 13, 2010. In this report, we describe (1) 
the status of NQF's work on the five duties related to health care 
quality measurement required under MIPPA, (2) the costs and fixed fees 
that NQF has reported under its contract, and (3) what NQF and HHS do 
in order to help ensure that NQF's reported costs are proper. 

To describe the status of NQF's work on duties related to health care 
quality measurement required under MIPPA, we focused our review on the 
status of NQF's work related to the five MIPPA duties as of the end of 
the first contract year, January 13, 2010. We reviewed relevant 
provisions in MIPPA, and HHS and NQF documents related to implementing 
health care quality measurement duties in MIPPA. Specifically, we 
reviewed HHS's contract with NQF and NQF's 2009 annual work plan, 
which established specific activities for implementing these duties as 
well as scheduled time frames for the activities. We also reviewed the 
monthly progress reports NQF is required to submit to HHS on its 
efforts for the first contract year, and we reviewed NQF's first 
annual report to HHS and Congress. We interviewed NQF officials 
responsible for implementing the contract and HHS officials 
responsible for managing the contract and overseeing NQF's 
performance. For NQF activities in progress at the end of the first 
contract year, we gathered information about their planned completion 
dates as of January 13, 2010. Our finding is limited to the duties 
established under MIPPA and does not include additional duties 
mandated by PPACA, which was enacted after the end of the first 
contract year. 

To describe the costs and fixed fees that NQF has reported under its 
contract, we reviewed NQF invoices submitted to HHS for the first 
contract year--January 14, 2009, through January 13, 2010. These 
invoices include the amounts of costs and fixed fees reported by NQF. 
We also reviewed NQF's monthly progress reports to HHS for the first 
contract year. We interviewed NQF officials responsible for reviewing 
and approving the costs and fixed fees submitted to HHS under the 
contract. We also interviewed HHS officials responsible for reviewing 
NQF's costs and fixed fees reported under the contract. Based on our 
review of relevant documents and interviews with NQF and HHS 
officials, we determined that the reported costs and fixed-fee data 
were sufficiently reliable for the purposes of this report. 

To describe what NQF and HHS do in order to help ensure that NQF's 
reported costs are proper, we interviewed NQF and HHS officials and 
reviewed relevant policies and procedures. For NQF, we interviewed 
officials about their process for reviewing NQF invoices submitted to 
HHS and about the other policies and procedures that NQF has in order 
to help ensure that the costs they report to HHS in the invoices are 
proper. We focused our discussions on policies and procedures related 
to employee labor costs and payments to contractors and consultants, 
which are the majority of NQF's direct costs. We also reviewed the 
invoices that NQF submitted to HHS during the first contract year for 
evidence of NQF approval by NQF officials. Additionally, we examined 
the files that NQF maintained on its subcontracts to review 
documentation for the eight subcontractors and consultants that 
performed work related to the HHS contract in the first contract year. 
We compared documentation that NQF maintains for the eight 
subcontractors and consultants with the requirements in NQF's January 
2010 procurement policy. For HHS, we interviewed officials responsible 
for reviewing NQF invoices, the project officer and the contracting 
officer for the NQF contract. We identified requirements in relevant 
HHS policies and procedures as well as relevant federal contracting 
requirements for oversight of cost-reimbursement contracts. We 
interviewed HHS officials about how they review NQF invoices. We also 
reviewed documentation in the file HHS maintained on the NQF contract 
for the first contract year to describe whether HHS officials followed 
the invoice review procedures they explained to us. Additionally, we 
reviewed documentation in the NQF contract file related to NQF's use 
of eight subcontractors and consultants that performed work under the 
contract for the first contract year. Our work was limited to 
describing what NQF and HHS do to help ensure that NQF's reported 
costs were proper and did not include a determination of whether the 
costs were proper. 

We conducted this performance audit from September 2009 through June 
2010 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

NQF is a nonprofit organization established in 1999 that fosters 
agreement on national standards for measurement and public reporting 
of health care performance data. Its membership includes more than 400 
organizations that represent multiple sectors of the health care 
system, including providers, consumers, and researchers.[Footnote 11] 
NQF uses a consensus development process to evaluate and endorse 
consensus standards, including quality measures, best practices, 
frameworks, and reporting guidelines. NQF has endorsed over 600 
quality measures in 27 areas, such as cancer and diabetes. NQF 
endorses quality measures developed by other organizations, such as 
the Joint Commission, the National Committee for Quality Assurance, 
and the American Medical Association, rather than developing quality 
measures itself. HHS has used a number of NQF-endorsed measures in 
initiatives to promote quality measurement, and NQF continues to 
endorse quality measures separate from this contract. 

Duties Established in MIPPA: 

MIPPA established five duties related to the use of quality measures. 
See table 1 for a description of the duties. 

Table 1: Description of Quality Measurement Duties as Specified in the 
Medicare Improvements for Patients and Providers Act of 2008 (MIPPA): 

MIPPA duties: Make recommendations on national strategy and priorities; 
Description of MIPPA duties: 
1. (1) The entity shall synthesize evidence and convene key 
stakeholders to make recommendations, with respect to activities 
conducted under this Act, on an integrated national strategy and 
priorities for health care performance measurement in all applicable 
settings. In making such recommendations, the entity shall: 
(A) ensure that priority is given to measures: 
i. that address the health care provided to patients with prevalent, 
high-cost chronic diseases; 
ii. with the greatest potential for improving the quality, efficiency, 
and patient-centeredness of health care; and; 
iii. that may be implemented rapidly due to existing evidence, 
standards of care, or other reasons; and; 
(B) take into account measures that: 
i. may assist consumers and patients in making informed health care 
decisions; 
ii. address health disparities across groups and areas; and; 
iii. address the continuum of care a patient receives, including 
services furnished by multiple health care providers or practitioners 
and across multiple settings. 

MIPPA duties: Endorsement of measures; 
Description of MIPPA duties: 
(2) The entity shall provide for the endorsement of standardized 
health care performance measures. The endorsement process under the 
preceding sentence shall consider whether a measure: 
(A) is evidence-based, reliable, valid, verifiable, relevant to 
enhanced health outcomes, actionable at the caregiver level, feasible 
to collect and report, and responsive to variations in patient 
characteristics, such as health status, language capabilities, race or 
ethnicity, and income level; and; 
(B) is consistent across types of health care providers, including 
hospitals and physicians. 

MIPPA duties: Maintenance of measures; 
Description of MIPPA duties: 
(3) The entity shall establish and implement a process to ensure that 
measures endorsed under the second duty are updated (or retired if 
obsolete) as new evidence is developed. 

MIPPA duties: Promotion of the development of electronic health 
records; 
Description of MIPPA duties: 
(4) The entity shall promote the development and use of electronic 
health records that contain the functionality for automated 
collection, aggregation, and transmission of performance measurement 
information. 

MIPPA duties: Annual report to Congress and the Secretary of Health 
and Human Services; 
Secretarial publication and comment; 
Description of MIPPA duties: 
(5)(A) The entity shall submit to Congress and the Secretary, by not 
later than March 1 of each year (beginning with 2009), a report 
containing a description of: 
i. the implementation of quality measurement initiatives under this 
Act and the coordination of such initiatives with quality initiatives 
implemented by other payers; 
ii. the recommendations made under the first duty; and; 
iii. the performance by the entity of the duties required under the 
contract entered into with the Secretary under subsection (a); 
(B) not later than 6 months after receiving a report under 
subparagraph (A) for a year, the Secretary shall: 
i. review such report; and; 
ii. publish such report in the Federal Register, together with any 
comments of the Secretary on such report. 

Source: GAO summary of MIPPA duties prior to amendments made by the 
Patient Protection and Affordable Care Act (PPACA). 

[End of table] 

NQF Contract: 

For the NQF contract, HHS selected a cost-plus-fixed-fee contract-- 
NQF's first cost-reimbursement contract. Under the cost-plus-fixed-fee 
contract, HHS will reimburse NQF for costs incurred under the contract 
in addition to a fixed fee that is paid regardless of other costs. 
Cost-plus-fixed-fee contracts are used for efforts such as research, 
design, or study efforts where cost and technical uncertainties exist 
and it is desirable to retain as much flexibility as possible in order 
to accommodate change. However, this type of contract provides only a 
minimum incentive to the contractor to control costs. As we reported 
in 2009, these contracts are suitable when the cost of work to be done 
is difficult to estimate and the level of effort required is unknown. 
[Footnote 12] 

Under the FAR, cost-reimbursement contracts may only be used when the 
contractor's accounting system is adequate for determining costs under 
the contract to help prevent situations where contractors bill the 
government for unallowable costs. One method an agency can use to 
determine if an accounting system is adequate is to perform a preaward 
survey of a potential contractor's accounting system prior to awarding 
a contract.[Footnote 13] This review serves as a key control to 
determine whether the potential contractor has an adequate accounting 
system in place to accurately and consistently record costs and submit 
invoices for costs. HHS conducted two preaward surveys of NQF's 
accounting system. HHS's initial review, in November 2007, found that 
NQF's accounting system was inadequate because the system could not 
identify and separate unallowable costs, among other issues. NQF 
subsequently replaced its accounting system, and a second HHS review 
in November 2008 found that the system was adequate. 

Under the FAR, contracts are to contain provisions for agency approval 
of a contractor's subcontracts.[Footnote 14] HHS's contract with NQF 
contains this provision and also requires the approval of consultants. 
This review requires appropriate support documentation provided by the 
contractor to the agency, including a description of the services to 
be subcontracted, the proposed subcontract price, and a negotiation 
memo that reflects the principal elements of the subcontract price 
negotiations between the contractor and subcontractor. 

Two HHS components are principally responsible for administering the 
NQF contract: the office of the Assistant Secretary for Planning and 
Evaluation (ASPE) and the Centers for Medicare & Medicaid Services 
(CMS)--an operational division within HHS.[Footnote 15] To conduct 
oversight of the NQF contract, HHS assembled staff in these two units 
with experience in acquisitions, contract management, and program 
management. Specifically, the project officer for the NQF contract, 
responsible for program management and performance assessment, is a 
representative of ASPE.[Footnote 16] The contracting officer for the 
NQF contract, responsible for administering the contract, is a 
representative of CMS.[Footnote 17] The contracting officer and 
project officer should perform a comprehensive review of contractor 
invoices to determine if the contractor is billing costs in accordance 
with the contract terms and applicable government regulations. 

NQF Has Begun Work for Each of the Five Duties Required by MIPPA 
Related to Quality Measures: 

As of January 13, 2010--the end of the first year of HHS's 4-year 
contract with NQF to implement the MIPPA duties--NQF had begun work 
for each of the five duties required by MIPPA related to health care 
quality measures: (1) make recommendations on a national strategy and 
priorities; (2) endorse quality measures; (3) maintain endorsed 
quality measures; (4) promote electronic health records; and (5) 
report annually to Congress and the Secretary of HHS. While NQF began 
work for each of the duties in the first contract year, HHS determines 
on an annual basis the specific work NQF will be expected to perform 
under the five MIPPA duties in each contract year. 

Recommendations on a National Strategy and Priorities for Quality 
Measurement. NQF has taken steps to begin the duty of making 
recommendations on a national strategy and priorities for quality 
measurement. In October 2009, NQF established a committee of 
stakeholders that is expected to develop recommendations about a 
national strategy and priorities for quality measurement. NQF 
published the recommended priorities in May 2010. The committee's 
recommendations are expected to be based on a synthesis of evidence 
that NQF has collected, using a subcontractor, on 20 conditions that 
account for the majority of Medicare's costs.[Footnote 18] The 
subcontractor collected evidence on existing quality measures for 
these conditions and identified gaps where quality measures did not 
exist. The subcontractor also collected evidence related to each 
condition, such as information on each condition's prevalence, 
treatment costs, variability in providers' treatment of the condition, 
disparities in treatment for patients with the condition, and 
potential to improve quality of care for the condition. The committee 
is expected to consider this evidence when developing recommendations 
on a national strategy and priorities for quality measurement. Under 
PPACA, NQF's recommendations[Footnote 19] on a national strategy and 
priorities must be considered by HHS when it develops a national 
strategy for quality improvement, which HHS is required to submit to 
Congress by January 1, 2011.[Footnote 20] 

Endorsement of Measures. NQF has taken steps to provide for the 
endorsement of quality measures. Prior to its contract with HHS, NQF 
established a process for endorsing quality measures. Under this 
process, organizations that develop quality measures submit them to 
NQF for consideration, in response to specific solicitations by NQF. 
[Footnote 21] NQF forms a committee of experts from its member 
organizations as well as other organizations and agencies to review 
these quality measures against NQF-established criteria, such as the 
usability and feasibility of the measure. After this committee 
evaluates the measures against these criteria, NQF's process allows 
for a period during which its member organizations and the public may 
comment on the committee's recommendation for each measure. The 
process also provides for a period for its member organizations to 
vote on whether the measures should be endorsed by NQF as a national 
standard. Ultimately, NQF's board of directors makes a final decision 
on whether NQF should formally endorse the measures. (See appendix I 
for a detailed description of NQF's endorsement process.) 

In order to provide for the endorsement of quality measures under this 
duty, NQF has taken several steps. Specifically, NQF initiated 
projects and solicited measures to be endorsed using its process for 
each of these projects. These projects relate to quality measurement 
in nursing homes, patient safety, and patient outcomes, and are 
scheduled to be completed between December 2010 and May 2011. In 
addition to endorsing measures, NQF also hired a subcontractor to 
evaluate its endorsement process and recommend ways to improve its 
efficiency and effectiveness. The subcontractor's report and NQF's 
approval of proposed enhancements to the process are due January 2011. 

Maintenance of Endorsed Quality Measures. NQF has taken steps to 
ensure that endorsed measures are maintained--that is, updated or 
retired. Prior to its contract with HHS, NQF established a process for 
maintenance of measures. According to NQF, once a quality measure has 
been endorsed, updated information on the measure's specifications 
should be submitted to NQF annually and the measure should be 
comprehensively reviewed under the maintenance process every 3 years. 
NQF's maintenance process is similar to NQF's endorsement process, in 
that it involves a review of measures against NQF-established 
criteria, a period for public comment, and a final decision by NQF's 
board of directors. In order to implement this process under its 
contract with HHS, NQF began maintenance reviews for 191 measures in 
14 areas such as diabetes and cardiovascular care. The measures were 
identified by HHS as being of interest to, or actually used by, HHS 
programs. By the end of the first contract year, NQF had not 
determined completion dates for maintenance of the 191 measures. As of 
May 2010, maintenance of the 191 measures identified by HHS is 
scheduled to be completed by the end of 2012. 

Promotion of the Development and Use of Electronic Health Records. NQF 
has taken steps towards completing the duty of promoting the 
development and use of electronic health records for use in quality 
measurement.[Footnote 22] As of January 13, 2010, NQF had begun to 
implement a framework that defines a standardized set of data that 
should be captured in patients' electronic health records. The 
framework, known as the Quality Data Set (QDS), is intended to allow 
data from electronic health records to be collected and used in 
quality measurement. Implementation and maintenance of the QDS is 
scheduled to continue through the end of the 4-year contract, which 
ends January 13, 2013. To further promote the development and use of 
electronic health records in quality measurement, NQF began additional 
activities. For example, NQF established a panel of experts to 
recommend additional capabilities to measure utilization. According to 
NQF officials, efforts under this duty are scheduled for completion 
between March 2010 and January 2013. 

Annual Report to Congress and the Secretary of HHS. NQF submitted its 
first annual report to Congress and the Secretary of HHS on March 1, 
2009.[Footnote 23] HHS published this report, with its comments, in 
the Federal Register on September 10, 2009. NQF submitted its second 
annual report, which also covers activities it performed during the 
first contract year, to Congress and the Secretary on March 1, 2010. 
[Footnote 24] 

While NQF has begun work for each of the duties in the first contract 
year, HHS determines on an annual basis the specific work NQF will be 
expected to perform under the five MIPPA duties each contract year. 
Specifically, HHS gives direction for and then approves annual plans 
that NQF develops. These plans can include work begun in prior 
contract years that has not been completed. HHS can adjust work in the 
annual plans in support of each of the five duties. For example, HHS 
officials told us that in future contract years, they may select 
additional projects for the endorsement of quality measures, and 
additional measures for maintenance reviews. 

NQF Reported about $6.5 Million in Costs and Fixed Fees for the First 
Contract Year: 

NQF reported costs and fixed fees totaling approximately $6.5 million 
for the first year of its contract with HHS, which ended January 13, 
2010.[Footnote 25] The amount NQF reported included direct and 
indirect costs, as well as fixed fees. Direct costs, which are costs 
incurred specifically for this contract, represented the largest 
percentage--about $3.2 million, or 49 percent--of the amount NQF 
reported (see figure 1). NQF's reported direct costs were largely 
labor costs for NQF employees and payments to subcontractors and 
consultants. In addition to direct costs, NQF reported about $2.9 
million in indirect costs for the first contract year. Indirect costs 
cover additional items, such as employee benefits, overhead, and 
administrative costs.[Footnote 26] NQF calculates its indirect costs 
based on a formula that takes into account an indirect-cost rate 
approved by HHS and the amounts of certain direct costs.[Footnote 27] 
For example, the formula estimates indirect costs such as employee 
benefits by multiplying an indirect-cost rate by the amount of direct 
costs for labor. Finally, in addition to its direct and indirect 
costs, NQF reported fixed fees of approximately $360,000 during the 
first contract year. HHS pays these fixed fees to NQF in addition to 
reimbursing the organization for its costs.[Footnote 28] 

Figure 1: National Quality Forum's (NQF) Costs and Fixed Fees Reported 
for the First Year of the Contract with HHS: 

[Refer to PDF for image: pie-chart] 

Direct costs: $3,205,503: 49%; 
Indirect costs: $2,939,901: 45%; 
Fixed fee: $357,228: 6%; 
Total costs: $6,502,631: 100%. 

Source: GAO analysis of NQF data. 

Note: Data are as of the end of the first contract year, January 13, 
2010. 

[End of figure] 

Of the approximately $6.5 million in costs and fixed fees NQF reported 
for the first contract year, most were incurred in the second half of 
the contract year. Costs and fixed fees in the second half of the 
contract year, from July 1, 2009, to January 13, 2010, totaled over $5 
million.[Footnote 29] NQF staff told us that costs in the first half 
of the contract year were primarily for activities such as development 
of solicitations for subcontractors. Costs in the second half of the 
contract year were primarily for activities related to quality 
measurement, such as endorsement of quality measures and promotion of 
electronic health records for use in quality measurement. 

NQF and HHS Rely on Reviews of NQF Invoices in Order to Help Ensure 
That NQF's Reported Costs Are Proper: 

NQF reviews invoices and carries out other activities prior to 
submitting them to HHS in order to help ensure that reported costs are 
proper. HHS requires its officials to follow certain procedures when 
reviewing these invoices. 

NQF Reviews Invoices and Carries out Other Activities in Order to Help 
Ensure That Its Reported Costs Are Proper: 

NQF officials told us their organization has several ways to help 
ensure that the contract costs it reports to HHS are proper. According 
to NQF officials, invoices are electronically generated using NQF's 
accounting system and then reviewed before submitting the invoices to 
HHS for payment. These reviews are conducted by two senior staff--the 
NQF Project Director, who manages the contract, and the Chief 
Financial Officer. These officials meet to review costs reported in 
each month's invoice. NQF officials told us that as part of their 
reviews, the two officials compare the current month's invoice to the 
previous month's invoice to identify discrepancies or cost trends that 
seem unusual and that the officials investigate such discrepancies or 
trends when necessary. After this review, the Chief Financial Officer 
signs the invoice. During our review of NQF's invoices for the first 
contract year, we found that the Chief Financial Officer signed the 
invoices as the officials described to us. 

In addition to the review of invoices, NQF officials described other 
ways the organization helps to ensure that the costs it reports to HHS 
are proper. In particular, NQF officials told us NQF uses an 
electronic timesheet system in order to track employee labor hours. 
[Footnote 30] NQF officials told us that the timesheet system allows 
NQF employees to track their labor hours by project and have their 
labor hours reviewed and approved by the appropriate NQF officials. In 
addition to the timesheet system, NQF officials told us that their 
organization established a written procurement policy in August 2009 
and revised it in January 2010 to guide how they track other direct 
costs--specifically, payments to subcontractors and consultants--that 
are reported in NQF's invoices.[Footnote 31] NQF officials told us 
that under its procurement policy, NQF officials are to obtain the 
appropriate approval signatures for payments on invoices as well as 
other payments for subcontractors and consultants once the services 
have been received. Furthermore, according to the policy, NQF 
officials are to document how key procurement decisions are made, such 
as the basis for setting an award cost or price for a subcontractor or 
consultant. Having a well-designed procurement policy can help reduce 
the risk of inappropriate payments or pricing related to 
subcontractors and consultants. During our review of NQF subcontractor 
and consultant files for the period prior to January 2010--before NQF 
revised its procurement policy--we found that NQF did not always 
document approvals for subcontractor payment and did not document that 
it had determined that its consultant pricing was reasonable. 

HHS Requires Its Officials to Review NQF Invoices following Certain 
Procedures in Order to Help Ensure That Reported Costs Are Proper: 

Like NQF, HHS relies on reviews of NQF's invoices in order to help 
ensure that reported costs are proper.[Footnote 32] Two HHS officials 
assigned to oversee the NQF contract, the project officer and the 
contracting officer, are responsible for these reviews.[Footnote 33] 
When conducting their reviews, the two officials are required to 
follow certain procedures established in HHS policies.[Footnote 34] 
For example, under these policies, the project officer is required to 
review NQF's invoices to determine whether billed services were 
actually provided and are supported with adequate documentation. 
Similarly, the contracting officer is required to review the invoices 
to determine whether NQF's reported costs are consistent with its 
contract, accurately calculated, and have adequate documentation. Both 
officials are required to document when they approve invoices for 
payment to NQF. When we reviewed HHS documentation and interviewed HHS 
officials during the course of our work, we found that the contracting 
officer and project officer had generally followed the review 
procedures required by HHS policy. 

Table 2 provides more detailed information on the procedures that the 
project and contracting officers are required to follow when reviewing 
NQF invoices. Table 2 also provides information we obtained from HHS 
officials on how they implemented these requirements. 

Table 2: Procedures Required under HHS Policy When Reviewing Invoices 
and Implementation of These Procedures for the NQF Contract: 

Required procedure for reviewing invoices: 
1. The project officer must review each of the cost categories 
reported in NQF's invoice to ensure that billed services were actually 
received, that they were appropriate, and that they are adequately 
supported with documentation submitted by NQF. The project officer can 
recommend to the contracting officer disapproval of costs that do not 
meet these criteria; 
Implementation of procedure for the NQF Contract: 
HHS officials told us that, as required, the project officer reviewed 
each of the cost categories in NQF invoices with the project manager, 
who works with the project officer to help provide technical direction 
to NQF. Officials told us that the project officer recommended the 
disapproval of certain costs to the contracting officer that were not 
appropriate. Our review of HHS documentation showed that the project 
officer questioned some of NQF's costs and recommended to the 
contracting officer disapproval of certain costs for services NQF 
should not have performed. For example, the project officer 
recommended disapproval of costs that NQF had billed for work on a 
project that had been placed on hold. 

Required procedure for reviewing invoices: 
2. The project officer is to document his or her approval of an 
invoice for payment; 
Implementation of procedure for the NQF Contract: 
For the NQF contract, the project officer told us that she sends an e-
mail each month to the contracting officer to document her approval of 
the invoice. We found this documentation during our review of the NQF 
contract file. 

Required procedure for reviewing invoices: 
3. The contracting officer or the contract specialist, who provides 
support to the contracting officer, is required to review the invoices 
to determine if, among other things, (1) all costs are consistent with 
the requirements of the contract, (2), all necessary supporting 
documentation for costs are attached to the invoice, (3) all 
calculations are correct and there are no obvious errors; 
Implementation of procedure for the NQF Contract: 
The contracting officer told us that he reviews NQF's invoices with 
the assistance of the contract specialist to ensure that HHS pays only 
for completed work that had been authorized by the project officer. In 
particular, he stated that he reviews all costs reported in the 
invoices to ensure they are consistent with the requirements of the 
contract, and that he reviews supporting documentation for the costs 
provided by NQF. HHS officials told us that because NQF invoices can 
range from 100 to 200 pages, the contracting officer or the contract 
specialist perform checks on a selection of costs within each invoice 
to verify that calculations are correct. In addition, they may rely on 
contract audits to determine if costs are proper.[A] 

Required procedure for reviewing invoices: 
4. Before approving costs associated with subcontractors and 
consultants, the contracting officer must confirm that the use of each 
subcontractor or consultant was approved[B]; 
Implementation of procedure for the NQF Contract: 
The contracting officer told us that he confirmed that NQF requests to 
use each subcontractor or consultant were approved. He stated that he 
reviews the NQF requests to approve subcontractors and consultants 
prior to reviewing the invoices. He also told us that he has 
disapproved costs associated with subcontractors. In our review of NQF 
and HHS's contract files, we found documentation of these reviews, 
including disapprovals. 

Required procedure for reviewing invoices: 
5. The contracting officer or the contract specialist is required to 
certify whether an invoice is approved for payment by signing it; 
Implementation of procedure for the NQF Contract: 
Our review of NQF invoices found no evidence of signatures indicating 
approval.[C] 

Source: GAO analysis of HHS data and interviews with HHS officials. 

[A] As of February 2010, CMS officials told us that they had not 
determined whether they will conduct an audit of final indirect-cost 
rates each year or after the contract is complete in 2013. 
Furthermore, a CMS official reported in March 2010 that the agency had 
not determined whether CMS or another auditing entity, such as the 
Defense Contract Audit Agency, would perform the audit. 

[B] According to the NQF contract, NQF must submit requests to use 
subcontractors and consultants to the contracting officer. The 
contracting officer must review NQF's request for subcontract or 
consultant approval and, while taking into consideration the project 
officer's recommendation, advise the contractor of the decision to 
consent to or dissent from the proposed subcontract or consultant 
arrangement in writing. 

[C] Reviewing invoices prior to payment is a preventative control that 
may result in the identification of unallowable billings, especially 
on cost-reimbursement contracts, before the invoices are paid, and a 
signature provides evidence of review. 

[End of table] 

Concluding Observations: 

While NQF has begun work in the first year of its contract for the 
five duties related to quality measurement established by MIPPA, it is 
too early for us to assess whether, or to what extent, NQF will be 
successful in carrying out these duties. This report describes NQF's 
work for the first of 4 contract years, and HHS has flexibility to 
determine on an annual basis the specific work it expects NQF to 
perform for each of the MIPPA duties. Therefore, it is not yet known 
exactly what work NQF will be expected to complete during the 
remainder of the contract period. In addition, other events related to 
quality measurement, such as the completion of HHS's national strategy 
for quality improvement, are expected to occur before the end of the 4-
year contract period and may have some influence on NQF's specific 
work for the five MIPPA duties. Our second report will provide another 
opportunity to review NQF's performance and costs. 

Agency and Other External Comments: 

We provided drafts of this report to HHS and NQF for comment. Both HHS 
and NQF provided technical comments, which we incorporated as 
appropriate. 

We are sending copies of this report to the Secretary of Health and 
Human Services and other interested parties. In addition, the report 
will be available at no charge on GAO's Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for 
our Office of Congressional Relations and Office of Public Affairs can 
be found on the last page of this report. Other major contributors to 
this report are listed in appendix II. 

Signed by: 

Linda T. Kohn: 
Director, Health Care: 

List of Committees: 

The Honorable Max Baucus: 
Chairman: 
The Honorable Charles E. Grassley: 
Ranking Member: 
Committee on Finance: 
United States Senate: 

The Honorable Tom Harkin: 
Chairman: 
The Honorable Michael B. Enzi: 
Ranking Member: 
Committee on Health, Education, Labor, and Pensions: 
United States Senate: 

The Honorable Henry A. Waxman: 
Chairman: 
The Honorable Joe Barton: 
Ranking Member: 
Committee on Energy and Commerce: 
House of Representatives: 

The Honorable Sander M. Levin: 
Chairman: 
The Honorable David Camp: 
Ranking Member: 
Committee on Ways and Means: 
House of Representatives: 

[End of section] 

Appendix I: National Quality Forum's Endorsement Process and Example 
Project: 

The National Quality Forum (NQF) established its endorsement process 
in 2000. NQF's process includes the nine steps described in table 3 
below. The table also provides information on the endorsement process 
as applied to a project to endorse a number of measures related to 
home health care, such as measures on education provided to patients 
and caregivers on medications for care and increases in the number of 
pressure ulcers. This project was initiated prior to the NQF contract 
with the Department of Health and Human Services (HHS) that was 
required by the Medicare Improvements for Patients and Providers Act 
of 2008. NQF announced a call for nominations for steering committee 
members for this project in August 2008 and the final set of 20 
endorsed measures was announced on March 31, 2009. 

Table 3: National Quality Forum's (NQF) Endorsement Process and 
Example Project: 

Steps in NQF endorsement process: 
1. Notice of Intent to Call for Measures for Endorsement Consideration; 
At the beginning of a project where NQF seeks to endorse measures, NQF 
usually issues a public notice of its intent to call for measures for 
endorsement consideration. The notice includes a brief background on 
the project and a statement on the scope of the project's activities; 
Home health measures project dates and details: 
NQF did not issue a notice of intent for this project because this 
step was added to the process in April 2009, after the project's 
completion. 

Steps in NQF endorsement process: 
2. Call for Nominations for Steering Committee Members; NQF issues a 
call for nominations for experts to serve on a steering committee, 
which will oversee the endorsement project. Any interested party can 
submit nominations for the steering committee during this 30-day 
period. NQF selects members of a steering committee based upon their 
expertise, their potential contribution to the project, and the need 
for input from a particular stakeholder perspective. Generally, a 
steering committee is composed of individuals affiliated with NQF 
member organizations, unless a necessary stakeholder perspective or 
specific expertise is not available among NQF's membership; 
Home health measures project dates and details: In August 2008, NQF 
issued a call for nominations to serve on the steering committee for 
this project. NQF selected 20 individuals representing the following 
member organizations: 8 were from provider organizations (5 of which 
were home health providers); 2 were from consumer advocacy groups; 
5 were health professional organizations; 3 were from quality 
measurement, research, and improvement groups; 1 was from a supplier 
and industry group; and 1 was from a health plan. 

Steps in NQF endorsement process: 
3. Call for Measures for Endorsement Consideration; Approximately 14 
days after the Intent to Call for Measures notice is issued, NQF 
issues a formal call for submission of measures that are candidates 
for endorsement. Any organization or agency, such as the Centers for 
Medicare & Medicaid Services (CMS), can submit measures for 
consideration during this 30-day period; 
Home health measures project dates and details: Between September 15 
and October 14, 2008, 57 measures were submitted to NQF, all from CMS. 

Steps in NQF endorsement process: 
4. Steering Committee Review of Measures for Endorsement Consideration; 
After the end of the 30-day period for submission of measures, the 
steering committee conducts a detailed review of all submitted 
measures. The duration of the steering committee's review can vary 
depending on the scope of the project, the number of standards under 
review, and the relative complexity of the standards. Submitted 
measures are evaluated against four criteria, but the measures must 
meet the first of these criteria in order to be evaluated against the 
remaining criteria. The four criteria are: 
Importance to measure and report: extent to which a measure is 
important for making significant gains in health care quality and for 
improving health outcomes within a high-impact aspect of health care 
where there is variation in or overall poor performance; 
Scientific acceptability of measure properties: extent to which a 
measure produces consistent (reliable) and credible (valid) results 
about the quality of care; 
Usability: extent to which intended audiences (e.g. consumers, 
purchasers, providers, policymakers) can understand the results of the 
measure and are likely to find them useful for decision making; 
Feasibility: extent to which the required data are readily available, 
retrievable without undue burden, and can be implemented for 
performance measurement; 
Based on its detailed evaluation, a steering committee can recommend 
either that (1) a measure continue through the process toward possible 
endorsement by NQF, or (2) a measure be returned for further 
development and refinement; 
Home health measures project dates and details: During October and 
November 2008, the steering committee reviewed the 57 measures and 
recommended endorsement of 22. 

Steps in NQF endorsement process: 
5. Member and Public Comment Period; After the steering committee 
completes its initial review of the submitted candidate measures, a 
draft of the committee's recommendations--or "draft report"--is posted 
on the NQF Web site for review and comment by NQF member organizations 
and the public. Member organizations have 30 days to comment on all 
submitted measures and the public has 21 days to comment. 
The comments are compiled by NQF staff and submitted to the steering 
committee for consideration. A steering committee may revise its draft 
report in direct response to these comments; 
Home health measures project dates and details: Between December 16, 
2008, and January 14, 2009, the recommended measures were posted for 
comment, and 92 comments were submitted by a total of 22 individuals 
and organizations. Twenty-four comments were from the public and 68 
comments were from NQF member organizations. Based on these comments 
and additional information received by the steering committee, the 
committee revised its recommendation to only include 20 measures. 

Steps in NQF endorsement process: 
6. Member Voting; Member organizations have 30 days to vote on the 
final version of the steering committee's recommendations for each 
measure. Each NQF member organization may cast one vote in favor of or 
against a steering committee's recommendations. A member organization 
may also abstain from voting on a particular consensus development 
project. Only measures that are approved will proceed to the next step 
in the process; 
Home health measures project dates and details: Between January 28 and 
February 26, 2009, 20 measures were posted for voting. Fifty-eight 
member organizations voted on each of the 20 measures. These 
organizations included consumer advocacy groups; health plans; 
health professional organizations; provider organizations; 
purchasers; quality measurement, research, and improvement groups; 
and supplier/industry groups. 

Steps in NQF endorsement process: 
7. Review of Measures by Consensus Standards Approval Committee (CSAC); 
The CSAC, which is a subcommittee of NQF's Board of Directors, reviews 
the measures under consideration for endorsement and voting results 
prior to making a recommendation to the NQF Board of Directors about 
endorsement of the measure. After detailed review of each measure, the 
CSAC determines if consensus has been reached. In this context, NQF 
considers consensus to mean that general agreement has been reached 
across the various member organizations, such as consumers and health 
care professionals and, if there are dissenters, that those opinions 
have been taken into consideration during the review process. The CSAC 
may seek further input from members if there is a lack of consensus. 
The CSAC can recommend full endorsement, time-limited endorsement, or 
denial of endorsement for a measure; 
Home health measures project dates and details: On March 10, 2009, 20 
measures were recommended by the CSAC. 

Steps in NQF endorsement process: 
8. Board of Directors Decision; 
CSAC recommendations regarding endorsement are submitted to the Board 
of Directors. The board can affirm or deny a CSAC decision; 
Home health measures project dates and details: On March 31, 2009, 20 
measures were endorsed by the Board of Directors. 

Steps in NQF endorsement process: 
9. Appeals; Any interested party may file an appeal with the NQF Board 
of Directors of the decision to endorse a measure. An interested party 
may not file an appeal regarding the decision to deny endorsement for 
a measure; An interested party may file a concern about any measure 
(whether endorsed or not endorsed) in the NQF endorsement process and 
this concern will be reviewed by the CSAC; 
Home health measures project dates and details: Between April 1 and 
April 30, 2009, no appeals were filed. 

Source: GAO analysis of NQF data. 

Note: Data are from documents, the Web site, and information provided 
during interviews. 

[End of table] 

[End of section] 

Appendix II: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Linda T. Kohn, (202) 512-7114 or kohnl@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Will Simerl, Assistant 
Director; La Sherri Bush; Helen Desaulniers; Krister Friday; Natalie 
Herzog; Carla Lewis; Lisa Motley; Ruth S. Walk; Rasanjali Wickrema; 
and William T. Woods made key contributions to this report. 

[End of section] 

Footnotes: 

[1] These programs include the Medicare Physician Quality Reporting 
Initiative for physicians and the Reporting Hospital Quality Data for 
Annual Payment Update Program (APU program) for hospitals. For more 
information on the APU program, see GAO, Hospital Quality Data: Issues 
and Challenges Related to How Hospitals Submit Data and How CMS 
Ensures Data Reliability, [hyperlink, 
http://www.gao.gov/products/GAO-08-555T] (Washington, D.C.: Mar. 6, 
2008). 

[2] This program, known as a value-based purchasing program, is 
required by the Patient Protection and Affordable Care Act (PPACA). 
Pub. L. No. 111-148, 124 Stat. 119 (2010). 

[3] Pub. L. No. 110-275, § 183, 122 Stat. 2494, 2583-86. The contract 
may be renewed at the end of the 4-year period after a subsequent 
bidding process. 

[4] MIPPA required HHS to use full and open competition to enter into 
the contract. HHS received only one proposal for the contract. 

[5] See Pub. L. No. 104-113, 110 Stat. 775 (1996). The act directs 
federal agencies and departments to use standards that are developed 
or adopted by voluntary consensus standards bodies, such as NQF, 
whenever possible. 

[6] PPACA requires that HHS choose endorsed measures for certain 
quality measurement programs if it is feasible and practical to do so. 

[7] HHS obligated $10 million for the contract on the date of the 
award and plans to increase the obligated amount each year of the 
contract. Funds obligated but not actually paid for NQF's costs and 
fees in a contract year remain available in subsequent contract years. 

[8] 48 C.F.R. ch. 1. The FAR establishes uniform policies for 
acquisition of supplies and services by executive agencies. Agency 
acquisition regulations may implement or supplement the FAR. 

[9] Cost principles applicable to contracts with nonprofit 
organizations are set forth in the Office of Management and Budget 
Circular A-122, the text of which is located at 2 C.F.R. pt. 230. See 
48 C.F.R. § 31.702 (2009). Under these provisions, costs are allowable 
if they are reasonable and allocable, consistent with any limitations 
and applicable policies, accorded consistent treatment, determined in 
accordance with generally accepted accounting principles, not counted 
elsewhere, and adequately documented. 

[10] MIPPA states that GAO's reports shall be submitted by 18 months 
and 36 months, respectively, after the effective date of the contract. 

[11] NQF classifies its membership as being composed of the following 
groups: provider organizations, which include hospitals, pharmacies, 
and other organizations (33 percent); health professional 
organizations, such as those representing doctors, nurses, and 
clinicians (20 percent); organizations that conduct research, 
education, or initiatives to improve health care quality, measurement, 
and reporting (16 percent); supplier/industry groups that provide 
devices, medications, and other products (8 percent); public/community 
health agencies (7 percent); consumer advocacy groups (7 percent); 
purchasers, such as private organizations and government agencies (6 
percent); and health plans and organizations involved in 
administration of health insurance programs (4 percent). These 
percentages add up to over 100 percent due to rounding. A list of 
NQF's member organizations is available at its Web site, [hyperlink, 
http://www.qualityforum.org/Membership/Membership_in_NQF.aspx]. 

[12] For more information on cost-plus-fixed-fee contracts, see GAO, 
Contract Management: Extent of Federal Spending under Cost- 
Reimbursement Contracts Unclear and Key Controls Not Always Used, 
[hyperlink, http://www.gao.gov/products/GAO-09-921] (Washington, D.C.: 
Sept. 30, 2009). 

[13] An agency's examination is to determine whether an accounting 
system is adequate. These reviews include a determination of whether 
the accounting system can meet generally accepted accounting 
principles and whether it provides for, among many things, proper 
segregation of direct costs from indirect costs. See 48 C.F.R. § 
53.301-1408, FAR Form 1408; 48 C.F.R. § 9.106-4. The scope of HHS's 
review of NQF's accounting system was limited to determining whether 
the design of the system was acceptable for accumulating costs under a 
government contract. 

[14] 48 C.F.R. §§ 44.204(a)(1), 52.244-2 (2009). 

[15] Within CMS, the Office of Acquisition and Grants Management is 
responsible for administering the NQF contract. 

[16] The project officer serves as the technical representative of the 
contracting officer, and provides technical direction to NQF for all 
tasks described in the NQF contract. In addition, the project officer 
monitors NQF's performance and reviews invoices for payment. 

[17] The contracting officer enters into, administers, and terminates 
government contracts. The contracting officer negotiates and prepares 
contract documents, modifies terms or conditions of the contract, and 
approves payment of invoices, among other tasks. 

[18] The 20 conditions are acute myocardial infarction, Alzheimer's 
disease and related disorders, atrial fibrillation, breast cancer, 
cataract, chronic kidney disease, chronic obstructive pulmonary 
disorder, colorectal cancer, congestive heart failure, diabetes, 
endometrial cancer, glaucoma, hip/pelvic fracture, ischemic heart 
disease, lung cancer, major depression, osteoporosis, prostate cancer, 
rheumatoid arthritis and osteoarthritis, and stroke/transient ischemic 
attack. 

[19] According to NQF officials, these recommendations to HHS will be 
based both on NQF's work under the contract and on other NQF 
initiatives. 

[20] Pub. L. No. 111-148, § 3011, 124 Stat. 119, 378-80 (2010). 

[21] Examples of organizations that have developed measures and 
submitted them to NQF include the Joint Commission, the National 
Committee for Quality Assurance, and the American Medical Association. 

[22] According to NQF, its efforts to promote the development and use 
of electronic health records for use in quality measurement support 
"meaningful use" of electronic health records under the American 
Recovery and Reinvestment Act of 2009. The American Recovery and 
Reinvestment Act authorizes CMS to provide reimbursement incentives 
for eligible professionals and hospitals who are successful in 
becoming "meaningful users" of electronic health records. The act 
states that one factor in determining if a provider is a "meaningful 
user" of electronic health records is whether it submits information 
on quality measures selected by HHS. The act also states that, in 
selecting these measures, HHS should give preference to measures 
endorsed by NQF. Pub. L. No. 111-5, §§ 4101-4102, 123 Stat. 115, 469-
70, 479-80 (2009) (codified at 42 U.S.C. §§ 1395w-4(o), 1395ww(n)). 

[23] NQF's first annual report to Congress and the Secretary of HHS 
covers a 6-week period, January 14, 2009, to February 28, 2009. 

[24] NQF's second annual report to Congress and the Secretary of HHS 
covers the period March 1, 2009, through February 28, 2010, which 
includes a portion of the first contract year. NQF's annual reports 
can be found at [hyperlink, 
http://www.qualityforum.org/projects/ongoing/hhs/]. 

[25] These are the costs and fixed fees that NQF reported for the 
first contract year as of May 31, 2010. 

[26] Employee benefits, known as fringe benefit costs, include items 
such as annual leave and holiday pay. Overhead includes items such as 
equipment rental and office supplies. Administrative costs, known as 
general and administrative costs, include bank fees, dues and 
subscriptions, and taxes. 

[27] HHS approved provisional indirect rates for NQF to use during the 
first contract year. These rates are intended to help ensure that 
indirect costs are reasonable for the services provided and within 
limits specified in the contract. The rates are provisional, which 
means that they are used until final indirect-costs rates can be 
established, generally at the end of the contractor's fiscal year. For 
more information on provisional and final indirect cost rates, see 
GAO, Centers for Medicare and Medicaid Services: Deficiencies in 
Contract Management Internal Controls Are Pervasive, [hyperlink, 
http://www.gao.gov/products/GAO-10-60] (Washington, D.C.: Oct. 23, 
2009). 

[28] NQF's contract requires that, in accordance with 48 C.F.R. § 
52.216-8, the payment of the fixed fee be paid monthly until fee 
payments reach 85 percent of the total amount of the fixed fee 
authorized, and after they reach 85 percent HHS may withhold a reserve 
up to 15 percent or $100,000, whichever is less. At such time, the 
contracting officer may withhold further payment of the fee to protect 
the government's interest. 

[29] The increase in costs and fixed fees throughout the year is due 
solely to increases in costs because NQF reported the same amount of 
fixed fees each month. 

[30] Labor costs represented NQF's largest category of direct costs 
during the first contract year. 

[31] Costs associated with subcontractors and consultants accounted 
for over one-third of NQF's direct costs during the first contract 
year. 

[32] In addition to the review of invoices, HHS officials conducted 
two preaward surveys of NQF's accounting system in 2007 and 2008 prior 
to the start of the contract. As a result of these surveys, HHS found 
the design of the contractor's accounting system to be adequate for 
determining costs related to the contract. The FAR requires that this 
determination be made prior to the start of the contract to help 
ensure costs are proper and to reduce improper payments. 

[33] Other HHS officials, such as the project manager and contract 
specialist, provide invoice review support to the project officer and 
contracting officer. 

[34] These policies include those specified in the HHS Project Officer 
manual and CMS's May 2008 invoice review policy. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: