This is the accessible text file for GAO report number GAO-06-67 
entitled 'DOD Business Transformation: Defense Travel System Continues 
to Face Implementation Challenges' which was released on January 18, 
2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to the Chairman, Subcommittee on Government Management, Finance,
and Accountability, Committee on Government Reform, House of 
Representatives: 

October 2005: 

Program Evaluation: 

OMB’s PART Reviews Increased Agencies’ Attention to Improving Evidence 
of Program Results: 

GAO-06-67: 

GAO Highlights: 

Highlights of GAO-06-67, a report to the Chairman, Subcommittee on 
Government Management, Finance, and Accountability, Committee on 
Government Reform, House of Representatives. 

Why GAO Did This Study: 

The Office of Management and Budget (OMB) designed the Program 
Assessment Rating Tool (PART) as a diagnostic tool to draw on program 
performance and evaluation information for forming conclusions about 
program benefits and recommending adjustments to improve results. To
assess progress in improving the evidence base for PART assessments, 
GAO was requested to examine (1) agencies’ progress in responding to 
OMB’s recommendations to evaluate programs, (2) factors facilitating or
impeding agencies’ progress, and (3) whether agencies’ evaluations
appear to be designed to yield the information on program results
that OMB expects. 

What GAO Found: 

GAO examined agency progress on 20 of the 40 evaluations OMB 
recommended in its PART reviews at four federal agencies: the Department
of Energy, Department of Health and Human Services, Department of Labor,
and Small Business Administration. About half the programs GAO reviewed
had completed an evaluation in the 2 years since those PART reviews were
published; 4 more were in progress and 3 were still being planned. 
Program restructuring canceled plans for the remaining 2 evaluations. 

Several agencies struggled to identify appropriate outcome measures and
credible data sources before they could evaluate program effectiveness.
Evaluation typically competed with other program activities for funds, 
so managers may be reluctant to reallocate funds to evaluation. Some 
agency officials thought that evaluations should be targeted to areas 
of policy significance or uncertainty. However, all four agencies 
indicated that the visibility of an OMB recommendation brought agency 
management attention—and sometimes funds—to get the evaluations done. 
Moreover, by coordinating their evaluation activities, agencies met 
these challenges by leveraging their evaluation expertise and 
strategically prioritizing their evaluation resources to the studies 
that they considered most important. 

Because the OMB recommendations were fairly general, agencies had
flexibility in interpreting the kind of information OMB expected. Some
program managers disagreed with OMB on the purpose of their evaluations,
their quality, and the usefulness of “independent” evaluations by third 
parties unfamiliar with their programs. Agency officials concerned 
about an increased focus on process said that they were more interested 
in learning how to improve program results than in meeting an OMB 
checklist. Since a few programs did not discuss their evaluation plans 
with OMB, it is not certain whether OMB will find their ongoing 
evaluations useful during the programs’ next PART review. 

GAO concludes that: 

* The PART review process stimulated agencies to increase their
evaluation capacity and available information on program results. 

* Agencies are likely to design evaluations to meet their own needs—that
is, in-depth analyses that inform program improvement. If OMB wants
evaluations with a broader scope, such as information that helps
determine a program’s relevance or value, it will need to take steps to
shape both evaluation design and execution. 

* Because agency evaluation resources tend to be limited, they are most
usefully focused on important areas of uncertainty. Regular performance
reporting is key to good management, but requiring all federal programs
to conduct frequent evaluation studies is likely to result in 
superficial reviews of little utility and to overwhelm agency 
evaluation capacity. 

What GAO Recommends: 

GAO recommends that OMB (1) encourage agencies to discuss evaluation 
plans with OMB and congressional and other program stakeholders to 
ensure that their findings will be timely, relevant, credible, and 
used; (2) discuss a risk-based allocation of evaluation resources with 
agencies and congressional stakeholders; and (3) continue to improve 
PART guidance and examiners’ training to acknowledge a wide range of
appropriate evaluation methods. OMB agreed that evaluation methodology 
should be appropriate to the size and nature of a program, and noted 
they intended to provide additional guidance in this area. 

[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-67]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Nancy Kingsbury, (202) 
512-2700, KingsburyN@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

About Half the Programs Completed Evaluations, and Three Evaluations 
Were Being Planned: 

Management Attention, Caught by OMB’s Recommendations, Overcame 
Measurement and Funding Barriers: 

Where OMB and Program Managers Do Not Share Expectations, Evaluations 
May Not Meet OMB Needs: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Agency Programs OMB Recommended Evaluations For in PART 
Reviews: 

Appendix II: Related Agency Program Evaluation Reports: 

Appendix III: Comments from the Office of Management and Budget: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Status of Evaluations OMB Recommended in PART Reviews, by 
Agency: 

Table 2: Federal Evaluators’ Views on Tailoring Designs for Program 
Effectiveness Evaluations: 

Abbreviations: 

DOE: Department of Energy: 

DOL: Department of Labor: 

GPRA: Government Performance and Results Act: 

HHS: Department of Health and Human Services: 

OMB: Office of Management and Budget: 

OSHA: Occupational Safety and Health Administration: 

PART: Program Assessment Rating Tool: 

SBA: Small Business Administration: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

October 28, 2005: 

The Honorable Todd R. Platts: 
Chairman: 
Subcommittee on Government Management, Finance, and Accountability: 
Committee on Government Reform: 
House of Representatives: 

Dear Mr. Chairman: 

In the 1990s, Congress and the executive branch laid out a statutory and
management framework for strengthening government performance and
accountability. The Government Performance and Results Act of 1993
(GPRA) was its centerpiece.[Footnote 1] The act was designed to provide
congressional and executive decision makers with objective information
on the relative effectiveness and efficiency of federal programs and
spending. The current administration has made integrating performance
information into budget deliberations one of five governmentwide
management priorities under its President’s Management Agenda.[Footnote 
2] 

A central element of this initiative is the Program Assessment Rating 
Tool (PART), designed by the Office of Management and Budget (OMB) to
provide a consistent approach to assessing federal programs in the
executive budget formulation process. PART is a standard series of
questions meant to serve as a diagnostic tool, drawing on available
program performance and evaluation information to form conclusions
about program benefits and recommend adjustments that may improve
results. 

However, PART’s ability to do this relies on OMB’s access to credible
information on program performance and on policy makers’ confidence in
the credibility of their analysis. In our January 2004 review of PART, 
we found that limited availability of credible evidence on program 
results constrained the ability of OMB staff to use PART to rate 
programs’ effectiveness.[Footnote 3] When OMB first applied PART, for 
the fiscal year 2004 budget, it judged fully half the programs it 
reviewed as not having adequate information on results. Moreover, 
although OMB’s assessments recommended improvements in program design, 
management, and assessment, half the recommendations were to improve 
program assessment—to identify outcome measures and obtain improved
performance data or program evaluations. 

To examine progress in improving the evidence base for the PART
assessments, you asked us to examine: 

1. progress agencies have made in responding to OMB’s PART 
recommendations that they obtain program evaluations; 

2. factors that facilitated or impeded agencies’ progress in obtaining
these evaluations, and; 

3. whether the evaluations appear to have been designed to yield the
information on program results that OMB anticipated. 

To answer these questions, we examined progress on 20 of the 40
evaluation recommendations in the President’s fiscal year 2004 budget
proposal. These 20 recommendations reflect a diverse array of programs
concentrated in the Department of Energy (DOE), the Department of
Health and Human Services (HHS), the Department of Labor (DOL), and
the Small Business Administration (SBA). We reviewed OMB and agency
documents and interviewed officials in the four agencies to learn the
status of the evaluations and the factors that influenced how they were
conducted. We also reviewed the available evaluation plans and reports 
to assess whether they were likely to yield the desired information on 
results. We conducted our review from December 2004 through August 2005 
in accordance with generally accepted government auditing standards. A 
list of the programs reviewed and their evaluation recommendations 
appears in appendix I. OMB provided written comments on a draft of this 
report that are reprinted in appendix III. 

Results in Brief: 

About half of the programs we reviewed (11 of the 20) had completed an
evaluation by June 2005—2 years after the fiscal year 2004 PART reviews
and recommendations were published. Four additional evaluations were in
progress, and 3 were still being planned. Program restructuring canceled
plans for the remaining 2 evaluations. The evaluations employed a 
variety of study designs, reflecting differences between the programs 
and the questions about their performance. For example, the quality of 
research project portfolios had been evaluated with external peer 
review, while occupational safety programs had been assessed on both 
the results of compliance investigations and reduction in workplace 
injuries. 

Several agencies had struggled to identify appropriate outcome measures
and credible data sources before they could conduct evaluations of
program effectiveness. Evaluation generally competes with other program
and department activities for resources, so managers may be reluctant to
reallocate resources to evaluation. Some agency officials thought that
evaluations should not be conducted for all programs but should be
targeted instead to areas of policy significance or uncertainty. 
However, all four agencies indicated that the visibility of an OMB PART
recommendation brought agency management attention—and sometimes
funds—to getting these evaluations done. Moreover, by coordinating their
evaluation activities, agencies met these challenges by leveraging their
evaluation expertise and strategically prioritizing their evaluation
resources to focus on the studies that they considered to be the most
important. 

Because the OMB evaluation recommendations were fairly general, it is
not always clear—and agencies had flexibility in interpreting—what kind
of information OMB expected. Some program managers disagreed with
OMB on the scope and purpose of their evaluations, their quality, and 
the usefulness of evaluations by independent third parties unfamiliar 
with their programs. Agency officials concerned about an increased 
focus on process said that they were more interested in learning how to 
improve program performance than in meeting an OMB checklist. Since a 
few programs did not discuss their evaluation plans with OMB, it is not 
certain whether OMB will find their ongoing evaluations useful during 
the programs’ next PART review. 

To help ensure that agency program evaluations are timely, relevant,
credible, and used, we reiterate and expand on our previous
recommendations to OMB to encourage agencies to discuss their
evaluation plans with OMB and congressional stakeholders, engage in
dialogue with agency and congressional stakeholders on a risk-based
allocation of evaluation resources across programs, and continue to
improve its PART guidance and training to acknowledge a wide range of
appropriate evaluation methods. 

Background: 

PART’s standard series of questions is designed to determine the 
strengths and weaknesses of federal programs by drawing on available 
program performance and evaluation information. OMB applies PART’s 25
questions to all programs under four broad topics: (1) program purpose
and design, (2) strategic planning, (3) program management, and (4)
program results (that is, whether a program is meeting its long-term and
annual goals).[Footnote 4] During the fiscal year 2004, 2005, and 2006 
budget cycles, OMB applied PART to approximately 20 percent of programs 
each year and gave each program one of four overall ratings: 
“effective,” “moderately effective,” “adequate,” or “ineffective,” 
depending on the program’s scores on those questions. OMB gave a fifth 
rating of “results not demonstrated” when it decided that a program’s 
performance information, performance measures, or both were 
insufficient or inadequate. 

The summary assessments published with the President’s annual budget
proposal include recommended improvements in program design,
management, and assessment. For example, a summary of the review’s
findings might be followed by the clause “the administration will 
conduct an independent, comprehensive evaluation of the program,” or 
“the Budget includes [funds] to conduct independent and quality 
evaluations,” both of which we interpreted as an OMB recommendation to 
the agency to conduct such an evaluation.[Footnote 5] In our previous 
analysis of the fiscal year 2004 PART reviews, we analyzed over 600 
recommendations made for the 234 programs assessed and found that half 
of those recommended improvements in program assessment.[Footnote 6] 

PART not only relies on previous program evaluation studies to answer
many of the questions but also explicitly asks, in the strategic 
planning section, “Are independent evaluations of sufficient scope and 
quality conducted on a regular basis or as needed to support program
improvements and evaluate effectiveness and relevance to the problem,
interest, or need?” Program evaluations are systematic studies that 
assess how well a program is working, and they are individually 
tailored to address the client’s research question. Process (or 
implementation) evaluations assess the extent to which a program is 
operating as intended. Outcome evaluations assess the extent to which a 
program is achieving its outcome-oriented objectives; they focus on 
program outputs and outcomes but may also examine program processes to 
understand how outcomes are produced.[Footnote 7] 

OMB first applied PART to the fiscal year 2004 budget during 2002, and 
the assessments were published with the President’s budget in February 
2003. In January 2004, we reported on OMB and agency experiences with 
PART in the fiscal year 2004 budget formulation process.[Footnote 8] We 
noted that PART had helped structure OMB’s use of performance 
information in its budget review and had stimulated agency interest in 
budget and performance integration. However, its effectiveness as a 
credible, objective assessment tool was challenged by inconsistency in 
OMB staff application of the guidance and limited availability of 
credible information on program results. Moreover, PART’s influence on 
agency and congressional decision making was hindered by failing to 
recognize differences in focus and issues of interest among the various 
parties involved in programmatic, policy, and budget decisions. We 
noted that PART’s potential value lay in recommended changes in program 
management and design but would require sustained attention if the 
anticipated benefits were to be achieved. 

To strengthen PART and its use, in our January 2004 report we 
recommended that OMB (1) centrally monitor and report on agency
progress in implementing the PART recommendations; (2) improve PART
guidance on determining the unit of analysis, and defining program
outcomes and “independent, quality evaluation”; (3) clarify expectations
regarding agency allocation of scarce evaluation resources among
programs; (4) target future reviews based on the relative priorities, 
costs, and risks associated with clusters of programs; (5) coordinate 
assessments to facilitate comparisons and trade-offs between related 
program; (6) consult with congressional committees on performance 
issues and program areas for review; and (7) articulate an integrated, 
complementary relationship between GPRA and PART. 

Requesting that we follow up on the findings in our January 2004 report,
you asked that we examine (1) OMB and agency perspectives on the 
effects of PART recommendations on agency operations and results, (2)
OMB’s efforts at ensuring an integrated relationship between PART and
GPRA, and (3) steps OMB has taken to involve Congress in the PART
process. A companion report addresses all three objectives—including
OMB’s outreach to Congress—with regard to all PART reviews. [Footnote 
9] Because of the fundamental role that the availability of program 
evaluations plays in conducting PART assessments, we conducted an in-
depth analysis of agencies’ responses to OMB recommendations that they 
conduct program evaluations. These recommendations were identified 
through the analysis of recommendations for our January 2004 review. 
This report focuses on agencies’ progress on those evaluations and the 
issues involved in obtaining them. For both analyses, we examined the 
same four agencies’ experiences with PART. The four agencies were 
selected to represent a range of program types (such as research and 
regulatory programs), large and small agencies, and, for the purposes 
of this report, a large proportion of the OMB evaluation 
recommendations. 

About Half the Programs Completed Evaluations, and Three Evaluations 
Were Being Planned: 

All but two of the programs we reviewed had responded to some extent to
OMB’s recommendations to conduct an evaluation; agencies did not plan
evaluations of the other programs because they were canceled or
restructured. However, after 2 years, only about half the programs had
completed evaluations, partly because of lengthy study periods and 
partly because of some lengthy planning phases. The evaluations used a 
variety of study designs, reflecting differences in the programs and in 
the questions posed about program performance. 

All Programs Responded to OMB’s Recommendations, but Only Half Completed
Evaluations: 

About half of the programs we reviewed (11 of the 20) had completed an
evaluation by June 2005—2 years after the fiscal year 2004 PART reviews
and recommendations were published. Four evaluations were in progress,
while 3 were still in the planning stage. Agencies did not plan an
evaluation of 2 programs because those programs had been canceled or
restructured. (See table 1.) Most of OMB’s evaluation recommendations
asked for evaluation of the specific program reviewed, while some PART
reviews at DOE and DOL asked the agencies to develop a plan for
conducting multiple evaluations. At DOL, where two entire regulatory
agencies had been assessed, these agencies had completed multiple
studies. 

Table 1: Status of Evaluations OMB Recommended in PART Reviews, by 
Agency: 

Agency (OMB recommendations): DOE (7); 
Completed by June 2005[a]: 5 expert panel reviews; 
In progress: [Empty]; 
Being planned: 1 outcome evaluation; 
None planned: Program discontinued. 

Agency (OMB recommendations): DOL (5); 
Completed by June 2005[a]: 2 comprehensive evaluations, multiple 
process and outcome evaluations, 2 regulatory reviews; 
In progress: [Empty]; 
Being planned: 1 comprehensive evaluation; Additional regulatory 
reviews scheduled; 
None planned: [Empty]. 

Agency (OMB recommendations): HHS (4); 
Completed by June 2005[a]: 1 process evaluation, 1 outcome evaluation; 
In progress: 2 comprehensive evaluations (1 interim report); 
Being planned: [Empty]; 
None planned: [Empty]. 

Agency (OMB recommendations): SBA (4); 
Completed by June 2005[a]: [Empty]; 
In progress: Customer outcome survey (1 interim report); 2 
comprehensive evaluations; 
Being planned: 1 comprehensive evaluation; 
None planned: Program discontinued. 

Source: GAO analysis. 

[a] Comprehensive evaluations combined assessment of program processes 
and outcomes. 

[End of table] 

OMB gave DOE seven evaluation recommendations in its fiscal year 2004
PART reviews. Six were for research programs in basic science and
nuclear energy and one was for its formula grant program to weatherize
the homes of low-income families. Since one research program in the
Office of Science had previously been evaluated by a panel of external
experts called a committee of visitors, OMB explicitly recommended that
the other research programs in that office also institute such a 
process by September 2003. 

In response, DOE completed evaluations of five of the six research
programs, but it did not plan to evaluate the sixth, the Nuclear Energy
Research Initiative, because it considered this not a stand-alone 
program but, rather, a source of funding for follow-up projects to 
other nuclear energy research programs. DOE revised this program’s 
objective and now authorizes funds for its projects through the other 
nuclear energy research programs; thus it is no longer considered a 
separately funded program to be evaluated. Finally, DOE officials 
indicated that they had only recently gained funding for planning the 
evaluation of the Weatherization Assistance program. (A bibliography of 
related agency evaluation reports appears in app. II.) 

OMB gave DOL five evaluation recommendations for fiscal year 2004. Two
were for evaluations of specific DOL programs: grants to state and local
agencies to provide employment-related training to low-income youths
and administration of the Federal Employees Compensation Act regarding
work-related injuries and illnesses. The three others were regulatory
enforcement offices or agencies of DOL that were reviewed in their
entirety: the Office of Federal Contract Compliance Programs, regarding
equal employment opportunity; the Employee Benefits Security
Administration; and the Occupational Safety and Health Administration
(OSHA). OMB recommended that the last, which is a large regulatory
agency, develop plans to evaluate the results of its regulatory and
nonregulatory programs. 

The two DOL regulatory administrations each completed several
evaluations of their enforcement activities by spring 2005, as did two 
of the three other DOL programs we reviewed. DOL is waiting to conduct 
an evaluation of the fifth—the youth employment program—until after its
reauthorization because that is expected to result in an increased 
focus on out-of-school youths and a significant change in program 
activities. In addition, OSHA completed two regulatory “lookback” 
reviews—assessing the cumulative effects of a regulation over time—one 
in 2004 and another in 2005. Program officials indicated that they had 
developed a plan for conducting lookback reviews of employee benefit 
regulations beginning in fiscal year 2006. 

OMB recommended evaluations for four diverse HHS programs: (1) grants
and technical assistance to states to increase childhood disease
immunization, (2) grants to states to help recently arrived refugees 
find employment, (3) education loan repayment and scholarships for 
nurses in return for serving in facilities facing a nursing shortage, 
and (4) direct assistance in constructing sanitation facilities for 
homes for American Indians and Alaskan Natives. Evaluations of the two 
state grant programs were still in progress during our review, although 
an interim report on the immunization program was available. Reports 
from the two other program evaluations had recently been completed and 
were under departmental review. 

OMB recommended evaluations for four SBA programs: (1) support for
existing Business Information Centers that provide information and 
access to technology for small businesses; (2) use of volunteer, 
experienced business executives to provide basic business counseling 
and training to current and prospective entrepreneurs; (3) Small 
Business Development Centers that provide business and management 
technical assistance to current and prospective entrepreneurs; and (4) 
the small business loan program that provides financing for fixed 
assets. OMB also asked all three counseling programs to develop outcome-
oriented annual and long-term goals and measures. 

SBA is conducting customer surveys and had recently initiated a
comprehensive evaluation of one its counseling programs, and is planning
one for the other in fiscal year 2006. Another evaluation has begun to
compare the costs, benefits, and potential duplication of its business 
loan programs. SBA planned no evaluation of the Business Information 
Centers program because the program was canceled, partly as a result of 
the PART review and an internal cost allocation study. In reassessing 
the need for the program, SBA decided that because of the increase in 
commercially available office supplies and services and the 
accessibility of personal computers over the years, such a program no 
longer needed federal government support. 

Evaluation Design and Focus Differed, Reflecting Different Program 
Purposes and Structures: 

Because evaluations are designed around programs and what they aim to
achieve, the form of the evaluations reflected differences in program
structure and anticipated outcomes. The evaluations were typically
multipurpose, including questions about results as well as the agency
processes that managers control in order to achieve those results, and
designed to respond to OMB and yield actionable steps that programs
could take to improve results. 

The Nursing Education Loan Repayment and Scholarship programs aim to
increase the recruitment and retention of professional nurses by 
providing financial incentives in exchange for service in health care 
facilities that are experiencing a critical shortage of nurses. The 
ongoing evaluation of the two programs combined was shaped by the 
reporting requirements of the Nurse Reinvestment Act of 2002.[Footnote 
10] The act requires HHS to submit an annual report to Congress on the 
administration and effect of the programs. Each yearly report is to 
include information such as the number of enrollees, scholarships, loan 
repayments and grant recipients, graduates, and recipient demographics 
to provide a clear description of program beneficiaries. Program 
beneficiaries are compared with the student, nurse applicant and 
general populations to assess success in outreach. Information 
pertaining to beneficiaries’ service in health care facilities is 
important for determining whether program conditions and program goals 
have been met.[Footnote 11] The number of defaulters, default rate, 
amount of outstanding default funds, and reasons for default are 
reported for each year. These data as well as follow-up data on whether
beneficiaries remain in targeted facilities after their term of 
commitment will be important in assessing the overall cost-benefit of 
the program. Subsequent data collection will establish trends and allow 
for a costbenefit analysis in the future. 

The Indian Health Service Sanitation Facilities Construction delivers
construction and related program services to provide drinking water and
waste disposal facilities for American Indian and Alaska Native homes, 
in close partnership with tribes. Among other issues, the evaluation
examined key areas of service delivery, while the health benefits of 
clean water were assumed. Specifically, project needs identification 
and project portfolio management were evaluated to see how well 
construction efforts are prioritized and targeted to areas of greatest 
need, and whether facilities construction projects are competently 
designed, timely, and costeffective. The completed evaluation 
recommended that the agency consider integrating its separate data 
systems into a single portfolio management system to represent all 
projects or, at least, to adopt standardized project management and 
financial tracking systems. 

The primary responsibility of DOL’s Office of Federal Contract
Compliance Programs is to implement and enforce rules banning 
discrimination and establishing affirmative action requirements for 
federal contractors and subcontractors. Because of the time and expense 
involved in conducting compliance reviews and complaint investigations, 
the office is attempting to target establishments for review based in 
part on an analytic prediction that they will be found to discriminate. 
The focus of its effectiveness evaluation, therefore, was on 
identifying a targeting approach and measuring change in the rate of 
discrimination among federal contractors during the period of 
oversight. The logic for this choice of outcome measure was based on 
the expectation that overall rates of discrimination would decrease if 
the oversight programs were effective. Using data on the 
characteristics of establishments that had already been reviewed, 
evaluators used statistical procedures to estimate a model of the
probability of discrimination. The coefficients from that model were 
then used to predict rates of discrimination among contractors who had 
not been reviewed and among noncontractors. The analysis showed that the
office effectively targeted selected establishments for review, but 
there was no measurable effect on reducing employment discrimination in 
the federal contractor workforce overall. To improve the office’s
effectiveness, the evaluators recommended that the office focus on
establishments with the highest predicted rates of discrimination rather
than employ its previous approach, targeting larger establishments that 
are likely to affect a greater number of workers. 

The DOE Office of Science used a peer review approach to evaluating its
basic research programs, adapting the committee of visitors model that
the National Science Foundation had developed. Because it is difficult 
to predict the findings of individual basic research projects, science
programs have adapted the peer review model they use for merit selection
of projects to evaluate their portfolios of completed (and ongoing)
research. The Office of Science convenes panels of independent experts 
as external advisers to assess the agency’s processes for selecting and
managing projects, the balance in the portfolio of projects awarded, and
progress in advancing knowledge in the research area and in contributing
to agency goals. Panel reviews generally found these programs to be
valuable and reasonably well-managed and recommended various
management improvements such as standardizing and automating
documentation of the proposal review process, adopting program-level
strategic planning, and increasing staffing or travel funds to increase
grantee oversight. OSHA, pursuant to section 610 of the Regulatory 
Flexibility Act and section 5 of Executive Order 12866, must conduct 
lookback studies on OSHA standards, considering public comments about 
rules, the continued need for them, their economic impacts, complexity, 
and whether there is overlap, duplicity, or conflict with other 
regulations.[Footnote 12] OSHA recently concluded a lookback review on 
its Ethylene Oxide standard and issued a final report on another 
lookback review that examined the Presence Sensing Device Initiation 
standard for mechanical power presses.[Footnote 13] A press
equipped with a sensing device initiates a press cycle if it senses 
that the danger zone is empty, and if something should enter the zone, 
the device stops the press. Accidents with mechanical presses result in 
serious injuries and amputations to workers every year. 

In the sensing device lookback review, OSHA examined the continued
need for the rule, its complexity, complaints levied against the rule,
overlap or duplication with other rules, and the degree to which
technology, economic conditions, or other factors have changed in the
area affected by the rule. Typically, once a standard is selected for a
lookback review, the agency gathers information on experience with the
standard from persons affected by the rule and from the general public
through an announcement in the Federal Register. In addition, available
health, safety, economic, statistical, and feasibility data are 
reviewed, and a determination is made about any contextual changes that 
warrant consideration. In conducting such reviews, OSHA determines 
whether the standards should be maintained without change, rescinded, 
or modified. OSHA found that there was a continued need for the rule 
but that to achieve the expected benefits of improved worker safety and 
employer productivity, the rule needed to be changed. Although the 
technology for sensing device systems had not changed since their 
adoption in 1988, the technology for controlling mechanical presses had 
changed considerably, with press operation now often controlled by 
computers, introducing hazards that were not addressed initially by the 
standard. 

Management Attention, Caught by OMB’s Recommendations, Overcame 
Measurement and Funding Barriers: 

Agency officials described two basic barriers to completing the
evaluations that OMB recommended: obtaining valid measures of program
outcomes to assess effectiveness and obtaining the financial resources 
to conduct independent evaluations. Although most of the program 
officials claimed that they had wanted to conduct such evaluations 
anyway, they noted that the visibility of an OMB recommendation brought 
evaluation to the attention of their senior management, and sometimes 
evaluation funds, so that the evaluations got done. Indeed, in response 
to the PART reviews and recommendations, two of the agencies initiated 
strong, centrally led efforts to build their evaluation capacity and 
prioritize evaluation spending. 

Measurement Challenges Delayed Evaluation Starts: 

To evaluate program effectiveness, agencies needed to identify
appropriate measures of the outcomes they intended to achieve and
credible data sources for those measures. However, as noted in our
previous report, many programs lacked these and needed to develop new
outcome-oriented performance measures in order to conduct evaluations. 

Agency officials identified a variety of conceptual and technical 
barriers to measuring program outcomes similar to those previously 
reported as difficulties in implementing performance reporting under 
GPRA.[Footnote 14] SBA officials acknowledged that before the PART 
reviews, they generally defined their programs’ performance in terms of 
outputs, such as number of clients counseled, rather than in outcomes, 
such as gains in small business revenue or employment. SBA revised its 
strategic plan in fall 2003 and worked with its program partners to 
develop common definitions across its counseling programs, such as who 
is the client or what constitutes a counseling session or training. 
Since SBA had also had limited experience with program evaluation, it 
contracted for assistance in designing evaluations of the economic 
impact of its programs. 

DOL had difficulty conceptualizing the outcomes of regulations in
monetary terms to produce the cost-benefit analyses that PART (and the
Regulatory Flexibility Act) asks of regulatory programs. For instance,
OSHA has historically considered the likely controversy of quantifying 
the value of a human life in calculating cost-benefit ratios for 
developing worker health and safety regulations. OSHA officials 
explained that the Assistant Secretary had helped to mitigate such a 
controversy by issuing a July 2003 memorandum that directed OSHA staff 
to identify costs, benefits, net benefits, and the impact of 
economically significant regulations and their significant 
alternatives, as well as discuss significant nonmonetized costs and 
benefits. 

DOL officials noted that designing a cumulative assessment of the net
benefits of employer reporting requirements for pension and health 
benefit plans was complicated. For example, a primary benefit of 
reporting is to aid the agency’s ability to enforce other benefit plan 
rules and thereby protect or regain employees’ benefits. They also 
pointed out that although health and safety regulations are mandatory, 
employers are not required to offer benefit plans, so a potential cost 
of regulators’ overreaching in their enforcement actions could be 
discouraging employers from offering these pension and health benefits 
altogether. 

DOE officials acknowledged that they could not continue to use state
evaluations to update the national estimates of energy savings from a
comprehensive evaluation of weatherization assistance conducted a
decade ago. They recognized that assumptions from the original national
evaluation could no longer be supported and that a new, comprehensive
national evaluation design was needed. They noted new hurdles to
measuring reductions in home heating costs since the previous 
evaluation: (1) monthly electric bills typically do not isolate how 
much is spent on heating compared with other needs, such as lighting, 
and (2) the increased privatization of the utility industry is expected 
to reduce government access to the utilities’ data on individual 
household energy use. 

Other barriers were more operational, such as the features of a 
program’s data system that precluded drawing the desired evaluative 
conclusions. For one, regulations need to be in place for a period of 
years to provide data adequate for seeing effects. HHS officials noted 
that their databases did not include the patient outcome measures OMB 
asked for and that they would need to purchase a longitudinal study to 
capture those data. They also noted that variation in the form of 
states’ refugee assistance programs and data systems, as well as 
regional variation in refugees’ needs, made it difficult to conduct a 
national evaluation. Their evaluation especially relied on the 
cooperation of state program coordinators. DOL officials pointed out 
that the federal employees’ compensation program’s data system was 
developed for employee and management needs and did not lend itself to 
making comparisons with the very different state employee compensation 
programs. 

Agencies, with Limited Funds, Delayed or Narrowed Evaluations and
Questioned the Need to Evaluate All Programs: 

Evaluation generally competes for resources with other program and
department activities. Contracts for external program evaluations that
collect and analyze new data can be expensive. In a time of tight
resources, program managers may be unwilling to reallocate resources to
evaluation. Agencies responded to such limitations by delaying 
evaluations or cutting back on an evaluation’s scope. Some agency 
officials thought that evaluations should not be conducted for all 
programs but should be targeted instead to areas of uncertainty. 

HHS’s Office of Refugee Resettlement—which was allotted funds
especially for its evaluation—is spending $2 million to evaluate its 
refugee assistance program over 2 years. Costs are driven primarily by 
the collection of data through surveys, interviews, and focus groups 
and the need for interpreters for many different languages. Given the 
size and scope of the program, even with $2 million, program officials 
would have liked to have more time and money to increase the coverage 
of their national program beyond the three sites they had selected. 

DOL program officials explained that although they had had a large 
program evaluation organization two decades ago, the agency downsized
in 1991, the office was eliminated, and now they must search for program
evaluation dollars. The program spent $400,000 for an 18-month 
evaluation of the Federal Employees Compensation Act program, which 
relied heavily on program administrative data, but they also spent a 
large amount of staff time educating and monitoring the contractor. 
Program officials were disappointed with the lack of depth in the 
evaluation. They believed that their evaluation contractor did not have 
enough time to plan and conduct a systematic survey, and consequently, 
their selective interview data were less useful than they would have 
liked. 

DOE program officials indicated that they have been discussing an
evaluation of Weatherization Assistance since spring 2003, but not 
having identified funds for an evaluation, they have not been able to 
develop a formal evaluation plan. They had no budget line item for 
evaluation, so they requested one in their fiscal year 2005 
appropriations. Although there was congressional interest in an 
evaluation, additional funds were not provided in fiscal year 2005. DOE 
instructed program officials to draw money for evaluation from the 10 
percent of the program’s funds that are set aside for training and 
technical assistance, increase the federal share from 1.5 percent to 2 
percent, and reduce the states’ share to 8 percent. Program officials 
indicated that the amount from the technical assistance account would 
cover only planning and initial implementation activities, not the bulk 
of the evaluation itself. And they were concerned about displacing 
existing training, so they were still looking for an evaluation
funding commitment. 

Agency officials also questioned PART’s assumption that all programs
should have evaluations. SBA officials indicated that some agency
appropriations generally precluded SBA’s spending program funds on any
but specifically identified program activities. Thus, evaluations had 
to be funded from agency administrative funds. They thought that it was
unreasonable to ask a small agency to finance several program
evaluations, as might be expected of a larger agency. SBA dealt with 
this by conducting evaluations sequentially as funds became available. 
DOL program officials also thought that spending several hundred 
thousand dollars for a comprehensive evaluation study was a reasonable 
investment for a $2.5 billion program but not for small programs. They 
did not believe that all programs need to be evaluated—especially in a 
time of budget deficits. They recommended that OMB and agencies should 
“pick their shots” and should be more focused in choosing evaluations 
to conduct. They suggested a risk-based approach, giving higher 
priority to evaluating programs for which costs are substantial and 
effectiveness uncertain. 

OMB’s Recommendations Increased Management Attention and Investment in 
Evaluation: 

Most of the agency officials we interviewed declared that they valued
evaluation. For example, HHS and DOE officials described evaluation as
part of their culture. Many said they had already been planning to do
something similar to the evaluation that OMB had recommended. In a
couple of cases, OMB’s recommendation appeared to have been shaped by
planned or ongoing activities. However, officials in all four agencies
indicated that the visibility of a PART recommendation and associated
OMB pressure brought management attention, and sometimes funds, to
getting the evaluations done. 

HHS departmental officials said that the agency was a federal leader in
terms of evaluation capacity, and that they spend approximately $2.6
billion a year on agency-initiated research, demonstrations, and
evaluation. They stated that it is part of their culture to conduct
evaluations—because their program portfolio is based in the physical and
social sciences. DOE officials said that they embraced the PART process
because, as an agency with a significant investment in advancing science
and technology, DOE had already been using similar processes, such as
peer review, to evaluate its programs. DOE officials noted that DOE had
developed a basic evaluation mechanism—independent peer review—that
all its research programs undertake. Officials in the Office of Energy
Efficiency and Renewable Energy developed a corporate peer review guide 
summarizing best practices in this field and considered their peer 
review process as “state of the art,” as it is used as a model 
nationally and globally.[Footnote 15] 

In other cases, agency or congressional interest in evaluation seemed to
set the stage for OMB evaluation recommendations. For example, while
OMB was reviewing the Nursing Education Loan Repayment program, the
Nursing Reinvestment Act of 2002 was enacted, expanding the program
and instituting a requirement for annual reports after the first 18 
months. The reports were to include data on the numbers of loan 
applicants and enrollees, the types of facilities they served in, and 
the default rates on their loans and service commitments and an 
evaluation of the program’s overall costs and benefits. OMB then 
recommended that the agency evaluate the program’s impact, develop 
outcome measures, and begin to track performance against newly adopted 
benchmarks. To respond to OMB’s request for a long-term outcome 
measure, the agency agreed to also collect information on how long 
beyond their service commitment nurses stay in service in critical 
shortage facilities. In another example previously discussed, the DOE 
Office of Science had already initiated committee of visitors reviews 
for its Basic Energy Sciences program, which OMB then recommended for 
other research programs in that office. 

The PART and President’s Management Agenda pressed agencies to report
progress on the recommendations. OMB published the cumulative set of
completed PART review summaries, including the recommendations, in
the President’s budget proposals for fiscal years 2004 through 2006. In 
the fiscal year 2006 budget, OMB reported on the status of its previous
recommendations in the PART summaries, whether action had been taken
or completed. OMB also asked agencies to report on their progress in
implementing PART recommendations to provide input into its quarterly
scorecards on agencies’ progress in implementing the President’s
Management Agenda initiatives. In addition, OMB precluded agencies from
being scored “green” on Budget and Performance Integration if more than
10 percent of their programs were rated “results not demonstrated” 2 
years in a row. DOE and DOL program officials reported being asked to 
update the status of the recommendations every 2 to 3 months. HHS 
officials noted that since fall 2004, they have been reporting on PART
recommendations to OMB twice a year, tracking approximately 100 PART
recommendations (with about 200 separate milestones) for the 62
programs reviewed for fiscal years 2004 through 2006. 

Most of the officials we interviewed believed that because of PART and
the President’s Management Agenda, their agencies were paying greater
attention to program results and evaluation. Officials at DOL noted that
the department spends much time and effort making sure it scores green
on the next President’s Management Agenda assessment; for example, the
department’s management review board, chaired by Labor’s Assistant
Secretary for Management and Administration, discusses these issues
monthly. In addition, DOL’s Center for Program Planning and Results
reviews programs’ progress on OMB’s recommendations, scores programs
internally on the Budget and Performance Integration scorecard, and
provides agencies with training and preparation before their PART
reviews. The SBA Administrator initiated a series of steps after August
2003 to increase the agency’s focus on achieving results. SBA rewrote 
its strategic plan to focus on a limited number of strategic goals and
integrated its strategic plan, annual performance plan, and performance
report. The agency formed a central Office of Analysis, Planning, and
Accountability to help each program office develop results-oriented
performance measures and conduct program assessments. 

Although HHS officials said that the department had invested in 
evaluation long before the PART reviews, Indian Health Service program 
officials indicated that they had not planned an evaluation of their 
sanitation facilities program before the PART review. However, they 
thought it was a good idea and said that the recommendation brought 
their lack of a recent evaluation to HHS’s attention, making it easier 
to justify efforts to quantify their program’s benefits. 

Centralized Coordination Helped Agencies Leverage Their Evaluation
Resources: 

SBA and DOL responded to demands for more performance information
by centrally coordinating their assessment activities, helping to 
address evaluation’s measurement and funding challenges. Centralization 
helped the agencies to leverage their evaluation expertise throughout 
the agency and helped them prioritize spending on the evaluations they 
considered most important. 

SBA program offices had little experience with outcome measurement and
evaluation before the 2002 PART reviews. The central planning office was
formed to help the program offices develop outcome measures linked to
the agency’s strategic goals and collect and validate their performance
data. The office also conducts an annual staff activity survey to 
support cost allocation across programs, a key step toward performance
budgeting. This office took advantage of the similarity in outcome goals
across SBA’s programs and the evaluation methodology developed for the
counseling programs to contract for the development of a standard
methodology for assessing other SBA programs’ economic impacts on
small businesses. The central office is also funding the subsequent
evaluations. For a small agency, this type of coordination can result in
important savings in contract resources as well as staff time. 

DOL, much larger than SBA, has measurement and evaluation experience,
but capacity had declined over time. DOL established the Center for
Program Planning and Results in 2001 to provide leadership, policy 
advice, and technical assistance to GPRA-related strategic and 
performance planning. The center was expanded in fiscal year 2003 to 
respond to the President’s Management Agenda and manage the PART 
process. With a budget of $5 million a year, the center solicits and 
selects evaluation proposals focusing on program effectiveness 
submitted by DOL’s component agencies, funds the studies, and helps 
oversee the external contractors. The center’s officials claimed that 
the Secretary’s and Assistant Secretary’s support for evaluation, 
combined with pressure from OMB, has led to increased interest by the 
component agencies in evaluation, resulting in $6 million to $7 million 
in proposals competing for $5 million in evaluation funds. Some DOL 
agencies retained their evaluation expertise and design, fund, and 
oversee their own evaluations. In addition to helping program offices 
develop research questions and evaluation designs, the center helps 
develop agency evaluation capacity by holding “Vendor Days,” when 
evaluation contractors are invited to exhibit for agency staff the 
specialized design, data collection, and analysis skills that could 
inform future studies. 

Where OMB and Program Managers Do Not Share Expectations, Evaluations 
May Not Meet OMB Needs: 

Because the OMB evaluation recommendations were fairly general,
agencies had flexibility in interpreting the information OMB expected 
and the evaluations to fund. Some program managers disagreed with OMB on
the scope and purpose of their evaluations, their quality, and the
usefulness of evaluations conducted by independent third parties.
Program managers concerned about an increased focus on process said
that they were more interested in learning how to improve program
performance than in meeting an OMB checklist. Since a few programs did
not discuss their evaluation plans with OMB, it is not certain whether 
OMB will accept their ongoing evaluations. 

Agencies Have Flexibility in Determining Evaluation Timing and Content: 

Agencies had a fair amount of flexibility to design their evaluations.
Except for the recommendations to the DOE Office of Science to conduct
committee of visitors reviews, OMB’s evaluation recommendations were
fairly general, typically telling agencies to conduct an independent
evaluation of a program’s effectiveness. Agencies reported little 
guidance from OMB on how to conduct these evaluations, beyond the PART 
written guidance and the rationale the examiner provided for not 
accepting their previous evaluations or measures of program outcomes. 
They said that follow-up on previous PART recommendations was generally 
limited to providing responses to the OMB reporting template, unless OMB
conducted a second formal PART review. 

Agencies also had flexibility to determine the timing of their 
evaluations. Agency officials reported that OMB did not prioritize its 
recommendations within or among programs. Moreover, because evaluation 
resources were limited, DOL and SBA officials reported that they had to 
choose which evaluations to conduct first. The recommendations for the 
two DOL regulatory agencies explicitly acknowledged their need to 
balance responsibility for several programs. OMB asked these agencies 
to develop plans to evaluate their programs or expand existing efforts 
for more comprehensive and regular evaluation. In the reviews of 
recommendation status for the fiscal year 2006 budget, OMB credited 
both agencies with having conducted one or more program reviews and 
planning others. Agencies were free to choose which programs to 
evaluate but were likely to be influenced by the potential effect of 
PART reassessments on their President’s Management Agenda scores and, 
thus, to attempt to reduce the number of programs rated “results not 
demonstrated.” Research and development programs were held to a 
somewhat higher standard than other programs were, since their agencies 
could not be scored “green” on the separate R&D Investment Criteria 
Initiative if less than 75 percent of their programs received a score 
of “moderately effective” or better. DOE officials noted that their 
Office of Energy Efficiency and Renewable Energy now requires programs 
to outline their plans for evaluations in their multiyear plans. 

Disagreements on the Scope and Purpose of Evaluation: 

OMB and the agencies significantly differed in defining evaluation scope
and purpose. Program officials were frustrated by OMB’s not accepting
their prior evaluations of program effectiveness in the PART review. 
Some of the difficulties seemed to derive from OMB expecting to find, 
in the agencies’ external evaluation studies, comprehensive judgments 
about program design, management, and effectiveness, like the judgments 
made in the OMB PART assessments. 

PART’s criteria for judging the adequacy of agency evaluations are
complex and may have created some tension as to the importance of one
dimension over another. For example, question 2.6 read: “Are independent
evaluations of sufficient scope and quality conducted on a regular 
basis or as needed to support program improvements and evaluate 
effectiveness and relevance to the problem, interest, or need?” OMB 
changed the wording of the question to help clarify its meaning and 
added the reference to “relevance.” However, while OMB’s revised 
guidance for this question defines quality, scope, and independence, it 
does not address the assessment of program “relevance.” Specifically, 
sufficient scope is defined as whether the evaluation focuses on 
achievement of performance targets and the cause and effect 
relationship between the program and target—i.e., program 
effectiveness. This is different from assessing the relevance—i.e., 
appropriateness—of the program design to the problem or need. Instead, 
questions in section 1 ask whether the design is free of major flaws 
and effectively targeted to its purpose. 

Another potential contribution to differences between OMB and agency
expectations for program evaluations is that evaluations designed for
internal audiences often have a different focus than evaluations 
designed for external audiences. Evaluations that agencies initiate 
typically aim to identify how to improve the allocation of program 
resources or the effectiveness of program activities. Studies requested 
by program authorizing or oversight bodies are more likely to address 
external accountability—to judge whether the program is properly 
designed or is solving an important problem. 

HHS officials reported differences with OMB over the acceptability of 
HHS evaluations. HHS officials were particularly concerned that OMB
sometimes disregarded their studies and focused exclusively on OMB’s
own assessments. One program official complained that OMB staff did not
adequately explain why the program’s survey of refugees’ economic
adjustment did not qualify as an “independent, quality evaluation,”
although an experienced, independent contractor conducted the
interviews and analysis. In the published PART review, OMB
acknowledged that the program surveyed refugees to measure outcomes
and monitored grantees on-site to identify strategies for improving
performance. In our subsequent interview, OMB staff explained that the
outcome data did not show the mechanism by which the program
achieved these outcomes, and grantee monitoring did not substitute for
obtaining an external evaluation, or judgment, of the program’s
effectiveness. Other HHS officials said that OMB had been consistent in
applying the standards for independent evaluation, but these standards
were set extremely high. 

In reviewing the vaccination program, OMB did not accept the several
research and evaluation studies offered, since they did not meet all key
dimensions of “scope.” OMB acknowledged that the program had conducted 
several management evaluations to see whether the program could be 
improved but found their coverage narrow and concluded “there have 
previously been no comprehensive evaluations looking at how well the 
program is structured/managed to achieve its overall goals.” OMB also
did not accept an external Institute of Medicine evaluation of how the
government could improve its ability to increase immunization rates
because the evaluation report had not looked at the effectiveness of the
individual federal vaccine programs or how this program complemented
the other related programs. However, in reviewing recommendation 
status, OMB credited the program with having contracted for a 
comprehensive evaluation that was focused on the operations, 
management, and structure of this specific vaccine program. 

DOE Office of Science officials described much discussion with OMB 
examiners about what was or was not a good committee of visitors review
in following up on the status of the evaluation recommendations. 
Although OMB had revised and extended its guidance on what constituted 
quality in evaluation, program officials still found this guidance 
difficult to apply to research programs. They also acknowledged that 
their first committee of visitors reviews might have been more useful 
to the program than to OMB. 

Disagreements about the Quality of Evaluation Designs: 

OMB and agencies differed in identifying which evaluation methods were
sufficiently rigorous to provide high-quality information on program
effectiveness. OMB guidance encouraged the use of randomized controlled
trials, or experiments, to obtain the most rigorous evidence of program
impact but also acknowledged that these studies are not suitable or
feasible for every program. However, as described above, without 
guidance on which—and when—alternative methods were appropriate, OMB 
and agency staff disagreed on whether specific evaluations were of
acceptable quality. To help develop shared understandings and 
expectations, federal evaluation officials and OMB staff held several
discussions on how to assess evaluation quality according to the type of
program being evaluated. 

When external factors such as economic or environmental conditions are
known to influence a program’s outcomes, an impact evaluation attempts 
to measure the program’s net effect by comparing outcomes with an
estimate of what would have occurred in the absence of the program
intervention. A number of methodologies are available to estimate
program impact, including experimental and quasi-experimental designs.
Experimental designs compare the outcomes for groups that were randomly 
assigned to either the program or to a nonparticipating control group 
prior to the intervention. The difference in these groups’ outcomes is 
believed to represent the program’s impact, assuming that random 
assignment has controlled for any other systematic difference between 
the groups that could account for any observed difference in outcomes. 
Quasiexperimental designs compare outcomes for program participants with
those of a comparison group not formed through random assignment, or
with participants’ experience prior to the program. Systematic 
selection of matching cases or statistical analysis is used to 
eliminate any key differences in characteristics or experiences between 
the groups that might plausibly account for a difference in outcomes. 

Randomized experiments are best suited to studying programs that are
clearly defined interventions that can be standardized and controlled, 
and limited in availability, and where random assignment of 
participants and nonparticipants is deemed feasible and ethical. Quasi-
experimental designs are also best suited to clearly defined, 
standardized interventions with limited availability, and where one can 
measure, and thus control for, key plausible alternative explanations 
for observed outcomes. In mature full-coverage programs where 
comparison groups cannot be obtained, program effects may be estimated 
through systematic observation of targeted measures under specially 
selected conditions designed to eliminate plausible alternative 
explanations for observed outcomes.[Footnote 16] 

Following our January 2004 report recommendation that OMB better
define an “independent, quality evaluation,” OMB revised and expanded 
its guidance on evaluation quality for the fiscal year 2006 PART 
reviews. The guidance encouraged the use of randomized controlled 
trials as particularly well suited to measuring program impacts but 
acknowledged that such studies are not suitable or feasible for every 
program, so it recommended that a variety of methods be considered. OMB 
also formed an Interagency Program Evaluation Working Group in the 
summer of 2004 to provide assistance on evaluation methods and 
resources to agencies undergoing a PART review that discussed this 
guidance extensively. Evaluation officials from several federal 
agencies expressed concern that the OMB guidance materials defined the 
range of rigorous evaluation designs too narrowly. In the spring of 
2005, representatives from several federal agencies participated in 
presentations about program evaluation purposes and methods with OMB 
examiners. They outlined the types of evaluation approaches they 
considered best suited for various program types and questions (see 
table 2).[Footnote 17] However, OMB did not substantively revise its 
guidance on evaluation quality for the fiscal year 2007 reviews beyond 
recommending that “agencies and OMB should consult evaluation experts, 
in-house and/or external, as appropriate, when choosing or vetting 
rigorous evaluations.”[Footnote 18] 

Table 2: Federal Evaluators’ Views on Tailoring Designs for Program 
Effectiveness Evaluations: 

Typical designs used to assess program effectiveness: Process and 
outcome monitoring or evaluation; 
Design features that help control for alternative explanations: 
Compares performance to pre-existing goal or standard. For example:
- OMB R&D criteria of relevance, quality and performance. 
- Productivity, cost effectiveness and efficiency standards; 
Best suited for (typical examples): Research, enforcement, information 
and statistical programs, and business-like enterprises with: 
- few, if any, alternative explanations for observed outcomes. 
- ongoing programs producing goods and services. 
- complete national coverage 

Typical designs used to assess program effectiveness: Quasi-experiments 
– Single Group; 
Design features that help control for alternative explanations: Compare 
outcomes for program participants or entities before and after the 
intervention; 
- Multiple data points over time are necessary. 
- Control for alternative explanations by statistical adjustments and 
analyses such as modeling; 
Best suited for (typical examples): Regulatory and other programs with: 
- clearly defined interventions with distinct starting times; 
- complete national coverage; 
- random assignment of participants or entities to groups is NOT 
feasible, practical, or ethical. 

Typical designs used to assess program effectiveness: Quasi-experiments 
– Comparison Groups; 
Design features that help control for alternative explanations: 
Compares outcomes for program participants or entities with outcomes 
for a comparison group selected to closely match the “treatment” group
on key characteristics; 
- Key characteristics are plausible alternative explanations for the 
outcome. 
- Measure outcomes before and after intervention (pretest, posttest). 
Best suited for (typical examples): Service and other programs with: 
- clearly defined interventions that can be standardized and 
controlled; 
- limited national coverage; 
- random assignment of participants or entities to groups is NOT 
feasible, practical, or ethical. 

Typical designs used to assess program effectiveness: Randomized 
experiments; 
Design features that help control for alternative explanations: 
Compares outcomes for randomly assigned program (treatment) 
participants or entities with outcomes for a randomly assigned “control”
group prior to intervention. 
- Measure outcomes before and after intervention (pretest, posttest). 
Best suited for (typical examples): Service and other programs with: 
- clearly defined interventions that can be standardized and 
controlled; 
- limited national coverage; 
- random assignment of participants or entities to groups is feasible 
and ethical. 

Source: Adapted from Eric Bernholz and others, Evaluation Dialogue 
between OMB Staff and Federal Evaluation Leaders: Digging a Bit Deeper 
into Evaluation Science (Washington, D.C.: April 2005). 

[End of table] 

Disagreements about Requiring Independent Third-Party Evaluations: 

A related source of tension between OMB and agency evaluation interests
was the importance of an evaluation’s independence. PART guidance
stressed that for evaluations to be independent, nonbiased parties with 
no conflict of interest, for example, GAO or an Inspector General, 
should conduct them. OMB subsequently revised the guidance to allow
evaluations to be considered independent if the program contracted them
out to a third party or they were carried out by an agency’s program
evaluation office. However, disagreements continued on the value and
importance of this criterion. 

HHS officials reported variation among examiners in whether their
evaluations were considered independent. Two programs objected to OMB
examiners’ claims that an evaluation was not independent if the agency
paid for it. OMB changed the fiscal year 2005 PART guidance to recognize
evaluations contracted out to third parties and agency program 
evaluation offices as possibly being sufficiently independent, subject 
to examination case by case. But HHS officials claimed that they were 
still having issues with the independence standard in the fiscal year 
2006 reviews and that OMB’s guidance was not consistently followed from 
one examiner to the next. 

DOL program officials stated that using an external evaluator who was 
not familiar with the program resulted in an evaluation that was not 
very useful to them. In part, this was because program staff were 
burdened with educating the evaluator. But more important, they claimed 
that the contractor designed the scope of the work to the broad 
questions of PART (such as questions on program mission) rather than 
focusing on the results questions the program officials wanted 
information on. In combination, this led to a relatively superficial 
program review, in their view, that provided the external, independent 
review OMB wanted but not the insights the program managers wanted. 

In reviewing the status of its PART recommendations, OMB did not accept
advisory committee reviews for two research programs that DOE offered
in response because OMB did not perceive the reviews as sufficiently
independent. These two program reviews involved standing advisory
committees of approximately 50 people who review the programs every 3
years. The OMB examiner believed that the committee was not truly
independent of the agency. DOE program officials objected, noting the
committee’s strong criticisms of the program, but have reluctantly 
agreed to plan for an external review by the National Academies. Program
officials expressed concern that because evaluators from the National
Academies may not be sufficiently familiar with their program and its
context, such reviews may not address questions of interest to them 
about program performance. 

HHS program officials were also concerned about the usefulness of an
evaluation of the sanitation facilities program if it was conducted by a
university-based team inexperienced with the program. The agency
deliberately guarded against this potential weakness by including two 
exagency officials (one an engineer) on the evaluation team, and by 
taking considerable effort with the team to define the evaluation 
questions. 

Agencies Not Consulting with OMB on Evaluation Plans May Not Meet OMB’s
Expectations: 

Agencies’ freedom to design their evaluations, combined with differences
in expectations between agencies and OMB, raises the strong possibility
that the evaluations that agencies conduct may not provide OMB with the
information it wants. Most of the agency officials we interviewed said 
that they had discussed their evaluation plans with their OMB 
examiners, often as part of their data collection review process. SBA 
and DOL, in particular, appeared to have had extensive discussions with 
their OMB examiners. However, a few programs have not discussed their 
plans with OMB, presumably on the assumption that they will meet OMB’s 
requirements by following its written guidance. 

Officials in SBA’s and DOL’s central planning offices described 
extensive discussions of their evaluation plans with their OMB 
examiners. SBA vetted the evaluation design for SBA’s counseling 
programs with OMB in advance, as well as the questionnaire used to 
assess client needs. DOL planning and evaluation officials noted that 
they had worked with OMB examiners to moderate their expectations for 
agencies’ evaluations. They said that OMB understands their “real 
world” financial constraints and is allowing them to “chip away” at 
their outcome measurement issues and not conduct net impact evaluations 
in program areas where they do not have adequate funds to do this type 
of evaluation. 

HHS program officials were concerned about whether OMB will accept 
their ongoing evaluation of the immunization program when they receive
their next PART review. The evaluation recommendation was general, so
they based their design on the fiscal year 2004 criteria and to provide
information useful to the program. However, the officials had heard that
the fiscal year 2007 evaluation quality criteria were more rigid than 
those previously used, so they were concerned about whether the program 
will meet OMB’s evaluation criteria when it is reviewed again. They 
said they would have liked OMB to consider its evaluation progress and 
findings so far and to have given them input as to whether the 
evaluation will meet the current criteria. OMB officials denied that 
the PART criteria for evaluation quality had changed much in the past 
two years. They also expected, from their review of the design, that 
this new evaluation would meet current PART criteria, assuming it was 
carried out as planned. 

Several program officials expressed the view that in designing their
evaluations, they were more concerned with learning how to improve their
programs than in meeting an OMB checklist. Program officials complained
that OMB’s follow-up on whether evaluations were being planned sent the
message that OMB was more interested in checking off boxes than in
having a serious discussion about achieving results. When one program
official was asked for the program’s new evaluation plan, he answered
“Who needs a plan? I’ve got an evaluation.” DOE program officials
indicated that they believe a comprehensive evaluation of Weatherization
Assistance should include all the questions that state, regional, and 
local officials would like to ask and not just establish a new national 
energy savings estimate. Those questions—also of interest to 
DOE—include: Which weatherization treatments correlate with energy 
savings? Should they use their own crews or hire contractors? What are 
the nonenergy benefits, such as improved air quality or employment 
impacts? Program officials indicated that they had conducted a great 
deal of planning and discussion with their stakeholders over the past 5 
to 6 months and expect to conduct five or six studies to meet those 
needs. 

Conclusions: 

The PART review process has stimulated agencies to increase their
evaluation capacity and available information on program results. The
systematic examination of the array of evidence available on program
performance has helped illuminate gaps and has helped focus evaluation
questions. The public visibility of the results of the PART reviews has
brought management attention to the development of agency evaluation
capacity. 

Evaluations are useful to specific decision makers to the degree that 
the evaluations are credible and address their information needs. 
Agencies are likely to design evaluations to meet their own needs—that 
is, in-depth analyses that inform program improvement. If OMB wants 
evaluations with a broader scope, such as information that helps 
determine a program’s relevance or value, it will need to take steps to 
shape both evaluation design and execution. 

Because agency evaluation resources tend to be limited, they are most
usefully focused on illuminating important areas of uncertainty. While
regular performance reporting is key to good program management and
oversight, requiring all federal programs to conduct frequent evaluation
studies is likely to result in many superficial reviews that will have 
little utility and that will overwhelm agency evaluation capacity. 

Recommendations for Executive Action: 

In light of our findings and conclusions in this report, we are making 
the following recommendations to OMB reiterating and expanding on
recommendations in our previous report: 

OMB should encourage agencies to discuss their plans for program
evaluations—especially those in response to an OMB recommendation—
with OMB and with congressional and other program stakeholders to
ensure that their findings will be timely, relevant, and credible and 
that they will be used to inform policy and management decisions. 

OMB should engage in dialogue with agencies and congressional 
stakeholders on a risk-based allocation of scarce evaluation resources
among programs, based on size, importance, or uncertain effectiveness,
and on the timing of such evaluations. 

OMB should continue to improve its PART guidance and training of
examiners on evaluation to acknowledge a wide range of appropriate
methods. 

Agency Comments: 

We provided a draft of this report to OMB and the agencies for review 
and comment. OMB agreed that evaluation methodology should be 
appropriate to the size and nature of the program and that randomized 
controlled trials may not be valuable in all settings. It noted its 
intent to provide additional guidance in this area. OMB disagreed with 
the reference to the PART as a checklist. This view was not ours but 
the view of agency officials who expressed concern about the focus of 
the assessment process. OMB also provided a number of technical 
comments, which we incorporated as appropriate throughout the report. 
OMB’s comments appear in appendix III. We also received technical 
comments from DOE, DOL, and HHS that we incorporated where appropriate 
throughout the report. SBA had no comments. 

We are sending copies of this report to the Director of the Office of
Management and Budget; the Secretaries of Energy, Labor, and Health and
Human Services; the Administrator of the Small Business Administration;
appropriate congressional committees; and other interested members of
Congress. We will also make copies available to others on request. In
addition, the report will be available at no charge on GAO’s Web site at
[hyperlink, http://www.gao.gov]. 

If you or your staff have questions about this report, please contact 
me at (202) 512-2700 or KingsburyN@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made key contributions to 
this report are listed in appendix IV. 

Sincerely, 

Singed by: 

Nancy Kingsbury: 
Managing Director: 
Applied Research and Methods: 

[End of section] 

Appendix I: Agency Programs OMB Recommended Evaluations For in PART 
Reviews: 

Agency: DOE; 
Program: Advanced Fuel Cycle Initiative; 
Program type: R&D; 
OMB recommendation: Establish plans for periodic independent 
evaluations to assess program progress and recommend program 
improvements. 

Agency: DOE; 
Program: Advanced Scientific Computing Research; 
Program type: R&D; 
OMB recommendation: Institute formal committee of visitors process by 
September 2003. 

Agency: DOE; 
Program: Generation IV Nuclear Energy Systems Initiative; 
Program type: R&D; 
OMB recommendation: Develop a plan for independent program evaluations 
to guide program managers and policy decision makers. 

Agency: DOE; 
Program: High Energy Physics; 
Program type: R&D; 
OMB recommendation: Institute formal committee of visitors process by 
September 2003. 

Agency: DOE; 
Program: Nuclear Energy Research Initiative; 
Program type: R&D; 
OMB recommendation: Will plan independent program evaluations to guide 
program management and development. 

Agency: DOE; 
Program: Nuclear Physics; 
Program type: R&D; 
OMB recommendation: Institute formal committee of visitors process by 
September 2003. 

Agency: DOE; 
Program: Weatherization Assistance; 
Program type: Block/formula grants; 
OMB recommendation: Recommends periodic independent evaluation of the 
program’s cost-effectiveness. 

Agency: DOL; 
Program: Employee Benefits Security Administration; 
Program type: Regulatory; 
OMB recommendation: Expand existing efforts for more comprehensive and 
regular program evaluation. 

Agency: DOL; 
Program: Federal Employees Compensation Act; 
Program type: Direct federal; 
OMB recommendation: An evaluation of strategic goals, the success of 
various program strategies, and state/industry best practices. 

Agency: DOL; 
Program: Occupational Safety and Health Administration; 
Program type: Regulatory; 
OMB recommendation: Develop a plan to evaluate the results and cost-
effectiveness of its regulatory and nonregulatory programs. 

Agency: DOL; 
Program: Office of Federal Contract Compliance Programs; 
Program type: Regulatory; 
OMB recommendation: Complete in 2003 an external evaluation and staff 
analysis to measure and improve program performance. 

Agency: DOL; 
Program: Youth Activities; 
Program type: Direct federal;
OMB recommendation: Plan and conduct an impact evaluation. 

Agency: HHS; 
Program: 317 Immunization Program; 
Program type: Competitive grants; 
OMB recommendation: Conduct a comprehensive evaluation of the structure,
management, and operations of the immunization program. 

Agency: HHS; 
Program: Indian Health Service Sanitation Facilities Construction 
Program; 
Program type: Capital assets; 
OMB recommendation: Conduct an independent, comprehensive evaluation of 
the program. 

Agency: HHS; 
Program: Nursing Education Loan Repayment and Scholarship Program; 
Program type: Competitive grants; 
OMB recommendation: Evaluate impact, develop outcome measures, and 
track performance. 

Agency: HHS; 
Program: Refugee and Entrant Assistance; 
Program type: Block/formula grants; 
OMB recommendation: The budget includes funds for ORR to conduct 
independent and quality evaluations. 

Agency: SBA; 
Program: Business Information Centers; 
Program type: Direct federal; 
OMB recommendation: Undertake an evaluation of the program’s 
effectiveness and measure whether it duplicates other federal and 
nonfederal mentoring programs. 

Agency: SBA; 
Program: SCORE;
Program type: Block/formula grant; 
OMB recommendation: Undertake an evaluation of the program’s 
effectiveness and measure whether it duplicates other federal and 
nonfederal mentoring programs. 

Agency: SBA; 
Program: Section 504 Certified Development Company Loan program; 
Program type: Credit; 
OMB recommendation: The 2004 budget proposes to increase program 
evaluations to determine the factors that affect both demand and 
performance in the 504 and 7(a) programs. 

Agency: SBA; 
Program: Small Business Development Centers; 
Program type: Block/formula grants; 
OMB recommendation: Undertake an evaluation of the program’s 
effectiveness and measure whether it duplicates other federal and 
nonfederal mentoring programs. 

Source: GAO analysis of the Budget of the United States Government, 
Fiscal Year 2004, Performance and Management Assessments (Washington, 
D.C.: 2003). 

Note: 
OMB = Office of Management and Budget; 
DOE = Department of Energy; 
R&D = Research and Development; 
DOL = Department of Labor; 
HHS = Department of Health and Human Services; 
ORR = Office of Refugee Resettlement; 
SBA = Small Business Administration. 

[End of section] 

Appendix II: Related Agency Program Evaluation Reports: 

Department of Energy Agency Reports: 

Advanced Fuel Cycle Initiative: Nuclear Energy Research Advisory
Committee (NERAC) Evaluation Subcommittee. Evaluation of DOE Nuclear 
Energy Programs. Washington, D.C.: Sept. 10, 2004. 

Advanced Scientific Computing Research Program: Advanced Scientific 
Computing Research. Committee of Visitors Report. Washington, D.C.: 
April 2004. 

Generation IV Nuclear Energy Systems Initiative: Nuclear Energy 
Research Advisory Committee (NERAC) Evaluation Subcommittee. Evaluation 
of DOE Nuclear Energy Programs. Washington, D.C.: Sept. 10, 2004. 

High Energy Physics Program: Committee of Visitors to the Office of
High Energy Physics. Report to the High Energy Physics Advisory Panel.
Washington, D.C.: Apr. 7, 2004. 

Nuclear Physics Program: Committee of Visitors. Report to the Nuclear
Science Advisory Committee. Washington, D.C.: Department of Energy, 
Office of Science, Feb. 27, 2004. 

Department of Health and Human Services Agency Reports: 

317 Immunization Program: RTI International. Section 317 Grant 
Immunization Program Evaluation: Findings from Phase I. Draft progress 
report. Atlanta, Ga.: Centers for Disease Control and Prevention, 
January 2005. 

Indian Health Service Sanitation Facilities Program: Department of
Health and Human Services, U.S. Public Health Service, Federal 
Occupational Health Service. Independent Evaluation Report Summary.
Prepared for Indian Health Service Sanitation Facilities Construction
Program, Rockville, Maryland. Seattle, Wash.: Mar. 8, 2005. 

Nursing Education Loan Repayment and Scholarship Program: Department of 
Health and Human Services, Health Resources and Services 
Administration, Bureau of Health Professions. HRSA Responds to the
Nursing Shortage: Results from the 2003 Nursing Scholarship Program
and the Nursing Education Loan Repayment Program: 2002–2003. First
report to the United States Congress. Rockville, Md.: n.d. 

Department of Labor Agency Reports: 

Employee Benefits Security Administration Reports: 

* Mathematica Policy Research, Inc. Case Opening and Results Analysis
(CORA) Fiscal Year 2002: Final Report. Washington, D.C.: Mar. 31,
2004. 

* Royal, Dawn. U.S. Department of Labor, Employee Benefits Security
Administration: Evaluation of EBSA Customer Service Programs 
Participant Assistance Program Customer Evaluation. Washington, D.C.: 
The Gallup Organization, February 2004. 

* Royal, Dawn. U.S. Department of Labor, Employee Benefits Security
Administration: Evaluation of EBSA Customer Service Programs 
Participant Assistance Mystery Shopper Evaluation. Washington, D.C.: 
The Gallup Organization, January 2004. 

* Royal, Dawn. U.S. Department of Labor, Employee Benefits Security
Administration: Evaluation of EBSA Customer Service Programs
Participant Assistance Outreach Programs Evaluation. Washington,
D.C.: The Gallup Organization, January 2004. 

* Royal, Dawn. U.S. Department of Labor, Employee Benefits Security
Administration: Evaluation of EBSA Customer Service Programs
Participant Assistance Web Site Evaluation. Washington, D.C.: The
Gallup Organization, January 2004. 

Federal Employees Compensation Act Program: ICF Consulting. Federal 
Employees Compensation Act (FECA): Program Effectiveness Study. 
Fairfax, Va.: U.S. Department of Labor, Office of Workers’ Compensation 
Programs, Mar. 31, 2004. 

Office of Federal Contract Compliance Programs: Westat. Evaluation of 
Office of Federal Contract Compliance Programs: Final Report. 
Rockville, Md.: December 2003. 

Occupational Safety and Health Administration Reports: 

* ERG. Evaluation of OSHA’s Impact on Workplace Injuries and Illnesses 
in Manufacturing Using Establishment–Specific Targeting of 
Interventions. Final report. Lexington, Mass.: July 23, 2004. 

* Marker, David and others. Evaluating OSHA’s National and Local 
Emphasis Programs. Draft Final Report for Quantitative Analysis of 
Emphasis Programs. Rockville, Md.: Westat, Dec. 24, 2003. 

* OSHA, Directorate of Evaluation and Analysis. Regulatory Review of
OSHA’s Presence Sensing Device Initiation (PSDI) Standard [29 CFR 
1910.217(h)]. Washington, D.C.: May 2004. [hyperlink, 
http://www.osha.gov/dcsp/compliance_assistance/lookback/psdi_final2004.h
tml] (Oct. 21, 2005). 

[End of section] 

Appendix III: Comments from the Office of Management and Budget: 

Executive Office Of The President: 
Office Of Management And Budget: 
Deputy Director For Management: 
Washington, D.C. 20503: 

October 17, 2005: 

Ms. Nancy Kingsbury: 
Managing Director: 
Applied Research and Methods: 
Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Kingsbury: 

Thank you for the opportunity to comment on the draft GAO report on 
program evaluation (Program Evaluation: OMB's PART Reviews Increased 
Agencies' Attention to Improving Evidence of Program Results, GAO-06-
67). 

We appreciate GAO's continued interest in the Program Assessment Rating 
Tool (PART) and our determination to use it to assess Federal programs 
in a consistent fashion. As is acknowledged in your report, the 
President's Management Agenda and the PART have raised awareness of the 
importance of program results.[Footnote 19] Indeed, we believe that 
without measurement, there is no certainty about how Federal programs 
are performing for the American people. 

Oftentimes, Federal programs proceed to carry out their business based 
on the assumption that their existing strategies and approaches are 
effective. High quality, independent evaluation may validate or 
disprove that assumption. Your draft report suggests that randomized 
controlled trials may not be valuable in every setting. OMB agrees and 
recognizes that the size and nature of the program should be considered 
when determining the appropriate evaluation methodology. We have tried 
to make this point clear and will provide additional guidance in this 
area. 

The report suggests that the PART requires too high a standard for 
evaluations and discards other useful evaluations. The evaluation 
question in Section II of the PART, the primary subject of this report, 
focuses on evaluations of program impact which must rely on rigorous 
methodologies to be credible; however, OMB's guidance makes clear that 
the methodology should be appropriate for the program. It is also 
important to note that other evaluations, such as those assessing 
processes or smaller aspects of a program, provide valuable information 
and are regularly used as evidence for other questions in the PART, 
particularly in Section III. 

The report also includes a few references to the PART as an OMB 
"checklist "[Footnote 20] Sections I — III of the PART essentially are 
a checklist — one to ensure that the program has key elements in place 
which are generally understood to make programs effective, such as a 
clear, focused mission; thoughtful strategic planning; and strong 
management practices. Section IV, on the other hand, measures whether 
these conditions and practices have produced results. In addition, 
after each PART assessment, programs commit to an improvement plan, key 
follow-up actions they will take to achieve performance results. PART 
is more than just a checklist. 

The particular interest that GAO has taken in the Administration's 
standards for measuring performance is noteworthy. Thank you for your 
continued enthusiasm about program evaluation and the PART, as well as 
for your willingness to take our oral and written comments into 
consideration in the final draft. I look forward to working with you to 
improve the ways in which we are making the Federal Government more 
results-oriented. 

Sincerely, 

Signed by: 

Clay Johnson III: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Nancy Kingsbury (202) 512-2700 or KingsburyN@gao.gov: 

Acknowledgments: 

In addition to the contact named above, Stephanie Shipman, Assistant
Director, and Valerie Caracelli made significant contributions to this
report. Denise Fantone and Jacqueline Nowicki also made key
contributions. 

[End of section] 

Related GAO Products: 

Performance Budgeting: PART Focuses Attention on Program Performance, 
but More Can Be Done to Engage Congress. GAO-06-28. Washington, D.C.: 
Oct. 28, 2005. 

Managing for Results: Enhancing Agency Use of Performance Information 
for Managerial Decision Making. GAO-05-927. Washington, D.C.: Sept. 9, 
2005. 

21st Century Challenges: Performance Budgeting Could Help Promote
Necessary Reexamination. GAO-05-709T. Washington, D.C.: June 14, 2005.
Performance Measurement and Evaluation: Definitions and Relationships. 
GAO-05-739SP. Washington, D.C.: May 2005. 

Results-Oriented Government: GPRA Has Established a Solid Foundation 
for Achieving Greater Results. GAO-04-38. Washington, D.C.: Mar. 10, 
2004. 

Performance Budgeting: Observations on the Use of OMB’s Program
Assessment Rating Tool for the Fiscal Year 2004 Budget. GAO-04-174.
Washington, D.C.: Jan. 30, 2004. 

Program Evaluation: An Evaluation Culture and Collaborative 
Partnerships Help Build Agency Capacity. GAO-03-454. Washington,
D.C.: May 2, 2003. 

Program Evaluation: Strategies for Assessing How Information 
Dissemination Contributes to Agency Goals. GAO-02-923. Washington,
D.C.: Sept. 30, 2002. 

Program Evaluation: Studies Helped Agencies Measure or Explain Program 
Performance. GAO/GGD-00-204. Washington, D.C.: Sept. 29, 2000. 

Performance Plans: Selected Approaches for Verification and Validation
of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.:
July 30, 1999. 

Managing for Results: Measuring Program Results That Are Under Limited 
Federal Control. GAO/GGD-99-16. Washington, D.C.: Dec. 11, 1998. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 103-62 (1993). 

[2] The agenda’s four other priorities are strategic management of 
human capital, expanded electronic government, improved financial 
performance, and competitive sourcing. See [hyperlink, 
http://www.whitehouse.gov/omb/budintegration/pma_index.html] (Oct. 21, 
2005). 

[3] GAO, Performance Budgeting: Observations on the Use of OMB’s 
Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04-
174 (Washington, D.C.: Jan. 30, 2004). 

[4] “Program” has no standard definition. For purposes of PART, OMB 
described program, its unit of analysis, as an activity or set of 
activities (1) clearly recognized as a program by the
public, OMB, or Congress; (2) having a discrete level of funding 
clearly associated with it; and (3) corresponding to the level at which 
budget decisions are made. 

[5] In subsequent PART reviews, OMB encouraged agencies to propose 
recommendations, which they refer to as “recommended follow-up actions” 
in the fiscal year 2006 PART summaries. 

[6] GAO-04-174, pp.12-13. 

[7] See GAO, Performance Measurement and Evaluation: Definitions and 
Relationships, GAO-05-739SP (Washington, D.C.: May 2005). 

[8] GAO-04-174. 

[9] GAO, Performance Budgeting: PART Focuses Attention on Program 
Performance, but More Can Be Done to Engage Congress, GAO-06-28 
(Washington, D.C.: Oct. 28, 2005). 

[10] Pub. L. No. 107-205 (2002). 

[11] The Nursing Education Loan Repayment Program offers registered 
nurses financial assistance to repay educational loans in exchange for 
service in a critical shortage facility. Participants contract to work 
full-time in a critical shortage facility. For 2 years of service,
the program pays up to 60 percent of the total qualifying loan balance. 
For the Nursing Scholarship Program, participants incur a year of full-
time obligated service for each full or partial year of support, with a 
minimum of a 2-year service obligation of full-time clinical service at 
a health facility with a critical shortage of nurses. 

[12] The Regulatory Flexibility Act, 5 U.S.C. § 610, and Executive 
Order No. 12866, Regulatory Planning and Review, Sept. 30, 1993, 58 
Fed. Reg. 51735 (Oct. 4, 1993), require certain regulatory agencies to 
conduct such periodic reviews of their rules. 

[13] 29 C.F.R. §§ 1910.1047, 1910.217 (2005). 

[14] GAO, Results-Oriented Government: GPRA Has Established a Solid 
Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.: 
Mar. 10, 2004), p. 88 noted these previously reported challenges: 
developing outcome-oriented measures, isolating the impact of a 
program, and obtaining timely, useful performance data. 

[15] DOE Office of Science also has a leading role in an international, 
informal professional organization—the Washington Research Evaluation 
Network, at [hyperlink, http://www.wren-network.net/]—exploring 
evaluation approaches for improving the management of public science 
and technology programs (Oct. 21, 2005). 

[16] For further discussion see Peter H. Rossi, Howard E. Freeman, and 
Mark W. Lipsey, Evaluation: A Systematic Approach, 6th ed. (Thousand 
Oaks, Calif.: Sage Publications, 1999). For additional examples of 
alternative evaluation designs, see GAO, Program Evaluation: Strategies 
for Assessing How Information Dissemination Contributes to Agency 
Goals, GAO-02-923 (Washington, D.C.: Sept. 30, 2002). 

[17] The entire evaluation dialogue presentation is at [hyperlink, 
http://www.epa.gov/evaluate/part.htm] (Oct. 21, 2005). 

[18] Office of Management and Budget, Guidance for Completing the 
Program Assessment Rating Tool (PART). (Washington, D.C.: March 2005) 
is at [hyperlink, http://www.whitehouse.gov/omb/part] (Oct. 21, 2005). 

[19] See, draft report "Management Attention, Caught by OMB's 
Recommendations, Overcame Measurement and Funding Barriers," p. 12. 

[20] See, for example, the report "Highlights." 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: