This is the accessible text file for GAO report number GAO-11-176 
entitled 'Program Evaluation: Experienced Agencies Follow a Similar 
Model for Prioritizing Research' which was released on January 14, 
2011. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to the Subcommittee on Oversight of Government Management, the 
Federal Workforce, and the District of Columbia, Committee on Homeland 
Security and Governmental Affairs, U.S. Senate: 

January 2011: 

Program Evaluation: 

Experienced Agencies Follow a Similar Model for Prioritizing Research: 

GAO-11-176: 

GAO Highlights: 

Highlights of GAO-11-176, a report to the Subcommittee on Oversight of 
Government Management, the Federal Workforce, and the District of 
Columbia, Committee on Homeland Security and Governmental Affairs, 
U.S. Senate. 

Why GAO Did This Study: 

Amid efforts to improve performance and constrain spending, federal 
agencies are being asked to expand the use of rigorous program 
evaluation in decision-making. In addition to performance data, 
indepth program evaluation studies are often needed for assessing 
program impact or designing improvements. Agencies can also use their 
evaluation resources to provide information needed for effective 
management and legislative oversight. 

GAO was asked to study federal agencies with mature evaluation 
capacity to examine (1) the criteria, policies, and procedures they 
use to determine programs to review, and (2) the influences on their 
choices. GAO reviewed agency materials and interviewed officials on 
evaluation planning in four agencies in three departments with 
extensive evaluation experience: Education, Health and Human Services 
(HHS), and Housing and Urban Development (HUD). 

HHS and HUD agreed with the description of how they plan evaluations. 
HHS noted that the optimal location of evaluation units will vary with 
the circumstances and purpose of evaluations. HUD felt the draft 
report did not emphasize enough the influence of the appropriations 
process. GAO has added text to note its influence on evaluation 
planning. Education provided technical comments. 

What GAO Found: 

Although no agency GAO reviewed had a formal policy describing 
evaluation planning, all followed a generally similar model for 
developing and selecting evaluation proposals. Agencies usually 
planned an evaluation agenda over several months in the context of 
preparing spending plans for the coming fiscal year. Evaluation staff 
typically began by consulting with a variety of stakeholders to 
identify policy priorities and program concerns. Then with program 
office staff they identified the key questions and concerns and 
developed initial proposals. Generally, the agencies reviewed and 
selected proposals in two steps: develop ideas to obtain initial 
feedback from senior officials and develop full-scale evaluation 
proposals for review and approval. 

The four general criteria these mature agencies use to plan 
evaluations were remarkably similar: (1) strategic priorities 
representing major program or policy area concerns or new initiatives, 
(2) program-level problems or opportunities, (3) critical unanswered 
questions or evidence gaps, and (4) the feasibility of conducting a 
valid study. 

The agencies’ procedures differed on some points. External parties’ 
participation in evaluation planning may reflect these agencies’ 
common reliance on nonfederal program partners. Only the offices GAO 
reviewed in HHS’ Centers for Disease Control and Prevention held 
formal competitions to rank-order proposals before submitting them for 
approval; in other agencies, senior officials assessed proposals in a 
series of discussions. When evaluation authority and funds are tied to 
the program, evaluators generally choose not which programs to 
evaluate but which research questions to answer. Sometimes this 
resulted in a program’s never being evaluated. 

Evaluation units at higher organizational levels conducted a wider 
range of analytic activities, consulted more formally with program 
offices, and had less control over approvals. The Congress influences 
an agency’s program evaluation choices through legislating evaluation 
authority, mandating studies, making appropriations, and conducting 
oversight. 

GAO concludes that: 

* all four criteria appear key to setting an effective evaluation 
agenda that provides credible, timely answers to important questions; 

* most agencies could probably apply the general model in which 
professional evaluators iteratively identify key questions in 
consultation with stakeholders and then scrutinize and vet research 
proposals; 

* agencies could adapt the model and decide where to locate evaluation 
units to meet their own organizational and financial circumstances and 
authorities; and; 

* agencies’ reaching out to key program and congressional stakeholders 
in advance of developing proposals could help ensure that their 
evaluations will be used effectively in management and legislative 
oversight. 

What GAO Recommends: 

GAO makes no recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-11-176] or key 
components. For more information, contact Nancy Kingsbury, Managing 
Director, (202) 512-2700 or kingsburyn@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

The Agencies' Generally Similar Informal Evaluation Planning Policies: 

Prioritization Depends on Funding Sources and Agency Circumstances: 

Concluding Observations: 

Agency Comments: 

Appendix I: Comments from the Department of Health and Human Services: 

Appendix II: Comments from the Department of Housing and Urban 
Development: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Bibliography: 

Related GAO Products: 

Figure: 

Figure 1: General Evaluation Planning Process and Key Differences 
across Agencies: 

Abbreviations: 

ACF: Administration for Children and Families: 

ASPE: Office of the Assistant Secretary for Planning and Evaluation: 

CDC: Centers for Disease Control and Prevention: 

DASH: Division of Adolescent and School Health: 

DHAP: Division of HIV/AIDS Prevention: 

DNPAO: Division of Nutrition, Physical Activity, and Obesity: 

GPRA: Government Performance and Results Act: 

HHS: Department of Health and Human Services: 

HUD: Department of Housing and Urban Development: 

IES: Institute of Education Sciences: 

NCEE: National Center for Education Evaluation and Regional Assistance: 

NRC: National Research Council: 

OMB: Office of Management and Budget: 

OPEPD: Office of Planning, Evaluation, and Policy Development: 

OPRE: Office of Planning, Research and Evaluation: 

PART: Program Assessment Rating Tool: 

PD&R: Office of Policy Development and Research: 

PHS: Public Health Service: 

PPSS: Policy and Program Studies Service: 

TANF: Temporary Assistance to Needy Families: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

January 14, 2011: 

The Honorable Daniel K. Akaka: 
United States Senate: 

Dear Senator Akaka: 

Amid efforts to improve federal government performance and constrain 
federal spending, the Obama Administration has emphasized expanding 
the availability and use of rigorous program evaluation. The aim is to 
establish an evaluation culture in which agencies regularly use 
evidence to inform program design and investment decisions, 
complementing existing efforts to strengthen performance measurement 
and management. Today, most federal agencies use performance measures 
to track progress toward goals. Few appear to conduct in-depth program 
evaluations regularly to assess their programs' impact or inform 
policymakers on how to improve results. Increasing demands for 
performance information prompted your expressed interest in how 
agencies, given their limited evaluation resources, can provide the 
information that is necessary for effective management and legislative 
oversight. To help agencies plan their evaluation activities 
strategically, you asked us to study federal agencies that have mature 
program evaluation capacity, to learn: 

1. What criteria, policies, and procedures do they use to determine 
which programs to review? 

2. What conditions influence an agency's choices? 

To find answers that would apply broadly to other federal agencies 
about how to prioritize program evaluations, we selected four agencies 
in three departments with known evaluation capacity--the ability to 
systematically collect, analyze, and use data on program results--in a 
variety of substantive areas. The four agencies were the Department of 
Education and the Department of Housing and Urban Development (HUD) 
and two agencies in the Department of Health and Human Services (HHS)--
the Administration for Children and Families (ACF) and the Centers for 
Disease Control and Prevention (CDC). 

To identify agencies fulfilling our criteria, we reviewed previous GAO 
reports and agency documents for evidence of emphasis on conducting 
evaluations. For example, we searched for examples of agencies' 
incorporating the results of program evaluations in their annual 
performance reports. To obtain a diverse set of cases, we selected 
agencies that addressed a variety of content areas and program types-- 
direct services or grants, regulation, and research--and that varied 
in the conditions likely to affect their decision process, such as 
whether they had a central office responsible for evaluation. To 
identify evaluation planning procedures and criteria, we reviewed 
agency materials and interviewed agency evaluation officials about 
their evaluation planning process. We identified conditions 
influencing their choices in interviews with agency officials and in 
comparisons of what we found across the agencies. 

We conducted this performance audit from May 2010 through December 
2010 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. We 
requested comments on a draft of this report from the Secretaries of 
Education, Health and Human Services, and Housing and Urban 
Development. HHS and HUD provided comments that are reprinted in 
appendixes I and II and described at the end of this letter. Education 
provided technical comments. 

Background: 

Under the Government Performance and Results Act of 1993 (GPRA) 
federal agencies are expected to focus on achieving results and to 
demonstrate, in annual performance reports and budget requests, how 
their activities help achieve agency or governmentwide goals. In 2002, 
to encourage greater use of program performance information in 
decision making, the Office of Management and Budget (OMB) created the 
Program Assessment Rating Tool (PART). PART was intended to provide a 
consistent approach for evaluating federal programs within the 
executive budget formulation process. However, because PART 
conclusions rely on available program performance and evaluation 
information, many of the initial recommendations focused on improving 
outcome and efficiency measures. Although GPRA and PART helped improve 
the availability of better performance measures, we and OMB have noted 
that this did not result in their greater use by the Congress or 
agencies.[Footnote 1] 

In October 2009, OMB announced a plan to strengthen federal program 
evaluation, noting that rigorous independent program evaluations can 
help determine whether government programs are achieving their 
intended outcomes as well as possible and at the lowest possible cost. 
[Footnote 2] Program evaluations are systematic studies that assess 
how well a program is working, and they are individually tailored to 
address the client's research question. 

* Process (or implementation) evaluations assess the extent to which a 
program is operating as intended. 

* Outcome evaluations assess the extent to which a program is 
achieving its outcome-oriented objectives but may also examine program 
processes to understand how outcomes are produced. 

When external factors such as economic or environmental conditions are 
known to influence a program's outcomes, 

* Impact evaluations may be used to measure a program's net effect by 
comparing outcomes with an estimate of what would have occurred had 
there been no program intervention. 

Thus, program evaluation can provide an important complement to agency 
performance data that simply track progress toward goals. 

In announcing the evaluation initiative, the OMB Director expressed 
concern that many important programs had never been evaluated, 
evaluations had not sufficiently shaped budget priorities or 
management practices, and many agencies lack an evaluation office 
capable of supporting an ambitious strategic research agenda. The 
initiative consisted of three efforts: posting information online on 
all agencies' planned and ongoing impact evaluations, establishing an 
interagency group to promote the sharing of evaluation expertise, and 
funding some new agency rigorous impact evaluations and capacity 
strengthening efforts. As part of the fiscal year 2011 budget process, 
OMB allocated approximately $100 million for the evaluation initiative 
to support 35 rigorous program evaluations and evaluation capacity-
building proposals. 

OMB made a similar evaluation solicitation for the fiscal year 2012 
budget in which nonsecurity agencies were asked to reduce their 
discretionary budgets by 5 percent. The budget process evaluation 
initiative is focused on impact evaluations and is not intended to 
cover the full range of an agency's evaluation activities. However, to 
be considered for additional evaluation funding, agencies must 
demonstrate that they are both using existing evaluation resources 
effectively and beginning to integrate evaluation into program 
planning and implementation. With significant efforts under way to 
increase agencies' evaluation resources, it is especially timely now 
to learn how agencies with more evaluation experience prioritize their 
resources. 

A recent GAO review identified three elements that leading national 
research organizations consider essential to a sound federal research 
and evaluation program: research independence, transparency and 
accountability, and policy relevance.[Footnote 3] These elements align 
well with OMB's new evaluation initiative and expectations for a 
better integration of evaluation into program design and management. 
In this report, we do not assess the quality of the agencies' research 
agendas or their achievement of these objectives. However, we do 
describe practices these agencies took that were designed to achieve 
those elements. 

Evaluation at Education: 

The Department of Education establishes policy for, administers, and 
coordinates most federal assistance to elementary, secondary, and 
postsecondary education. The department has supported educational 
research, evaluation, and dissemination not only since the Congress 
created it in 1979 but also earlier, when it was the Office of 
Education. For several years, two central offices in the Department of 
Education have been responsible for program and policy evaluation. The 
Policy and Program Studies Service (PPSS), in the Office of Planning, 
Evaluation, and Policy Development (OPEPD), advises the Secretary on 
policy development and review, strategic planning, performance 
measurement, and evaluation. The Institute of Education Sciences 
(IES), established in 2002 (replacing the Office of Educational 
Research and Improvement), is the research arm of the department. IES 
is charged with producing rigorous evidence on which to ground 
education practice and policy, with program evaluation housed 
primarily in the National Center for Education Evaluation and Regional 
Assistance (NCEE). 

In 2009, the Department of Education launched a review of its 
evaluation activities, comparing them to those of other government 
agencies, seeking to build analytic capacity, and intending to use 
available knowledge and evidence more effectively. This review 
resulted in a comprehensive, departmentwide evaluation planning 
process and clarified the distinct evaluation responsibilities of the 
two offices. Starting in 2010, OPEPD was to lead the planning process, 
in partnership with IES, to identify the department's key priorities 
for evaluation and related knowledge-building activities. Starting in 
fiscal year 2011, NCEE in IES will be responsible for longer-term (18 
months or longer) program implementation and impact studies, while 
PPSS in OPEPD will focus on shorter-term evaluation activities (fewer 
than 18 months), policy analysis, performance measurement, and 
knowledge management activities. Some program offices also conduct 
evaluation activities separate from studies conducted by either of the 
central offices, such as supporting grantee evaluations or analyzing 
grantee performance data for smaller programs where larger-scale 
evaluations are not practical. 

Evaluation at HUD: 

The Department of Housing and Urban Development is the principal 
federal agency responsible for programs on housing needs, fair housing 
opportunities, and community improvement and development. It insures 
home mortgages, subsidizes housing for low-and moderate-income 
families, promotes and enforces fair housing and equal opportunity 
housing, and provides grants to states and communities to aid 
community development. At HUD, program evaluation is primarily 
centralized in one office--the Office of Policy Development and 
Research (PD&R)--created in 1973. It conducts a mix of surveys, 
independent research, demonstrations, policy analyses, and short-and 
long-term evaluations that inform HUD's decisions on policies, 
programs, and budget and legislative proposals. PD&R provides HUD's 
program offices with technical support, data, and materials relevant 
to their programs. Although the primary responsibility for evaluating 
programs falls to PD&R, some evaluation is found in program offices, 
such as the Office of Housing, which routinely conducts analyses to 
update its loan performance models for assessing credit risk and the 
value of its loan portfolio. 

In 2006, the Congress, concerned about the quality of HUD research, 
commissioned the National Research Council (NRC) to evaluate the PD&R 
office and provide recommendations regarding the course of future HUD 
research. A 2008 NRC report noted declining resources for data 
collection and research and insufficient external input to its 
research agenda.[Footnote 4] On the heels of the report, the scope of 
the current economic and housing crisis led the incoming 
administration to acknowledge a need both to reform and transform HUD 
and to sustain a commitment of flexible budget resources for these 
efforts. In 2009, HUD proposed a departmentwide Transformation 
Initiative of organizational and program management improvements to 
position HUD as a high-performing organization. In fiscal year 2010, 
much of PD&R's research and evaluation activities are funded through a 
set-aside created for the initiative, which also supports program 
measures, demonstrations, technical assistance, and information 
technology projects. 

Evaluation at HHS: 

Evaluation planning is decentralized at the Department of Health and 
Human Services. We reviewed ACF and CDC because they have significant 
evaluation experience. HHS's centrally located Office of the Assistant 
Secretary for Planning and Evaluation (ASPE) coordinates agency 
evaluation activities, reports to the Congress on the department's 
evaluations, and conducts studies on broad, cross-cutting issues while 
relying on agencies to evaluate their own programs.[Footnote 5] In 
some cases, ASPE conducts independent evaluations of programs housed 
within other HHS operating and staff divisions (for example, ACF and 
CDC). 

ACF oversees and helps finance programs to improve the economic and 
social well-being of families, individuals, and communities--the Head 
Start program is an example. It also assists state programs for child 
support enforcement as well as Temporary Assistance to Needy Families 
(TANF). The Office of Planning, Research, and Evaluation (OPRE) is the 
principal office for managing evaluation at ACF. It also provides 
guidance, analysis, technical assistance, and oversight related to 
strategic planning, performance measurement, research, and evaluation 
methods. It conducts statistical, policy, and program analyses and 
synthesizes and disseminates research and demonstration findings. OPRE 
consults with outside groups on ideas that feed into program and 
evaluation planning. In each policy area with substantial evaluation 
resources, OPRE consults with a group of researchers, program 
partners, and other content area experts who share their knowledge and 
ideas for research and evaluation. 

CDC, as part of the Public Health Service, is charged with protecting 
the public health by developing and providing to persons and 
communities information and tools for preventing and controlling 
disease, promoting health, and preparing for new health threats. It 
supports some evaluation activities through the Public Health Service 
(PHS) evaluation set-aside; in 2010 the Secretary was authorized to 
use up to 2.5 percent of appropriations for evaluating programs funded 
under the PHS Act. The set-aside is also used to fund databases of the 
National Center for Health Statistics and programs that cut across 
CDC's divisions. Presently, the divisions within CDC control most 
evaluation funding focused on their respective programs, but 
evaluation planning across CDC is currently under review. CDC recently 
created an Office of the Associate Director for Program which will 
have responsibility for supporting performance measurement and 
evaluation across CDC, among other duties. 

We interviewed staff from evaluation offices in three CDC divisions: 
Nutrition, Physical Activity, and Obesity (DNPAO); HIV/AIDS Prevention 
(DHAP); and Adolescent and School Health (DASH). These three divisions 
oversee cooperative agreements with state and local agencies and plan 
a portfolio of evaluations. CDC officials suggested that variation in 
evaluation planning in these three offices could provide insight into 
how CDC's centers generally prioritize evaluations to conduct. 

DNPAO is charged with leading strategic public health efforts to 
prevent and control obesity, chronic disease, and other health 
conditions through physical activity and healthy eating. DNPAO 
supports the First Lady's Lets Move! campaign to curb childhood 
obesity, which is considered an important public health issue but has 
a limited body of research on effective practices. The Nutrition, 
Physical Activity, and Obesity Program is a cooperative agreement 
between CDC and 25 state health departments to support a range of 
activities, including process and outcome evaluations. A consulting 
group of state evaluators, outside experts, and divisional 
representation advises DNPAO on proposing evaluation projects that 
would be useful to grantees and the divisions. 

DHAP, charged with leadership in helping control the HIV/AIDS 
epidemic, has a fairly large program evaluation branch that supports 
national performance monitoring and evaluation planning. The 
evaluation branch is responsible for monitoring CDC-funded HIV 
prevention programs, including 65 health units and 150 community 
organizations. Within the branch, the Evaluation Studies Team conducts 
specific evaluations of interest and in-depth process evaluations and 
outcome monitoring studies of selected HIV prevention interventions 
delivered by community-based organizations, state and local health 
departments, and health-care providers. In addition to the Division's 
strategic plan, the governmentwide National HIV/AIDS Strategy for the 
United States, released in July 2010, informs evaluation planning. 
DHAP's work is also shaped by an advisory committee and findings from 
an external peer review that provided input into programs and 
evaluations through the strategic plan.[Footnote 6] 

DASH is considered somewhat unique among CDC's divisions because it is 
not focused on disease or exposure but has a mission to promote the 
health and well-being of a particular population--children and 
adolescents. DASH funds tribal governments and state, territorial and 
local educational agencies to address child and adolescent health 
issues, including nutrition, risky sexual behavior, tobacco 
prevention, school infrastructure, and asthma management. DASH 
typically funds evaluations in one health risk area each year. Its 
framework, Coordinated School Health, involves community, family, 
teachers, and schools in addressing a diverse set of health issues. It 
also partners with nongovernmental and community-based organizations 
to reach children who are not in school. DASH supports rapid 
evaluations to identify innovative programs and practices. These 
evaluations typically last 12 to 24 months and data are collected 
within a school calendar year. The evaluation team also has a small 
portfolio of evaluation research that includes large longitudinal 
randomized controlled trials that assess effectiveness over a 5-to-6-
year period. 

The Agencies' Generally Similar Informal Evaluation Planning Policies: 

The agencies we reviewed use a similar but informal evaluation 
planning process that involves collaboration between each agency's 
evaluation office and program offices, external groups, and senior 
officials. Typically, the evaluation office leads an iterative two-
step process to develop ideas into full-scale proposals by obtaining 
feedback from senior officials and considering available resources. 
The process varies across agencies in the breadth of the studies and 
programs considered, the use of ranked competitions, and the amount of 
oversight by senior officials. Figure 1 depicts the general process 
and the agencies' significant differences. 

Figure 1: General Evaluation Planning Process and Key Differences 
across Agencies: 

[Refer to PDF for image: illustration] 

Start of Fiscal Year (October) to End of Fiscal Year (September): 

1. Evaluation staff begins process. 

2. Develop concepts: Some agencies use same process for evaluations 
and other research; 
* Program staff contributes ideas; 
* External Advisory Groups; 
* Public solicitations for projects; 
Considerations: 
* Identify policy and program management questions; 
* Vet ideas to intended users; 
* Scrutinize proposals within available data and resources. 

[Feedback within agencies occurs at step 2 and 3] 

3. Seek feedback within agency. 

4. Develop full-scale proposals. 

5. Review proposals: 
* Some agencies hold competitions and rank proposals; 
* Some agencies consult external stakeholders; 
* Consult Internal stakeholders. 

6. Send finalists to senior agency officials. 

7. Approval. 

Source: GAO. 

[End of figure] 

The General Process for Developing and Selecting Evaluation Proposals: 

In most of the agencies we reviewed, evaluation planning generally 
starts and ends in the same fiscal year. General procedures for 
submitting and clearing annual spending plans structure the evaluation 
planning process at several of these agencies because the approved 
evaluations may involve external expenditures. The agencies must 
approve their evaluation plans by the start of the next fiscal year, 
or when appropriated funds become available, so that they can issue 
requests for proposals from the contractors that conduct the 
evaluations. Planning evaluations can include reviews by policy 
officials, such as deputy and assistant secretaries, and budget 
officials, such as an agency's chief financial officer. For example, 
ACF's evaluation staff develop evaluation proposals in the fall and 
early winter, before sending them to the agency's assistant secretary 
for approval in the late winter to allow the assistant secretary to 
make approval decisions in time to meet the deadlines for awarding 
contracts in that fiscal year. Although most of the agencies finish 
their planning by the start of the next fiscal year, the process can 
start as late as July at CDC's DNPAO or in the fall of the current 
fiscal year at ACF. 

Planning begins at each agency with internal coordination to define 
the goals and procedures for developing evaluation proposals. At ACF, 
this process begins informally, with evaluation and program staff 
meeting to discuss their priorities for the coming year. The other 
agencies we reviewed (including Education beginning in 2010) issue 
memorandums describing the planning process to the staff members 
involved. They may describe the staff members who will lead proposal-
development teams, the substantive focus of the year's process, the 
evaluation plan's connection to spending plans, and the role of senior 
officials. They may also give a schedule of key deadlines. CDC's DASH 
distributes a broader call for project nominations to agency staff 
members and researchers, state and local education agencies, and other 
program partners. In recent years, the call has specified the type of 
interventions the division seeks to evaluate, stated deadlines for 
submitting nominations, and solicited information from nominators 
about particular interventions. CDC's DNPAO issues a call for 
proposals that addresses the process and broad criteria for project 
selections that can involve many people and proposals. The calls at 
each agency are informal planning documents, however, as no agency we 
reviewed has an official policy that specifies the process for 
developing and selecting evaluations. Having developed informal 
processes over time, senior officials, and evaluation and program 
office staff have a common understanding of how they will develop, 
review, and select evaluations. 

After the agencies identify their planning goals and steps, the 
evaluation and program staff begin to develop evaluation proposals. At 
some agencies, the program staff may develop the initial proposals 
independently of the evaluation staff, in response to the same call 
for proposals. The program staff may later consult with the evaluation 
staff to improve the proposals before they are reviewed further. This 
process is common in the CDC divisions we reviewed, where the 
evaluation staff are located inside program offices dedicated to 
particular health issues, so both program and evaluation staff may 
individually or jointly submit proposals for consideration. 

At other agencies, the evaluation staff meet with the program staff 
specifically to discuss ideas for evaluation and then develop initial 
proposals from the input they receive. The evaluation staff at one of 
these agencies said they incorporate the priorities, questions, and 
concerns the program staff conveyed from their day-to-day experience 
into evaluation planning and that collaboration helps ensure later 
support for the approved evaluations. Alternatively, HUD's evaluation 
unit includes program staff on the teams that develop proposals in 
specific policy areas, such as fair housing and homelessness. The 
program offices also contribute to the initial proposals by providing 
comments to senior officials. At all the agencies, the evaluation 
staff use their expertise in designing social research and assessing 
the reliability of data, among other skills, to ensure the quality and 
feasibility of proposals. 

In addition to consulting internal program staff, most of the agencies 
we examined consult external groups to obtain ideas for evaluation 
proposals. Evaluation staff members cited a number of reasons for 
consulting external groups in developing proposals: the ability to 
identify unanswered research questions, coordinate evaluations across 
federal agencies, uncover promising programs or practices to evaluate, 
and inform strategic goals and priorities. Some evaluation staff 
reported consulting external groups as they develop program priorities 
and strategic plans, which they cited as criteria for planning 
evaluations. Over the past 2 years, PD&R has participated in a 
philanthropic foundation-funded partnership with research 
organizations that conducted several research projects to help inform 
the Department's development of an evidence-based housing and urban 
policy agenda. Other staff said that they consult with state and local 
program partners, such as state welfare offices, to identify 
potentially useful projects. 

External groups have formal roles in developing proposals at two 
agencies. CDC's DASH directly consults with external researchers, 
state and local education officials, and school health professionals 
for nominations of promising interventions to evaluate. In planning 
for fiscal year 2011, HUD asked the public to submit ideas for 
evaluation on its "HUD User" Web site.[Footnote 7] At most of the 
agencies, however, external groups do not explicitly develop 
evaluation proposals. For example, ACF staff said they informally 
consult with researchers about possible evaluation topics, partly in 
regular research conferences, but they do not ask their advisory 
panels or individual researchers to review specific evaluation 
proposals. In recent years, PD&R also contacted the office of HUD's 
Inspector General for evaluation ideas that build on that office's 
work. 

Generally, the agencies review and approve evaluation proposals in two 
steps. First, evaluation or program staff members develop ideas or 
brief concept papers for initial feedback from senior officials. The 
feedback can involve a series of proposal development meetings, as at 
Education and ACF, where senior officials give staff members strategic 
direction to develop and refine their proposals. Alternatively, senior 
officials may review all draft proposals that the evaluation and 
program staff have developed independently, as at HUD and CDC's DNPAO 
and DHAP. Initial feedback helps prevent staff from investing large 
amounts of time in proposals that would have a small chance of being 
approved. The feedback expands proposal development beyond the 
evaluation and program offices and helps ensure that the proposals 
support the agency's broader strategic goals. 

Second, once the initial proposals are sufficiently well developed, 
senior officials review and select from a group of revised, full-scale 
proposals. These may contain detailed information about cost and 
design for later use in the contracting process. Evaluation officials 
at ACF and HUD select from the pool of revised proposals those they 
wish to present to agency leaders, such as the secretary or assistant 
secretary, for final approval. Branch leaders at CDC's DNPAO and DHAP 
choose a group of proposals to compete for division resources against 
proposals from other branches within their divisions. Review panels 
rank-order all proposals (discussed below), and then division leaders 
decide, based on the rankings and available resources, which proposals 
the division will fund. In fiscal year 2011, Education staff plan to 
present to senior officials the entire proposed evaluation portfolio, 
identifying how evaluation studies address key questions and agency 
priorities. 

The Significant Differences in Agency Processes: 

Some of the agencies we reviewed focus specifically on planning 
program evaluations, while others use the same annual process to plan 
a variety of analytic studies. The central evaluation offices at ACF, 
Education, and HUD perform a continuum of research and evaluation 
studies that may include collecting national survey data, conducting 
policy analyses, and describing local program activities, among other 
activities. These agencies use the same process to make funding 
decisions across these various analysis proposals, which allows them 
to weigh the pros and cons of evaluation against other information 
needs when sufficient funds are available. Consequently, program 
evaluations may compete with other types of studies that require 
specific funding each year. 

Although the evaluation branch of CDC's DNPAO provides a narrower 
range of services, the division uses a similar, unified process to 
decide how to develop proposals for all evaluations and research 
activities. In contrast, DASH plans their different types of studies 
separately. It uses one annual process to develop evaluation proposals 
for promising practices and interventions often implemented by 
grantees. It uses a different process to develop "evaluation research" 
proposals, which evaluation staff defined as national-level 
evaluations or long-term studies of program impact, often involving 
randomized controlled trials. By considering these types of study 
separately, DASH does not require longer-term evaluations to compete 
with shorter-term studies for the same funds. 

Programs compete against one another for evaluation resources at some 
but not most of the agencies we reviewed. The scope of evaluation 
planning at one group of agencies is limited to the same programs or 
policy areas each year. These agencies have designed their planning 
processes to select not programs to evaluate but evaluation questions 
to answer in a program area. For example, the ACF evaluation staff 
indicated that they identify important questions for each program with 
evaluation funding and then allocate funds to the most important 
questions for each program. Consequently, the agency typically 
conducts evaluations in programs with evaluation funds (such as TANF) 
every year but has not evaluated programs which do not have evaluation 
funding (such as the Community Services and Social Services Block 
Grants). 

HUD and, to a certain extent, two CDC divisions seek to identify which 
programs are important to evaluate as well as what questions are 
important to answer about those programs. Agency staff have the 
flexibility to direct resources to the programs that they believe most 
need evaluation. HUD evaluation staff said that this broad scope 
allows them to build a portfolio of evaluations across policy areas 
and serve the agency's most pressing evaluation needs. Senior 
officials consider the value of proposals from all policy areas 
combined but make some effort to achieve balance across policy areas. 

Only CDC's divisions hold formal, ranked competitions to review and 
select proposals. In each division, staff members or external panels 
rate and rank all evaluations the evaluation and program offices 
propose, once they have been fully developed. Senior leaders at CDC's 
DHAP and DNPAO select proposals by rank and available funds. In 
addition, senior leaders at DNPAO rank and select proposals within 
each of its three policy areas: nutrition, physical activity, and 
obesity. Senior leaders at DASH consider information collected from 
site visits and interviews for a small group of semi-finalists that 
were selected based on the input of the external panel. CDC staff 
reported that CDC often uses ranked competitions to award grants and 
contracts across the agency. At the other agencies we reviewed, 
evaluation staff said that proposals are reviewed and selected in a 
series of discussions between the agency's policy officials, such as 
assistant or deputy secretaries, and the senior leaders of its 
evaluation and program offices. None of these agencies reported 
formally ranking their proposed evaluations but, instead, 
qualitatively consider the relative strengths and weaknesses of each 
evaluation. 

Proposal review and selection in the CDC divisions involves less 
department-level input than at ACF, Education, and HUD. CDC's 
evaluation staff reported that division leadership makes the final 
decision on evaluation projects and does not need the approval of the 
Office of the CDC Director or HHS officials, although a key criterion 
in project ranking and selection is often alignment with larger CDC, 
HHS, and national priorities. CDC is studying evaluation planning 
across the agency, however, and may increase central oversight in the 
future. The assistant secretary at ACF, not departmental officials, 
makes final approval decisions, but the evaluation staff reported 
consulting informally with staff of the Office of the Assistant 
Secretary for Planning and Evaluation (ASPE) when proposals are 
developed. 

The processes at Education and HUD are more centralized than at CDC or 
ACF. At these agencies, senior department officials--such as the 
secretary, deputy secretary, or assistant secretary--make the final 
selection decisions, after the evaluation and program staff have 
developed and reviewed the initial proposals. Beginning in fiscal year 
2010, HUD staff indicated that the agency funded many of its 
evaluations from a departmental Transformation Initiative fund, whose 
board must approve proposed evaluations and other projects. Board 
members include the assistant secretaries of PD&R and Community 
Planning and Development, the Chief Information Officer, as well as 
the Director of Strategic Planning and Management. 

One agency does not strictly plan evaluations for the next fiscal 
year. CDC's DASH staff plan evaluations that are funded during the 
current fiscal year rather than evaluations that will be funded in the 
next fiscal year. Local education agencies typically partner with the 
agency to conduct evaluations during the school year, when parents, 
students, and teachers are available to researchers. As a result, the 
agency cannot wait until funds are scheduled for appropriation in 
October or later, because their data collection plans and site 
selections must be final before the school year begins, typically in 
late August or early September. 

Education adjusted its evaluation planning guidance in 2010 to 
explicitly plan evaluations to be conducted in fiscal year 2011 as 
well as to inform its budget request for fiscal year 2012. The agency 
links its evaluation planning to the budget, partly to ensure that 
funding or authority will be available and that evaluations are 
aligned with program goals and objectives, congressional mandates, and 
the agency's strategic priorities. In addition, Education has 
proposed, for reauthorization of the Elementary and Secondary 
Education Act, to submit a biennial evaluation plan to the Congress 
and establish an independent advisory panel to advise the department 
on these evaluations.[Footnote 8] These plans align well with the 
American Evaluation Association's (AEA) recommendation, made in a 
recent policy paper on federal government evaluation, that federal 
agencies prepare annual and multiyear evaluation plans to guide 
program decision-making and consult with the Congress and nonfederal 
stakeholders in defining program and policy objectives, critical 
operations, and definitions of success.[Footnote 9] 

Common Types of Criteria Agencies Use to Prioritize Evaluation 
Proposals: 

We found these mature agencies remarkably similar in the four general 
criteria they used for selecting evaluations to conduct during the 
next fiscal year: strategic priorities, program concerns, critical 
unanswered questions, and the feasibility of conducting a valid 
evaluation study. Another important consideration, in situations in 
which several program offices draw on the same funding source, was 
establishing balance across program areas. Most agencies indicated no 
hierarchy among these criteria. Rather, they considered them 
simultaneously to create the best possible portfolio of studies. 

The first criterion, strategic priorities, represents major program or 
policy areas identified as a focus of concern and reflected in a new 
initiative or increased level of effort. Strategic priorities might be 
expressed by a department or the White House as strategic goals or 
secretarial initiatives or by the Congress in program authorizations 
or study mandates. Under GPRA, agencies are expected to revise their 
strategic plans at least every 3 years, providing an opportunity to 
align their plans with current conditions, goals, and concerns. The 
plans can chart expectations for both program and evaluation planning. 
CDC's DHAP officials described waiting for the White House release of 
the National AIDS Strategy in July to finalize their strategic plan 
and objectives and to prioritize evaluation activities that would 
address them. In addition to national priorities, division priorities 
are informed by their own research, surveillance, and program 
evaluation, identifying the subpopulations and geographic areas most 
affected by the disease. HUD's PD&R conducts the national Housing 
Discrimination Study every 10 years, which provides a unique benchmark 
and input to the department's long-term planning. 

Strategic priorities may also arise from congressional mandates. 
Education officials noted that the Congress generally mandates 
evaluations when it reauthorizes large formula grant programs, such as 
the national assessments of title I of the Elementary and Secondary 
Education Act, and that it has also mandated the evaluation of major 
new programs that might have great public interest or promise. They 
said that they schedule evaluations so that they will produce useful 
information for reauthorization, usually every 6 to 8 years. 

The second criterion, program-level concerns, represents more narrowly 
focused issues concerning an identified problem or opportunity. 
Evaluation staff reported that valuable ideas for evaluations often 
reflected the questions and concerns that arise in daily program 
operations. ACF noted that Head Start teachers' reports of disruptive 
children who prevented other children from learning led to a large- 
scale evaluation of several potentially effective practices to enhance 
children's socio-emotional development and teachers' classroom 
management practices. 

Accountability concerns that OMB, GAO, and Inspector General reports 
raise may lead to follow-up studies to assess the effectiveness of 
corrective actions. For example, PD&R staff stated that after a GAO 
report criticized the Section 202 demonstration grant program for not 
building housing projects in a timely fashion, the Congress introduced 
a competition for grants to speed up development. A follow-up 
evaluation will assess whether timeliness has improved. Other 
evaluation questions may address crosscutting issues that influence 
program success, such as a provider's ability to leverage resources or 
promote partnerships with other stakeholders. CDC's DNPAO places a 
priority on proposals that develop collaborations with external 
partners and among operational units within the division. 

The third criterion, critical unanswered questions, reflects the state 
of knowledge and evidence supporting government policies and programs. 
For example, agency staff talk with advisory groups, academics and 
other researchers in their field to identify useful research and 
evaluation questions that could advance understanding of the field and 
improve practice. CDC staff indicated that filling knowledge gaps is a 
particularly important criterion for project selection, because some 
public health areas lack an extensive research base or evidence on 
effective practices. An OPRE senior official described OPRE staff as 
looking for compelling, essential questions of enduring interest. ACF 
programs attempt to solve persistent social problems, such as testing 
diverse strategies to promote employment retention and advancement for 
low-wage workers and current or former TANF recipients. Because formal 
impact evaluations of these efforts may take 5 or 6 years to complete, 
OPRE staff look for questions that are persistent and studies that are 
likely to advance knowledge. Gathering information on emerging, 
promising practices was a consideration, particularly where evidence 
of effective practice has not yet been demonstrated. This was 
particularly important to the CDC divisions, DNPAO and DASH, where the 
public health research base was limited and effectiveness evaluations 
of promising practices were needed to expand the pool of evidence-
based interventions to offer grantees. 

The fourth criterion, evaluation feasibility, encompassed a range of 
pragmatic issues such as whether data were available and at what cost, 
whether the proposed evaluation could answer questions persuasively, 
and whether grantees had the interest and capacity to participate in 
evaluation. Naturally, agencies weigh their evaluation priorities in 
the context of their fiscal and budget constraints. Evaluators 
described determining whether the most important questions could be 
answered and the resources that would be needed to answer them. When 
"hard" data are lacking, some evaluators find that in-house 
exploratory work and investment in data gathering may be needed before 
scaling up to a contracted evaluation. Like the other evaluation 
units, PD&R compares the feasibility and cost of a study to 
alternative proposals. The evaluation staff noted that cost cannot be 
the sole criterion, however, because studies of some programs, such as 
the large block grants, are more resource intensive than the 
approaches available for studying other programs, such as housing 
voucher programs. 

When working with community-based organizations, agencies find grantee 
evaluation capacity can be very important. To ensure that the selected 
grantee is implementing the program faithfully, is ready for 
evaluation, and can collect valid and reliable data, CDC's DASH staff 
conduct site visits to assess candidate projects on such issues as 
appropriate logical links between program components and expected 
outcomes, staff turnover, political conflicts, fiscal sustainability, 
and staff interest in and capacity to conduct the evaluation. ACF 
evaluators were pleased to note that many state and local TANF 
officials participate in OPRE's annual welfare research conference, 
show interest in conducting evaluations, and have been willing to 
randomly assign recipients to new programs for research purposes. 

Prioritization Depends on Funding Sources and Agency Circumstances: 

Although the agencies generally followed a similar process in 
developing their evaluation agendas, some agency characteristics or 
conditions appeared to influence their choices and may be important 
for other agencies to consider as they develop their own evaluation 
agendas. The four conditions we identified as strongly influencing the 
evaluation planning process were: 

1. the location of evaluation funding, whether with the program or 
evaluation office; 

2. the scope of the evaluation unit's responsibility within the agency; 

3. how much the evaluators rely on program partners; and: 

4. the extent and form of congressional oversight over agency program 
evaluations. 

Evaluation Funding's Location: 

Where evaluation funds come from largely controls the selection of 
programs to evaluate. In ACF, CDC, and Education, authority and funds 
for evaluation are primarily attached to the programs, not to the 
evaluation office. This has implications for both how evaluation 
offices conduct their planning and for whether a program is likely to 
be evaluated. 

Where evaluation funds and authority are tied to the program, and 
funds are available, evaluation staff generally choose not which 
programs to evaluate but which research questions to answer. Thus, 
evaluators in ACF and Education work separately with each program 
office that has evaluation funds to develop proposals. 

In contrast, at HUD, when the evaluation office has uncommitted 
evaluation funds, selecting proposals can involve deciding between 
programs. Therefore, besides considering policy priorities and 
feasibility issues, HUD senior managers try to balance available 
evaluation funding across programs or policy areas after proposals are 
developed within program areas. This involves soliciting input from 
program office leaders on the preliminary agenda and discussing 
competing needs in the final selection process. CDC's DNPAO, with its 
three distinct program areas--nutrition, physical activity, and 
obesity--made similar efforts to obtain a balanced portfolio by 
forming teams to rank order proposals separately and having senior 
division leaders consider program balance in selecting proposals. 

One consequence of tying evaluation funds and authority to programs is 
that programs that do not have their own evaluation authority may not 
get evaluated at all. Staff at ACF and Education told us that because 
their evaluation offices did not have significant discretionary funds 
for external contracts, they had not conducted any evaluations of 
several programs, even though they believed that some of those 
programs should be evaluated. Not discussing the pros and cons of 
evaluating a particular program can lead to inappropriately excluding 
some from evaluation. HUD officials noted that it was important to 
attempt to balance evaluation spending across program areas because, 
otherwise, some programs might be avoided as too difficult or 
expensive to evaluate. Education officials said they plan to address 
this issue by developing a departmental portfolio of strong evaluation 
proposals based on policy and management needs, without regard to 
existing evaluation authority, and then request funds for them. Then, 
in future legislative proposals, they plan to ask the Congress for 
more flexibility in evaluation funds to better meet the field's needs. 

Scope of Responsibility: 

The agency evaluation offices we examined were located at different 
organizational levels, affecting the scope of their program and 
analytic responsibilities as well as the range of issues they 
considered. 

At CDC, the evaluation offices are generally within program offices, 
so they do not need a separate step for consulting with program staff 
to identify their priorities. Instead, the divisions solicit 
evaluation proposals from staff throughout the division. In the other 
agencies we examined, evaluation offices are either parallel to 
program offices (ACF) or at the departmental level (Education and 
HUD), which leads them to consult more formally with the program 
offices during both development and selection. 

Location and scope of responsibilities also influenced evaluation 
approval. CDC's divisions, with the narrowest scope among the units we 
examined, exerted considerable control over their evaluation funds and 
did not require approval of their evaluation agendas by either the 
director or the department. DASH did, however, report coordinating 
evaluation planning with other agencies and HHS offices on specific 
cross-cutting programs, and DHAP reported delaying its selection of 
evaluation proposals this past spring to coordinate with the new 
National AIDS Strategy. In contrast, at Education and HUD, where 
evaluation offices have departmental scope, final approval decisions 
are made at the department level. In the middle, OPRE selections are 
approved by the ACF assistant secretary and do not require 
departmental approval. 

Being responsible for a wide range of analytic activities also 
influenced an evaluation office's choice of evaluations. Evaluators in 
the more centralized offices in ACF and HUD described having the 
flexibility to address the most interesting questions feasible. For 
example, if it is too early to obtain hard data on an issue, PD&R 
staff said that they might turn to in-house exploratory research on 
that issue. ACF staff noted that they often conducted small 
descriptive studies of the operations of state TANF programs because 
of the decentralized nature of that program. This flexibility can 
mean, however, that they must also consider the range of the program 
office's information needs when developing their portfolio of studies. 
PD&R staff noted that they try to ensure that some studies are 
conducted in-house to meet program staff interest in obtaining quick 
turnaround on results. DNPAO aims to achieve a balanced portfolio of 
studies by ranking cross-cutting proposals within categories of 
purpose, such as monitoring or program evaluation. Education officials 
propose to create a comprehensive departmental evaluation plan that 
identifies the department's priorities for evaluation and other 
knowledge-building activities, is aligned with their strategic plan, 
and will support resource allocation. 

Several of the evaluation offices we examined also provide technical 
assistance in performance monitoring and evaluation. While this may 
help strengthen relationships with program staff and understanding of 
program issues, the responsibility can also reduce the resources 
available for evaluation studies. All three CDC divisions require 
evaluations or performance monitoring from their grantees; therefore, 
providing grantees with technical assistance is a major activity for 
these evaluation offices. In DHAP and DNPAO, staff workload, including 
providing technical assistance, was cited among the resource 
constraints in developing evaluation proposals. ACF staff noted that 
if program offices prioritize their available funds on technical 
assistance and monitoring, there may not be enough to conduct an 
evaluation. 

In our cases, placing the evaluation office inside the program office 
(as in the CDC divisions we examined) was associated with conducting 
more formal proposal ranking. We considered several possible 
explanations for this: (1) staff adopted the competitive approach they 
generally take in assessing proposals for project and research funds, 
(2) a large volume of proposals required a more systematic process for 
making comparisons, or (3) the visibility of the selections created 
pressure for a transparent rationale. The first point may be true but 
does not explain why the other agencies are also deliberative in 
assessing and comparing evaluation proposals but do not rate them 
numerically. 

The two other explanations appear to be more relevant and may be 
related to the fact that evaluations are being selected within the 
program office and thus cover a relatively narrow range of options. 
CDC staff said that they did not need to formally rate and rank the 
three or four proposals they submitted for OMB's Evaluation Initiative 
but might have done so had the number of proposals to consider been 
greater. DASH and DHAP issue broad calls each year for nominations of 
promising practices to evaluate and, thus, gather a large number of 
proposals to assess. Staff in DASH, which also solicits project 
nominations from the public, indicated that over time their process 
has become more formal, accountable, and transparent so that 
selections appear to the public to be more systematic and less 
idiosyncratic. Although information is limited, we believe that 
systematically rating and ranking proposals may be a useful procedure 
to consider case by case. 

Reliance on Program Partners: 

The influence of nonfederal program partners on developing and 
selecting evaluation proposals was observed in most of the agencies we 
examined, although it did not vary much among them. The importance of 
program stakeholders to planning should be expected because these 
particular agencies generally rely on external partners--state and 
local agencies and community-based organizations--to implement their 
programs. However, the extent of coordination with external parties on 
evaluation planning seen here may not be necessary in agencies that 
are not so reliant on third parties. ACF's evaluation staff pointed 
out that they cannot evaluate practices that a state or local agency 
is not willing to use. Efforts to engage the academic and policy 
communities in discussing ideas for future research at ACF and 
Education also reflect these agencies' decades-long history of 
sponsoring research and evaluation. CDC's DHAP and DNPAO also employ 
advisory groups, including CDC staff and external experts, to advise 
them on strategic planning and topics that will help meet the needs of 
their grantees, but only DASH involved external experts directly in 
assessing evaluation proposals. DASH evaluators assemble panels to 
assess nominations of sites implementing a promising practice; 
depending on the topic and stage of the process, these panels might 
include external experts and experts from across CDC or other agencies 
serving children and families. 

Program partners' evaluation capacity is especially important to 
evaluation planning in the CDC divisions we examined because their 
evaluations tend to focus on the effectiveness of innovative programs 
or practices. Each year, DASH publicly solicits nominations of 
promising projects of a designated type of intervention and uses 
review panels and site visits to rank dozens of sites on an 
intervention's strength and promise, as well as the feasibility of 
conducting an evaluation. Staff said that it was important to ensure 
that the grantee organization was stable and able to cooperate fully 
with an evaluation and noted that evaluation is sometimes difficult 
for grantees. 

The Form of Congressional Oversight: 

Congress influences agencies' evaluation choices in a variety of ways. 
Congress provides agencies with the authority and the funds with which 
to conduct evaluations and may mandate specific studies. The 
evaluation offices in ACF, Education, and HUD all noted their 
responsibility to conduct congressionally mandated evaluation studies 
in describing the criteria they used in evaluation planning. The CDC 
offices indicated that they did not have specific study mandates but, 
rather, authority to conduct studies with wide discretion over the 
particular evaluation topics or questions. Of course, in addition to 
legislatively mandating studies, the Congress expresses interest in 
evaluation topics through other avenues, such as oversight and 
appropriations hearings. DHAP evaluators noted that they receive a lot 
of public scrutiny and input from the Congress and the public health 
community that works its way into project selection through the 
division's setting of priorities. 

Agency evaluators described a continuum of evaluation mandates, from a 
general request for a study or report to a list of specific questions 
to address. Education officials noted that the Congress generally 
mandates evaluations of the largest programs when they are 
reauthorized or new programs or initiatives for which public interest 
or promise might be great. Some evaluators noted that sometimes the 
Congress and agency leaders want answers to policy questions that 
research cannot provide. They indicated that, where legislative 
language was vague or confusing, they did their best to interpret it 
and create a feasible evaluation. In a previous study of agency 
studies' not meeting congressional information needs, we suggested 
that expanding communication between congressional staff and agency 
program and evaluation staff would help ensure that information needs 
are understood and that requests and reports are suitably framed and 
adapted as needs evolve.[Footnote 10] 

Evaluators told us that whether and how much funding was attached to 
an evaluation mandate also influenced how the mandate was implemented. 
They said that when appropriate funding was available, they always 
conducted congressionally mandated evaluations. However, sometimes the 
amounts available do not reflect the size of the evaluation needs in a 
program. This was particularly a problem for small programs where a 
fixed set-aside of program funds for evaluation might yield funds 
inadequate for rigorous evaluation. 

Evaluators described a related challenge when evaluation authorities 
are attached to single programs which preclude pooling funds across 
programs. Such limitations on using evaluation funds could lead to 
missed opportunities to address cross-cutting issues. In cases where 
no additional funding was provided for legislatively mandated studies, 
agencies had to decide how and whether to fund them. Some agency 
evaluators told us that they generally conducted what they saw as 
"unfunded mandates" but would interpret the question and select an 
approach to match the funds they had available. This might mean that 
without funds to collect new data, a required report might be limited 
to simply analyzing or reporting existing data. 

HUD receives considerable congressional oversight of its research and 
evaluation agenda, reflecting congressional concern about its past 
research priorities and greater decision-making flexibility under the 
new Transformation Initiative. In 2008, a congressionally requested 
National Research Council review of HUD's research and evaluation 
lauded most of PD&R's work as "high quality, relevant, timely, and 
useful" but noted that its resources had declined over the previous 
decade, its capacity to perform effectively was deteriorating, and its 
research agenda was developed with limited input from outside the 
department.[Footnote 11] 

NRC recommended that, among other things, HUD actively engage external 
stakeholders in framing its research agenda. In response, PD&R 
solicited public suggestions online for research topics for fiscal 
year 2011 and beyond. In addition, HUD proposed a Transformation 
Initiative of organizational and program management improvement 
projects in 2009 and asked that up to 1 percent of its program budget 
be set aside in a proposed Transformation Initiative fund to support 
research and evaluation, information technology, and other projects. 
The House and Senate Appropriations Committees approved the fund (at 
somewhat less than the requested amount) with a proviso that HUD 
submit a plan for appropriations committee approval, detailing the 
projects and activities the funds would be used for.[Footnote 12] 

Concluding Observations: 

An effective evaluation agenda aims to provide credible, timely 
answers to important policy and program management questions. In 
setting such agendas, agencies may want to simultaneously consider the 
four general criteria we identified: strategic priorities, program 
concerns, critical unanswered questions, and the feasibility of 
conducting a valid study. In the short run, because agency evaluation 
resources are limited, ensuring balance in evaluations across programs 
may not be as important as addressing strategic priorities. However, 
developing a multiyear evaluation plan could help ensure that all an 
agency's programs are examined over time. 

To produce an effective evaluation agenda, agencies may want to follow 
the general model we identified at the agencies we reviewed: 
professional evaluators lead an iterative process of identifying 
important policy and program management questions, vetting initial 
ideas with the evaluations' intended users, and scrutinizing the 
proposed portfolio of studies for relevance and feasibility within 
available resources. Since professional evaluators have the knowledge 
and experience to identify researchable questions and the strengths 
and limitations of available data sources, they are well suited to 
leading a consultative process to ensure that decision makers' 
information needs can be met. 

However, agencies may need to adapt the general model's steps to match 
their own organizational and financial circumstances. For example, 
they may not need to formally rank proposals unless they have many 
more high-quality proposals than they can fund. They may find 
advantages to placing evaluation offices within program offices (for 
focusing on program needs, for example) and at higher levels (for 
addressing broader policy questions). Where analytic demands are 
significant and resources permit, they may find a combined approach 
best-suited to their needs. 

To ensure that their evaluations provide the information necessary for 
effective management and legislative oversight, evaluation offices are 
likely to need to seek out in advance the interests and concerns of 
key program and congressional stakeholders, especially program 
partners, and discuss preliminary proposals with the intended users. 

Agency Comments: 

The Departments of Health and Human Services and Housing and Urban 
Development provided comments on a draft of this report, which are 
reprinted in appendixes I and II. 

HHS appreciated the attention that this report gives to the importance 
of strong prioritization processes for selecting evaluation studies 
and allocating resources to complete them, and was pleased that the 
practices of ACF and CDC in this area are models for emulation by 
others. It also noted that, given the diversity of purposes for 
evaluations, the optimal location and organization of evaluation 
activities will vary with the circumstances. This is consistent with 
our concluding observation that agencies may need to adapt the general 
model--including where to locate evaluation offices--to match their 
own organizational and financial circumstances. 

HUD agreed with our description of how it plans evaluations but was 
concerned that the report did not place enough emphasis on the 
appropriations process as a major influence on what projects it funds 
and when it can begin the contracting process. We have added text to 
note that the Congress influences the agencies' evaluation processes 
through providing them with both the authority and funds with which to 
conduct evaluations, as well as mandating specific studies. 

Education, HHS, and HUD also provided technical comments that were 
incorporated where appropriate throughout the text. 

We are sending copies of this report to the Secretaries of Education, 
Health and Human Services, and Housing and Urban Development; the 
Director of the Office of Management and Budget; and appropriate 
congressional committees. The report is also available at no charge on 
GAO's Web site at [hyperlink, http://www.gao.gov]. 

If you have questions about this report, please contact me at (202) 
512-2700 or kingsburyn@gao.gov. Contacts for our Office of 
Congressional Relations and Office of Public Affairs are on the last 
page. Key contributors are listed in appendix III. 

Sincerely yours, 

Signed by: 

Nancy Kingsbury, Ph.D. 
Managing Director: 
Applied Research and Methods: 

[End of section] 

Appendix I: Comments from the Department of Health and Human Services: 

DEPARTMENT OF HEALTH and HUMAN SERVICES: 
OFFICE OF THE SECRETARY: 
Assistant Secretary for Legislation: 
Washington, DC 20201: 

December 17, 2010: 

Nancy R. Kingsbury, Ph.D. 
Managing Director: 
Applied Research and Methods: 
U.S. Government Accountability Office: 
441 G Street N.W. 
Washington, DC 20548: 

Dear Ms. Kingsbury: 

Attached are comments on the U.S. Government Accountability Office's 
(GAO) report entitled: "Program Evaluation: Experienced Agencies 
Follow a Similar Model for Prioritizing Research" (GAO-11-176).
The Department appreciates the opportunity to review this report 
before its publication. 

Sincerely, 

Jim R. Esquea: 
Assistant Secretary for Legislation: 

Attachment: 

[End of letter] 

General Comments Of The Department Of Health And Human Services (HHS) 
On The Government Accountability Office's (GAO) Draft Report Entitled, 
"Program Evaluation: Experienced Agencies Follow A Similar Model For 
Prioritizing Research" (GAO-11-176): 

The Department appreciates the opportunity to review and comment on 
this draft report. 

We appreciate the attention that this report gives to the importance 
of strong prioritization processes for selecting evaluation studies 
and allocating resources to complete them. We are pleased that the 
practices of the Administration for Children and Families (ACF) and 
the Centers for Disease Control and Prevention (CDC) in this area are 
models for emulation by others. We also note, as reported in the 
draft, that recent organizational restructuring has led to even more 
attention to performance, accountability and program evaluation by 
CDC's Office of the Director. 

As GAO noted, the Office of the Assistant Secretary for Planning and 
Evaluation (ASPE) coordinates agency evaluation activities, reports to 
Congress on the Department's evaluations, and conducts studies on 
broad, cross-cutting issues. ASPE works in close collaboration with 
departmental agencies to evaluate their programs, frequently taking 
the lead on such program evaluations. ASPE is also uniquely positioned 
in the Department to identify programs serving similar purposes but 
housed in different agencies within HHS and at other Departments, and 
works to coordinate their program efforts as well as evaluation 
activities. As part of this role, ASPE has led the Department's 
response to OMB's Evaluation Initiative in recent years. 

The diversity of evaluation activities at HHS is a result of our being 
the largest and most diverse civilian executive department in the 
federal government. As a large department that manages many programs, 
some programs conduct evaluations for the purpose of program 
management and monitoring. In other instances, external program 
evaluations are conducted for policy and budget allocation purposes. 
Sometimes, evaluation research supports technical assistance 
activities with states, localities and individual service providers. 
Depending on the specific purposes and circumstances the optimal 
location and organizations of evaluation activities will vary. 
Evaluations aimed at improved management practices within an agency 
may be best conducted by the management of the agency in question. In 
contrast, when the purpose is to inform policy questions, external 
evaluations conducted outside the agency operating the program may be 
most appropriate. In this manner, the efficient and effective division 
of labor in evaluation is different across agencies of different 
sizes, with complex arrays of programs, policies and regulations that 
they oversee. 

[End of section] 

Appendix II Comments from the Department of Housing and Urban 
Development: 

U.S. DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT: 
ASSISTANT SECRETARY FOR POLICY DEVELOPMENT AND RESEARCH
WASHINGTON, DC 20410-6000: 

December 15, 2010: 

Dr. Nancy Kingsbury: 
Managing Director: 
Applied Research and Methods: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Dr. Kingsbury: 

On behalf of Secretary Donovan, thank you for the opportunity to 
review your draft report titled "Program Evaluation: Experienced 
Agencies Follow a Similar Model for Prioritizing Research." I am 
responding because, where your report considers HUD, it is primarily 
about the Office of Policy Development and Research. 

The GAO team did a nice job in preparing this report. I am attaching a 
few comments, most of which are editorial. I do have one substantive 
concern, which is that the report does not place enough emphasis on 
the appropriations process. The report rightly considers how we plan 
evaluations. However, what gets funded and when we can begin to 
procure research contracts is very much contingent on the amount of 
funding Congress provides and the time our appropriations bill is 
enacted. The report appears to presume this occurs on October 1, but 
as we know, it has been many years since this has happened. In FY 
2010, for example, the bill was not enacted until December 16, 2009 - 
two and a half months after October I. Such delays have significant 
implications for PD&R's performance in executing our recent program. 

Thank you again for the opportunity to review this report. 

Very sincerely yours, 

Signed by: 

Raphael W. Bostic: 

Attachment: 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Nancy Kingsbury (202) 512-2700 or kingsburyn@gao.gov: 

Staff Acknowledgments: 

In addition to the person named above, Stephanie Shipman, Assistant 
Director; Valerie Caracelli; and Jeff Tessin made significant 
contributions to this report. 

[End of section] 

Bibliography: 

American Evaluation Association. An Evaluation Roadmap for a More 
Effective Government. September 2010. [hyperlink, 
http://www.eval.org/EPTF.asp]. 

Leviton, Laura C., Laura Kettel Khan, and Nicola Dawkins, eds. "The 
Systematic Screening and Assessment Method: Finding Innovations Worth 
Evaluating." New Directions for Evaluation no. 125, 2010. 

National Research Council, Committee to Evaluate the Research Plan of 
the Department of Housing and Urban Development, Center for Economic, 
Governance, and International Studies, Division of Behavioral and 
Social Sciences and Education. Rebuilding the Research Capacity at 
HUD. Washington, D.C.: National Academies Press, 2008. 

Office of Management and Budget. Analytical Perspectives--Budget of 
the United States Government, Fiscal Year 2011. Washington, D.C.: 
Executive Office of the President, Feb. 1, 2010. 

Office of Management and Budget. Evaluating Programs for Efficacy and 
Cost-Efficiency. M-10-32 Memorandum for the Heads of Executive 
Departments and Agencies. Washington, D.C.: Executive Office of the 
President, July 29, 2010. [hyperlink, 
http://www.whitehouse.gov/sites/default/files/omb/memoranda/2010/m10-
32.pdf]. 

Office of Management and Budget. Increased Emphasis on Program 
Evaluations. M-10-01 Memorandum for the Heads of Executive Departments 
and Agencies. Washington, D.C.: Executive Office of the President, 
Oct. 7, 2009. [hyperlink, 
http://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/
m10-01.pdf]. 

U.S. Department of Education, Office of Planning, Evaluation, and 
Policy Development. A Blueprint for Reform: The Reauthorization of the 
Elementary and Secondary Education Act. Washington, D.C.: March 2010. 

U.S. Department of Health and Human Services. Evaluation: Performance 
Improvement 2009. Washington, D.C.: 2010. 

[End of section] 

Related GAO Products: 

Employment and Training Administration: Increased Authority and 
Accountability Could Improve Research Program. [hyperlink, 
http://www.gao.gov/products/GAO-10-243]. Washington, D.C.: January 29, 
2010. 

Program Evaluation: A Variety of Rigorous Methods Can Help Identify 
Effective Interventions. [hyperlink, 
http://www.gao.gov/products/GAO-10-30]. Washington, D.C.: November 23, 
2009. 

Continuing Resolutions: Uncertainty Limited Management Options and 
Increased Workload in Selected Agencies. [hyperlink, 
http://www.gao.gov/products/GAO-09-879]. Washington, D.C.: September 
24, 2009. 

Results-Oriented Management: Strengthening Key Practices at FEMA and 
Interior Could Promote Greater Use of Performance Information. 
[hyperlink, http://www.gao.gov/products/GAO-09-676]. Washington, D.C.: 
August 17, 2009. 

Performance Budgeting: PART Focuses Attention on Program Performance, 
but More Can Be Done to Engage Congress. [hyperlink, 
http://www.gao.gov/products/GAO-06-28]. Washington, D.C.: October 28, 
2005. 

Program Evaluation: OMB's PART Reviews Increased Agencies' Attention 
to Improving Evidence of Program Results. [hyperlink, 
http://www.gao.gov/products/GAO-06-67]. Washington, D.C.: October 28, 
2005. 

Managing for Results: Enhancing Agency Use of Performance Information 
for Management Decision Making. [hyperlink, 
http://www.gao.gov/products/GAO-05-927]. Washington, D.C.: September 
9, 2005. 

Performance Measurement and Evaluation: Definitions and Relationships. 
[hyperlink, http://www.gao.gov/products/GAO-05-739SP]. Washington, 
D.C.: May 2005. 

Program Evaluation: An Evaluation Culture and Collaborative 
Partnerships Help Build Agency Capacity. [hyperlink, 
http://www.gao.gov/products/GAO-03-454]. Washington, D.C.: May 2, 2003. 

Program Evaluation: Improving the Flow of Information to the Congress. 
[hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1]. Washington, 
D.C.: January 30, 1995. 

[End of section] 

Footnotes: 

[1] GAO, Managing for Results: Enhancing Agency Use of Performance 
Information for Management Decision Making,. [hyperlink, 
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9, 
2005); Executive Office of the President, Office of Management and 
Budget, Analytical Perspectives--Budget of the United States 
Government, Fiscal Year 2011 (Washington, D.C.: The White House, Feb. 
1, 2010), pp. 73-74. 

[2] Office of Management and Budget, Increased Emphasis on Program 
Evaluations, M-10-01, Memorandum for the Heads of Executive 
Departments and Agencies (Washington, D.C.: The White House, Oct. 7, 
2009). 

[3] See GAO, Employment and Training Administration: Increased 
Authority and Accountability Could Improve Research Program, 
[hyperlink, http://www.gao.gov/products/GAO-10-243] (Washington, D.C.: 
Jan. 29, 2010), which draws on guidelines issued by the American 
Evaluation Association and National Research Council of the National 
Academy of Sciences. 

[4] National Research Council, Rebuilding the Research Capacity at HUD 
(Washington, D.C.: National Academies Press, 2008). 

[5] U.S. Department of Health and Human Services, Evaluation: 
Performance Improvement 2009 (Washington, D.C.: 2010). 

[6] White House Office of National AIDS Policy, National HIV/AIDS 
Strategy for the United States (Washington, D.C.: July 2010). 
[hyperlink, http://www.whitehouse.gov/onap]. 

[7] [hyperlink, http://www.huduser.org]. 

[8] U.S. Department of Education, Office of Planning, Evaluation, and 
Policy Development, A Blueprint for Reform: The Reauthorization of the 
Elementary and Secondary Education Act (Washington, D.C.: March 2010). 

[9] American Evaluation Association, An Evaluation Roadmap for a More 
Effective Government (September 2010). [hyperlink, 
http://www.eval.org/EPTF.asp]. 

[10] GAO, Program Evaluation: Improving the Flow of Information to the 
Congress, [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1] 
(Washington, D.C.: Jan. 30, 1995). 

[11] National Research Council, Rebuilding the Research Capacity at 
HUD (Washington, D.C.: National Academies Press, 2008), pp. 1-4. 

[12] Consolidated Appropriations Act, 2010, Pub. L. No. 111-117, 123 
Stat. 3034 (Dec. 16, 2009). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: