This is the accessible text file for GAO report number GAO-11-285 
entitled 'Employment And Training Administration: More Actions Needed 
to Improve Transparency and Accountability of Its Research Program' 
which was released on April 14, 2011. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

March 2011: 

Employment And Training Administration: 

More Actions Needed to Improve Transparency and Accountability of Its 
Research Program: 

GAO-11-285: 

GAO Highlights: 

Highlights of GAO-11-285, a report to congressional committees. 

Why GAO Did This Study: 

To help guide the nation’s workforce development system, the 
Department of Labor’s (Labor) Employment and Training Administration 
(ETA) conducts research in areas related to job training and 
employment. Building upon our earlier work, GAO examined the 
following: (1) To what extent do ETA’s research priorities reflect key 
national employment and training issues and how useful were the 
studies funded under them? (2) What steps has ETA taken to improve its 
research program? (3) How has ETA improved the availability of its 
research since our last review in January 2010? To answer these 
questions, GAO reviewed ETA’s research reports disseminated between 
January 2008 and March 2010 costing $1 million or more, as well as 
ongoing studies costing $2 million or more. GAO also convened a 
virtual expert panel, interviewed Labor and ETA officials, and 
reviewed relevant documents. 

What GAO Found: 

ETA’s 2007 to 2012 research plan generally addressed key employment 
and training issues, but some studies were limited in their 
usefulness. Most experts on our panel reported that the areas in ETA’s 
plan reflected key national employment and training issues at least to 
a moderate extent. ETA invested most of its research and evaluation 
resources in the areas of Unemployment Insurance and increasing labor 
market participation of underutilized groups. Of the $96 million that 
ETA invested in the 58 research reports we reviewed, more than half—or 
about $56 million—funded studies in these two areas. The 
methodological approaches and statistical procedures researchers used 
in the studies we reviewed were generally consistent with the 
questions posed, but the studies were not always useful for informing 
policy and practice. For example, in one study, shortcomings in the 
data collection phase limited the strength of the findings. Experts 
suggested that ETA include more varied and rigorous methodologies in 
its future research projects. They also reported that future research 
should address additional areas, including a focus on employment and 
training approaches that work and for whom. 

Labor and ETA have taken steps to improve the way research is 
conducted, but additional actions are needed. In acknowledging the 
need for more rigorous evaluations to inform its policies, Labor 
recently established the Chief Evaluation Office to oversee 
departmentwide research and evaluation efforts. In addition, ETA made 
changes to some of its research practices. For example, ETA has begun 
involving outside experts in developing its research plan. Despite 
these improvements, ETA’s process lacks critical elements needed to 
ensure that current improvements become part of its routine practices. 
For example, ETA’s process lacks a formal provision to consult with 
the newly established Chief Evaluation Officer at important points in 
the research process. Moreover, ETA’s current research practices fall 
short of ensuring research transparency and accountability—essential 
elements of a sound research program. For example, its research and 
evaluation center lacks safeguards to protect it from undue outside 
influence. ETA has recently begun efforts to increase the rigor of its 
research designs, but has faced design and implementation challenges. 
For example, some of ETA’s ongoing research studies face challenges in 
recruiting large enough sample sizes to meet the studies’ objectives. 

ETA has improved the availability of its research findings, but it 
lacks a plan for assessing the usability of its Web-based search page—
the primary tool for making ETA’s research publicly available. ETA 
recently improved the timeliness with which it disseminates its 
research reports, decreasing the average number of days to release its 
reports to the public from 804 days in 2008 to 76 days in 2009. ETA 
has taken steps to update its online, Web-based search page. However, 
the agency has not developed a formal plan for assessing the overall 
effectiveness of its Web-based search page, including user 
satisfaction. In addition to its research database, ETA uses a variety 
of other methods to disseminate its research, including providing its 
research reports at conferences and internal briefings. Experts 
suggested that ETA consider other effective dissemination methods, 
such as publishing a one-page summary of research findings. 

What GAO Recommends: 

GAO recommends that ETA formally include the Chief Evaluation Officer 
in its research process, create a mechanism to enhance the 
transparency and accountability of its research program, and develop a 
plan to ensure that research reports are accessible through its Web-
based search page. Labor agreed with our recommendations and noted its 
ongoing efforts to improve its research program. While these efforts 
are important, GAO stresses the need for additional actions to fully 
address the recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-11-285] or key 
components. For more information, contact George Scott at (202) 512-
7215 or scottg@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

ETA's Research Areas Generally Reflect Key Issues, but Some Studies 
Are of Limited Usefulness: 

ETA Has Taken Steps to Improve Its Research Program, but Additional 
Actions Are Needed: 

ETA Has Improved the Availability of Its Research but There Are 
Opportunities to Improve Its Search Page and Dissemination Methods: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Status of Prior GAO Recommendations to the Department of 
Labor, as of January 2011: 

Appendix II: Scope and Methodology: 

Appendix III: The Panel's Ratings of Key Employment and Training 
Issues, Populations, and Programs That ETA Should Address in Its 
Future Research: 

Appendix IV: Characteristics of Research Studies Disseminated between 
January 2009 and March 2010 That Cost $1 Million or More: 

Appendix V: Delphi Phase I and Phase II Questionnaires: 

Appendix VI: Experts Who Agreed to Participate in GAO's Delphi Panel: 

Appendix VII: Comments from the Department of Labor: 

Appendix VIII: GAO Contact and Staff Acknowledgments: 

GAO Related Products: 

Tables: 

Table 1: Research and Evaluation Terms and Definitions: 

Table 2: ETA's Research Areas, with Examples of Topics Covered in the 
2007 to 2012 Research Plan: 

Table 3: Estimated Funding by Research Area for Research Studies 
Disseminated between January 2008 and March 2010: 

Table 4: Characteristics of Three Ongoing Studies: 

Table 5: Descriptive Statistics on Key Employment and Training Issues 
ETA Should Address in Its Future Research: 

Table 6: Descriptive Statistics on Key Populations ETA Should Address 
in Its Future Research: 

Table 7: Descriptive Statistics on Key Employment and Training 
Programs ETA Should Address in Its Future Research: 

Figures: 

Figure 1: ETA's Combined Evaluations and Pilots, Demonstrations, and 
Research Budgets from Fiscal Years 2007 to 2010: 

Figure 2: Key Phases of the Research Process: 

Figure 3: Issue Areas Ranked among the Top Three by Experts for ETA to 
Address in Future Research: 

Figure 4: Unemployment Issues Frequently Cited by Experts as Important 
for ETA to Address in Future Research: 

Figure 5: Expert Panel Views on the Extent to Which ETA's Research Has 
Informed Policy and Practice: 

Figure 6: Methodological Approaches Frequently Cited by Experts as 
Very Important for ETA to Address in Future Research: 

Figure 7: ETA's 8-Step Research Process: 

Figure 8: Experts' Opinion on the Effectiveness of Various Research 
Dissemination Methods: 

Abbreviations: 

ETA: Employment and Training Administration: 

Project GATE II: Project Growing America Through Entrepreneurship II: 

Labor: Department of Labor: 

OMB: Office of Management and Budget: 

OPDR: Office of Policy Development and Research: 

TEN: Training and Employment Notice: 

UI: Unemployment Insurance: 

WIA:Workforce Investment Act of 1998: 

WIRED:Workforce Innovation in Regional Economic Development: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

March 15, 2011: 

The Honorable Tom Harkin:
Chairman:
Subcommittee on Labor, Health and Human Services, Education, and 
Related Agencies:
Committee on Appropriations:
United States Senate: 

The Honorable Michael B. Enzi:
Ranking Member:
Committee on Health, Education, Labor, and Pensions:
United States Senate: 

The Honorable Patty Murray:
Chairman:
The Honorable Johnny Isakson:
Ranking Member:
Subcommittee on Employment and Workplace Safety:
Committee on Health, Education, Labor, and Pensions:
United States Senate: 

Across the country, the workforce development systems' one-stop 
centers serve as the key access point for services that are crucial in 
today's economy--services that include Unemployment Insurance (UI) 
benefits, job training, and employment assistance. The Department of 
Labor's (Labor) Employment and Training Administration (ETA) is 
responsible for guiding the nearly $13 billion public workforce 
development system. Its mission is to help make the U.S. labor market 
function more efficiently by developing policies that lead to high-
quality job training, employment, labor market information, and income 
maintenance services. To help shape its policies, ETA conducts 
evaluations and research studies on a range of employment-related 
topics. Congress appropriated about $103 million to ETA's research and 
evaluation budget line items for 2010.[Footnote 1] 

Over the last decade, however, ETA's research and evaluation program 
has fallen short in its efforts to conduct research that can help 
answer urgent workforce policy questions. In 2008 and again in 2009, 
we faulted ETA for failing to conduct research and evaluations that 
would lead to an understanding of what works and what 
doesn't.[Footnote 2] For example, in January 2010, we reported on 
shortcomings in ETA's research structure and processes.[Footnote 3] We 
found that ETA's research and evaluation center, the Office of Policy 
Development and Research (OPDR), 

* lacked independent authority to make key decisions about its 
research; 

* maintained processes that were unclear and that lacked transparency 
and accountability; 

* lacked a standard process for ensuring stakeholder involvement or 
other strategies to ensure that research addressed key national 
priorities; and: 

* had been slow to distribute its research findings and slow to 
respond to its statutory mandate to evaluate the Workforce Investment 
Act of 1998 (WIA). 

Based on these findings, we made several recommendations to Labor to 
improve ETA's research program. (For information on the status of 
those recommendations, see appendix I.) 

As ETA's leadership moves forward to help the nation meet its current 
employment challenges, questions remain about how well ETA's research 
has prepared the workforce development system for the challenges of 
today. Against this backdrop, you asked us to build upon our January 
2010 review and further examine ETA's research program. Specifically, 
we answered the following questions: 

1. To what extent do ETA's research priorities reflect key national 
employment and training issues and how useful were the studies funded 
under them? 

2. What steps has ETA taken to improve its research program? 

3. How has ETA improved, if at all, the availability of its research 
since our last review in January 2010 and what other steps could ETA 
take to further ensure its research findings are readily available? 

To address our objectives, we reviewed the 58 research and evaluation 
reports that ETA disseminated between January 2008 and March 2010 and 
assessed the methodological soundness of the 11 completed studies that 
cost $1 million or more. In addition, we reviewed the 10 ongoing 
studies costing $2 million or more to determine if research practices 
or the soundness of research designs had changed over time. In 
addition, we convened a virtual (Delphi) expert panel of academics, 
researchers, and consultants to obtain their opinions on ETA's 
research priorities and dissemination methods. To learn how ETA 
determines what research to conduct, we interviewed Labor officials 
and reviewed relevant agency and budget documents. In addition, we 
conducted site visits to two local workforce agencies in Pennsylvania 
and Virginia that are implementing ETA's ongoing research studies to 
obtain information about challenges and issues associated with 
participating in studies. To evaluate the availability of ETA's 
research, we analyzed dissemination time frames for all publications 
released between January 2008 and March 2010 and we tested the ability 
of ETA's research database to support searches generally available to 
users of research databases. (See appendix II for more details on our 
objectives, scope, and methodology.) 

We conducted this performance audit from March 2009 through March 2011 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Background: 

WIA sets forth various requirements for the Secretary of Labor 
relating to research and evaluation of federally funded employment-
related programs and activities. The law calls upon the Secretary of 
Labor to publish in the Federal Register every 2 years a plan that 
describes its pilot, demonstration, and research priorities for the 
next 5 years regarding employment and training. Specifically, WIA 
requires the Secretary to: 

* develop the research plan after consulting with states, localities, 
and other interested parties; 

* send the plan to the appropriate committees of Congress; and: 

* take into account such factors as the likelihood that the results of 
the projects will be useful to policymakers and stakeholders in 
addressing employment and training problems.[Footnote 4] 

Within ETA, OPDR's Division of Research and Evaluation plans, 
conducts, and disseminates employment and training-related research 
and evaluations. Nearly all of the agency's research and evaluation 
studies are conducted under contract; these contractors represent a 
range of research organizations and academic institutions. 
Furthermore, OPDR plans and conducts its research and evaluation 
activities in consultation with ETA's program offices, such as the 
Office of Workforce Investment and the Office of Trade Adjustment 
Assistance. 

ETA's research and evaluation funding is divided into two separate 
budget line items: 

Pilots, demonstrations, and research. Efforts in this category are 
focused on developing and testing new ways to approach problems and to 
deliver services. Under WIA, pilots and demonstrations shall be 
carried out "for the purpose of developing and implementing techniques 
and approaches, and demonstrating the effectiveness of specialized 
methods, in addressing employment and training needs." WIA also states 
that the Secretary shall "carry out research projects that will 
contribute to the solution of employment and training problems in the 
United States." 

Evaluations. Efforts in this category are focused on continuing 
evaluations of certain programs and activities carried out under WIA. 
These evaluations must address: 

* the effectiveness of these programs and activities carried out under 
WIA in relation to their cost; 

* the effectiveness of the performance measures relating to these 
programs and activities; 

* the effectiveness of the structure and mechanisms for delivery of 
services through these programs and activities; 

* the impact of the programs and activities on the community and 
participants involved, and on related programs and activities; 

* the extent to which such programs and activities meet the needs of 
various demographic groups; and: 

* such other factors as may be appropriate. 

In program year 2010, ETA's combined budget appropriation for 
conducting evaluations and pilots, demonstrations, and research was 
about $103 million--or nearly $34 million above what the agency 
requested.[Footnote 5] (See figure 1.) About $84 million of the 2010 
funds were designated by the Congress for specific projects, including 
$30 million for Transitional Jobs activities for ex-offenders, and 
another $5.5 million for competitive grants addressing the employment 
and training needs of young parents. According to agency documents, in 
2008 and 2009, the Congress similarly increased ETA's requested budget 
for pilots, demonstrations, and research, at the same time 
specifically designating how the majority of those funds would be 
used, including $4.9 million in 2008 and $5 million in 2009 for the 
young parents' demonstration. 

Figure 1: ETA's Combined Evaluations and Pilots, Demonstrations, and 
Research Budgets from 2007 to 2010: 

[Refer to PDF for image: stacked vertical bar graph] 

Year: 2007; 
Funds requested: $22.6 million; 
Funds designated for congressionally mandated projects: $0; 
Funds not designated for congressionally mandated projects: $19.6 
million; 
ETA's combined budgets: $19.6 million. 

Year: 2008; 
Funds requested: $20.0 million; 
Funds designated for congressionally mandated projects: $48.5 million; 
Funds not designated for congressionally mandated projects: $4.8 
million; 
ETA's combined budgets: $53.3 million. 

Year: 2009; 
Funds requested: $25.0 million; 
Funds designated for congressionally mandated projects: $46.3 million; 
Funds not designated for congressionally mandated projects: $9.4 
million; 
ETA's combined budgets: $55.7 million. 

Year: 2010; 
Funds requested: $69.1 million; 
Funds designated for congressionally mandated projects: $84.4 million; 
Funds not designated for congressionally mandated projects: $18.7 
million; 
ETA's combined budgets: $103.1 million. 

Source: GAO analysis of ETA budget documents. 

[End of figure] 

Key Elements of Sound Research and Evaluation Programs: 

While there is no single or ideal way for government agencies to 
conduct research, several leading national organizations have 
developed guidelines that identify key elements that promote a sound 
research program.[Footnote 6] These guidelines identify five elements 
as key: agency resources, professional competence, independence, 
evaluation policies and procedures, and evaluation plans. 

Resources. Research should be supported through stable, continuous 
funding sources and through special one-time funds for evaluation 
projects of interest to executive branch and congressional 
policymakers. 

Professional competence. Research should be performed by professionals 
with appropriate training and experience for the evaluation activity 
(such as performing a study, planning an evaluation agenda, reviewing 
evaluation results, or performing a statistical analysis). 

Independence. Although the heads of federal agencies and their 
component organizations should participate in establishing evaluation 
agendas, budgets, schedules, and priorities, the independence of 
evaluators must be maintained with respect to the design, conduct, and 
results of their evaluation studies. 

Evaluation policy and procedures. Each federal agency and its 
evaluation centers should publish policies and procedures and adopt 
quality standards to guide evaluations within its purview. Such 
policies and procedures should identify the kinds of evaluations to be 
performed and the criteria and administrative steps for developing 
evaluation plans and setting priorities, including selecting 
evaluation approaches to use, consulting experts, ensuring evaluation 
product quality, and publishing reports. 

Evaluation plans. Each federal agency should require its major program 
components to prepare annual and multiyear evaluation plans and to 
update these plans annually. The planning should take into account the 
need for evaluation results to inform program budgeting, 
reauthorization, agency strategic plans, program management, and 
responses to critical issues concerning program effectiveness. These 
plans should include an appropriate mix of short-and long-term studies 
to produce results for short-or long-term policy or management 
decisions. To the extent practical, the plans should be developed in 
consultation with program stakeholders. 

Furthermore, leading organizations, including the American Evaluation 
Association and the National Academy of Sciences, emphasize the need 
for research programs to establish specific policies and procedures to 
guide research activities. Based on several key elements identified by 
these organizations, we developed a framework comprised of five 
phases--agenda setting, selecting research, designing research, 
conducting research, and disseminating research results. (See figure 
2.) 

Figure 2: Key Phases of the Research Process: 

[Refer to PDF for image: illustration] 

Agenda setting: 
Identify general research priority areas where the agency will focus 
its attention. 

Selecting research: 
Identify the criteria agencies use to select the research studies that 
they will fund. 

Designing research: 
Develop research questions and appropriate methodologies for research 
projects. 

Conducting research: 
Perform and monitor the implementation of research projects. 

Disseminating research: 
Make available sound research to policymakers, researchers, government 
officials, and the general public. 

Source: GAO analysis of key elements of a sound research process 
identified by leading national organizations,including the American 
Evaluation Association and the National Academy of Sciences. 

[End of figure] 

Agenda setting. Agencies should establish a structured process for 
developing their research priorities. The process should identify how 
agencies set research priority areas and provide for updating the 
areas on a regular basis. The process should also allow for the 
consideration of critical issues and state how internal and external 
stakeholders will be included in developing the plan. 

Selecting research. At this phase, the process should identify how the 
research program's staff identifies and selects studies to fund, 
including the criteria it uses to make those decisions. Steps might 
describe how the staff assembles a list of potential studies, works 
with internal program offices, and makes final decisions. 

Designing research. During the design phase, the process should 
identify steps taken to select appropriate research approaches and 
methods and the safeguards in place to ensure appropriate tradeoffs 
are made between what is desirable and what is practical and between 
the relative strengths and weaknesses of different methods. 

Conducting research. At this stage, the process should include 
policies and procedures to guide the conduct of research. The process 
should ensure that key events, activities, and time frames are 
specified and that knowledgeable staff in the sponsoring agency 
monitor the implementation of the research. 

Disseminating research. This process should describe how research 
findings are made available to the public and disseminated to all 
potential users. These dissemination methods should include safeguards 
to ensure research findings are disseminated in a timely manner and 
are accessible through the Internet with user-friendly search and 
retrieval technologies. 

Research Terminology in This Report: 

In this report, we use several technical terms in describing ETA's 
research designs and study characteristics. (See table 1.) 

Table 1: Research and Evaluation Terms and Definitions: 

Term: Bias; 
Definition: The extent to which a measurement or a sampling or 
analytic method systematically underestimates or overestimates a value. 

Term: Cross-sectional data; 
Definition: Observations collected on subjects or events at a single 
point in time. 

Term: Descriptive study; 
Definition: Provides descriptive information about specific conditions 
or events, such as the number of individuals who received a particular 
type of job training. 

Term: Experimental design; 
Definition: Units of study are randomly assigned to a treatment group 
or to one or more comparison groups. The program's effects are 
estimated by comparing outcomes for the treatment group with outcomes 
for each comparison group. 

Term: External validity/generalizability (used interchangeably with 
generalizability); 
Definition: The extent to which a finding applies (or can be 
generalized) to persons, objects, settings, or times other than those 
that were the subject of study. 

Term: Longitudinal data; 
Definition: Sometimes called "time series data," observations 
collected over a period of time; the sample (instances or cases) may 
or may not be the same each time but the population remains constant. 

Term: Meta-analysis; 
Definition: A systematic approach to summarizing or analyzing 
evaluation findings across studies, usually quantitative, allowing 
analysis of variation in program effect sizes by study methods or 
program characteristics. 

Term: Random assignment; 
Definition: A method for assigning subjects to two or more groups by 
chance, as in the flip of a coin. 

Term: Secondary analysis; 
Definition: The reanalysis of data collected in previous studies or 
surveys to address new questions or use methods not previously 
employed. 

Term: Quasi-experimental design; 
Definition: Units of study are assigned to a treatment and a 
comparison group without random assignment. 

Term: Representative sample; 
Definition: A sample that has approximately the same distribution of 
characteristics as the population from which it was drawn. 

Term: Treatment group; 
Definition: The subjects of the intervention being studied. 

Source: GAO analysis of program evaluation and methodology documents. 

[End of table] 

ETA's Research Areas Generally Reflect Key Issues, but Some Studies 
Are of Limited Usefulness: 

Experts Thought ETA's 2007 to 2012 Research Plan Reflected Key Areas, 
but They Also Suggested New Ones for Future Research: 

Our expert panel generally considered ETA's research areas to be the 
right ones for the period the research plan covered. About three- 
fourths of the panel members reported that ETA's 2007 to 2012 research 
agenda reflected key employment and training issues to at least a 
moderate extent. However, a few experts commented that some of ETA's 
research areas may be too broad and lack specificity. 

The areas in ETA's 2007 to 2012 research plan covered a range of 
issues, from job training to postsecondary education. Table 2 
illustrates the scope of ETA's research areas. 

Table 2: ETA's Research Areas, with Examples of Topics Covered in the 
2007 to 2012 Research Plan: 

Research area: Integration of workforce and regional economic 
development; 
Example topic areas[A]: 
* Job training initiatives to produce high-skill, high-wage jobs; 
* Strategic partnerships between private business sector and public 
entities; 
* Regional and economic development. 

Research area: Methods of expanding U.S. workforce skills; 
Example topic areas[A]: 
* Rapid response services for dislocated workers; 
* Costs and benefits of apprenticeship; 
* Trade adjustment assistance. 

Research area: Increasing the Labor market participation of 
underutilized populations; 
Example topic areas[A]: 
* Enhanced services for the hard-to-employ; 
* Employment-centered programs for ex-offenders; 
* Current strategies to employ and retain older workers. 

Research area: Using state-level administrative data to measure 
progress and outcomes; 
Example topic areas[A]: 
* Examination of local workforce investment areas; 
* WIA and employment outcomes. 

Research area: Postsecondary education and job training; 
Example topic areas[A]: 
* The role of community colleges in workforce development and training; 
* Career advancement accounts; 
* Growing America through entrepreneurship. 

Research area: Unemployment Insurance; 
Example topic areas[A]: 
* UI benefits; 
* Characteristics of unemployed workers. 

Source: GAO analysis of ETA's research agenda. 

[A] This is not a complete or comprehensive list of examples of the 
types of research conducted by ETA under each research area. 

[End of table] 

With regard to the specific studies within these research areas, ETA 
invested most of its research and evaluation resources in work that 
focused on increasing the labor market participation of underutilized 
workers and on UI. Of the estimated $96 million that supported the 58 
research reports published between January 2008 and March 2010, more 
than half--about $56 million--funded research that addressed these two 
research areas. Other areas received far less funding. For example, 
funding for studies addressing the methods of expanding U.S. workforce 
skills and using state-level administrative data to measure progress 
and outcomes accounted for about $6.5 million, or about 6.7 percent of 
the cost of studies published during the period we examined. (See 
table 3.) Overall, the individual studies that ETA funded addressed a 
wide variety of issues and ranged in cost from about $15,000 to a high 
of about $22 million. 

Table 3: Estimated Funding by Research Area for Research Studies 
Disseminated between January 2008 and March 2010: 

Research area: Increasing the Labor market participation of 
underutilized populations; 
Total amount: $28.9 million. 

Research area: Unemployment Insurance; 
Total amount: $26.8 million. 

Research area: Postsecondary education and job training; 
Total amount: $12.4 million. 

Research area: Integration of workforce and regional economic 
development; 
Total amount: $9.9 million. 

Research area: Methods of expanding U.S. workforce skills; 
Total amount: $4.0 million. 

Research area: Using state-level administrative data to measure 
progress and outcomes; 
Total amount: $2.5 million. 

Research area: Other[A]; 
Total amount: $11.6 million. 

Research area: Total; 
Total amount: $96.1 million. 

Source: GAO analysis of study cost information provided by ETA. 

[A] "Other" includes studies that were not part of ETA's 2007 to 2012 
priorities. 

[End of table] 

In addition to the research areas covered in ETA's 2007 to 2012 
research plan, experts from our virtual panel suggested that ETA 
incorporate additional research areas in its future research agenda. 
Of the research areas identified, over half of our experts (28 of 39) 
ranked the identification of employment and training approaches that 
work, and for whom, as one of the top areas that ETA's future research 
should address. (See figure 3.) Without such focus, experts commented 
that it will be difficult to know how to improve the nation's 
workforce system. Other issues ranked at the top by experts included 
research on job creation strategies and the impact of long-term and 
short-term training. (See appendix III for more information on issue-
area rankings.) 

Figure 3: Issue Areas Ranked among the Top Three by Experts for ETA to 
Address in Future Research: 

[Refer to PDF for image: horizontal bar graph] 

Which areas would you rank as the top 3 areas that ETA should address 
in future research? 

Employment and training approaches that work and for whom: 
Number of respondents answering: 28. 

Job creation strategies: 
Number of respondents answering: 18. 

Impact of long-term and/or short-term training: 
Number of respondents answering: 18. 

Long-term outcomes of employment and training programs: 
Number of respondents answering: 15. 

Effective performance measurement systems: 
Number of respondents answering: 9. 

Issues related to Unemployment Insurance: 
Number of respondents answering: 8. 

Linkages between the public workforce system and economic development 
entities: 
Number of respondents answering: 6. 

Short-term outcomes of employment and training programs: 
Number of respondents answering: 4. 

Needs of different labor markets or industries: 
Number of respondents answering: 4. 

One-stop center management and operations: 
Number of respondents answering: 3. 

Value of various credentials: 
Number of respondents answering: 3. 

Employment and training strategies for various economic conditions: 
Number of respondents answering: 1. 

Source: GAO’s survey of ETA’s research priorities and dissemination 
methods. 

[End of figure] 

In addition to identifying overall employment and training areas, 
including issues related to UI, experts also identified specific 
aspects of the UI system that could be examined in ETA's future 
research. In particular, most experts (34 of 39 respondents) reported 
that it would be at least moderately important, in the future, for ETA 
to research the linkage between UI and employment and safety net 
programs, such as Temporary Assistance for Needy Families or the 
Supplemental Nutrition Assistance Program. (See figure 4.) This area 
of research may be particularly important given the role that these 
programs play in supporting individuals during economic downturns. In 
addition, many experts (24 of 39 respondents) mentioned that ETA 
should make the examination of the incentives and disincentives in the 
UI system a research priority, given the challenge of supporting 
unemployed workers during difficult economic times, while promoting 
self-sufficiency through employment. 

Figure 4: Unemployment Issues Frequently Cited by Experts as Important 
for ETA to Address in Future Research: 

[Refer to PDF for image: stacked horizontal bar graph] 

Linkage between UI programs and safety net programs: 
Number of respondents answering: 
Not at all important: 0; 
Somewhat important: 8; 
Moderately important: 8; 
Very important: 11; 
Extremely important: 15. 

Examination of incentives and disincentives of UI system: 
Number of respondents answering: 
Not at all important: 0; 
Somewhat important: 5; 
Moderately important: 9; 
Very important: 17; 
Extremely important: 7. 

Modernization of UI system: 
Number of respondents answering: 
Not at all important: 0; 
Somewhat important: 3; 
Moderately important: 13; 
Very important: 15; 
Extremely important: 6. 

Costs and benefits of UI system in different business cycles: 
Number of respondents answering: 
Not at all important: 1; 
Somewhat important: 6; 
Moderately important: 8; 
Very important: 18; 
Extremely important: 3. 

Accountability mechanisms in UI system: 
Number of respondents answering: 
Not at all important: 1; 
Somewhat important: 7; 
Moderately important: 12; 
Very important: 15; 
Extremely important: 2. 

Source: GAO’s survey of ETA’s research priorities and dissemination 
methods. 

[End of figure] 

Experts also reported that it is important to fund research on what 
works for selected population groups. Of the population groups 
identified, the experts on our virtual panel most often ranked the 
long-term unemployed, economically disadvantaged workers, and adults 
with low basic skills as the top populations on which to focus future 
research. Specifically, several experts commented that research could 
help to identify the challenges some of these groups face, as well as 
identify effective strategies that may help these population groups 
obtain employment. (See appendix III for a complete list of responses 
to these items.) 

In addition to population groups, experts also identified several 
employment and training programs that they believe warrant research 
attention. In particular, experts most often ranked three components 
of the WIA program--WIA Adult, WIA Dislocated Workers, and WIA Youth 
[Footnote 7]--as key to evaluate in ETA's future research. Among those 
three, WIA Adult was ranked the highest. (See appendix III for a 
complete list of experts' responses on employment and training 
programs to evaluate.) 

ETA's Research Studies Generally Answered the Questions Posed, but 
Their Usefulness Was Limited: 

Research organizations and academic institutions with responsibility 
for implementing ETA-funded research generally used methodologies 
appropriate for the questions posed, but the studies were not always 
useful for informing policy and practice. From January 2008 through 
March 2010, ETA published 17 large research and evaluation reports--14 
evaluations and 3 research reports--that each cost $1 million or more. 
Four of these reports were designed to demonstrate what works and for 
whom. Each of these four reports compared the employment-related 
outcomes of individuals or regions who participated in training or 
employment programs with the employment outcomes of similar 
individuals who did not participate in the programs. The remaining 13 
reports were descriptive and were not designed to assess program 
outcomes. 

In several studies we examined that cost $1 million or more, we found 
that, for a number of reasons, ETA's research studies were limited in 
their usefulness and in their ability to inform policy and practices. 
For example, in a study of the Prisoner Re-entry Initiative, 
shortcomings in the data collection phase limited the strength of the 
findings and, as a result, limited the study's opportunity to 
influence policy directions. Among other things, while the study 
provided information on employment-centered opportunities for ex-
offenders, the study relied on self-reported baseline data, did not 
account for differences across sites where services were received, 
lacked the capacity to record differences in the intensity of those 
services, and researchers failed to ensure that data collectors were 
properly trained. 

In another study, researchers did not control for bias in selecting 
participants, compromising their ability to draw conclusions about the 
cause and effect of program outcomes. Authors of this study on the 
Workforce Innovations in Regional Economic Development (WIRED) 
initiative acknowledged that the study would be unable to attribute 
outcomes to program services because it did not use random assignment 
in selecting participating regions. We have previously criticized ETA 
for failing to adequately provide for evaluating the effectiveness of 
its WIRED initiative.[Footnote 8] 

Moreover, some studies were limited due to observation periods that 
did not match the needs of the studies' objectives. For example, an 
evaluation of an entrepreneurship training project was unable to 
assess the effectiveness of the project in meeting its long-term goals 
of increasing business ownership and self-sufficiency because the time 
frames for the study were too short. In this study, data collection 
was limited to 18 months after participants were randomly assigned, a 
period far shorter than the 60-month period recommended by experts. 
[Footnote 9] (See appendix IV for additional information on the 
methodological characteristics of these studies.) 

Experts generally agreed that ETA's research had limited usefulness in 
informing policy and practice. Over one-third of the 39 experts 
reported that over the past 5 years, ETA's research informed 
employment and training policy and state and local practices to a 
little extent or not at all. (See figure 5.) Some experts commented 
that the design of these studies and the length of time to complete 
them and disseminate results reduced their usefulness. For example, 
many of the reports that we reviewed costing $1 million or more were 
multiyear projects that took, in most cases, about 3 to 5 years to 
complete. Some experts commented that the inclusion of shorter-length 
studies may be useful in times of rapidly changing economic 
conditions. At least one expert noted that some mixed-methods studies 
would be useful--studies that would allow for short-term interim 
findings that could facilitate changes in practice during the course 
of the research study. 

Figure 5: Expert Panel Views on the Extent to Which ETA's Research Has 
Informed Policy and Practice: 

[Refer to PDF for image: stacked horizontal bar graph] 

In the past 5 years, to what extent has ETA’s research informed 
employment and training policy: 
Number of respondents answering: 
Don’t know: 1; 
Not at all: 1; 
Little extent: 12; 
Some extent: 17; 
Moderate extent: 8; 
Great extent: 0. 

In the past 5 years, to what extent has ETA’s research informed state 
and local employment and training practices: 
Number of respondents answering: 
Don’t know: 4; 
Not at all: 1; 
Little extent: 15; 
Some extent: 13; 
Moderate extent: 5; 
Great extent: 1. 

Source: GAO’s survey of ETA’s research priorities and dissemination 
methods. 

[End of figure] 

Members of our expert panel stressed the importance of ETA 
incorporating varied methodological approaches into its future 
research proposals to best position the agency to address key 
employment and training issues. Twenty-seven of the 39 experts 
reported it was very important that ETA evaluate its pilots and 
demonstrations. Twenty-three reported that it was very important that 
more randomized experimental research designs be integrated into ETA's 
future research.[Footnote 10] (See figure 6.) While several experts 
noted that these randomized experiments will allow ETA to identify the 
effectiveness of particular interventions or strategies, at least one 
expert suggested that ETA should be strategic in choosing the 
interventions it tests more rigorously, basing those decisions on what 
appears most promising in preliminary studies. 

Figure 6: Methodological Approaches Frequently Cited by Experts as 
Very Important for ETA to Address in Future Research: 

[Refer to PDF for image: stacked horizontal bar graph] 

Evaluation of pilots and demonstrations: 
Number of respondents answering: 
Very unimportant: 0; 
Somewhat unimportant: 2; 
Neither important nor unimportant: 0; 
Somewhat important: 9; 
Very important: 27. 

Randomized experimental design: 
Number of respondents answering: 
Very unimportant: 0; 
Somewhat unimportant: 0; 
Neither important nor unimportant: 3; 
Somewhat important: 12; 
Very important: 23. 

Quasi-experimental design: 
Number of respondents answering: 
Very unimportant: 1; 
Somewhat unimportant: 1; 
Neither important nor unimportant: 3; 
Somewhat important: 17; 
Very important: 16. 

Analysis of administrative data for research purposes: 
Number of respondents answering: 
Very unimportant: 0; 
Somewhat unimportant: 1; 
Neither important nor unimportant: 5; 
Somewhat important: 19; 
Very important: 14. 

Descriptive studies: 
Number of respondents answering: 
Very unimportant: 0; 
Somewhat unimportant: 3; 
Neither important nor unimportant: 6; 
Somewhat important: 18; 
Very important: 12. 

Meta-analyses (i.e., review or analysis of prior research): 
Number of respondents answering: 
Very unimportant: 2; 
Somewhat unimportant: 2; 
Neither important nor unimportant: 11; 
Somewhat important: 18; 
Very important: 6. 

Secondary analysis of existing research data: 
Number of respondents answering: 
Very unimportant: 1; 
Somewhat unimportant: 4; 
Neither important nor unimportant: 13; 
Somewhat important: 17; 
Very important: 4. 

Source: GAO’s survey of ETA’s research priorities and dissemination 
methods. 

[End of figure] 

Furthermore, 16 of the 39 experts also reported that it is very 
important for ETA to consider including more quasi-experimental 
studies in the future. As previously discussed, such studies would 
include designs that compare outcomes between groups with similar 
characteristics, but do not use random assignment. By including more 
quasi-experimental designs, ETA may be able to better understand the 
link between services and outcomes in those settings where random 
assignment is not possible, ethical, or practical.[Footnote 11] 

ETA Has Taken Steps to Improve Its Research Program, but Additional 
Actions Are Needed: 

Labor Has Taken Steps to Reform Its Research Program: 

Labor has taken several steps designed to improve the way it conducts 
research, both at the department level and within ETA. 

Department-level efforts. Labor has changed the organizational 
structure of research within the department. In 2010, acknowledging 
the need for better and more rigorous evaluations to inform its 
policy, Labor established the Chief Evaluation Office to oversee the 
department's research and evaluation efforts. The office, which 
resides within the Office of the Assistant Secretary for Policy, has 
no authority to direct research within Labor's agencies, according to 
officials. It does, however, manage evaluations supported by funds 
from a departmentwide account, oversee departmentwide evaluations, and 
provide consultation to Labor agencies, including ETA. Specifically, 
the office is responsible for creating and maintaining a comprehensive 
inventory of past, ongoing, and planned evaluation activities within 
Labor and for ensuring that Labor's evaluation program and findings 
are transparent, credible, and accessible to the public. In fiscal 
year 2010, the Chief Evaluation Office had an estimated budget of $8.5 
million, and two of its four staff were on board by the beginning of 
fiscal year 2011. 

ETA efforts. ETA has recently made changes to some of its research 
practices--chief among them is the involvement of stakeholders and 
outside experts in the research process. We previously criticized ETA 
for failing to consistently involve a broad range of stakeholders, 
outside experts, or the general public in deciding what areas of 
research it should undertake. We recommended that ETA take steps to 
routinely involve outside experts in the research agenda-setting 
process.[Footnote 12] For the upcoming 2010 to 2015 research plan, ETA 
has awarded a grant to the Heldrich Center at Rutgers University to 
convene an expert panel to help inform the research plan. The center 
is expected to issue a report in May 2011 that outlines the panel's 
recommendations for research areas to include in the plan. In 
addition, ETA will work with other Labor agencies, as well as the 
Departments of Education and Health and Human Services, before 
finalizing its research agenda. Officials told us that they will also 
solicit public comments before the research plan is finalized. 

In addition to engaging stakeholders, ETA has also established a 
formal research process. As we previously reported, ETA developed and 
documented its research process in 2007.[Footnote 13] The agency's 
actions were in response to a request by the Office of Management and 
Budget (OMB) to establish more formal policies and procedures to guide 
its research--a request that came out of OMB's concerns about the 
manner in which ETA's research was being carried out. Prior to 2007, 
ETA lacked a documented research process, and its research was often 
conducted in an ad hoc manner. ETA's current research process 
identifies the steps, activities, and time frames it uses to carry out 
its research. Figure 7 illustrates critical components of ETA's 8-step 
research process. 

Figure 7: ETA's 8-Step Research Process: 

[Refer to PDF for image: illustrated table] 

Research stage: Agenda setting; 
Steps in ETA’s process: No formalized process; 
Critical components of each step: (none). 

Research stage: Selecting research; 
Steps in ETA’s process: ETA Step 1: Select projects for evaluation and 
pilots, demonstrations, and research (PD&R) funding; 
Critical components of each step: 
* Review ETA/Labor stated priorities and recommendations in the 5-year 
plan and solicit project ideas from within; 
* Review congressional mandates and requests to co-fund projects from 
other agencies; 
* Review existing project commitments; 
* Submit agenda to Assistant Secretary of ETA and finalize plan; 
* Prepare approval memos for approved projects. 

Research stage: Designing research; 
Steps in ETA’s process: ETA Step 2: Design Statements of Work (SOW), 
Requests for Proposal (RFP) and Solicitations for Grant Applications 
(SGA); 
Critical components of each step: 
* Review time frames, available funding, and methodology for each 
project idea and determine appropriate funding method; 
* Prepare draft reflecting research questions and methodology and 
obtain feedback from program offices; 
* Revise SOWs, RFPs, or SGAs, as appropriate, and obtain final 
approval from Assistant Secretary; 
* Submit to OMB for review any SGA, SOW, or RFP that costs $250,000 or 
more. 

Research stage: Designing research; 
Steps in ETA’s process: ETA Step 3: Contractor/grantee selection 
process; 
Critical components of each step: 
* Work with the Office of Grants and Contracts Management to select 
contractors or grantee; 
* Ensure contracts and/or grants are awarded in accordance with 
Federal Acquisition Regulations and departmental procurement rules. 

Research stage: Conducting research; 
Steps in ETA’s process: ETA Step 4: Initial project design (Post-award 
implementation); 
Critical components of each step: 
* Hold initial meetings to discuss study objectives and gather 
information; 
* Review contractor’s draft design report and approve? Review 
grantee’s implementation plans and approve; 
* Submit contractor’s final design report, and if applicable, submit 
report for peer review. 

Research stage: Conducting research; 
Steps in ETA’s process: ETA Step 5: Paperwork Reduction Act clearance 
process; 
Critical components of each step: 
* Prepare and submit an information collection request for internal 
review to reduce burdens associated with federal information 
collection and reporting requirements; 
* Inform the public about the information collection request and allow 
60 days for comment; 
* Finalize information collection request and submit to OMB for review 
and approval. 

Research stage: Conducting research; 
Steps in ETA’s process: ETA Step 6: Project monitoring; 
Critical components of each step: 
* Monitor project implementation and raise any issues to senior 
management; 
* Develop corrective action plans (as needed) and review/approve 
project progress and finance reports. 

Research stage: Disseminating research; 
Steps in ETA’s process: ETA Steps 7 and 8: Review and dissemination of 
reports. 
Critical components of each step: 
* Review and finalize report deliverables, working with contractors 
and internal offices to incorporate revisions; 
* Finalize research reports and obtain approval from the Assistant 
Secretary; 
* Produce Training and Employment Notice (TEN) and abstract to 
announce the availability of approved research reports; 
* Disseminate reports through ETA’s research Web site and in hard copy 
(as available); 
* Alert OMB of reports submitted for dissemination and not approved 
within 9 months. 

Source: GAO analysis of ETA documentation. 

[End of figure] 

ETA's process contains several of the key elements identified by 
leading organizations as important for guiding research activities. 
For example, the process includes specific steps the agency should 
take to identify the types of evaluations it will perform, as well as 
the administrative steps it should take to develop evaluation plans 
and select the research projects to fund.[Footnote 14] In addition, 
the process also specifies key events and time frames, and provides 
for monitoring the implementation of the research. For example, the 
process stipulates that ETA should alert OMB of research reports that 
have not been approved for dissemination within 9 months of being 
submitted and allows contractors to publicly release their research 
reports within those same time frames.[Footnote 15] 

Some Areas of ETA's Research Program Merit Further Attention: 

Despite ETA's efforts, more action is needed to improve its research 
program. While ETA has taken steps to document its research process, 
its process lacks specific details in some areas, creating ambiguities 
that could undermine efforts to adhere to a formal process. For 
example, as we previously reported, its process lacks clear criteria, 
such as a dollar threshold or a particular methodological design 
feature, for determining which projects require peer review. And while 
the process specifies the actions project officers should take if 
reports are not released in a timely manner, it does not specify the 
consequences for failing to do so. We previously recommended that ETA 
establish more specific processes, including time frames for 
disseminating research reports. ETA has taken some action, such as 
revising the performance standards for project officers to hold them 
accountable for meeting time frames, but these steps do not fully 
satisfy the recommendation because the changes are not yet reflected 
in the formal research process. 

Moreover, ETA's process is missing some critical elements that are 
needed to ensure that the current improvements become routine 
practices. 

* Consulting with the Chief Evaluation Officer. ETA's process lacks a 
formal provision requiring consultation with the newly established 
Chief Evaluation Officer at important points in the research process. 
For example, it contains no provision for consulting with the Chief 
Evaluation Officer when developing its annual list of research 
projects or when determining how ETA will invest its research and 
evaluation resources. Such consultation could help Labor better 
coordinate its research and evaluation efforts and better leverage its 
research funding. Moreover, the process contains no provision for 
involving the Chief Evaluation Officer in the early stages of 
developing its research projects. In the recent past, Labor officials 
told us that ETA has had difficulty developing requests for research 
and evaluation proposals that can pass OMB technical reviews. In 
particular, OMB has been critical of ETA's research designs because 
they failed to provide for adequate sample size and appropriate 
methodologies that are needed to obtain useful results. In addition, 
OMB has also expressed concerns with ETA's reliance on process 
evaluations rather than focusing on outcomes. These difficulties have 
resulted in delays in the research process. ETA has begun to consult 
with the Chief Evaluation Officer; however, these consultations are 
not a routine component in the formal process. 

* Setting the research agenda. ETA's current process, as documented, 
begins with phase two--selecting specific research studies--and misses 
the important first step of setting the overall research agenda. This 
first phase of the process should include the steps that ETA will take 
to establish its research priorities and to update them on a regular 
basis. It should also include provisions for ensuring critical issues 
are considered and internal and external stakeholders are included in 
developing the plan. Officials noted that they plan to incorporate the 
agenda-setting phase into its formal process, but have not yet done 
so. Setting the research agenda is key to ensuring that an appropriate 
mix of studies is included in future research. Failing to make this 
phase part of the formal process, including the specific steps to 
involve outside stakeholders that are currently under way, may leave 
ETA with little assurance that these efforts will continue in the 
future. 

Beyond ETA's process for conducting research, current research 
practices fall short of ensuring research transparency and 
accountability--essential elements of a sound research and evaluation 
program. The research program has few, if any, safeguards to protect 
it from undue influence. According to officials, at times in the past 
decade, many key research decisions have been made outside of the 
office that is responsible for research. For example, decisions about 
which research studies would and would not be publicly released were 
made at the highest levels within ETA, and the criteria used to make 
those decisions were unclear. Of the 34 reports that ETA released to 
the public in 2008, 20 had waited between 2 and 5 years to be approved 
for public release.[Footnote 16] Several reports that had experienced 
long delays had relatively positive and potentially useful findings 
for the workforce system, according to our analysis. Among the studies 
delayed by almost 5 years was an evaluation of labor exchange services 
in the one-stop system that found certain employment services to be 
highly cost-effective in some situations. Another study, delayed for 
about 3.5 years, was a compendium of past and ongoing experimental 
studies of the workforce system, including early findings and 
recommendations for future research. 

In our previous report, we noted that ETA's research and evaluation 
center lacked a specific mechanism to insulate it from undue 
influence. We reported that other federal agencies, such as the 
Department of Education's Institute of Education Sciences and the 
National Science Foundation, engage advisory bodies in the research 
process. While not without tradeoffs in terms of additional time and 
effort, such an approach may serve to protect the research program 
from undue influence and improve accountability. ETA is currently 
involving outside experts in setting the research agenda for 2010 to 
2015, but is not involving experts more broadly on research policy and 
practices. 

ETA Has Recently Included More Random Assignment Studies in Its 
Research Program: 

ETA has recently begun to include more rigorous studies in its ongoing 
research. Of the 10 large, ongoing studies costing $2 million or more 
that began during the period of our review, three--the WIA Gold 
Standard Evaluation of the Adult and Dislocated Worker Programs, the 
Impact Evaluation of the Young Parents Demonstration, and the 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II)--use experimental design with random assignment, as 
recommended by our experts.[Footnote 17] These ongoing studies--which 
range in cost from $2 million to nearly $23 million--have the 
potential to determine the effectiveness of some of the program 
services. Table 4 outlines some key characteristics of these three 
studies. 

Table 4: Characteristics of Three Ongoing Studies: 

Study: Purpose; 
WIA Gold Standard Evaluation: This experimental evaluation will 
provide rigorous, nationally representative estimates of the WIA's net 
impacts. The study will also produce a detailed description of the 
program's implementation and estimates of its benefits and costs. Key 
questions include: 
* Does access to WIA-intensive training services lead adult and 
dislocated workers to achieve better educational and employment 
outcomes than they would achieve in the absence of those services? 
* Does the effectiveness of WIA vary by population subgroup? 
* Is the effectiveness of WIA--and its components--commensurate with 
its costs? 
Impact Evaluation of the Young Parents Demonstration: This 
experimental evaluation will determine the impacts of supplemental 
short-term mentoring services to complement core workforce development 
programs, including the following: 
* Measure differences in services that the participants in the 
treatment and control groups receive; 
* Assess how services received differ across sites and among 
individuals in the treatment and control groups; 
* Identify the resource requirements involved in providing services 
and identify implementation issues that might have affected outcomes; 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II): This experimental evaluation will compare the 
outcomes of randomly assigned Project GATE II participants to the 
outcomes of individuals who did not receive Project GATE II services; 
* Measure the impacts of Project GATE on participants' labor market 
and self-employment outcomes; 
* Test the viability of providing Project GATE services to dislocated 
workers in rural areas over the age of 45; 
* Assess the linkages between the local one-stop centers' programs and 
other key groups providing services. 

Study: Scope; 
WIA Gold Standard Evaluation: Thirty local workforce investment areas, 
selected at random, stratified to ensure that they are representative 
in terms of the size, geography, and customer training rates. 
Following this selection, a subset of participants at each of the 30 
selected sites will be randomly assigned to one of three groups; 
Impact Evaluation of the Young Parents Demonstration: Five to 10 sites 
serving approximately 2,500 participants; 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II): Examination of four state grants. 

Study: Performance period; 
WIA Gold Standard Evaluation: 7 years. A subset of participants will 
be tracked and administered surveys at 15 and 36 months after random 
assignment; 
Impact Evaluation of the Young Parents Demonstration: 7 years. A 
sample of participants will be tracked and administered a follow-up 
survey approximately 12 months after random assignment. An additional 
survey will be conducted 30 months after random assignment; 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II): 3 years. A sample of project participants will be 
tracked and administered a follow-up survey approximately 12 months 
after random assignment. 

Study: Peer review; 
WIA Gold Standard Evaluation: The contractor will coordinate three 
meetings of the peer review board made up of 10 experts. ETA will 
select seven and the contractor three. The purpose of the board will 
be to review and comment on the project methodology and key project 
deliverables; 
Impact Evaluation of the Young Parents Demonstration: The contractor 
will convene up to four peer review panels composed of four to five 
researchers selected by ETA to review the evaluation methodology, 
analytic approaches and key deliverables; 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II): No requirements under contract. However, ETA 
officials told us that they have plans for an independent peer review. 

Study: Cost; 
WIA Gold Standard Evaluation: Total: $22,951,040 (Evaluation contract); 
Impact Evaluation of the Young Parents Demonstration: Total: 
$6,154,570; (Impact evaluation contract: $4.8 million; process 
evaluation: $717,000; and technical assistance contracts: $199,998 and 
$437,572); 
Evaluation of Project Growing America Through Entrepreneurship II 
(Project GATE II): Total: $2,014,996; (Impact evaluation: $1.5 million; 
technical assistance contracts: $485,000 and $29,996). 

Source: GAO analysis of ETA's documents. 

[End of table] 

Experimental designs with random assignment are an important means to 
understand whether various program components or services are 
effective, but they are also often difficult to design and implement 
in real-world settings.[Footnote 18] For example, in doing evaluations 
of employment and training programs, researchers often have difficulty 
in recruiting sample sizes large enough to detect meaningful outcomes. 
Because employment and training services may vary by location, and 
participants and their socio-economic environments are diverse, 
researchers must find ways to standardize procedures and treatment or 
service options. This often means recruiting relatively large samples. 
However, studies can be intrusive, often requiring program sites to 
change how they operate or to increase the resources available to 
participants. As a result, recruiting sites and sufficient numbers of 
participants may be difficult. 

Some of ETA's ongoing research studies face challenges in recruiting 
sample sizes large enough to meet the studies' objectives. For 
example, based on an OMB review, it was determined that the sample 
size for the Impact Evaluation of the Young Parents Demonstration had 
to be much larger in order to be able to assess the effectiveness of 
the program. At that time, ETA had already awarded two phases of 
grants. After consulting with the new Chief Evaluation Officer, ETA 
changed the number of participants required for the third phase from 
100 to 400 to obtain a sample large enough to address OMB's concerns 
and provide reliable estimates. However, grantees found it difficult 
to recruit even the 100 participants in the smaller sample, and it 
remains unclear whether they will be able to recruit all of the needed 
participants for the expanded design. 

The WIA Gold Standard Evaluation of the Adult and Dislocated Worker 
Programs: 

The WIA Gold Standard Evaluation illustrates ETA's difficulties in 
planning and executing large-scale, rigorous random assignment 
studies. WIA required that the Secretary of Labor conduct at least one 
multi-site control-group evaluation of the services and programs under 
WIA by the end of fiscal year 2005.[Footnote 19] ETA, however, delayed 
executing such a study, finally soliciting proposals in November 2007 
and awarding the contract in June 2008. The contractor submitted the 
initial design report in January 2009 and provided ETA with design 
revisions in May 2010. Officials tell us researchers will soon begin 
randomly assigning participants. ETA expects to receive the first 
report (on implementation) during the winter of 2012-2013 and the 
final report in 2015--10 years later than the WIA-mandated time frame. 

An OMB-selected panel of government experts--a technical working group 
composed of experts chosen by ETA, the evaluation contractor, and OMB 
staff--reviewed the original design for this study.[Footnote 20] 
Reviewers agreed the design contained many strengths, including: 

* the selection of an experimental design and a net impact approach; 

* the addition of a process or implementation study to evaluate 
differences among sites and other implementation and data collection 
issues; 

* the use of administrative and survey data; 

* the collection of information on services received by participants 
in the control group; and: 

* the collection of a wide range of outcome data for participants. 

However, reviewers raised several concerns regarding the design. For 
example, they were skeptical that the researchers would be able to 
obtain a sufficiently large and representative sample to draw 
meaningful conclusions about the effectiveness of the national 
workforce system. In order to maximize participation, officials told 
us that the Assistant Secretary of ETA made personal phone calls to 
all selected sites to emphasize the importance of the study, offered 
an open door policy to site officials to discuss issues, and followed 
up with an appreciation letter. Furthermore, ETA required the 
evaluation contractor to provide reimbursement payments to each site 
to offset implementation costs. 

Reviewers also had several other concerns regarding which groups would 
be included in the study and which groups would not. For example, some 
experts raised concerns about getting accurate information on the 
youth program because of the large, one-time infusion of funds the 
program received from the American Recovery and Reinvestment Act of 
2009. Reviewers were further concerned about the appropriateness of 
the evaluation objectives, the adequacy of steps taken to account for 
the effect of variation in services across sites on evaluation 
outcomes, and the external validity or generalizability of the study. 
In order to address these concerns, ETA made substantial adjustments 
to the original design. Specifically, ETA officials told us that based 
on an agreement with OMB, they instructed the contractor to drop the 
youth component from the evaluation and to focus only on the Adult and 
Dislocated Worker programs. While we received information on the new 
design and time frames for the WIA Gold Standard Evaluation, a 
finalized design plan is not yet available. According to officials, a 
finalized design is being prepared and will be available in June 2011. 

ETA Has Improved the Availability of Its Research but There Are 
Opportunities to Improve Its Search Page and Dissemination Methods: 

ETA Has Improved the Timeliness of Its Disseminated Research: 

ETA has recently improved the timeliness with which it disseminates 
its research reports. In our last review in January 2010, we found 
that 20 of the 34 reports that ETA disseminated in 2008 had been 
waiting 2 to 5 years to be publicly released.[Footnote 21] The 34 
research reports published by ETA in 2008 took, on average, 804 days 
from the time the report was submitted to ETA until the time it was 
posted to ETA's research database. By contrast, from 2009 through the 
first quarter of 2010, the average time between submission and public 
release was 76 days, which represents a more than 90 percent 
improvement in dissemination time compared with 2008. Additionally, 
there were no research reports in 2009 that were delayed for more than 
6 months. Further, the average time to dissemination improved 
significantly even when we excluded such outliers as the 20 research 
reports that were delayed for 2 years or more. Without these outliers, 
average time to dissemination for reports in 2008 was 100 days, 
indicating that time to dissemination in 2009 through the first 
quarter of 2010 still improved by 24 percent. 

ETA Has Improved Its Research Database but Lacks Plans for Assessing 
the Usability of Its Search Page: 

In 2010, ETA updated its online, Web-based search page in order to 
improve the usability of its research database--the primary tool for 
making ETA research available to policymakers and the general public. 
Officials told us that ETA's old Web-based search page was so error- 
prone and difficult to use that they opted to substitute it with one 
that had not yet completed internal testing. Our review of the old Web-
based search page confirmed that it had serious limitations and did 
not consistently return the same results. For example, when we 
searched the database by title for a known ETA research report titled 
Registered Apprenticeship, we successfully retrieved that report once. 
One month later, when we entered the exact same search terms, we were 
unable to retrieve the report. (For a more complete description of our 
analysis of ETA's search capability, see appendix II.) 

In our review of the updated Web-based search page, we found that the 
updates make the research database more usable. Labor officials told 
us they have taken other steps, as well, in efforts to improve its Web-
based search page. For example, they have developed a project plan 
that articulates the steps Labor will take to update ETA's Web-based 
search page. In addition, they have assigned a database administrator 
whose responsibilities include performing daily quality control spot 
checks in order to monitor performance and address technical problems. 

Although these changes have the potential to improve the usability of 
ETA's database, Labor has not developed a formal plan for assessing 
the overall effectiveness of its Web-based search page, including user 
satisfaction. Labor has made a number of changes to the way the page 
operates, but it has not provided users with tips on how to use the 
search functions, even though it is an industry standard to do 
so.[Footnote 22] Even skilled users who were familiar with the old Web-
based search page may need guidance on the exact meaning of new terms 
and functions now available on the new page. For example, the old Web- 
based search page gave users the option of searching by "key word," 
which is no longer an option in the new page. Instead, "key word" 
searches have been replaced with a variety of other options, including 
the ability to search the full text or abstract of a research report. 
However, there is no guidance on the Web site on how to use these new 
search options. Industry best practices suggest that a Web site 
evaluation plan that incorporates data from routine reviews of Web 
site performance and that assesses user satisfaction can help agencies 
ensure the usability of their Web sites. ETA currently has no plans to 
do such assessments. 

ETA Uses Various Methods to Disseminate Research, but Experts Suggest 
Additional Methods: 

At present, ETA's research database is the primary method that ETA 
uses to make its research reports publicly available, according to 
officials. In order to call attention to new reports available in that 
database, ETA sends a Training and Employment Notice, also commonly 
known as a TEN, to an e-mail list of the more than 40,000 subscribers 
who have signed up to receive them. ETA's research process specifies 
that for each new research report that is approved for dissemination, 
ETA must draft a TEN and an abstract before it is posted to ETA's Web 
site. Beyond posting reports to its database, ETA also distributes 
hard copies of some of its research reports. 

In addition to electronic distribution, ETA also organizes various 
presentations to disseminate its research findings. These 
presentations, however, are done on an ad hoc basis. As mentioned in 
our prior report, ETA hosted a research conference in 2009 to present 
some of its research findings, renewing a practice that had been 
discontinued in 2003. As ETA looks to the future, officials tell us 
they will plan and organize similar research conferences as resources 
permit. In addition to these research conferences, ETA's regional 
offices occasionally hold smaller, regional conferences as well. 
Beyond these formal conferences, ETA also hosts an internal briefing 
series at Labor headquarters where research contractors present their 
findings to various officials. For each of these briefings, ETA has a 
list of stakeholders that it invites, including various Labor 
officials, outside agency officials, congressional staff, and other 
outside stakeholders. 

Experts who participated in our virtual panel provided their views on 
the effectiveness of different methods for disseminating research 
reports, and several of those rated more highly are methods currently 
employed by ETA. (See figure 8.) 

Figure 8: Experts' Opinion on the Effectiveness of Various Research 
Dissemination Methods: 

[Refer to PDF for image: stacked horizontal bar graph] 

E-mail notifications announcing newly released research: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 3; 
Moderately effective: 6; 
Very effective: 19; 
Extremely effective: 11. 

Searchable database of papers on ETA’s Web site: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 3; 
Moderately effective: 10; 
Very effective: 10; 
Extremely effective: 16. 

Distributing a one-page summary of the research findings from each 
report: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 4; 
Moderately effective: 9; 
Very effective: 17; 
Extremely effective: 9. 

Briefings at ETA for external audiences (including stakeholders and 
policymakers): 
Number of respondents answering: 
No response or left blank: 1; 
Not at all effective: 0; 
Somewhat effective: 7; 
Moderately effective: 9; 
Very effective: 14; 
Extremely effective: 8. 

Publicizing research findings through press articles, industry 
publications, etc.: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 7; 
Moderately effective: 13; 
Very effective: 14; 
Extremely effective: 5. 

Presenting at outside conferences: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 6; 
Moderately effective: 14; 
Very effective: 15; 
Extremely effective: 4. 

National ETA-sponsored conferences: 
Number of respondents answering: 
No response or left blank: 2; 
Not at all effective: 0; 
Somewhat effective: 5; 
Moderately effective: 16; 
Very effective: 13; 
Extremely effective: 3. 

Regional ETA-sponsored conferences: 
Number of respondents answering: 
No response or left blank: 1; 
Not at all effective: 0; 
Somewhat effective: 4; 
Moderately effective: 19; 
Very effective: 13; 
Extremely effective: 2. 

Publishing articles in journals: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 0; 
Somewhat effective: 14; 
Moderately effective: 11; 
Very effective: 8; 
Extremely effective: 6. 

Contracting with relevant organizations or associations to disseminate 
research: 
Number of respondents answering: 
No response or left blank: 1; 
Not at all effective: 0; 
Somewhat effective: 7; 
Moderately effective: 17; 
Very effective: 10; 
Extremely effective: 4. 

Webinars: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 1; 
Somewhat effective: 9; 
Moderately effective: 17; 
Very effective: 9; 
Extremely effective: 3. 

Distributing a compendium of ETA’s research: 
Number of respondents answering: 
No response or left blank: 0; 
Not at all effective: 2; 
Somewhat effective: 7; 
Moderately effective: 20; 
Very effective: 5; 
Extremely effective: 5. 

Videos or podcasts: 
Number of respondents answering: 
No response or left blank: 2; 
Not at all effective: 4; 
Somewhat effective: 13; 
Moderately effective: 12; 
Very effective: 8; 
Extremely effective: 0. 

Social media (such as Twitter): 
Number of respondents answering: 
No response or left blank: 3; 
Not at all effective: 11; 
Somewhat effective: 12; 
Moderately effective: 10; 
Very effective: 3; 
Extremely effective: 0. 

Source: GAO’s survey of ETA’s research priorities and dissemination 
methods. 

[End of figure] 

Most of the experts (30 of the 39 respondents) in our panel reported 
that using e-mail notifications, a searchable database of ETA papers, 
and briefings at ETA for external audiences (including stakeholders 
and policymakers) would be very effective or extremely effective 
approaches for disseminating research. In addition, a majority of the 
experts (26 of the 39 respondents) in our panel reported that 
publishing one-page summaries of research findings, not currently done 
by ETA, would be very or extremely effective. 

Conclusions: 

ETA plays an important role in developing workforce policies and 
helping to identify the most effective and efficient ways to train and 
employ workers for jobs in the twenty-first century. With the current 
economic crisis and high unemployment rates, ETA's role has become 
even more critical. The agency has made some improvements in its 
research program, even since our last review a year ago. But officials 
can do more to ensure that the progress continues in the years to come. 

ETA has taken a major step forward in establishing a formal research 
process--one that documents most actions that must be taken in the 
life cycle of a research or evaluation project. But, it is missing 
some key elements that could help ensure the continuation of current 
practices. While ETA is currently using outside advisory bodies to 
help it establish its research agenda, the formal process does not 
include the agenda-setting phase. Officials tell us they have plans to 
incorporate this phase in the future, and we urge them to do so. 
Without a formalized agenda-setting phase, ETA may miss opportunities 
to ensure that its research agenda addresses the most critical 
employment and training issues and that outside stakeholders are 
routinely involved. Moreover, ETA's process has not formalized the now 
ad hoc advisory role of the Chief Evaluation Officer. Absent the 
routine involvement of the Chief Evaluation Officer at key steps in 
the process, ETA may find it difficult to ensure that research 
proposals are asking the right questions, are methodologically sound, 
and that they can quickly pass critical OMB reviews. 

ETA's research findings are now available to the public on its Web 
site in far less time than it took in 2008. Despite this clear 
improvement, ETA has not taken the necessary steps to ensure that 
research products remain readily available to the public. The decision 
regarding what and when to make research publicly available is left in 
the hands of too few, and the process lacks needed safeguards to 
ensure transparency and accountability. Absent safeguards, key 
research decisions may again be made in ways that harm the credibility 
of the program and prevent important research findings from being used 
to inform policy and practice. 

ETA's Web-based search page is the primary means ETA uses to make the 
research studies it funds readily available to the public. And, while 
ETA has improved the functionality of its Web site, no effort has been 
made to ensure that the problems that plagued the system in the past 
do not recur. Absent such efforts, ETA will have little assurance that 
its research findings are actually available to users. 

Recommendations for Executive Action: 

To improve ETA's research program, we recommend that the Secretary of 
Labor require ETA to take the following three actions: 

* Formally incorporate into its research process the routine 
involvement of the Chief Evaluation Officer at key milestones, 
including at the development of ETA's annual research agenda and 
spending priorities, as well as at the early stages of developing 
specific research projects. 

* Develop a mechanism to enhance the transparency and accountability 
of ETA's research program. For example, such a mechanism might include 
involving advisory bodies or other entities outside ETA, in efforts to 
develop ETA's research policies and processes. 

* Develop a formal plan for ensuring that ETA's research products are 
easily accessible to stakeholders and to the general public through 
its searchable database. Such a plan could involve requiring Labor to 
assess the overall effectiveness of its Web-based search page, 
including user satisfaction with search features. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the Department of Labor for 
review and comment. Labor provided written comments, which are 
reproduced in appendix VII. In addition, ETA provided technical 
comments, which we incorporated where appropriate. 

In its response, Labor generally agreed with our findings and all of 
our recommendations, noting its ongoing efforts in support of the 
recommendations. 

* Regarding our recommendation to formally incorporate into its 
research process the routine involvement of the Chief Evaluation 
Officer at key research milestones, Labor noted that it is currently 
taking steps to do so. Officials reported that they have worked 
closely with this office in various aspects of its research, including 
discussing research, demonstration projects, and evaluations in the 
early stages of development and plans to continue this collaboration 
in the future. However, ETA's comments did not discuss plans to update 
its documentation on the formal research process. We found in our 
review that involving the Chief Evaluation Officer was not an official 
component of ETA's documented research process, and it occurred on an 
ad hoc basis. As ETA moves forward, we urge the agency to modify its 
current research process and document the involvement of the Chief 
Evaluation Officer at critical research milestones. 

* Regarding our recommendation for ETA to develop a mechanism to 
enhance the transparency and accountability of its research program, 
officials cited several steps they are taking to improve the program, 
including involving outside experts in the development of their 5-year 
research plan and establishing advisory and peer review groups to 
review major evaluations. While officials note they plan to engage 
outside experts in broader research policies and processes, we 
encourage ETA to formalize this involvement. Moreover, we encourage 
ETA to continue to move forward in its efforts to further clarify 
components of its research process that are not well defined, 
including, for example, the criteria to be used when deciding when a 
peer review should be performed. 

* Regarding our recommendation to develop a formal plan to ensure that 
disseminated research is easily accessible to stakeholders and the 
general public, officials cited specific steps the agency has taken to 
improve its Web-based research database. While these actions are a 
step in the right direction, we believe that it is still important for 
Labor to develop a formal and comprehensive plan to ensure that 
disseminated research continues to be accessible to the public. 

Furthermore, Labor expressed concerns about how we characterized the 
agency's budget for pilots, demonstrations, and research. Recognizing 
these concerns, we made changes to the report to better capture the 
amount of funding ETA has available for research. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the 
appropriate congressional committees, the Secretary of Labor, and 
other interested parties. The report will also be available at no 
charge on GAO's Web site at [hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions about this report, please 
contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made key contributions to 
this report are listed in appendix VIII. 

Signed by: 

George A. Scott: 
Director, Education, Workforce and Income Security Issues: 

[End of section] 

Appendix I: Status of Prior GAO Recommendations to the Department of 
Labor, as of January 2011: 

GAO's recommendations: The Secretary of Labor should 
Department of Labor's response: [Empty]; 
Status: [Empty]. 

GAO's recommendations: The Secretary of Labor should take steps to 
clarify the Employment and Training Administration's (ETA) revised 
organizational structure and ensure that the Office of Policy 
Development and Research (OPDR) reports directly to ETA's Assistant 
Secretary; 
Department of Labor's response: The Department of Labor (Labor) does 
not agree with this recommendation as written. According to Labor 
officials, the Administrator of OPDR currently reports to the Deputy 
Assistant Secretary, not directly to ETA's Assistant Secretary. 
However, Labor officials acknowledge that important functions such as 
research and evaluation should not have too many intermediary 
reporting layers. To facilitate communication, officials further noted 
that the OPDR Administrator, the Deputy Assistant Secretary, and the 
Chief Evaluation Officer meet on a monthly basis with the Assistant 
Secretary to discuss evaluations; 
Status: Labor has taken no action. 

GAO's recommendations: The Secretary of Labor should provide 
sufficient authority to ETA's research and evaluation center to plan, 
conduct, and disseminate research; 
Department of Labor's response: Labor agrees with this recommendation, 
but authority to make key decisions still resides with the Office of 
the Assistant Secretary for ETA. OPDR currently provides 
recommendations to this office regarding plans for conducting and 
disseminating research. In an effort to improve evaluations 
departmentwide, the Secretary of Labor recently established the Chief 
Evaluation Office to monitor evaluation efforts across the department. 
OPDR has begun to work informally with the Chief Evaluation Officer 
and the Chief Economist to design and implement research and 
evaluation projects; 
Status: Labor's actions do not completely satisfy recommendation. 

GAO's recommendations: The Secretary of Labor should direct ETA's 
research and evaluation center to establish more specific processes, 
including time frames for dissemination of research, to promote 
transparency and accountability; 
Department of Labor's response: Labor agrees with this recommendation. 
ETA reports that it has taken some steps to establish more specific 
processes regarding dissemination of research, citing changes in 
performance standards for project officers. However, our 
recommendation would make broader changes to their research process 
and no such changes are reflected in the documents the agency provided; 
Status: Labor's actions do not completely satisfy recommendation. 

GAO's recommendations: The Secretary of Labor should create an 
information system to track research projects at all phases to ensure 
timely completion and dissemination; 
Department of Labor's response: Labor agrees with this recommendation. 
Officials report that they have begun working on a centralized, 
electronic tracking system for its research projects. However, the 
work is still under way and no time frames have been provided for its 
completion. Currently, OPDR uses an Excel document to keep inventory 
of all research, demonstration, and evaluation projects; 
Status: Labor's actions do not completely satisfy recommendation. 

GAO's recommendations: The Secretary of Labor should instruct ETA's 
research and evaluation center to develop processes to routinely 
involve outside experts in setting its research agenda and to the 
extent required, do so consistent with the Federal Advisory Committee 
Act; 
Department of Labor's response: Labor agrees with this recommendation. 
OPDR has taken steps to engage outside experts in setting its 5-year 
research plan for 2011 and collaborate with the research and 
evaluation centers of other federal agencies, such as the Departments 
of Education and Health and Human Services. OPDR also plans to convene 
an expert panel, solicit public comments, and incorporate feedback 
from its 2009 Reemployment Research Conference and its 2010 ETA 
Reemployment Summit. However, despite these current efforts, OPDR has 
not formally incorporated them in its standard research process; 
Status: Labor's actions do not completely satisfy recommendation. 

Source: GAO analysis of GAO-10-243. 

[End of table] 

[End of section] 

Appendix II: Scope and Methodology: 

We were asked to review the Employment and Training Administration's 
(ETA) research program to better understand its approach to conducting 
and disseminating research. Specifically, we answered the following 
research questions: (1) To what extent do ETA's research priorities 
reflect key national employment and training issues and how useful 
were the studies funded under them? (2) What steps has ETA taken to 
improve its research program? (3) How has ETA improved, if at all, the 
availability of its research since our last review in January 2010 and 
what other steps could ETA take to further ensure its research 
findings are readily available? 

To answer our research questions, we convened a virtual panel using a 
modified Delphi technique to obtain selected employment and training 
experts' opinions on ETA's research priorities and dissemination 
methods. We also visited two workforce agencies in Pennsylvania and 
Virginia that are implementing two of ETA's ongoing research studies 
to learn about implementation issues and how research is being 
conducted. In addition, we reviewed 58 ETA-funded research and 
evaluation reports disseminated between January 2008 and March 2010 
and assessed the methodological soundness of completed studies that 
cost $1 million or more. We also reviewed ETA's ongoing studies that 
cost $2 million or more. To determine the availability of ETA's 
research, we measured the time between when the final version of a 
research report was submitted to ETA's Office of Policy Development 
and Research (ODPR) and when it was posted on ETA's Web site. We also 
conducted a series of systematic searches to test the reliability of 
ETA's research database. Furthermore, we interviewed Department of 
Labor (Labor) and ETA officials to better understand ETA's research 
capacity, processes, and the use of research findings to inform policy 
and practice. Lastly, we reviewed relevant agency documents and 
policies, as well as relevant federal laws. 

Web-Based Expert Panel: 

We convened a nongeneralizable Web-based virtual panel of 41 
employment and training experts to obtain their opinions on ETA's 
research priorities and dissemination methods. We employed a modified 
version of the Delphi method to organize and gather these experts' 
opinions.[Footnote 23] To encourage participation by our experts, we 
promised that responses would not be individually identifiable and 
that results would generally be provided in summary form. To select 
the panel, we asked several employment and training experts, on the 
basis of their experience and expertise, to identify other experts who 
were knowledgeable of ETA and the research it conducts and 
disseminates. After receiving nominations from experts, we reviewed 
the list to ensure that it reflected a range of perspectives and 
backgrounds, including academics, researchers, and consultants. 

Our Delphi process entailed two survey phases. (See appendix V for a 
copy of our phase I and phase II questionnaires.) In phase I, which 
ran from June 22, 2010, to August 9, 2010, we asked the panel to 
respond to five open-ended questions about ETA's research priorities 
and dissemination methods. We developed these questions based on our 
study objectives and pretested them with four experts by phone to 
ensure the questionnaire was clear, unbiased, and did not place an 
undue burden on respondents. All relevant changes were made before we 
deployed the first Web-based questionnaire to experts. 

After the experts completed the open-ended questions in the first 
questionnaire, we performed a content analysis of the responses in 
order to identify the most important issues raised by our experts. Two 
members of our team categorized experts' responses to each of the 
questions. Any disagreements were discussed until consensus was 
reached. Thirty-six of the 41 panelists selected completed phase I of 
the survey (about an 88 percent response rate). Those that did not 
complete phase I were allowed to participate in phase II. (For a list 
of experts who participated in phase I and phase II, see appendix VI.) 

The experts' responses to phase I were used to create the questions 
for phase II. In phase II, we gathered more specific information on 
ETA's research and dissemination practices. Phase II, which ran from 
October 29, 2010, to December 14, 2010, consisted of 16 follow-up 
questions where panelists were asked to either rank or rate the 
responses from phase I. We pretested the questionnaire for the second 
phase with three experts to ensure the clarity of the instrument. We 
conducted two of our expert pretests in-person and one by phone. 
Thirty-nine of the 41 experts completed phase II (about a 95 percent 
response rate). 

Site Visits to Workforce Agencies Implementing ETA-Funded Research 
Studies: 

To further enhance our understanding of how ETA conducts its research, 
we visited two workforce agencies that are implementing ETA's ongoing 
research studies. First, we visited the Lancaster County Workforce 
Investment Board in Lancaster, Pa., which received funding from ETA to 
implement the Young Parents Demonstration project.[Footnote 24] This 
project provides educational and occupational skills training to 
promote employment and economic self-sufficiency for mothers, fathers, 
and expectant mothers ages 16 to 24. Second, we visited the Northern 
Virginia Workforce Investment Board in Falls Church, Va., which 
received funding from ETA to implement the second round of the Project 
Growing America Through Entrepreneurship, also referred to as Project 
GATE II.[Footnote 25] This grant helps dislocated workers aged 50 and 
over obtain information, classroom training, one-to-one technical 
assistance, counseling, and financial assistance to establish new 
businesses in order to help them start and sustain successful self- 
employment. 

We selected these workforce agencies because they were identified by 
ETA as having active research projects in the implementation stage. 
These sites also required minimum travel expenditure. During our site 
visits, we toured each workforce agencies' facilities and used a 
semistructured interview protocol to interview the project director 
and staff about their role and responsibilities, the extent to which 
they communicate with ETA, and whether or not they face challenges 
with regards to implementation. At the Lancaster County site, we 
participated in an informal on-site lunch forum where local community 
programs that the agency partners with talked with us about their 
collaboration with the program. At the Northern Virginia GATE II site, 
we observed a focus group operated by the program to facilitate 
information-sharing among participants. 

After our site visits, we conducted phone interviews with the 
contractors that received funding from ETA to evaluate the outcomes of 
two research projects. Specifically, we interviewed the Urban 
Institute, which evaluates the Young Parents Demonstration project, 
and IMPAQ International, which evaluates Project GATE II. Both 
projects include an experimental component with control and comparison 
groups to determine the effects of program interventions on 
participants. During our interviews we used a semistructured 
questionnaire and asked questions to better understand their roles and 
responsibilities for the project, the extent to which they communicate 
with ETA, and whether or not they experience methodological and 
implementation challenges. 

Analysis of Methodological Characteristics of ETA: 

We reviewed the 58 research and evaluation reports that ETA 
disseminated between January 2008 and March 2010 and assessed the 
methodological soundness of 11 completed studies that cost $1 million 
or more. In addition, we reviewed 10 ongoing studies costing $2 
million or more to determine if research practices or the soundness of 
research designs had changed over time. We categorized the 58 studies 
disseminated between January 2008 and March 2010 by study type, cost, 
and research area. For the larger studies costing $1 million or more, 
we analyzed key characteristics including design features, scope, 
generalizability, and the appropriateness of analytical approaches and 
statistical procedures. These studies were analyzed independently by 
two analysts and the agreement between their ratings was 100 percent. 
(For results of this analysis, see appendix VI.) 

Analysis of the Timeliness and Effectiveness of ETA's Dissemination 
Activities: 

To evaluate the availability of ETA's research, we measured the time 
between when the final version of a research report was submitted to 
ODPR and when it was posted on ETA's Web site. Specifically, we 
measured the dissemination time frames for reports posted in 2008 and 
compared that with the dissemination time frames for reports issued 
between January 2009 through March 2010. In addition, we conducted a 
series of systematic searches to test the reliability of ETA's Web- 
based research database. To perform our searches, we selected a random 
sample of 30 reports from the 312 reports available on ETA's research 
database at the time of our review. Specifically, we tested a variety 
of search functions available at the time of our review to determine 
the extent to which research reports could be easily retrieved on 
ETA's research database. These functions included searches by title, 
keywords, author, and/or dates. We classified a report as retrievable 
if it appeared anywhere in our search results. We conducted our 
initial searches between June 30, 2010, and July 6, 2010. A second 
round of searches was conducted between August 6, 2010, and August 10, 
2010. Further, we interviewed Labor and ETA officials to learn more 
about the search capabilities of ETA's research database and the 
processes used to address errors and implement changes. Finally, we 
interviewed officials to gather information about ETA's dissemination 
methods, including its current techniques and future plans for 
disseminating research reports. 

Interviews with Labor and ETA Officials: 

To better understand the agency's research capacity, we interviewed 
ETA officials and reviewed relevant agency and budget documentation. 
Similarly, to obtain information on ETA's research process and how 
research findings are used to inform employment and training policy 
and practice, we interviewed officials and reviewed agency 
documentation, including relevant policies and procedures that guide 
ETA's research. We also reviewed relevant federal laws. 

We conducted this performance audit from March 2009 through March 2011 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix III: The Panel's Ratings of Key Employment and Training 
Issues, Populations, and Programs That ETA Should Address in Its 
Future Research: 

In our Delphi phase II Web-based questionnaire, we asked the panel of 
experts to rate and rank the key employment and training issues, 
populations, and programs that ETA should address in its future 
research. These issues were identified by the panel during phase I. 
For our analysis, we calculated basic descriptive statistics on these 
issues, which are presented in tables 5 through 7. 

Table 5: Descriptive Statistics on Key Employment and Training Issues 
ETA Should Address in Its Future Research: 

The responses in this table are based on the following questions: 

Question 1: Taking into account ETA's limited resources, how 
important, if at all, is it for ETA to address the following in future 
research? (Q1): 

Question 2: Among the areas you checked as at least moderately 
important in question 1, which would you rank as the top 3 areas that 
ETA should address in future research. (Please rank only 3 areas, with 
1 as your top priority area, 2 as your second highest area, etc.) 

Long-term outcomes of employment and training programs: 
Responses to question 2: Ranked 1: 9; 
Responses to question 2: Ranked 2: 3; 
Responses to question 2: Ranked 3: 3; 
Responses to question 2: Ranked 1, 2, or 3: 15; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 4; 
Responses to question 1: Moderately important: 4; 
Responses to question 1: Very Important: 14; 
Responses to question 1: Extremely important: 17. 

Short-term outcomes of employment and training programs: 
Responses to question 2: Ranked 1: 2; 
Responses to question 2: Ranked 2: 0; 
Responses to question 2: Ranked 3: 2; 
Responses to question 2: Ranked 1, 2, or 3: 4; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 5; 
Responses to question 1: Moderately important: 12; 
Responses to question 1: Very Important: 13; 
Responses to question 1: Extremely important: 9. 

Value of various credentials: 
Responses to question 2: Ranked 1: 0; 
Responses to question 2: Ranked 2: 1; 
Responses to question 2: Ranked 3: 2; 
Responses to question 2: Ranked 1, 2, or 3: 3; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 7; 
Responses to question 1: Moderately important: 12; 
Responses to question 1: Very Important: 15; 
Responses to question 1: Extremely important: 4. 

Impact of long-term and/or short-term training: 
Responses to question 2: Ranked 1: 5; 
Responses to question 2: Ranked 2: 9; 
Responses to question 2: Ranked 3: 4; 
Responses to question 2: Ranked 1, 2, or 3: 18; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 0; 
Responses to question 1: Moderately important: 5; 
Responses to question 1: Very Important: 18; 
Responses to question 1: Extremely important: 16. 

Job creation strategies, such as providing wage subsidies, tax 
credits, or public service employment: 
Responses to question 2: Ranked 1: 2; 
Responses to question 2: Ranked 2: 9; 
Responses to question 2: Ranked 3: 7; 
Responses to question 2: Ranked 1, 2, or 3: 18; 
Responses to question 1: Not at all important: 1; 
Responses to question 1: Somewhat important: 2; 
Responses to question 1: Moderately important: 5; 
Responses to question 1: Very Important: 18; 
Responses to question 1: Extremely important: 13. 

Employment and training approaches that work and for whom: 
Responses to question 2: Ranked 1: 9; 
Responses to question 2: Ranked 2: 11; 
Responses to question 2: Ranked 3: 8; 
Responses to question 2: Ranked 1, 2, or 3: 28; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 1; 
Responses to question 1: Moderately important: 1; 
Responses to question 1: Very Important: 18; 
Responses to question 1: Extremely important: 19. 

One-stop center management and operations: 
Responses to question 2: Ranked 1: 0; 
Responses to question 2: Ranked 2: 1; 
Responses to question 2: Ranked 3: 2; 
Responses to question 2: Ranked 1, 2, or 3: 3; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 10; 
Responses to question 1: Moderately important: 14; 
Responses to question 1: Very Important: 12; 
Responses to question 1: Extremely important: 3. 

Effective performance measurement systems: 
Responses to question 2: Ranked 1: 2; 
Responses to question 2: Ranked 2: 3; 
Responses to question 2: Ranked 3: 4; 
Responses to question 2: Ranked 1, 2, or 3: 9; 
Responses to question 1: Not at all important: 0; 
Responses to question 1: Somewhat important: 4; 
Responses to question 1: Moderately important: 10; 
Responses to question 1: Very Important: 16; 
Responses to question 1: Extremely important: 9. 

Linkages between the public workforce system and economic development 
entities: 
Responses to question 2: Ranked 1: 4; 
Responses to question 2: Ranked 2: 1; 
Responses to question 2: Ranked 3: 1; 
Responses to question 2: Ranked 1, 2, or 3: 6; 
Responses to question 1: Not at all important: 2; 
Responses to question 1: Somewhat important: 11; 
Responses to question 1: Moderately important: 11; 
Responses to question 1: Very Important: 8; 
Responses to question 1: Extremely important: 7. 

Needs of different labor markets or industries: 
Responses to question 2: Ranked 1: 1; 
Responses to question 2: Ranked 2: 1; 
Responses to question 2: Ranked 3: 2; 
Responses to question 2: Ranked 1, 2, or 3: 4; 
Responses to question 1: Not at all important: 2; 
Responses to question 1: Somewhat important: 6; 
Responses to question 1: Moderately important: 15; 
Responses to question 1: Very Important: 14; 
Responses to question 1: Extremely important: 2. 

Employment and training strategies for various economic conditions: 
Responses to question 2: Ranked 1: 0; 
Responses to question 2: Ranked 2: 0; 
Responses to question 2: Ranked 3: 1; 
Responses to question 2: Ranked 1, 2, or 3: 1; 
Responses to question 1: Not at all important: 1; 
Responses to question 1: Somewhat important: 4; 
Responses to question 1: Moderately important: 13; 
Responses to question 1: Very Important: 15; 
Responses to question 1: Extremely important: 6. 

Issues related to unemployment insurance: 
Responses to question 2: Ranked 1: 5; 
Responses to question 2: Ranked 2: 0; 
Responses to question 2: Ranked 3: 3; 
Responses to question 2: Ranked 1, 2, or 3: 8; 
Responses to question 1: Not at all important: 2; 
Responses to question 1: Somewhat important: 5; 
Responses to question 1: Moderately important: 10; 
Responses to question 1: Very Important: 12; 
Responses to question 1: Extremely important: 10. 

Source: GAO's survey of ETA's research priorities and dissemination 
methods. 

[End of table] 

Table 6: Descriptive Statistics on Key Populations ETA Should Address 
in Its Future Research: 

The responses in this table are based on the following questions: 

Question 3: How important, if at all, would it be for ETA to fund 
research that focuses on what works for the following populations 
given its resource constraints? 

Question 4: Among the populations you checked as at least moderately 
important in question 3, which would you rank as the top 5 populations 
on which ETA should fund research in the future? (Please rank only 5 
populations, with 1 as your top population, 2 as your second highest 
population, etc.) 

Short-term unemployed: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 1; 
Responses to question 4: Ranked 4: 0; 
Responses to question 4: Ranked 5: 0; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 3; 
Responses to question 3: Not at all important: 3; 
Responses to question 3: Somewhat important: 16; 
Responses to question 3: Moderately important: 8; 
Responses to question 3: Very Important: 7; 
Responses to question 3: Extremely important: 2. 

Long-term unemployed: 
Responses to question 4: Ranked 1: 15; 
Responses to question 4: Ranked 2: 10; 
Responses to question 4: Ranked 3: 6; 
Responses to question 4: Ranked 4: 1; 
Responses to question 4: Ranked 5: 1; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 33; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 1; 
Responses to question 3: Moderately important: 0; 
Responses to question 3: Very Important: 19; 
Responses to question 3: Extremely important: 19. 

Dislocated workers: 
Responses to question 4: Ranked 1: 5; 
Responses to question 4: Ranked 2: 4; 
Responses to question 4: Ranked 3: 4; 
Responses to question 4: Ranked 4: 2; 
Responses to question 4: Ranked 5: 3; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 18; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 4; 
Responses to question 3: Moderately important: 7; 
Responses to question 3: Very Important: 19; 
Responses to question 3: Extremely important: 9. 

Older workers: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 2; 
Responses to question 4: Ranked 4: 1; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 7; 
Responses to question 3: Not at all important: 2; 
Responses to question 3: Somewhat important: 11; 
Responses to question 3: Moderately important: 9; 
Responses to question 3: Very Important: 13; 
Responses to question 3: Extremely important: 3. 

Veterans: 
Responses to question 4: Ranked 1: 1; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 2; 
Responses to question 4: Ranked 4: 0; 
Responses to question 4: Ranked 5: 0; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 5; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 9; 
Responses to question 3: Moderately important: 13; 
Responses to question 3: Very Important: 11; 
Responses to question 3: Extremely important: 4. 

Economically disadvantaged workers: 
Responses to question 4: Ranked 1: 11; 
Responses to question 4: Ranked 2: 8; 
Responses to question 4: Ranked 3: 8; 
Responses to question 4: Ranked 4: 0; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 29; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 2; 
Responses to question 3: Moderately important: 2; 
Responses to question 3: Very Important: 22; 
Responses to question 3: Extremely important: 13. 

Racial and ethnic minorities: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 0; 
Responses to question 4: Ranked 4: 4; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 8; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 8; 
Responses to question 3: Moderately important: 13; 
Responses to question 3: Very Important: 13; 
Responses to question 3: Extremely important: 3. 

Immigrants: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 0; 
Responses to question 4: Ranked 4: 0; 
Responses to question 4: Ranked 5: 1; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 3; 
Responses to question 3: Not at all important: 2; 
Responses to question 3: Somewhat important: 10; 
Responses to question 3: Moderately important: 12; 
Responses to question 3: Very Important: 11; 
Responses to question 3: Extremely important: 2. 

Adults with low basic skills: 
Responses to question 4: Ranked 1: 3; 
Responses to question 4: Ranked 2: 3; 
Responses to question 4: Ranked 3: 6; 
Responses to question 4: Ranked 4: 3; 
Responses to question 4: Ranked 5: 7; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 22; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 2; 
Responses to question 3: Moderately important: 6; 
Responses to question 3: Very Important: 19; 
Responses to question 3: Extremely important: 10. 

Workers with physical and mental disabilities: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 0; 
Responses to question 4: Ranked 3: 0; 
Responses to question 4: Ranked 4: 3; 
Responses to question 4: Ranked 5: 1; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 4; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 8; 
Responses to question 3: Moderately important: 15; 
Responses to question 3: Very Important: 13; 
Responses to question 3: Extremely important: 1. 

In-school youth: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 0; 
Responses to question 4: Ranked 3: 3; 
Responses to question 4: Ranked 4: 0; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 5; 
Responses to question 3: Not at all important: 2; 
Responses to question 3: Somewhat important: 14; 
Responses to question 3: Moderately important: 12; 
Responses to question 3: Very Important: 9; 
Responses to question 3: Extremely important: 1. 

Out-of-school youth: 
Responses to question 4: Ranked 1: 4; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 4; 
Responses to question 4: Ranked 4: 8; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 20; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 3; 
Responses to question 3: Moderately important: 10; 
Responses to question 3: Very Important: 17; 
Responses to question 3: Extremely important: 9. 

Ex-offenders: 
Responses to question 4: Ranked 1: 0; 
Responses to question 4: Ranked 2: 2; 
Responses to question 4: Ranked 3: 3; 
Responses to question 4: Ranked 4: 3; 
Responses to question 4: Ranked 5: 2; 
Responses to question 4: Ranked 1,2, 3, 4, or 5: 10; 
Responses to question 3: Not at all important: 0; 
Responses to question 3: Somewhat important: 5; 
Responses to question 3: Moderately important: 10; 
Responses to question 3: Very Important: 17; 
Responses to question 3: Extremely important: 6. 

Source: GAO's survey of ETA's research priorities and dissemination 
methods. 

[End of table] 

Table 7: Descriptive Statistics on Key Employment and Training 
Programs ETA Should Address in Its Future Research: 

The responses in this table are based on the following questions: 

Question 6: How important, if at all, is it for ETA to evaluate the 
following key employment or training programs (excluding UI)? 

Question 7: Given a limited amount of resources, which three of the 
key employment or training programs that you checked as at least 
moderately important in question 6 should ETA address in future 
research? (Please rank only 3 employment or training programs, with 1 
as your top program, 2 as your second highest program, etc.) 

WIA Adult: 
Responses to question 7: Ranked 1: 12; 
Responses to question 7: Ranked 2: 9; 
Responses to question 7: Ranked 3: 7; 
Responses to question 7: Ranked 1, 2, or 3: 28; 
Responses to question 6: Not at all important: 0; 
Responses to question 6: Somewhat important: 2; 
Responses to question 6: Moderately important: 11; 
Responses to question 6: Very Important: 11; 
Responses to question 6: Extremely important: 13. 

WIA Dislocated Worker: 
Responses to question 7: Ranked 1: 8; 
Responses to question 7: Ranked 2: 11; 
Responses to question 7: Ranked 3: 3; 
Responses to question 7: Ranked 1, 2, or 3: 22; 
Responses to question 6: Not at all important: 0; 
Responses to question 6: Somewhat important: 3; 
Responses to question 6: Moderately important: 8; 
Responses to question 6: Very Important: 13; 
Responses to question 6: Extremely important: 13. 

WIA Youth: 
Responses to question 7: Ranked 1: 7; 
Responses to question 7: Ranked 2: 6; 
Responses to question 7: Ranked 3: 4; 
Responses to question 7: Ranked 1, 2, or 3: 17; 
Responses to question 6: Not at all important: 0; 
Responses to question 6: Somewhat important: 4; 
Responses to question 6: Moderately important: 11; 
Responses to question 6: Very Important: 15; 
Responses to question 6: Extremely important: 8. 

Wagner-Peyser employment Service: 
Responses to question 7: Ranked 1: 3; 
Responses to question 7: Ranked 2: 3; 
Responses to question 7: Ranked 3: 6; 
Responses to question 7: Ranked 1, 2, or 3: 12; 
Responses to question 6: Not at all important: 0; 
Responses to question 6: Somewhat important: 7; 
Responses to question 6: Moderately important: 13; 
Responses to question 6: Very Important: 9; 
Responses to question 6: Extremely important: 7. 

Job Corps: 
Responses to question 7: Ranked 1: 1; 
Responses to question 7: Ranked 2: 3; 
Responses to question 7: Ranked 3: 1; 
Responses to question 7: Ranked 1, 2, or 3: 5; 
Responses to question 6: Not at all important: 1; 
Responses to question 6: Somewhat important: 11; 
Responses to question 6: Moderately important: 12; 
Responses to question 6: Very Important: 10; 
Responses to question 6: Extremely important: 2. 

Trade Adjustment Assistance: 
Responses to question 7: Ranked 1: 2; 
Responses to question 7: Ranked 2: 2; 
Responses to question 7: Ranked 3: 3; 
Responses to question 7: Ranked 1, 2, or 3: 7; 
Responses to question 6: Not at all important: 2; 
Responses to question 6: Somewhat important: 5; 
Responses to question 6: Moderately important: 15; 
Responses to question 6: Very Important: 11; 
Responses to question 6: Extremely important: 5. 

Veterans' Employment and Training: 
Responses to question 7: Ranked 1: 1; 
Responses to question 7: Ranked 2: 2; 
Responses to question 7: Ranked 3: 4; 
Responses to question 7: Ranked 1, 2, or 3: 7; 
Responses to question 6: Not at all important: 1; 
Responses to question 6: Somewhat important: 9; 
Responses to question 6: Moderately important: 16; 
Responses to question 6: Very Important: 6; 
Responses to question 6: Extremely important: 6. 

Apprenticeship program: 
Responses to question 7: Ranked 1: 3; 
Responses to question 7: Ranked 2: 5; 
Responses to question 7: Ranked 3: 4; 
Responses to question 7: Ranked 1, 2, or 3: 3; 
Responses to question 6: Not at all important: 7; 
Responses to question 6: Somewhat important: 14; 
Responses to question 6: Moderately important: 1; 
Responses to question 6: Very Important: 7; 
Responses to question 6: Extremely important: 12. 

Source: GAO's survey of ETA's research priorities and dissemination 
methods. 

[End of table] 

[End of section] 

Appendix IV: Characteristics of Research Studies Disseminated between 
January 2009 and March 2010 That Cost $1 Million or More: 

Report name: 1. Implementation Analysis of the High Growth Job 
Training Initiative (HGJI) Programs; 
ETA report number: 2008-10; 
ETA research area: Integration of the workforce and regional economic 
development; 
Cost: $1,500,000; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Implementation review; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 2. Workforce Investment Act Non-Experimental Net Impact 
Evaluation: Final Report; 
ETA report number: 2009-10; 
ETA research area: Using state-level administrative data to measure 
progress and outcomes; 
Cost: $1,000,000; 
Study type: Evaluation; 
Study design: Quasi-experimental; 
Methods and techniques: 
* Other (comparison v. treatment group using propensity score 
matching); 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 3. Evaluation of the Prisoner Re-Entry Initiative - Final 
Report; 
ETA report number: 2009-3; 
ETA research area: Increasing the labor market participation of 
underutilized populations; 
Cost: $1,204,078; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Post-intervention only data collection; 
* Other (some characteristics collected before the intervention, but 
these were not used to make comparisons); 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 4. Implementing the National Fund for Workforce 
Solutions: The Baseline Evaluation Report; 
ETA report number: 2009-21; 
ETA research area: Integration of the workforce and regional economic 
development; 
Cost: $1,000,000; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Implementation review; 
* Secondary analysis; 
Consistency of analytical approach with research question: No basis to 
judge; 
Generalizability of findings: No response. 

Report name: 5. The Power of Partnership: American Regions 
Collaborating for Economic Competitiveness (Generation I WIRED Interim 
Eval); 
ETA report number: 2009-18; 
ETA research area: Integration of the workforce and regional economic 
development; 
Cost: $3,433,478[A]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Implementation review; 
* Survey; 
* Secondary analysis; 
* Other (social network analysis); 
Consistency of analytical approach with research question: No; 
Generalizability of findings: No. 

Report name: 6. Early Implementation of Generation I of the Workforce 
Innovation in Regional Development (WIRED) Initiative, 2007 Interim 
Evaluation Report; 
ETA report number: 2008-03; 
ETA research area: Integration of the workforce and regional economic 
development; 
Cost: [Empty]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Implementation review; 
* Survey; 
* Secondary analysis; 
* Other (social network analysis); 
Consistency of analytical approach with research question: No; 
Generalizability of findings: No. 

Report name: 7. Nurturing America's Growth in the Global Marketplace: 
An Interim Report on the Evaluation of Generations II and III of WIRED; 
ETA report number: 2009-19; 
ETA research area: Integration of the workforce and regional economic 
development; 
Cost: $3,345,036; 
Study type: Evaluation; 
Study design: Quasi-experimental; 
Methods and techniques: 
* Survey; 
* Secondary analysis; 
* Other (comparison v. treatment group analysis ); 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 8. Recent Changes in the Characteristics of Unemployed 
Workers; 
ETA report number: 2009-13; 
ETA research area: Unemployment insurance; 
Cost: $6,507,262[B]; 
Study type: Research; 
Study design: Descriptive; 
Methods and techniques: 
* Prestest/post-test intervention data collection; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: Yes. 

Report name: 9. Trends in the Structure of the Labor Market and 
Unemployment; 
ETA report number: 2009-09; 
ETA research area: Unemployment insurance; 
Cost: [Empty]; 
Study type: Research; 
Study design: Descriptive; 
Methods and techniques: 
* Secondary analysis; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: Yes. 

Report name: 10. Reemployment and Eligibility Assessment (REA) Study 
FY 2005 Initiative; 
ETA report number: 2008-02; 
ETA research area: Unemployment insurance; 
Cost: [Empty]; 
Study type: Research; 
Study design: 
* Experimental (randomized control trials; 
* Descriptive; 
Methods and techniques: 
* Survey; 
* Secondary analysis; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 11. Growing America Through Entrepreneurship: Findings 
from the Evaluation of Project GATE; 
ETA report number: 2008-08; 
ETA research area: Postsecondary education and job training; 
Cost: $11,400,000; 
Study type: Evaluation; 
Study design: Experimental (randomized control trials); 
Methods and techniques: 
* Survey; 
* Secondary analysis; 
* Prestest/post-test intervention data collection; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 12. Evaluation of Youth Build Offender Grants; 
ETA report number: 2009-11; 
ETA research area: Increasing the labor market participation of 
underutilized populations; 
Cost: $1,151,449; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
* Other (data collected on characteristics upon entry and outcome 
characteristics collected after completion); 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 13. Initial Implementation of the Trade Act; 
ETA report number: 2009-14; 
ETA research area: Methods of expanding U.S. workforce skills; 
Cost: $10,453,957[C]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
* Implementation review; 
* Secondary analysis; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 14. Assessment, Case Management, and Post-Training 
Assistance for TAA Participants; 
ETA report number: 2009-15; 
ETA research area: Methods of expanding U.S. workforce skills; 
Cost: [Empty]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 15. Linkages Between TAA, One-Stop Career Center Partners 
and Economic Development Agencies; 
ETA report number: 2009-16; 
ETA research area: Methods of expanding U.S. workforce skills; 
Cost: [Empty]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 16. Rapid Response and TAA; 
ETA report number: 2009-17; 
ETA research area: Methods of expanding U.S. workforce skills; 
Cost: [Empty]; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Report name: 17. Youth Offender Demonstration Project Process 
Evaluation Round Two; 
ETA report number: 2004-10; 
ETA research area: Increasing the labor market participation of 
underutilized populations; 
Cost: $1,734,393; 
Study type: Evaluation; 
Study design: Descriptive; 
Methods and techniques: 
* Case studies; 
* Secondary analysis; 
Consistency of analytical approach with research question: Yes; 
Generalizability of findings: No. 

Source: GAO analysis of ETA's disseminated studies between January 
2008 and March 2010. 

[A] Two of four reports produced or expected from the WIRED Generation 
I Evaluation cost a total of $3,433,478 for the full evaluation. 

[B] Three of five reports produced from the UI Benefits Study cost a 
total of $6,507,262 for the full study. 

[C] Four of approximately 10 reports produced or expected from the 
National Evaluation of Trade Adjustment Assistance cost a total of 
$10,453,957 for the full evaluation. 

[End of table] 

[End of section] 

Appendix V: Delphi Phase I and Phase II Questionnaires: 

GAO's Review of ETA's Research Priorities and Dissemination
Methods: Phase 1: 

U.S. Government Accountability Office: 

Introduction: 

The Congress has asked the U.S. Government Accountability Office
(GAO), an independent agency that gathers information for the 
Congress, to review the Employment and Training Administration's (ETA) 
research activities. 

As part of this review, we have asked you to participate in this 
virtual expert panel to gather your input on national priorities in 
the area of employment and training, and on ETA's research agenda and 
dissemination practices. As you know, you have been identified as an 
expert in employment and training services or programs, service 
delivery approaches, and/or evaluation methodologies for these 
programs. Your participation allows you an opportunity to assist us in 
informing ETA's future work. 

The virtual expert panel will consist of multiple phases. For the first
stage of the virtual panel, we ask that you complete a brief, five-
question survey. We will use the group's responses to these questions 
to create a more detailed questionnaire in the second stage. This 
questionnaire will allow you to consider information provided by other 
members of the group. No information will be attributed to specific 
individuals. 

Thank you very much for agreeing to participate in the panel. Your 
participation in our study is essential for us to provide requested 
information to the Congress. Our work in this review will cover other 
areas in addition to those addressed in the virtual panel. To learn 
more about these additional areas,click here. 

Contact: 

If your response will be delayed or if you have any questions, please 
call or e-mail Ashanta Williams or Kathy White. Click here for their 
contact information. 

Confidentiality of Responses: 

We will not release individually identifiable information from this 
survey while our work is in progress unless compelled by law or 
required to do so by the Congress. Once the report is issued, we will 
destroy the file that links names to individual responses. While the 
results will generally be provided in summary form in our report, 
individual answers may be discussed, but they will not include any 
information that could be used to identify individual respondents. 
Names of participants will be listed in our final report; however, 
names will not be associated with responses. Furthermore, your 
comments and answers will not be provided to other participants during 
any stage in the process. 

Navigation, Exiting and Printing the Survey: 

To learn more about navigation, exiting and printing the survey, 
please click here. 

Key Employment and Training Issues: 

1. In your opinion, what are the key employment and training issues that
ETA's research agenda should address? Please consider population 
groups, programs, and/or methodological approaches in your response. 

ETA's research agenda: 

2. We understand that ETA is currently reviewing its research agenda 
and that its research priorities may change moving forward. However, 
we would like you to consider the priority areas addressed in the 2007-
2012 research agenda along with selected examples of the topics 
covered in the agenda listed below. 

Note: This is not a complete or comprehensive list of ETA's
disseminated research studies. We compiled this list to give you a 
better sense of the types of research conducted by ETA. 

ETA Research Priority Areas, with Example Topics Covered in ETA's 
Research Agenda: 

1. Integration of Workforce and Regional Economic Development: 
* Job training initiatives to produce high-skill, high-wage jobs; 
* Strategic partnerships between private business sector and public 
entities; 
* Regional and economic development. 

2. Methods of Expanding U.S. Workforce Skills: 
* Rapid response services for dislocated workers; 
* Costs and benefits of apprenticeship; 
* Trade adjustment assistance. 

3. Increasing the Labor Market Participation of Underutilized 
Populations: 
* Enhanced services for the hard-to-employ; 
* Employment-centered programs for ex-offenders; 
* Current strategies to employ and retain older workers. 

4. Using State-Level Administrative Data to Measure Progress and
Outcomes: 
* Examination of local workforce investment areas o WIA and employment 
outcomes. 

5. Post-Secondary Education and Job Training: 
* The role of community colleges in workforce development and training; 
* Career advancement accounts; 
* Growing America through entrepreneurship. 

6. Unemployment Insurance: 
* Unemployment insurance benefits
* Characteristics of unemployed workers
* Trends in the structure of the labor market and unemployment 

Based on your knowledge, how, if at all, do you think ETA's 2007-2012 
research agenda reflects the key employment and training issues you 
identified in question one?	 

3. In your opinion, how well does ETA's research inform policy and 
practice? Please consider ongoing research studies, of which you are 
familiar, during the past 5 years.	
		
4.	What additional thoughts, if any, do you have about ETA's research or
research agenda?	
				
ETA's	Dissemination Methods:	 

5.	What dissemination methods do you think would allow ETA to 
effectively share its research findings?	
		
Completed: 

6. Have you finished this questionnaire?
(Check Only One Answer)		
1.	Yes. 
2.	No. 

Print. 
Submit. 
Cancel. 

[End of section] 

Menu: 

Display list of headings: 

GAO's Review of ETA's	Research Priorities and	
Dissemination	Methods: Phase 2: 

Prioritization for Future	Work:
	
Prioritization of ETA's	Research:	 

Policy and Practice:	 

ETA's Dissemination Methods: 

Demographics:	 

Summary: 

View and print a summary of your responses.	 

Help: 

If you have any questions about this GAO survey, please contact 
Ashanta Williams at (202) 512-5110 (williamsar@qao.qoy) or	Kathy White 
at (202)	512-8512 (whitek@gao.gov).	 

Progress: 

Navigate: 

GAO's Review of ETA's Research Priorities and Dissemination Methods: 
Phase 2: 

U.S. Government Accountability Office: 

In phase 1, we asked you to provide information on: 

* key employment and training issues that ETA's research agenda should 
address; 

* the extent to which ETA's 2007-2012 research agenda addresses these 
key employment and training issues; and 

*	dissemination methods that would allow ETA to effectively share its
	research findings. 
	
This is Phase II of our virtual panel. We developed our questions in 
this 	phase using the responses of 36 experts who provided input in 
Phase 1. Your responses will allow us to gather more specific 
information for the Congress on ETA's research and dissemination 
practices. 

We thank you again for your participation in our panel. We appreciate 
your time and found the information that you provided extremely 
helpful. 

Contact: 

If your response will be delayed or if you have any questions, please 
contact	Ashanta Williams at (202) 512-5110 (williamsa@agao.gov) or 
Kathy White at (202) 512-8512 (whitek@gao.gov). 

Directions: 

* Click the "Exit" button at the bottom of the screen if you wish to 
save your responses and continue at a later time. Always use the 
"Exit" button to close the survey to avoid losing data you have 
entered. 

Confidentiality of Responses: 

We will not release individually identifiable information from this 
survey while our work is in progress unless compelled by law or 
required to do so by the Congress. Once the report is issued, we will 
destroy the file that links names to individual responses. While the 
results will generally be provided in summary form in our report, 
individual answers may be discussed, but they will not include any 
information that could be used to identify individual respondents. 
Names of participants may be listed in our final report; however, 
names will not be associated with responses. Furthermore, your 
comments and answers will not be provided to other participants during 
any stage in the process. 

Prioritization for Future Work: 

The questions in this section are based on respondents' answers to the 
question:		 

In your opinion, what are the key employment and training issues that		
ETA's research agenda should address?	 

1. Taking into account ETA's limited resources, how important, if at 
all, is it for ETA to address the following in future research?		 

a. Long-term outcomes of	Employment and Training Programs: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

b. Short-term outcomes of	Employment and Training Programs: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

c. Value of various credentials: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

d. Impact of long-term and/or short-term training: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

e. Job creation strategies, such as providing wage subsidies, tax 
credits,	or public service employment: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

f. Employment and training approaches that work and for whom: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

g. One-stop center management and operations: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

h. Effective performance measurement systems: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 
	
i. Linkages between the public	workforce system and economic 
development entities: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

j. Needs of different labor markets or industries:	
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

k. Employment and training strategies for various economic conditions: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

1. Issues related to Unemployment Insurance: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

2. Among the areas you checked at least moderately important in 
question 1, which would you rank as the top 3 areas that ETA should 
address in future research. (Please rank only 3 areas, with 1 as your 
top priority area, 2 as your second highest area, etc.)	  

a. Long-term outcomes of Employment and Training Programs: 
Rank: 

b. Short-term outcomes of Employment and	Training Programs: 
Rank: 

c. Value of various credentials: 
Rank: 

d. Impact of long-term and/or short-term training: 
Rank: 

e. Job creation strategies, such as providing wage subsidies, tax 
credits, or public service employment: 
Rank: 

f. Employment and training approaches that work and for whom: 
Rank: 

g. One-stop center management and operations: 
Rank: 

h. Effective performance measurement systems: 
Rank: 

i. Linkages between the public workforce system and economic 
development entities: 
Rank: 

j. Needs of different labor markets or industries: 
Rank: 

k. Employment and training strategies for various economic conditions: 
Rank: 

I. Issues related to Unemployment Insurance: 
Rank: 

3. How important, if at all, would it be for ETA to fund research that 
focuses on what works for the following populations given its resource 
constraints?		 

a. Short-term unemployed: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

b. Long-term unemployed: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

c. Dislocated workers: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

d. Older workers: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

e. Veterans: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

f. Economically disadvantaged workers: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

g. Racial and ethnic minorities: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

h. Immigrants: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

i. Adults with low basic skills: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

j. Workers with physical and mental disabilities: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

k. In-school youth (ages 14 - 21): 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

I. Out-of-school youth: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

m. Ex-offenders: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

4. Among the populations you checked as least moderately important in 
question 3, which would you rank as the top 5 populations on which ETA 
should fund research in the future, please rank only 5 populations, 
with 1 as your top population, 2 as your second highest population, 
etc.)	 

a. Short-term unemployed: 
Rank: 

b. Long-term unemployed: 
Rank: 

c. Dislocated workers: 
Rank: 

d. Older workers: 
Rank: 

e. Veterans: 
Rank: 

f. Economically disadvantaged workers: 
Rank: 

g. Racial and ethnic minorities: 
Rank: 

h. Immigrants: 
Rank: 

i. Adults with low basic skills: 
Rank: 

j. Workers with physical and mental disabilities: 
Rank: 

k. In-school youth (ages 14 - 21): 
Rank: 

I. Out-of-school youth: 
Rank: 

m. Ex-offenders: 
Rank: 

5. How important are each of the following Unemployment Insurance (UI) 
system issues for ETA to include in future	research efforts?	 

a. Examination of the incentives and disincentives in the UI system: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 
	
b. Modernization of UI system (such as extending coverage to include 
additional income groups):	
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 
						
c. Benefits and costs of UI system in different business cycles: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

d. Accountability mechanisms in the UI system (such as data accuracy, 
enforcement of work search requirement): 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

e. Linkage between UI and employment and training programs or safety-
net programs (such as SNAP,	TANF): 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

6. How important, if at all, is it for ETA to evaluate the following 
key employment or training programs (excluding Up?	 

a. WIA Adult: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

b. WIA Dislocated Worker: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

c. WIA Youth: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

d. Wagner-Peyser Employment	Service: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

e. Job Corps: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

f. Trade Adjustment Assistance: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

g. Veterans' Employment and Training: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

h. Apprenticeship program: 
Extremely important:	
Very important:	
Moderately important:	
Somewhat important:	
Not at all important:	
No response: 

7. Given a limited amount of resources, which three of the key 
employment or training programs that you checked as at least 
moderately important in question 6 should ETA address in future 
research. Please rank only 3 employment or training programs, with 1 
as your top program, 2 as your second highest program, etc.)		 

a. WIA Adult: 
Rank: 

b. WIA Dislocated Worker: 
Rank: 

c. WIA Youth: 
Rank: 

d. Wagner-Peyser Employment Service: 
Rank: 

e. Job Corps: 
Rank: 

f. Trade Adjustment Assistance: 
Rank: 

g. Veterans' Employment and Training: 
Rank: 

h. Apprenticeship program: 
Rank: 

8. How important, if at all, is it for ETA to integrate the following 
into its future research proposals? 

a. Randomized experimental design: 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

b. Quasi-experimental design: 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

c. Descriptive studies (including program implementation and	
operations): 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

d. Analysis of administrative	data for research purposes: 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

e. Meta-analyses (including literature reviews or quantitative 
analyses of prior research): 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 
		
f. Secondary analysis of existing	research data: 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

g. Evaluation of pilots and	demonstrations: 
Very important:	
Somewhat important:	
Neither important nor unimportant: 
Somewhat unimportant:	
Very unimportant:	
No response: 

Prioritization of ETA's Research: 

9. Overall, to what extent do you feel ETA's 2007-2012 research agenda 
reflect key employment and training issues? (For a reminder of the 
topics covered during	the 2007-2012 research agenda click here.)		
* Great extent; 		
* Moderate extent; 
* Some extent; 
* Little extent; 
* Not at all; 
* No response. 

Policy and Practice: 
The questions in this section are based on respondents' answers to the 
following questions: 

"In your opinion, how well does ETA's research inform policy and 
practice? Please consider ongoing research studies, of which you are 
familiar, during the past 5 years."	 

"What additional thoughts, if any, do you have about ETA's research or 
research agenda?"	 

10. In the past 5 years, to what extent has ETA's research informed 
employment and training policy?
* Great extent; 
* Moderate extent; 
* Some extent; 
* Little extent; 
* Not at all; 
* Don't know; 
* No response. 

11. In the past 5 years, to what extent has ETA's research informed 
state and local employment and training practices?
* Great extent; 
* Moderate extent; 
* Some extent; 
* Little extent; 
* Not at all; 
* Don't know; 
* No response. 

ETA's Dissemination Methods: 

The questions in this section are based on respondents' answers to the 
following question: 

"What dissemination methods do you think would allow ETA to 
effectively share its research findings?"	 

12. How effective, if at all, do you think the following methods would 
be for	ETA to disseminate its research findings? 

a. Searchable database of papers on	ETA's website: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

b. Distributing a compendium of ETA's research: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

c. Distributing a one-page summary of the research findings from each 
report: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

d. Email notifications announcing newly released research: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

e. Publishing articles in journals: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

f. Publicizing research findings through press articles, industry 
publications, etc. 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

g. Contracting with relevant organizations and associations to 
disseminate research: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

h. Webinars: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

i. Videos or podcasts: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

j. Social media (such as Twitter): 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

k. National ETA-sponsored conferences: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

l. Regional ETA-sponsored conferences: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

m. Presenting at outside conferences: 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

n. Briefings at ETA for external audiences (including stakeholders and	
policymakers): 
Extremely effective:	
Very	effective:	
Moderately effective:	
Somewhat	effective:	
Not at	all	effective:	
No response: 

13. Given a limited amount of resources, which five of the 
dissemination methods that you checked as at least moderately 
effective in question 12 should ETA focus on for disseminating 
research in the future. Please rank only 3 methods, with 1 as your top 
method, 2 as your second highest method, etc.). 

a. Searchable data base of papers on ETA's website: 
Rank: 

b. Distributing a compendium of ETA's	research: 
Rank: 

c. Distributing a one-page summary of the research findings from each 
report: 
Rank: 

d. Email notifications announcing newly released research: 
Rank: 

e. Publishing articles in journals: 
Rank: 

f. Publicizing research findings through press articles, industry 
publications, etc. 
Rank: 

g. Contracting with relevant organizations and associations to 
disseminate research: 
Rank: 

h. Webinars: 
Rank: 

i. Videos or podcasts: 
Rank: 

j. Social media (such as Twitter): 
Rank: 

k. National ETA-sponsored conferences: 
Rank: 

l. Regional ETA-sponsored conferences: 
Rank: 

m. Presenting at outside conferences: 
Rank: 

n. Briefings at ETA for external audiences (including stakeholders and 
policymakers): 
Rank: 

14. Do you have recent experience with any of the following ETA 
research dissemination methods? If yes, how satisfied or dissatisfied 
were you with each of the following ETA research dissemination methods? 

Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
a. Searchable database of	research papers on ETA's website: 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

b. ETA's Training and	Employment Notice (TENs) e-mail list: 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

c. Webinars: 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

d. National ETA-sponsored conferences: 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

e. Regional ETA-sponsored conferences: 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

f. Briefings at ETA for external audiences (including stakeholders and 
policymakers): 
Do you have recent	experience using any of the following ETA	research 
dissemination methods?	
Yes: (Go to next column); 
No. 
If yes, how satisfied or	dissatisfied were you with the following ETA 
research dissemination methods? 
Very satisfied: 
Satisfied: 
Neither satisfied nor dissatisfied: 
Dissatisfied: 
Very dissatisfied. 

15. Are there any additional thoughts you would like to share with us 
about ETA's research and dissemination practices? 

Demographics: 

16. Which of the following best describe your recent professional 
experience? (CHECK ALL THAT APPLY)
* Researcher
* Professor/academic
* Consultant; 
* State Official; 
* Local official; 
* Other; 
* No response. 

17. Are you ready to submit your final completed survey to GAO? (This 
is equivalent to mailing a completed paper survey to us. It tells us 
that your answers are official and final.) 
* Yes, my survey is complete - To submit your final responses, please 
click on "Exit" below. 
* No, my survey is not yet complete -To save your responses for later, 
please click on "Exit" below. 

You may view and print your completed survey by clicking on the 
Summary link in the menu to the left. 

Print: 
Submit: 

[End of section] 

Appendix VI: Experts Who Agreed to Participate in GAO's Delphi Panel: 

Expert: Burt S. Barnow; 
Affiliation: Trachtenberg School of Public Policy and Public 
Administration, George Washington University. 

Expert: Jon Baron; 
Affiliation: Coalition for Evidence-Based Policy. 

Expert: Stephen Bell; 
Affiliation: Abt Associates Inc. 

Expert: Jacob Benus; 
Affiliation: IMPAQ International, LLC. 

Expert: Dan Bloom; 
Affiliation: MDRC. 

Expert: Gary Burtless; 
Affiliation: Brookings Institution. 

Expert: Paul Decker; 
Affiliation: Mathematica Policy Research, Inc. 

Expert: Randall Eberts; 
Affiliation: W.E. Upjohn Institute for Employment Research. 

Expert: Richard Freeman; 
Affiliation: Harvard University; 
and; 
National Bureau of Economic Research. 

Expert: Robert Giloth; 
Affiliation: Annie E. Casey Foundation. 

Expert: David Heaney; 
Affiliation: Maximus. 

Expert: Carolyn J. Heinrich; 
Affiliation: Robert M. LaFollette School of Public Affairs, University 
of Wisconsin-Madison. 

Expert: Kevin Hollenbeck; 
Affiliation: W.E. Upjohn Institute for Employment Research. 

Expert: Louis Jacobson; 
Affiliation: New Horizons Economic Research. 

Expert: Richard Kazis; 
Affiliation: Jobs for the Future. 

Expert: Jacob Alex Klerman; 
Affiliation: Abt Associates Inc. 

Expert: Kathy Krepcio; 
Affiliation: John J. Heldrich Center for Workforce Development, 
Rutgers, The State University of New Jersey. 

Expert: Robert L. Lerman; 
Affiliation: Department of Economics, College of Arts and Sciences, 
American University. 

Expert: Alberto Martini; 
Affiliation: Progetto Valutazione. 

Expert: Sheena McConnell; 
Affiliation: Mathematica Policy Research, Inc. 

Expert: Rick McGahey; 
Affiliation: Ford Foundation. 

Expert: Peter Mueser; 
Affiliation: Department of Economics, University of Missouri-Columbia. 

Expert: Lee Munnich; 
Affiliation: Humphrey School of Public Affairs, University of 
Minnesota. 

Expert: Burke Murphy; 
Affiliation: Minnesota Department of Employment and Economic 
Development. 

Expert: Sigurd R. Nilsen; 
Affiliation: Formerly of GAO. 

Expert: Demetra Smith Nightingale; 
Affiliation: Urban Institute. 

Expert: Christopher O'Leary; 
Affiliation: W.E. Upjohn Institute for Employment Research. 

Expert: Marion Pines; 
Affiliation: Institute for Policy Studies, Johns Hopkins University. 

Expert: James Riccio; 
Affiliation: MDRC. 

Expert: Neil Ridley; 
Affiliation: Center for Law and Social Policy. 

Expert: Howard Rosen; 
Affiliation: Peterson Institute for International Economics. 

Expert: Peter Z. Schochet; 
Affiliation: Mathematica Policy Research, Inc. 

Expert: Kenneth R. Troske; 
Affiliation: University of Kentucky, Department of Economics. 

Expert: Jason Turner; 
Affiliation: Heritage Foundation. 

Expert: John Twomey; 
Affiliation: New York Association of Training and Employment 
Professionals. 

Expert: Ray Uhalde; 
Affiliation: Job for the Future. 

Expert: Carl Van Horn; 
Affiliation: John J. Heldrich Center for Workforce Development, 
Rutgers, The State University of New Jersey. 

Expert: John W. Wallace; 
Affiliation: Formerly of MDRC. 

Expert: Jeffrey B. Wenger; 
Affiliation: Department of Public Administration and Policy, School of 
Public and International Affairs, University of Georgia. 

Expert: Michael Wiseman; 
Affiliation: George Washington Institute of Public Policy, George 
Washington University. 

Expert: Steve A. Woodbury; 
Affiliation: Department of Economics, Michigan State University; 
and; W.E. Upjohn Institute for Employment Research. 

Source: GAO. 

[End of table] 

[End of section] 

Appendix VII: Comments from the Department of Labor: 

U.S. Department of Labor:	
Assistant Secretary for Employment and Training: 	
Washington, D.C. 20210 

March 7, 2011: 

Mr. George Scott: 
Director, Education, Workforce and Income Security Issues: 
U.S. Government Accountability Office: 
441 G. Street, N.W. 
Washington, D.C. 20548: 

Dear Mr. Scott: 

Thank you for the opportunity to review the Government Accountability 
Office (GAO) draft report entitled: "Employment and Training 
Administration: More Actions Needed to Improve Transparency and 
Accountability of Its Research Program." We continue to look for ways 
to strengthening the Employment and Training Administration's (ETA) 
research program, we appreciate the information provided in the report 
and the opportunity to comment. 

The Department of Labor (DOL) agrees with all three recommendations. 

GAO Recommendation 1: 

Formally incorporate into its research process the routine involvement 
of Chief Evaluation Office at key milestones, including at the 
development of ETA's annual research agenda and spending priorities, 
as well as at the early stages of developing specific research 
projects. 

DOL Response: The Department agrees with this recommendation and is in 
the process of formally involving the Chief Evaluation Officer (CEO) 
into its research process. In fact, ETA has worked closely with the 
CEO since its establishment in spring 2010. Before setting formal 
monthly meetings that began in fall 2010, ETA routinely sought 
involvement of CEO to discuss research, demonstration projects, and 
evaluations from the early development stages. Topics covered include 
implications of research findings, strategic planning, and project and 
evaluation development of the Green Jobs High Growth Health Care 
Impact Evaluation, the Young Parents Demonstration and Impact 
Evaluation, the Enhanced Transitional Jobs Demonstrations and Random 
Assignment Evaluation, and the Unemployment Insurance-related American 
Recovery and Reinvestment Act studies. 

Currently, ETA also routinely involves the CEO in: 

1) Reviewing research and demonstration statements of work; 

2) Discussing research and evaluation methodologies with ETA and the 
Office of Management and Budget and periodically, with other federal 
agencies; 

3) Discussing, on a monthly basis, on-going ETA research and 
evaluation projects; 

4) Reviewing research and evaluation design reports, interim reports 
and final reports prior to their dissemination; 

5) Participating in meetings with grantees involved in major research 
demonstrations to help explain the importance of rigorous evaluations; 
and; 

6) Providing input on how to best incorporate research in ETA's 
Solicitations for Grant Applications (SGAs) by strengthening the 
evaluation and report sections of the SGAs and crafting language to 
encourage applicants' use of evidence-based practices. 

Going forward, ETA plans to continue its collaboration with the CEO 
and involve the CEO in earlier stages of the research development 
process, including meetings to discuss potential research, 
demonstration and evaluation projects. 

GAO Recommendation 2: 

Develop a mechanism to enhance the transparency and accountability of 
ETA's research program. For example, involve advisory bodies or other 
entities outside ETA, in efforts to develop research policies and 
processes. 

DOL Response: The Department agrees with this recommendation and ETA 
already has taken several steps to engage outside experts more fully 
to enhance the transparency and accountability of its research 
program. Specifically, ETA is: (1) including outside experts in the 
development of its five-year research plan; (2) forming advisory and 
peer review groups for major evaluations, (3) developing, with the 
Coalition for Evidence-Based Policy, the capability to conduct peer 
reviews on an ad hoc basis, and (4) having the CEO, who is in the 
Department but independent of ETA, also review reports. 

In addition, ETA is collecting and synthesizing feedback from a 
variety of sources for developing its five-year research plan 
including: (1) input received from experts and workforce system 
practitioners at ETA's conferences, and (2) feedback solicited from 
stakeholders in the workforce community. ETA also will analyze public 
comments that it will solicit when the draft plan is announced in the 
Federal Register and posted on the ETA Web site. ETA has strengthened 
its collaboration with other Federal agency experts at the U.S. 
Departments of Education and Health and Human Services in planning and 
carrying out research and evaluation studies. 

Moving forward, ETA plans to continue engaging outside experts in 
developing research policies and processes. 

GAO Recommendation 3: 

Develop a formal plan for ensuring that ETA's research products are 
easily accessible to stakeholders and to the general public through 
its searchable database. Such a plan could involve requiring labor to 
assess overall effectiveness of its Web based search page, including 
user satisfaction with search features. 

DOL Response: The Department agrees with this recommendation, and ETA 
has taken steps to establish specific processes for disseminating 
research and evaluation findings in an effort to ensure that its 
research products are easily accessible to stakeholders and to
the general public. Specifically, ETA has improved the web-based 
publication search engine, and featured research and evaluation 
findings at agency-sponsored or other Federally-sponsored conferences. 

ETA supports continuous improvement of its dissemination efforts to 
ensure public accessibility. For example, last summer ETA upgraded the 
web-based research database to improve its overall effectiveness and 
user satisfaction by implementing the Google Search on all doleta.gov 
sites and developing an interface between the Google Search Appliance 
and ETA Research Publication Database. Since the improvements cited in
GAO's report, additional improvements have been made; when a keyword 
search is performed in the search box, located in the top navigation 
bar on doleta.gov, it returns a list of relevant publications housed 
in the ETA Research Publication Database. 

As part of ETA's plan to continue improving its dissemination 
practices, we have linked to the customer satisfaction survey form 
provided at the bottom of all Departmental Web pages. This survey 
provides feedback from visitors to help ensure that the Web site is 
useful, accurate and complete. ETA's Web Services team monitors the 
system log daily and takes immediate action necessary to rectify any 
issues. The Web Services team will periodically obtain feedback on 
operation of the Research Publication Database and will gather new 
requirements for further cost-effective enhancements that will be 
specified in an improvement plan. 

Additional Remarks: 

The Department believes that without further clarifications on the 
agency pilots, demonstration and research budget, the draft report 
would be misleading as to the funds available for undesignated pilots, 
demonstrations and research. We would encourage you to consider the 
Department's technical comments on the draft report, enclosed for your 
reference. 

Again, thank you for the opportunity to review the draft report. If 
you would like additional information, please do not hesitate to call 
me at (202) 693-2700. 

Sincerely, 

Signed by: 

Jane	Oates: 
Assistant Secretary: 

Enclosure: 

[End of section] 

Appendix VIII: GAO Contact and Staff Acknowledgments: 

Contact: 

George A. Scott, (202) 512-7215 or scottg@gao.gov: 

Staff Acknowledgments: 

In addition to the contact listed above, Dianne Blank, Assistant 
Director, and Kathleen White, analyst-in-charge, managed all phases of 
the engagement. Ashanta Williams assisted in managing many aspects of 
the work and was responsible for final report preparation. Lucas 
Alvarez and Benjamin Collins made significant contributions to all 
aspects of this report. In addition, Amanda Miller assisted with study 
and questionnaire design; Joanna Chan performed the data analysis; 
Stephanie Shipman advised on evaluation approaches; James Bennett 
provided graphics assistance; David Chrisinger provided writing 
assistance; Alex Galuten and Sheila McCoy provided legal support; and 
Sheranda Campbell and Ryan Siegel verified our findings. 

[End of section] 

GAO Related Products: 

Program Evaluation: Experienced Agencies Follow a Similar Model For 
Prioritizing Research. [hyperlink, 
http://www.gao.gov/products/GAO-11-176]. Washington, D.C.: January 14, 
2011. 

Employment and Training Administration: Increased Authority and 
Accountability Could Improve Research Program. [hyperlink, 
http://www.gao.gov/products/GAO-10-243]. Washington, D.C.: January 29, 
2010. 

Workforce Investment Act: Labor Has Made Progress in Addressing Areas 
of Concern, but More Focus Needed on Understanding What Works and What 
Doesn't. [hyperlink, http://www.gao.gov/products/GAO-09-396T]. 
Washington, D.C.: February 26, 2009. 

Employment and Training Program Grants: Evaluating Impacts and 
Enhanced Monitoring Would Improve Accountability. [hyperlink, 
http://www.gao.gov/products/GAO-08-486]. Washington, D.C.: May 7, 2008. 

Federal Research: Policies Guiding the Dissemination of Scientific 
Research from Selected Agencies Should Be Clarified and Better 
Communicated. [hyperlink, http://www.gao.gov/products/GAO-07-653]. 
Washington, D.C.: May 17, 2007. 

Data Quality: Expanded Use of Key Dissemination Practices Would 
Further Safeguard the Integrity of Federal Statistical Data. 
[hyperlink, http://www.gao.gov/products/GAO-06-607]. Washington, D.C.: 
May 31, 2006. 

Workforce Investment Act: Substantial Funds Are Used for Training, but 
Little Is Known Nationally about Training Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-05-650]. Washington, D.C.: June 29, 
2005. 

Program Evaluation: An Evaluation Culture and Collaborative 
Partnerships Help Build Agency Capacity. [hyperlink, 
http://www.gao.gov/products/GAO-03-454]. Washington, D.C.: May 2, 2003. 

Workforce Investment Act: Improvements Needed in Performance Measures 
to Provide a More Accurate Picture of WIA's Effectiveness. [hyperlink, 
http://www.gao.gov/products/GAO-02-275]. Washington, D.C.: February 1, 
2002. 

[End of section] 

Footnotes: 

[1] A program year begins on July 1 of a year and ends on June 30 of 
the following year. Thus, program year 2010 began on July 1, 2010, and 
ends on June 30, 2011. Of the $103 million appropriated for program 
year 2010, Congress designated about $84 million for specific 
projects, including $30 million for Transitional Jobs activities and 
$5.5 million for the employment and training needs of young parents. 
The remainder of the funds, approximately $18.7 million, was available 
to ETA for undesignated pilots, demonstrations, and research. 

[2] GAO, Workforce Investment Act: Labor Has Made Progress in 
Addressing Areas of Concern, but More Focus Needed on Understanding 
What Works and What Doesn't, [hyperlink, 
http://www.gao.gov/products/GAO-09-396T] (Washington, D.C.: Feb. 26, 
2009); Employment and Training Program Grants: Evaluating Impact and 
Enhancing Monitoring Would Improve Accountability, [hyperlink, 
http://www.gao.gov/products/GAO-08-486] (Washington, D.C.: May 7, 
2008); and Employment and Training Program Grants: Labor Has Outlined 
Steps for Additional Documentation and Monitoring but Assessing Impact 
Still Remains an Issue, [hyperlink, 
http://www.gao.gov/products/GAO-08-1140T] (Washington, D.C.: Sept. 23, 
2008). 

[3] GAO, Employment and Training Administration: Increased Authority 
and Accountability Could Improve Research Program, [hyperlink, 
http://www.gao.gov/products/GAO-10-243] (Washington, D.C.: Jan. 29, 
2010). 

[4] 29 U.S.C. § 2916(a). 

[5] While ETA's program year 2010 began on July 1, 2010, ETA's funding 
for pilots, demonstrations, and research was available starting on 
April 1, 2010. 

[6] American Evaluation Association, An Evaluation Roadmap for a More 
Effective Government, (September 2010), available at [hyperlink, 
www.eval.org/eptf.asp]. 

[7] Congress enacted WIA in part to increase employment, retention, 
earnings, and occupational skill attainment of participants, thus 
improving the quality of the workforce. WIA Adult primarily focuses on 
disadvantaged adults and WIA Dislocated Workers focuses largely on 
workers who have been laid off or have been notified that they will be 
laid off. WIA Youth helps to prepare youth for employment and/or 
postsecondary education through linkages between academic and 
occupational learning. 

[8] See [hyperlink, http://www.gao.gov/products/GAO-08-486]. 

[9] ETA now has a contract for $1 million with the firm that performed 
the original GATE I evaluation to perform a 5-year follow-up to assess 
long-term outcomes for Project GATE II. 

[10] These studies should include designs that randomly assign 
individuals to groups that receive enhanced program services and to 
groups that do not. 

[11] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help 
Identify Effective Interventions, [hyperlink, 
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23, 
2009). 

[12] See [hyperlink, http://www.gao.gov/products/GAO-10-243]. 

[13] See [hyperlink, http://www.gao.gov/products/GAO-10-243]. 

[14] For more information on the criteria several other federal 
agencies use to prioritize their research, see GAO, Program 
Evaluation: Experienced Agencies Follow a Similar Model for 
Prioritizing Research, [hyperlink, 
http://www.gao.gov/products/GAO-11-176] (Washington, D.C.: Jan. 14, 
2011). 

[15] While ETA has a provision that allows contractors to publicly 
release reports that have not been approved by the Assistant Secretary 
after 9 months, researchers may be reluctant to do so out of concern 
that this action may damage their credibility with the agency and 
limit their ability to win future research contracts. 

[16] See [hyperlink, http://www.gao.gov/products/GAO-10-243]. 

[17] A fourth study, the Impact Evaluation of the Senior Community 
Service Employment Program, was also designed as a random assignment 
study. We omitted it from our analysis because ETA was reconsidering 
the funding of this evaluation at the time of our report. 

[18] See [hyperlink, http://www.gao.gov/products/GAO-10-30]. 

[19] 29 U.S.C. § 2917(c). 

[20] In addition to OMB's review, the evaluation contractor also 
conducted a peer review of the study design. 

[21] [hyperlink, http://www.gao.gov/products/GAO-10-243]. 

[22] See Interagency Committee on Government Information, Recommended 
Policies and Guidelines for Federal Public Websites (2004). The 
Interagency Committee on Government Information was established by OMB 
in 2003 to improve the usability of federal public Web sites. In 
particular, they recommend that "organizations should provide help, 
hints, or tips, and include examples" for search users. 

[23] The Delphi method, developed by the RAND Corporation in the 
1950s, is most commonly applied in a group-discussion forum. We 
modified the approach to have the group discussion take place in the 
form of a Web-based forum. 

[24] Labor awarded about $9.9 million to various entities to carry out 
its Young Parents Demonstration for fiscal years 2008 and 2009. 
Lancaster County Workforce Investment Board was 1 of 13 grantees that 
received funding. 

[25] In June 2008, ETA awarded GATE II grants to four states--one of 
which was to Virginia--for the extension of the GATE model for helping 
selected dislocated workers create their own business. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: