This is the accessible text file for GAO report number GAO-10-243 
entitled 'Employment and Training Administration: Increased Authority 
and Accountability Could Improve Research Program' which was released 
on January 29, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 
GAO: 

January 2010: 

Employment and Training Administration: 

Increased Authority and Accountability Could Improve Research Program: 

GAO-10-243: 

Contents: 

Letter: 

Appendix I: Briefing Slides: 

Appendix II: Dissemination Time Frames for ETA's Research Products 
Published in 2008: 

Appendix III: Comments from the Department of Labor: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Table: 

Table 1: ETA's 2008 Research Products by Dissemination Time Frames: 

Abbreviations: 

EPA: Environmental Protection Agency: 

ERS: Economic Research Service: 

ETA: Employment and Training Administration: 

HUD: Department of Housing and Urban Development: 

IES: Institute of Education Sciences: 

NSF: National Science Foundation: 

OMB: Office of Management and Budget: 

OPDR: Office of Policy Development and Research: 

USDA: U.S. Department of Agriculture: 

WIA: Workforce Investment Act of 1998: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

January 29, 2010: 

The Honorable Tom Harkin: 
Chairman: 
The Honorable Michael B. Enzi: 
Ranking Member: 
Committee on Health, Education, Labor, and Pensions: 
United States Senate: 

The Honorable Patty Murray: 
Chairman: 
The Honorable Johnny Isakson: 
Ranking Member: 
Subcommittee on Employment and Workplace Safety: 
Committee on Health, Education, Labor, and Pensions: 
United States Senate: 

The Honorable Tom Harkin: 
Chairman: 
Subcommittee on Labor, Health and Human Services, Education and 
Related Agencies: 
Committee on Appropriations: 
United States Senate: 

With current rising unemployment rates and the need for a more skilled 
workforce, it is important for the Department of Labor's (Labor) 
Employment and Training Administration (ETA) to invest in sound 
research that identifies the most effective and efficient ways to 
train and employ workers for 21st century jobs. While ETA 
traditionally has played an important role in providing job training, 
employment assistance, and labor market information for the nation's 
workers, the current unemployment crisis has made this role more 
critical. The Bureau of Labor Statistics estimates that jobless rates 
have increased over the past year in all 50 states and the District of 
Columbia. In particular, several states have reported unemployment 
rates well over 10 percent. 

As ETA's new leadership works to help the nation meet these economic 
challenges, it must have solid information that is supported by sound 
research to guide decision-making. Since 2002, GAO and others have 
criticized ETA for not focusing sufficient attention on its research 
program, particularly with regard to complying with congressional 
mandates, conducting policy-relevant research, and disseminating key 
research findings in a timely way. In this context, we have examined 
the structure and processes of ETA's research and evaluation center in 
terms of the elements that leading national research organizations 
cite as essential to a sound program; that is, research independence, 
transparency and accountability, and policy relevance. Based on these 
elements, we addressed the following questions: (1) How does ETA's 
organizational structure provide for research independence? (2) What 
steps has ETA taken to promote transparency and accountability in its 
research program? (3) How does ETA ensure that its research is 
relevant to workforce development policy and practice? 

On December 3, 2009, we briefed your staff on the results of our 
analysis. This report formally conveys the information provided during 
this briefing (see app. I for the briefing slides). In summary, we 
found that ETA's research center lacks independent authority for 
research, has limitations with regard to transparency and 
accountability processes, has not routinely involved stakeholders in 
developing its research agenda, and has been slow to evaluate the 
programs and activities carried out under the Workforce Investment Act 
of 1998 (WIA). 

* ETA is currently in the process of revising its organizational 
structure. Previously, its research and evaluation center, within the 
Office of Policy Development and Research (OPDR), was several levels 
removed from the Office of the Assistant Secretary and lacked 
independent authority for conducting research. Under ETA's revised 
structure, it is unclear whether OPDR will report directly to the 
Assistant Secretary or the Deputy Assistant Secretary. Furthermore, 
unlike the heads of some other research and evaluation centers, the 
head of ETA's research and evaluation center does not have the 
authority to set the research agenda, to approve requests for funded 
projects, or to disseminate research and evaluation reports. 

* While ETA has recently made improvements to its research program, 
some limitations remain with regard to its accountability processes 
and timely dissemination of research products. In 2007, ETA documented 
all stages of its research process from project selection to 
dissemination. ETA also began coordinating its research process with 
the Office of Management and Budget (OMB) and renewed its practice of 
sponsoring biennial research conferences to address a range of policy 
and program issues. Despite these recent changes, ETA's research 
processes lack specificity, including specific time frames for key 
milestones and an established criteria for peer review. In addition, 
ETA's research program lacks an information system to track its 
research from completion to dissemination and does not have a 
mechanism to ensure that research findings are disseminated in a 
timely manner. In 2008, ETA disseminated 34 research products to the 
public. However, 20 of these products, which had a combined cost of 
about $28 million, were delayed for between 2 and 5 years. 

* Labor, consistent with recent efforts of OMB, has taken steps to 
emphasize the value of research and evaluation, but ETA does not have 
sufficient mechanisms to ensure that its research can inform policy 
decisions. OMB currently has efforts underway to help agencies to 
enhance their research and evaluation programs. Labor also announced 
plans to create a position of chief evaluation officer and a new 
evaluation center within the Office of the Assistant Secretary for 
Policy to foster research relevant to policy. However, ETA has been 
slow to comply with a congressional mandate to evaluate WIA[Footnote 
1] and it also lacks a standard process and advisory bodies to 
consistently involve stakeholders in the development of its research 
agenda. Similarly, ETA has no formal process in place to ensure that 
research findings are used to inform strategic planning and policy. 

GAO is making several recommendations concerning the structure and 
processes of ETA's research and evaluation center. Specifically, GAO 
recommends that the Secretary of Labor: 

* take steps to ensure that ETA clarifies its organizational structure 
and OPDR reports directly to ETA's Assistant Secretary; 

* provide sufficient decision-making authority to ETA's research and 
evaluation center regarding its research; 

* direct ETA's research and evaluation center to establish more 
specific processes, including time frames for dissemination of 
research; 

* create an information system to track research projects; and: 

* instruct ETA's research and evaluation center to develop processes 
to involve outside experts in setting its research agenda. 

We used several methodologies to develop our findings. To identify 
elements of sound government-sponsored research, we reviewed relevant 
guidelines established by the American Evaluation Association and the 
National Research Council of the National Academy of Sciences. To 
apply these guidelines to ETA's research and evaluation center, we 
examined ETA's organizational structure and processes in terms of 
three basic elements of sound research--independence, transparency and 
accountability, and policy relevance. As part of this examination, we 
analyzed information on ETA's disseminated research to determine 
whether research products were released in a timely manner. (App. II 
contains additional information regarding the time frames of these 
research products.) In addition, we reviewed the research structures 
and processes of five other federal agencies' research and evaluation 
centers that had some of the characteristics that were identified by 
the American Evaluation Association and the National Research Council 
of the National Academy of Sciences to support sound research. 
[Footnote 2] We did not evaluate the operation or quality of the 
research produced by the centers but rather the structures and 
policies supporting their research. We also reviewed relevant federal 
laws. 

We conducted this performance audit from October 2009 through December 
2009 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

We provided a draft of this report to ETA for its review and comment. 
ETA provided written comments, which are reproduced in appendix III. 
In addition, ETA provided technical comments, which we incorporated 
where appropriate. 

In its response, ETA concurred with two of our recommendations and 
cited ongoing activities addressing the areas covered in the other 
three. Regarding our recommendation to clarify its organizational 
chart to ensure that OPDR reports directly to ETA's Assistant 
Secretary, ETA concurred and has since clarified this point in an 
updated organizational chart that it has posted on its Web site. ETA 
also concurred with our recommendation to create an information-
tracking system for its research products, noting that it has plans to 
implement such a system to track product milestones. However, 
officials did not provide specific time frames for implementation. 
Because of the importance of developing such a system, we urge ETA to 
give this effort high priority. 

On the remaining three recommendations, ETA did not agree or disagree, 
noting instead its relevant ongoing activities. 

* Regarding our recommendation that ETA provide its research and 
evaluation center with sufficient decision-making authority, officials 
cited ongoing efforts to provide recommendations to the Assistant 
Secretary and to collaborate with relevant program offices on ETA's 
research plans. We considered these activities in our review and 
discussed them in our report, but these efforts alone fall short of 
providing OPDR with the necessary authority to make key decisions, and 
they do not serve to insulate OPDR from undue political influence. 

* Regarding our recommendation that ETA establish more specific 
processes to guide research, officials cited their work with OMB to 
establish time frames for disseminating research. However, specifying 
time frames is but one component of ETA's research process that needs 
to be addressed. Our report cites other steps that ETA should take 
including specifying the criteria to be used in deciding whether a 
report should be peer reviewed. In ETA's technical comments, officials 
noted that they will be updating their process steps in early 2010. As 
ETA reviews and updates its processes, we urge the agency to make the 
steps more specific in order to help ensure transparency and 
accountability. 

* Regarding our recommendation to develop processes to routinely 
involve outside experts in setting its research agenda, ETA officials 
cited the various informal methods they currently use to gather input 
from outside experts. These efforts may serve to increase outside 
involvement in setting the agenda, but they are informal and ad hoc 
and, therefore, do not provide ETA with a formal mechanism to ensure 
outside experts--including researchers, policy makers, and 
practitioners--consistently have a role in setting its research agenda. 

We are sending copies of this report to relevant congressional: 

committees. In addition, this report will be available at no charge on 
GAO's Web site at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff members who made key 
contributions to this report are listed in appendix IV. 

Signed by: 

George A. Scott: 
Director, Education, Workforce, and Income Security Issues: 

[End of section] 

Appendix I: Briefing Slides: 

Employment and Training Administration: Increased Authority and 
Accountability Could Improve Research Program: 

Briefing to Congressional Committees: 

December 3, 2009: 

Overview: 
* Introduction: 
* Research Objectives: 
* Scope and Methodology: 
* Summary of Findings: 
* Background: 
* Findings: 
* Conclusions and Recommendations: 

Introduction: 

A sound program of research and evaluation is essential to finding 
effective ways for training and employing workers for 21st century 
jobs. 

Since 2002, we and others have criticized the Employment and Training 
Administration (ETA) for not focusing sufficient attention on its 
research program, particularly with regard to: 

* complying with statutory research mandates; 

* conducting research to help policy makers understand what works and 
what does not; and; 

* disseminating key research findings in a timely way. 

* Most recently, we have been asked to review the extent to which 
ETA’s structure and processes promote sound research; 

* At the time of our review, ETA was transitioning from one 
administration to another, and; 

* ETA’s new leadership is formulating new plans on how best to carry 
out its research and evaluations. 

Leading national research organizations, including the American 
Evaluation Association and the National Research Council of the 
National Academy of Sciences—have identified a number of elements of a 
sound research and evaluation program. 

Three of the elements of sound research are: 
* Research Independence; 
* Transparency and Accountability; 
* Policy Relevance. 

[End of Introduction] 

Objectives: 

Based on these elements, we examined ETA’s research program to answer 
the following questions: 

1. How does ETA’s organizational structure provide for research 
independence? 

2. What steps has ETA taken to promote transparency and accountability 
in its research program? 

3. How does ETA ensure that its research is relevant to workforce 
development policy and practice? 

[End of Objectives] 

Scope and Methodology: 

To identify basic principles of sound government-sponsored research, 
we reviewed relevant guidelines as developed by the American 
Evaluation Association and the National Research Council of the 
National Academy of Sciences. 

These guidelines are suitably applicable to a full range of government 
research and evaluation centers. 

To examine ETA’s organizational structure and processes, we: 

* Interviewed agency officials, stakeholders, and experts. 

* Examined agency documents, policies, and procedures. 

To examine the structure and processes of other government research 
and evaluation centers, we: 

* Interviewed research experts. 

* Examined agencies’ documents and external reports. 

* Obtained information from GAO officials with knowledge of these 
centers. 

In addition to ETA, we present information for comparative purposes on 
the following research centers: 

* The Department of Agriculture’s (USDA) Economic Research 
Service(ERS) and National Institute of Food and Agriculture; 

* The Department of Education’s (Education) Institute of Education 
Sciences (IES); 

* The Department of Housing and Urban Development’s (HUD) Office of 
Policy Development and Research; 

* The Environmental Protection Agency’s (EPA) Office of Research and 
Development; and; 

* The National Science Foundation (NSF). 

We did not evaluate the operation or quality of the research produced 
by the centers but rather the structures and policies supporting their 
research. 

We selected these centers as examples because they had some of the 
characteristics that support sound research as identified by the 
American Evaluation Association and the National Research Council of 
the National Academy of Sciences guidelines. 

We conducted this performance audit from October 2009 to December 2009 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Key Terminology: 

* Governing or advisory board: This term generally refers to an 
independent body that provides advice to senior leadership on a range 
of issues, including recommending research priorities, helping to 
safeguard the independence of the agency, and providing critical links 
to practice and policy communities. 

* Peer review: A quality assurance process that involves selecting 
independent and knowledgeable individuals to review and provide 
objective feedback on an agency’s research. 

* Research and evaluation center: The entity within an agency 
responsible for planning and conducting research and evaluations. 

[End of Scope and Methodology] 

Summary of Findings: 

ETA is currently in the process of revising its organizational 
structure. Previously, its research and evaluation center, within the 
Office of Policy Development and Research (OPDR), was several levels 
removed from the Office of the Assistant Secretary and lacked 
independent authority for conducting research. 

While ETA has recently taken steps to improve its research program, 
some limitations remain with regard to its accountability processes 
and timely dissemination of research products. 

The Office of Management and Budget (OMB) and Labor have both 
emphasized the value of research and evaluation. However, ETA has not 
routinely involved stakeholders in developing its research agenda and 
has also been slow to evaluate the Workforce Investment Act of 1998 
(WIA). 

[End of Summary of Findings] 

Background: 

Some Essential Elements of a Sound Government Research and Evaluation 
Program: 

Some elements of sound research practices that can help agencies’ 
strengthen their research and evaluation function, regardless of key 
differences related to the agency’s overall mission and the size of 
its research and evaluation center: 
Research Independence; 
Transparency and Accountability; 
Policy Relevance. 

Research Independence: Some Essential Elements of Sound Practices: 

According to leading experts, to achieve research independence, 
research and evaluation centers should be: 

* Located at a level of influence within the agency or department. 
That is, research centers should be located at a level that allows its 
leaders to report directly to the senior executive of the organization. 

* Responsible for key aspects of its research program, including the 
design, conduct, and dissemination of research studies. 

Transparency and Accountability: Some Essential Elements of Sound 
Practices: 

To achieve transparency and accountability, research and evaluation 
centers should: 

* Release their research agendas to the public; 

* Document research policies and procedures; and; 

* Publicly disseminate research findings in a timely and accessible 
manner. 

Policy Relevance: Some Essential Elements of Sound Practices: 

To achieve policy relevance, research and evaluation centers should: 

* Use research results to both inform and respond to agency strategic 
planning; 

* Focus research agenda on issues of importance to policy and 
practice; and; 

* Involve key stakeholders (researchers, policy makers, and 
practitioners) in planning, designing, and conducting research. 

Objective 1: Research Independence: 

Previously, ETA’s OPDR Did Not Report Directly to the Assistant 
Secretary: 

Figure: Organization chart: 

[Refer to PDF for image] 

Research and evaluation center was several levels below the
Assistant Secretary: 

Top level: 
Secretary of Labor; 
* Deputy Secretary. 

Second level: Report to Deputy Secretary; 
* Assistant Secretary: Administration and Management; 
* Bureau of Labor Statistics; 
* Assistant Secretary: ETA; 
* Women's Bureau; 
* Occupational Safety & Health Administration. 

Third level: Report to Assistant Secretary: ETA; 
* Deputy Assistant Secretary: Workforce Investment System; 
* Deputy Assistant Secretary: Administration and National Activity. 

Fourth level: Reporting to Deputy Assistant Secretary, Administration 
and National Activity: 
* Trade Adjustment Assistance; 
* Foreign Labor Certification; 
* Apprenticeship; 
* Policy Development and Research; 
* Financial and Administrative Management; 
* Performance and Technology. 

Fifth level: Reporting to Policy Development and Research: 
* Policy Legislation and Dissemination; 
* Research and Evaluations. 

Source: Department of Labor's Employment and Training Administration 
and GAO analysis. 

[End of figure] 

ETA’s Emerging Organizational Structure Does Not Ensure OPDR is 
Located at a Level of Influence: 

At the time of our review, ETA was revising its organizational 
structure. 

Under the revised structure, OPDR is placed at the same level as 
several program offices such as the Offices of Apprenticeship and 
Trade Adjustment Assistance. 

It is unclear from ETA’s revised organizational chart whether OPDR 
reports directly to the Assistant Secretary or the Deputy Assistant 
Secretary. 

It is also unclear whether, as part of OPDR, ETA’s research and 
evaluation center is located at a level of sufficient influence within 
ETA. 

Alternative Example: Education’s Research Center (IES) Reports 
Directly to the Secretary: 

Figure: Organization chart: 

[Refer to PDF for image] 

Top level: 
* Secretary of Education. 

Second level: Reporting to Secretary of Education: 
* Planning, Evaluation and Policy Development; 
* Communications and Outreach; 
* General Counsel; 
* Institute of Education Sciences: 
- National Board for Education Sciences. 

Third level: Reporting to Institute of Education Sciences: 
* Administration and Policy: 
- Management and Operations; 
- Outreach and Communications; 
- Grants Administration; 
* Science: 
- Standards and Review. 

Fourth level: Reporting to Institute of Education Sciences: 
* National Center for Education Research; 
* National Center for Education Statistics; 
* National Center for Education Evaluation and Regional Assistance; 
* National Center for Special Education Research. 
						
Source: Department of Education and GAO analysis. 

[End of figure] 

ETA’s Research and Evaluation Center Lacks Sufficient Independence: 

The head of ETA’s research and evaluation center lacks sufficient 
authority to approve: 

* its research agenda; 

* requests for funded proposals, grant announcements, and statements 
of work; or; 

* dissemination of research and evaluation publications. 

According to ETA officials, these authorities reside with ETA’s Office 
of the Assistant Secretary. 

Neither ETA’s research and evaluation center, nor the Office of Policy 
Development and Research of which it is a part, has a specific 
mechanism that could provide insulation from undue influence. 

Other Agencies Have Mechanisms Designed to Promote Research 
Independence: 

For example, the directors of IES and NSF take the following actions: 

* Implement their research agenda; 

* Design funded projects; 

* Submit reports for peer review and publish research, statistics,and 
evaluations with or without the approval of any other office within 
their organizations; and; 

* Work with governing boards on research agenda and policies. 

[End of Objective 1] 

Objective 2: Transparency and Accountability: 

ETA Has Recently Documented Its Research Process: 

In 2007, ETA began documenting its research process. 

* This documentation details the eight steps ETA uses to conduct its 
research, from project selection to dissemination, and the expected 
time frames for each step. 

* ETA developed this documentation at the request of OMB. 

* Prior to this, ETA did not have a documented process. 

However, ETA has not made this documentation available to the public 
and includes little information regarding its processes on its Web 
site. 

ETA’s Research Process Involves Coordination with OMB: 

In September 2007, ETA agreed to inform OMB of sizable research 
projects and evaluations. 

* OMB reviews the statement of work for all projects that exceed 
$250,000. 

* OMB informs ETA if it wants to discuss the statement of work. 

* OMB has requested that ETA alert it of research reports submitted to 
ETA for dissemination and not approved by the Office of the Assistant 
Secretary within 9 months. 

ETA Now Allows Contractors to Release Products Delayed for More Than 9 
Months: 

Since January 2008, ETA has included a provision in its research 
contracts that allows the contractor to release products submitted to 
ETA but not approved by the Assistant Secretary within 9 months. 

However, ETA does not broadly advertise the availability of these 
products on its Web site to the public. 

ETA Has Also Renewed Its Practice of Sponsoring National Conferences 
to Highlight Research Findings: 

In the past, ETA has sponsored biennial research conferences to 
address a range of policy and program issues and document promising 
program practices. This practice ended in 2003. 

In September 2009, ETA renewed this practice by sponsoring a national 
research conference on “Recovery and Reemployment” that highlighted 
ETA sponsored research. 

* The conference also assembled leading researchers from a variety of 
government and private entities and fields of research and practice. 

* ETA has made the PowerPoint presentations of conference sessions 
available on the Internet and is in the process of making the audio 
and visual transcripts available to the public. 

Despite Recent Changes, ETA’s Research Processes Lack Specificity: 

Time frames for ETA’s key milestones are stated loosely. 

For example: 

* During the final review and approval process for dissemination, 
project managers are instructed to inform management when products 
have not been approved by the Office of the Assistant Secretary,and as 
a result, have stalled in the process for an “excessive” amount of 
time. 

* However, ETA has not explained what constitutes an "excessive” 
amount of time. 

ETA Does Not Have Specific Guidance for Peer Review: 

ETA’s research process provides for peer review but does not specify 
the criteria that will be used to determine whether a report should be 
peer reviewed. 

ETA officials told us that the scope of a project determines whether 
it will be peer reviewed, but did not provide specific criteria. 

ETA does not provide information on or guidelines for the peer review 
process, including how comments will be used and who will be selected 
to serve as peer reviewers. 

ERS Reported That It Has Specific Policies and Procedures for Peer 
Review: 

According to USDA information: 

* ERS conducts peer reviews of its reports and has established review 
guidelines, including criteria for selecting qualified peer reviewers. 

* Each report is subjected to both internal and formal, independent, 
external peer review. 

* ERS requires thorough review of data products by knowledgeable staff 
prior to dissemination. 

ETA Lacks a System to Track Research Progress: 

ETA has not established an information system to track its research 
products from completion to dissemination. 

* As a result, ETA is unable to track the status of products and 
determine which products are delayed and where in the process they are 
delayed. 

* Without a tracking system, it is difficult for ETA to determine if 
it is following its processes or complying with established time 
frames. 

ETA Has Not Consistently Disseminated Research Findings in a Timely 
Manner: 

ETA disseminated 34 research products to the public in 2008. 

* Twenty of these products had been waiting 2 to 5 years for approval. 
(See appendix II for information on report dissemination time frames.) 

* ETA officials attributed the delays to inadequate methodologies and 
poorly written products. However, ETA had previously agreed to these 
methodologies at design and reviewed and approved earlier drafts of 
these reports. 

* Presently ETA is reviewing its procedures to identify areas where it 
can obtain organizational buy-in earlier in the process. 

Dissemination Delays Could Have Cost, Policy, and Practice 
Implications: 

Most of ETA’s sponsored research whose publication was delayed until 
2008 were evaluations of workforce development strategies. 

The combined cost of the 20 delayed research products was about $28 
million. 

Due to the delays, findings from these studies could have a limited 
impact in influencing policy and practice. 

* Delayed products covered a range of topics including evaluations of 
labor exchanges in the one-stop delivery system, the youth offender 
demonstration project, and workforce development in rural areas. 

ETA’s Dissemination of Research Findings Is Limited: 

ETA’s dissemination of research findings is largely limited to 
postings on its Web site and consists of research and evaluation 
reports that it has sponsored. 

ETA also issues e-mail advisories to notify its subscribers of newly 
released research publications. 

ETA has not used dissemination strategies, such as the use of 
clearinghouses, to more broadly disseminate its research. 

HUD Disseminates Its Research, including Data Sets, through 
Clearinghouses and a Dedicated Web Site: 

According to department information: 

* HUD uses a dedicated Web site to disseminate information that it and 
others have collected on housing issues, including housing market 
trends, economic and demographic data, housing finance, and energy and 
resource-efficient housing designs and construction. 

* It has two clearinghouses that provide (1) information on state and 
local regulatory reform strategies to support affordable housing and 
(2) access to HUD-sponsored research and publications. 

* HUD also makes data sets collected for both research and 
administrative purposes routinely available to the public. 

EPA Disseminates Its Research Products through Partnerships and a 
Range of Media: 

According to department information: 

* EPA partners with other federal agencies and with states to 
disseminate and publicize its research. 

* EPA’s Web site contains a searchable database of EPA science 
activities and scientific and technical products produced by EPA and 
through EPA-funded assistance agreements. 
- It also provides public access to its data sets. 
- Users can receive e-mail notifications of new products based on key 
words, topics, or general research. 

* EPA issues press releases and holds research and press conferences. 

[End of Objective 2] 

Objective 3: Policy Relevance: 

OMB Has Taken Steps to Promote and Strengthen Evaluation Efforts: 

Current efforts are underway at OMB to encourage agencies to increase 
their emphasis on the importance of research and evaluations and 
strengthen evaluation methods. 

OMB plans to work with agencies to: 

* Expand information about program evaluations that are made public. 

* Establish an interagency working group to build evaluation capacity 
and create effective evaluations that work. 

* Provide additional funding to agencies for high-priority evaluation 
activities. 

Labor Has Taken Recent Steps to Foster Research Relevant to Policy: 

In September 2009, Labor announced plans to create a position of chief 
evaluation officer and a new evaluation center within the Office of 
the Assistant Secretary for Policy. 

* This position is designed to link evaluation with policy development 
and strategic planning. 

* However, the linkage between the new position and ETA’s research and 
evaluation center is unclear. 

Recently, ETA’s Assistant Secretary has acknowledged the importance of 
research-driven policy and the need to conduct rigorous evaluations. 

ETA has taken steps to increase its evaluation activities and has 
applied to OMB for additional funds to conduct rigorous evaluations. 

However, ETA Has Been Slow to Evaluate WIA Program Activities: 

Congress required an impact evaluation of WIA activities by 2005. 
[Footnote 3] 

* However, ETA did not award a contract to conduct the evaluation 
until 2008. 

* As of mid-November 2009, the evaluation’s research design has not 
yet been finalized. 

* Officials told us that they plan to implement the evaluation in 
January 2010. 

Key Stakeholders Have Not Been Consistently Involved to Ensure ETA’s 
Research Is Relevant: 

ETA has had no standard process for selecting stakeholders to review 
its research agenda or incorporating their comments. 

ETA’s documents indicate that, in recent years, ETA’s Assistant 
Secretary had set and implemented the agenda with limited involvement 
from external stakeholders. 

* For example, ETA did not solicit public comment on its finalized2007 
research plan. 

* Moreover, ETA never finalized its 2003 and 2005 revisions to its 5-
year research plan or made the plans available to the public. 

* However, in 1999, ETA advertised the availability of a draft of its 
plan through a Federal Register notice and solicited public comments 
for consideration before the plan was finalized. 

Unlike Other Agencies, ETA Does Not Have Advisory Bodies to Involve 
Stakeholders in the Development of Its Research Agenda: 

For example: 

* EPA’s Board of Scientific Counselors provides advice, information, 
and recommendations about its research program. 

* IES’ National Board for Education Sciences approves or disapproves 
the priorities for the Institute proposed by the Director and ensures 
that its priorities are consistent with its mission. 

* NSF’s National Science Board establishes its research policies and 
approves its strategic budget direction. 

Labor Is Currently Assisted by 14 Advisory Boards That Perform a 
Variety of Functions: 

Currently, 14 advisory boards assist Labor and serve a variety of 
purposes. For example: 

* The Advisory Committee on Apprenticeship provides assistance and 
advice to the Assistant Secretary of ETA on policies and programs 
regarding apprenticeship. 

* The Bureau of Labor Statistics Data Users Committee provides advice 
on the collection and analysis of the Bureau's statistics, its 
published reports, and its overall mission. 

ERS Routinely Consults with Stakeholders to Ensure Its Research Is 
Relevant and Useful: 

According to USDA, ERS’ procedures call for consulting with 
stakeholders—including policy makers and key institutions that 
influence public policy. 

* ERS involves stakeholders in discussions of past research 
accomplishments, and program effectiveness and impact. 

* ERS also consults with and works across department offices and 
programs in devising cross-cutting research. 

While Advisory Committees Can Promote Research Relevance, They Involve 
Some Trade-Offs: 

* Involving advisory committees requires additional time and effort, 
as well as some minimal expense. 

* In creating advisory committees, federal agencies are generally 
required to follow the provisions of the Federal Advisory Committee 
Act, which include provisions limiting when such committees can be 
established. 

* Most of the agencies we reviewed had advisory committees.For 
example, during FY 2008, the General Services Administration reported 
a total of 917 active committees with a total of nearly 64,000 members 
who provided advice and recommendations to 50 federal agencies. 

ETA Does Not Have a Formal Process in Place to Ensure Strategic 
Planning Is Informed by Research: 

Although ETA coordinates with its strategic planning office, there is 
no formal process in place to ensure that research findings are used 
to inform strategic planning and policy. 

For example: 

* Officials told us that they respond to requests from other 
departments to provide information on research and evaluation findings. 

* However, there is no formal process in place to ensure that all 
research and evaluation findings are shared on a consistent basis. 

In Contrast, EPA’s Structure Has the Potential to Encourage 
Coordination Between Research and Strategic Planning: 

According to agency documents, EPA has established two offices within 
its research office to help ensure its policy and planning are 
informed by its research. 

* Office of Science Advisor: establishes specific mechanisms for 
ensuring that scientific results and hypotheses, with technical 
evaluation and peer-review, play a prominent role in all regulatory 
decisions. 

* Office of Science Policy: ensures that the scientific information 
generated through EPA and other research is used to inform EPA 
decisions. 

USDA’s Advisory Board Reported That It Serves as a Link to Strategic 
Planning: 

According to department information, USDA’s advisory board plays a key 
role in coordinating strategic planning for the National Institute of 
Food and Agriculture. 

The board is to help this institute to: 

* establish policies and priorities on a quarterly basis, 

* evaluate the effectiveness of those policies and priorities, and, 

* develop its 5-year strategic plan. 

In addition, according to the National Institute of Food and 
Agriculture, it revises its strategic plan on an ongoing basis so that 
its research, education, and extension strategies align with 
Agriculture’s goals and objectives. 

Additionally, IES’ Board Facilitates Stakeholder Involvement: 

The National Board for Education Services is required by law to: 

* Solicit advice and information from those in the field of education 
to recommend topics that require research, and; 

* Recommend ways to enhance partnerships and collaboration among other 
federal and state research agencies. 

IES also involves stakeholders by establishing: 

* Fellowships for education research at universities, and; 

* Standing review panels of scientists from university and industry 
settings who review grant applications. 

[End of Objective 3] 

Conclusions: 

A variety of elements—including research independence, transparency 
and accountability, and policy relevance—may foster conditions that 
can help achieve sound and relevant research. 

ETA’s OPDR and its component research center have limited decision-
making authority and lack formal access to executive leadership 
leaving it potentially vulnerable to undue political influence. 

Moreover, lacking specific processes and a tracking system to monitor 
progress through review inhibits transparency and may lead to 
continued delays in approving and disseminating research products. 

The involvement of stakeholders and advisory bodies can help ensure 
that decisions to conduct research are grounded in policy 
implications. Such involvement also serves to foster public 
accountability. While deciding to involve advisory bodies comes at 
some cost in time and resources, such a move may lend credibility to 
the research program. 

[End of Conclusions] 

Recommendations: 

To improve ETA’s research program, we recommend that the Secretary of 
Labor: 

* Take steps to clarify ETA’s revised organizational structure and 
ensure that OPDR reports directly to ETA’s Assistant Secretary; 

* Provide sufficient authority to ETA’s research and evaluation center 
to plan, conduct, and disseminate research; 

* Direct ETA’s research and evaluation center to establish more 
specific processes, including time frames for dissemination of 
research to promote transparency and accountability; 

* Create an information system to track research projects at all 
phases to ensure timely completion and dissemination; and; 

* Instruct ETA’s research and evaluation center to develop processes 
to routinely involve outside experts in setting its research agenda 
and to the extent required, do so consistent with the Federal Advisory 
Committee Act. 

[End of Briefing Slides] 

Appendix II: Dissemination Time Frames for ETA's Research Products 
Published in 2008: 

Table 1: ETA's 2008 Research Products by Dissemination Time Frames: 

1; 
Publication title: Current Strategies to Employ and Retain Older 
Workers; 
ETAOP#: 2008-01; 
Date of final submission to ETA/OPDR: 1/1/2008; 
Date posted on ETA publication database/Web site: 3/3/2008; 
Time frames between submission and dissemination[A]: 2.2 months. 

2; 
Publication title: Reemployment and Eligibility Assessment (REA) Study; 
ETAOP#: 2008-02; 
Date of final submission to ETA/OPDR: 3/1/2008; 
Date posted on ETA publication database/Web site: 3/12/2008; 
Time frames between submission and dissemination[A]: 11 days. 

3; 
Publication title: Assessment of Strategies to Retain Experienced 
Technical and Professional Healthcare Personnel After Retirement Age: 
Mature Healthcare Workers Focus on Group Research - Final Report; 
ETAOP#: 2008-04; 
Date of final submission to ETA/OPDR: 1/22/2008; 
Date posted on ETA publication database/Web site: 5/8/2008; 
Time frames between submission and dissemination[A]: 4 months. 

4; 
Publication title: Early Implementation of Generation I of the 
Workforce Innovation in Regional Economic Development (WIRED) 
Initiative: 2007 Interim Evaluation Report; 
ETAOP#: 2008-03; 
Date of final submission to ETA/OPDR: 5/16/2008; 
Date posted on ETA publication database/Web site: 5/21/2008; 
Time frames between submission and dissemination[A]: 5 days. 

5; 
Publication title: Evaluation of State Implementation of Section 303 
(K), Social Security Act, "SUTA Dumping" - Final Report; 
ETAOP#: 2008-05; 
Date of final submission to ETA/OPDR: 5/7/2008; 
Date posted on ETA publication database/Web site: 6/12/2008; 
Time frames between submission and dissemination[A]: 1 month. 

6; 
Publication title: Evaluation of Prisoner Re-Entry Initiative- Interim 
Report; 
ETAOP#: 2008-06; 
Date of final submission to ETA/OPDR: 9/28/2007; 
Date posted on ETA publication database/Web site: 6/16/2008; 
Time frames between submission and dissemination[A]: 8.5 months. 

7; 
Publication title: Unemployment Insurance: Assessment of the Impact of 
the 2002 Reed Act Distribution; 
ETAOP#: 2004-11; 
Date of final submission to ETA/OPDR: 12/1/2004; 
Date posted on ETA publication database/Web site: 7/30/2008; 
Time frames between submission and dissemination[A]: 3.8 years. 

8; 
Publication title: Responses to Personal Remployment Accounts (PRAs): 
Findings from the Demonstration States --Final Evaluation Report; 
ETAOP#: 2008-07; 
Date of final submission to ETA/OPDR: 6/6/2008; 
Date posted on ETA publication database/Web site: 8/11/2008; 
Time frames between submission and dissemination[A]: 2 months. 

9; 
Publication title: Navigating the U.S. Labor Market: Trends and 
Prospects for Workers; 
ETAOP#: 2003-09; 
Date of final submission to ETA/OPDR: 9/22/2003; 
Date posted on ETA publication database/Web site: 8/14/2008; 
Time frames between submission and dissemination[A]: 4.10 years. 

10; 
Publication title: Review of Recent Pilot, Demonstration, Research, 
and Evaluation Initiatives to Assist in the Implementation of Programs 
under the Workforce Investment Act; 
ETAOP#: 2003-10; 
Date of final submission to ETA/OPDR: 9/22/2003; 
Date posted on ETA publication database/Web site: 8/14/2008; 
Time frames between submission and dissemination[A]: 4.11 years. 

11; 
Publication title: Net Impact Estimates for Services Provided through 
the Workforce Investment Act; 
ETAOP#: 2005-06; 
Date of final submission to ETA/OPDR: 10/1/2005; 
Date posted on ETA publication database/Web site: 8/14/2008; 
Time frames between submission and dissemination[A]: 2.10 years. 

12; 
Publication title: Review of Alternative Methodologies for Employment 
and Training Research; 
ETAOP#: 2003-11; 
Date of final submission to ETA/OPDR: 9/22/2003; 
Date posted on ETA publication database/Web site: 8/15/2008; 
Time frames between submission and dissemination[A]: 4.9 years. 

13; 
Publication title: Anatomy of a One-Stop: Baltimore City East Side 
Career Center; 
ETAOP#: 2006-07; 
Date of final submission to ETA/OPDR: 8/20/2008; 
Date posted on ETA publication database/Web site: 8/29/2008; 
Time frames between submission and dissemination[A]: 9 days. 

14; 
Publication title: Anatomy of Two One-Stops: Camdenton and Columbia, 
Missouri; 
ETAOP#: 2006-08; 
Date of final submission to ETA/OPDR: 8/20/2008; 
Date posted on ETA publication database/Web site: 8/29/2008; 
Time frames between submission and dissemination[A]: 9 days. 

15; 
Publication title: Youth Offender Demonstration Project Evaluation 
Final Report - Volume One; 
ETAOP#: 2006-06; 
Date of final submission to ETA/OPDR: 6/1/2006; 
Date posted on ETA publication database/Web site: 9/4/2008; 
Time frames between submission and dissemination[A]: 2.3 years. 

16; 
Publication title: Workforce Development in Rural Areas - Changes in 
Access, Service Delivery and Partnerships; 
ETAOP#: 2005-07; 
Date of final submission to ETA/OPDR: 6/30/2005; 
Date posted on ETA publication database/Web site: 9/4/2008; 
Time frames between submission and dissemination[A]: 3.2 years. 

17; 
Publication title: Unemployment Insurance and Reemployment among Older 
Workers; 
ETAOP#: 2006-09; 
Date of final submission to ETA/OPDR: 7/1/2006; 
Date posted on ETA publication database/Web site: 9/4/2008; 
Time frames between submission and dissemination[A]: 2.2 years. 

18; 
Publication title: Youth Offender Demonstration Project Process 
Evaluation Final Report - Round Two; 
ETAOP#: 2004-10; 
Date of final submission to ETA/OPDR: 1/30/2004; 
Date posted on ETA publication database/Web site: 9/8/2008; 
Time frames between submission and dissemination[A]: 4.2 years. 

19; 
Publication title: Evaluation of Labor Exchange in the One-Stop 
Delivery System Environment; 
ETAOP#: 2004-09; 
Date of final submission to ETA/OPDR: 2/1/2004; 
Date posted on ETA publication database/Web site: 9/11/2008; 
Time frames between submission and dissemination[A]: 4.7 years. 

20; 
Publication title: Use of Experimental Methods in Workforce 
Evaluations; 
ETAOP#: 2005-08; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

21; 
Publication title: On the Use of Administrative Data for Workforce 
Development Program Evaluations; 
ETAOP#: 2005-09; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

22; 
Publication title: Immigration and the Effects on the US Labor Market; 
ETAOP#: 2005-10; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

23; 
Publication title: The Labor Market Effects of Globalization and TAA; 
ETAOP#: 2005-11; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

24; 
Publication title: Systemic Disincentive Effects of the UI Program; 
ETAOP#: 2005-12; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

25; 
Publication title: Older workers and the Labor Market/Labor Market 
Policies for Older Workers; 
ETAOP#: 2005-13; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

26; 
Publication title: Programs to Support Out-of-School Youth; 
ETAOP#: 2005-14; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

27; 
Publication title: Community College Training and the Workforce 
Investment System; 
ETAOP#: 2005-15; 
Date of final submission to ETA/OPDR: 4/18/2005; 
Date posted on ETA publication database/Web site: 9/16/2008; 
Time frames between submission and dissemination[A]: 3.5 years. 

28; 
Publication title: Project GATE; 
ETAOP#: 2008-08; 
Date of final submission to ETA/OPDR: 5/1/2008; 
Date posted on ETA publication database/Web site: 9/17/2008; 
Time frames between submission and dissemination[A]: 4.5 months. 

29; 
Publication title: Flexible Learning Options for Adult Students; 
ETAOP#: 2008-09; 
Date of final submission to ETA/OPDR: 4/1/2008; 
Date posted on ETA publication database/Web site: 10/16/2008; 
Time frames between submission and dissemination[A]: 6.5 months. 

30; 
Publication title: Implementation Analysis of High Growth Job Training 
Initiative (HGJTI); 
ETAOP#: 2008-10; 
Date of final submission to ETA/OPDR: 6/1/2008; 
Date posted on ETA publication database/Web site: 10/16/2008; 
Time frames between submission and dissemination[A]: 4.5 months. 

31; 
Publication title: Literature Review: Intermediaries and FBCOs Working 
Together; 
ETAOP#: 2008-11; 
Date of final submission to ETA/OPDR: 10/2/2008; 
Date posted on ETA publication database/Web site: 10/29/2008; 
Time frames between submission and dissemination[A]: 27 days. 

32; 
Publication title: Youth Opportunity Grants Initiative (YO); 
ETAOP#: 2008-12; 
Date of final submission to ETA/OPDR: 12/1/2005; 
Date posted on ETA publication database/Web site: 12/23/2008; 
Time frames between submission and dissemination[A]: 3.0 years. 

33; 
Publication title: What's Known About the Effects of Publicly-Funded 
Employment and Training Programs; 
ETAOP#: 2006-10; 
Date of final submission to ETA/OPDR: 4/21/2006; 
Date posted on ETA publication database/Web site: 12/15/2008; 
Time frames between submission and dissemination[A]: 2.8 years. 

34; 
Publication title: Competency Models - A Review of the Literature and 
the Role of the Employment and Training Administration (ETA); 
ETAOP#: 2008-13; 
Date of final submission to ETA/OPDR: 1/29/2008; 
Date posted on ETA publication database/Web site: 12/23/2008; 
Time frames between submission and dissemination[A]: 11 months. 

Source: Department of Labor's Employment and Training Administration 
and GAO analysis of data. 

Note: The information presented in this table was provided by ETA 
officials. 

[A] Time frames were calculated using the date each research product 
was submitted to ETA and the date it was posted on ETA's Web site. 

[End of table] 

[End of section] 

Appendix III: Comments from the Department of Labor: 

U.S. Department of Labor: 
Assistant Secretary for Employment and Training: 
Washington, D.C. 20210: 

January 12, 2010: 

Mr. George A. Scott: 
Director: 
Education, Workforce, and Income Security Issues: 
U.S. Government Accountability Office: 
441 G. Street, N.W. 
Washington, D.C. 20548: 

Dear Mr. Scott: 

The Employment and Training Administration (ETA) is in receipt of the 
draft Government Accountability Office (GAO) draft report number 10-
243 entitled, "Employment and Training Administration: Increased 
Authority and Accountability Could Improve Research Program." We 
appreciate the opportunity to comment on the draft. 

Responses to Recommendations: 

On page 5 and slide 50, GAO makes five recommendations regarding ETA's 
research program. Following is ETA's response. 

Recommendation #1: Organizational Structure: 
Take steps to clarify ETA revised organizational structure and ensure 
that OPDR reports directly to ETA's Assistant Secretary. 

The most recent organizational structure for the Employment and 
Training Administration (ETA) has the Office of Policy Development and 
Research (OPDR) reporting directly to the Assistant Secretary for 
Employment and Training. See [hyperlink, 
http://www.doleta.gov/pdf/OrgChart.pdf]. Consistent with the Essential 
Elements of a Sound Research and Evaluation Program,[Footnote 4] under 
the most recent organizational structure, the Administrator for ETA's 
research and evaluation center has direct access to and reports to the 
Assistant Secretary for Employment and Training. The center, lead by 
the OPDR Administrator, is placed organizationally within ETA's 
leadership structure to have influence and input to the organization's 
ultimate decision maker, the Assistant Secretary.[Footnote 5] 

Recommendation #2: Decision-Making Authority: 
Provide sufficient decision-making authority to ETA's research and 
evaluation center to plan, conduct, and disseminate research. 

ETA's research and evaluation center currently provides 
recommendations to the Assistant Secretary regarding plans for 
conducting and disseminating research. Meetings with ETA's decision-
makers are scheduled to discuss such plans and decisions are 
communicated with all relevant parties within the agency for 
implementation. ETA's research and evaluation center develops the 
agency's plans for conducting research collaboratively with the 
relevant program offices. This ensures that research and evaluation 
efforts are coordinated and aligned with agency programmatic 
priorities. In addition, program offices participate in the 
dissemination decision-making process by reviewing and commenting on 
final draft reports. 

Recommendation #3: Establish Specific Time Frames: 
Direct ETA's research and evaluation center to establish more specific 
processes, including time frames for dissemination of research to 
promote transparency and accountability. 

ETA's process improvement steps, developed in 2007 and finalized in 
2008 in agreement with the Office of Management and Budget, provide 
set time frames for the dissemination of research. The time frames arc 
based on ranges in order to accommodate the variance in reviewing and 
finalizing reports. This iterative process involves the Federal 
project officer, OPDR staff, the relevant program offices, and 
contractor staff. The time frames can range from one week to one 
month, or one month to several months to show the ideal review 
schedule but allows for a realistic schedule that takes into 
consideration the workload of staff serving as project officer, the 
availability of program office staff and other OPDR staff in reviewing 
reports, and the contractor's completion schedule of a final draft 
report. 

ETA is committed to publishing reports that disseminate evidence-based 
knowledge of what works and what does not on a timely basis to assist 
policymakers with policy and programmatic decisions. ETA has shared 
and disseminated research and evaluation findings at the agency's 
national conferences that showcases research. Recently, ETA's 
commitment to dissemination was evidenced at the recent Recovery and 
Reemployment Research Conference which featured findings from many 
Department-funded research and evaluation projects and engaged a broad 
spectrum of communities, including workforce investment, research, 
education, oversight agencies, non-profit organizations, and public 
policy makers. 

In addition, ETA has sponsored a series of briefings titled Coffee 
House Briefing Series that featured findings and results from ETA-
funded research. These briefings are announced on the agency's 
Intranet website. The briefing announcements have also been shared 
with colleagues in other Department of Labor agencies such as the 
Office of the Assistant Secretary for Policy and the Veterans 
Employment and Training Service as well as with other Federal 
agencies, including the Departments of Health and Human Services and 
Education. These agencies and departments have attended several of the 
briefing sessions. These briefing sessions are well attended by ETA 
staff from different program offices. 

Lastly, in keeping with the Department's commitment to the President's 
Open Government philosophy, ETA's research will also be disseminated 
as part of the Department's efforts to make all DOL agencies' research 
products available through its Open Government Web site. 

Recommendation #4: Track Research: 
Create an information system to track research projects at all phases 
to ensure timely completion and dissemination. 

Currently, ETA research projects are tracked by individual project 
officers who report any significant delays to their Team Leaders and 
the Division Director. A departmental research inventory is produced 
semi-annually which updates the status of all projects, however, this 
inventory has only limited project management value. Therefore, ETA is 
planning to implement a centralized, electronic tracking system for 
its research projects. The new system will track major project 
milestones during the course of the study and steps in the review and 
dissemination of reports, once reports have been completed. The 
project officers will be responsible for updating the system as events 
occur. 

Recommendation #5: Involve Outside Experts: 
Instruct ETA's research and evaluation center to develop processes to 
routinely involve outside experts in setting its research agenda and 
to the extent required, do so consistent with the Federal Advisory 
Committee Act. 

ETA has in the past, and most recently, increased its efforts to 
involve outside experts in setting its research agenda. Under Section 
171 of the Workforce Investment Act, ETA is required to develop a Five-
Year Strategic Research Plan in consultation with states, localities 
and other interested parties. The strategic plan provides guidance on 
projects for pilots, demonstrations, research and evaluations. The 
plan provides information on the types of research topics and projects 
that should be taken into consideration. The first plan transmitted to 
the Congress covered the years 2000-2005. For the years covering 2002-
2007 and 2004-2009, the strategic plans remained in draft form and 
were not transmitted to the Congress since they did not pass 
Departmental clearance as they were compendiums of various research 
papers rather than a plan to help guide the agency's pilots, 
demonstrations, research and evaluations. The most recent strategic 
research plan for 2007-2012, was transmitted to the Congress on July 
20, 2007. All of these plans were developed in consultation with 
outside experts. 

In addition, ETA has engaged informally in dialogues with outside 
experts through conferences as well as through interagency 
collaborations in the development of the research and evaluation 
agenda. For instance, ETA has collaborated with the research and 
evaluation centers of the Departments of Education and Health and 
Human Services on areas of shared interest. These efforts are designed 
to identify ways each agency's resources and expertise can be 
leveraged in support of each agency's research agenda. Another 
informal, however routine, method for leveraging input from 
programmatic and research experts on ETA's research agenda has been 
through the sponsorship of the Research Showcase annually at the past 
Workforce Innovations Conferences and the most recent Recovery and 
Reemployment Research Conference. These venues afford ETA 
opportunities to engage in a dialogue with research and program 
experts on completed research and evaluation projects and discuss 
opportunities for future research. 

Thank you for the opportunity to comment on this report. Attached 
you'll find technical comments on report and slides. 

Sincerely, 

Signed by: 

Jane Oates: 
Assistant Secretary: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

George A. Scott, Director (202) 512-7215 or scottg@gao.gov: 

Acknowledgments: 

In addition to the contact listed above, Dianne Blank (Assistant 
Director) and Kathleen White (Analyst-in-Charge) supervised the 
development of this product. Linda Stokes, Amy Sweet, and Ashanta 
Williams made significant contributions to all aspects of this report. 
In addition, Stephanie Shipman assisted with the conceptual 
development of our study; Amanda Miller assisted with design and 
analyses; James Bennett provided graphic assistance; Susan Bernstein 
provided writing assistance; Alex Galuten provided legal support; and 
Karen Brown and Melissa Jaynes verified our findings. 

[End of section] 

Related GAO Products: 

Workforce Investment Act: Labor Has Made Progress in Addressing Areas 
of Concern, but More Focus Needed on Understanding What Works and What 
Doesn't. GAO-09-396T. Washington, D.C.: February 26, 2009. 

Federal Research: Policies Guiding the Dissemination of Scientific 
Research from Selected Agencies Should Be Clarified and Better 
Communicated. GAO-07-653. Washington, D.C.: May 17, 2007. 

Data Quality: Expanded Use of Key Dissemination Practices Would 
Further Safeguard the Integrity of Federal Statistical Data. GAO-06-
607. Washington, D.C.: May 31, 2006. 

Workforce Investment Act: Substantial Funds Are Used for Training, but 
Little Is Known Nationally about Training Outcomes. GAO-05-650. 
Washington, D.C.: June 29, 2005. 

Program Evaluation: An Evaluation Culture and Collaborative 
Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.: 
May 2, 2003. 

Workforce Investment Act: Improvements Needed in Performance Measures 
to Provide a More Accurate Picture of WIA Effectiveness. GAO-02-275. 
Washington, D.C.: February 1, 2002. 

[End of section] 

Footnotes: 

[1] 29 U.S.C. § 2917(c). 

[2] In addition to ETA, we present information for comparative 
purposes on the following research centers: the Department of 
Agriculture's (USDA) Economic Research Service (ERS) and National 
Institute of Food and Agriculture; the Department of Education's 
(Education) Institute of Education Sciences (IES); the Department of 
Housing and Urban Development's (HUD) Office of Policy Development and 
Research; the Environmental Protection Agency's (EPA) Office of 
Research and Development; and the National Science Foundation (NSF). 

[3] 29 U.S.C. § 2917(c). 

[4] GAO Draft Report titled "Employment and Training Administration: 
Increase Authority and Accountability Could Improve Research Program", 
[hyperlink, http://www.gao.gov/products/GA0-10-243], page 21 (slide 
15). 

[5] Ibid; page 25 (slide 19). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: