Employment and Training Administration:

More Actions Needed to Improve Transparency and Accountability of Its Research Program

GAO-11-285: Published: Mar 15, 2011. Publicly Released: Apr 14, 2011.

Additional Materials:

Contact:

George A. Scott
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

To help guide the nation's workforce development system, the Department of Labor's (Labor) Employment and Training Administration (ETA) conducts research in areas related to job training and employment. Building upon our earlier work, GAO examined the following: (1) To what extent do ETA's research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved the availability of its research since our last review in January 2010? To answer these questions, GAO reviewed ETA's research reports disseminated between January 2008 and March 2010 costing $1 million or more, as well as ongoing studies costing $2 million or more. GAO also convened a virtual expert panel, interviewed Labor and ETA officials, and reviewed relevant documents.

ETA's 2007 to 2012 research plan generally addressed key employment and training issues, but some studies were limited in their usefulness. Most experts on our panel reported that the areas in ETA's plan reflected key national employment and training issues at least to a moderate extent. ETA invested most of its research and evaluation resources in the areas of Unemployment Insurance and increasing labor market participation of underutilized groups. Of the $96 million that ETA invested in the 58 research reports we reviewed, more than half--or about $56 million--funded studies in these two areas. The methodological approaches and statistical procedures researchers used in the studies we reviewed were generally consistent with the questions posed, but the studies were not always useful for informing policy and practice. For example, in one study, shortcomings in the data collection phase limited the strength of the findings. Experts suggested that ETA include more varied and rigorous methodologies in its future research projects. They also reported that future research should address additional areas, including a focus on employment and training approaches that work and for whom. Labor and ETA have taken steps to improve the way research is conducted, but additional actions are needed. In acknowledging the need for more rigorous evaluations to inform its policies, Labor recently established the Chief Evaluation Office to oversee departmentwide research and evaluation efforts. In addition, ETA made changes to some of its research practices. For example, ETA has begun involving outside experts in developing its research plan. Despite these improvements, ETA's process lacks critical elements needed to ensure that current improvements become part of its routine practices. For example, ETA's process lacks a formal provision to consult with the newly established Chief Evaluation Officer at important points in the research process. Moreover, ETA's current research practices fall short of ensuring research transparency and accountability--essential elements of a sound research program. For example, its research and evaluation center lacks safeguards to protect it from undue outside influence. ETA has recently begun efforts to increase the rigor of its research designs, but has faced design and implementation challenges. For example, some of ETA's ongoing research studies face challenges in recruiting large enough sample sizes to meet the studies' objectives. ETA has improved the availability of its research findings, but it lacks a plan for assessing the usability of its Web-based search page--the primary tool for making ETA's research publicly available. ETA recently improved the timeliness with which it disseminates its research reports, decreasing the average number of days to release its reports to the public from 804 days in 2008 to 76 days in 2009. ETA has taken steps to update its online, Web-based search page. However, the agency has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. In addition to its research database, ETA uses a variety of other methods to disseminate its research, including providing its research reports at conferences and internal briefings. Experts suggested that ETA consider other effective dissemination methods, such as publishing a one-page summary of research findings. GAO recommends that ETA formally include the Chief Evaluation Officer in its research process, create a mechanism to enhance the transparency and accountability of its research program, and develop a plan to ensure that research reports are accessible through its Web-based search page. Labor agreed with our recommendations and noted its ongoing efforts to improve its research program. While these efforts are important, GAO stresses the need for additional actions to fully address the recommendations.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Department of Labor concurred with our recommendation, and stated in FY14 that ETA has fully incorporated routine involvement of the Chief Evaluation Officer (CEO) in its research process for the past two years. Some of the key research milestones where the CEO is involved include: agenda-setting, planning and design, procurement, implementation, report review, and dissemination. The CEO and senior staff meet regularly with ETA's Division of Research and Evaluation to review progress on current projects, and planning for new projects.

    Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to formally incorporate into its research process the routine involvement of the Chief Evaluation Officer at key milestones, including at the development of ETA's annual research agenda and spending priorities, as well as at the early stages of developing specific research projects.

    Agency Affected: Department of Labor

  2. Status: Closed - Implemented

    Comments: ETA stated that it has engaged in efforts to involve outside experts to develop its research policies and processes. In 2010, ETA began a series of meetings with a panel of outside experts to develop its 5-year research plan, which was submitted to Congress and posted on ETA's web page in May 2013. For large scale ETA contracted evaluations, the agency now requires the identification of experts as members of a technical working group to provide assistance at key points in the research process. Members are involved in reviewing evaluation design plans and preliminary report drafts. The agency has also engaged third party organizations to help with peer review, and engaged colleagues at the Departments of Health and Human Services and Education to review reports prior to publication.

    Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to develop a mechanism to enhance the transparency and accountability of ETA's research program. For example, such a mechanism might include involving advisory bodies or other entities outside ETA, in efforts to develop ETA's research policies and processes.

    Agency Affected: Department of Labor

  3. Status: Closed - Implemented

    Comments: ETA has established several methods to ensure its products are easily accessible to stakeholders and the general public through its research database. ETA maintains contractor support to enhance and facilitate the dissemination of its research findings and methods, and evaluation activities. The contractor, for example, updates annotated bibliographies of ETA-funded and other research, pilots and demonstrations, and evaluation studies; analyzes alternative approaches to disseminate findings; and provides peer review of ETA-funded or other research studies, and other dissemination support activities. ETA is also collaborating with DOL's Chief Evaluation Officer (CEO) in the development of a web-based clearinghouse of program research. In addition, ETA hosts briefings to disseminate new research and evaluation findings to internal and external stakeholders. In May 2013, ETA sponsored a workshop for DOL agencies that provided an overview of evidence-based policy making. Finally, all research products are available on ETA's website and by request.

    Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to develop a formal plan for ensuring that ETA's research products are easily accessible to stakeholders and to the general public through its searchable database. Such a plan could involve requiring Labor to assess the overall effectiveness of its Web-based search page, including user satisfaction with search features.

    Agency Affected: Department of Labor

 

Explore the full database of GAO's Open Recommendations »

Dec 17, 2014

Nov 13, 2014

Jul 23, 2014

May 15, 2014

Apr 9, 2014

Mar 5, 2014

Jan 23, 2014

Dec 20, 2013

Dec 6, 2013

Dec 2, 2013

Looking for more? Browse all our products here