Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts
Highlights
What GAO Found
The Department of Education's (Education) Institute of Education Sciences (IES) supports high-quality research, but lacks certain key procedures needed to fulfill other aspects of its mission. According to stakeholders, IES has substantially improved the quality of education research since its inception over a decade ago. However, GAO identified concerns with IES's ability to produce timely and relevant research. For example, IES's efforts are slow to respond to stakeholders' needs, in part, because the time IES's products have spent in peer review has substantially increased in recent years--from an average of 117 days in fiscal year 2011 to 150 days in fiscal year 2013--and IES does not monitor some aspects of these timeframes. In addition, IES does not have a structured process for incorporating stakeholder input into its research agenda, which previous GAO work has shown to be key to sound federal research programs. Lastly, IES's performance measures do not fully reflect its current programs, which is not consistent with leading practices GAO has identified for performance management. For example, IES does not publicly report on the overall performance of the Regional Educational Laboratories (REL) program, which constitutes one of the agency's largest investments. IES officials said that they have begun to develop new performance measures for all of their programs, but these will not be in place until fiscal year 2015.
Although Education's research and technical assistance groups have taken steps to disseminate relevant research to the education field, IES does not always assess these efforts. Some stakeholders raised concerns about the dissemination of relevant products from the RELs and Research and Development Centers (R & D Center). For example, they told GAO that these groups do not always adapt their products for use by both policymaker and practitioner audiences. Further, IES has not fully assessed REL and R & D Center dissemination efforts. As a result, IES does not know if these efforts are effective in meeting their mandated goal of providing usable research and information to stakeholders. GAO's prior work on information dissemination suggests that further assessment could help inform IES's oversight of the RELs and R & D Centers to improve these groups' dissemination to key audiences.
IES coordinates with other relevant federal research agencies to increase the use of research evidence in federal decision-making. For example, IES and the National Science Foundation recently released guidelines to help improve the quality of evidence resulting from federally-funded education research, which stakeholders said will benefit the education field. Within the department, IES plans evaluations of Education programs through coordination with various other offices. However, Education lacks statutory authority to combine evaluation funds for Elementary and Secondary Education Act (ESEA) programs, which officials said limits the department's ability to conduct high-quality evaluations of programs it considers most important. GAO's prior work has shown that many Education programs, especially smaller programs, have not been evaluated, limiting the ability of Congress to make informed policy decisions. For example, in 2009, GAO reported that 11 of Education's teacher quality programs had not been evaluated in more than 7 years. Officials said that the ability to combine evaluation funds would allow the department to conduct needed evaluations of ESEA programs.
Why GAO Did This Study
The federal government has a longstanding role in conducting education research and collecting education data, and IES has a broad mission to provide this information to a wide variety of stakeholders. With a fiscal year 2013 budget of just under $600 million, IES is Education's primary research and evaluation arm. GAO was asked to review IES's performance. This report examines: (1) the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, (2) the extent to which selected Education research and technical assistance groups disseminate relevant products to the education field, and (3) how IES coordinates its activities with other relevant federal research agencies and within Education. GAO reviewed relevant federal laws and regulations, agency documents and data, interviewed agency officials and stakeholders, and analyzed information from selected research and technical assistance groups. GAO also compared IES's practices to federal internal control standards and leading practices for performance management and collaboration.
Recommendations
GAO recommends that Congress consider granting Education authority to combine funds authorized for evaluation of ESEA programs, and IES use available data to manage its peer review process, develop a structured process to gather stakeholder input, develop performance measures that reflect all key agency activities, and assess REL and R & D Center dissemination strategies. Education agreed with our recommendations.
Matter for Congressional Consideration
| Matter | Status | Comments |
|---|---|---|
| To ensure that Education can conduct critical program evaluation work, Congress should consider granting Education authority to combine funds authorized for evaluations of ESEA programs and target them to high-priority evaluations, with appropriate measures to ensure transparency and accountability for how the funds will be used. | The fiscal year 2014 and 2015 Department of Education appropriations acts authorized Education to use evaluation funds reserved under section 9601 of ESEA to evaluate any ESEA program without respect to the source of the funds. According to Education, in 2014, the department combined funds totaling $4.8 million from a number of ESEA programs in order to support new, high-priority evaluations, including short turnaround impact evaluations on technology-enabled strategies that are being used to improve instruction and outcomes for elementary and secondary school students. |
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Institute of Education Sciences | To improve the management and accountability of IES's research and evaluation efforts, IES should use available data to routinely monitor all stages of its peer review process and identify opportunities to improve the timeliness of its reviews. |
In order to ensure that all the steps in the peer review process are implemented in a timely manner, Institute of Education Sciences (IES) has developed a comprehensive system for monitoring the time required for all steps in the report production process. Since January 2014, IES has had a tracking system in place to track all of the National Center for Education Evaluation's (NCEE's) evaluation reports that are peer-reviewed through the Standards and Review Office (SRO). The tracking system includes dates for receipt of the first draft of a report, as well as all subsequent steps in the review and revision process, including receipt of peer review comments and a response memo from SRO, receipt of subsequent drafts from the contractor, final approval for publication by SRO, and posting on the IES website. NCEE uses summary statistics for each report and the Center as a whole to monitor the peer review process. According to IES officials, these data are used to identify delays in the peer review process for each report and to develop strategies to resolve them.
|
| Institute of Education Sciences | To improve the management and accountability of IES's research and evaluation efforts, IES should develop a structured process to systematically gather input from policymakers and practitioners and use this input when developing its research agenda. |
IES currently obtains input from practitioners and policymakers in a variety of ways, including through the National Board for Education Sciences and by convening technical working groups for input on its research programs. In order to further expand opportunities for practitioner and policymaker input, Institute of Education Sciences (IES) will annually solicit practitioner and policy input on its research agenda through the IES website and with an IES Newsflash e-mail alert. We believe this is an important step towards increasing input from these key IES stakeholder groups and will close this recommendation once these solicitations have been issued. ED reported in FY15 that IES had addressed this recommendation in 2014 in two ways: by convening Technical Working Groups of researchers and practitioners and by seeking broader public feedback through the notices on the IES website and its listserve. In 2015, IES used this feedback to shape its activities, including the development of the Pathways to the Education Sciences Research Training program (as described on the IES research blog: http://ies.ed.gov/blogs/research/2015/06/04/default). This program, modeled on similar efforts at NSF and NIH, responds to concerns identified through these actions.
|
| Institute of Education Sciences | To improve the management and accountability of IES's research and evaluation efforts, IES should develop performance measures, including targets and goals that clearly demonstrate results, and that reflect its current programs and all key agency activities, such as the performance of the RELs and its new grant programs supporting researcher-practitioner partnerships. |
As part of the Department of Education's strategic planning process, Institute of Education Sciences (IES) has reviewed its existing performance measures to align them with its current research investments, and is developing new performance measures for its knowledge utilization activities, such as the Regional Educational Laboratories. The Department plans to include these new and revised measures in a future Congressional budget justification. ED reported in FY15 that IES revised its existing performance measures for the Research, Development, and Dissemination (RD&D), Statistics, and Assessment programs. These measures were reported in the Congressional budget justification for FY16.
|
| Institute of Education Sciences | To improve the management and accountability of IES's research and evaluation efforts, in order to identify leading practices and target areas for improvement, IES should assess the effectiveness of REL and R & D Center dissemination strategies by, for example, collecting consistent data and lessons learned from these groups to inform future dissemination efforts. |
Institute of Education Sciences (IES) is evaluating how to collect better data on its dissemination activities in order to assess their effectiveness, including activities conducted by the Regional Educational Laboratories (REL) and Research and Development (R & D) Centers. ED reported in FY15 that IES has collected data on the dissemination activities of the RELs and the R&D centers. For the R&D centers, IES compiled a list of all peer-reviewed publications produced by the current R&D Center grantees since 2012. During this period, R&D Center grantees published 59 peer-reviewed publications, 32 of which have already been indexed in the Education Resources Information Center (ERIC). For grants issued in or after FY12 (5 of the 11 grants covered by this publication review), IES requires that all its grantees submit a final manuscript for all peer-reviewed publications within 12 months of publication for inclusion in ERIC, which will facilitate broader, more timely access to the full text of scientific publications resulting from IES-funded research in the near future. Each REL surveys all participants in each of its dissemination activities about the quality and usefulness of these offerings, using a set of questions that is partially standardized. These data are compiled and reported at the program level in the Congressional budget justifications. The National Center for Education Evaluation (NCEE) uses this information to inform its monitoring of the REL contracts and to share "lessons learned" across the RELs.
|