This is the accessible text file for GAO report number GAO-10-644 
entitled 'Department Of Education: Improved Dissemination and Timely 
Product Release Would Enhance the Usefulness of the What Works 
Clearinghouse' which was released on July 23, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 
GAO: 

July 2010: 

Department Of Education: 

Improved Dissemination and Timely Product Release Would Enhance the 
Usefulness of the What Works Clearinghouse: 

GAO-10-644: 

GAO Highlights: 

Highlights of GAO-10-644, a report to congressional committees. 

Why GAO Did This Study: 

In connection with the Omnibus Appropriations Act, 2009, GAO was 
required to study the What Works Clearinghouse (WWC), a federal source 
of evidence about effective education practices. Operating through a 5-
year contract awarded by the U.S. Department of Educations Institute 
of Education Sciences (IES), the WWC reviews education research and 
disseminates its findings. GAO examined: (1) the extent to which the 
WWC review process meets accepted standards for research evaluation 
and how the WWC has responded to recommendations and criticism, (2) 
how WWC output and costs have changed over time and how its 
performance is measured, and (3) how WWC products are disseminated and 
how useful educators find them to be. To conduct its work, GAO 
reviewed WWC-related documents, analyzed cost and performance data, 
surveyed all states and a nationally representative sample of school 
districts, and interviewed IES officials, WWC contractors, 
researchers, and others. 

What GAO Found: 

GAO as well as a congressionally mandated panel of experts, found that 
the WWCs review process, which includes screening studies to 
determine if they meet WWC criteria, follows accepted standards for 
evaluating research on the effectiveness of education interventions. 
WWC is responding to recommendations made by the expert panel to 
further improve its review and reporting processes. For example, the 
panel recommended improvements in the way the WWC presents information 
to readers on the reasons why studies do not qualify for review. The 
WWC is revising a report template to include a table summarizing which 
studies met or did not meet WWC criteria for evaluating research. The 
WWC has also responded to researchers who have criticized the WWC for 
presenting limited information because its screening criteria exclude 
some rigorous research designs that may be appropriate for evaluating 
certain education programs, such as special education. The WWC 
responded to this criticism by creating new standards that include two 
additional study designs and by creating a new product, called a 
practice guide, which includes a wider range of research. 

WWCs report output and scope increased under the current contract. 
For example, the WWC increased its production of various reports, 
introduced new products, and developed new processes for evaluating 
research. However, IES had a substantial backlog in its product review 
process from January 2009 to May 2010. The backlog generally decreased 
the timeliness of WWC reports, with 20 reports being delayed by up to 
6 months. To support the increases in output and scope, WWCs costs 
doubled from the previous contract to the current one. Both contracts 
designated about 60 percent of costs to production, while the other 40 
percent of costs support other tasks, such as communications, 
dissemination, and process development. IES performance goals for the 
WWC primarily relate to the number of reports produced. However, IES 
has not developed performance measures related to the cost or 
usefulness of WWC products. 

Education uses WWC contractors, Regional Educational Laboratories 
(RELS) and the Doing What Works (DWW) Web site to disseminate 
information about WWC products; however, awareness and use of the WWC 
varies among states, districts, teachers, and principals. WWC 
contractors disseminate product information in various ways including 
email alerts and presentations. The RELs host events featuring WWC 
products for state, district, and school officials and DWW provides 
resources to educators based on WWC products. Based on our survey, 
officials from 33 of 38 state education agencies that responded to our 
survey and an estimated 42 percent of school districts have heard of 
the WWC. Those states and school districts generally used the WWC to a 
small or moderate extent to inform decisions on effective practices. 
Based on our survey, states and school districts reported that they 
would likely increase their use of the WWC if it included a broader 
array of information or more timely information. 

What GAO Recommends: 

GAO recommends that IES: develop and implement strategies to avoid 
backlogs in WWC product reviews; establish performance measures 
related to costs and usefulness; and improve dissemination efforts to 
promote awareness and use of the WWC. Education generally agreed with 
GAOs recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-10-644] or key 
components. For more information, contact Cornelia Ashby at (202) 512-
7215 or ashbyc@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

WWC Reviews Research in Accordance with Accepted Standards and Has 
Responded to Recommendations and Criticisms: 

WWC's Output and Costs Increased; However, IES Has Not Developed 
Adequate Performance Measures Related to Cost or Product Usefulness: 

Education Has Three Primary Ways to Disseminate Information about WWC 
Products, but Awareness and Use Vary among Target Audiences: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Other Sources of Information Districts Use To Identify 
Effective Education Practices: 

Appendix III: IES and WWC Response to Expert Panel Recommendations: 

Appendix IV: Comments from the Department of Education: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Products and Registries Available on the WWC Web Site: 

Table 2: Two Study Designs That Meet the WWC Standards with or without 
Reservations: 

Table 3: New WWC Publications and Reports: 

Table 4: Task Category Definitions and Changes between Contracts: 

Table 5: Estimates and Confidence Intervals for Figure 8: 

Table 6: Estimates and Confidence Intervals for Figure 10: 

Table 7: Estimates and Confidence Intervals for Figure 12: 

Table 8: Estimates and Confidence Intervals for Figure 14: 

Table 9: Conferences Attended to Administer Questionnaires to Teachers 
and Principals: 

Figures: 

Figure 1: WWC Research Review Process for Interventions: 

Figure 2: Percentage of Interventions with Positive or Potentially 
Positive Ratings Categorized by the Amount of the Evidence Supporting 
Those Ratings: 

Figure 3: Studies Reviewed That Meet WWC Evidence Standards: 

Figure 4: Publication Quantities, by Contract Year (CY) for Current 
WWC Contract: 

Figure 5: Average Time for IES Peer Reviews of Released Intervention 
Reports and Quick Reviews, by Contract Year (CY) and Quarter (Q) for 
Current WWC Contract: 

Figure 6: IES Peer Review Backlog for Intervention Reports and Quick 
Reviews, by Contract Year (CY) and Quarter (Q) for Current WWC 
Contract: 

Figure 7: WWC Costs, by Task Categories and Contracts: 

Figure 8: Sources from Which District Officials Heard of the WWC: 

Figure 9: Extent to Which States Use WWC for Various Purposes: 

Figure 10: Extent to Which School Districts That Have Used the 
Clearinghouse Used It for Various Purposes: 

Figure 11: Extent to Which States Use Specific WWC Products: 

Figure 12: Extent of Specific Product Use among Districts That Use the 
Clearinghouse: 

Figure 13: Number of States That Reported They Would Likely Increase 
Their Use of WWC Given Certain Changes: 

Figure 14: Estimated Percent of School Districts That Have Accessed 
the WWC That Would Likely Increase Their Use of the WWC Given Various 
Changes: 

Figure 15: GAO's Web-based Survey of State Departments of Education 
and Local Educational Agencies in the 50 States and the District of 
Columbia: 

Abbreviations: 

AYP: adequate yearly progress: 

DWW: Doing What Works: 

Education: U.S. Department of Education: 

ESEA: Elementary and Secondary Education Act of 1965: 

IES: Institute of Education Sciences: 

LEA: local educational agency: 

Recovery Act: American Recovery and Reinvestment Act of 2009: 

REL: Regional Educational Laboratories: 

WWC or Clearinghouse: What Works Clearinghouse: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

July 23, 2010: 

Congressional Committees: 

The U.S. Department of Education's What Works Clearinghouse (WWC or 
Clearinghouse) was established as a federal source of scientific 
evidence about "what works" in education. The Institute of Education 
Sciences (IES), a division of the U.S. Department of Education 
(Education), created the WWC in 2002, in part to help educators 
identify and use scientifically-based practices as specified in the 
Elementary and Secondary Education Act of 1965 (ESEA).[Footnote 1] The 
WWC, which is operated by an independent contractor, conducts 
systematic reviews of education research and disseminates information 
on its Web site about the effectiveness of the practices reported in 
these research studies. Currently operating under a $50 million 5-year 
contract, the Clearinghouse has generated criticism in the education 
research evaluation field on the timeliness of its reviews, its 
standards for study inclusion, and the methodological soundness of its 
research review process.[Footnote 2] 

An explanatory statement submitted in lieu of a conference report for 
the Omnibus Appropriations Act, 2009, directed GAO to examine how the 
WWC reviews education research and to address concerns about the 
operation, cost, and usefulness of the WWC.[Footnote 3] Specifically, 
GAO was required to determine whether the WWC review process met 
current standards for evaluating research and to examine the output 
and cost for completing reviews, the degree of consistency of review 
procedures across the various topics addressed, and the usefulness of 
the Clearinghouse for practicing educators. To conduct this work, we 
examined (1) the extent to which the WWC review process meets accepted 
standards for research evaluation and how the WWC has responded to 
recommendations and criticisms of its processes, (2) how the WWC's 
output and costs have changed over time and how IES measures WWC 
performance, and (3) how WWC products are disseminated and how useful 
education professionals find them to be. 

To address all of our objectives, we interviewed and obtained 
information from IES officials and the current and former WWC 
contractors, as well as representatives from various educational 
organizations. In addition, to address objective 1, we reviewed a 
prior GAO report that examined WWC procedures and standards, an expert 
panel report that previously assessed the validity of the WWC review 
process, literature, and procedures used by other organizations that 
conduct systematic reviews of research. We also reviewed the 
Clearinghouse's response to the expert panel and to specific 
criticisms in education research literature. To determine how 
performance and costs changed over time (objective 2), we analyzed the 
costs and productivity of the WWC contractors by reviewing budget, 
expenditure, and performance data. For objective 3, we administered a 
Web-based survey to state education agencies in all 50 states and the 
District of Columbia and a nationally representative sample of school 
districts;[Footnote 4] interviewed IES's 10 Regional Educational 
Laboratories; and gathered nongeneralizeable information from teachers 
and principals at four conferences. Appendix I explains our scope and 
methodology in more detail. We performed our work from September 2009 
to July 2010 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

The mission of the WWC is to be a central source of scientific 
evidence for what works in education.[Footnote 5] To accomplish this, 
the WWC reviews existing education research and posts information 
based on its reviews on the WWC Web site, http://ies.ed.gov/ncee/wwc/. 

The types of products currently available on the WWC Web site are 
described in table 1. 

Table 1: Products and Registries Available on the WWC Web Site: 

Product: Intervention reports; 
Description: Summarize all of the research reviewed for a particular 
intervention within a topic area. Each report offers an overview of 
the intervention, summarizes all relevant research, and provides a 
rating of effectiveness. Studies featured in intervention reports must 
meet WWC evidence standards with or without reservations; 
Number: 130; 
Example: Accelerated Reader: WWC reviewed the evidence pertaining to 
the effectiveness of this specific curriculum with respect to certain 
reading outcomes. 

Product: Practice guides; 
Description: Contain recommendations for educators to address 
challenges in their classrooms. Assign strength of evidence ratings to 
each recommendation (strong, moderate, low). Rely to some extent on 
expert opinion; 
Number: 12; 
Example: Structuring Out-of-School Time to Improve Academic 
Achievement: WWC published general recommendations on how to design 
out-of-school time programs that will increase student learning. 

Product: Quick reviews; 
Description: Assess the quality of research evidence from single 
studies recently featured in the media to determine if they meet WWC 
evidence standards; 
Number: 40; 
Example: Recess and Classroom Behavior: WWC reviewed a study profiled 
in the news that examined whether providing daily recess to third 
graders improves their classroom behavior. 

Product: Multimedia; 
Description: Audio files, video files, presentations, and transcripts 
from WWC events; 
Number: N/A; 
Example: Reducing Behavior Problems in the Elementary School 
Classroom: WWC held a webinar featuring a practice guide on this topic. 

Product: Registry of evaluation researchers; 
Description: An online database of researchers who conduct evaluations 
of the effectiveness of educational interventions to help schools, 
school districts, and educational program developers identify 
potential researchers; 
Number: N/A; 
Example: Individual researchers and various organizations. 

Product: Registry of randomized controlled trials; 
Description: An online database of completed and in-progress 
randomized controlled trials in education. This resource is designed 
to help schools, school districts, and educational program developers 
identify research regarding the effectiveness of educational 
interventions; 
Number: N/A; 
Example: A Randomized Trial of Two Promising Interventions for 
Students with Attention Problems: WWC included this randomized 
controlled trial in its registry. 

Source: GAO analysis of WWC information. 

Note: This table summarizes the WWC products and registries available 
as of May 18, 2010. The Clearinghouse previously published topic 
reports summarizing findings from all studies on all relevant 
interventions for a particular topic, such as beginning reading. These 
reports were replaced by dynamically generated summaries of evidence. 

[End of table] 

In addition to the Clearinghouse, Education provides other technical 
assistance and research-related resources to assist states, districts, 
and schools. Examples of research-related resources include the 
Regional Educational Laboratories (REL) and the Doing What Works (DWW) 
Web site (http://dww.ed.gov): 

Regional Educational Laboratories. IES's Regional Educational 
Laboratory Program is a network of 10 laboratories that conduct 
research and provide policymakers and practitioners with expert 
advice, training, and technical assistance on how to interpret 
findings from scientifically valid research.[Footnote 6] 

Doing What Works. Led by Education's Office of Planning, Evaluation 
and Policy Development, DWW is a Web-based resource intended to help 
teachers, schools, districts, states and technical assistance 
providers implement research-based instructional practice. 

Initial Years of the What Works Clearinghouse: 

In 2002, IES awarded a $27 million 5-year contract to the initial 
contractors to operate the Clearinghouse.[Footnote 7] The WWC 
contractors developed the Clearinghouse's research review standards 
with IES and reviewed research related to topic areas considered to be 
pressing issues in education.[Footnote 8] One of the goals of the 
Clearinghouse was to promote informed education decision making 
through a Web-based dissemination system that featured rigorous 
reviews of studies on the effectiveness of educational interventions. 

The WWC experienced a slow start due in part to the amount of work 
involved in developing a research review and reporting process that 
was valid, transparent, and replicable, according to the initial 
contractors. In developing the research review process, the 
contractors and IES addressed over 60 technical issues, such as 
determining what constitutes an acceptable level of participant loss 
(attrition) from a study and what methods should be in place to 
accommodate common education research techniques. In addition, initial 
plans for topic areas and reporting formats were modified. For 
example, IES decided to drop one planned topic area because IES 
officials determined it to be too broad.[Footnote 9] The WWC and IES 
also spent a substantial amount of time developing and refining a 
reporting format to communicate research results to a lay audience. As 
a result, the WWC began releasing reports in 2006. By September 2007, 
the WWC had released 89 intervention reports, six topic reports, and 
three practice guides. 

WWC Research Review Process for Intervention Reports: 

The WWC uses a three-step review process to assess the quality of 
studies and report on what the research indicates about the 
effectiveness of interventions. The WWC definition of interventions 
includes programs (such as whole school reform), products (such as a 
textbook or curriculum), practices (such as mixed-age grouping), or 
policies (such as class size reduction).[Footnote 10] The process 
begins with an initial screening of published and unpublished studies 
relevant to the intervention being reviewed. Studies are collected 
from electronic databases, journals, conference proceedings, and 
nominations solicited from the general public. The studies that pass 
initial screens are reviewed to determine whether they provide valid 
evidence of an intervention's effectiveness. Using these studies, the 
WWC then synthesizes the evidence about the effectiveness of an 
intervention and publishes a report describing its findings. The 
Clearinghouse categorizes interventions as either having positive 
effects, potentially positive effects, mixed effects, no discernible 
effects, potentially negative effects, or negative effects (see figure 
1). 

Figure 1: WWC Research Review Process for Interventions: 

[Refer to PDF for image: illustration] 

Step 1: Initial screening: 
Is the study: 
* Randomized or quasi-experimental? 
* Published within 20 years of the beginning of the topic area review? 
* Focused on a relevant intervention to the topic under review? 
And does the study: 
* Target students in the topic areas age or grade range and specified 
location? 
* Focus on populations relevant to the topic area (e.g., students with 
learning disabilities, English language learners)? 
* Report on at least one outcome relevant to the review? 
Screened out: 
Studies not meeting initial screening criteria. 

Step 2: Quality review: 
Studies meeting initial screening criteria: 
Two Ph.D.-level research analysts independently rate studies using a 
codebook that considers study design and execution, validity and 
reliability of outcome measures, and data analysis and reporting to 
evaluate the strength of the evidence in the study. 
Screened out: 
Studies not meeting evidence standards. 

Step 3: Synthesize evidence: 
Studies meeting evidence standards (with or without reservations): 
Evidence from all studies meeting standards is synthesized and 
summarized for use in reports; 
Intervention reports summarize evidence on the effects of a specific 
intervention. 

Source: GAO analysis of WWC guidelines. 

[End of figure] 

The WWC uses evidence standards to assess the strengths and weaknesses 
of a study's methodology, such as the type of design it uses, the 
quality of the study's data, and the appropriateness of the study's 
statistical procedures. Until recently, the WWC accepted two types of 
study designs--randomized experiments and quasi-experimental studies. 
[Footnote 11] Only randomized controlled trials (or randomized 
experiments) that WWC has determined to be well-designed and well- 
implemented are considered strong evidence and can receive the highest 
rating of "meets evidence standards without reservations." The WWC 
also considers evidence from quasi-experiments it has determined to be 
well-designed and well-implemented. The highest rating a study using 
quasi-experimental design can receive is "meets evidence standards 
with reservations." This rating category is intended to inform 
educators to interpret the study results with caution, as the results 
may reflect other factors, in addition to the impact of the 
intervention (see table 2). 

Table 2: Two Study Designs That Meet the WWC Standards with or without 
Reservations: 

Study design: Randomized control-group experiments; 
Description: Compare the outcomes of groups that were randomly 
assigned either to the intervention group or to a nonparticipating 
control group before the intervention. Such an assignment helps ensure 
that any differences in outcomes can be attributed to the intervention; 
Highest rating category WWC will assign if well-conducted, and why: 
Meets evidence standards: Considers randomized experiments as the 
design that is most likely to yield unbiased estimates of a program's 
impact on student outcomes. 

Study design: Comparison-group quasi-experiments; 
Description: Compare the outcomes of groups in which individuals are 
assigned to an intervention or control group in a way that minimizes 
observable differences between the groups that could affect outcomes. 
The researcher must demonstrate that the groups are equivalent on 
observable participant characteristics, such as age, grade level, 
prior academic achievement, or pretest results; 
Highest rating category WWC will assign if well-conducted, and why: 
Meets evidence standards with reservations: Even with equivalent 
observable characteristics, there may be differences in other 
participant characteristics related to the desired outcomes--for 
example, certain family or social structures that are unknown to the 
researcher. 

Source: GAO analysis of WWC information. 

[End of table] 

IES Oversight and Support of the WWC: 

The WWC is administered by IES through a contract with a private 
research organization. IES monitors implementation of the specific 
tasks detailed in the WWC contract by reviewing an annual work plan 
and monthly performance and expenditure reports submitted by the 
contractor. IES tracks implementation of the tasks, completion of 
performance goals, and adherence to the budget outlined in the 
contractor work plan.[Footnote 12] The contractor monitors the work of 
any subcontractors that it uses to perform services such as research 
reviews, technological support, and communications support. 

IES is also involved in the development and dissemination of WWC 
products. IES reviews and approves proposed topics for WWC products, 
product formats, and the research review procedures. It also 
coordinates a group of independent researchers to peer review WWC 
products and reviews and approves all WWC products prior to public 
release. IES required the contractor to develop a communications plan 
to inform WWC customers about features of the Web site. 

WWC Reviews Research in Accordance with Accepted Standards and Has 
Responded to Recommendations and Criticisms: 

WWC Follows Accepted Review Standards and Is Improving Its Review 
Process in Response to a Congressionally Mandated Expert Panel Report: 

We found that the WWC review process follows generally accepted 
practices for reviewing research. Specifically, GAO's November 2009 
report reviewing federally supported efforts to identify effective 
interventions found that the WWC determines whether a study provides 
credible evidence on effectiveness based on several dimensions, 
including the quality of the research design, how the study was 
implemented, and other technical considerations.[Footnote 13] Our 2009 
report also noted that WWC follows a common approach to conducting its 
reviews,[Footnote 14] and provides information to help educators 
understand the body of existing research on specific interventions. 
[Footnote 15] 

Additionally, a congressionally mandated panel of experts found in 
October 2008 that WWC's research review process was based on 
appropriate methods for judging the strength of the evidence regarding 
the effectiveness of interventions.[Footnote 16] For example, the 
panel agreed that the minimum qualifications a study must meet in 
order to be reviewed by the WWC are appropriate. The panel also found 
that WWC's reporting process is reasonable and that the WWC provides 
succinct and relevant evidence on the effectiveness of education 
interventions. While the panel concluded that the WWC's processes are 
generally appropriate, the panel made several recommendations to the 
WWC for continued improvement. The recommendations primarily related 
to establishing or clarifying procedures, reviewing statistical 
methods, and documenting the screening process. 

The WWC implemented or is considering implementing 14 of the panel's 
17 recommendations.[Footnote 17] The WWC implemented nine 
recommendations, in part by modifying some procedures and creating a 
procedures and standards handbook.[Footnote 18] For example, in 
response to the panel's recommendation that the WWC include a table of 
study dispositions (e.g., whether studies meet WWC evidence standards) 
at the front of intervention reports, the WWC is modifying the report 
template to include a summary table along with the existing listing of 
dispositions in the reference section. The WWC also addressed panel 
concerns about technical issues in its review process by making its 
treatment of study attrition--the rate at which subjects drop out of a 
study--more consistent across topic areas. The panel noted that the 
WWC's practice of determining acceptable attrition levels by topic 
area led to arbitrary inconsistencies across the topic areas. In 
response to the panel's recommendation that the WWC reconsider this 
practice, the WWC took steps to increase its consistency by developing 
attrition guidance that applies to all topic areas.[Footnote 19] (See 
appendix III for a table detailing the recommendations, WWC and IES's 
response, and the status of any changes made in response to 
recommendations.) 

In addition, the WWC is considering implementing five other panel 
recommendations. For example, the panel raised concerns that the WWC 
does not document some potential conflicts of interest for the studies 
it reviews. In response to this concern, the WWC is considering 
tracking and publishing whether studies of a program are funded or 
conducted by the program's developers.[Footnote 20] Further, in 
response to the panel's concern that the WWC's screening process may 
exclude some eligible studies, the WWC is undertaking an evaluation of 
the reliability of its screening process. According to IES officials, 
they will postpone decisions about the recommendations until the newly 
appointed Commissioner for the WWC is on board and actively involved 
in the decision making. 

WWC Also Responded to Criticism That It Produces Limited and 
Potentially Misleading Information: 

Some researchers claim that the WWC presents potentially misleading 
information by including brief experiments involving small numbers of 
students when evaluating interventions.[Footnote 21] As a result, 
according to critics, educators may accept the WWC's rating of the 
intervention's effectiveness, even though the evidence behind the 
rating is limited. One researcher suggested the WWC emphasize larger 
studies that span significant periods of time and set a minimum sample 
size requirement. According to WWC staff, such changes would exclude 
valuable research and prevent the WWC from providing educators with 
research-based information about some interventions.[Footnote 22] 
Instead of changing its treatment of sample size and study duration, 
the WWC began publishing information on the extent of the evidence 
supporting its findings in 2007. The WWC's "extent of evidence" rating 
alerts educators when the WWC effectiveness ratings are based on a 
small amount of evidence. As figure 2 shows, 76 percent of 
interventions with positive or potentially positive ratings of 
effectiveness are based on a small amount of evidence (see figure 2). 

Figure 3: Percentage of Interventions with Positive or Potentially 
Positive Ratings Categorized by the Amount of the Evidence Supporting 
Those Ratings: 

[Refer to PDF for image: pie-chart] 

Small (51 interventions): 76%; 
Medium to large (13): 19%; 
Not rated (3): 4%. 

Source: GAO analysis of WWC data. 

Note: The figure excludes seven interventions that were rated with 
different amounts of evidence as of April 27, 2010. Currently, the 
extent of evidence rating has two categories: small and medium to 
large. A rating of "medium to large" requires at least two studies and 
two schools across studies and a total sample size across studies of 
at least 350 students or 14 classrooms. Otherwise, the rating is 
"small." 

[End of figure] 

Further, researchers suggested that the WWC presents misleading 
information by rating interventions based on studies in which measures 
of student performance closely match the content taught to the 
intervention group, but not the control group.[Footnote 23] In such 
studies, higher test scores among the intervention group may not 
accurately represent the effectiveness of the intervention more 
generally. The researchers suggested that the WWC exclude such 
measures, or at least report on them separately. However, the WWC 
includes these measures because, according to IES officials, they 
answer questions about whether different interventions lead to 
different content knowledge. The WWC agrees that there is a concern 
regarding the reliability of outcome measures that are overly similar 
to the intervention, but maintains that WWC procedures attempt to 
exclude such measures. In addition, in response to researcher concerns 
that tests created by intervention developers may be biased,[Footnote 
24] the WWC added information to the intervention reports noting 
whether outcome measures are based on tests created by the developer. 

Some researchers and education professionals we interviewed suggested 
that the WWC produces limited information because its screening 
criteria are too restrictive--currently screening out about 90 percent 
of studies initially identified as potentially relevant (see figure 
3). Until recently, the WWC reviewed only two types of study designs-- 
randomized experiments and quasi-experimental studies--and according 
to critics, this limited the amount and type of information available 
to educators.[Footnote 25] For example, staff from one REL noted that 
educators may not be able to find reviews of the interventions they 
are using or considering because so few studies meet WWC standards. 
[Footnote 26] Staff from another REL told us that if educators cannot 
find relevant and useful information, they may be discouraged from 
using evidence-based practices. Staff from a third lab noted that the 
narrow focus prevents educators from learning from less rigorous but 
nonetheless useful research, such as case studies describing an 
intervention's costs and implementation requirements. 

Figure 3: Studies Reviewed That Meet WWC Evidence Standards: 

[Refer to PDF for image: illustration] 

Initially reviewed: 2,669 studies; 
Met evidence standards with or without reservations: 226 studies; 
Out of scope or did not meet evidence standards: 2,443 studies. 

Source: GAO analysis of WWC data. 

[End of figure] 

The WWC maintains that its screening criteria and study inclusion 
standards focus on studies that provide strong evidence of an 
intervention's effectiveness, and lowering these standards could 
undermine the validity of the findings reported by the WWC. Although 
the Clearinghouse screens out most studies, many of its reports have 
identified interventions with positive effects. Data from the 
contractor indicate that 58 percent of WWC's intervention reports 
identify positive or potentially positive effects of interventions. 
While the WWC plans to continue using its methodological standards for 
reviewing randomized and quasi-experimental studies, the Clearinghouse 
acknowledges that the emphasis on randomized experiments and quasi- 
experiments can exclude useful information on interventions in certain 
topic areas, such as special education, that do not lend themselves to 
these study designs. The WWC created new standards to include 
additional study designs.[Footnote 27] 

The WWC also introduced practice guides in 2007 in response to 
criticisms that its intervention reviews exclude too much research and 
consequently provide limited information to educators. Written by a 
panel of experts, practice guides include recommendations for 
educators on various topics, such as reducing high school drop-out 
rates and reducing behavioral problems in the classroom.[Footnote 28] 
Whereas WWC's intervention reviews are based entirely on studies that 
meet WWC evidence standards, practice guides also incorporate studies 
that do not have designs that are eligible for WWC review, or in some 
cases, are reviewed and do not meet WWC evidence standards, and 
include the views of experts. To develop recommendations, the practice 
guide panel reviews available literature about the particular topic 
and then meets several times to discuss the topic. Through consensus, 
the panel identifies effective practices based on the evidence. Once 
the practice guide is developed, it undergoes a quality assurance 
review by WWC and IES staff and external peer review. The following 
text box provides an example of practice guide recommendations and the 
level of evidence supporting them. 

Table: Example of Practice Guide Recommendations and Evidence 
Levels[A]: 

In 2009, the WWC published a practice guide to help educators assist 
students struggling with reading in the primary grades. The practice 
guide authors used an early detection and prevention framework known 
as Response to Intervention. The panel that authored the practice 
guide consisted of six researchers and one expert in implementation of 
the Response to Intervention model. Two WWC staff also assisted in the 
practice guide development. The panel's recommendations follow. 

Recommendation: 
Screen all students for potential reading problems twice per year and 
monitor those with higher risk; 
Basis for recommendation: Numerous studies with designs that did not 
meet WWC evidence standards or that did not use samples that 
adequately resembled the population of interest; 
Level of evidence[B]: Moderate. 

Recommendation: 
Provide time for differentiated reading instruction for all students 
based on assessments of students' current reading level; 
Basis for recommendation: One descriptive study and expert opinion; 
Level of evidence[B]: Low. 

Recommendation: 
Provide intensive, systematic instruction on foundational reading 
skills in small groups to students who score below the benchmark score; 
Basis for recommendation: 11 studies that met WWC evidence standards; 
Level of evidence[B]: Strong. 

Recommendation: 
Monitor the progress of these students at least once a month; 
Basis for recommendation: 3 studies that met WWC evidence standards, 
but did not evaluate the effectiveness of monitoring so no conclusive 
inferences could be made, and expert opinion; 
Level of evidence[B]: Low. 

Recommendation: 
Provide intensive interaction on a daily basis to students who show 
minimal progress after reasonable time in small group instruction; 
Basis for recommendation: 5 studies that met WWC evidence standards 
but did not report statistically significant impacts on reading 
outcomes; Level of evidence[B]: Low. 

Source: GAO review of a WWC practice guide. 

[A] A strong rating indicates that studies supporting the 
recommendation generally meet WWC standards. A moderate rating 
indicates that studies supporting the recommendation generally meet 
WWC standards with reservations. A low rating indicates the 
recommendation is based on expert opinion, derived from theory or 
experience, and supported with evidence that does not rise to the 
moderate or strong levels. 

[B] Our analysis of practice guide recommendations found that almost 
half of the 67 recommendations made in the 12 practice guides released 
as of May 2010 were based on a low level of evidence. 

[End of table] 

WWC's Output and Costs Increased; However, IES Has Not Developed 
Adequate Performance Measures Related to Cost or Product Usefulness: 

WWC Increased Output and Introduced New Products: 

WWC's report output increased under the current contract, and its 
scope expanded to include new products and processes to support 
production. Under the current contract, the WWC increased its total 
number of publications from the first contract year to the second 
contract year and generally kept pace with its increased scope, as 
specified in the Clearinghouse's annual plans.[Footnote 29] For 
example, the current contract calls for the WWC to increase the number 
of topic areas and intervention reports. Under the current contract, 
the WWC added three new topic areas and released 60 intervention 
reports, including 5 in the new topic areas as of June 2010.[Footnote 
30] In addition, the WWC produces practice guides and quick reviews 
and increased its production of both of these products between the 
first and second year of the current contract. Figure 4 shows the 
production of all three WWC products as of June 30, 2010, the end of 
the third contract year. 

Figure 4: Publication Quantities, by Contract Year (CY) for Current 
WWC Contract: 

[Refer to PDF for image: 3 horizontal bar charts] 

Intervention reports: 
Contract year 1: 4 publications; 
Contract year 2: 37 publications; 
Contract year 3: 19 publications. 

Practice guides: 
Contract year 1: 0 publications; 
Contract year 2: 3 publications; 
Contract year 3: 3 publications. 

Quick reviews: 
Contract year 1: 5 publications; 
Contract year 2: 21 publications; 
Contract year 3: 16 publications. 

Source: GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010. 

[End of figure] 

The WWC's scope of work increased under the current contract with the 
addition of new products and work processes, as well as 
responsibilities related to the American Recovery and Reinvestment Act 
of 2009 (Recovery Act), which provided additional innovation and 
improvement funding.[Footnote 31] The WWC is developing three new 
types of publications and conducts an annual review of Education-
sponsored studies for IES's internal use. Specifically, the WWC is 
developing research briefs, research perspectives, and practice 
briefs, which will focus on Education policy priorities. Like practice 
guides, the new publications will incorporate expert opinion and a 
broad range of research. Table 3 provides more information on these 
new initiatives.[Footnote 32] 

Table 3: New WWC Publications and Reports: 

Product: Research briefs; 
Description: Short summaries of what research indicates about the 
effectiveness and implementation challenges of policies, practices, or 
issues in education; 
Status: In process and template has been approved. First publications 
projected for release in 2010. 

Product: Research perspectives; 
Description: Researchers' perspectives on what research has found will 
work in addressing pressing educational issues. Topics will initially 
focus on issues relevant to the Recovery Act; 
Status: In process and template has been approved. First publications 
projected for release in 2010. 

Product: Practice briefs; 
Description: Provide explicit information on how to implement one 
practice from a WWC practice guide, and provides educators with 
research-based, how-to steps and strategies for overcoming roadblocks, 
and tools for educators; 
Status: Template has been drafted but further work on this product is 
on hold pending direction from IES. 

Product: Reviews of IES-sponsored studies (annual); 
Description: IES uses this report to evaluate the research it funds. 
WWC reviews this research using WWC standards and reports on whether 
the research studies identify effective or promising practices; 
Status: First produced in 2008, with plans for annual reporting to IES. 

Source: GAO review of WWC contracts and annual plans. 

[End of table] 

IES's Reviews Have Delayed the Release of Some Reports: 

While the WWC contractor increased its report production, IES's review 
process did not keep pace with output. IES is responsible for 
administering independent peer reviews of all products and conducting 
final reviews and approvals before products are released, and has 
internal time frame estimates used in scheduling and completing such 
reviews. For example, according to IES planning documents, IES 
estimates 15 business days for the completion of peer reviews for 
intervention reports and 6 business days for WWC quick reviews. 
However, throughout 2009, IES took increasingly more time to schedule 
and coordinate the completion of peer reviews for some intervention 
reports and quick reviews. As a result, the release of 20 reports--11 
intervention reports and nine quick reviews--was delayed by more than 
6 months. For example, in the first quarter of the current contract 
year (third contract year, 2009 to 2010), IES took an average of over 
50 business days to have intervention reports and quick reviews peer 
reviewed, compared to an average of 7 business days during the first 
quarter of second contract year (see figure 5). 

Figure 5: Average Time for IES Peer Reviews of Released Intervention 
Reports and Quick Reviews, by Contract Year (CY) and Quarter (Q) for 
Current WWC Contract: 

[Refer to PDF for image: 2 vertical bar charts] 

IES peer review goal: 	
Intervention reports: 15; 
Quick reviews: 6. 

Quarter released: CY2-Q1; 
Intervention reports: 7; 
Quick reviews: 7. 

Quarter released: CY2-Q2; 
Intervention reports: 10; 
Quick reviews: 6. 

Quarter released: CY2-Q3; 
Intervention reports: 13; 
Quick reviews: 33. 

Quarter released: CY2-Q4; 
Intervention reports: 16; 
Quick reviews: 45. 

Quarter released: CY3-Q1; 
Intervention reports: 54; 
Quick reviews: 56. 

Quarter released: CY3-Q2; 
Intervention reports: 55; 
Quick reviews: NA. 

Quarter released: CY3-Q3; 
Intervention reports: 54; 
Quick reviews: 134. 

Quarter released: CY3-Q4; 
Intervention reports: 111; 
Quick reviews: 99. 

Source: GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010. In CY3-Q2, no 
quick reviews were released. 

[End of figure] 

These delays in the IES-administered peer review process resulted in 
significant backlogs of intervention reports and quick reviews 
awaiting release. For example, as shown in figure 6, reports that 
entered the peer review process in the first quarter of the second 
contract year (CY2-Q1) were completed within that quarter. However, 
the majority of reports entering review the first quarter of the third 
contract year (CY3-Q1) remained in process for subsequent quarters. 
While the backlog persisted through the third quarter of the third 
contract year (CY3-Q3), the number of reports that completed peer 
review in the third and fourth quarters increased from prior quarters. 
Figure 6 shows that 11 intervention reports completed peer review in 
CY3-Q3 and an additional 27 completed peer review in CY3-Q4, compared 
with 4, 5, and 8 intervention reports in the prior three quarters. 

Figure 6: IES Peer Review Backlog for Intervention Reports and Quick 
Reviews, by Contract Year (CY) and Quarter (Q) for Current WWC 
Contract: 

[Refer to PDF for image: horizontal bar graph] 

CY2-Q1: 
Intervention reports entering peer review: 11; 
Intervention reports completing peer review: 10; 
Intervention reports remaining in peer review: 1; 
Quick reviews entering peer review: 3; 
Quick reviews completing peer review: 3; 
Quick reviews remaining in peer review: 0. 

CY2-Q2: 
Intervention reports entering peer review: 10; 
Intervention reports completing peer review: 10; 
Intervention reports remaining in peer review: 1; 
Quick reviews entering peer review: 7; 
Quick reviews completing peer review: 5; 
Quick reviews remaining in peer review: 2. 

CY2-Q3: 
Intervention reports entering peer review: 19; 
Intervention reports completing peer review: 14; 
Intervention reports remaining in peer review: 6; 
Quick reviews entering peer review: 7; 
Quick reviews completing peer review: 4; 
Quick reviews remaining in peer review: 5. 

CY2-Q4: 
Intervention reports entering peer review: 11; 
Intervention reports completing peer review: 8; 
Intervention reports remaining in peer review: 9; 
Quick reviews entering peer review: 11; 
Quick reviews completing peer review: 7; 
Quick reviews remaining in peer review: 9. 

CY3-Q1: 
Intervention reports entering peer review: 8; 
Intervention reports completing peer review: 5; 
Intervention reports remaining in peer review: 12; 
Quick reviews entering peer review: 3; 
Quick reviews completing peer review: 1; 
Quick reviews remaining in peer review: 11. 

CY3-Q2: 
Intervention reports entering peer review: 6; 
Intervention reports completing peer review: 4; 
Intervention reports remaining in peer review: 14; 
Quick reviews entering peer review: 4; 
Quick reviews completing peer review: 3; 
Quick reviews remaining in peer review: 12. 

CY3-Q3: 
Intervention reports entering peer review: 16; 
Intervention reports completing peer review: 11; 
Intervention reports remaining in peer review: 19; 
Quick reviews entering peer review: 3; 
Quick reviews completing peer review: 10; 
Quick reviews remaining in peer review: 5. 

CY3-Q4: 
Intervention reports entering peer review: 10; 
Intervention reports completing peer review: 27; 
Intervention reports remaining in peer review: 2; 
Quick reviews entering peer review: 7; 
Quick reviews completing peer review: 11; 
Quick reviews remaining in peer review: 1. 

Source: GAO analysis of WWC data. 

Notes: CY is July 1 to June 30. CY1 covers this period for 2007 to 
2008, CY2 for 2008 to 2009, and CY3 for 2009 to 2010. At the end of 
each quarter, any report remaining in peer review would carry over to 
the next quarter. For example, for intervention reports in CY2-Q3, 19 
new reports entered peer review, joining the 1 report that remained 
from the previous quarter. Fourteen of these 20 reports completed peer 
review, and 6 remained. 

[End of figure] 

IES attributed these delays to several factors and recently took steps 
to eliminate the backlog. IES officials told us that delays were, in 
part, attributable to difficulty in identifying and scheduling 
independent peer reviewers, vacancies in WWC-related positions at IES, 
and an increasing amount of research that met WWC standards.[Footnote 
33] For example, IES officials told us identifying and scheduling a 
sufficient number of qualified, independent peer reviewers had become 
increasingly difficult because several former peer reviewers were now 
associated in some way with the WWC and therefore were no longer 
independent. To reduce the delays and eliminate the backlog, IES 
recently implemented a new database to help staff track and manage the 
work of peer reviewers and other WWC-related tasks. IES officials also 
told us that they began identifying additional potential peer 
reviewers using the WWC online registry of researchers. In addition, 
IES increased a staff member's responsibilities related to scheduling 
and coordinating peer reviews. These efforts reduced the amount of 
time reports remain in the IES peer review process and eliminated the 
backlog as of June 2010.[Footnote 34] 

In addition to delays in the peer review process, WWC contractors told 
us that many of their daily decisions need IES approval, and slow 
responses delayed contractor processes. For example, the contractor 
needs IES approval on the format and content of the products in 
development, hindering further work when responses are delayed. IES 
officials acknowledged that some delays in the approval process 
occurred during contract year three and told us that this was largely 
due to staff vacancies that they anticipate filling. 

The Cost of the Current WWC Contract Has Increased from the Previous 
One: 

WWC's contracted costs have doubled from about $5.3 million per year 
under the previous 5-year contract to the current level of about $10.7 
million per year.[Footnote 35] The increase in contracted costs 
reflects the expanded scope--more publications and new products and 
processes--of the second contract compared to the first. IES's 
contract for the WWC includes a variety of tasks that the contractor 
is responsible for, including tasks related to report production and 
product development. Table 4 provides a description of six broad task 
categories and how they changed between contracts. 

Table 4: Task Category Definitions and Changes between Contracts: 

Task category: WWC products; 
Task category includes expenditures related to: Conducting research 
reviews and developing and publishing WWC products; 
Changes from first contract to current contract: New product types, 
expanded practice guide review process; 
Stopped producing topic reports (2008). 

Task category: Strategic planning and coordination with IES; 
Task category includes expenditures related to: Preparing annual 
plans, managing reporting requirements, and communications and 
workflow with IES; 
Changes from first contract to current contract: New contractor 
database increased process documentation and reporting capabilities to 
IES. 

Task category: Communications, collaboration, and dissemination of WWC 
products; 
Task category includes expenditures related to: Maintaining WWC Help 
Desk; Promoting WWC through various means; Developing/implementing a 
communications/dissemination plan; 
Changes from first contract to current contract: WWC staff attend 
conferences and coordinate some dissemination efforts with other IES 
departments in the current contract. 

Task category: WWC development, process revisions, and maintenance; 
Task category includes expenditures related to: Revising and 
developing review processes and policies; Administrating and 
supporting technical staff training, technical advisory group, online 
registries, and conflict of interest procedures; 
Changes from first contract to current contract: Enhanced review 
processes and standards, added new research designs; Developed policy 
and procedures handbooks, new products, and staff training. 

Task category: Web site and technical maintenance; 
Task category includes expenditures related to: Coordinating content, 
maintaining databases/search functions, and processing federal data 
collection forms; 
Changes from first contract to current contract: New online searchable 
system and database. 

Task category: Award fees; 
Task category includes expenditures related to: Fixed and performance-
based contractor award fees based on a percentage of the overall 
contract total; 
Changes from first contract to current contract: No change. 

Source: GAO review of WWC contracts, annual plans, and budget 
documents. 

[End of table] 

Our analyses of costs associated with these six broad task categories 
shows that the proportion of funds dedicated to producing WWC reports 
was about 60 percent under both contracts (see figure 7).[Footnote 36] 

Figure 7: WWC Costs, by Task Categories and Contracts: 

[Refer to PDF for image: 2 pie-charts] 

First 5-year contract ($26.5 million): 

WWC products: 60%: 
- Intervention and topic reports: 57%; 
- Practice guides: 3%; 
Other costs: 41%: 
- A: 6%; 
- B: 12%; 
- C: 5%; 
- D: 9%; 
- E: 9%. 

Second 5-year contract ($53.3 million): 
WWC products: 60%: 
- Intervention and topic reports: 29%; 
- Practice guides: 21%; 
Other costs: 41%: 
- A: 7%; 
- B: 6%; 
- C: 11%; 
- D: 7%; 
- E: 9%. 

Breakdown of other costs: 

A: Strategic planning and coordination with Education. 
B: Communications, collaboration, and dissemination. 
C: WWC development, process revisions, and staff training. 
D: Web site and technical maintenance. 
E: Award fees. 

Source: GAO analysis of WWC budget data. 

Notes: Figures reflect actual expenditures for the first 5-year 
contract and updated budgeted expenditures for the current 5-year 
contract, which began on July 1, 2007. Percentages do not add to 100% 
due to rounding. Total WWC expenditures for the first 5-year contract 
were about $26,527,760, but cost category percentages do not reflect 
$1,222,714 billed by a co-contractor but not itemized by task. 
Category proportions for the first contract are estimates because IES 
could not provide documentation that included final adjusted 
expenditures by tasks. The WWC budget is $53,315,166 for the current 5-
year contract, of which $23,643,891 had been spent as of October 31, 
2009. Cost category proportions for the current contract do not 
include $104,559 related to transition from the first contract to the 
second contract. 

[End of figure] 

The proportion of funds dedicated to some tasks changed from the first 
contract to the second. For example, costs for tasks related to 
process development and revisions doubled from 5 percent to 11 
percent, supporting various activities such as expanding the practice 
guide review process and revising the Clearinghouse's procedures and 
standards handbook. According to IES officials, the current WWC 
contractor developed and implemented new or enhanced processes that 
affect all publications and deliverables. For example, the current 
contractor developed a standardized system for conducting and 
recording the WWC's searches of research studies.[Footnote 37] 

Most WWC cost increases supported additional output and expansions in 
product scope. While under both contracts more resources were devoted 
to intervention reports than any other product, the proportion devoted 
to practice guides increased significantly, currently comprising about 
21 percent of total budgeted costs. IES noted that practice guides 
were only added during the last year of the prior contract, but are 
now a primary product. Other new WWC products make up a relatively 
small proportion of budgeted costs in the current contract, 
representing about 9 percent of the total contract budget combined. 

IES Has Not Developed Performance Measures Related to Production Costs 
or Product Usefulness: 

IES established performance goals, which the WWC met or exceeded; 
however, these goals do not address production costs or the usefulness 
of WWC products. IES established WWC-related performance goals in its 
annual organizational assessment, but Education discontinued the use 
of these performance measures for fiscal year 2010.[Footnote 38] In 
addition, IES established performance goals for its WWC contractor in 
the contractor award fee plan, which IES uses to determine the amount 
of performance-based funds awarded to the contractor.[Footnote 39] 

IES measured WWC program performance from fiscal year 2003 to fiscal 
year 2009, as part of Education's Organizational Assessment--its 
departmentwide performance management system. The WWC-related 
performance goals included in Education's Organizational Assessment 
focused on WWC Web site visits and the quantity of publications, both 
of which were areas of concern as the WWC was getting established. 
[Footnote 40] Specifically, these performance goals included increased 
WWC Web site visits, sustained productivity in the release of 
intervention reports and quick reviews, and increased practice guide 
production. The WWC met or exceeded these performance goals annually; 
however, according to IES officials, these performance goals will not 
be included in Education's fiscal year 2010 Organizational Assessment, 
in part because IES is now satisfied with WWC activity in these areas. 
[Footnote 41] 

IES has not developed performance measures related to the cost of 
specific WWC products.[Footnote 42] IES officials noted that the costs 
per WWC publication vary greatly depending on the amount of available 
research relevant to the specifications of a product. For example, 
intervention reports based on a large number of studies meeting WWC 
standards take longer and cost more to produce than do reports for 
which few studies qualify for review. IES has tasked the current WWC 
contractor to develop ways to streamline production processes and to 
conduct a cost study, the results of which would improve budget 
estimates and strengthen IES's monitoring of the contract. While the 
contractor has begun this work, IES officials told us that they do not 
know when cost-related performance measures, such as acceptable cost 
ranges for each type of product, will be established. 

WWC does not currently have a performance measure related to product 
usefulness. While Web site visits were tracked as a measure of WWC 
utilization in IES's Organizational Assessment through fiscal year 
2009, this metric did not assess the degree to which WWC products were 
reaching their target audience and did not provided any information on 
the extent to which educators find WWC products to be useful. IES's 
2010 budget justification calls for a representative survey of WWC use 
among education decision makers to be conducted by 2013. However, IES 
officials told us that they were unsure whether the survey would take 
place, and IES does not currently have a plan in place to implement 
this survey.[Footnote 43] 

Education Has Three Primary Ways to Disseminate Information about WWC 
Products, but Awareness and Use Vary among Target Audiences: 

Education Has Various Ways to Disseminate Information about WWC 
Products, but Awareness of the Clearinghouse Is Generally Limited: 

Education uses the WWC contractor, RELs, and DWW to disseminate 
information about the Clearinghouse to its target audience, which 
includes state and school district officials, as well as teachers and 
principals. In accordance with its contract, the WWC contractor 
disseminates information about its products electronically and through 
various events, such as formal presentations at conferences. The 
Clearinghouse's electronic dissemination methods include an e-mail 
listserv, Web-based seminars (webinars), and newsletters. For example, 
the WWC sends out notices to its e-mail listserv, alerting subscribers 
of the availability of new products, including intervention reports, 
practice guides, and quick reviews.[Footnote 44] WWC staff told us 
that the webinars cover the same topics as their reports and are a 
relatively cost-effective way to disseminate information about 
products and methodology. In addition, WWC staff disseminate 
information about WWC products at education conferences, such as 
teacher, principal, and researcher conferences. At these conferences, 
WWC staff may conduct formal presentations, have an exhibit featuring 
their products, or both. At conference exhibits, Clearinghouse staff 
answer questions about their products and provide literature to 
conference attendees. From July 2009 through June 2010, WWC staff were 
scheduled to present or have an exhibit at 14 conferences. WWC staff 
also told us that they work with other groups, such as education, 
research, and legislative organizations, in order to further 
disseminate information about WWC products to their members. 

In addition, Education disseminates information about WWC products 
through IES's 10 RELs, which hold events that may feature information 
based on practice guides and refer educators to Clearinghouse 
products. Officials at all 10 RELs told us that they spent time 
disseminating information about WWC, in part by holding events that 
bridge research and practice. According to REL officials, these bridge 
events are attended primarily by school-, district-, and state-level 
education professionals and provide an opportunity for educators to 
discuss ways to implement research-based practices. Officials at all 
10 RELs told us that bridge events focused on practice guides to some 
extent, and 7 indicated that WWC practice guides were the primary 
focus of these events. According to REL officials and WWC staff, these 
events sometimes included a WWC staff member to discuss methodology 
and panelists who helped develop the practice guides. RELs also 
disseminate research from WWC when responding to educator questions or 
concerns.[Footnote 45] Officials from 7 of the 10 RELs told us their 
respective RELs generally use relevant WWC products (practice guides 
and others) when searching for research-based information to address 
educator questions. 

In addition, Education's Office of Planning, Evaluation, and Policy 
Development disseminates information about WWC practice guides on its 
DWW Web site, which provides an online library of resources designed 
to help educators implement research-based instructional practice. 
This Web site uses different formats to present content based 
primarily on WWC Practice Guides and provides examples of possible 
ways educators might apply WWC research findings. For instance, to 
help educators implement the recommendations from the practice guide 
on dropout prevention, the DWW Web site features slideshows with 
examples of supportive academic environments and interviews with 
educators and experts on dropout prevention. In addition, the Web site 
includes sample materials, such as lesson plan templates, that provide 
an example of how to implement recommendations. The DWW also includes 
information on the research behind the recommendations and a link to 
the WWC Web site and the individual practice guides. According to IES 
officials, a recent analysis of the DWW Web site traffic showed that 
49 state Web sites have links to the DWW Web site, which helps 
disseminate WWC products further to the education community. 

We found that 33 of the 38 states[Footnote 46] that responded to our 
survey reported that they had heard of the WWC. Based on our survey 
results, we estimate that 42 percent of school districts have heard of 
the WWC and that the percentage is greater for school districts that 
rely to a very large extent on external sources for information on 
research-based practices.[Footnote 47] 

School districts identified several sources of information about the 
Clearinghouse, including conferences and Education (see figure 8). 

Figure 8: Sources from Which District Officials Heard of the WWC: 

[Refer to PDF for image: vertical bar graph] 

Potential sources of WWC information: Education; 
Not a source of WWC information: 23; 
Don't know: 5; 
Source of WWC information: 72. 

Potential sources of WWC information: Conferences; 	
Not a source of WWC information: 22; 
Don't know: 5; 
Source of WWC information: 73. 

Potential sources of WWC information: Peers; 
Not a source of WWC information: 28; 
Don't know: 6; 
Source of WWC information: 66. 

Potential sources of WWC information: RELs; 
Not a source of WWC information: 45; 
Don't know: 8; 
Source of WWC information: 47. 

Potential sources of WWC information: Other; 
Not a source of WWC information: 24; 
Don't know: 40; 
Source of WWC information: 37. 

Source: GAO analysis of school district survey responses. 

Notes: Estimates shown are based upon a probability survey. See 
appendix I for associated confidence intervals. The responses for 
other sources included state departments of education, Internet 
searches, and journals. 

[End of figure] 

While the majority of states have accessed the WWC Web site, we 
estimate that only 34 percent of school districts have done so. 
Specifically, among the states that responded to our survey, 33 of 38 
states[Footnote 48] reported that they had accessed the WWC Web site 
at least once. In addition, 19 of states reported visiting the Web 
site at least seven times per year.[Footnote 49] In contrast, an 
estimated 34 percent of school districts accessed the WWC Web site at 
least once.[Footnote 50] Further, we estimate that only 11 percent of 
school districts visited the Web site at least seven times per year. 
[Footnote 51] States and school districts that visited the WWC Web 
site less than seven times per year most often cited time constraints 
as the primary reason for their infrequent use.[Footnote 52] 

In addition to the WWC, states and school districts use a variety of 
other sources of information to identify effective education 
practices. Most states and school districts use several broad sources 
of information, such as academic journals, education periodicals, and 
associations of educators. For example, 37 states reported using 
academic journals to identify such practices, and we estimate that 
about 97 percent of school districts used academic journals.[Footnote 
53] Overall, more school districts and states that responded to our 
survey used the WWC than used other research synthesis organizations. 
[Footnote 54] 

While the WWC also includes teachers and principals in its target 
audience, we found that relatively few of the teachers and principals 
we contacted at education conferences had heard of the WWC. While not 
a generalizeable sample, we found that out of a total of 391 teachers 
who completed our questionnaire at four education conferences, only 18 
had accessed the WWC Web site.[Footnote 55] In addition, 341 teachers 
who had not accessed the WWC Web site told us they had not heard of 
the Web site. Similarly, among the 208 principals and other school 
administrators who completed the questionnaire, only 32 had accessed 
the WWC Web site. Further, 135 principals and other school 
administrators told us they had not heard of the WWC. 

States and School Districts Generally Used the Clearinghouse to a 
Small or Moderate Extent to Inform Decisions and Used Specific WWC 
Products to Varying Extents: 

Based on our survey, most states and school districts that reported 
accessing the WWC Web site used it to inform decisions on effective 
education practices--a stated purpose of the WWC--to a small or 
moderate extent. Specifically, 25 of the 33 states that use the 
Clearinghouse indicated that they use it to a small or moderate extent 
to inform their decisions, while 6 reported using it to a large or 
very large extent.[Footnote 56] We estimate that 72 percent of school 
districts that have accessed the Clearinghouse use the WWC to inform 
education decisions to a small or moderate extent, while only 18 
percent use it to a large or very large extent.[Footnote 57] 

States that used the WWC to inform decisions reported that they used 
the Clearinghouse for various purposes, including informing 
professional development and curriculum decisions. For example, 25 
states reported using the Clearinghouse to inform professional 
development programs for teachers, and 22 reported using it to inform 
curriculum decisions. Fewer states used the Clearinghouse to advise 
districts that were not making adequate yearly progress (AYP) in 
meeting academic targets or to develop improvement plans for such 
districts. (Figure 9 provides a breakdown of the extent to which these 
states use the Clearinghouse for various purposes.) 

Figure 9: Extent to Which States Use WWC for Various Purposes: 

[Refer to PDF for image: vertical bar graph] 

Number of states: 

Inform curriculum decisions: 
Not at all: 4; 
Small extent: 2; 
Moderate extent: 12; 
Large or very large extent: 8. 

Inform professional development of teachers: 
Not at all: 2; 
Small extent: 4; 
Moderate extent: 12; 
Large or very large extent: 9. 

Develop school improvement plan: 
Not at all: 4; 
Small extent: 6; 
Moderate extent: 9; 
Large or very large extent: 4. 

Advise schools not making AVP on potential interventions: 
Not at all: 4; 
Small extent: 4; 
Moderate extent: 11; 
Large or very large extent: 5. 

Source: GAO analysis of state responses. 

Note: "I Don't Know" was also a response option, and is not displayed 
in the figure. 

[End of figure] 

In addition, we estimate that among school districts that use the WWC 
to inform decisions on effective education practices, about 90 percent 
used it to inform curriculum decisions at least to a small extent, 
similar to the percentage that used the WWC to inform professional 
development decisions. However, fewer school districts used it to 
advise schools that did not meet academic goals or to develop school- 
level plans to help such schools improve.[Footnote 58] Figure 10 
provides a breakdown of the extent to which these school districts use 
the Clearinghouse for various purposes. 

Figure 10: Extent to Which School Districts That Have Used the 
Clearinghouse Used It for Various Purposes: 

[Refer to PDF for image: vertical bar graph] 

Estimated percentage of school districts: 

Inform curriculum decisions: 
Not at all: 3%; 
Small extent: 20%; 
Moderate extent: 37%; 
Large or very large extent: 37%. 

Inform professional development of teachers: 
Not at all: 6%; 
Small extent: 19%; 
Moderate extent: 38%; 
Large or very large extent: 32%. 

Develop school improvement plan: 
Not at all: 13%; 
Small extent: 19%; 
Moderate extent: 35%; 
Large or very large extent: 27%. 

Advise schools not making AVP on potential interventions: 	
Not at all: 29%; 
Small extent: 11%; 
Moderate extent: 28%; 
Large or very large extent: 23%. 

Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See 
appendix I for associated confidence intervals. "I Don't Know" was 
also a response option, and is not displayed in the figure. 

[End of figure] 

States reported using specific WWC products--intervention reports and 
practice guides--more than quick reviews. Specifically, of the states 
that had used the Clearinghouse, 21 reported that they used 
intervention reports and 20 reported using practice guides, while only 
12 reported using quick reviews. States used intervention reports and 
practice guides to a similar extent to inform education decisions. For 
example, for each product, six states reported using them to large or 
very large extent to inform such decisions (see figure 11). 

Figure 11: Extent to Which States Use Specific WWC Products: 

[Refer to PDF for image: vertical bar graph] 

Number of states: 

Intervention reports: 
Not at all: 4; 
Small extent: 3; 
Moderate extent: 12; 
Large or very large extent: 6. 

Practice guides: 
Not at all: 3; 
Small extent: 3; 
Moderate extent: 11; 
Large or very large extent: 6. 

Quick reviews: 
Not at all: 10; 
Small extent: 3; 
Moderate extent: 7; 
Large or very large extent: 2. 

Source: GAO analysis of state survey responses. 

[End of figure] 

However, the relative use of specific WWC products was different among 
school districts. We estimate that among school districts that use the 
Clearinghouse to inform decisions on effective education practices, 
more school districts use intervention reports relative to practice 
guides or quick reviews. Specifically, we estimate that 74 percent of 
those school districts that use the WWC have used its intervention 
reports to inform education decisions,[Footnote 59] while practice 
guides and quick reviews were each used by about half of such 
districts.[Footnote 60] Based on our survey, an estimated 21 percent 
of school districts that use the WWC have used intervention reports to 
a large or very large extent,[Footnote 61] while about 10 percent use 
practice guides to a large or very large extent (see figure 12). 
[Footnote 62] 

[Refer to PDF for image] 

[End of figure] 

Figure 12: Extent of Specific Product Use among Districts That Use the 
Clearinghouse: 

[Refer to PDF for image: vertical bar graph] 

Intervention reports: 
Not at all: 21; 
Small extent: 24; 
Moderate extent: 29; 
Large or very large extent: 21. 

Practice guides: 
Not at all: 42; 
Small extent: 17; 
Moderate extent: 28; 
Large or very large extent: 10. 

Quick reviews: 
Not at all: 40; 
Small extent: 25; 
Moderate extent: 17; 
Large or very large extent: 13. 

Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See 
appendix I for associated confidence intervals. 

[End of figure] 

States and School Districts Would Likely Increase Their Use of the 
Clearinghouse If the WWC Made Certain Changes: 

Many states and school districts that had accessed the Clearinghouse 
reported that they would likely increase their use of the WWC if the 
Clearinghouse provided a broader array of information. For example, 
many states and school districts would be likely to increase their use 
of the Clearinghouse if it reviewed more studies, covered additional 
topics, or provided more relevant or timely reports. For example, 21 
of the 33 states that had used the Clearinghouse reported that they 
would be somewhat or very likely to use the Clearinghouse more often 
if it had reviews that were more timely (see figure 13).[Footnote 63] 

Figure 13: Number of States That Reported They Would Likely Increase 
Their Use of WWC Given Certain Changes: 

[Refer to PDF for image: horizontal bar graph] 

Potential changes: 

A greater number of intervention reports showing positive effects; 
Likely or very likely: 26. 

Reviews of efficacy of programs being used or considered in my state; 
Likely or very likely: 23. 

Additional practices with positives reviews; 
Likely or very likely: 22. 

Reviews that are more timely; 
Likely or very likely: 21. 

Additional topic areas; 
Likely or very likely: 19. 

Additional studies reviewed; 
Likely or very likely: 19. 

Additional practice guides; 
Likely or very likely: 18. 

Additional information on interventions based on studies that may not 
meet WWC standards; 
Likely or very likely: 15. 

A broader definition of what studies meet WWC standards; 
Likely or very likely: 15. 

An easier web site to navigate; 
Likely or very likely: 14. 

Source: GAO analysis of state survey responses. 

In addition, based on our survey, we estimate that about two thirds of 
school districts that had accessed the Clearinghouse would likely 
increase their use if it included reviews of programs or interventions 
being used or considered in their school district.[Footnote 64] An 
estimated 50 percent of school districts would likely increase their 
use of the Clearinghouse if it had reviews that were more timely (see 
figure 14).[Footnote 65] 

Figure 14: Estimated Percent of School Districts That Have Accessed 
the WWC That Would Likely Increase Their Use of the WWC Given Various 
Changes: 

[Refer to PDF for image: horizontal bar graph] 

Potential changes: 

Reviews of efficacy of programs being used or considered in my 
district: 68%. 

A greater number of intervention reports showing positive effects: 
Likely or very likely: 67%. 

Additional practices with positives reviews: 
Likely or very likely: 60%. 

Additional topic areas: 
Likely or very likely: 57%. 

Additional information on interventions based on studies that may not 
meet WWC standards: 
Likely or very likely: 56%. 

Additional practice guides: 
Likely or very likely: 56%. 

Additional studies reviewed: 
Likely or very likely: 55%. 

Reviews that are more timely: 
Likely or very likely: 50%. 

A broader definition of what studies meet WWC standards: 
Likely or very likely: 47%. 

An easier web site to navigate: 
Likely or very likely: 47%. 

Source: GAO analysis of school district survey responses. 

Note: Estimates shown are based upon a probability survey. See 
appendix I for associated confidence intervals. 

[End of figure] 

Conclusions: 

In 2007, Education substantially increased its financial investment in 
the WWC, and the Clearinghouse is significantly expanding its scope in 
an effort to better serve its target audience. Some of the new 
products aim to be more responsive to educators and education decision 
makers by providing timely information about evidence-based practices 
relevant to pressing needs. Such information could help states and 
districts identify strategies as they implement educational reform 
efforts--such as reforming low-performing schools or improving 
professional development--under ESEA and the Recovery Act. For 
example, WWC research perspectives, still in development, are intended 
to help education decision makers as they address challenges related 
to spending Recovery Act funds. However, the development of these 
products and the release of other products were delayed, in part, by a 
substantial backlog in IES's review and approval processes. These 
delays hindered the timely release of several publications, and some 
products were released months after they were completed by the 
contractor. While IES recently eliminated the backlog, educators need 
to be able to rely on the Clearinghouse for timely and relevant 
information. According to our survey, many states and school districts 
reported that they would likely increase their use of the 
Clearinghouse if it released information more quickly. 

While IES has increased annual report production, IES has not 
established reasonable production cost ranges or specific cost-related 
performance measures related to each product type. Without acceptable 
per product cost ranges, it is difficult for IES to assess the 
reasonableness of costs associated with certain products, even as IES 
takes steps to streamline production. IES's current study on costs may 
help IES establish acceptable cost ranges that could inform IES's 
performance measurements related to the WWC. In addition, such 
information could inform cost comparisons between the WWC and other 
research evaluation organizations or provide baselines for future 
contractor work. 

In addition, IES has not established meaningful performance measures 
related to product usefulness. Until fiscal year 2010, IES tracked 
visits to its Web site and annual report production as a way to 
measure the productivity of the Clearinghouse. While these measures 
were important to accurately track the WWC's initial growth, they did 
not evaluate the degree to which the products were meeting the needs 
of educators. Specifically, IES currently does not have a way to gauge 
user satisfaction with WWC products, which is a common practice when 
developing and providing new products. Further, while IES currently 
incorporates some feedback from the WWC Web site users, to inform 
future topic areas, it does not systematically gauge its target 
audience's major areas of interest or concern--such as gathering 
information on interventions currently being used or considered in 
specific school districts or states. IES decides how to spread its 
limited resources across the various product types without directly 
measuring the extent to which educators use the WWC or how useful they 
find the various products to be. Measuring the use and usefulness of 
its products could help IES continue to improve content, develop 
products, and respond to the needs of educators and policymakers. 

While some educators and policymakers find WWC products useful, many 
other educators are not familiar with the Clearinghouse. IES has spent 
a substantial amount of money, time, and effort producing various 
summaries of evidence-based practices, which cover both specific 
education interventions and general practices. This investment in the 
WWC was made in order to inform education professionals at all levels--
from classroom teachers to policymakers--as they make decisions on how 
best to educate the nation's children. Improved dissemination of WWC 
products could increase awareness and use of the WWC. Increased use of 
the Clearinghouse could help education professionals identify and 
implement effective educational interventions and practices, and 
potentially lead to increased student achievement. 

Recommendations for Executive Action: 

We are making the following four recommendations based on our review. 

To consistently release WWC products in a timely manner, we recommend 
the Secretary of Education direct IES to develop and implement 
strategies that help avoid future backlogs and ensure that IES's 
review and approval processes keep pace with increased contractor 
production. Strategies could include shifting IES resources to ensure 
sufficient staff time for managing the peer review process and 
streamlining its approval processes. 

To better track the costs and usefulness of the WWC, we recommend that 
the Secretary of Education direct IES to: 

* incorporate findings from its cost studies to develop performance 
measures related to costs, such as identifying a range of acceptable 
costs per product and using that information to monitor contractor 
spending; and: 

* develop performance measures related to product usefulness and 
periodically assess whether WWC products are meeting the needs of 
target audiences by gathering information on product usefulness in the 
proposed survey or through other means. 

To reach more members of the target audience, we recommend the 
Secretary of Education direct IES to assess and improve its 
dissemination efforts to promote greater awareness and use of the WWC, 
for example, by developing a way to inform school districts of new 
products or encouraging educator professional development programs to 
focus on research-based practices such as those discussed in practice 
guides. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the U.S. Department of Education 
for review and comment. Education officials provided written comments 
on a draft of this report, which are reproduced in appendix IV. 
Education also provided technical comments, which we incorporated into 
the report as appropriate. 

Education generally agreed with our recommendations. Specifically, 
Education agreed to our recommendations on consistently releasing WWC 
products in a timely manner and assessing and improving its 
dissemination efforts. In its response to our recommendation on 
tracking the cost and usefulness of the WWC and its products, 
Education noted that IES has taken some steps that address the 
recommendation. With regard to costs, Education stated that it intends 
to incorporate the results of current cost studies into future work 
plans and monitoring efforts. We continue to recommend that these 
results be used to inform performance measures related to costs for 
future operations. With regard to tracking the usefulness of the WWC, 
Education noted that it uses a variety of tools to gather consumer 
input, such as a Help Desk and online voting for future report topics. 
While such feedback provides some information to the WWC, it relies on 
existing users and reflects the views of those users who provide 
feedback, rather than those of the broader population. However, as 
shown in our survey, only an estimated 34 percent of school districts 
have accessed the WWC Web site at least once--and fewer have used the 
Web site frequently. Education also noted that it would include a 
customer satisfaction survey in IES's review of its own performance, 
but whether the survey would be directed at current Clearinghouse 
customers or a broader audience, or whether the survey would identify 
how useful various WWC products are and how the WWC can be improved is 
unclear. More nationally representative information could help IES 
prioritize topics for intervention reports and practice guides and 
inform budget priorities. 

We are sending copies of this report to the appropriate congressional 
committees, the Secretary of the U.S. Department of Education, and 
other interested parties. In addition, the report will be available at 
no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions regarding this report, please 
contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made major contributors to 
this report are listed in appendix V. 

Signed by: 

Cornelia M. Ashby: 
Director, Education, Workforce, and Income Security Issues: 

List of committees: 

The Honorable Tom Harkin:
Chairman:
The Honorable Thad Cochran:
Ranking Member:
Subcommittee on Labor, Health and:
Human Services, Education and Related Agencies:
Committee on Appropriations:
United States Senate: 

The Honorable David Obey:
Chairman:
The Honorable Todd Tiahrt:
Ranking Member:
Subcommittee on Labor, Health and:
Human Services, Education and Related Agencies:
Committee on Appropriations:
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

To address all three objectives, we interviewed officials from the 
Institute of Education Sciences (IES), What Works Clearinghouse (WWC 
or Clearinghouse) contractors, and representatives from various 
educational organizations. To assess the research review process used 
by the IES's WWC, we reviewed WWC standards and procedures, reviewed 
an expert panel report that assessed the validity of the WWC review 
process, and collected information about the extent to which the WWC 
has implemented the panel's recommendations. To determine how 
performance and costs changed over time, we analyzed the costs and 
productivity of the two WWC contractors. To obtain information about 
the usefulness of WWC products, we conducted a Web-based survey of all 
state education agencies and a nationally representative sample of 
school districts. We also collected information about the usefulness 
of the WWC from teachers and principals at four education conferences. 
We conducted our work from September 2009 through July 2010 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Assessment of WWC Research Review Process: 

GAO previously assessed the procedures and criteria used by the WWC by 
reviewing documents and interviewing IES officials and WWC 
contractors.[Footnote 66] We reviewed WWC standards and procedures and 
examined the degree of consistency of these standards and procedures 
across education topic areas. We also reviewed the findings and 
recommendations from an expert panel report that assessed the validity 
of the WWC review process.[Footnote 67] We obtained information from 
IES officials and WWC contractors on the extent to which the WWC has 
implemented the panel's recommendations. Further, we identified other 
concerns about the WWC review process through a literature review and 
interviews with researchers, and we interviewed IES officials and WWC 
contractors to assess the extent to which the Clearinghouse has 
addressed these concerns. We also examined the degree to which the 
WWC's review process is similar to that used by other entities engaged 
in systematic research review efforts. 

Performance and Cost Data Analyses: 

To determine how performance and costs changed over time, we analyzed 
the costs and productivity of the two WWC contractors. We reviewed 
budget data and product release dates to analyze cost and productivity 
trends of the WWC. To examine performance, we interviewed the two 
contractors, as well as IES officials, and compared IES's performance 
measures and goals to actual outcomes. 

We assessed the reliability of the WWC performance and cost data by 
(1) reviewing existing information about the data and the system that 
produced them and (2) interviewing agency officials knowledgeable 
about the data. We determined that the data were sufficiently reliable 
for the purposes of this report. 

Survey of States and School Districts: 

To determine how WWC products are disseminated, we interviewed 
officials from IES and all 10 RELs, as well as WWC contractors. To 
determine how useful education professionals find WWC products to be, 
we designed and administered a Web-based survey of state education 
agencies in the 50 states and the District of Columbia and a 
nationally representative sample of local educational agencies (LEA). 
Specifically, the survey asked officials about (1) their general 
sources of information on effective educational practices, (2) the 
extent to which they use WWC products to inform curriculum decisions 
(including questions on specific intervention reports and practice 
guides), (3) how useful the officials find the information in the WWC, 
(4) the likelihood they would increase their usage if certain changes 
were made to the WWC Web site, and (5) the extent to which the 
officials use the Doing What Works and Best Evidence Encyclopedia Web 
sites to inform curriculum decisions and how useful the officials find 
these other information sources to be. We reproduce the questions we 
used in our analysis in figure 15. The survey was administered from 
February 18, 2010 to April 14, 2010. 

To determine how the WWC was being used at the state level, we 
surveyed the state Secretary, Commissioner, or Superintendent of 
Education in the 50 states and the District of Columbia. Out of the 51 
state officials surveyed, 38 responded to the survey. 

To determine how the WWC was being used at the school district level, 
we surveyed a nationally representative sample of school districts 
across the country. We selected a stratified random sample of 625 LEAs 
from the population of 17,620 LEAs included in our sample frame of 
data obtained from the Common Core of Data for the 2007-08 school 
year. A total of 454 LEAs responded, resulting in a final response 
rate of 74 percent. Because we surveyed a sample of LEAs, survey 
results for the district are estimates of a population of LEAs and 
thus are subject to sampling errors that are associated with samples 
of this size and type. Our sample is only one of a large number of 
samples that we might have drawn. As each sample could have provided 
different estimates, we express our confidence in the precision of our 
particular sample's results as a 95 percent confidence interval (e.g., 
plus or minus 10 percentage points). We excluded 12 of the sampled 
LEAs for various reasons--6 were closed, 3 did not administer any 
schools, 2 managed schools in a correctional facility, and 1 was a 
private school--and therefore were considered out of scope. All 
estimates produced from the sample and presented in this report are 
representative of the in-scope population. 

The practical difficulties of conducting any survey may introduce 
nonsampling errors, such as difficulties interpreting a particular 
question, which can introduce unwanted variability into the survey 
results. We took steps to minimize nonsampling errors by pretesting 
the questionnaire over the phone with officials from two school 
districts and one state department of education in November and 
December 2009. We conducted pretests to verify that (1) the questions 
were clear and unambiguous, (2) terminology was used correctly, (3) 
the questionnaire did not place an undue burden on officials, and (4) 
the questionnaire was comprehensive and unbiased. An independent 
reviewer within GAO also reviewed a draft of the questionnaire prior 
to its administration. We made revisions to the questionnaire based on 
feedback from the pretests and independent review before administering 
the survey. 

The survey-related data used in this report is based on the state and 
school district responses to the survey questions. 

Figure 15: GAO's Web-based Survey of State Departments of Education 
and Local Educational Agencies in the 50 States and the District of 
Columbia: 

[Refer to PDF for image: survey] 

3. To what extent, if at all, does your	rely on external evidence-
based research to inform curriculum decisions? 
(Check only one answer) 
1. Very large extent: 
2. Large extent: 
3. Moderate extent: 
4. Small extent: 
5. Does not use external evidence-based research: 
6. Don't know: 

4. How useful, if at all, are each of the following research evaluation
resources to you or your staff in identifying effective practices to
implement in your______? (Please choose one response for each 
resource.) 

4a. Best Evidence Encyclopedia (Johns Hopkins University): 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4b. Child Trends: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4c. Coalition for Evidence Based Policy: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4d. Doing What Works: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4e. RAND's Promising Practices: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4f. What Works Clearinghouse: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

4g. Other research synthesis clearinghouses: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5. Recognizing that research evaluation resources are not necessarily 
the primary sources of information used to identify effective 
education practices, GAO is also interested in the role of other 
information sources. 

How useful, if at all, are each of the following sources to you or your
staff in identifying effective practices to implement in your ______? 
(Please choose one response for each resource.)	
			
5a. Academic journals: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5b. Education-related periodicals: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5c. Online databases (ERIC or others): 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5d. University-based research institutions: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5e. Non-profit organizations: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5f. Associations of educators or researchers: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5g. Peer conferences: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5h. Regional Education Laboratories (Department of Ed): 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5i. Other federal outreach center: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5j. State government offices and/or outreach centers: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5k. Local data and/or internal research: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

51. Community and parent input: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5m. Mentors/Colleagues: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5n. Personal experience: 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

5o. Other resource (Please specify below): 
Very useful: 
Useful: 
Somewhat useful: 
Slightly useful: 
Not at all useful: 
No opinion: 
Have not used this source of information: 

Other resource: 

7. Have you or your staff heard of the What Works Clearinghouse (WWC)?
(Check only one answer) 

1. Yes: 
2. No (Go to Section 3: Use of Doing What Works): 
3. Don't know (Go to Section 3: Use of Doing What Works): 

8. From which of the following source(s) did you or your staff hear 
about the WWC? 

(Please choose one response for each source.) 

8a. Conferences: 
Yes: 
No: 
Don't know: 

8b. Peers: 
Yes: 
No: 
Don't know: 

8c. Regional Education Labs: 
Yes: 
No: 
Don't know: 

8d. U.S. Department of Education: 
Yes: 
No: 
Don't know: 

8e. Other source (Please specify below): 
Yes: 
No: 
Don't know: 

Other source: 

9. How frequently, if at all, do you or your staff access the WWC 
website? (Check only one answer) 

1. Never (Go to question 28): 
2. Less than twice a year (Go to question 10): 
3. Between 2 and 6 times per year (Go to question 10): 
4. Between 7 and 11 times per year (Go to question 11): 
5. Monthly (Go to question 11): 
6. More than once a month (Go to question 11): 

10. You indicated that you or your staff access the WWC website less 
than 7 times per year. Which of the following reasons best describes 
why you and your staff do not access the website more frequently?
(Check only one answer) 

1. Time Constraints: 
2. Content is not relevant to our decisions: 
3. Disagree with recommendations on the site: 
4. Site is difficult to navigate: 
5. Other reason (Please specify below) Other reason: 

11. To what extent, if at all, have you or your staff used the WWC 
website to inform decisions on effective education practices?
(Check only one answer) 

1. Very large extent (Go to question 13): 
2. Large extent (Go to question 13): 
3. Moderate extent (Go to question 13): 
4. Small extent (Go to question 13): 
5. Have not used the VVWC to inform any decisions (Go to question 12): 
6. Don't know (Go to question 12): 

13. To what extent, if at all, have you or your staff used information 
in the WWC to do any of the following? (Please choose one response for 
each action.) 

13a. Inform professional development of teachers: 
To a very large extent: 
To a large extent: 
To a moderate extent: 
To a small extent: 
Not at all: 
Don't know: 

13b. Advise ______s that are not making AYP on potential interventions: 
To a very large extent: 
To a large extent: 
To a moderate extent: 
To a small extent: 
Not at all: 
Don't know: 

13c. Develop ______improvement plans: 
To a very large extent: 
To a large extent: 
To a moderate extent: 
To a small extent: 
Not at all: 
Don't know: 

13d. Inform curriculum decisions: 
To a very large extent: 
To a large extent: 
To a moderate extent: 
To a small extent: 
Not at all: 
Don't know: 

13e. Other use (Please specify below): 
To a very large extent: 
To a large extent: 
To a moderate extent: 
To a small extent: 
Not at all: 
Don't know: 

Other use: 

14. To what extent have you or your staff used the WWC's Intervention 
Reports to inform decisions on effective education practices?
Intervention Reports provide an assessment of the efficacy of 
interventions based on existing research that meets certain standards.
(Check only one answer) 

1. Very large extent: 
2. Large extent: 
3. Moderate extent: 
4. Small extent: 
5. Don't know: 
6. Have not used the VVWC Intervention Reports to inform any decisions 
(Go to question 18): 

18. To what extent have you or your staff used the WWC's Practice
Guides to inform decisions on effective education practices?
Practice Guides are developed by a panel of experts and provide 
recommendations to help educators address common classroom or school-
wide challenges. (Check only one answer) 

1. Very large extent: 
2. Large extent: 
3. Moderate extent: 
4. Small extent: 
5. Don't know: 
6. Have not used the WWC Practice Guides to inform any decisions (Go 
to question 22): 

22. To what extent have you or your staff used the WWC's Quick Reviews 
to inform decisions on effective education practices? 

Quick Reviews are designed to help educators and policy makers assess 
the quality of recently released research papers and reports.
(Check only one answer) 

1. Very large extent: 
2. Large extent: 
3. Moderate extent: 
4. Small extent: 
5. Don't know: 
6. Have not used the VVWC Quick Reviews to inform any decisions (Go to 
question 25): 

25. How likely or unlikely would you or your staff be to increase your 
usage of the WWC if any of the following information were added to the 
website? (Please choose one response for each type of information.) 
				
25a. Additional topic areas: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25b. Additional practice guides: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25c. Additional studies reviewed: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25d. Additional information on interventions based on studies that may 
not meet WWC standards: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25e. A broader definition of what studies meet WWC standards: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25f. Additional practices with positive reviews: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25g. Reviews of efficacy of programs being used or considered in 
my______: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25h. Reviews that are more timely: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25i. A greater number of intervention reports showing positive effects: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25j. An easier website to navigate: 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

25k. Other information (Please specify below): 
Very Likely: 
Somewhat Likely: 
Neither likely nor unlikely: 
Somewhat unlikely: 
Very unlikely: 
No opinion: 

Other information: 

Source: GAO survey of states' and school districts' use of educational 
clearinghouses. 

[End of figure] 

The following tables contain the estimates and associated confidence 
intervals for the data displayed in figures 8, 10, 12, and 14. 

Table 5: Estimates and Confidence Intervals for Figure 8: 

Q8. From which of the following source(s) did you or your staff hear 
about the WWC? 

Label: Conferences; 
Response: Yes; 
Percentage: 72.72; 
Lower bound: 65.26; 
Upper bound: 80.19. 

Label: Conferences; 
Response: No; 
Percentage: 21.83; 
Lower bound: 15.17; 
Upper bound: 29.77. 

Label: Conferences; 
Response: Don't know; 
Percentage: 5.45; 
Lower bound: 2.36; 
Upper bound: 10.50. 

Label: Peers; 
Response: Yes; 
Percentage: 66.43; 
Lower bound: 58.39; 
Upper bound: 74.47. 

Label: Peers; 
Response: No; 
Percentage: 27.80; 
Lower bound: 20.22; 
Upper bound: 35.37. 

Label: Peers; 
Response: Don't know; 
Percentage: 5.77; 
Lower bound: 2.32; 
Upper bound: 11.62. 

Label: Regional Education Labs; 
Response: Yes; 
Percentage: 47.14; 
Lower bound: 38.40; 
Upper bound: 55.89. 

Label: Regional Education Labs; 
Response: No; 
Percentage: 45.24; 
Lower bound: 36.51; 
Upper bound: 53.97. 

Label: Regional Education Labs; 
Response: Don't know; 
Percentage: 7.62; 
Lower bound: 3.81; 
Upper bound: 13.34. 

Label: U.S. Department of Education; 
Response: Yes; 
Percentage: 71.93; 
Lower bound: 64.41; 
Upper bound: 79.46. 

Label: U.S. Department of Education; 
Response: No; 
Percentage: 22.85; 
Lower bound: 16.17; 
Upper bound: 30.72. 

Label: U.S. Department of Education; 
Response: Don't know; 
Percentage: 5.22; 
Lower bound: 2.00; 
Upper bound: 10.80. 

Source: GAO analysis. 

[End of table] 

Table 6: Estimates and Confidence Intervals for Figure 10: 

Q13: To what extent, if at all, have you or your staff used 
information in the WWC to do any of the following? 

Label: Inform professional development of teachers: 

Response: To a very large extent; 
Percentage: 12.33; 
Lower bound: 6.24; 
Upper bound: 21.15. 

Response: To a large extent; 
Percentage: 19.49; 
Lower bound: 11.74; 
Upper bound: 29.45. 

Response: To a moderate extent; 
Percentage: 38.12; 
Lower bound: 28.25; 
Upper bound: 48.00. 

Response: To a small extent; 
Percentage: 19.48; 
Lower bound: 11.77; 
Upper bound: 29.35. 

Response: Not at all; 
Percentage: 5.90; 
Lower bound: 2.40; 
Upper bound: 11.80. 

Response: Don't know; 
Percentage: 4.67; 
Lower bound: 1.09; 
Upper bound: 12.44. 

Label: Advise schools that are not making adequate yearly progress on 
potential interventions: 

Response: To a very large extent; 
Percentage: 13.23; 
Lower bound: 6.90; 
Upper bound: 22.23. 

Response: To a large extent; 
Percentage: 10.12; 
Lower bound: 5.12; 
Upper bound: 17.48. 

Response: To a moderate extent; 
Percentage: 28.21; 
Lower bound: 18.82; 
Upper bound: 39.24. 

Response: To a small extent; 
Percentage: 11.42; 
Lower bound: 5.69; 
Upper bound: 19.82. 

Response: Not at all; 
Percentage: 28.65; 
Lower bound: 19.55; 
Upper bound: 39.23. 

Response: Don't know; 
Percentage: 8.36; 
Lower bound: 3.54; 
Upper bound: 16.16. 

Label: Develop school improvement plans: 

Response: To a very large extent; 
Percentage: 12.07; 
Lower bound: 6.19; 
Upper bound: 20.56. 

Response: To a large extent; 
Percentage: 14.79; 
Lower bound: 7.59; 
Upper bound: 24.99. 

Response: To a moderate extent; 
Percentage: 34.83; 
Lower bound: 24.68; 
Upper bound: 44.98. 

Response: To a small extent; 
Percentage: 19.15; 
Lower bound: 11.72; 
Upper bound: 28.63. 

Response: Not at all; 
Percentage: 13.37; 
Lower bound: 7.35; 
Upper bound: 21.73. 

Response: Don't know; 
Percentage: 5.79; 
Lower bound: 1.80; 
Upper bound: 13.36. 

Label: Inform curriculum decisions: 

Response: To a very large extent; 
Percentage: 14.95; 
Lower bound: 8.53; 
Upper bound: 23.61. 

Response: To a large extent; 
Percentage: 21.99; 
Lower bound: 14.15; 
Upper bound: 31.65. 

Response: To a moderate extent; 
Percentage: 36.50; 
Lower bound: 26.71; 
Upper bound: 46.28. 

Response: To a small extent; 
Percentage: 19.50; 
Lower bound: 12.02; 
Upper bound: 29.00. 

Response: Not at all; 
Percentage: 3.31; 
Lower bound: 0.92; 
Upper bound: 8.23. 

Response: Don't know; 
Percentage: 3.75; 
Lower bound: 0.63; 
Upper bound: 11.47. 

Source: GAO analysis. 

[End of table] 

Table 7: Estimates and Confidence Intervals for Figure 12: 

Q14: To what extent have you or your staff used the WWCs intervention 
reports to inform decisions on effective education practices? 

Q18: To what extent have you or your staff used the WWCs practice 
guides to inform decisions on effective education practices? 

Q22: To what extent have you or your staff used the WWCs quick reviews 
to inform decisions on effective education practices? 

Label: Intervention reports: 

Response: Very large or large extent; 
Percentage: 20.98; 
Lower bound: 13.66; 
Upper bound: 29.99. 

Response: Moderate extent; 
Percentage: 29.42; 
Lower bound: 20.13; 
Upper bound: 38.71. 

Response: Small extent; 
Percentage: 23.99; 
Lower bound: 15.26; 
Upper bound: 34.67. 

Response: Don't know; 
Percentage: 4.60; 
Lower bound: 1.48; 
Upper bound: 10.51. 

Response: Have not used the WWC intervention report to inform 
decisions; 
Percentage: 21.01; 
Lower bound: 13.23; 
Upper bound: 30.73. 

Label: Practice guides: 

Response: Very large or large extent; 
Percentage: 10.06; 
Lower bound: 5.28; 
Upper bound: 16.95. 

Response: Moderate extent; 
Percentage: 27.78; 
Lower bound: 18.84; 
Upper bound: 38.23. 

Response: Small extent; 
Percentage: 16.77; 
Lower bound: 10.12; 
Upper bound: 25.44. 

Response: Don't know; 
Percentage: 2.91; 
Lower bound: 0.70; 
Upper bound: 7.75. 

Response: Have not used the WWC practice guides to inform decisions; 
Percentage: 42.48; 
Lower bound: 32.26; 
Upper bound: 52.70. 

Label: Quick reviews: 

Response: Very large or large extent; 
Percentage: 12.93; 
Lower bound: 6.66; 
Upper bound: 21.92. 

Response: Moderate extent; 
Percentage: 16.66; 
Lower bound: 9.79; 
Upper bound: 25.70. 

Response: Small extent; 
Percentage: 24.65; 
Lower bound: 15.93; 
Upper bound: 35.21. 

Response: Don't know; 
Percentage: 5.78; 
Lower bound: 2.35; 
Upper bound: 11.55. 

Response: Have not used the WWC quick reviews to inform decisions; 
Percentage: 39.98; 
Lower bound: 30.16; 
Upper bound: 49.81. 

Source: GAO analysis. 

[End of table] 

Table 8: Estimates and Confidence Intervals for Figure 14: 

Q25: How likely or unlikely would you or your staff be to increase 
your usage of the WWC if any of the following information were added 
to the Web site? 

Label: Reviews of efficacy of programs being used or considered in my 
district; 
Response: Very likely or somewhat likely; 
Percentage: 67.92; 
Lower bound: 59.77; 
Upper bound: 76.06. 

Label: A greater number of intervention reports showing positive 
effects; 
Response: Very likely or somewhat likely; 
Percentage: 66.54; 
Lower bound: 58.30; 
Upper bound: 74.77. 

Label: Additional practices with positives reviews; 
Response: Very likely or somewhat likely; 
Percentage: 60.49; 
Lower bound: 51.95; 
Upper bound: 69.03. 

Label: Additional Topic Areas; 
Response: Very likely or somewhat likely; 
Percentage: 57.27; 
Lower bound: 48.66; 
Upper bound: 65.88. 

Label: Additional information on interventions based on studies that 
may not meet WWC standards; 
Response: Very likely or somewhat likely; 
Percentage: 55.97; 
Lower bound: 47.43; 
Upper bound: 64.51. 

Label: Additional Practice Guides; 
Response: Very likely or somewhat likely; 
Percentage: 55.80; 
Lower bound: 47.12; 
Upper bound: 64.47. 

Label: Additional studies reviewed; 
Response: Very likely or somewhat likely; 
Percentage: 54.89; 
Lower bound: 46.24; 
Upper bound: 63.53. 

Label: Reviews that are more timely; 
Response: Very likely or somewhat likely; 
Percentage: 49.95; 
Lower bound: 41.32; 
Upper bound: 58.57. 

Label: A broader definition of what studies meet WWC standards; 
Response: Very likely or somewhat likely; 
Percentage: 47.38; 
Lower bound: 38.70; 
Upper bound: 56.05. 

Label: An easier web site to navigate; 
Response: Very likely or somewhat likely; 
Percentage: 47.01; 
Lower bound: 38.30; 
Upper bound: 55.73. 

Label: Other Information; 
Response: Very likely or somewhat likely; 
Percentage: 17.37; 
Lower bound: 8.69; 
Upper bound: 29.58. 

Source: GAO analysis. 

[End of table] 

Information from Teachers, Principals, and Researchers: 

In addition to interviews with teacher, principal, and research 
organizations, we obtained information about the usefulness of the WWC 
by administering a questionnaire at four conferences of teachers and 
principals. Table 9 provides more information about the conferences we 
attended. 

Table 9: Conferences Attended to Administer Questionnaires to Teachers 
and Principals: 

Conference: National Council of Teachers of Mathematics; 
Regional/national: Regional; 
Location: Nashville, Tennessee; 
Conference dates: November 18-20, 2009; 
Attendance dates: November 19-20, 2009. 

Conference: National Council of Teachers of English; 
Regional/national: National; 
Location: Philadelphia, Pennsylvania; 
Conference dates: November 19-24, 2009; 
Attendance dates: November 21-22, 2009. 

Conference: ASCD (formerly the Association of Supervisors and 
Curriculum Developers); 
Regional/national: National; 
Location: San Antonio, Texas; 
Conference dates: March 6-8, 2010; 
Attendance dates: March 7-8, 2010. 

Conference: National Association of Secondary School Principals; 
Regional/national: National; 
Location: Phoenix, Arizona; 
Conference dates: March 12-14, 2010; 
Attendance dates: March 12-13, 2010. 

Source: GAO. 

[End of table] 

We selected these conferences because they were relevant to segments 
of the WWC's target population that we were not reaching through our 
survey and they were held at times that coincided with our report time 
frames. At each of these conferences, conference organizers agreed to 
have GAO have a table either inside the exhibit hall or just outside 
it. The questionnaires included questions on awareness and use of WWC-
-including use of specific products and use of other information 
sources to identify effective educational practices. For those who had 
not used the WWC, we also asked them to specify the reason they had 
not used it. The information gathered through the questionnaires is 
not generalizable and does not represent the views of teachers and 
principals nationwide. 

[End of section] 

Appendix II: Other Sources of Information Districts Use To Identify 
Effective Education Practices: 

Source of information: Personal Experience; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 80.6; (76.46, 84.79); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 16.6; (12.73, 20.53); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 2.3; (0.90, 4.67); 
Not used: 0.5; (0.06, 1.67). 

Source of information: Local data and/or internal research; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 77.4; (72.85, 81.84); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 14.9; (11.30, 19.13); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 6.2; (3.79, 9.58); 
Not used: 1.5; (0.48, 3.55). 

Source of information: Peer Conferences; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 77.3; (72.93, 81.74); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 16.7; (12.88, 20.58); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 4.2; (2.28, 7.14); 
Not used: 1.7; (0.60, 3.73). 

Source of information: Mentors/Colleagues; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 74.5; (69.97, 79.08); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 21; (16.74, 25.17); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 3.2; (1.53, 5.84); 
Not used: 1.3; (0.42, 3.05). 

Source of information: Education-related periodicals; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 70.9; (66.00, 75.72); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 23; (18.50, 27.50); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 3.2; (1.48, 6.02); 
Not used: 2.9; (1.34, 5.49). 

Source of information: Associations of educators or researchers; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 65; (59.94, 70.04); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 23.1; (18.59, 27.57); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 6.3; (3.83, 9.66); 
Not used: 5.6; (3.39, 8.72). 

Source of information: Academic journals; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 64.3; (59.22, 69.41); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 27.7; (22.90, 32.45); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 4.7; (2.59, 7.81); 
Not used: 3.3; (1.64, 5.85). 

Source of information: Online databases (ERIC or others); 
Estimated percent of district officials that find this source Very 
Useful or Useful: 54.4; (49.10, 59.63); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 30.6; (25.62, 35.53); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 7.9; (5.12, 11.59); 
Not used: 7.1; (4.58, 10.52). 

Source of information: Community and parent input; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 51.7; (46.48, 56.97); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 39.9; (34.67, 45.06); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 6.2; (3.82, 9.53); 
Not used: 2.2; (0.95, 4.19). 

Source of information: University based research institutions; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 50.3; (45.11, 55.49); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 35.4; (30.25, 40.45); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 9.3; (6.30, 13.16); 
Not used: 5; (2.94, 7.98). 

Source of information: Regional Educational Laboratories; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 45.9; (40.66, 51.03); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 31.3; (26.32, 36.18); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 11.5; (8.18, 15.59); 
Not used: 11.4; (8.16, 15.36). 

Source of information: State government offices and/or outreach 
centers; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 39.8; (34.68, 45.01); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 37.8; (32.64, 42.88); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 15.79; (11.87, 20.10); 
Not used: 6.7; (4.33, 9.91). 

Source of information: Other federal outreach centers; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 17.9; (13.99, 21.83); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 34.7; (29.65, 39.77); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 24; (19.41, 28.67); 
Not used: 23.3; (18.82, 27.86). 

Source of information: What Works Clearinghouse; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 24.4; (20.07 28.76; 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 14.8; (11.20, 19.09); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 16.1; (12.25 20.57); 
Not used: 44.7; (39.37 49.96). 

Source of information: Doing What Works; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 22.8; (18.40, 27.10); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 13.8; (10.42, 17.68); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 14.2; (10.52, 18.50); 
Not used: 49.3; (44.03, 54.62). 

Source of information: Non-profit organization; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 21.7; (17.63, 25.83); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 43.9; (38.58, 49.11); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 17; (13.06, 21.60); 
Not used: 17.4; (13.22, 21.59). 

Source of information: RANDs Promising Practices; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 10.1; (7.34, 13.36); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 14; (10.48, 18.09); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 18.9; (14.54, 23.25); 
Not used: 57.1; (51.77, 62.40). 

Source of information: Child Trends; 
Estimated percent of district officials that find this source Very 
Useful or Useful: 9.7; (6.74, 13.29); 
Estimated percent of district officials that find this source Somewhat 
or Slightly Useful: 14.5; (10.93, 18.77); 
Estimated percent of district officials find this source not at all 
useful or no opinion: 16.73; (12.84, 21.25); 
Not used: 59.1; (53.84, 64.33). 

Source: GAO analysis of survey results from the following questions: 
(4) How useful, if at all, are each of the following research 
evaluation resources to you or your staff in identifying effective 
practices to implement in your district?; and (5) Recognizing that 
research evaluation resources are not necessarily the primary sources 
of information used to identify effective education practices, GAO is 
also interested in the role of other information sources. How useful, 
if at all, are each of the following sources to you or your staff in 
identifying effective practices to implement in your district? 

[End of table] 

[End of section] 

Appendix III: IES and WWC Response to Expert Panel Recommendations: 

Expert panel recommendation: 1. Full review. IES should commission a 
full review of the WWC, including a review of the Clearinghouse's 
mission and of the WWC practice guides, which the panel did not 
attempt to evaluate. The panel also recommends that IES consider 
instituting a regular review process to ensure that the WWC is using 
the most appropriate standards in its work; 
IES/WWC response: IES is considering an appropriate mechanism and time 
for conducting a complete review of the WWC. IES believes that the 
first 2 years of the current contract necessitated a tremendous 
development effort to transfer the infrastructure of the Clearinghouse 
in year one from one contractor to another, and in year two, to 
complete reviews in a consistent manner that began under the original 
contract. Now that the Clearinghouse is more clearly in the production 
phase, this may be the appropriate time to plan for a complete review; 
Implementation status: Under consideration. 

Expert panel recommendation: 2(i). Protocol templates. WWC should 
develop standards for crossover and assignment noncompliance, and for 
adjusting intention to treat effects across studies; 
IES/WWC response: IES is currently considering having the WWC develop 
a standard for assessing crossover compliance, following the process 
recently used to revise its attrition standard; Currently, the WWC 
documents crossover reported in studies. Principal investigators have 
discretion to use this information to determine whether a study 
represents a reasonable test of the intervention. Evidence of 
crossover and assignment noncompliance is documented in the 
intervention report and its appendix table A.1. Readers can use that 
information to assess the findings. IES agrees there is value in 
adjusting intent-to-treat effects for compliance, but believes this 
adjustment is inconsistent with its goal of having the WWC be 
transparent in how it reports findings. Making its own estimates to 
account for compliance will lead to differences between what the WWC 
reports and what is found in publicly available literature; Currently 
the WWC does adjust for clustering when authors report their findings 
incorrectly. However, the purpose of the clustering adjustment is to 
correct for an analytic problem in the methods authors use to estimate 
variances, which generally causes them to overstate the precision of 
their findings. In contrast, adjusting for compliance will yield an 
alternate estimate of effects that may differ from the one reported by 
the study; 
Implementation status: Under consideration. 

Expert panel recommendation: 2(ii). Protocol templates. Develop 
standards for documenting the program received in the control arm of 
randomized experiments (or by members of the comparison group in quasi-
experimental designs), and potentially incorporating this information 
in making comparisons across studies and/or interventions; 
IES/WWC response: Though not based on a standard, WWC practice is for 
reviewers to document the counterfactual in study review guides and in 
intervention reports (the information is reported in appendix table 
A.1). Reviewers routinely send author queries for this information, if 
it is not provided in the study; IES has asked the WWC to assess how 
other review organizations report counterfactual information and the 
utility of incorporating this information into its reports. IES 
officials are also considering an alternative approach that would code 
information about the counterfactual in a study into the study 
database, which then would generate summary tables that would report 
results for studies that have similar counterfactuals. This approach 
has downsides as well, since the set of counterfactuals could be quite 
varied and many assumptions would have to be made to group 
counterfactuals together. We are therefore proceeding cautiously in 
making any changes to current WWC practice; 
Implementation status: Considered but not planning to implement. 

Expert panel recommendation: 2 (iii). Protocol templates. Revise 
standards for multiple comparisons in light of the recent research 
report by Peter Schochet entitled Guidelines for Multiple Testing in 
Experimental Evaluations of Educational Intervention; 
IES/WWC response: WWC staff consulted with Dr. Schochet to investigate 
the possibility of revising the multiple comparison standards; 
Dr. Schochet indicated that his report focused on issues related to 
multiple comparisons within single studies. It did not tackle issues 
related to multiple comparisons issues that may arise when 
synthesizing evidence for a set of studies. WWC procedures are 
consistent with his report for handling multiple comparisons within a 
study; 
Implementation status: Considered but not planning to implement. 

Expert panel recommendation: 2(iv). Protocol templates. Reconsider the 
current process of setting different attrition standards in different 
topic areas; 
IES/WWC response: At the time of the National Board of Education 
Sciences Expert Panel's data collection, the WWC was already reviewing 
its attrition standards. The WWC released new attrition standards in 
December 2008 in the Procedures and Standards Handbook. The new 
standards requires a principal investigator in a topic area to choose 
one of two well-specified attrition boundaries, and the standards 
include guidance on how to choose between the boundaries based on the 
nature of research in the topic area; The attrition discussion is in 
the WWC Procedures and Standards Handbook (Version 2.0) posted on the 
Clearinghouse's Web site; 
Implementation status: Implemented. 

Expert panel recommendation: 2(v). Protocol templates. Establish a 
protocol to keep track of potential conflicts of interest, such as 
cases where a study is funded or conducted by a program developer, and 
consider making that information available in its reports; 
IES/WWC response: IES is considering options for collecting and 
documenting potential conflicts of interest; Sources of funding are 
rarely included in published documents beyond government and 
foundation support. An alternate source of information for tracking 
potential conflicts of interest would be for the WWC to request that 
study authors identify their source of funding, which would provide 
the WWC with a basis for flagging a potential conflict of interest. 
Any effort would depend on cooperation from authors because the WWC 
has no leverage to formally require authors to declare potential 
conflicts (which some academic journals require as a condition for 
publication). WWC's experience to date is that study authors 
frequently fail to respond to requests for additional information, and 
IES officials expect that many study authors likewise will not respond 
to requests for information about funding sources, or may judge that 
it is not in their proprietary interest to provide the information. 
Currently the WWC only queries authors in cases where the 
Clearinghouse needs additional information. Querying all authors and 
tracking their responses would increase costs for intervention reports; 
Another potential option is to ask developers, when they are reviewing 
the list of studies WWC found during the literature search for 
comprehensiveness, to note any studies that they funded; 
Implementation status: Under consideration. 

Expert panel recommendation: 2(vi). Protocol templates. Define 
precisely the standards for "randomization" in a multilevel setting; 
IES/WWC response: The current version of the handbook gives guidance 
on standards for random assignment in simple cases. The next version 
of the handbook (forthcoming in 2010) will provide guidance and 
examples for multilevel settings, with explicit guidance on acceptable 
practice and potential issues with random assignment in a multilevel 
setting; 
Implementation status: Implemented. 

Expert panel recommendation: 3. Documentation of search process. WWC 
should expand the protocol templates to specify more explicit 
documentation of the actual search process used in each topic area and 
maintain a record of the results of the process that can be used to 
guide decision making on future modifications; 
IES/WWC response: IES asked the WWC to review the search process. The; 
WWC now takes steps to ensure that search records are maintained. Each 
team and the library maintain a record of conducted searches. More 
documentation on the process will be included in the forthcoming 
revision of the handbook; 
Implementation status: Implemented. 

Expert panel recommendation: 4. Reliability of eligibility screening. 
WWC should conduct regular studies of the reliability of the 
eligibility screening process, using two independent screeners, and 
use the results from these studies to refine the eligibility screening 
rules and screening practices; 
IES/WWC response: The WWC is undertaking a pilot using five recent 
evidence reports in different topic areas. Because WWC screeners are 
encouraged to pass to the next stage any study for which they are 
uncertain about eligibility, the proportion of eligible studies that 
are excluded is the salient error rate (the other source of error is 
when screeners include an ineligible study in a review, but this error 
is then offset by the review); 
IES officials are not aware of any established standards for 
acceptable error rates (there are tradeoffs between making Type I vs. 
Type II errors relating to cost), but will examine this issue further. 
If the screening error rate is larger than the IES and the WWC 
believes is acceptable, IES officials will assess whether additional 
training or two screeners is appropriate given the different costs and 
benefits of each approach; 
Implementation status: Under consideration. 

Expert panel recommendation: 5. Documentation of screening process. 
WWC reports should include a flow chart documenting the flow of 
studies through each review and number of studies excluded at each 
point, and a table of excluded studies, listing specific reasons for 
exclusion for each study; 
IES/WWC response: Currently, reference lists for WWC intervention 
reports include all studies, both eligible and ineligible, located in 
the search process. Ineligible studies are flagged with the primary 
reason for not qualifying for further WWC review. Intervention reports 
do not list materials such as product descriptions or reviews of 
products that are deemed not relevant to the intervention being 
reviewed; To make the number of studies (both eligible and ineligible 
studies) more apparent to readers, the WWC will add a text box to 
intervention reports located in front of the listing of reports. The 
text box will summarize the number of studies that met different 
conditions (this approach currently is used for reports in which none 
of the studies meet standards). The box will serve the same purpose as 
a flow chart but the codes used to describe the final status for 
reports will be the same ones currently used in the citation appendix. 
The WWC plans to begin including the text box in reports released in 
2010 and thereafter; 
Implementation status: Implemented. 

Expert panel recommendation: 6. Misalignment adjustment. In cases 
where a study analysis is "misaligned," WWC staff should request that 
study authors reanalyze their data correctly, taking into account the 
unit of randomization and clustering. The panel recommends that the 
results from the process be compared to the adjustment procedure 
currently specified, to develop evidence on the validity of the latter; 
IES/WWC response: Ideally, the primary source for reanalyses of data 
would be study authors. However, as noted above in response to 
recommendation 2(v), it is common for authors not to respond to the 
WWC's requests for additional information. Reanalyzing the data also 
would require additional effort by the authors and would run into 
difficulties when studies are dated or are based on data that has been 
destroyed to comply with confidentiality or privacy restrictions; The 
WWC recently undertook a survey of published clustering estimates. It 
found that the WWC's current default clustering correction is 
consistent with published estimates for achievement and behavioral 
outcomes. The WWC will continue to monitor research developments on 
this topic; 
Implementation status: Considered but not planning to implement. 

Expert panel recommendation: 7. Combining evidence across multiple 
studies. WWC should re-evaluate its procedures for combining evidence 
across studies, with specific attention to the issue of how the rules 
for combining evidence can be optimally tuned, given the objectives of 
the WWC review process and the sample sizes in typical studies for a 
topic area; 
IES/WWC response: There are, of course, many possible ways to 
summarize evidence. Given its intended broad and primarily 
nontechnical audience, the WWC's current approach is designed to be 
transparent and easily explained. IES believes that having the WWC 
conduct its own analyses to estimate intervention effects, as 
statistical meta-analyses do, would be inconsistent with these goals; 
However, as an alternative to modifying the WWC's main approach for 
reporting findings, IES is considering having the WWC conduct 
supplemental meta-analyses related to specific questions of interest, 
and releasing these findings as a separate report that would 
complement intervention reports. For example, a report could analyze 
whether computer mediated approaches to teaching reading are more or 
less effective than approaches that rely solely on teachers, based on 
already-released interventions reports. Having a separate report 
enables the WWC to continue using its current transparent approach, 
while also using statistical techniques that combine evidence in other 
ways; 
Implementation status: Under consideration. 

Expert panel recommendation: 8. Reporting. (i) Published reports on 
the Web site should include the topic area protocols, as well as more 
information on the screening process results that led to the set of 
eligible studies actually summarized in the topic area reports. (ii) 
WWC should make available its Standards and Procedures Handbook, 
including appendixes, as well as all other relevant documents that 
establish and document its policies and procedures; 
IES/WWC response: Topic area protocols are available on the topic area 
home pages; Just after the expert panel's report, the WWC released its 
Procedures and Standards Handbook in December 2008. A revision 
currently is under way that will include more detail on the screening 
process. See the response to (3) above related to results of the 
screening process and the response to (5) regarding the results of the 
screening process; 
Implementation status: Implemented. 

Expert panel recommendation: 9. Practice guides. Clearly separate 
practice guides from the topic and intervention reports; 
IES/WWC response: IES agrees that these products need to remain 
distinct. Practice guides are on a separate Web site tab that 
separates them from intervention reports. The next revision of the 
handbook (forthcoming in 2010) will include a chapter describing the 
practice guide development process and how it is different from that 
of the evidence reports; The recently released guide on What Works for 
Practitioners also provides more information on reports and practice 
guides, and the WWC is preparing a video tutorial that will explain 
the differences to users; 
Implementation status: Implemented. 

Expert panel recommendation: 10. Outreach and collaboration with other 
organizations. (i) The WWC should build and maintain a relationship 
with national and international organizations focusing on systematic 
reviews to engage in the broader scientific community and learn about 
the latest standards and practices. (ii) The WWC should convene 
working groups with a mixture of researchers (including specialists in 
education research and systematic reviews) to address the development 
of new standards for the review and synthesis of studies; 
IES/WWC response: The WWC tries to keep abreast of developments in the 
field, for example, by routinely checking materials from the Cochrane 
Collaborative when developing new standards or approaches; Most 
recently, the WWC has undertaken the following outreach efforts to 
connect with other organizations conducting systematic reviews: 
* The WWC sponsored a forum on research methods in December 2008 that 
featured speakers from the National Academy of Sciences, the National 
Cancer Institute, and the Cochrane Collaboration; 
* In June 2009 WWC staff attended the Cochrane conference on practice 
guides in June to learn about state of the art methods in research 
synthesis and practice guides; 
* The WWC is presenting a workshop on WWC standards at the upcoming 
annual conference of the Association of Public Policy and Management; 
The WWC has also met with six international contacts (from Sweden, 
Denmark, Hungary, England, Interamerican Development Bank, and 
Trinidad/Tobago) in response to inquiries about how governments or 
organizations could implement their own clearinghouse operations; 
Recently the WWC began a webinar series to disseminate its new 
practice guides. The webinar includes researchers and practitioners in 
its audience; The WWC convened two groups of researchers to develop 
its forthcoming standards on single-subject designs and regression 
discontinuity designs. It will continue to bring together researchers 
as needs for new standards are identified. This approach will continue 
to be used for developing new standards; 
Implementation status: Implemented. 

Source: GAO analysis of IES and WWC data. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Department of Education: 

United States Department Of Education: 
Institute Of Education Sciences: 
The Director: 
555 New Jersey Ave., NW: 
Washington, D.C. 20208: 

July 6, 2010: 

Ms. Cornelia M. Ashby: 
Director, Education, Workforce, and Income Security Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Ashby: 

Thank you for the opportunity to respond to the Government 
Accountability Office (GAO) draft report, Improved Dissemination and 
Timely Product Release Would Enhance the Usefulness of the What Works 
Clearinghouse (GA040-644). 

This report indicates that GAO, as well as a congressionally mandated 
panel of experts, found that the What Works Clearinghouse's review 
process follows accepted standards for evaluating research on the 
effectiveness of education interventions. It notes that the 
Clearinghouse is responding to recommendations made by the expert 
panel to further improve its review and reporting processes, and to 
researchers who have criticized the WWC for excluding some types of 
research designs that may be appropriate for evaluating certain 
education programs, such as special education. It also recognized that 
product output and scope of the WWC have increased. 

While the report emphasized the need for more timely release of WWC 
reports, it noted, but did not highlight, other important findings 
from the school district survey that point to important avenues for 
improvement. For example, a higher proportion of district respondents 
cited having evidence reviews of programs that they might use in their 
local areas (68 percent) or that show positive effects (67 percent) as 
a reason to use WWC products than cited getting reviews more quickly 
(50 percent) (Figure 15, page 36). Given that the timetable of WWC 
reports is not posted publicly, we are not sure how to interpret 
districts' concerns with timeliness, but we have heard their need for 
more relevant evidence reports covering more educational areas and 
programs. As GAO noted, a backlog of reports that built up during a 
time of senior leadership turnover has been resolved, and many new 
reports are soon to be released. In addition, the Institute of 
Education Sciences is continuing its efforts to ensure that expansion 
into new topic areas is responsive to the needs of the field. 

We appreciate the opportunity to comment on the draft report. The 
Department of Education has prepared the attached comments in response 
to your draft report. If you have any questions regarding this 
response, please contact Dr. Rebecca Maynard, Commissioner for 
Education Evaluation and Regional Assistance, in the Institute of 
Education Sciences at (202) 208-1289. 

Sincerely, 

Signed by: 

John Q. Easton: 
Director: 

Attachments: 

[End of letter] 

Attachment 1: 

IES Response to GAO Report: Improved Dissemination and Timely Product 
Release Would Enhance the Usefulness of the What Works Clearinghouse, 
Draft: 

Recommendation 1: 

To consistently release WWC products in a timely manner, we recommend 
the Secretary of Education direct IES to develop and implement 
strategies that help avoid future backlogs and ensure that IES' review 
and approval processes keep pace with increased contractor production. 
Strategies could include shifting IES resources to ensure sufficient 
staff time for managing the peer review process and streamlining its 
approval process. 

Response: 

The Department of Education (Department) agrees with GAO's 
recommendation and is developing and implementing strategies to ensure 
that IES' review and approval processes keep pace with production. The 
Director of the Institute of Education Sciences has recently appointed 
a Commissioner of the National Center for Education Evaluation and 
Regional Assistance who will oversee the content and operations of the 
What Works Clearinghouse. 

Recommendation 2: 

To better track the usefulness and costs of the WWC, we recommend that 
the Secretary of Education direct IES to: 

* Incorporate findings from its cost studies to develop performance 
measures related to costs, such as identifying a range of acceptable 
costs per product and using that information to monitor contractor 
spending. 

* Develop performance measures related to product usefulness and 
periodically assess whether WWC products are meeting the needs of 
target audiences, by gathering information on product usefulness in 
the proposed survey or through other means. 

Response: 

First, as noted in the report, WWC is currently conducting assessments 
for streamlining procedures and costs at IES' request. The results, 
due at the end of October 2010, will be incorporated into future WWC 
work plans and IES's monitoring of contractor performance and costs. 
IES intends to undertake such assessments on an ongoing basis for 
continuous improvement. 

Second, the WWC will continue to monitor and evaluate consumer use and 
assessment of WWC products, as well as gather information regarding 
suggestions for improvements. In addition, the WWC will be included as 
part of IES' review of its own performance, which will include 
surveying customers on satisfaction. 

In addition, the Clearinghouse already has in place a number of tools 
for users to provide input, and we will continue to emphasize their 
availability to the education community. The WWC Help Desk allows 
users to contact the WWC to send suggestions and ask questions about 
specific products. Users contact the Help Desk through an 800 number 
or through the WWC Web site. Since 2007, the WWC has responded to over 
2,100 inquiries to the Help Desk. 

Users have suggested over 200 interventions for the WWC to review, and 
have suggested over 30 different topics for practice guides. This 
information is continuously reviewed by WWC staff and informs 
priorities. The WWC has used Web-based tools for soliciting specific 
feedback from users. For example, the WWC has posted potential 
practice guide topics on a Web-based survey, having users "vote" for 
their favorite topics. The forthcoming practice guides on Reading 
Comprehension, Fractions, and Teaching Writing were all identified 
through this process. 

In 2007, 2008, and 2009, the WWC conducted focus groups to solicit 
feedback from educators on several aspects of its work. These included 
such topics as the usability of the WWC Web site, which led to 
improvement to the site functionality, and the format of practice 
guides, which led to changes in the layout of the guides. 

Visits to the WWC Web site have increased substantially over time. 
Between fiscal years 2008 and 2009, the number of visits rose by 45 
percent to 772,000. And, to date in 2010, there have been 618,000 
visits to the Web site, an increase of 21 percent relative to the same 
period in fiscal year 2009. IES will continue to track all of these 
measures to ensure that we are using solid information on customer use 
and satisfaction. 

Recommendation 3: 

To reach more members of the target audience, we recommend the 
Secretary of Education direct IES to assess and improve its 
dissemination efforts to promote greater awareness and use of the WWC, 
for example, by developing a way to inform school districts of new 
products or encouraging educator professional development programs to 
focus on research-based practices such as those discussed in practice 
guides. 

Response: 

The Department agrees with GAO's recommendation and is developing 
further means to assess and improve dissemination efforts. IES' 
monitoring of the WWC contract requires implementation of an Annual 
Communication Plan that specifies how the contractor will conduct 
outreach, dissemination, and communication activities that include, 
for example, the WWCFlash, media and trade organization outreach, 
development of dissemination partnerships with organizations and the 
education community, WWC sponsored events and forums, and targeted 
product outreach. IES has made all WWC products available through the 
Department's Educational Resources Information Center (ERIC), an 
online digital library of education research and information, and 
provides RSS feeds of WWC products to ERIC users. The Department's 
National Library of Education makes all WWC products, together with 
all items indexed in ERIC, available to Google, MSN, and Yahoo search 
engines for harvesting and, therefore, accessible to Web users 
worldwide. All WWC products are now available to WorldCat, an 
international bibliographic database that is used by libraries around 
the world to identify and locate resources. 

IES will continue to develop and implement the Communication Plan 
vehicle to assess and improve dissemination, as well as fully utilize 
other Department dissemination resources. One example of a new vehicle 
that the Clearinghouse has in the works to expand its dissemination 
partnerships with organizations and the education community is a 
monthly WWC Newsletter on up-to-date information on recently released 
Clearinghouse products as well as those in line for release in the 
month ahead. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Cornelia M. Ashby, (202) 512-7215, ashbyc@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, individuals making key 
contributions to this report include Elizabeth Morrison (Assistant 
Director), Nagla'a El-Hodiri (analyst-in-charge), James Ashley, Carl 
Barden, James Bennett, Valerie Caracelli, Laura Henry, Geoffrey King, 
Jill Lacey, Luann Moy, Robert Owens, Cathy Roark, Stephanie Shipman, 
Kate Van Gelder, and Craig Winslow. 

[End of section] 

Related GAO Products: 

Program Evaluation: A Variety of Rigorous Methods Can Help Identify 
Effective Interventions. [hyperlink, 
http://www.gao.gov/products/GAO-10-30]. Washington, D.C.: November 23, 
2009. 

Teacher Quality: Sustained Coordination among Key Federal Education 
Programs Could Enhance State Efforts to Improve Teacher Quality. 
[hyperlink, http://www.gao.gov/products/GAO-09-593]. Washington, D.C.: 
July 6, 2009. 

Teacher Preparation: Multiple Federal Education Offices Support 
Teacher Preparation for Instructing Students with Disabilities and 
English Language Learners, but Systematic Departmentwide Coordination 
Could Enhance This Assistance. [hyperlink, 
http://www.gao.gov/products/GAO-09-573]. Washington, D.C.: July 20, 
2009. 

No Child Left Behind Act: Education Actions Could Improve the 
Targeting of School Improvement Funds to Schools Most in Need of 
Assistance. [hyperlink, http://www.gao.gov/products/GAO-08-380]. 
Washington, D.C.: February 29, 2008. 

Program Evaluation: Strategies for Assessing How Information 
Dissemination Contributes to Agency Goals. [hyperlink, 
http://www.gao.gov/products/GAO-02-923]. Washington, D.C.: September 
30, 2002. 

The Evaluation Synthesis. [hyperlink, 
http://www.gao.gov/products/PEMD-10.1.2]. Washington, D.C.: March 1992. 

[End of section] 

Footnotes: 

[1] 20 U.S.C.  6301-7941. The mission and functions to be performed 
by IES are set out at 20 U.S.C.  9511. 

[2] This is the second 5-year contract for the Clearinghouse. The 
first contract for about $27 million expired in 2007. 

[3] H.R. Comm. on Appropriations, 111th Cong., Omnibus Appropriations 
Act, 2009: Comm. Print of the Comm. on Appropriations U.S. 
Representatives on H.R. 1105/Public Law 111-8, at 1483 (Comm. Print 
2009). 

[4] Thirty-seven states and the District of Columbia and 74 percent of 
surveyed districts responded to our survey. 

[5] WWC's mission is consistent with IES's broader mission to bring 
rigorous and relevant research, evaluation, and statistics to the 
nation's education system. 

[6] The current priority for the 2006-2010 REL contract period is 
providing policymakers and practitioners with expert advice, training, 
and technical assistance on how to interpret the latest findings from 
scientifically valid research pertaining to requirements of the ESEA. 

[7] This contract was awarded to the American Institutes for Research 
and the Campbell Collaboration. In 2007, Education awarded the second 
5-year contract to Mathematica Policy Research, Inc. to operate the 
WWC. 

[8] The initial topic areas chosen in 2003 were: beginning reading, K- 
12 math achievement, dropout prevention, adult literacy, peer-assisted 
learning, reducing delinquency, and English language acquisition. 

[9] IES dropped the peer-assisted learning topic area, which would 
have covered interventions related to students learning with and from 
other students--generally in the same class and at a similar academic 
level. 

[10] The WWC intervention reports primarily focus on branded products. 

[11] In June 2010 IES made public its standards for two additional 
study designs: regression discontinuity and single case design studies. 

[12] IES uses some of this information to determine a performance-
based award for the contractor. 

[13] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help 
Identify Effective Interventions, [hyperlink, 
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23, 
2009). 

[14] For example, WWC rates the credibility of a study's evidence 
along a continuum. 

[15] To help educators understand the research behind a WWC report, 
the WWC (1) combines information on the size and number of studies 
reviewed to rate the extent of evidence as small or medium/large; (2) 
includes an overall rating of effectiveness on each measured outcome, 
which combines the size and direction of effects, statistical 
significance, and the quality of the research designs; and (3) reports 
the average improvement index across studies as the expected change in 
percentile rank for an average control group student if the student 
had received the intervention. 

[16] H. Brown, D. Card, K. Dickersin, J. Greenhouse, J. Kling, and J. 
Littell, Expert Report on the What Works Clearinghouse, a report 
prepared by the National Board for Education Sciences, 2008. The 
expert panel was convened by the National Board for Education Sciences 
in 2008 in response to the Senate Appropriations Committee. S. Rep. 
No. 110-410 at 228-29. The National Board for Education Sciences, 
consisting of 15 voting members appointed by the President with the 
advice and consent of the Senate, provides IES guidance and oversight. 
20 U.S.C.  9516. The mandate directed the Board to convene leading 
experts in rigorous evaluations to assess the WWC, specified that 
panel members should be free of conflicts of interest and that a 
report with any recommendations was to be submitted within 4 months. 
Expert panel members included economists, statisticians, and 
professors with expertise in other systematic review efforts in the 
fields of health care, and social policy. 

[17] The WWC considered but is not implementing three of the panel's 
recommendations. One recommendation suggested the WWC develop 
standards for documenting the program received by the comparison group 
that did not receive the intervention and potentially incorporating 
this information when making comparisons across studies and/or 
interventions. The other two related to WWC procedures for combining 
evidence across studies and asking study authors to reanalyze their 
data to correct a common error associated with the use of classrooms 
rather than individual students in data analysis. This mismatch can 
result in an overstatement of the statistical significance of the 
effects of the intervention. The WWC maintains that its current 
procedures are consistent with standard practices and has elected not 
to ask authors to reanalyze their data. See appendix III for more 
detail. 

[18] The handbook documents the actions that WWC staff must take when 
reviewing research and the items that must be included in the reports, 
among other things. It is available at [hyperlink, 
http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf]
 

[19] WWC requires reviewers to select one of two levels of attrition 
(higher or lower) depending on the topic area and context. WWC allows 
a higher level of attrition for topic areas in which it assumes that 
attrition is due to factors that are not strongly related to the 
intervention. WWC allows a lower level of attrition for topic areas in 
which attrition may be due to certain individuals choosing not to 
participate in the intervention. 

[20] In cases in which a study of a program is funded by the program 
developer, the study authors may have incentives to find positive 
effects of the program. Such incentives could call the validity of the 
study's results into question. 

[21] For example, see Robert Slavin and Dewi Smith, "The Relationship 
Between Sample Sizes and Effect Sizes in Systematic Reviews in 
Education," Educational Evaluation and Policy Analysis, vol. 31, no. 4 
(2009): 500-506. 

[22] WWC staff also contend that there is no statistical basis for 
setting a minimum sample size and doing so would arbitrarily ignore 
available evidence and potentially bias the findings of a systematic 
review. 

[23] Robert E. Slavin and Nancy A. Madden, "Measures Inherent to 
Treatments in Program Effectiveness Reviews," paper presented at the 
annual meetings of the Society for Research on Effective Education, 
Crystal City, Virginia, March 3-4, 2008; and Robert E. Slavin, "What 
Works? Issues in Synthesizing Educational Program Evaluations," 
Educational Researcher, vol. 37, no. 1 (Jan./Feb. 2008): 5. 

[24] Intervention developers may intentionally or unintentionally 
create a test that is more likely to favor the intervention because 
they have financial or other interests in the success of the 
intervention. 

[25] While randomized control trials and quasi-experiments are 
considered to be rigorous approaches in assessing program 
effectiveness, they are not the only rigorous research designs 
available and may not always be appropriate. For example, such 
comparison group designs may not be appropriate for research on small 
numbers of students receiving special education services in a self- 
contained classroom. In such a case, an in-depth case study may be 
more appropriate. Examples of other research methods include 
statistical analyses of observational data, such as student records, 
or analyses of surveys of an intervention's participants. 

[26] The WWC produces intervention reports noting when no studies meet 
standards. 

[27] Specifically, the WWC developed standards--which were made 
publicly available in June 2010--for reviewing single-case and 
regression discontinuity designs. The WWC anticipates reviewing many 
studies with single-case designs--studies that involve repeated 
measurement of a single subject (e.g., a student or a classroom)--as 
it evaluates interventions for special education. Regression 
discontinuity designs compare outcomes for a treatment and control 
group that are formed based on the results of a preintervention 
measure. 

[28] The WWC has published 12 practice guides as of May 2010. The 
topics of these 12 practice guides are using data to support decision 
making, helping students navigate the path to college, structuring out-
of-school time, assisting students in math, assisting students in 
reading, reducing behavior problems, dropout prevention, improving 
literacy, turning around low-performing schools, instruction for 
English language learners, encouraging girls in math and science, and 
organizing instruction and study. 

[29] IES requires the contractor to file WWC annual plans that outline 
planned product releases and other deliverables. IES and the 
contractor update these plans once a year with revised estimates. 

[30] Under the first contract (2002 to 2007), the WWC released 89 
intervention reports, six topic reports, and three practice guides. 

[31] Pub. L. No.111-5, 123 Stat. 115, 182. 

[32] The current contract also requires WWC to create and maintain 
other resources on its Web site, such as registries of researchers and 
randomized trials and the WWC Policy and Procedures Handbook. IES 
noted that these deliverables are either new or significantly enhanced 
from those produced under the first WWC contract. 

[33] IES indicated that the amount of research available meeting WWC 
inclusion standards for a given report varied and had an impact on the 
number of staff hours required in the production of reports. Reports 
based on larger numbers of studies took more staff hours to complete 
than those based on less available evidence. 

[34] We discussed the backlog and its causes with IES officials in 
February and May 2010. For the first six months of 2010, IES completed 
the review of 59 report products (intervention reports and quick 
reviews)--compared to 46 for the entirety of calendar year 2009--thus 
eliminating the backlog. 

[35] IES also spends about $200,000 per year on noncontracted WWC 
expenses--including internal salaries, independent peer review 
honorariums, and Web site support--which have not changed 
significantly between the two 5-year contracts. In addition, three 
practice guides were completed outside of the WWC contract, at a total 
cost of about $319,000. IES noted that these preliminary guides were 
produced through a less thorough process than the current process. 

[36] Both contractors dedicated the same proportion of funds to WWC 
products. The first contractor primarily published products in the 
final year of the first contract (2007); however, products were 
produced, reviewed, and modified--but not published--prior to that 
year. As a result, despite limited publication in the first 4 years, a 
large portion of the first contractor's expenditures were designated 
for direct product costs. 

[37] According to WWC staff, this system allows them to use results 
from prior literature searches for related topics, rather than 
conducting new searches. The current contractor also designed and 
implemented standardized training for staff and subcontractors who 
evaluate research. All WWC research evaluators complete 2 days of 
training; are tested on WWC products, review standards, and policy; 
and have initial reviews with monitored before working independently. 

[38] The Organizational Assessment--Education's performance management 
system--was developed in response to the requirements of Executive 
Order 13450, Improving Government Program Performance, as well as the 
Office of Personnel Management's requirement that each federal agency 
evaluate its principal offices on an annual basis. 

[39] The contract award fee plan includes performance measures related 
to production, business management of the contract, and timeliness. 
Business management of the contract includes cost management, business 
relationships, efforts to meet small business subcontracting goals, 
and accurate billing. These measures are linked to work specified in 
the WWC contract and annual plans. 

[40] While the WWC annually exceeded performance targets, it is 
difficult to interpret these results as the performance measures 
changed annually and, according to IES officials, the criteria for 
meeting them were negotiated well into the fiscal year. 

[41] In addition, IES determined that the current and prior 
contractors generally met the award fee plan performance measures. 

[42] The WWC's award fee plan includes cost management components but 
has no cost per product measurements. 

[43] This survey would ascertain whether IES has met its goal that at 
least 25 percent of decision makers surveyed will have consulted the 
Clearinghouse prior to making decisions on interventions in reading, 
writing, mathematics, science, or teacher quality by 2013. 

[44] WWC staff told us that the listserv had over 10,000 subscribers 
and that Web site visits increase after conferences. 

[45] IES's Web site hosts an "Ask A REL" page, where educators can 
submit questions. "Ask A REL" is described on the Web site as being a 
collaborative reference desk service provided by the 10 RELs that 
functions much in the same way as a technical reference library. 

[46] Thirty-seven states and the District of Columbia responded to our 
survey. While the District of Columbia is not a state, we will refer 
to the survey respondents as representing 38 states. 

[47] The 95 percent confidence interval for this estimate is (36.7, 
46.6). 

[48] While based on 38 state-level respondents, this analysis provides 
the minimum number (33) of states (overall) that have accessed the WWC 
Web site. Regardless of whether or not the 13 states that did not 
respond to our survey have accessed the Clearinghouse Web site, 33 is 
about two-thirds of the 51 states and constitutes a majority of states. 

[49] Our survey asked respondents to indicate the number of times they 
had accessed the WWC Web site. Answer choices included never; less 
than twice a year; between 2 and 6 times per year; between 7 and 11 
times per year; monthly; and more than once a month. 

[50] The 95 percent confidence interval for this estimate is (29.4 , 
38.8). 

[51] The 95 percent confidence interval for this estimate is (8.3, 
14.8). 

[52] Fourteen states reported accessing the Clearinghouse six or fewer 
times a year, as did an estimated 72 percent of districts. In addition 
to time constraints (cited by 7 of the 14 states), five states 
reported that they did not access the WWC more frequently because its 
content was not relevant to their decisions. 

[53] The 95 percent confidence interval for this estimate is (94.2, 
98.4). 

[54] Our survey asked states and districts to report how useful 
certain sources of information were in identifying effective education 
practices. The sources ranged from general (personal experience, 
education periodicals) to specific (RELs, federal outreach centers). 
In addition, we listed several research synthesis organizations by 
name, including the WWC and the Best Evidence Encyclopedia. See 
appendix II for more details. 

[55] Between November 2009 and February 2010, we attended a regional 
conference for the National Council for Teachers of Mathematics in 
Nashville, Tenn., as well as three national conferences: National 
Council for Teachers of English, ASCD (formerly the Association of 
Supervision and Curriculum Development), and the National Association 
of Secondary School Principals. 

[56] One state official responded that he and his staff had not used 
the WWC to inform any decisions on effective education practices, 
while another state official responded "Don't know." 

[57] The 95 percent confidence intervals for these estimates are 
(55.5, 87.7) and (10.3, 29.8), respectively. 

[58] The estimates and their 95 percent confidence intervals were as 
follows: inform curriculum decisions--93 percent (85.5, 97.3); inform 
professional development of teachers--89.4 percent (81.7, 94.7). 

[59] The 95 percent confidence interval for this estimate is (64.3, 
82.8). 

[60] The estimates and their 95 percent confidence intervals were as 
follows: practice guides--54.6 percent (44.4, 64.8); quick reviews-- 
54.2 percent (44.4, 64.3). 

[61] The 95 percent confidence interval for this estimate is (13.7, 
30). 

[62] The 95 percent confidence interval for this estimate is (5.3, 17). 

[63] Twenty-six states that had used the Clearinghouse reported that 
they would be somewhat likely or very likely to increase their use of 
the WWC if it had a greater number of intervention reports showing 
positive effects--a number which depends both on the number of 
interventions that the WWC reviews and whether the results of 
available research meeting WWC standards show positive effects. 

[64] The 95 percent confidence interval for this estimate is (59.8, 
76.1). 

[65] The 95 percent confidence interval for this estimate is (41.3, 
58.6). 

[66] GAO, Program Evaluation: A Variety of Rigorous Methods Can Help 
Identify Effective Interventions, [hyperlink, 
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23, 
2009). 

[67] The expert panel was convened by the National Board for Education 
Sciences in 2008 in response to a mandate from the Senate 
Appropriations Committee. The National Board for Education Sciences, 
consisting of 15 voting members appointed by the President with the 
advice and consent of the Senate, provides IES guidance and oversight. 
The mandate directed the board to convene leading experts in rigorous 
evaluations to assess the WWC and specified that panel members should 
be free of conflicts of interest. Expert panel members included 
economists, statisticians, and professors with expertise in other 
systematic review efforts in the fields of health care, social policy, 
and education. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAOs actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAOs Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: