International Education Assistance: USAID Has Implemented Primary Grade Reading Programs but Has Not Yet Measured Progress toward Its Strategic Goal
Highlights
What GAO Found
U.S. Agency for International Development (USAID) missions in all five countries where GAO conducted fieldwork were implementing primary grade reading interventions recommended in USAID's education strategy guidance. A key intervention in four of these countries is developing new reading instruction materials in mother-tongue languages—in some instances, multiple languages. In Ethiopia, for example, USAID's reading program developed new student textbooks and teacher's guides for each of eight grades and seven languages and involved teams of local teachers, language experts, and story writers, among others. The complexity of these interventions in Ethiopia, Malawi, the Philippines, and Uganda increased the time before they could be introduced in classrooms. As a result, some targeted student populations are not expected to benefit from improved reading instruction until late 2015 or 2016, too late to be measured to contribute to USAID's strategic goal of 100 million students with improved reading skills by the end of 2015. In addition, the missions were monitoring their reading programs and planned to conduct program evaluations consistent with USAID guidance.
USAID is currently unable to estimate progress toward the education strategy's reading goal because of four factors. First, because it took some missions longer to implement reading programs than USAID estimated, only about two-thirds of missions are expected to have data to estimate progress toward the goal by the end of 2015. Second, only a small number of missions provided USAID the complete data and supporting information that it needs to aggregate and analyze reading assessment results. Agency officials attributed the lack of complete data and information to an absence of timely guidance. In 2014, USAID issued updated reporting guidance. Third, USAID has not selected a methodology for calculating percentages of assessed children demonstrating improved reading skills, and extrapolating the results to estimate the total numbers of children with improved reading skills. Along with incomplete data and information, other factors contributed to a delay in identifying a methodology—for example, USAID did not ask its contractor to examine the methodology proposed in 2012 and explore alternatives until 2014. Fourth, the education strategy did not set interim targets for assessing progress toward the reading goal as suggested by leading performance management practices. Without a methodology and interim targets, USAID officials and others lack aggregate information about progress toward the goal to assess the current strategy and plan a realistic goal for a future strategy.
Status of Missions' Implementation of Primary Grade Reading Programs 
Why GAO Did This Study
While many developing countries have achieved important gains in primary school enrollment, students' reading skills remain very low. In response, USAID's 2011-2015 Education Strategy set a goal of improved reading skills for 100 million primary grade children by the end of 2015 (Goal 1). The 42 missions with funding for primary grade reading programs were to start programs aligned with the strategy by the end of fiscal year 2012 to allow time to assess results by 2015.
GAO was asked to review USAID's efforts to implement primary grade reading programs—Goal 1 of its education strategy. This report (1) examines five USAID missions' implementation, monitoring, and evaluation of reading programs and (2) assesses USAID's efforts to estimate progress toward its reading goal. GAO analyzed USAID's education strategy and other guidance, reading assessment data and methodologies, performance monitoring plans and reports, and program evaluations, and conducted fieldwork in five countries selected based on geographic diversity, funding for basic education, and availability of reading assessment data.
Recommendations
USAID should select a methodology for estimating total numbers of children with improved reading skills, document a description of the methodology when reporting results, and set interim targets to assess progress toward a reading goal in any future education strategy. USAID concurred with GAO's recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| U.S. Agency for International Development | To help ensure that USAID reports a reliable estimate of the results of its efforts to achieve the goal of improving reading skills of 100 million children, the Acting USAID Administrator should select a methodology for estimating the total numbers of children with improved reading skills as a result of exposure to primary grade reading programs. |
In comments on the draft report, USAID concurred with the recommendation and noted that the aggregation of reading assessment data across countries and languages into a single agency-level indicator posed a complex challenge that required time to vet and test the details of the methodology before formalizing a final approach. In October 2015, USAID issued a report describing the methodology that USAID selected for measuring progress toward the agency's reading goal. The selected methodology ranks assessment results for a sample of children assessed at baseline and a sample assessed at endline and matches results for individuals in each sample to calculate percentages with improved reading skills. For example, assessment results for the child reading the 10th fewest words correctly per minute at baseline are matched with results for the child reading the 10th fewest words correctly at endline. The percentages of children with improved reading skills are calculated on the basis of the numbers of matched pairs whose reading skills improved by a minimum number of words read correctly per minute. The percentage of pairs that meet this definition of improved reading is multiplied by the number of unique beneficiaries over the life of USAID's reading programs to estimate the total numbers of students with improved reading skills. USAID reported that it will apply this methodology to all reading programs initiated during and after the agency's 2011-2015 Education Strategy and the agency will use this methodology to report aggregate progress toward its strategic reading goal annually.
|
| U.S. Agency for International Development | To help ensure that USAID reports a reliable estimate of the results of its efforts to achieve the goal of improving reading skills of 100 million children, the Acting USAID Administrator should document a description of the selected methodology as well as the information necessary to evaluate the estimate (i.e., sampling designs, sample sizes, and modes of data collection) when reporting progress toward the reading goal. |
In written comments on the report, USAID agreed to implement GAO's recommendation. In August 2015, USAID told GAO that it is committed to an open and consultative process to developing and documenting its selected methodology for measuring progress toward the aggregate goal of 100 million primary grade students with improved reading skills. USAID noted that this also involves establishing the capacity to share information on the individual programs that contribute to this goal including their reading assessment data sets. In December 2016, the USAID released the 2011-2015 USAID Education Strategy Progress Report, which documents and reflects on accomplishments achieved under the strategy, including progress towards the strategy?s reading goal. USAID reported that of the 10 million learners reached so far for whom it has measured change in reading outcomes, 1.5 million meet the definition of improved reading. USAID also launched website to serve as a technical appendix to the progress report which, among other things, provides additional information on aggregate beneficiary reports, reading assessment data sets and documentation, including code books. In October 2015, USAID issued a report describing the methodology it would use to produce this estimate.
|
| U.S. Agency for International Development | To improve USAID's ability to measure progress in achieving a quantitative reading goal in any future education strategy, the Acting USAID Administrator should ensure that the future strategy includes targets that will allow USAID to monitor interim progress toward its goal in comparison with planned performance. |
In written comments on the report, USAID agreed to implement GAO's recommendation. USAID noted that as it expands its body of knowledge surrounding achievements of current reading projects, it will be better able to set achievable project and country level targets and report interim progress toward the new strategy's aggregate primary grade reading goal. The current strategy was originally scheduled to end in December 2015, however, USAID extended the USAID Education Strategy 2011-2015 to December 31, 2018, or until a new strategy is approved, whichever is sooner. In May 2018, USAID officials told GAO that USAID had decided not to establish a new education "strategy" with specific time bound goals and uniform quantitative targets. On November 15, 2018, USAID released a new Education Policy which confirms USAID's intention not to establish quantitative targets. With the absence of a quantitative target, there is no need to measure interim progress as called for in GAO's recommendation. Because of this change in circumstances, the recommendation is closed as unimplemented.
|