Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries
Highlights
What GAO Found
The Millennium Challenge Corporation (MCC)--a U.S. government corporation-- recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results. However, it reduced the scopes of its early transportation infrastructure projects and reports mixed success in meeting key performance targets. In addition, problems with data quality call into question the reliability of those reported results. GAO found the following for the seven compacts ending in 2010 and 2011, each with a road project or a port project. Road Projects: (1) MCC reduced kilometers to be paved under six compacts--Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, and Armenia--by a combined 63 percent (from about 1,800 to 600 km) because of increased construction costs and political problems in partner governments. MCC reported meeting reduced targets for five compacts. However, for three compacts, MCC did not consistently account for kilometers completed with funding from third parties. (2) MCC reported meeting revised targets for road roughness--a measure of pavement quality--for five of the compacts. However, reported data have quality issues, including the inconsistent application of measurement methodologies and calculation errors that resulted in overstated results. (3) MCC reported meeting targets for annual average daily traffic--a measure of the volume of traffic using the road--for three of the compacts. However, weaknesses in traffic baseline estimates may have affected the establishment of targets and therefore MCC's ability to measure results. Port Projects: (1) In Cape Verde, MCC funding ($53.7 million) was insufficient to construct all planned port elements. As a result, MCC reduced the project's scope and deferred measuring the results of key indicators. (2) In Benin, MCC completed most of the envisioned scope of the port project. MCC's data show that the compact met the original target for one of three key performance indicators, volume of merchandise. (The other two indicators relate to the measurement of shipping costs.) However, GAO found that MCC's estimation of this indicator's baseline may lead to overstated results. In addition, data quality reviews identified problems with the data used, which MCA-Benin did not formally address. In 2009, MCC improved its methodology for estimating beneficiaries--people who realize income gains or expenditure savings as a result of its investment--by standardizing its approach. MCC subsequently revised its beneficiary numbers for all compacts. However, the new approach did not include a formal quality review process. As a result, implementation of the new approach suffered from varying degrees of quality problems. For example, (1) MCC did not implement its beneficiary estimation methodology consistently across early transportation infrastructure projects; (2) beneficiary calculations contained incorrect formulas and numbers, and differed from supporting documents; and (3) beneficiary figures in MCC's public documents were sometimes inaccurate.
Why GAO Did This Study
To help developing countries reduce poverty and stimulate economic growth, MCC has approved 26 bilateral compact agreements totaling about $9.3 billion. In the seven compacts that ended in 2010 and 2011--Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia, and Benin--transportation infrastructure projects generally received about 50 percent of the compact's total funding. To measure the results of its compacts, MCC sets targets for various performance indicators--such as number of kilometers paved or volume of merchandise passing through a port--and estimates the number of beneficiaries. This report, responding to a congressional mandate, examines the extent to which MCC has, for transportation infrastructure projects, (1) achieved expected performance targets and (2) consistently estimated numbers of beneficiaries. GAO analyzed MCC documents, interviewed MCC officials, and drew on fieldwork completed for related work in four of the seven countries.
Recommendations
MCC should strengthen existing policies and practices regarding measuring and evaluating results data and formalize a quality review process to improve its beneficiary calculations. MCC agreed with all of our recommendations and outlined some steps the agency will take or has already taken to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Millennium Challenge Corporation | To improve the reliability of results measurement, MCC should improve guidance for common indicators by requiring that they are included in data quality reviews, measured uniformly across compacts, and reported in a standardized manner--including when third-party funding is used to complete the original scope of a project. |
Closed – Implemented
The Millennium Challenge Corporation (MCC) has taken steps to improve guidance for common indicators. Regarding data quality review, in September 2016, MCC amended its Policy for Monitoring and Evaluation of Compacts and Threshold Programs (M&E policy) to stipulate that, "At least one [Data Quality Review] is required for each compact and, at a minimum, must include all common indicators from the M&E plan." Regarding uniform measurement, a previous iteration of the M&E policy included language on MCC's right to adjust M&E plans in order to standardize units of measurement or indicator names. In addition, MCC added some guidance in the description of certain indicators on how they should be measured. For example, the policy specifies how to measure the common indicator, Road Roughness. While MCC has not clarified measurement methodology for all indicators, these steps suggest that MCC is working toward uniformly measuring common indicators. Regarding standardized reporting, MCC's revised Guidance on Common Indicators now notes how to measure on report common indicators when third-party funding is introduced. Specifically, the guidance states that for joint projects, MCAs should report only on MCC's contribution to a particular common indicator.
|
| Millennium Challenge Corporation | To improve the reliability of results measurement, MCC should improve guidance for baseline measurements by requiring MCAs to document the date, source, and methodology for establishing the baseline. |
Closed – Not Implemented
While MCC has developed a Policy for Monitoring and Evaluation of Compacts and Threshold Programs, it does not include language specifying the need to document the date, source, and methodology for establishing baseline measurements. In March 2016, MCC said it would update its monitoring and evaluation policy in June 2016. On August 5, 2016, MCC provided us with a draft of their updated policy. However, the updated version did not include language specifying the need to document the date, source, and methodology for establishing baseline measurements.
|
| Millennium Challenge Corporation | To improve the reliability of results measurement, MCC should enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues, discussing how recommendations will be implemented or explaining why changes may not be made. |
Closed – Not Implemented
Although MCC has taken some steps to improve enforcement of the requirement to conduct data quality reviews, it has not provided evidence of written responses. MCC has developed a Policy for Monitoring and Evaluation of Compacts and Threshold Programs, which requires thoroughly documented data quality reviews. In addition, MCC provided several examples of MCC efforts to enforce this requirement. Some examples included final data quality reviews from Senegal, Burkina Faso, and Lesotho. However, MCC did not provide any evidence that the MCAs provided it with written comments discussing how the recommendations would be addressed or explaining why changes would not be made.
|
| Millennium Challenge Corporation | To ensure more accurate beneficiary numbers, MCC should incorporate into the "Guidelines for Economic and Beneficiary Analysis" a formal process for reviewing beneficiary calculations and analysis. |
Closed – Implemented
MCC has updated its Guidelines for Country Team and Peer Review of Cost-Benefit Analysis to include clarification that these guidelines also apply to Beneficiary Analysis (BA). The update specifies that BA is an extension of Economic Rates of Return analysis that seeks to disaggregate the total increase in income to determine specifically which segments of society will benefit from the proposed activities. Although there is no new guidance in the Guidelines for Economic and Beneficiary Analysis, the incorporation of BA in the Guidelines for Country Team and Peer Review of Cost-Benefit Analysis is sufficient to respond to our recommendation.
|