This is the accessible text file for GAO report number GAO-12-631 entitled 'Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries' which was released on September 12, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Committees: September 2012: Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries: GAO-12-631: GAO Highlights: Highlights of GAO-12-631, a report to congressional committees. Why GAO Did This Study: To help developing countries reduce poverty and stimulate economic growth, MCC has approved 26 bilateral compact agreements totaling about $9.3 billion. In the seven compacts that ended in 2010 and 2011— Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia, and Benin— transportation infrastructure projects generally received about 50 percent of the compact’s total funding. To measure the results of its compacts, MCC sets targets for various performance indicators—-such as number of kilometers paved or volume of merchandise passing through a port—-and estimates the number of beneficiaries. This report, responding to a congressional mandate, examines the extent to which MCC has, for transportation infrastructure projects, (1) achieved expected performance targets and (2) consistently estimated numbers of beneficiaries. GAO analyzed MCC documents, interviewed MCC officials, and drew on fieldwork completed for related work in four of the seven countries. What GAO Found: The Millennium Challenge Corporation (MCC)-—a U.S. government corporation—-recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results. However, it reduced the scopes of its early transportation infrastructure projects and reports mixed success in meeting key performance targets. In addition, problems with data quality call into question the reliability of those reported results. GAO found the following for the seven compacts ending in 2010 and 2011, each with a road project or a port project. Road Projects: * MCC reduced kilometers to be paved under six compacts-—Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, and Armenia-—by a combined 63 percent (from about 1,800 to 600 km) because of increased construction costs and political problems in partner governments. MCC reported meeting reduced targets for five compacts. However, for three compacts, MCC did not consistently account for kilometers completed with funding from third parties. * MCC reported meeting revised targets for road roughness—-a measure of pavement quality-—for five of the compacts. However, reported data have quality issues, including the inconsistent application of measurement methodologies and calculation errors that resulted in overstated results. * MCC reported meeting targets for annual average daily traffic-—a measure of the volume of traffic using the road-—for three of the compacts. However, weaknesses in traffic baseline estimates may have affected the establishment of targets and therefore MCC’s ability to measure results. Port Projects: * In Cape Verde, MCC funding ($53.7 million) was insufficient to construct all planned port elements. As a result, MCC reduced the project’s scope and deferred measuring the results of key indicators. * In Benin, MCC completed most of the envisioned scope of the port project. MCC’s data show that the compact met the original target for one of three key performance indicators, volume of merchandise. (The other two indicators relate to the measurement of shipping costs.) However, GAO found that MCC’s estimation of this indicator’s baseline may lead to overstated results. In addition, data quality reviews identified problems with the data used, which MCA-Benin did not formally address. In 2009, MCC improved its methodology for estimating beneficiaries— people who realize income gains or expenditure savings as a result of its investment—by standardizing its approach. MCC subsequently revised its beneficiary numbers for all compacts. However, the new approach did not include a formal quality review process. As a result, implementation of the new approach suffered from varying degrees of quality problems. For example, (1) MCC did not implement its beneficiary estimation methodology consistently across early transportation infrastructure projects; (2) beneficiary calculations contained incorrect formulas and numbers, and differed from supporting documents; and (3) beneficiary figures in MCC’s public documents were sometimes inaccurate. What GAO Recommends: MCC should strengthen existing policies and practices regarding measuring and evaluating results data and formalize a quality review process to improve its beneficiary calculations. MCC agreed with all of our recommendations and outlined some steps the agency will take or has already taken to address them. View [hyperlink, http://www.gao.gov/products/GAO-12-631]. For more information, contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. [End of section] Contents: Letter: Background: MCC Data Show that Some Key Performance Targets Were Met, but Data Quality Problems Call into Question Reported Results: Data Resulting from Beneficiary Calculations May Not Be Valid: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Compact Timeframes: Appendix III: Roads Data: Appendix IV: Port Data: Appendix V: Comments from the Millennium Challenge Corporation: Appendix VI: GAO Contacts and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Key Performance Results for the Kilometers of Road Completed Indicator, MCC Compacts Ending in 2010 and 2011: Table 2: Scope Revisions of Road Projects for MCC Compacts Ending in 2010 and 2011: Table 3: Key Performance Results for the Road Roughness Indicator, Compacts Ending in 2010 and 2011: Table 4: Key Performance Results for the Average Annual Daily Traffic Indicator, Compacts Ending in 2010 and 2011: Table 5: Data Quality Reviews Performed for Road Project Common Indicators: Table 6: Key Performance Results for MCC Port Project in Cape Verde: Table 7: Key Performance Results for MCC Port Project in Benin: Table 8: Kilometers of Roads Completed for MCC Road Projects in Compacts Ending in 2010 and 2011: Table 9: Road Roughness of MCC Road Projects in Compacts ending in 2010 and 2011, as Measured by the International Roughness Index (IRI): Table 10: Average Annual Daily Traffic on MCC-Funded Roads in Compacts Ending in 2010 and 2011: Table 11: Volume of Merchandise through Port for Projects in MCC Compacts That Ended in 2010 and 2011: Table 12: Container Ship Time at Berth for Port Projects in MCC Compacts That Ended in 2010 and 2011: Table 13: Container Ship Time at Anchor for Port Projects in MCC Compacts That Ended in 2010 and 2011: Figures: Figure 1: Funds for Transportation Infrastructure and Other Projects, by MCC Compact: Figure 2: Relationship between MCC Indicators, Baselines, Targets, and Results: Figure 3: Timeframes for MCC Compacts Ending in 2010 and 2011, by Date of Signature: Abbreviations: IRI: International Roughness Index: MCC: Millennium Challenge Corporation: MCA: Millennium Challenge Account: [End of section] United States Government Accountability Office: Washington, DC 20548: September 12, 2012: The Honorable Patrick Leahy: Chairman: The Honorable Lindsey Graham: Ranking Member: Subcommittee on the Department of State, Foreign Operations, and Related Programs: Committee on Appropriations: United States Senate: The Honorable Kay Granger: Chairwoman: The Honorable Nita Lowey: Ranking Member: Subcommittee on State, Foreign Operations, and Related Programs: Committee on Appropriations: House of Representatives: The Millennium Challenge Corporation (MCC), a U.S. government corporation, was established in 2004 to provide aid to developing countries that have demonstrated a commitment to ruling justly, encouraging economic freedom, and investing in people. MCC provides assistance to eligible countries through multiyear compact agreements to fund specific projects aimed at reducing poverty and stimulating economic growth. MCC's core principles include assessing the results of its funding to determine how its activities affect poverty and economic growth. To do this, MCC has instituted a monitoring and evaluation approach that includes identifying relevant indicators to measure results during and at the end of project implementation and establishing performance targets for each indicator. MCC's efforts to determine how its activities will affect poverty and economic growth also include estimating the number of people who will benefit from its projects and analyzing the impact of its projects on populations of specific interest. As of June 2012, MCC had signed 26 compacts, committing a total of approximately $9.3 billion.[Footnote 1] Seven of these compacts ended in 2010 or 2011: Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia, and Benin.[Footnote 2] (See appendix II for compact time frames.) In the fiscal year 2008 Consolidated Appropriations Act, Congress mandated that we review the results achieved by MCC compacts.[Footnote 3] We have previously reported on the results of MCC's transportation infrastructure projects in four countries. In July 2011, we reported that MCC had met reduced targets for its compacts with Cape Verde and Honduras, the first two compacts to reach completion.[Footnote 4] In June 2012, we reported that MCC transportation infrastructure projects in Georgia and Benin varied in quality and may not be sustainable. [Footnote 5] Because much of MCC's funding for compacts that ended in 2010 and 2011 focused on transportation infrastructure projects, this report examines the extent to which MCC has, for transportation infrastructure projects, (1) achieved expected performance targets and (2) used a consistent methodology to estimate numbers of beneficiaries. To assess the extent to which MCC has achieved its performance targets for transportation infrastructure projects under compacts that ended in 2010 and 2011, we reviewed MCC guidance and policy documents and analyzed compact agreements, monitoring and evaluation plans, and results data. We compared actual results achieved at the end of the compact for select performance indicators with MCC's original targets and, in some cases, with revised targets associated with each indicator. Since MCC tracks many performance indicators for each compact project, we selected a subset of indicators to examine for this report. * For the road projects, we examined kilometers of road completed, roughness as a measure of road quality, and average annual daily traffic. We selected those road project indicators because they address the projects' key objectives and because they are among the "common indicators" MCC requires for all road projects, so that it can aggregate results across countries. * For the port projects, we examined the volume of merchandise through the port and two indicators related to the measurement of shipping costs: container ship time at berth and container ship time at anchor. Because MCC has not established common indicators for ports, we selected indicators that addressed the projects' key objectives, corresponded to the common road indicators, and were originally included in both port projects. Additionally, we interviewed MCC officials about data quality and indicator tracking tables. As part of our related engagements--focused on Cape Verde and Honduras, and on Georgia and Benin[Footnote 6]--we visited compact projects in-country and met with partner country government officials, including those responsible for compact implementation, and with contractors, project managers, construction supervisors, and relevant private businesses. To assess the extent to which MCC has consistently applied its beneficiary estimation methodology, we reviewed MCC's Guidelines for Economic and Beneficiary Analysis as well as its beneficiary calculations and accompanying supporting documents for the seven compacts that ended in 2010 and 2011. We compared original and revised beneficiary estimates for the transportation projects and assessed the justification behind any changes. Data are considered reliable when they are accurate, complete, consistent, and valid, given the uses for which they are intended. MCC enters into a legal relationship with partner country governments that vest an accountable entity with responsibility for day-to-day management of compact project implementation, including monitoring and evaluation activities such as setting and revising performance targets. Because such actions require MCC's direct oversight and approval, throughout this report, we attribute all decisions related to project rescoping and compact targets to MCC. (See appendix I for further details of our objectives, scope, and methodology.) We conducted this performance audit from August 2011 to September 2012, as part of a body of work on this subject, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: MCC Organization: MCC is managed by a chief executive officer, appointed by the President with the advice and consent of the Senate, and is overseen by a Board of Directors. The Secretary of State serves as board chair and the Secretary of the Treasury as vice-chair.[Footnote 7] MCC's model of providing foreign aid is based on a set of core principles deemed essential for effective development assistance, including good governance, country ownership, focus on results, and transparency. According to MCC, country ownership of an MCC compact occurs when a country's national government controls the prioritization process during compact development, is responsible for implementation, and is accountable to its domestic stakeholders for decision making and results. To implement the compact, the partner government establishes an accountable entity, referred to as a Millennium Challenge Account (MCA).[Footnote 8] MCC provides the framework and guidance for compact implementation, monitoring, and evaluation that MCAs are to use in implementing compact projects. Transportation Infrastructure Projects: The seven MCC compacts completed in 2010 and 2011 each included a transportation infrastructure project that--except in the case of Armenia--received 50 percent or more of the compact's total funding (see figure 1). For the purposes of this report, we have defined transportation infrastructure as public works that convey passengers or goods from one place to another; infrastructure includes structures such as roads, seaports, airports, and railways. We reviewed five compacts with road projects, one with a port project, and one with both types of projects. Figure 1: Funds for Transportation Infrastructure and Other Projects, by MCC Compact: [Refer to PDF for image: illustrated stacked horizontal bar graph] Type of project: road; Honduras; Amount at signature: Transportation infrastructure projects: $125.7 million; Other projects: $89.3 million; Total amount at signature: $215.0 million; Amount disbursed: Transportation infrastructure projects: $120.6 million; Other projects: $83.4 million; Total amount disbursed: $204.0 million. Type of project: road and port; Cape Verde; Amount at signature: Transportation infrastructure projects: $78.8 million; Other projects: $31.3 million; Total amount at signature: $110.1 million; Amount disbursed: Transportation infrastructure projects: $82.5 million; Other projects: $26.0 million; Total amount disbursed: $108.5 million. Type of project: road; Nicaragua; Amount at signature: Transportation infrastructure projects: $92.8 million; Other projects: $82.3 million; Total amount at signature: $175.1 million; Amount disbursed: Transportation infrastructure projects: $56.7 million; Other projects: $56.0 million; Total amount disbursed: $112.7 million. Type of project: road; Georgia; Amount at signature: Transportation infrastructure projects: $102.2 million; Other projects: $193.1 million; Total amount at signature: $295.3 million; Amount disbursed: Transportation infrastructure projects: $212.9 million; Other projects: $174.3 million; Total amount disbursed: $387.2 million. Type of project: port; Benin; Amount at signature: Transportation infrastructure projects: $169.4 million; Other projects: $137.9 million; Total amount at signature: $307.3 million; Amount disbursed: Transportation infrastructure projects: $187.9 million; Other projects: $113.9 million; Total amount disbursed: $301.8 million. Type of project: road; Vanuatu; Amount at signature: Transportation infrastructure projects: $60.7 million; Other projects: $5.0 million; Total amount at signature: $65.7 million; Amount disbursed: Transportation infrastructure projects: $60.1 million; Other projects: $5.3 million; Total amount disbursed: $65.4 million. Type of project: road; Armenia; Amount at signature: Transportation infrastructure projects: $67.1 million; Other projects: $168.6 million; Total amount at signature: $235.7 million; Amount disbursed: Transportation infrastructure projects: $8.4 million; Other projects: $168.2 million; Total amount disbursed: $176.6 million. Sources: GAO analysis. Notes: Amount at signature is the amount allocated when MCC and the partner country sign the compact; amount disbursed is the actual amount disbursed by compact closure. Any undisbursed funds are returned to MCC. "Other projects" includes funds disbursed before the compact's entry into force, to facilitate implementation of the compact. [End of figure] MCC Monitoring and Evaluation Framework: According to MCC's Policy for Monitoring and Evaluation of Compacts and Threshold Programs,[Footnote 9] performance monitoring helps track progress toward compact goals and objectives and serves as a management tool. Therefore, MCC requires partner countries to create a monitoring and evaluation plan that they update throughout the course of the compact. The plan's monitoring component outlines the performance indicators by which the compact results will be measured. It also establishes a performance target for each indicator, including a baseline value (the starting value of an indicator) and the expected year by which the target will be achieved.[Footnote 10] For some indicators, baseline measurements are necessary to establish interim and final targets. MCC and the MCAs monitor the progress of compact activities using an indicator tracking table, which is a reporting tool that displays targets and tracks progress against them. (See figure 2.) Figure 2: Relationship between MCC Indicators, Baselines, Targets, and Results: [Refer to PDF for image: illustration] Indicator: An Indicator is a measurement used to track project progress throughout the compact. Baseline: At the beginning of each project, baseline values are measured as a starting point. Yearly target values: MCA sets yearly performance targets for some indicators to assess interim progress. End-of-compact target: An end-of-compact target is established for each indicator. This is how MCC determines whether it met compact goals. Actual results: Compact actual results are the final results achieved. MCC policy requires that MCA contract with independent entities to conduct data quality reviews of compact performance data. Source: GAO analysis. Notes: The indicators shown are for a road project. The terms indicator, baseline values, actual results, performance targets, end- of-compact target, and data quality reviews pertain to all MCC projects. [End of figure] Transportation Infrastructure Indicators: MCC established common indicators in 2009 to aggregate results across countries.[Footnote 11] For road projects, MCC has established several common indicators, including the following:[Footnote 12] * Kilometers of road completed: A measure of the total length of road completed under a compact. This indicator is a cumulative measure of works completed. * Road roughness: A measure of pavement quality that affects ride quality, vehicle delay costs, fuel consumption, and maintenance costs. Roughness is used to determine the reduction in road-user costs and indicates when resurfacing work is needed. New pavement roughness ratings--as measured by the international roughness index (IRI)-- should range from 1.5 to 3.5; the higher the IRI, the rougher the pavement, with a roughness of 0 being perfectly smooth. * Average annual daily traffic: A measure of the volume and type of traffic using a road. MCC has not established common indicators for ports, because ports are a small part of its portfolio (it has funded port projects in only two countries). However, we selected certain key indicators to review because they addressed the projects' key objective of economic growth: [Footnote 13] * Volume of merchandise through the port: A measure of the total volume of exports and imports passing through the port each year. * Container ship time at berth: A measure of the average number of days a container ship spends at berth to load and/or unload its cargo. * Container ship time at anchor: A measure of the average number of hours a container ship spends at anchor waiting for a berth. Independent Data Quality Reviews: MCC requires that MCAs contract an independent entity (such as a local or international firm, research organization, or individual consultant) to review the quality of compact performance data. These reviews determine the utility, objectivity, and integrity of information gathered, identify problems with data quality, and recommend actions to remedy these issues. Reviews should ensure that data used to measure indicators meet the following criteria: validity, reliability, timeliness, precision, and integrity. MCC policy allows each country to determine the frequency and timing of each review as well as the indicators to be evaluated. This requirement was fulfilled for every compact except Nicaragua. MCC policy requires the MCA to comment on the review, including noting the recommendations it will implement. In addition, the MCA should reflect, in a subsequent monitoring and evaluation plan, any indicator changes made in response to data quality reviews. MCC policy also requires the MCA to make publicly available on its website a summary of the final reviews and its comments. MCAs are responsible for ensuring that MCC-approved recommendations are implemented. Beneficiary Assessment: MCC defines beneficiaries as people who realize income gains or expenditure savings as a result of its investment.[Footnote 14] MCC's authorizing legislation requires that compacts contain an identification of the intended beneficiaries, disaggregated by income level, gender, and age, to the maximum extent practicable. MCC notes that "the use of these [beneficiary] analyses and their availability to the general public are trademarks of MCC's commitment to transparency and results-based aid." In addition, MCC's 2009 Guidelines for Economic and Beneficiary Analysis maintain that identification of intended beneficiaries should not be limited to counting the number of people who benefit from compacts but should include a more detailed analysis of the effect of compacts on different subgroups, including the poor, the elderly, women, children, and ethnic subpopulations. MCC has not undertaken such detailed beneficiary analysis for the seven compacts completed in 2010 or 2011. [Footnote 15] MCC Data Show that Some Key Performance Targets Were Met, but Data Quality Problems Call into Question Reported Results: MCC reduced the scope of early road and port projects and reports mixed success in meeting key performance targets. In addition, problems with data quality call those reported results into question. MCC Reduced Targets for Completed Roads by 63 Percent and Had Data Quality Problems: In each of the six compacts with road projects, MCC reduced the number of kilometers to be completed. In addition, the roughness and annual average daily traffic indicators had data quality problems. Most compacts did not examine the three common road indicators in their data quality reviews. In addition, MCC did not consistently account for third-party funding. According to MCC Data, Most Compacts Met Reduced Targets for Kilometers of Roads Completed: MCC reduced by 63 percent the total number of road kilometers to be completed for the six compacts with road projects, from an original combined target of 1,822.8 kilometers to a final combined target of 623.9 kilometers.[Footnote 16] Five of six compacts met their reduced targets for kilometers completed. Honduras was the only compact that did not meet its target (see table 1). Table 1: Key Performance Results for the Kilometers of Road Completed Indicator, MCC Compacts Ending in 2010 and 2011: Honduras[A]; Original target: 200.0; Revised target: 174.5; Percentage change: -12.8; Final result: 115.0; Target met: met original target. Cape Verde; Original target: 63.0; Revised target: 39.3; Percentage change: -37.6; Final result: 40.6; Target met: met revised target. Nicaragua; Original target: 158.0; Revised target: 67.0; Percentage change: -57.6; Final result: 74.0; Target met: met revised target. Georgia; Original target: 245.0; Revised target: 220.2; Percentage change: -10.1; Final result: 220.2[B]; Target met: met revised target. Vanuatu; Original target: 213.8; Revised target: 149.7; Percentage change: -30.0; Final result: 149.7; Target met: met revised target. Armenia; Original target: 943.0; Revised target: 24.4; Percentage change: -97.4; Final result: 24.4; Target met: met revised target. Total; Original target: 1,822.8; Revised target: 675.1; Percentage change: -63.0; Final result: 623.9. Source: GAO analysis of MCC data. Note: See appendix III for additional details on targets and results for the kilometers of road indicator in each country. [A] Kilometers include secondary roads and sections 3 and 4 of the highway in Honduras. Sections 1 and 2 of the highway were to be completed post-compact with additional funding provided by the Central American Bank for Economic Integration; as of March 9, 2012, construction of these segments was not complete. [B] Of this amount, 217 kilometers were fully rehabilitated using MCC funds. For the remaining about 3 kilometers, MCC funded maintenance items, such as road painting. [End of table] In general, MCC decreased the scope of the road projects because of increased construction costs, changes in exchange rates, or actions taken by host country governments that were inconsistent with MCC's eligibility criteria. For example, MCC decreased the scope of the compacts in Nicaragua and Armenia because of concerns about democratic governance in the respective partner governments (see table 2). Table 2: Scope Revisions of Road Projects for MCC Compacts Ending in 2010 and 2011: Compact: Honduras; Scope change: Reduced kilometers of secondary roads completed. Removed two sections of the highway project, to be completed with additional funding from the Central American Bank for Economic Integration post- compact; Reason for scope change: Increased construction costs. Compact: Cape Verde; Scope change: Eliminated two roads; Reason for scope change: Increased construction costs. Compact: Nicaragua; Scope change: In June 2009, MCC terminated funding for the transportation activities not already under contract. At this time, 74 kilometers of road were already under construction; Reason for scope change: Electoral irregularities in the municipal elections of November 2008, which were inconsistent with MCC eligibility criteria.[A]. Compact: Georgia; Scope change: Reduced kilometers of road to 70 percent of the original plan; Reason for scope change: Increased construction costs and changes in exchange rates. Scope change: Subsequently increased kilometers of road to roughly 90 percent of the original plan; Reason for scope change: MCC obligated an additional $100 million to the compact following the end of Georgia's war with Russia over South Ossetia in 2008, $60 million of which went to the road project. Compact: Vanuatu; Scope change: Reduced scope from the rehabilitation of several roads, airstrips and wharfs to two roads; Reason for scope change: Increased construction costs and changes in exchange rates. Scope change: Increased kilometers completed on road in Santo; Reason for scope change: New Zealand provided additional funding to support MCC road project in June 2009. Compact: Armenia; Scope change: Reduced kilometers of road to roughly 32 percent of the original plan prior to June 2008; Reason for scope change: Increased construction costs and changes in exchange rates. Scope change: In June 2009, MCC put an indefinite hold on funding for road construction that was not already underway. At this time, construction of 24.4 kilometers had been completed; Reason for scope change: Concerns about the status of democratic governance in Armenia, which were inconsistent with MCC's eligibility criteria[A]. Source: GAO analysis of MCC documents. [A] The country also failed to meet MCC's corruption indicator, which measures the extent to which public power is exercised for private gain. [End of table] MCC-Reported Results for Road Roughness Have Data Quality Issues: MCC data show that five of six compacts met original targets for road roughness. In addition, MCC reports that the Georgia and Armenia compacts surpassed their original roughness targets and built a smoother road than originally planned.[Footnote 17] Honduras did not did not meet its roughness target. (See table 3.) Table 3: Key Performance Results for the Road Roughness Indicator, Compacts Ending in 2010 and 2011: Honduras[A]; Original target: 2.2; Revised target: [Empty]; Final result: 3.0; Target met: Did not meet original or revised target. Cape Verde; Original target: 2.3; Revised target: [Empty]; Final result: 2.3; Target met: Met original target. Nicaragua; Original target: 3.1; Revised target: [Empty]; Final result: 2.5; Target met: Met original target. Georgia; Original target: 3.2; Revised target: 2.5; Final result: 1.5; Target met: Met original target; Met revised target. Vanuatu; Original target: 3.5; Revised target: [Empty]; Final result: 3.0; Target met: Met original target. Armenia; Original target: 5.0; Revised target: 4.0; Final result: 3.5; Target met: Met original target; Met revised target. Source: GAO analysis of MCC data. Notes: A higher roughness number indicates a rougher road, while zero is a perfectly smooth road. A missing value for the revised target indicates that the original target was not revised. See appendix III for additional details on targets and results for the roughness indicator in each country. [A] The roughness measure includes secondary roads and sections 3 and 4 of the highway in Honduras. Sections 1 and 2 of the highway were to be completed post-compact with additional funding provided by the Central American Bank for Economic Integration; as of March 9, 2012, construction of these segments was not complete. [End of table] We identified problems with the measurement of road roughness for some of the six compacts with road projects, including the use of inconsistent application of methodologies and calculation errors. These problems call into question MCC's reported roughness results. Inconsistent application of methodologies. MCC determined roughness using a visual scale for some roads and mechanical measurements for others. While both methods are legitimate means of measuring roughness, they have different degrees of accuracy. Moreover, MCC did not apply either measurement method consistently across compacts. First, for the two countries measuring roughness visually, Cape Verde and Vanuatu, MCC inappropriately provided the MCAs a World Bank scale for unpaved roads to assess the paved roads. Second, for compacts where roughness was measured mechanically, the MCAs used different machines that had varying levels of accuracy. Third, for three compacts--Nicaragua, Honduras, and Armenia--the MCAs measured roughness on both sides of the road and then averaged these measurements to calculate a final measure. In contrast, MCA-Georgia calculated roughness using measurements taken on one side of the road only, potentially producing a less accurate measurement since roughness can vary from one side of the road to the other.[Footnote 18] Calculation errors. We found that errors calculating roughness led to an overstatement of results. For example, MCA-Nicaragua recorded zeros--an IRI rating for a perfectly smooth road--for sections that could not be measured, incorrectly lowering the overall roughness calculation from 4.8 to 3.4 for one road and indicating that the road was smoother than it actually was.[Footnote 19] Additionally, MCA- Honduras did not account for the differing lengths of each segment of secondary road when aggregating the roughness values and therefore overstated its reported results at 3.2 instead of 3.3. Although this difference is not large, consistent methodologies should be used across compacts to allow a valid basis for comparing and evaluating results. MCC-Reported Results for Road Traffic Have Data Quality Issues: MCC data show that three of the six compacts with road projects met original or revised targets for average annual daily traffic.[Footnote 20] The Nicaragua and Armenia compacts met their original targets and the Vanuatu compact met a reduced target. For the other three compacts, Cape Verde did not meet its original target; Georgia did not meet its reduced target; and MCC stated that a final traffic measure in Honduras is not expected until late 2012. (See table 4.) Table 4: Key Performance Results for the Average Annual Daily Traffic Indicator, Compacts Ending in 2010 and 2011: Honduras[A]; Original target: 3,772; Revised target: [Empty]; Final result: N/A; Target met: N/A. Cape Verde[B]; Original target: 559; Revised target: [Empty]; Final result: 295; Target met: Did not meet original or revised target. Nicaragua; Original target: 932; Revised target: [Empty]; Final result: 1,216; Target met: Met original target. Georgia; Original target: 6,700; Revised target: 1,183; Final result: 1,092; Target met: Did not meet original or revised target. Vanuatu; Original target: 969; Revised target: 196; Final result: 334; Target met: Met revised target. Armenia; Original target: 460; Revised target: 706; Final result: 735; Target met: Met original target; Met revised target. Legend: N/A: Not available. Source: GAO analysis of MCC data. Notes: A missing value for the revised target indicates that the original target was not revised. See appendix III for additional details on targets and results for the traffic indicator in each country. [A] The calculation for original and revised average annual daily traffic measures include the secondary roads and sections 3 and 4 of the highway in Honduras. Sections 1 and 2 of the highway were to be completed post-compact with additional funding provided by the Central American Bank for Economic Integration; as of March 9, 2012, construction of these segments was not complete. In addition, MCC officials stated that a final traffic measure in Honduras is not expected until late 2012. [B] The final result for Cape Verde is not annualized. [End of table] MCC did not consistently update the baseline estimates for the annual average daily traffic indicator either by projecting growth or by using new information when it became available. Because MCC establishes the targets for some indicators by applying estimated growth rates to the baselines, these identified weaknesses may have affected MCC's ability to create accurate traffic targets, thus affecting its reported results. * For the secondary roads project in Nicaragua, MCC stated that it updated the baseline estimates established in 2007 by applying 1 year's growth rate to the traffic measures, because construction was expected to begin in fall 2008. However, MCC did not use this method in Cape Verde, where there was a similar time lag.[Footnote 21] * For the section of highway to be rehabilitated in Nicaragua, MCC retained a baseline calculated in a 2000 traffic study[Footnote 22] even though it had 2007 data for the specific 18 kilometers of road it planned to rehabilitate. * In Honduras, MCC used a baseline established before 2005 despite the availability of data from a 2008 traffic survey. The 2008 traffic survey results showed that traffic volumes already met or exceeded the end-of-compact targets for most road segments. Because MCC used the 2005 data, the final results for the project indicate a larger change over the compact implementation timeframe than actually occurred. Common Indicators for Road Projects Not Consistently Reviewed for Quality: The data quality reviews conducted for most compacts did not include the kilometers-paved, roughness, or average annual daily traffic indicators. MCC requires that MCAs contract an independent entity to review the quality of compact performance data. According to MCC, this requirement was fulfilled for every compact except Nicaragua. However, only the data quality reviews for Georgia included all three indicators. (The reviews did not identify any data problems with these indicators).[Footnote 23] Although the data quality reviews for Armenia included two of the three indicators, the compact was put on hold before MCA-Armenia could fully correct identified problems. In Vanuatu, data quality reviews conducted in 2008 and 2009 identified problems with the measurement of annual average daily traffic. [Footnote 24] MCA-Vanuatu did not provide a response to either of the recommendations in the review and made no changes to the definition of the traffic indicator in the final monitoring and evaluation plan. The data quality reviews for the Honduras and Cape Verde compacts did not include any of the three road indicators. Table 5 summarizes the treatment of the three common road indicators we examined in data quality reviews for the six compacts. Table 5: Data Quality Reviews Performed for Road Project Common Indicators: Honduras; Kilometers of road completed: Indicator not reviewed; International roughness index: Indicator not reviewed; Average annual daily traffic: Indicator not reviewed; Recommendations implemented: N/A. Cape Verde; Kilometers of road completed: Indicator not reviewed; International roughness index: Indicator not reviewed; Average annual daily traffic: Indicator not reviewed; Recommendations implemented: N/A. Nicaragua[A]; Kilometers of road completed: No data quality review performed; International roughness index: No data quality review performed; Average annual daily traffic: No data quality review performed; Recommendations implemented: N/A. Georgia; Kilometers of road completed: Indicator reviewed without recommendation; International roughness index: Indicator reviewed without recommendation; Average annual daily traffic: Indicator reviewed without recommendation; Recommendations implemented: N/A. Vanuatu; Kilometers of road completed: Indicator not reviewed; International roughness index: Indicator not reviewed; Average annual daily traffic: Indicator reviewed with recommendation; Recommendations implemented: No. Armenia; Kilometers of road completed: Indicator not reviewed; International roughness index: Indicator reviewed with recommendation; Average annual daily traffic: Indicator reviewed with recommendation; Recommendations implemented: No[B]. [End of table] Legend: N/A: Not applicable. [A] MCC officials stated that no data quality reviews were performed in Nicaragua. [B] Recommendations were not implemented in Armenia, because the road project was put on hold in June 2009. Source: GAO analysis of MCA data quality reviews. MCC Did Not Consistently Account for Third-Party Funding: The governments of three countries--Honduras, Cape Verde, and Vanuatu-- secured funding from a third party to rehabilitate some of the road lengths that MCC removed from the scope of the compact.[Footnote 25] However, we found that MCC did not consistently account for this third- party funding when reporting the final results of kilometers completed. As a result, MCC understated its results in Honduras and overstated them in Vanuatu. * The government of Honduras secured money from the Central American Bank for Economic Integration to complete 57.6 kilometers of the highway project, which MCC removed from the compact's original scope. [Footnote 26] However, MCA-Honduras did not revise its kilometers- completed target to reflect this reduction in scope. Therefore, MCC reports that it did not meet the compact's final kilometers-completed target, even though data suggest that the kilometers MCC funded were complete at the end of the compact: * The government of Cape Verde secured funding from the government of Portugal to rehabilitate 22.9 kilometers of secondary roads that MCC removed from the scope of its compact. MCA-Cape Verde revised its kilometers-completed target to reflect this reduction in scope and did not report these 22.9 kilometers as a part of its final results. Therefore, MCC reports that it met the compact's revised kilometers- completed target. * The government of Vanuatu secured money from New Zealand to assist in rehabilitating a 57.2 kilometer road, because the MCC compact funds were not sufficient to complete all 57.2 kilometers. However, MCC reported all 57.2 kilometers as a part of its final results, even though it did not fund all of the work. MCC Reduced the Cape Verde Port Project Scope and Did Not Meet Many Targets for the Benin Port Project: For the Cape Verde compact, MCC reduced the scope of the port project, deferred the measurement of key indicators, and therefore did not conduct any data quality reviews of those indicators. In Benin, MCC completed most of the originally envisioned port project scope, and its data show that the compact met the original target for one of the three key indicators. However, we identified issues with the indicators' baseline estimates. In addition, data quality reviews identified problems with the data behind the indicators, which MCA- Benin did not formally address. In Cape Verde, MCC Reduced the Port Project's Scope and Deferred the Measurement of Key Indicators: MCC split the Cape Verde project into two phases in May 2008 because, owing to increased construction costs, the $53.7 million allocated to the port projects was insufficient to fund all of the planned improvements. MCC funded phase 1 of the improvements, including construction of a cargo storage area and an access road, and rehabilitation of Wharf 2. The government of Cape Verde secured a loan from the government of Portugal for $87 million to complete phase 2, that is, those elements removed from the scope of the MCC project.[Footnote 27] The MCC-funded portion of the construction (phase 1) represents about one-third of the total expected cost of both phases. Nearly 100 percent of the works for phase 1 were completed by compact end. Phase 2 was awarded late in the compact, and work was under way as of July 2012. MCC deferred the measurement of key performance indicators when it revised the project scope, because results for key indicators could be measured only after full completion of the port (see table 6). MCC intends to measure results for the key indicators when phase 2 is complete. Table 6: Key Performance Results for MCC Port Project in Cape Verde: Indicator: Volume of merchandise (thousand tons/year); Original target: 710.5; Revised target: Final result: Target met: Performance indicators were eliminated once the Cape Verde project was descoped. Indicator: Container ship time at berth (days); Original target: 1.0; Revised target: Final result: Target met: Performance indicators were eliminated once the Cape Verde project was descoped. Indicator: Container ship time at anchor (hours); Original target: 4.0; Revised target: Final result: Target met: Performance indicators were eliminated once the Cape Verde project was descoped. Source: GAO analysis of MCC data. Note: See appendix IV for additional details on targets and results for key indicators in Cape Verde. [End of table] MCA-Cape Verde contracted for data quality reviews in 2007 and 2010. The port project was reviewed in 2010; however, key indicators were not evaluated, because MCA-Cape Verde deferred measurement of those indicators until phase 2 was complete. MCC's Reported Results for the Benin Port Project Have Data Quality Issues: MCC data show that Benin maintained most of the original project scope and met the original target for one of three key indicators.[Footnote 28] The project scope included constructing or rehabilitating a jetty, a new wharf, and port roads and installing lighting, security, and an electricity distribution system. MCC reports that the Benin compact met the original target for volume of merchandise through the port, but it did not meet targets for the other two key indicators, container ship waiting time at berth and container ship waiting time at anchor. See table 7 for a summary of key performance results for the Benin port project. Table 7: Key Performance Results for MCC Port Project in Benin: Indicator: Volume of merchandise (thousand tons/year); Original target: 6,944.6; Revised target: [Empty]; Final result: 7,605.9; Target met: Met original target. Indicator: Container ship time at berth (days); Original target: 1.0; Revised target: [Empty]; Final result: 1.3; Target met: Did not meet original or revised target. Indicator: Container ship time at anchor (hours); Original target: 4.0; Revised target: [Empty]; Final result: 34.6; Target met: Did not meet original or revised target. Source: GAO analysis of MCC data. Note: A missing value for the revised target indicates that the original target was not revised. See appendix IV for additional details on targets and results for key indicators in Benin. [End of table] We identified problems with MCC's measurement of the volume-of- merchandise indicators, and MCA-Benin's data quality reviews identified additional problems with all three key indicators. For the volume-of-merchandise indicator in Benin, MCC used 2004 data from the Port Authority to establish a baseline of 4.1 million metric tons in the compact's monitoring and evaluation plans and indicator tracking tables. MCA-Benin did not subsequently update the baseline. However, the compact did not enter into force until October 2006 and port improvement construction did not begin until August 2009. Available annual data for Benin showed that in 2005 the volume of merchandise had already increased to 5.2 million metric tons. Without an updated baseline, the estimate may not accurately reflect preconstruction conditions and may lead MCC to overstate the degree to which port traffic increased because of the compact. MCA-Benin contracted data quality reviews in 2008, 2009, and 2011. The 2008 and 2009 data quality reviews identified issues with the three key port indicators that we reviewed, but MCC did not address all of the issues.[Footnote 29] The 2011 review did not examine these specific port indicators. * For the volume-of-merchandise indicator, MCA-contractors noted that although data collection methods were consistent and the data were valid, the Port Authority of Cotonou needed to improve the timeliness of data related to this indicator because the data entry and control procedures created delays in the availability of information. MCC policy indicates that data should be sufficiently current to inform management decisions because decisions depend on regular collection of performance information. In 2008 and 2009, the contractors recommended that MCA-Benin set up an automatic data transfer system for information collected on ship stopover, to avoid redundant data entry efforts by the port's statistical services. Although the 2009 review indicated that the port was resolving problems through the creation of a port information system when the evaluation was conducted, we found that MCA-Benin did not resolve the issues identified for this indicator. No changes were made in the final monitoring and evaluation plan to indicate that adjustments were made in the data collection methods for this indicator. In addition, MCA-Benin responses to the review did not address this issue. * For the container ship time at berth and container ship time at anchor indicators, the 2008 and 2009 reviews noted that, while the data collected were a direct measure of what the indicators were intended to capture and the data collection procedures were consistent, there were no historical data to support the baseline values for these indicators. The reviews recommended using baseline values established in a 2005 report by an international consulting firm. We found that MCA-Benin did not resolve issues identified for these two indicators. MCC and MCA officials stated that they decided not to revise indicator baselines and targets as recommended, and no revisions were made to baseline figures in subsequent monitoring and evaluation plans. MCC and MCA officials provided no additional explanation. Data Resulting from Beneficiary Calculations May Not Be Valid: In 2009, MCC improved its beneficiary estimates by adopting a more standardized approach, which it documented in its Guidelines for Economic and Beneficiary Analysis; however, the updated guidelines do not include a formal quality review process. MCC subsequently revised its beneficiary estimates for ongoing compacts, but the implementation of the new approach was not always consistent and suffered from varying degrees of quality control problems. Although some of the individual problems we identified were small in nature, taken as a whole they reduce confidence in MCC's estimates. MCC's Updated Beneficiary Analysis Guidelines Do Not Include a Formal Quality Review Process: In our previous work, we found that MCC did not have a consistent methodology for estimating the number of beneficiaries across compacts.[Footnote 30] As a result, MCC adopted a new, more standardized approach, which it published in its 2009 Guidelines for Economic and Beneficiary Analysis. MCC subsequently conducted a "beneficiary scrub," revising beneficiary numbers for ongoing compacts based on the 2009 guidelines. However, we found errors and inconsistencies in the calculations. MCC stated that these scrubs are informally reviewed but that they do not undergo a formal quality review process.[Footnote 31] Such a process would ensure that the information and data provided have been verified for consistency and accuracy. Weaknesses Call into Question the Validity and Reliability of MCC's Beneficiary Data: We identified three general weaknesses with MCC's beneficiary calculations. First, the beneficiary calculations in MCC's internal documents contain weaknesses such as incorrect formulas and numbers as well as discrepancies with supporting documents. Second, MCC did not apply a consistent methodology in estimating beneficiaries for its early transportation infrastructure projects. Third, the beneficiary figures in MCC's public documents are sometimes inaccurate. MCC's Beneficiary Calculations Contained Mistakes: Some beneficiary calculations contained mistakes in the formulas used. For example, the computations of the population growth rate for Georgia and Cape Verde contained an error in the mathematical formula, causing MCC to slightly overstate the growth rate in the first case and understate it in the second case. In addition, erroneous numbers were used in four of the seven compacts we reviewed. * For the Georgia road project, MCC stated in the beneficiary calculation documents that "the original beneficiary estimate of 53,988 people counts all households living in the four rayons, or districts, through which the road passes…The revised beneficiary estimate [of 58,079] is based on a 5 km catchment area of the Road." MCC thus originally claimed a lower number of beneficiaries for a larger geographic area (the districts) than the more limited catchment area--the geographic area in which benefits may be expected to accrue-- of 5 kilometers on either side of the road.[Footnote 32] * For the Benin port project, the population projection in the beneficiary scrub began with a 2005 original baseline figure of 8,490,000, since the port project is expected to raise the incomes of the entire population of Benin. However, this number could not be derived from its purported source, the 2002 population census. [Footnote 33] * For the Armenia road project, the document supporting the beneficiary scrub calculation considered a catchment area for an erroneous revised road length of 68 kilometers rather than the correct figure of 24.4 kilometers. * In the Cape Verde roads and bridges activity, the original beneficiary numbers provided in the document supporting the beneficiary scrub spreadsheet did not match the figures in the spreadsheet. MCC's Methodology to Estimate Beneficiaries Is Not Consistently Applied across Compacts: In the 2009 revision of its Guidelines for Economic and Beneficiary Analysis, MCC standardized its methodology to estimate beneficiaries and accordingly modified its beneficiary numbers in the spreadsheets for its beneficiary scrubs. However, the amount of detail in the analysis varied across compacts, and population growth rates were not always calculated in a consistent manner. Of the beneficiary scrubs we reviewed, four of seven contained disaggregated data at the project level. Specifically, the Armenia, Georgia, Nicaragua, and Vanuatu scrubs are made up of multiple, relatively detailed spreadsheets. By contrast, one beneficiary scrub, for Honduras, contained one page of macro-level summary data with no explanation or sources for the figures. The remaining two of the seven scrubs contained additional data but were not as detailed as the first four. For example, the Cape Verde scrub included a second sheet of World Bank development indicators on the country, and the Benin scrub had some details on one of the compact projects. These varying levels of detail suggest that the scrubs were executed in an ad hoc, rather than consistent, manner. MCC uses population growth rates to estimate the number of beneficiaries in a 20-year projection period; for each compact, MCC publicly presents the number of beneficiaries projected for the end of the 20-year period.[Footnote 34] However, the scrub sheets we reviewed showed that MCC sometimes computed population growth rates differently across compacts, even when the scrubs provided the same underlying information. * For Honduras, MCC indicated that it took the population growth rates for 2005, 2006, and 2007 from World Bank data and averaged the three rates to get an average annual growth rate for the projection. [Footnote 35] * For Georgia and Cape Verde, MCC did not use the historical growth rates provided but instead took total population figures in 2005 and 2007 and used them to compute an average annual growth rate for the beneficiary projection. * For Vanuatu, MCC computed an average annual growth rate based on four historical growth rates without specifying a source or year. For Some Countries, MCC Did Not Update Beneficiary Numbers or Reported Them Incorrectly: In three of seven compacts, we found that some numbers in MCC's public documents had not been updated or were inaccurate. * In the case of the Georgia road, a December 2009 report and a July 2011 close-out document used original beneficiary estimates (53,988), instead of using the revised estimate (58,079) from July 2009, when MCC reported having recalculated the beneficiary numbers. In addition, in the 2011 status report, the total number of beneficiaries for the compact as a whole (143,000) was smaller than the beneficiaries of the infrastructure project alone (300,000). * For the Benin port project, the beneficiary estimate that appeared in a 2011 status report (8,791,832) was an original estimate from 2009 instead of the revised estimate of 16,119,058. In addition, the supporting document accompanying the 2009 scrub spreadsheet mistakenly reported 13,421,086 beneficiaries for the port project by 2026. However, the data in the scrub indicated that this figure referred to the number of beneficiaries in 2020; the correct number of projected beneficiaries for 2026 is 16,119,058. * For Armenia, the beneficiary estimate for the road project in a 2011 publication (6,216) did not reflect a revision of the estimate done in 2009 (6,356). Conclusions: MCC recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results to make well-informed decisions about U.S. investments. It has established an extensive monitoring and evaluation program that includes guidance and tools for the partner countries and controls on data quality. In addition, MCC has created and implemented a revised beneficiary calculation methodology to ensure standardization across projects and compacts. Even with these steps, problems exist with the data behind several of MCC's key performance indicators, limiting the reliability of MCC's reported results. * MCAs have not measured the results of common indicators in a uniform manner--including how they account for additional third-party funding-- nor have they consistently reviewed the quality of data used to report the indicators' results. As such, MCC cannot confidently compare or aggregate the results of these common indicators as intended. * MCC requires the establishment of baselines to measure the impact its assistance has on partner countries. However, because some incorrect baselines were used, results may be over-or understated. * MCAs have not consistently fulfilled the MCC requirements that they conduct independent data quality reviews and respond to issues the reviews raised. As a result, MCC and its partners have missed opportunities to improve data quality and facilitate data reporting and aggregation. In addition, MCC's beneficiary calculations have problems that limit their usefulness. Because MCC's new approach to estimating beneficiary numbers does not include a formal quality review process, the beneficiary estimates contain incorrect numbers and formulas and inconsistent calculations. While the impact of these problems is generally small, the lack of a robust, formal quality review process nonetheless calls into question the reliability of MCC's estimates. Without a correct representation of the compacts' results and beneficiaries, MCC, Congress, and other key stakeholders cannot accurately evaluate the extent to which MCC is achieving its goals of poverty reduction and economic growth. Recommendations for Executive Action: We recommend that MCC's Chief Executive Officer take the following four actions. To improve the reliability of results measurement, MCC should: * improve guidance for common indicators by requiring that they are included in data quality reviews, measured uniformly across compacts, and reported in a standardized manner--including when third-party funding is used to complete the original scope of a project; * improve guidance for baseline measurements by requiring MCAs to document the date, source, and methodology for establishing the baseline; and: * enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues, discussing how recommendations will be implemented or explaining why changes may not be made. To ensure more accurate beneficiary numbers, MCC should: * incorporate into the Guidelines for Economic and Beneficiary Analysis a formal process for reviewing beneficiary calculations and analysis. Agency Comments and Our Evaluation: In written comments on a draft of this report, MCC stated that it agrees with our four recommendations and outlined steps it will take or has taken to address them. * With respect to our first recommendation--to improve guidance for common indicators--MCC stated that it has taken preliminary actions to address it. We reviewed MCC's May 2012 Guidance on Common Indicators and found that it defines the indicators and specifies each indicator's unit of measurement and how each should be disaggregated. In addition, when MCC activities are conducted jointly with other organizations (e.g. when the activities include third-party funding), MCC's guidance instructs MCAs to report only MCC's contribution to a particular common indicator. In addition, the guidance states that MCC will be explicit in reporting which changes in outcome indicators are more likely the result of MCC investments and which changes might also be influenced by other interventions. However, the guidance lacks enough specificity to address other weaknesses we identified. For example, when discussing road roughness, the guidance does not instruct MCAs to measure both sides of the road to account for variation in roughness from one side of the road to the other. In addition, this guidance does not specify that MCAs should include common indicators in their data quality reviews. * With respect to our second recommendation--to improve guidance for baseline measurements--MCC stated that its current monitoring and evaluation policy requires that compacts' monitoring and evaluation plans document the date, source, and methodology for establishing baseline measurements. MCC further stated that current MCA monitoring and evaluation plans are compliant with this requirement. * With respect to our third recommendation--to enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues--MCC stated that it works with MCAs to ensure that recommendations of data quality reviews are incorporated into monitoring and evaluation plans and then implemented. * With respect to our fourth recommendation--to incorporate a formal process for reviewing beneficiary calculations and analysis--MCC noted that it will institute a formal process for reviewing beneficiary calculations and address the weaknesses we identified in this report. MCC asserted that the results of its compact investments are not diminished by the data quality challenges highlighted in this report. MCC noted, for example, that more than 623 kilometers of rehabilitated roads now serve poor and rural households in the countries we reviewed, linking farmers to markets and bolstering important regional trade routes. Regarding our finding that the total target for kilometers of road completed for the six compacts was reduced by 63 percent, MCC also noted that the final target for Honduras, Georgia, Cape Verde, and Vanuatu would be only 19 percent below the original target if we were to exclude the road projects that MCC terminated or discontinued as a result of the Armenian and Nicaraguan governments' undemocratic behavior. We have reprinted MCC's comments in appendix V. We have also incorporated technical comments from MCC in our report where appropriate. We are sending copies of this report to interested congressional committees and the Millennium Challenge Corporation. In addition, this report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff has any questions about this report, please contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Signed by: David Gootnick: Director: International Affairs and Trade: [End of section] Appendix I: Objectives, Scope, and Methodology: The fiscal year 2008 Consolidated Appropriations Act, Public Law 110- 161, mandated that GAO review the results of the Millennium Challenge Corporation's (MCC) compacts. This report examines the extent to which MCC has, for transportation infrastructure projects in compacts ending in 2010 and 2011, (1) achieved expected performance targets and (2) used a consistent methodology in estimating numbers of beneficiaries. The seven countries with compacts relevant to our scope are Honduras, Cape Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin. MCC enters into a legal relationship with partner country governments, which vests responsibility for day-to-day management of compact project implementation with an accountable entity--a Millennium Challenge Account (MCA)--including monitoring and evaluation activities such as setting and revising targets. Because such MCA actions require MCC's direct oversight and approval, throughout this report we attribute all decisions related to project rescoping and compact targets to MCC. To assess the extent to which MCC has achieved its performance targets for transportation infrastructure projects in compacts ending in 2010 and 2011, we reviewed MCC guidance and policy documents and analyzed compacts, monitoring and evaluation plans, and indicator tracking tables for the compacts with Honduras, Cape Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin. In assessing and reporting MCC's results, we compared actual results achieved at the end of the compact for select performance indicators with the original and, in some cases, revised targets associated with each indicator. We considered the original target to be the one first documented for each performance indicator and the final target to be the one last documented in MCC monitoring documents. Given that MCC tracks several performance indicators for each compact project, we selected a subset of indicators to examine for this report. For the roads projects, we examined kilometers of road completed, road roughness, and average annual daily traffic. We selected these road project indicators because (1) they address the projects' key objectives and (2) they are among the "common indicators" that MCC requires all road projects to measure so that it can aggregate results across countries. For the port projects, we examined volume of merchandise through the port, container ship time at berth, and container ship time at anchor. Although MCC has not established common indicators for the port projects, we selected these port project indicators because they (1) addressed the projects' key objectives, (2) corresponded to the common indicators established for the roads projects, and (3) were originally included in both port projects. To report the average value of targets and results for compact road projects, we calculated a weighted average using the length of each road segment. We performed this calculation for the average annual daily traffic and roughness indicators for Cape Verde, Honduras, Vanuatu, and Nicaragua because these MCAs report targets and results by road segment. For Honduras, we did not weight the indicators using values from sections 1 and 2 because the government of Honduras completed these sections with funding from the Central American Bank for Economic Integration. For Cape Verde, we did not include the average annual daily traffic and roughness for bridge projects, because the kilometer lengths of these sections were not available. To determine the amount of funding used for transportation infrastructure projects, we reviewed MCC financial data. We included compact implementation funding--funds disbursed before entry into force to facilitate the implementation of the compact--with other projects not related to transportation infrastructure. To determine which MCAs contracted for independent data quality reviews, which transportation infrastructure indicators were evaluated, and whether recommendations were made for common and key indicators, we reviewed third-party data quality review reports. To determine whether recommendations were implemented, we reviewed MCA comments on the data quality reviews that MCC provided to us and we reviewed final monitoring and evaluation plans. To assess the extent to which MCC has consistently applied its beneficiary estimation methodology, we reviewed MCC's beneficiary calculations and accompanying supporting documents for the seven compacts that ended in 2010 and 2011. In particular, we examined for consistency and accuracy the data and formulas on projected beneficiaries for road and port projects in the Excel spreadsheets that MCC used to update--or "scrub"--the beneficiary estimates. We reviewed the documents explaining the calculations. We compared original beneficiary figures for these transportation projects with the revised ones and assessed the justification behind the changes. We also examined quarterly status reports and the monitoring and evaluation plans for each compact to check the accuracy of the publicly reported beneficiary counts. We consulted with MCC officials when we found discrepancies and errors in the data. As part of related GAO engagements focusing on Cape Verde and Honduras [Footnote 36] and on Georgia and Benin,[Footnote 37] we visited compact projects in-country and met with MCA officials, partner country government officials, contractors, project managers, construction supervisors, and relevant private businesses. Data are considered reliable when they are accurate, complete, consistent and valid, given the uses for which they are intended. To assess the reliability of MCC indicator and beneficiary data, we (1) reviewed MCC policy for the monitoring and evaluation of compacts and for calculating beneficiary numbers, (2) interviewed MCC and MCA officials regarding the reliability and validity of the data, (3) reviewed independent data quality reviews required by MCC, (4) compared performance indicators across original and final monitoring and evaluation plans and indicator tracking tables to identify inconsistencies, and (5) reviewed beneficiary calculations for accuracy. We identified some weaknesses in the data that may affect the data's reliability, as discussed in this report. Finally, some of the reports and documents referenced above were written in French and Spanish. We translated these documents as needed, creating English summaries to enable our analysis. We conducted this performance audit from August 2011 to September 2012 in accordance with generally accepted government auditing standards. [Footnote 38] Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Compact Timeframes: The Millennium Challenge Corporation (MCC) completed seven compacts in 2010 and 2011--Honduras, Cape Verde, Nicaragua, Georgia, Benin, Vanuatu, and Armenia (see figure 3). MCC commits funding when it signs a compact, but MCC does not obligate funding until the compact enters into force. MCC funding that the partner country does not spend for compact activities by the end of the 5-year implementation timeframe must be deobligated.[Footnote 39] Figure 3: Timeframes for MCC Compacts Ending in 2010 and 2011, by Date of Signature: [Refer to PDF for image: illustrated table] Honduras; Compact signature: 6/13/2550; Preparation for entry-into-force: 6-13-05-9/29/05; Entry-into-force: 9/29/2005; 5-year compact implementation timeframe: 9/29/05-9/28/10; End of compact: 9/28/2010. Cape Verde; Compact signature: 9/07/2005; Preparation for entry-into-force: 7/04/05-10/17/05; Entry-into-force: 10/17/2005; 5-year compact implementation timeframe: 10/17/05-10/16/2010; End of compact: 10/16/2010. Nicaragua; Compact signature: 7/14/2005; Preparation for entry-into-force: 7/14/05-5/26/06; Entry-into-force: 5/26/2006; 5-year compact implementation timeframe: 5/26/06-5/25/2011; End of compact: 5/25/2001. Georgia; Compact signature: 9/12/2005; Preparation for entry-into-force: 9/12/05-4/7/06; Entry-into-force: 4/7/2006; 5-year compact implementation timeframe: 4/7/06-4/6/2011. End of compact: Benin; Compact signature: 2/22/2006; Preparation for entry-into-force: 2/22/06-10/6/06; Entry-into-force: 10/6/2006; 5-year compact implementation timeframe: 10/6/06-10/5/11; End of compact: 10/5/2011. Vanuatu; Compact signature: 3/2/2006; Preparation for entry-into-force: 3/2/06-4/28/06; Entry-into-force: 4/28/2006; 5-year compact implementation timeframe: 4/28/06-4/27/11; End of compact: 4/27/2011. Armenia; Compact signature: 3/27/2006; Preparation for entry-into-force: 3/27/06-9/29/06; Entry-into-force: 9/29/2006; 5-year compact implementation timeframe: 9/29/06-9/28/11; End of compact: 9/28/2011. Source: GAO analysis of MCC compact data. [End of figure] [End of section] Appendix III: Roads Data: Table 8: Kilometers of Roads Completed for MCC Road Projects in Compacts Ending in 2010 and 2011: Honduras; Original target: 200.0; Final target: 174.5[A]; Final result: 115.0; Percentage of original target met: 57.5%; Percentage of final target met: 65.9%. Cape Verde; Original target: 63.0; Final target: 39.3; Final result: 40.6; Percentage of original target met: 64.4%; Percentage of final target met: 103.3%. Nicaragua; Original target: 158.0; Final target: 67.0; Final result: 74.0; Percentage of original target met: 46.8%; Percentage of final target met: 110.4%. Georgia; Original target: 245.0; Final target: 220.2; Final result: 220.0; Percentage of original target met: 89.8%; Percentage of final target met: 100.0%. Vanuatu; Original target: 213.8; Final target: 149.7; Final result: 149.7; Percentage of original target met: 70.0%; Percentage of final target met: 100.0%. Armenia[B]; Original target: 943.0; Final target: 24.4; Final result: 24.4; Percentage of original target met: 2.6%; Percentage of final target met: 100.0%. Source: GAO analysis of MCC data. [A] After reducing the scope of the Honduras compact, MCC formally reduced the final target for the secondary road activities but did not reduce the final target for the highway activity. [B] MCC provided documentation of kilometers completed; however, the documentation did not include takeover certificates from contractors, the data source for final kilometers in other countries. [End of table] Table 9: Road Roughness of MCC Road Projects in Compacts ending in 2010 and 2011, as Measured by the International Roughness Index (IRI): Honduras[B]: Road section: CA-5 Highway: Section 1; Roughness baseline: 4.7; Original target: 1.9; Final target: 1.9; Final result: N/C; IRI methodology[A]: [Empty]. Road section: CA-5 Highway: Section 2; Roughness baseline: 4.4; Original target: 1.9; Final target: 1.9; Final result: N/C; IRI methodology[A]: [Empty]. Road section: CA-5 Highway: Section 3; Roughness baseline: 4.0; Original target: 1.9; Final target: 1.9; Final result: 2.2; IRI methodology[A]: Mechanical. Road section: CA-5 Highway: Section 4; Roughness baseline: 4.0; Original target: 1.9; Final target: 1.9; Final result: 3.2; IRI methodology[A]: Mechanical. Road section: Secondary roads; Roughness baseline: 13.6; Original target: 2.5; Final target: 2.5; Final result: 3.2; IRI methodology[A]: Mechanical. Cape Verde: Road section: Road 1: Orgăos-Pedra Badejo; Roughness baseline: 19.0; Original target: 2.0; Final target: 2.0; Final result: 2.0; IRI methodology[A]: Visual. Road section: Road 2: Cruz Grande-Calhetona; Roughness baseline: 18.0; Original target: 2.0; Final target: 2.0; Final result: 2.0; IRI methodology[A]: Visual. Road section: Road 4: Assomada-Rincăo; Roughness baseline: 20.7; Original target: 2.0; Final target: 2.0; Final result: 2.0; IRI methodology[A]: Visual. Road section: Vila das Pombas-Eito; Roughness baseline: 18.0; Original target: 10.0; Final target: 10.0; Final result: 10.0; IRI methodology[A]: Visual. Nicaragua: Road section: Highway N-1: Villanueva-Guasaule; Roughness baseline: 12.0; Original target: 3.4; Final target: 3.4; Final result: 1.8; IRI methodology[A]: Mechanical. Road section: S1: Somotillo-Cinco Pinos; Roughness baseline: 13.2; Original target: 3.0; Final target: 3.0; Final result: 3.4; IRI methodology[A]: Mechanical. Road section: Leon-Poneloya; Roughness baseline: 12.0; Original target: 3.0; Final target: 3.0; Final result: 1.8; IRI methodology[A]: Mechanical. Georgia; Road section: [Empty]; Roughness baseline: 16.6; Original target: 3.2; Final target: 2.5; Final result: 1.5; IRI methodology[A]: Mechanical. Vanuatu: Road section: Efate Ring Road; Roughness baseline: 17.5; Original target: 3.5; Final target: 3.5; Final result: 3.0; IRI methodology[A]: Visual. Road section: Santo East Coast Road; Roughness baseline: 22.0; Original target: 3.5; Final target: 3.5; Final result: 3.0; IRI methodology[A]: Visual. Armenia; Road section: [Empty]; Roughness baseline: 14.2; Original target: 5.0; Final target: 4.0; Final result: 3.5; IRI methodology[A]: Mechanical. Source: GAO analysis of MCC data. N/C = not complete. Note: We found weaknesses in some of the reported data on road roughness, including the use of inconsistent application of methodologies and calculation errors. [A] There are two standard methods for measuring IRI: mechanical reading and visual observation. Mechanical reading involves using a device such as a bump integrator, walking profiler, or car-mounted software equipment to measure the IRI for each kilometer of road, which is then averaged across the length of the road to provide a single IRI estimate. The margin of error for mechanical readings differs depending on the type of device used. Visual observation is based on interpreting the smoothness of an observers' drive of the road at different speeds in a standard passenger vehicle, as corresponding to an IRI visual assessment scale. The accuracy of the visual inspection varies with the experience of the observer. [B] Because of the rescoping of the Honduras compact, sections 1 and 2 of the CA-5 Highway were completed post-compact with additional funding provided by the Central American Bank for Economic Integration; as of March 9, 2012, construction of these segments was not complete. [End of table] Table 10: Average Annual Daily Traffic on MCC-Funded Roads in Compacts Ending in 2010 and 2011: Honduras[A]: CA-5 Highway: Section 1; Average annual daily traffic baseline: 8,374; Original target: 9,447; Final target: 9,447; Final result: N/C. CA-5 Highway: Section 2; Average annual daily traffic baseline: 5,411; Original target: 6,104; Final target: 6,104; Final result: N/C. CA-5 Highway: Sections 3 & 4; Average annual daily traffic baseline: 6,732; Original target: 7,594; Final target: 7,594; Final result: N/C. Secondary roads; Average annual daily traffic baseline: 676; Original target: 887; Final target: 887; Final result: N/C. Cape Verde[B]: Road 1: Orgăos-Pedra Badejo; Average annual daily traffic baseline: 298; Original target: 375; Final target: 375; Final result: 432. Road 2: Cruz Grande-Calhetona; Average annual daily traffic baseline: 440; Original target: 554; Final target: 554; Final result: 235. Road 4: Assomada-Rincăo; Average annual daily traffic baseline: 543; Original target: 684; Final target: 684; Final result: 268. Road: Vila das Pombas-Eito; Average annual daily traffic baseline: 410; Original target: 517; Final target: 517; Final result: 184. Bridge Ribeira Grande-Paul; Average annual daily traffic baseline: 552; Original target: 695; Final target: 695; Final result: 76. Bridge Ribeira da Torre-Ponta de Sol; Average annual daily traffic baseline: 765; Original target: 963; Final target: 963; Final result: 644. Bridge A Paul (Vila das Pombas); Average annual daily traffic baseline: 671; Original target: 845; Final target: 845; Final result: 50. Nicaragua: Highway N-1: Villanueva-Guasaule; Average annual daily traffic baseline: 1,413; Original target: 1,580; Final target: 1,580; Final result: 1,962. S1: Somotillo-Cinco Pinos; Average annual daily traffic baseline: 234; Original target: 278; Final target: 278; Final result: 561. S9: Leon-Poneloya; Average annual daily traffic baseline: 1,103; Original target: 1,276; Final target: 1,276; Final result: 1,462. Georgia: Average annual daily traffic baseline: 612; Original target: 6,700; Final target: 1,183; Final result: 1,092. Vanuatu: Efate Ring Road; Average annual daily traffic baseline: 85; Original target: 950; Final target: 98; Final result: 305. Santo East Coast Road; Average annual daily traffic baseline: 307; Original target: 1,000; Final target: 355; Final result: 381. Armenia: Average annual daily traffic baseline: 637; Original target: 460; Final target: 706; Final result: 735. Source: GAO analysis of MCC data. N/C = not complete. Note: We found weaknesses in some of the reported data on average annual daily traffic, including inconsistent updating of baseline measurements. [A] A traffic count for completed roads in Honduras was conducted from December 19, 2010, through January 25, 2011. MCC stated that it does not expect a final traffic measure in Honduras until late 2012. [B] As of January 2012, a final traffic study for completed roads in Cape Verde had not been conducted; results shown reflect traffic counts conducted at the compact end date and do not reflect annualized average daily traffic volumes. [End of table] [End of section] Appendix IV: Port Data: Table 11: Volume of Merchandise through Port for Projects in MCC Compacts That Ended in 2010 and 2011: Thousand tons/year: Cape Verde[A]; Baseline: 482; Original target: 711; Final target: 711; Final result: N/C. Benin; Baseline: 4,519[B]; Original target: 6,945; Final target: 6,945; Final result: 7,606. Source: GAO analysis of MCC data. N/C = not complete. [A] MCC eliminated performance indicators after decreasing the scope of the Cape Verde port project, because the indicators could not be measured prior to completion of the Phase II works, including the wharf 1, container yard, and breakwater. Phase II was funded by the government of Portugal and was expected to be completed after the compact closed. [B] Baseline from 2004. According to MCC, the value in 2005 was 5,680 thousand tons. If MCC had used 2005 data for its baseline, the volume of merchandise would not show as large an increase. [End of table] Table 12: Container Ship Time at Berth for Port Projects in MCC Compacts That Ended in 2010 and 2011: Days: Cape Verde[A]; Baseline: 1.41; Original target: 1.01; Final target: 1.01; Final result: N/C. Benin; Baseline: 2; Original target: 1; Final target: 1; Final result: 1.31. Source: GAO analysis of MCC data. N/C = not complete: [A] MCC eliminated performance indicators after descoping the Cape Verde port project, because the indicators could not be measured prior to completion of the Phase II works, including the wharf 1, container yard, and breakwater. Phase II was funded by the government of Portugal and was expected to be completed after the compact closed. [End of table] Table 13: Container Ship Time at Anchor for Port Projects in MCC Compacts That Ended in 2010 and 2011: Hours: Cape Verde[A]; Baseline: 5; Original target: 4; Final target: 4; Final result: N/C. Benin; Baseline: 16; Original target: 4; Final target: 4; Final result: 34.6. Source: GAO analysis of MCC data. N/C = not complete. [A] MCC eliminated performance indicators after descoping the Cape Verde Port project, because the indicators could not be measured prior to completion of the Phase II works, including the wharf 1, container yard, and breakwater. Phase II was funded by the government of Portugal and was expected to be completed after the compact closed. [End of table] [End of section] Appendix V: Comments from the Millennium Challenge Corporation: Millennium Challenge Corporation: United States of America: August 13, 2012: David Gootnick: Director, International Affairs and Trade: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Gootnick: Subject: Results of Transportation Infrastructure Projects in Seven Countries, Report# GAO-12-631. Thank you for the opportunity to review and comment on the U.S. Government Accountability Office's (GAO) draft report, "Results of Transportation Infrastructure Projects in Seven Countries." I was pleased to see the report acknowledge MCC's commitment to accountability and measuring results. As GAO concluded: "MCC recognizes the importance of a disciplined, transparent, and accountable approach to tracking compact results to make well-informed decisions about U.S. investments. It has established an extensive monitoring and evaluation program that includes guidance and tools for the partner countries and controls on data quality." MCC is committed to rigorously monitoring and evaluating the results of its investments, and welcomes GAO's findings and recommendations for continued improvement in this area. MCC agrees with the four recommendations made in the report, and outline below how the agency is addressing the recommendations. Before discussing the recommendations I would like to note that MCC's compact investments have produced significant, demonstrable results that are not diminished by the data quality challenges highlighted in the report. For example, over 623 kilometers of rehabilitated roadways are now serving poor and rural households in Honduras, Cape Verde, Nicaragua, Georgia, Vanuatu, and Armenia, linking farmers to markets and bolstering important regional trade routes. As GAO reports, performance targets for road roughness were met or exceeded in five of those six countries, and half met or exceeded targets for annual average daily traffic. The report states that the performance target for the number of kilometers to be paved across the six countries was reduced by 63 percent. However, if GAO excluded road projects that MCC terminated or discontinued as a result of undemocratic behavior by the governments of Armenia and Nicaragua, consistent with MCC's rigorous approach to holding partner countries accountable for good policy performance, the target for kilometers of roadway completed in Honduras, Georgia, Cape Verde, and Georgia was reduced by only 19 percent below the original target. MCC agrees with GAO's recommendations to improve the reliability of results measurement and has already taken action to address many of them. In response to lessons learned from the implementation of the compacts analyzed by GAO, which were among the first implemented by MCC, the agency updated its Policy for Monitoring and Evaluation of Compacts and Threshold Programs (M&E Policy) in 2012 and published specific guidance for common indicators to improve the reliability of results measurement across all compact programs. GAO recommends that MCC "improve guidance for common indicators by requiring that they are included in data quality reviews, measured uniformly across compacts, and reported in a standardized manner including when third-party funding is used to complete the original scope of a project." MCC agrees and has already taken preliminary actions to address this recommendation. The 2012 revision to MCC's M&E Policy and accompanying guidance require Millennium Challenge Account entities (MCAs) to include compact-relevant common indicators in their M&E Plans, including definitions and descriptions. The guidance will ensure that MCC's common indicators are measured uniformly and reported in a standardized manner across compacts. In the implementation of the M&E Policy, MCC will ensure data quality reviews include common indicators. MCC will also ensure that it accounts for third-party funding consistently across countries in its reporting. GAO also recommends that MCC "improve guidance for baseline measurements by requiring MCAs to document the date, source, and methodology for establishing the baseline." MCC agrees with this recommendation, and the agency's M&E Policy already requires that MCA M&E Plans document the date, source, and methodology for establishing baseline measurements. Current MCA M&E Plans are compliant with this requirement. With respect to data quality reviews, GAO recommends that MCC "enforce current monitoring and evaluation policy requiring MCAs to conduct data quality reviews and to provide written comments in response to identified issues." MCC agrees with this recommendation. MCC's M&E Policy states, "MCAs are responsible for ensuring that MCC-approved recommendations of data quality reviews are.followed through and implemented' (emphasis added). In performing its oversight role, MCC works with MCAs to ensure that recommendations of data quality reviews are incorporated into M&E Plans and then implemented. GAO notes that MCC improved its beneficiary estimates in 2009 by adopting a more standardized approach and revising its beneficiary numbers. Despite these improvements, the report identified remaining weaknesses in MCC's beneficiary calculations, including mistakes in formulas, inconsistent application of estimation methodologies, and inaccurate reporting of beneficiary numbers. While GAO notes that "the impact of these problems is generally small," MCC will take steps to address the weaknesses, including re-examining formulas and beneficiary data and correcting errors or inconsistencies in analysis or reporting as necessary. GAO also reports that MCC's approach to estimating beneficiary numbers does not include a formal quality review process. Although beneficiary estimates are reviewed informally by MCC staff economists, a formal review process is not in place. As recommended by GAO, MCC will institute a formal process for reviewing beneficiary calculations. This should not only reduce the frequency of possible errors, but also will ensure consistent application of MCC guidelines for beneficiary estimates. I thank you and your staff for the professional manner in which this audit was conducted and for the opportunity to provide additional information and feedback on the GAO draft report. MCC looks forward to continued engagement with GAO to improve its compact assistance programs. Sincerely, Signed by: Andrew Mayock: Acting Vice President: Department of Compact Operations: [End of section] Appendix VI: GAO Contacts and Staff Acknowledgments: GAO Contact: David Gootnick, (202) 512-3149 or gootnickd@gao.gov: Staff Acknowledgments: In addition to the contact named above, Emil Friberg, Jr. (Assistant Director), Miriam Carroll Fenton, Aryn Ehlow, Heather Hampton, and Seyda Wentworth made key contributions to this report. In addition, Michael Armes, Lynn Cothern, Reid Lowe, Ernie Jackson, Leslie Locke, and Jena Sinkfield provided technical assistance. [End of section] Related GAO Products: Millennium Challenge Corporation: Georgian and Benin Transportation Infrastructure Projects Varied in Quality and May Not Be Sustainable. [hyperlink, http://www.gao.gov/products/GAO-12-630]. Washington, D.C.: June 27, 2012. Millennium Challenge Corporation: Compacts in Cape Verde and Honduras Achieved Reduced Targets. [hyperlink, http://www.gao.gov/products/GAO-11-728]. Washington, D.C.: July 25, 2011. Millennium Challenge Corporation: Summary Fact Sheet for 17 Compacts. [hyperlink, http://www.gao.gov/products/GAO-10-797R]. Washington, D.C.: July 14, 2010. Millennium Challenge Corporation: MCC Has Addressed a Number of Implementation Challenges, but Needs to Improve Financial Controls and Infrastructure Planning. [hyperlink, http://www.gao.gov/products/GAO-10-52]. Washington, D.C.: November 6, 2009. Millennium Challenge Corporation: Independent Reviews and Consistent Approaches Will Strengthen Projections of Program Impact. [hyperlink, http://www.gao.gov/products/GAO-08-730]. Washington, D.C.: June 17, 2008. Management Letter: Recommendations for Improvements to MCC's Internal Controls and Policies on Premium Class Air Travel. [hyperlink, http://www.gao.gov/products/GAO-08-468R]. Washington, D.C.: February 29, 2008. Millennium Challenge Corporation: Projected Impact of Vanuatu Compact Is Overstated. [hyperlink, http://www.gao.gov/products/GAO-07-1122T]. Washington, D.C.: July 26, 2007. Millennium Challenge Corporation: Vanuatu Compact Overstates Projected Program Impact. [hyperlink, http://www.gao.gov/products/GAO-07-909]. Washington, D.C.: July 11, 2007. Millennium Challenge Corporation: Progress and Challenges with Compacts in Africa. [hyperlink, http://www.gao.gov/products/GAO-07-1049T]. Washington, D.C.: June 28, 2007. Millennium Challenge Corporation: Compact Implementation Structures Are Being Established; Framework for Measuring Results Needs Improvement. [hyperlink, http://www.gao.gov/products/GAO-06-805]. Washington, D.C.: July 28, 2006. Analysis of Future Millennium Challenge Corporation Obligations. [hyperlink, http://www.gao.gov/products/GAO-06-466R]. Washington, D.C.: February 21, 2006. Millennium Challenge Corporation: Progress Made on Key Challenges in First Year of Operations. [hyperlink, http://www.gao.gov/products/GAO-05-625T]. Washington, D.C.: April 27, 2005. Millennium Challenge Corporation: Progress Made on Key Challenges in First Year of Operations. [hyperlink, http://www.gao.gov/products/GAO-05-455T]. Washington, D.C.: April 26, 2005. [End of section] Footnotes: [1] MCC commits funding when a compact is signed and obligates funds after the compact enters into force. As of June 2012, MCC had signed initial compacts with, in order of signature, Madagascar, Honduras, Cape Verde, Nicaragua, Georgia, Benin, Vanuatu, Armenia, Ghana, Mali, El Salvador, Mozambique, Lesotho, Morocco, Mongolia, Tanzania, Burkina Faso, Namibia, Senegal, Moldova, the Philippines, Jordan, Malawi, Indonesia, and Zambia. In February 2012, MCC signed a second compact with Cape Verde. [2] We did not include the Madagascar compact in this review because, as the result of an undemocratic transfer of power in Madagascar in March 2009, MCC formally terminated the compact effective August 31, 2009, before the end of the 5-year statutory compact timeframe. [3] Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, § 668(d)(1)(A). The Act also required us to examine the financial control and procurement practices of MCC and its accountable entities. We responded to this requirement separately in GAO, Millennium Challenge Corporation: MCC Has Addressed a Number of Implementation Challenges, but Needs to Improve Financial Controls and Infrastructure Planning, [hyperlink, http://www.gao.gov/products/GAO-10-52] (Washington, D.C.: Nov. 6, 2009). [4] GAO, Millennium Challenge Corporation: Compacts in Cape Verde and Honduras Achieved Reduced Targets, [hyperlink, http://www.gao.gov/products/GAO-11-728] (Washington, D.C.: Jul. 25, 2011). [5] GAO, Millennium Challenge Corporation: Georgia and Benin Transportation Infrastructure Projects Varied in Quality and May Not Be Sustainable, [hyperlink, http://www.gao.gov/products/GAO-12-630] (Washington, D.C.: Jun. 27, 2012). [6] [hyperlink, http://www.gao.gov/products/GAO-11-728] and [hyperlink, http://www.gao.gov/products/GAO-12-630]. [7] Other board members are the U.S. Trade Representative, the Administrator of the U.S. Agency for International Development, the Chief Executive Officer of MCC, and up to four Senate-confirmed nongovernmental members appointed by the President from lists of individuals submitted by congressional leadership. [8] MCC generally refers to each country's accountable entity by combining MCA with the country's name (e.g., MCA-Benin). [9] MCC developed guidelines in 2006 to assist eligible countries in the preparation of monitoring and evaluation plans and issued an updated policy in 2009 and in 2012. [10] In many cases, indicators are chosen because they relate to inputs for the economic rate of return analysis that MCC conducts to estimate a project's likely impact on the partner country's economic growth and poverty reduction prior to compact approval. More specifically, a project's estimated economic rate of return is the expected annual average return to the country's firms, individuals, or sectors for each dollar that MCC spends on the project. [11] MCC issued its Common Indicators Directive in 2009, establishing the common indicators and requiring MCAs to use them in project monitoring and evaluation plans. Although the seven compacts that we reviewed were signed before 2009, initial monitoring and evaluation plans for three of the six compacts with road projects included common indicators for kilometers completed, road roughness, and average annual daily traffic. The MCAs for the three remaining compacts introduced the common indicators into their monitoring and evaluation activities before the compacts ended. [12] MCC has developed additional common indicators for roads, including the percentage of funds disbursed for contracted studies, value of signed contracts for road works, percentage of funds for contracted road works disbursed, kilometers of roads under works contracts, and value of signed contracts for feasibility design supervision and program management contracts. [13] MCC has developed other indicators for ports, including percentage of funds for contracted port works disbursed, average time for goods to clear customs, average duration of stay of trucks at port, tons of merchandise shipped per year, and whether the port is meeting the International Ship and Port Facility Code, which establishes security requirements for ports. [14] According to its Guidelines for Economic and Beneficiary Analysis, MCC takes the household as the unit of measurement, counting as beneficiaries all members of the household. MCC's projects include targeted projects that benefit specific individuals or households--for example, projects focused on agricultural development or school construction--but also includes projects that are national or broad- based in scope and benefit people living in a large geographical area. Beneficiaries include everyone who realizes income gain or expenditure savings regardless of the magnitude of these changes, although the income gain or expenditure savings are likely to vary significantly across individuals and projects. [15] MCC officials stated that detailed beneficiary analyses for its compacts are carried out retrospectively. MCC provided us with detailed beneficiary analyses, including "poverty scorecards" focusing on the impact of compacts on the poor, for six other compacts out of the 26 current and completed compacts. [16] In response to a draft copy of this report, MCC noted that if one excludes road projects that MCC terminated or discontinued as a result of undemocratic behavior by the governments of Armenia and Nicaragua, the final target for kilometers of roads completed in Honduras, Georgia, Cape Verde, and Vanuatu is only 19 percent below the original target. See appendix V for a copy of MCC's comments. [17] In Georgia, MCC reduced the roughness target to account for a delayed construction start date. A road's surface is smoothest directly following construction. Because the delay in construction pushed the expected completion date closer to the compact end date, this reduced the amount of time MCC-funded roads could deteriorate before roughness measurements were taken at the end of the compact. Therefore, the roads were expected to have a better roughness measure than originally planned. In Armenia, MCC reduced the target for road roughness once the project was placed on indefinite hold. The 24.4 kilometers that MCC completed before putting the project on hold had a lower roughness target than the full package of roads MCC originally planned to complete. Therefore, removing the terminated roads from the project resulted in a better roughness measure for the compact overall. [18] During construction, because all lanes of a road often are not paved at the same time, uneven construction quality (e.g., nonuniform compaction of asphalt pavements), construction techniques that allow some variation in the surface, and other factors may lead to different roughness measures for each lane. [19] For two of the road segments, the nonmeasurable sections of road were left blank and the calculation of the final IRI used the value from the one measurable side of the road as the overall measure for that kilometer. [20] MCC revised the original traffic targets for the Georgia, Vanuatu, and Armenia compacts. In Georgia and Vanuatu, MCC reduced traffic targets when it corrected the original baseline estimates. In Armenia, MCC increased the traffic target once the road project was placed on indefinite hold; the single road that MCC completed prior to the project hold had a higher traffic target than the full package of roads MCC originally planned to complete. [21] MCC used a traffic count from a study conducted in 2005; however, construction of the road project began in 2006. [22] In addition, the 2000 traffic study was conducted for a 72- kilometer section of highway, rather than the 18 kilometers designated for the project. [23] MCC policy allows each country to determine the frequency and timing of each review, along with which indicators will be evaluated. [24] The 2008 review provided several recommendations on how to structure the traffic count survey. In addition, a 2009 review found that the Vanuatu compact was at risk of double counting vehicles and recommended that the monitoring and evaluation plan indicate that traffic refers to the sum of the average number of vehicles passing over checkpoints. [25] Third-party funding is funding from a source other than MCC or the partner government. [26] MCC originally planned to fund all reconstruction included in the four sections of the Honduras highway project. However, because estimated costs had increased, MCC determined that it would be unable to complete all four sections within the 5-year compact timeframe and within the funding allocation. As a result, MCC funded about 94 percent of construction on sections 3 and 4, funded 31 percent of section 2, and funded none of section 1. The third-party funding from the Central American Bank of Economic Integration funded the difference. For more information on this project, see [hyperlink, http://www.gao.gov/products/GAO-11-728]. [27] Phase 2 works included the expansion of Wharf 1 and construction of a new breakwater. [28] While the project was largely completed as planned, components for a proposed fourth lot--which included a storage facility for dry bulk goods such as grains and sand and a fish quality inspection station--were deemed not viable once MCA-Benin conducted its feasibility studies. As a result, MCA-Benin did not tender a bid for that lot. According to MCC officials, the funds originally planned for those items were shifted to the other infrastructure components. The funds also helped cover cost increases and additional work on the wharf such as increasing wall length and dredging the berth. [29] Reviews also identified other port indicators for improvement in Benin. For example, the 2008 and 2009 reviews indicated that the average duration of stay of trucks at port, which measures the reduction in time necessary for vehicles to cross the port due to reduced congestion, had a baseline value that was set with no justification of its actual level. The review recommended that this information be collected in the annual survey of port user satisfaction. MCA-Benin did so and determined that the baseline value was 7 hours. However, no changes were made to the final monitoring and evaluation plan to reflect this and MCC and MCA officials indicated that they agreed not to revise indicator baselines at the time. [30] See GAO, Millennium Challenge Corporation: Independent Review and Consistent Approaches Will Strengthen Projections of Program Impact, [hyperlink, http://www.gao.gov/products/GAO-08-730] (Washington, D.C.: Jun. 17, 2008). [31] According to Standards for Internal Control in the Federal Government, a variety of control activities should be used in information processing, including checking the data entered. In addition, GAO's Internal Control Management and Evaluation Tool states that data validation and editing should be performed to identify erroneous data, which need to be reported and promptly corrected (GAO, Internal Control Management and Evaluation Tool, [hyperlink, http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 2001). [32] MCC officials told us that the original number of 53,988 beneficiaries was erroneous and should have been 111,442, in accordance with the 2004 Georgian Agricultural Census. [33] The 2002 census indicates 6,769,914 for the population of Benin; adjusting population growth for 3 years leads to 7,419,235, not 8,490,000, in 2005. In addition, the supporting document that accompanied the beneficiary scrub spreadsheet and explained the computations had a different "original" baseline number--8,791,832-- for 2005. [34] MCC officials indicated that the standard time horizon for the beneficiary analysis is generally 20 years but that exceptions exist. [35] In the case of Benin and Nicaragua, the growth rate for the year 2000 was added to the formula. [36] [hyperlink, http://www.gao.gov/products/GAO-11-728]. [37] [hyperlink, http://www.gao.gov/products/GAO-12-630]. [38] We began work for this report in 2010; however, the job was suspended while we conducted work for a related engagement. [39] The statutory 5-year maximum timeframe begins once a compact enters into force. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. [End of document] Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.