This is the accessible text file for GAO report number GAO-12-920 entitled 'Public Financial Management: Improvements Needed in USAID's and Treasury's Monitoring and Evaluation Efforts' which was released on September 13, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Ranking Member, Committee on Foreign Relations, U.S. Senate: September 2012: Public Financial Management: Improvements Needed in USAID's and Treasury's Monitoring and Evaluation Efforts: GAO-12-920: GAO Highlights: Highlights of GAO-12-920, a report to the Ranking Member, Committee on Foreign Relations, U.S. Senate. Why GAO Did This Study: Effective management of public resources can play an important role in a country’s development. In recent years, developing countries committed to strengthen their PFM systems and donors committed to use those systems as much as possible. The United States provides assistance to strengthen PFM systems primarily through USAID and Treasury. USAID conducts capacity building activities to strengthen PFM systems as part of its development programs and has also set a target to obligate 30 percent of its annual assistance through local systems by 2015. Treasury provides technical assistance through advisors who work in country, typically with the finance ministry. GAO was asked to examine the processes U.S. agencies use to (1) develop programs to strengthen PFM systems and (2) monitor and evaluate those programs. GAO reviewed agency guidance and program documents, interviewed U.S. agency officials, and selected case studies to serve as illustrative examples of PFM-related programs. What GAO Found: To develop programs to strengthen developing countries’ Public Financial Management (PFM) systems, the U.S. Agency for International Development (USAID) and the U.S. Department of the Treasury (Treasury) rely on assessments of the host country government’s systems. In 2011, USAID implemented new processes that place a greater emphasis on PFM in its development efforts as the agency aims to increase its use of country systems to deliver assistance. The agency traditionally included PFM capacity-building efforts only as components of broader programs, as it identified relevant weaknesses during the country assessment or program design process. USAID’s new strategy and program development processes include a mandatory assessment of a country’s institutional capacity, including its financial systems, and a requirement to consider the use of country systems to deliver assistance. Most USAID country offices are required to develop a strategy using the new guidance by the end of fiscal year 2013. Treasury’s process for developing programs begins with an initial assessment of the host country’s capabilities. Treasury staff then draft objectives for the program. For example, a Treasury program in Honduras set four objectives, including improving operational efficiency and enhancing accountability by strengthening the organization of the ministry of finance. Once in country, the advisor develops an annual workplan, outlining more specific goals aimed at meeting the overall objectives. USAID and Treasury use several processes to monitor and evaluate their PFM assistance, but weaknesses exist. USAID uses its regular procedures, which may include performance management plans, periodic progress reporting, site visits, and evaluations, to monitor and evaluate its PFM-related programs. Prior reports by USAID’s Inspector General and GAO have found weaknesses in USAID’s implementation of its monitoring procedures in other programs, including programs from the USAID offices that provide PFM assistance. In addition, USAID is currently unable to monitor overall progress toward its target to obligate 30 percent of its program funds through local systems by 2015. USAID, and GAO in prior reports, have identified a number of weaknesses in evaluation practice. To address weaknesses the agency had identified, USAID adopted a new evaluation policy in January 2011 that states that all large projects are required to have an external evaluation, 3 percent of program budgets should be devoted to external evaluation, and evaluations must use methods that generate the highest quality evidence. Treasury’s processes for monitoring and evaluating its programs include monthly reports, annual quantitative performance measures, voluntary customer feedback surveys, and on-site management reviews, but Treasury does not fully evaluate the performance of its completed technical assistance programs. In addition, Treasury’s quantitative performance measures have been a useful project-level indicator of performance but have not been a useful indicator of overall performance due in part to inherent challenges associated with summarizing program performance and errors introduced when aggregating the performance data. Furthermore, a senior Treasury official reported that Treasury had not yet fully implemented a requirement to conduct independent postproject evaluations of its technical assistance programs. What GAO Recommends: USAID should improve its capacity to measure its use of local systems and ensure adequate monitoring of its PFM programs. Treasury should implement additional controls to improve the process for computing program-wide annual performance measures and fully implement its requirement to evaluate the impact of its completed assistance. USAID and Treasury both concurred with GAO’s recommendations. View [hyperlink, http://www.gao.gov/products/GAO-12-920]. For more information, contact Thomas Melito at (202) 512-9601 or melitot@gao.gov. [End of section] Contents: Letter: Background: USAID's and Treasury's Processes for Developing PFM Programs Involve Consultation with Stakeholders and Rely on Assessments of Host-Country Systems: USAID and Treasury Use Certain Processes to Monitor and Evaluate PFM- Related Programs, but Implementation Has Some Weaknesses: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Several International Organizations Reported Results of Their Public Financial Management Diagnostic Tools: Appendix III: State Department Conducts Fiscal Transparency Reviews and Administers the Domestic Finance for Development Initiative: Appendix IV: Examples of Public Financial Management-related Projects in Six Countries: Appendix V: Comments from the U.S. Agency for International Development: Appendix VI: Comments from the Department of the Treasury: Appendix VII: GAO Contact and Staff Acknowledgments: Tables: Table 1: Selected PFM-Related Commitments in the Paris Declaration and the Accra Agenda: Table 2. Status of Countries in the PFM Risk-Assessment Framework Process: Table 3: USAID's Kosovo Growth and Fiscal Stability Initiative: Selected Objectives and Expected Results: Table 4: USAID's Liberia GEMS: Selected Objectives, Subobjectives, and Expected Results: Table 5: USAID's Peru ProDecentralization 2: Selected Objectives and Expected Results: Table 6: Treasury's Technical Advisory Services to Cambodia on Budgeting: Selected Objectives and Activities: Table 7: Treasury's Technical Advisory Services on Honduran Budgeting: Selected Objectives and Action Plan Activities: Table 8: Treasury's Technical Advisory Services to South Africa on Budgeting: Selected Objectives and Activities: Figures: Figure 1: Illustrative Example of a Country's Budget Cycle: Figure 2: Countries Receiving U.S. PFM-Related Assistance since 2007: Figure 3. Select Components of USAID's New Strategy Development and Project Design Processes That May Identify the Need for PFM Capacity Building: Figure 4: Open Budget 2010 Index Scores: Figure 5: Transparency International 2010 Corruption Perceptions Scores: Figure 6. Case Study Country Percentile Rankings on PFM Diagnostic Tools: Abbreviations: DF4D: Domestic Finance for Development: DG: Democracy and Governance Office, U.S. Agency for International Development: EG: Economic Growth Office, U.S. Agency for International Development: IMF: International Monetary Fund: OECD: Organization for Economic Cooperation and Development: OTA: Office of Technical Assistance, U.S. Department of the Treasury: PEFA: Public Expenditure and Financial Accountability Program: PFM: Public Financial Management: PFMRAF: Public Financial Management Risk Assessment Framework: State: U.S. Department of State: Treasury: U.S. Department of the Treasury: USAID: U.S. Agency for International Development: [End of section] United States Government Accountability Office: Washington, DC 20548: September 13, 2012: The Honorable Richard G. Lugar: Ranking Member: Committee on Foreign Relations: U.S. Senate: Dear Senator Lugar: A country's development is influenced, in part, by how effectively its government raises, manages, and expends public resources. In 2005, as part of the Paris Declaration on Aid Effectiveness, developing countries committed to strengthen their public financial management (PFM) systems and donors committed to use those systems to the maximum extent possible. More transparent and effective PFM systems can make governments more accountable and give donors more assurance that funds will not be misspent if they deliver more aid through recipient countries' budgets and finance systems. Highlighting the need for stronger PFM, international organizations have raised concerns about low levels of transparency in countries' budgets and high levels of corruption. The United States provides assistance to strengthen other countries' PFM systems primarily through the U.S. Agency for International Development (USAID) and the U.S. Department of the Treasury (Treasury). USAID has set a target of obligating 30 percent of the agency's total annual assistance, or program funds, through local systems, including both partner country PFM systems and local not-for- profit and for-profit organizations, by fiscal year 2015.[Footnote 1] Over the last decade, USAID has provided most of its assistance through agreements with large international implementing partners. USAID assesses countries' PFM systems to determine their ability to effectively receive and manage U.S. assistance directly. USAID primarily provides PFM capacity-building assistance as a component of a broader development program, whereas the Treasury's Office of Technical Assistance (OTA) provides PFM-related technical assistance. Since 2007, USAID and Treasury have provided PFM assistance to 70 countries. The U.S. Department of State (State) assesses the budget transparency of countries that receive U.S. assistance and helps to leverage existing PFM assistance, but has not funded PFM assistance. In this context, you asked us to examine: (1) the processes U.S. agencies use to develop programs to strengthen PFM systems in developing countries and (2) the processes U.S. agencies use to monitor and evaluate their programs to strengthen PFM systems. To address these objectives, we reviewed USAID, Treasury, and State guidance and program documents and interviewed U.S. officials in Washington, D.C. To address the first objective, we reviewed relevant documents, including assessment reports, agency guidance, and assistance agreements.[Footnote 2] To address the second objective, we reviewed agency and Organization for Economic Cooperation and Development (OECD)[Footnote 3] guidance and program documents, including assistance agreements, progress reports, evaluations, work plans, and performance management plans; and quantitative performance data. We spoke with USAID and Treasury officials about their development, monitoring, and evaluation of PFM-related programs. We also selected six case study countries in which Treasury or USAID has relevant ongoing or recently completed projects focused on strengthening PFM systems. We chose these case study countries for their geographic and income level diversity, as illustrative rather than generalizable examples. For Treasury, we selected Cambodia, Honduras, and South Africa; and for USAID, Kosovo, Liberia, and Peru. For each of our case study countries, we interviewed U.S. officials and reviewed program development and monitoring and evaluation documents related to PFM-focused projects. Finally, we interviewed officials at the World Bank and International Monetary Fund to discuss the tools that they use to assess PFM systems. We conducted this performance audit from October 2011 through September 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. See appendix I for a more detailed discussion of our scope and methodology. Background: PFM Focuses on a Country's Budget Cycle: The broad objectives of PFM are to achieve overall fiscal discipline, allocation of resources to priority needs, and efficient and effective allocation of public services, according to OECD. While donors may use different definitions of PFM, most definitions focus on a country's budget cycle, the process used to manage public resources. The budget cycle centers around four main phases: (1) budget formulation, (2) budget execution, (3) accounting and reporting, and (4) external oversight (see figure 1). OECD states that PFM includes all components of a country's budget cycle. Figure 1: Illustrative Example of a Country's Budget Cycle: [Refer to PDF for image: illustration] Cycle is a repeatable loop. Budget Formulation: Processes by which the government plans for the use of the coming year's resources in accordance with policy priorities. Components include: * Budget preparation; * Expenditure identification and classification; * Medium-term expenditure estimates; * Setting policy priorities and expenditure ceilings; * Approval by legislature. Budget Execution: Processes by which the programming agencies and ministry of finance, or equivalent, implement the approved budget. Components include: * Cash and commitment management; * Adjustments; * Payroll; * Procurement; * Transfers; * Internal control and audit; Accounting and Reporting: Throughout the fiscal year, each agency records its expenditures. These accounts are then consolidated by the ministry of finance, or equivalent. Components include: * Accounting; * Reporting, including timeliness and accuracy; * Budget monitoring. External Oversight: The accounting reports are then subjected to external review, including reviewing the government's actual revenue collection and spending compared to the approved budget. Components include: * Auditing; * Legislative scrutiny; * Legal reform. Source: GAO analysis of information from the World Bank and Overseas Development Institute. [End of figure] A budget cycle starts with the budget formulation process, in which the government, often with legislative oversight, plans for the use of the coming year's resources in accordance with policy priorities.[Footnote 4] After the government approves the budget and the new fiscal year begins, programming agencies and the ministry of finance, or appropriate entity, are responsible for executing the budget. They use the resources allocated to them for items such as salaries for public servants, operating costs for their offices, and goods and services delivered to their beneficiaries. The ministry of finance, or equivalent, manages the flow of funds and monitors and makes in-year adjustments to help ensure compliance with the budget and PFM rules. Throughout the fiscal year, each programming agency is to account for and record its expenditures. The ministry of finance centrally consolidates these accounts. At the end of the fiscal year, the ministry of finance, or equivalent, issues an accounting report that demonstrates how the budget was implemented. External or independent entities, such as a country's supreme audit institution, may review this report.[Footnote 5] The audit institution reviews the government's revenue collection and spending and issues its own statement on the execution of the budget and the strength of the PFM systems. In many countries, the institution presents this audit report to an appropriate government entity for further scrutiny and action. PFM processes involve a number of governmental entities. While the ministry of finance is generally the focus of a country's PFM system, PFM extends to all public ministries that are charged with delivering services or have spending authority. Civil society, donors, and oversight institutions, such as a country's supreme audit institution, also help to ensure the proper management of public funds through external scrutiny and review. Assessment Tools Highlight Both Improvements and Weaknesses in Developing Countries' PFM Systems: Broad PFM assessments conducted by international organizations report improvements in countries' PFM systems, as well as continued weaknesses. Several international organizations, including the Public Expenditure and Financial Accountability (PEFA) program, World Bank, International Budget Partnership, and Transparency International, have developed assessment tools to assess broad aspects of PFM systems. [Footnote 6] These assessments have highlighted varying levels of progress in improving PFM systems around the world, and the last two have identified relatively low levels of transparency in many countries' budgets and high levels of corruption, respectively. More specifically: * PEFA: A 2010 monitoring report on the PEFA program, released by the PEFA Secretariat, found that while PFM systems were improving overall, progress varied among elements of countries' systems.[Footnote 7] The PEFA secretariat's analysis, based on a comparison of 33 repeat PEFA assessments from 2005 through 2010, showed that more countries had a higher number of improved versus worsened scores, indicating a broad trend of PFM improvement across the countries surveyed. According to the analysis, country actions taken to strengthen PFM features in the earlier stages of the budget cycle, such as planning, are more likely to improve or maintain a high score than actions taken later in the budget cycle, such as control and oversight of actual spending. * World Bank: According to our analysis of the World Bank's quality of budgetary and financial management indicator scores from 2005 through 2010, slightly more than one-third of the countries assessed showed improvements in the quality of their PFM systems, while one-quarter of the countries showed a worsening in the quality of their PFM systems. [Footnote 8] * International Budget Partnership:[Footnote 9] In its 2010 open budget survey of 94 countries, the International Budget Partnership concluded that the state of budget transparency was poor.[Footnote 10] Only about 21 percent of the countries surveyed had open budgets while 44 percent of countries provided limited to no budget information. Nonetheless, the survey found the trend toward open budgets was favorable based on substantial improvements in budget transparency, especially among countries that had provided little information in the past. Some of these governments achieved improvements by simply making budget documents available on their websites. * Transparency International:[Footnote 11] The organization's Corruption Perceptions Index showed that nearly three quarters of the 178 countries in the index scored below 5, on a scale of 10 (very clean) to 0 (highly corrupt), indicating what it categorizes as a serious corruption problem.[Footnote 12] For further information on each organization's assessment tools and selected results, including the percentile rankings of our six case study countries in selected PFM diagnostic tools, see appendix II. International Attention to Strong and Effective PFM Systems Has Increased since 2003: Donors and recipient governments have increased their attention to strengthening PFM systems, recognizing that strong and effective PFM systems underpin fiscal and macroeconomic stability, guide the allocation of public resources to national priorities, support the efficient delivery of services for poverty reduction and economic development, and make possible the transparency and scrutiny of public funds. In 2003, OECD's Development Assistance Committee established a Working Party on Aid Effectiveness, which has played a critical role in establishing an initial international donor coordination framework and setting goals for donors and aid recipient countries to strengthen PFM. The working party has sponsored four global development forums since 2003. The second forum, held in Paris in 2005, resulted in the Paris Declaration on Aid Effectiveness which included broad commitments by recipient governments and donors to strengthen PFM systems and use those systems as appropriate. More than 100 countries and aid agencies, including the United States, endorsed the declaration. By signing the declaration, recipient governments made clear commitments to strengthen their systems to the maximum extent possible, and donor governments made clear commitments to use those systems wherever possible. These commitments were renewed and refined in the subsequent forum in Accra, Ghana, in 2008. The most recent forum was held in Bussan, South Korea, in 2011. (See table 1.) Table 1: Selected PFM-Related Commitments in the Paris Declaration and the Accra Agenda: Partner countries committed to: * Carry out diagnostic reviews and undertake reforms to ensure national systems, institutions, and procedures for managing resources are effective, accountable, and transparent; * Intensify efforts to mobilize domestic resources, strengthen fiscal sustainability, and create an enabling environment for public and private investments; * Publish timely, transparent, and reliable reporting on budget execution; * Take leadership of PFM reform process. Donors committed to: * Use country systems and procedures to the maximum extent possible. Where use is not feasible, establish safeguards and measures in ways that strengthen rather than undermine country systems; * Provide reliable indicative commitments of aid over a multi-year framework and disburse aid in a timely and predictable manner according to agreed schedules; * Rely to the maximum extent possible on transparent partner government budget and accounting mechanisms. Partner countries and donors jointly committed to: * Implement harmonized diagnostic reviews and performance assessment frameworks in PFM. Source: GAO analysis of Paris Declaration on Aid Effectiveness, 2005, and the Accra Agenda for Action, 2008. [End of table] U.S. Agencies Provide a Variety of PFM-Related Assistance: USAID and Treasury are the two main U.S. agencies providing PFM- related assistance.[Footnote 13] Combined, the two agencies have PFM- related projects in 70 countries in all regions of the world, as shown in figure 2. State also conducts some PFM-related activities, although it has not funded programs. See appendix III for details on State's PFM activities. Figure 2: Countries Receiving U.S. PFM-Related Assistance since 2007: [Refer to PDF for image: illustrated world map] USAID: Afghanistan; Angola; Bangladesh; Bolivia; Burundi; Cambodia; Colombia; Democratic Republic of the Congo; Djibouti; Dominican Republic; Ecuador; Egypt; El Salvador; Ethiopia; Georgia; Ghana; Guatemala; Guinea; Haiti; Honduras; India; Indonesia; Iraq; Jamaica; Jordan; Kenya; Kosovo; Lebanon; Liberia; Madagascar; Malawi; Mali; Mexico; Mongolia; Morocco; Mozambique; Namibia; Nepal; Nicaragua; Nigeria; Pakistan; Panama; Paraguay; Peru; Philippines; Rwanda; Senegal; Sierra Leone; Somalia; South Africa; South Sudan; Sri Lanka; Sudan (pre-July 2011); Tanzania; Thailand; Timor-Leste; Uganda; Vietnam; West Bank and Gaza; Yemen; Zambia; Zimbabwe. Treasury[A]: Afghanistan; Algeria; Angola; Argentina; Bolivia; Botswana; Burundi; Cambodia; Chile; Colombia; Costa Rica; Democratic Republic of the Congo; Dominican Republic; Ecuador; Egypt; El Salvador; Ethiopia; Ghana; Guatemala; Guinea; Haiti; Honduras; India; Indonesia; Jamaica; Jordan; Lesotho; Liberia; Madagascar; Mexico; Mongolia; Morocco; Mozambique; Namibia; Nicaragua; Niger; Nigeria; Pakistan; Paraguay; Peru; Philippines; Rwanda; Senegal; South Africa; Sri Lanka; Tanzania; Uganda; Uruguay; Vietnam; West Bank and Gaza; Zambia. Nontransparent waivers[B]: State: Afghanistan; Algeria; Angola; Cambodia; Cameroon; Central African Republic; Chad; Cote d'Ivoire; Dominican Republic; Egypt; Ethiopia; Gabon; Gambia; Guinea; Guinea Bissau; Kyrgyzstan; Lebanon; Libya; Nicaragua; Niger; Somalia ; Swaziland; Tajikistan; Turkmenistan; Uzbekistan; Yemen. Source: GAO analysis of USAID, Treasury, and State data; Map Resources (map). [A] Only countries where assistance is at least partially funded by OTA are listed. OTA projects funded completely by other agencies are not included. [B] U.S. appropriations law requires State to evaluate the fiscal transparency of foreign governments before providing certain U.S. assistance. If State determines it is in the U.S. national interest, it processes waivers for countries deemed nontransparent thereby allowing U.S. agencies to provide assistance to them. See Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, § 668(e), 121 Stat. 1844, 2353 (2007) and Consolidated Appropriations Act, 2012, Pub. L. No. 112-74, § 7031(b), 125 Stat. 786, 1211 (2011). [End of figure] USAID: USAID provides PFM capacity-building activities through its development programs and is seeking to provide more of its assistance through recipient countries' financial systems. Capacity-building activities to strengthen PFM systems are typically part of broader democracy and governance (DG) or economic growth (EG) programs.[Footnote 14] PFM activities that are included as components of DG programs typically address the areas of legislative function and processes, public sector executive function, local government decentralization, and anticorruption reforms. PFM activities included as components of EG programs typically address fiscal and monetary policy issues. USAID has identified DG and EG programs with PFM components in over 60 countries since 2007. However, according to a USAID official, USAID cannot determine the total funding for PFM activities because it does not collect data at a sufficiently detailed level to precisely identify PFM activities. Therefore, the official reported that USAID is unable to separate PFM assistance from other assistance in larger DG and EG development programs. In 2011, total funding for DG programs was $1.7 billion, and for EG programs, $4.2 billion.[Footnote 15] For illustrative examples of USAID PFM-related projects, see appendix IV. In addition to PFM capacity-building programs, USAID is seeking to increase the use of recipient country PFM systems to deliver assistance. As part of its implementation and procurement reforms under its institutional reform agenda, known as USAID Forward, USAID has announced an agency target of obligating 30 percent of its annual assistance through local systems, including both partner country PFM systems and local not-for-profit and for-profit organizations, by fiscal year 2015. Treasury: Treasury's OTA provides technical assistance through advisors who work in-country within the finance ministry, the central bank, or other government entities. OTA's program consists of five core areas, as follows: * Budget and Financial Accountability helps foreign governments reform and strengthen their PFM systems in order to promote control, accountability, and transparency over resources, and to improve a country's overall financial condition. * Banking and Financial Services promotes strong financial sectors in which institutions are well regulated, stable, and accessible; serve as efficient intermediaries between savers and investors; and are resistant to criminal activity. * Government Debt Issuance and Management helps host countries implement sound public debt management practices and develop markets through which the government can finance itself. * Economic Crimes helps counterpart governments build their capacity to prevent, detect, investigate, and prosecute complex financial crimes. * Revenue Policy and Administration provides assistance to ministries of finance and other relevant organizations that strengthens their ability to serve the country and its people through the efficient and responsible collection of revenues. OTA provides both long-term resident and intermittent advisors. Long- term resident advisors provide advice and training to ministers of finance, central bank governors, and other government officials. Country engagement typically lasts between 2 and 10 years, according to an OTA official. Intermittent advisors provide highly specialized technical assistance in support of long-term projects, projects requiring several different specialties, and projects of short duration. According to an OTA official, they currently have about 70 advisors in roughly 50 countries. In 2011, OTA's total funding was $44.5 million, with $25.5 million in direct OTA appropriations and $19 million in transfers from other agencies, including State, USAID, and the Millennium Challenge Corporation. For illustrative examples of OTA technical assistance projects, see appendix IV. USAID's and Treasury's Processes for Developing PFM Programs Involve Consultation with Stakeholders and Rely on Assessments of Host-Country Systems: USAID's Processes: USAID is implementing new processes for developing programs that reflect new agency reform priorities to increase the use of country systems to deliver U.S. assistance. USAID's work in PFM has traditionally involved capacity building under broader programs designed to improve fiscal management and promote good governance. According to agency officials, USAID has increased its attention to PFM issues. USAID's new country strategy development and project design processes include various analyses and assessments that may identify opportunities to strengthen and use countries' PFM systems, as prioritized by USAID's reform agenda. In these new processes, USAID may identify the need for PFM assistance during countrywide development assessments or through other required assessments. Furthermore, according to USAID guidance, country offices are to develop these assistance programs in collaboration with country stakeholders throughout the program development process. USAID's new processes are similar in structure and approach to prior processes, but, according to USAID, aim to incorporate more analytical rigor at all stages of the strategic planning framework so that USAID's efforts are better aligned with the recipient country's development efforts. USAID may identify the need for PFM assistance during the country strategy development process. According to USAID, most country offices are required to develop a Country Development Cooperation Strategy, a 5-year strategy document, by the end of fiscal year 2013.[Footnote 16] USAID's latest draft of the guidance for developing the country strategy, released in September 2011, states that the strategy should demonstrate how it is integrating the goals of USAID's assistance reform effort, USAID Forward, such as working through host-country systems and developing the capacity of civil society and private sector partners. In developing the country strategy document, the USAID country office is to consult with various country stakeholders and conduct several assessments to understand the development context and develop goals and objectives. According to the guidance, the country office is required to develop the strategy document with a focus on maximizing the impact of USAID resources and build the capacity of specific institutions and related governance systems at the national, regional, and local levels. For example, if a USAID country office determines that a nontransparent and inefficient financial system is a key obstacle to economic growth, the country office is to work with the host government to improve its capacity for sound financial management and equitable allocation of resources. According to USAID officials, one indicator of need for a PFM project would be an assessment that the country has a major fiscal imbalance that needs to be corrected by a combination of increased revenue mobilization or reduction of budget expenditures. Another indicator would be a determination, arrived at through repeated reporting on corruption in the media, or internal and external publications, and supported by stakeholder interviews, that anticorruption programs would improve PFM. The goals and objectives outlined in the country strategy are to provide the basis for project design, monitoring, and evaluation. As of June 2012, USAID stated it had approved 15 country strategies, and 73 USAID country offices are scheduled to complete a strategy by October 2013. See figure 3 for highlights of the elements of USAID's new strategy and project design processes that could identify the need for PFM capacity building. Figure 3. Select Components of USAID's New Strategy Development and Project Design Processes That May Identify the Need for PFM Capacity Building: [Refer to PDF for image: illustration] Country development cooperation strategy: * Seek to build capacity of institutions and related governance systems at the national, regional, and local levels. * Demonstrate how mission will implement USAID Forward priorities, including working through host country systems and developing capacity of civil society and private sector partners. Project design: * Assess the likelihood of using country systems and conduct PFM risk assessments. * Build in steps to mitigate risks associated with using country systems identified during the PFM risk assessment. * Conduct a mandatory sustainability analysis that will describe how relevant activities or services will gradually be tied to sustainable financing models. * Conduct an institutional analysis to assess local institutions and systems and increase the capacity of these institutions as implementation partners. Source: GAO analysis of USAID documents. Note: USAID's new processes have several steps to identify the need for PFM capacity-building programs. We included the components that pertain most directly to PFM. [End of figure] According to the new USAID guidance on project design, after USAID approves the strategy document and identifies the need for PFM-related assistance, USAID country offices are to begin the project design phase. This phase is to begin with the formation of a design team that is responsible for the project's development from planning to implementation.[Footnote 17] Overall, the project design phase is to comprise three stages, as follows: * Conceptual stage. During the conceptual stage, the project design team is to conduct stakeholder outreach, assessments, analysis, and implementation planning. This stage results in a concept paper, which provides a summary of a proposed project that country-office management can review to assess how the project aligns with the country strategy, the likelihood of success, and the assumptions underlying project success. After USAID management approves the concept paper, the design team transitions to the analytical stage. * Analytical stage. During the analytical stage, the design team seeks to understand the problem or constraints identified during the conceptual stage, and identify and assess critical assumptions. The team is to conduct a series of targeted assessments, including required gender, environmental, and sustainability analyses and other social, political, and institutional analyses. Of these analyses, we identified two that pertain directly to PFM systems, as follows: * Sustainability analysis. The sustainability analysis assesses the partner government's ability to manage the continuation of the project after the project has concluded. According to USAID guidance, to build sustainability into a project, the design should consider how the country office will increase the skills and capacity of local stakeholders involved in maintaining development gains after the project ends--as well as how USAID will ensure that relevant activities or services are gradually tied to sustainable financing models through private sector participation or through sustainable, publicly managed arrangements and government processes. * Institutional analysis. An institutional analysis is an in-depth assessment of the local institutions and systems most critical to the implementation of the project's development interventions, including an assessment of the quality of their leadership, structure, and staff; and identification of their administrative and financial management strengths and weaknesses. Using the analysis, USAID is to develop a plan for project activities that are necessary and sufficient to bring these institutions up to the level of performance or engagement as partners appropriate for their roles in the project's implementation and their eligibility for direct USAID funding. According to USAID guidance, the analytical stage results in a project authorization document that will be the basis for project implementation, adaptation, and evaluation, which includes a summary of the analyses that underlie the rationale for the project design. * Approval stage. A successful project design results in an approved project authorization, which enables a project to move from the planning stage to implementation. The project authorization sets out the purpose and duration of the project, defines fundamental terms and conditions of the assistance when a partner country agreement is anticipated, and approves an overall total budget level, subject to the availability of funds. In addition to the above analyses, USAID has developed a PFM risk- assessment framework (PFMRAF) to measure the fiduciary risk, or the risk of funds being misspent or mismanaged, when USAID plans to provide aid through the country's finance systems. USAID guidance commits country offices to offer, if appropriate, a USAID assessment of partner country PFM systems with the goal of providing funding for project implementation through the use of those systems. If the offer is accepted, the assessment must be carried out using the PFMRAF. Whenever possible, USAID should begin conducting the PFMRAF during the conceptual stage of the project design process. The PFMRAF consists of the following five stages: 1. The Rapid Appraisal provides a high-level snapshot of country-level fiduciary risks associated with the use of partner country PFM systems and helps inform decisions on whether to undertake a more rigorous, formal risk assessment. 2. The Risk Assessment, Analysis, Mitigation, and the Approval for Use of Partner Country Systems establishes the baseline level of risk corresponding to contemplated funding levels and identifies vulnerabilities of the partner country PFM sector in which USAID is considering use of the system for project implementation. USAID determines whether any systemic risk can be reasonably mitigated and, if so, what kind of mitigating measures might be introduced to reduce the risk. USAID establishes an accountability framework with a set of conditions that would, if complied with, constitute formal approval for the use of a partner country PFM system. 3. The Project Design, Approval, Designation of Responsibilities, and Selection of the Funding Mechanism incorporates the approval of use of country systems into the project design, includes risk mitigation measures, such as capacity-building technical assistance, concurrent audits, and disbursements in tranches, as appropriate, and the appropriate funding mechanism. 4. Negotiating and Preparing the Bilateral Project Agreement with the Partner Country Government involves the preparation of the bilateral agreement in accordance with the risk mitigation measures outlined in the approval for use of partner country systems and in collaboration with the partner country government. 5. Implementation, Monitoring, and Evaluation occurs once both countries agree to the bilateral agreement and as outlined in the project design. According to USAID, the country office is to incorporate recommendations on actions to mitigate identified risks to providing assistance directly to the host government into various stages of the project design. According to one USAID official, the country office's decision to implement the recommended steps for mitigating risk may depend on the availability of funds. According to USAID, if risks cannot be mitigated, USAID is not to deliver assistance through the partner country's financial systems. Mitigation steps may include adding a PFM component to a project, such as technical assistance to improve an aspect of the financial system. As of June 2012, USAID had completed the rapid assessment in 20 countries and was in the process of finalizing the report for an additional 12. For 4 of the countries for which USAID had completed the rapid assessment, USAID had decided to either delay or not to proceed to stage 2 for various reasons, including funding cuts and the political situation in the country. For the second stage of the PFMRAF, USAID had completed the risk assessment for 3 countries and was in the process of conducting the assessment for 11 others. USAID had completed four stages of the PFMRAF for 3 countries. Table 2 contains a summary of the status of countries where USAID has begun the PFMRAF process. Table 2: Status of Countries in the PFM Risk-Assessment Framework Process: Stage 1: Complete: 20[A]; Stage 1: In Progress: 12; Stage 2: Complete: 3; Stage 2: In Progress: 11; Stage 3: Complete: 3; Stage 3: In Progress: 0; Stage 4: Complete: 3; Stage 4: In Progress: 0; Stage 5: Complete: 0[B]. Source: USAID. [A] Stage 2 on hold or no plan to continue to stage 2 in four of these countries. [B] The three countries that have completed stage 4 of the process are pending obligation of funds, which will move them to stage 5. [End of table] Treasury's Processes: Treasury OTA's processes for developing PFM programs involve collaboration with country government officials whom Treasury has determined have a strong commitment to reform and are seeking to develop strong PFM, as well as OTA's assessment of the country system. OTA receives requests for assistance from foreign governments, other U.S. government agencies, U.S. embassies, international organizations, and OTA advisors and other donors already working in countries on other projects. According to OTA officials, OTA receives two to three requests for assistance per month, but due to the agency's limited financial resources and the financial commitment required for each project, it selectively chooses its projects. OTA's principles for providing assistance are as follows: * work with partners committed to ownership of their reform; * advocate self-reliance by providing countries with knowledge and skills required to generate and manage their own resources and reduce dependence on international aid; and: * work with ministry staff daily to build capacity through mentoring and on-the-job training on PFM practices. After receiving the request, OTA begins a preliminary assessment process to identify weaknesses and areas for assistance, according to OTA officials. The preliminary assessment is primarily a review of available information on the country's PFM system, which may include information from the U.S. embassy, as well as relevant country reports and assessments prepared by international financial institutions, other bilateral assistance providers, and nongovernmental organizations. After reviewing documents, if it decides to proceed with the assessment, OTA completes a more in-depth, in-country assessment of PFM, which consists of meetings with U.S. embassy staff, other donors, and relevant ministry officials, including the minister of finance or the head of the central bank, as well as high-level policy staff and mid-level supervisors. According to OTA officials, the purpose of the in-country assessment is to review the structural issues related to the country system and determine whether the country has dedicated staff committed to working toward their reform efforts. At any point during the assessment process, OTA may determine that OTA assistance is not suitable for the country's needs. OTA officials told us that in one such case, government officials requested that OTA manage their budget for them, which is against OTA's principles because doing so would not promote self-reliance or help the country develop the capacity to generate and better manage its own revenues. Finally, OTA officials told us that their decision on whether to provide assistance depends on whether sufficient funding is available. The in-country assessment typically results in a draft of official terms of reference, which identifies weaknesses to be addressed during OTA's engagement. According to OTA officials, the terms of reference describes the broad goals of the project and represents agreement between Treasury and the host-country counterpart, the ministry staff with whom the advisor will work on a daily basis. For example, the 2010 terms of reference for a budget project in Honduras identified the following four agreed-upon areas for assistance: * improve operational efficiency and enhance accountability by strengthening the organization of the ministry of finance; * move toward compliance with International Public Sector Accounting Standards; * enhance capacity to conduct fiscal analysis and produce more reliable macroeconomic estimates and revenue and expenditure projections; and: * conduct workshops on basic finance methods and terminology. The terms of reference also identifies the advisor's host-country counterparts and agreements regarding each party's responsibilities. According to OTA officials, after OTA fully vets a project and secures funding, the parties finalize and sign the terms of reference. However, OTA and its counterparts may seek to revise the terms of reference if the central purpose of the technical assistance project changes or during times of transition, such as when a finance minister changes. In Honduras, the 2010 terms of reference replaced an existing OTA project following a gap in OTA assistance during a political crisis in the Honduran government in 2009. The OTA advisor assigned to the country uses the broad goals expressed in the terms of reference to develop a work plan. The work plan contains specific objectives, milestones, and planned completion dates designed to work toward goals agreed upon in the terms of reference, and the plan must be developed within 60 days of the advisor arriving in-country. Each work plan covers a 1-year period and is the primary basis for regular monthly progress reports to OTA headquarters. OTA considers work plans, and the monthly reports on which they are based, to be dynamic documents that reflect the project's progress in real time, and advisors can and should change or modify the work plan during the course of a project as appropriate and in consultation with the ministry staff and OTA management. Moreover, according to OTA officials, a 1-week in-country assessment may not provide all of the information needed and may require that the advisor rewrite the plan after a few months in the country. While the work plan is considered a management tool to monitor projects, OTA officials told us that in recent years it has also become a joint document between the OTA advisor and ministry staff as a shared tool to monitor and discuss progress. USAID and Treasury Use Certain Processes to Monitor and Evaluate PFM- Related Programs, but Implementation Has Some Weaknesses: USAID's Processes: USAID Uses Multiple Tools to Monitor Its PFM Assistance but Lacks a Process for Tracking Use of Country Systems: To monitor PFM-related programs, USAID develops performance management plans, reviews periodic progress reports, and conducts site visits, among other activities, but reviews of USAID programs have identified agency-wide weaknesses in implementation, including using unreliable baseline data and inaccurate reporting of results. USAID develops the monitoring and evaluation framework during the project design, and it generally is to include the following: * A performance management plan: This is a tool to plan and manage the process of monitoring, evaluating, and reporting progress toward achieving the project's development objectives. The performance management plan includes performance indicators and targets that link to the project objectives.[Footnote 18] * A work plan: This specifies activities to be undertaken and the proposed schedule for these activities during the life of the project. [Footnote 19] [Side bar: The following illustrates the key elements that are built into the performance management plan and the work plan for USAID's Growth and Fiscal Stability Initiative Program in Kosovo. Agreement: The USAID/Kosovo program established 3 objectives: Objectives: 1. Build professionalism and expertise in sound PFM in municipalities. 2. Support private sector participation through public-private partnerships. 3. Develop and implement an action-oriented government strategy for profitable private sector growth. The performance management plan incorporated these objectives, along with indicators and targets. For example: Objective 1: Build professionalism and expertise in sound PFM in municipalities. Indicator: Number of PEFA assessments conducted by municipalities. Target: Conduct 5 assessments in year 1 and 10 in year 2. The work plan linked tasks, activities, and timelines to each of the objectives from the performance management plan. For example: Objective 1: Build professionalism and expertise in sound PFM in municipalities: Task: Implement the PEFA framework methodology for self-assessments by municipalities. Activity: Conduct inventory of current PFM practices through PEFA self- assessment. Timeline: First 3 months. End of side bar] * Periodic progress reports: USAID assistance agreements with nongovernmental organizations generally also require implementing partners to submit periodic progress reports, the frequency of which vary depending on the assistance agreement, but these reports may not be required more frequently than quarterly or less frequently than annually. The progress reports should generally contain a comparison of actual accomplishments with the goals and objectives established for the period and reasons why established goals were not met, if appropriate. Implementing partners should immediately notify USAID of developments that have a significant impact on the award-supported activities. * Site visits: USAID guidance states that site visits are conducted as needed by the technical officers, who are responsible for monitoring. USAID works with the implementing partner to produce and approve the work plan and the performance management plan. USAID's technical officer approves the plans. USAID guidance also requires staff to conduct at least one portfolio review each year that covers all activities included in their various programs. These reviews determine, among other things, whether the activities are achieving the desired results. Prior reviews of USAID programs have identified challenges in the agency's implementation of its monitoring processes across many types of programs, including DG and EG programs. In its fiscal year 2011 memo on management and performance challenges, USAID's Inspector General identified specific monitoring-related weaknesses as one of USAID's most serious management and performance challenges. The USAID Inspector General reported problems with assistance planning in 25 out of 80 performance audits conducted in fiscal year 2011. Assistance planning is important because it provides the means for program implementers to track progress toward program objectives and helps to ensure that USAID assistance programs achieve planned results. In addition, 37 of the 80 audit reports the Office of the Inspector General issued in fiscal year 2011 identified cases in which USAID operating units or their implementing partners reported misstated, unsupported, or unvalidated results. In recent reports, we also identified deficiencies in USAID's monitoring practices, including the lack of an integrated plan for monitoring and evaluating nonemergency food aid,[Footnote 20] monitoring practices that do not correspond to agency performance guidelines,[Footnote 21] difficulties in developing meaningful outcome indicators related to building trade capacity, [Footnote 22] undocumented site visits for assistance programs in Burma,[Footnote 23] and lack of performance targets and baseline data for indicators related to PFM efforts in Afghanistan.[Footnote 24] Moreover, in our review of the fiscal year 2011 and 2012 Inspector General's audits of USAID's DG and EG programs, we found monitoring weaknesses cited in 20 of the 32 audit reports.[Footnote 25] The audit reports cited monitoring weaknesses, including unreliable or nonexistent baseline data; performance data weaknesses, such as results that were not reported, lack of data verification, or inaccurate reporting of data; lack of a current performance management plan; and inadequate monitoring of program activities, including lack of regular submission of progress reports. With regard to our three case study countries, we found that the country offices generally applied USAID's monitoring processes in all three PFM-related programs; however, we did find some conditions that could make monitoring more difficult, as described below. Our review was not intended to be comprehensive or applicable to all USAID PFM- related programs. * Performance management plans: All three programs were clearly defined in terms of overall objectives, project objectives, tasks, expected results or outcomes, and required plans and reports. All three programs--the Kosovo Growth and Fiscal Stability Initiative, the Peru ProDecentralization Project, and Liberia's Governance and Economic Management Support Program--had performance management plans containing objectives and indicators. Two of the three performance management plans also included targets. However, the targets for Liberia's program had not been determined 9 months into the program. [Footnote 26] * Work plans: All three programs also had work plans that specified tasks and timelines. However, Liberia's Governance and Economic Management Support Program initially operated without an approved work plan, although it did establish an action plan to guide its activities [Footnote 27] * Periodic progress reports: Assistance agreements for all three USAID programs required quarterly progress reports, and the implementing partners submitted all reports according to agreed-upon time frames. In our review of the three programs, we found that the language used to describe the objectives and results in the quarterly progress reports did not correspond to the language in the work plans, which could make it difficult to track progress against the work plans' objectives. For example, the quarterly reports for the Kosovo Growth and Fiscal Stability Initiative Program describe progress on five separate lines of effort under the objective to improve fiscal stewardship, while the statement of work and work plan discuss only three lines of effort under the same objective. * Site visits: USAID officials responsible for monitoring progress on all three projects said that they do not rely solely on progress reports to track progress, and that additional monitoring tools included weekly progress meetings, frequent site visits, and telephone and e-mail communications with implementers. For example, the USAID official in Kosovo reported that he works on a daily basis with the implementing partners who produce the quarterly and annual reports and that they exchange e-mails and phone conversations daily. See appendix IV for descriptions of USAID PFM-related assistance programs from our case study countries. In addition to USAID's reported program monitoring weaknesses, we found that the agency does not have a process to fully identify and measure its use of country systems. USAID has set an agency target of obligating 30 percent of the agency's annual assistance through local systems, including both partner country PFM systems and local not-for- profit and for-profit organizations, by fiscal year 2015. [Footnote 28] However, USAID currently cannot track progress toward its 30 percent target because its accounting system cannot identify the full range of such assistance, which includes a variety of implementing mechanisms from host-country contracting to direct cash transfers. The project data in USAID's accounting system does not provide sufficiently detailed information, such as the location of the organization receiving assistance, to identify qualifying assistance. According to a senior USAID official, the agency is working on a system whereby it will tag each entity receiving assistance with an identifier, such as government or not-for-profit organization, as well as a vendor location signifying U.S.-based or non-U.S. based. According to the official, this tagging process will facilitate identifying use of country systems. However, USAID's financial system currently cannot distinguish whether non-U.S. based not-for-profit and for-profit organizations are located in the host country or in a third- party country. USAID Uses Evaluations to Assess the Effectiveness of Its Programs, but Weaknesses in Its Practices Have Been Reported: USAID's process to assess the effectiveness of its PFM-related programs involves independent evaluations, but weaknesses in the agency's overall evaluation practices have been reported. USAID may use independent external evaluations in the middle or at the end of a program based on the need to inform program decisions. USAID adopted a new evaluation policy in January 2011 and updated its guidance in February 2012 to reflect it. The new evaluation policy requires that all large projects have at least one evaluation and that evaluations use methods that generate the highest-quality and most credible evidence, including experimental methods.[Footnote 29] The policy also states that 3 percent of program budgets should be devoted to external evaluation and classifies two types of evaluations: * performance evaluations, which focus on descriptive and normative questions and other questions pertinent to program design, management, and operational decision making; and: * impact evaluations, which define a counterfactual (what would have happened had the program not been implemented) to control for external factors and measure the change in a development outcome that is attributable to a defined intervention. USAID reported that experimental methods generate the strongest evidence for impact evaluations; however, experimental methods and impact evaluations may be difficult to apply to PFM capacity-building efforts.[Footnote 30] For example, USAID evaluations of PFM programs have highlighted difficulties associated with conducting impact evaluations, including lack of data or resources, and too short a time period to identify impact.[Footnote 31] A February 2012 USAID report on the first year of implementation of the new evaluation policy noted that USAID had not yet completed an evaluation under the new policy. [Footnote 32] USAID adopted the new policy to address a number of weaknesses it had identified in its evaluation practices. USAID reported that the number of evaluations submitted to USAID's Development Experience Clearinghouse--its main repository for agency evaluations--decreased from nearly 500 in 1994 to approximately 170 in 2009 despite an almost three-fold increase in program dollars managed. Furthermore, according to USAID, the majority of evaluations in recent years relied heavily on anecdotal information and expert opinion, rather than systematically collected evidence. We have also reported weaknesses in USAID evaluation practices in several areas in recent years, including not planning for evaluations or not using evaluation results.[Footnote 33] Furthermore, the weaknesses in performance indicators that the USAID IG identified in DG and EG programs could create difficulties for evaluating PFM programs. For example, weaknesses in baseline data or performance indicators would make it difficult to use quantitative measures to evaluate these programs. We reviewed external evaluations from our case study countries to illustrate how evaluations have been applied to PFM programs and noted a number of program accomplishments as well as examples of methodological weaknesses.[Footnote 34] For example, an external team conducted a final evaluation of the Governance and Economic Management Assistance Program in Liberia in 2010 and found that the program made corrupt practices more difficult, improved the accuracy of the budget, and exerted central control over governmental processes.[Footnote 35] However, the evaluation team was hampered by insufficient time for data collection and analysis and a disorganized monitoring and evaluation framework. In addition, in Kosovo in 2009, an external team conducted a midterm evaluation of the Economic Management for Stability and Growth program and found that the program improved the government's institutional capacity in fiscal and monetary policy. [Footnote 36] However, the team did not document its evaluation methodology. Treasury's Processes: Treasury's OTA Uses Various Processes to Monitor PFM-Related Programs: Treasury's OTA uses various processes to monitor PFM-related programs, including narrative monthly and trip reports, annual quantitative performance measures, and customer feedback surveys, but weaknesses exist in some of its evaluation processes. The work plan is the key document OTA uses to monitor a project. OTA uses monthly reports to document and monitor progress toward the milestones identified in the annual work plan, including changes to milestones or timelines, and to facilitate communication between the resident advisors and headquarters. OTA advisors may also share monthly reports with host- country counterpart institutions, U.S. embassy staff, other Treasury bureaus and offices, and other interested partners such as USAID, the Millennium Challenge Corporation, and relevant international financial institutions to monitor progress and coordinate donor assistance as relevant. In addition to updates on work-plan progress, the monthly reports may also include other information such as activities completed; significant meetings held; and important political, social, and economic developments. OTA also uses monthly reports to track changes made to the work plan agreed to by the advisor and his or her counterpart. For example, an OTA budget project in Cambodia listed implementation of program budgeting as one its primary objectives and improving budget classification as one its secondary objectives. In the monthly reports, the resident advisor documented the need to refocus the reform efforts on budget classification to better align with the Cambodian government's ongoing PFM reform efforts. Further, in the case of OTA's 2011 Honduras budget project to implement international public sector accounting standards, the objectives remained unchanged, but milestones were changed in the monthly reports. According to the advisor, some milestones had to be extended due to overly ambitious initial expectations that were purposely established to encourage the ministry staff to undertake reforms. See appendix IV for descriptions of OTA-funded PFM projects in our case study countries. OTA headquarters also maintains regular communication with its advisors. Senior OTA officials monitor project performance through regular contact with advisors in the field by e-mail, telephone, and site visits. One advisor told us that their director would already be aware of any issues raised in the monthly reports, given the frequency of their communications. Resident advisors also told us that senior Treasury staff visited Cambodia and Honduras to review OTA projects and talk with counterparts. Following site visits, OTA officials are required to prepare trip reports. For example, in a trip report from November 2011, a senior OTA official reported that he met with host- country counterparts to discuss the four current OTA engagements in Cambodia, including the budget project. The visit confirmed the reported difficulties in implementing the government financial management information system as a result of a procurement delay, but also affirmed OTA's role in its eventual implementation. OTA also uses voluntary customer surveys to collect feedback on OTA projects from the host-country counterpart. For example, a Honduran government official who completed the voluntary customer feedback survey for the budget project indicated that the project met expectations and made a significant contribution by strengthening the host-country staff's technical capacity through the training sessions that OTA staff offered. We found that OTA applied its monitoring processes to all three of our case study country projects, but monthly reports lacked some details on progress. We reviewed OTA's monitoring activities in three PFM- related projects to examine how OTA applied its monitoring process to these projects. Our review was not intended to be comprehensive or applicable to all OTA projects. We found the projects to be clearly defined with specific objectives, milestones, proposed completion dates, and regularly submitted monthly reports. In addition, in all three cases, the resident advisor reported frequent communication on project progress with headquarters. However, in two of the case study countries, Cambodia and South Africa, work plans had major changes in objectives, but only Cambodia documented its change in a monthly report. Additionally, our review found that the level of detail of the monthly report narratives on progress varied, with a few reports having little detail, which could make tracking progress for these specific projects difficult without other communication between the advisor and OTA management. However, as noted above, OTA headquarters maintained frequent communication with advisors in the field. Treasury OTA's Performance Measures Are Useful Project-Level Indicators but Have Weaknesses: OTA's quantitative performance measures have been a useful project- level indicator of performance but have not been a useful indicator of OTA-wide performance because of conceptual problems and errors introduced by OTA when aggregating the performance data. OTA advisors and management score technical assistance projects annually based on project-specific indicators under four main categories.[Footnote 37] The projects are scored on a scale of one (lowest) to five (highest) for "traction," or the degree of engagement with host-country counterparts, and "impact," or the results of project activities that bring about change in counterpart law, systems, processes, and procedures. OTA officials told us that the traction and impact measures also reflect the language and values of OTA, and are useful to management, advisors, and host-country counterparts. The primary purpose of the quantitative performance scores was to respond to an Office of Management and Budget reporting requirement for evaluation data. While OTA reported that it met or exceeded targets for traction and impact set in 2008, in every year from 2009 through 2011, OTA officials acknowledged that the aggregate values associated with its annual goals were of limited value due to lack of comparability across programs and over time. In previous work on the Office of Management and Budget, we highlighted the difficulty of representing program performance with a single rating.[Footnote 38] Using a single rating can force agencies to simplify more nuanced and complex performance results, a circumstance similar to OTA's aggregation of traction and impact scores. A senior official told us that OTA, in complying with the OMB requirement, designed the measures to be as useful as possible at the project level. For example, OTA uses the project-level traction and impact measures in setting expectations and discussing progress with both the host-country counterpart and the advisor. In addition, the official reported that OTA is continually looking for additional ways to use the traction and impact scores. One example of how researchers have used PFM performance data is by developing analytical approaches to identify the determinants of strong PFM systems and elements of successful PFM reform efforts, which can help to control for important differences across countries.[Footnote 39] Our analysis of OTA performance data found several errors that OTA introduced when aggregating project-level data; these errors collectively raised questions about the reliability of the instrument used to aggregate OTA's quantitative performance measures and, thus, suggested there may be limitations in its ability to provide insight into performance across OTA projects. OTA uses this instrument to calculate the annual performance averages which are compared against annual targets and provided to OMB and to Congress in annual reports. Our analysis of the spreadsheet containing these data, and limited spot checking of some underlying data for our three case-study countries, identified a number of errors, including those introduced when transcribing data to the spreadsheet. In one instance, OTA listed a performance score under the wrong country, which resulted in inaccurate information for two countries' projects. OTA Does Not Fully Evaluate Technical Assistance after Completion: OTA has not yet fully implemented its requirement to conduct independent evaluations--that is, evaluations conducted by someone other than the resident advisor--of its completed technical assistance. OTA guidance for project reporting and documentation includes requirements for end-of-tour reports when a resident advisor leaves and end-of-project reports when a technical assistance project is completed. The end-of-project report is a postproject evaluation whose purpose is to (1) compare accomplishments with the initial objectives of the project and (2) improve the planning and execution of future projects. A senior OTA official said that the requirement for the end-of-tour report was fully implemented and that OTA intended to fully implement the end-of-project report requirement by the end of 2012. According to the guidance, the evaluation must be conducted independently; the report is the responsibility of the associate director and can be delegated to a senior advisor, but it should not be prepared by an advisor who prepared an end-of-tour report. Both OECD guidance and USAID guidance also highlight the importance of independence in evaluation. In contrast to the guidance, OTA provided three end-of-project reports that were conducted by OTA staff who had been involved in providing some of the technical assistance being evaluated. While no comprehensive and fully independent end-of-project evaluation has been conducted, OTA officials described relevant insights from a postproject trip to one country that was undertaken by a senior OTA official in part to better understand the longer-term impact of OTA assistance. During the trip, that official identified some factors that significantly limited the impact of this project, notably a lack of counterparty commitment at the policy level to implement and sustain reforms in key areas identified in the terms of reference. Conclusions: As more donors, including the United States, seek to provide additional assistance through countries' systems, the need for strong PFM systems and accountability in recipient countries become even more important to help lessen corruption and ensure countries effectively manage resources. Given USAID's stated goal of obligating 30 percent of annual assistance through local systems, including partner country PFM systems, by 2015, it must ensure that those systems are functioning properly and transparently. Efforts to strengthen PFM systems can lower the risks that assistance delivered through country systems will be misspent, while also increasing the capacity of the recipient country to effectively manage its own resources. Given concerns about transparency and corruption in many aid-recipient countries, achieving the expected benefits of using recipient countries' PFM systems may be difficult without concerted efforts by the donors and countries to strengthen these systems. In addition, the risks to using these systems are increased without efforts to strengthen them. To help strengthen these systems, USAID provides developing countries with PFM capacity-building assistance. In light of this assistance, and given the difficulties USAID has experienced in the past with implementing monitoring and evaluation practices, the importance of developing efforts to strengthen PFM systems and effectively monitoring and evaluating these efforts has increased significantly. Previous difficulties USAID has experienced with regard to monitoring, including poor quality of reported data on projects, and the lack of reliable baseline data, could affect USAID's ability to conduct effective evaluations of these projects, as evaluators will not have access to reliable data. Also, while USAID's new evaluation policy places greater emphasis on impact evaluations and experimental methods, it may be difficult for USAID to evaluate its PFM capacity- building efforts using these approaches. Finally, for USAID to achieve its target for use of local systems, by fiscal year 2015, it must be able to identify and measure the full extent of assistance that qualifies. Treasury OTA advisors provide technical assistance targeted at weaknesses in countries' PFM systems. Although OTA has taken numerous steps to monitor this assistance, errors in its aggregated performance data and the lack of comprehensive postproject evaluations limit OTA's ability to effectively evaluate its assistance. OTA has adapted a challenging mandate from OMB to create a useful measure of its efforts' performance for management, advisors, and host-country counterparts; however, conceptual problems with and errors in the aggregated measures undermine the measures' reliability. With greater confidence in the quality of the data, opportunities exist to better identify patterns in performance across OTA programs, such as economic or institutional factors that influence program performance. These patterns could help OTA better understand the strengths and weaknesses of its assistance programs and make appropriate changes. OTA could also use the quantitative performance measurement system it has developed to experiment and document the results of new approaches. Finally, although OTA guidance recognizes the importance of postproject evaluations by including in its guidance the requirements for an end-of-project evaluation, the agency has yet to enforce this requirement. Without a postproject evaluation OTA may not fully understand results of its technical assistance or be able to apply lessons learned to new projects. Recommendations for Executive Action: To monitor progress toward USAID's target to obligate 30 percent of its annual assistance through local systems by 2015, we recommend that the Administrator of USAID direct the appropriate offices to develop a process to reliably identify and track the agency's use of local systems in all countries receiving assistance. To help ensure that USAID conducts effective evaluations of PFM- related programs under its new policy, we recommend that the Administrator of USAID direct the appropriate offices to ensure that they are establishing adequate monitoring practices for PFM-related programs. Such practices may include selecting proper indicators, collecting reliable baseline data, and ensuring the reliability of reported results. To improve the effectiveness of OTA's technical assistance, we recommend that the Secretary of the Treasury direct OTA to take the following two actions to improve monitoring and evaluation: * implement additional controls to improve the process for computing OTA-wide annual performance measures, and: * fully implement OTA's existing requirement for end-of-project evaluations and, consistent with its existing guidance, have an independent party conduct the evaluations. Agency Comments and Our Evaluation: We provided a draft of this report to USAID, Treasury, and State for their review and comment. Both USAID and Treasury concurred with our recommendations in their written comments, which are reproduced in appendix V and appendix VI respectively. These agencies along with State provided technical comments and updated information, which we have incorporated throughout this report, as appropriate. In concurring with our two recommendations, USAID reported that it is in the process of refining definitions that will identify and help measure the assistance that qualifies to meet the agency's target of obligating 30 percent of its annual assistance through local systems by 2015. USAID also reported that it has implemented two complementary reforms that will help ensure effective evaluations and adequate monitoring of its PFM assistance. The first reform involves USAID planning for the monitoring and evaluation of assistance during the early stages of project design, including defining indicators and collecting baseline data. The second reform requires USAID to plan final monitoring and evaluation schedules during project design. In concurring with our two recommendations, Treasury reported that OTA has corrected several errors in the 2011 annual performance measures, and has taken steps to strengthen data controls, including conducting additional reviews and increasing staff resources dedicated to computing the performance measures. In addition, OTA has begun to implement its requirement for independent end-of-project evaluations of its technical assistance and intends to fully implement the requirement by the end of 2012. We are sending copies of this report to appropriate congressional committees. We are also sending copies of this report to the Administrator of USAID and the Secretaries of the Treasury and State. In addition, this report is available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this reported are listed in appendix VII. Sincerely yours, Signed by: Thomas Melito: Director, International Affairs and Trade: [End of section] Appendix I: Scope and Methodology: To examine the process the U.S. Agency for International Development (USAID) uses to develop programs to strengthen public financial management (PFM) systems, we reviewed USAID's official policy and procedures, as contained in the Automated Directives System, as well as USAID's new guidance documents for developing country strategies and designing projects, including the Country Development Cooperation Strategy and Project Design Guidance. We also interviewed USAID officials, including officials from the democracy and governance (DG) and economic growth (EG) offices. Most USAID country offices are required to have an approved country strategy by the end of fiscal year 2013 and to follow the new project design guidance starting July 2012. Because USAID was still in the process of transitioning to these new processes during the course of our audit, we did not review USAID's implementation of these processes. To examine the process the U.S. Department of Treasury's Office of Technical Assistance (OTA) uses to develop programs to strengthen PFM systems, we interviewed senior OTA officials regarding OTA's policies and procedures. We interviewed resident advisors regarding how OTA assessed and developed projects in our three OTA case study countries. We also reviewed OTA's official policy guidance document and project- specific documents, including assessment reports, signed terms of reference, work plans, and monthly reports. To examine and assess the processes USAID uses to monitor and evaluate its PFM-related programs, we reviewed USAID program development guidance; monitoring and evaluation policies and procedures; USAID reports; and program documents, including assistance agreements, monitoring and evaluation plans, and progress reports; USAID external project evaluations; past GAO reports; and USAID Inspector General reports. We used the review of the documents for our three USAID country case studies to illustrate the implementation of USAID's monitoring processes for its PFM-related programs. We also interviewed agency officials in Washington, D.C., and conducted telephone interviews and had e-mail communications with key country-office staff for each of our case study countries. To examine and assess OTA's processes for monitoring and evaluating PFM-related programs, we reviewed OTA project reporting and documentation instructions, project work plans, and monthly reports, end-of-tour and end-of-project reports, annual quantitative performance data, Organization for Economic Cooperation and Development guidance on evaluation, and conducted interviews with OTA headquarters staff. We used the review of the documents for our three OTA case studies to illustrate the implementation of OTA's monitoring processes. We supplemented the document review with interviews with current or former advisors for each of the case study countries. In addition, we assessed the reliability of the instrument OTA uses to aggregate quantitative performance measures across projects. We examined spreadsheets provided by OTA for consistency, examined data for outliers and missing values, and spot-checked the transcription of data to the spreadsheet for our case study countries. Due to a number of errors in the OTA data, we could not determine if the aggregated performance data were sufficiently reliable for identifying patterns in performance across projects or over time. In selecting our case study countries, we focused on countries in which OTA or USAID had relevant ongoing or recently completed projects designed to strengthen PFM systems. We selected six countries: Cambodia, Honduras, and South Africa for OTA, and Kosovo, Liberia, and Peru for USAID. In selecting these countries, we considered the following two factors: * Geographic diversity: For each agency, we selected countries from three different geographical regions. * Country income group diversity: For each agency, we chose a country that the World Bank has listed as (1) lower income, (2) lower-middle income, and (3) upper-middle income in order to report examples from different income levels, which may also be associated with different institutional characteristics. In cases where more than one country would be acceptable under our decision criteria, we considered additional criteria, such as the availability of other broad-based indicators. For OTA, we focused our selection on countries receiving technical assistance from OTA's Budget and Financial Accountability team, given its focus on traditional PFM aspects. For USAID, we selected our countries from a list of countries with significant PFM-related programs that USAID provided. USAID excluded some countries with PFM-related programs from the list because staff were not available to discuss their programs with us. Our review of USAID and OTA case study countries was not intended to be comprehensive or applicable to all their respective programs and projects or generalizable to all countries. To describe recent trends in country PFM systems, we reviewed the data and publications of five international organizations that conduct broad assessments of country PFM systems. These broad assessment tools include the Public Expenditure and Financial Accountability Program, the World Bank's Country Policy and Institutional Assessment's quality of budgetary and financial management indicator, International Budget Partnership's Open Budget Survey, Transparency International's Corruption Perceptions Index, and the International Monetary Fund's fiscal transparency Reports on the Observance of Standards and Codes. We converted our six case study country scores for the three PFM diagnostic tools for which scores were available (Open Budget Survey, Corruption Perceptions Index, and quality of budget indicator) into percentile rankings to illustrate each country's performance as measured by the three PFM diagnostic tools. We also interviewed officials at the World Bank and International Monetary Fund to discuss their PFM-related diagnostic tools. To describe the Department of State's PFM-related efforts, we conducted interviews with agency officials in Washington, D.C. We reviewed State documents, including agency guidance, waiver packages, and program documents. We also reviewed relevant appropriations laws to identify the requirements for State's fiscal transparency reviews. We conducted this performance audit from October 2011 through September 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Several International Organizations Reported Results of Their Public Financial Management Diagnostic Tools: Five international organizations --the Public Expenditure and Financial Accountability Program, the World Bank, International Budget Partnership, Transparency International, and the International Monetary fund--have developed tools to assess various aspects of countries' public financial management (PFM) systems, and some have published recent findings or results. We illustrate the percentile ranking for each of our case study countries in three broad PFM diagnostic tools - the Open Budget Survey, the Corruption Perceptions Index, and the World Bank's quality of budgetary and financial management indicator. Public Expenditure and Financial Accountability Program's PFM Performance Measurement Framework: In December 2001, a multiagency partnership founded the Public Expenditure and Financial Accountability Program (PEFA) to strengthen the ability of aid recipients and donors to assess and improve country public expenditure, procurement, and financial accountability systems. [Footnote 40] In 2005, the program developed the PFM Performance Measurement Framework, known as the PEFA Framework, to provide a measure of the strengths and weaknesses of a country's PFM system. The PEFA Framework incorporates a PFM performance report, which includes an assessment of the evaluated country's PFM performance along six core dimensions of PFM. The six dimensions are the following: * Credibility of the budget: The budget is realistic and is implemented as intended. * Comprehensiveness and transparency: The budget and the fiscal risk oversight are comprehensive, and fiscal and budget information is accessible to the public. * Policy-based budgeting: The budget is prepared with due regard to government policy. * Predictability and control in budget execution: The budget is implemented in an orderly and predictable manner, and arrangements for the exercise of control and stewardship in the use of public funds exist. * Accounting, recording, and reporting: Adequate records and information are produced, maintained, and disseminated to meet decision-making control, management, and reporting purposes. * External scrutiny and audit: Arrangements for scrutiny of public finances and follow up by executive are operating. The six core dimensions include 28 high-level indicators, each of which is assigned a letter score from A to D. The initial assessment helps establish performance baselines, while repeat assessments help in monitoring performance progress over time. In 2011, the PEFA secretariat released the 2010 PEFA Monitoring Report. Based on a comparison of 33 repeat PEFA assessments over the 2005-2010 period, the analysis shows that more countries had a higher number of highest or improved scores (23 countries) than lowest or worsened scores (8 countries), indicating a broad trend of PFM improvement across the countries surveyed.[Footnote 41] According to the analysis, while PFM systems are improving overall, systems features vary significantly. Formal PFM features where progress can be achieved through adopting a new law, regulation, or technical tool, or focusing on no more than a few agencies, or at an early stage in the budget cycle are more likely to improve or maintain a high score than functional PFM features where progress requires actually implementing a new law or regulation, or coordinating the work of many agencies, or working later in the budget cycle. Starting July 2012, the PEFA partners are extending the PEFA program by 5 years and conducting a comprehensive review of the PEFA Framework, the first since it was launched in 2005. One of the objectives is to improve confidence in PEFA assessments through an endorsement process that provides an incentive to ensure adherence to PEFA good practices in undertaking an assessment. World Bank's Quality of Budgetary and Financial Management Indicator: The World Bank undertakes annual Country Policy and Institutional Assessments to assess the quality of a country's present policy and institutional framework. One of the 16 indicators the World Bank uses to assess the performance of a country's current policies and institutions is the quality of budgetary and financial management. This criterion assesses the extent to which the country has a comprehensive and credible budget linked to policy priorities; effective financial management systems to ensure that the budget is implemented as intended in a controlled and predictable way; and timely and accurate accounting and fiscal reporting. Over the 5-year period from 2005 through 2010, 26 out of 73 countries, or slightly more than one-third, showed improvements in the quality of their PFM systems, while 19 countries' scores worsened.[Footnote 42] Most countries, 62 percent in 2010, are clustered in the mid-range. [Footnote 43] In addition to the quality of budgetary and financial management indicator, the World Bank uses other broad PFM diagnostic tools, including the Country Financial Accountability Assessments and the Public Expenditure Reviews. The objective of the Country Financial Accountability Assessments is to support the World Bank's development objectives by identifying strengths and weaknesses in country PFM systems. The assessments are intended to help identify priorities and inform the design and implementation of capacity-building programs. The assessments also describe and analyze financial management and expenditure controls, including expenditure monitoring, accounting and financial reporting, internal controls, internal and external auditing, and legislative review. Information obtained from the assessments, taken together with that obtained from other World Bank diagnostic products and other sources, supports the preparation of an integrated fiduciary assessment. The results of these assessments inform the preparation of the World Bank's Country Assistance Strategy, particularly the sections dealing with the size of the support program, the sectors to be supported, selection of lending instruments, and approaches to risk management. The assessments are particularly important where World Bank resources are managed by the country's own PFM system, as in the case of budget support. The Public Expenditure Review's objectives are to strengthen budget analysis and processes to achieve a better focus on growth and poverty reduction, and to assess public expenditure policies and programs to provide governments with an external review of their policies. Public Expenditure Reviews may also address the incentives and institutions needed to improve the efficacy of public spending in major sectors such as health and education. International Budget Partnership's Open Budget Survey: The International Budget Partnership's Open Budget Survey assesses the availability in each country of eight key budget documents, as well as the comprehensiveness of the data contained in these documents.[Footnote 44] The survey also examines the extent of effective oversight provided by legislatures and supreme audit institutions, as well as the opportunities available to the public to participate in national budget decision-making processes. In its 2010 survey of budget transparency in 94 countries, Open Budget concluded that the state of budget transparency is poor. Only a minority of countries, 20 out of the 94, or 21 percent, can be considered to have open budgets, while a larger number of countries, 41, or 44 percent, provide minimal to scant budget information. The following figure includes the 2010 Open Budget scores for 5 of our 6 case study countries.[Footnote 45] (See figure 4.) Figure 4: Open Budget 2010 Index Scores: [Refer to PDF for image: horizontal bar graph] Extent of information provided: Score: 81-100 (Extensive); Number of countries in each category: 7 (South Africa: 92). Extent of information provided: Score: 61-80 (Significant); Number of countries in each category: 13 (Peru: 65). Extent of information provided: Score 41-60 (Some); Number of countries in each category: 33. Extent of information provided: Score: 21-40 (Minimal); Number of countries in each category: 19 (Liberia: 40). Extent of information provided: Score: 0-20 (Scant or no information); Number of countries in each category: 22 (Cambodia: 15; Honduras: 11). Source: GAO analysis of the 2010 Open Budget Index. Note. The Open Budget Index assigns countries covered by the survey a transparency score on a 100-point scale. Scores assigned to questions in the Open Budget Survey are used to determine an overall transparency score for each country surveyed. These scores are then compiled to create the Open Budget Index, providing a ranking of each country's relative level of transparency. [End of figure] Nonetheless, the survey finds the trend toward open budgets is favorable based on substantial improvements in budget transparency, especially among countries that provided little information in the past. Some of these governments, especially those that scored very low in earlier rounds of the Open Budget Index, largely achieved these improvements by taking one basic and inexpensive step: They began to make available on their web-sites the budget documents that they previously produced but had made available only to internal government audiences or to donors. According to the Open Budget Survey, countries performing poorly on the Open Budget Index tend to share certain characteristics, such as low levels of income, low levels of democracy, and geographical location in Africa and the Middle East. Transparency International's Corruption Perceptions Index: Transparency International's Corruption Perceptions Index ranks countries according to the perception of corruption in the public sector.[Footnote 46] The Corruption Perceptions Index is an aggregate indicator that combines different sources of information about corruption, making it possible to compare countries. Broadly speaking, the surveys and assessments used to compile the index include questions relating to bribery of public officials, kickbacks in public procurement, embezzlement of public funds, and questions that probe the strength and effectiveness of public sector anticorruption efforts. Transparency International's Corruption Perceptions Index shows that nearly three-quarters of the 178 countries in the index score below 5, on a scale from 10 (very clean) to 0 (highly corrupt). These results, shown in figure 5, indicate a serious corruption problem, according to Transparency International. Figure 5 includes the scores of our six case study countries. Figure 5: Transparency International 2010 Corruption Perceptions Scores: [Refer to PDF for image: horizontal bar graph] Highly corrupt (1) to Very clean (10). Average score: 4.5. Corruption Perceptions Index score: 9.0-10; Number of countries in category: 5; Corruption Perceptions Index score: 8.0-8.9; Number of countries in category: 9. Corruption Perceptions Index score: 7.0-7.9; Number of countries in category: 9. Corruption Perceptions Index score: 6.0-6.9; Number of countries in category: 9. Corruption Perceptions Index score: 5.0-5.9; Number of countries in category: 15. Corruption Perceptions Index score: 4.0-4.9; (South Africa: 4.5); Number of countries in category: 19. Corruption Perceptions Index score: 3.0-3.9; (Peru: 3.5; Liberia: 3.3); Number of countries in category: 38 Corruption Perceptions Index score:2.0-2.9; (Kosovo: 2.8; Honduras: 2.4); Cambodia: 2.1); Number of countries in category: 43 Corruption Perceptions Index score: 1.0-1.9; Number of countries in category: 11. Source: GAO analysis of Transparency International's 2010 Corruption Perception Index. [End of figure] According to Transparency International, the Corruption Perceptions Index is not suitable for trend analysis or for monitoring changes in the perceived levels of corruption over time for all countries as year- to-year changes in a country's or territory's score can result from a change in the perceptions of performance, a change in the ranking provided by original sources, or changes in the methodology resulting from Transparency International's efforts to improve the index. However, using specific criteria, Transparency International determined that nine countries showed an improvement, and seven countries showed deterioration from 2009 to 2010.[Footnote 47] International Monetary Fund's Fiscal Transparency Reports on the Observance of Standards and Codes: The International Monetary Fund (IMF) developed a Code of Good Practices on Fiscal Transparency (the Code) in 1998 to help ensure that governments are providing a clear picture of the structure and finances of government. The Code, updated in 2007, focuses particularly on transparency and accountability, and is based on four principles: clarity of roles and responsibilities, open budget processes, public availability of information, and assurances of integrity. IMF's fiscal transparency Reports on the Observance of Standards and Codes is a voluntary fiscal transparency assessment program intended to assess member countries' observance of the good fiscal transparency practices in the Code and recent achievements in fiscal transparency; identify and prioritize reforms necessary to enhance the openness of accountability of public institutions; and in resource-rich countries, assess and advise on revenue transparency issues. Case Study Country Performance Based on PFM Diagnostic Tools: Figure 6 illustrates the percentile ranking for each of our six case study countries in three broad PFM diagnostic tools: the Open Budget Survey, the Corruption Perceptions Index, and the World Bank's quality of budgetary and financial management indicator. Percentiles measure position from the bottom, so the percentile rank indicates the percent of cases falling at or below that score. A higher percentile represents a better relative standing. For example, Kosovo's ranking on the Corruption Perceptions Index is at the 36th percentile, meaning that Kosovo's standing is as high or higher than 36 percent of the other 178 countries in the index. Figure 6. Case Study Country Percentile Rankings on PFM Diagnostic Tools: [Refer to PDF for image: vertical bar graph] Percentile rankings: Country: Cambodia; 2010 Open Budget Survey: 20%; 2010 Corruption Perception Index: 8%; 2010 Budget and Financial Management Indicator: 39%. Country: Honduras; 2010 Open Budget Survey: 14%; 2010 Corruption Perception Index: 20%; 2010 Budget and Financial Management Indicator: 39%. Country: Kosovo; 2010 Open Budget Survey: 0%; 2010 Corruption Perception Index: 36%; 2010 Budget and Financial Management Indicator: 82%. Country: Liberia; 2010 Open Budget Survey: 43%; 2010 Corruption Perception Index: 50%; 2010 Budget and Financial Management Indicator: 5%. Country: Peru; 2010 Open Budget Survey: 85%; 2010 Corruption Perception Index: 53%; 2010 Budget and Financial Management Indicator: 0%. Country: South Africa; 2010 Open Budget Survey: 100%; 2010 Corruption Perception Index: 69%; 2010 Budget and Financial Management Indicator: 0%. Source: GAO analysis of the 2010 Open Budget Index, Corruption Perception Index, and the World Bank's budgetary and financial management indicator. Note: Kosovo was not in the 2010 Open Budget Survey. World Bank budgetary and financial management indicator scores are not available for Peru and South Africa because the World Bank does not disclose Country Policy and Institutional Assessment scores for countries that are not eligible for funding from the World Bank's International Development Association, which provides interest-free loans and grants to governments of the poorest countries. [End of figure] [End of section] Appendix III: State Department Conducts Fiscal Transparency Reviews and Administers the Domestic Finance for Development Initiative: State Department Conducts Two Public Financial Management-Related Activities: The State Department (State) does not directly fund public financial management (PFM) programs, but its Office of Monetary Affairs (OMA) is responsible for two PFM-related activities. First, since 2008, U.S. appropriations laws have required State to evaluate the fiscal transparency of foreign governments receiving U.S. assistance.[Footnote 48] When State determines it is important to the United States' national interest, it may approve waivers for countries that are deemed to be nontransparent that allow U.S. agencies to provide assistance to these countries in accordance with the legal prohibition against providing such assistance without waivers. State processed 28 such waivers in 2011. Second, State supports the presidential initiative called Domestic Finance for Development (DF4D), which President Obama announced in 2011. Under this initiative, State is to help countries use their own resources and leverage other donor resources to meet development goals. State is piloting this initiative in five countries: El Salvador, Honduras, Kyrgyzstan, Tunisia, and Zambia. The goal of DF4D is to strengthen the commitment to reform within partner countries; provide technical assistance in partner countries, such as taxation expertise, including through innovative public-private partnerships; and elevate the importance and interrelation of domestic resource mobilization, fiscal transparency, and anticorruption efforts in public finance as key components for sustainable economic development. Because State has not funded programs in the past for this initiative, State has reached out to other organizations operating in these countries, including international and bilateral organizations, to leverage their programs and resources. State Processes for PFM Activities Derive from Legal Requirements and Presidential Initiatives: State has developed processes for carrying out its PFM-related activities required by legislative mandate and presidential initiative. Under the Fiscal Transparency Review Process (FTRP), OMA reviews central governments expected to receive bilateral economic assistance and international security assistance for several dimensions of transparency: publicly disclosed budget, budget breakdown by ministry, standards for awarding natural resource contracts, and timely and accurate documents.[Footnote 49] Each year, OMA reviews all countries it deemed nontransparent in the prior year, and countries where recent events may have affected fiscal transparency to evaluate whether they meet the threshold of fiscal transparency. State uses the IMF's definition of fiscal transparency as a guideline for the FTRP.[Footnote 50] During this process, OMA officials said they obtain information about the level of transparency of each country by collecting information from U.S. embassy staff working in those countries. In addition, OMA staff review publicly available reports published by international organizations and civil society representatives, such as the World Bank, International Monetary Fund, International Budget Partnership, and the World Economic Forum. Because these organizations do not always prepare country reports on an annual basis, officials told us they use these reports as a check on their internally generated information, rather than relying on them as primary sources of information. For countries that State finds to be nontransparent, State can issue a waiver that allows the country to receive otherwise restricted assistance. As part of the process of requesting a waiver, embassy staff in country develop action plans to assist the country in improving the level of transparency in its budget process. The action plan should address specific fiscal transparency issues identified in the transparency review process and should include recommendations of short-and long- term steps that the countries can take to improve budget transparency. Embassies work with host governments to encourage implementation of action plans, which may include activities such as continued daily engagement with country officials working on budget reforms, providing training and coordinating training on issues related to the budget process, and funding local nongovernmental organizations to perform budget oversight functions. For OMA's DF4D initiative, State officials, working with other agencies and organizations, helped develop programs based on needs expressed by country government officials that built on existing reform efforts and development priorities. As an interagency effort, OMA helps identify and leverage existing programs and resources, such as those of multilateral organizations and other donors, such as USAID's Innovation Fund and the Global Health Initiative. For example, USAID is implementing a $7.6 million program to advance El Salvador's fiscal reform agenda by building capacity and improving systems for public expenditure and management and tax revenue mobilization, promoting private sector engagement, and creating a $2 million Revenue Challenge Fund to support improved tax collection at the municipal level. With five countries piloting DF4D, State plans to proceed by selecting additional partner countries based on performance against quantitative DF4D-related measures, consultations with posts, and expressions of interest from ministers of finance and other economic leaders. Moreover, State plans to use action plans developed for FTRP for countries to be considered for DF4D. State encourages posts to report on PFM issues and opportunities to mobilize domestic resources, raise the issues with relevant stakeholders in the public and private sectors, and work with OMA staff to further the objectives of the DF4D initiative. State Is Taking Steps to Monitor Progress of Nontransparent Aid Recipients: State is beginning to conduct additional monitoring as part of its fiscal transparency evaluations. Starting in 2012, State is requiring benchmarks in its action plans for nontransparent countries so that it can compare progress annually. According to State officials, some country action plans had benchmarks but the quality of the benchmarks varied. The action plan attempts to capture all the steps necessary to improve a country's fiscal transparency and includes more than just State actions. Lastly, appropriations law for 2012 requires State to evaluate whether or not the country has made progress toward improved fiscal transparency included in any waiver request submitted.[Footnote 51] State guidance reports that progress toward implementation of embassy action plans will factor into its decision of whether to renew waivers. [End of section] Appendix IV: Examples of Public Financial Management-related Projects in Six Countries: To provide illustrative examples of U.S. projects to strengthen public financial management (PFM) systems, we chose six case study countries. For the U.S. Agency for International Development (USAID), we selected projects in Kosovo, Liberia, and Peru and for the U.S. Department of the Treasury (Treasury), we selected projects in Cambodia, Honduras, and South Africa. We used the following criteria to select the three countries for each agency: * The agency has relevant ongoing or recently completed projects focused on strengthening PFM systems. * The countries represent different geographic regions. * The countries have different income levels, which may be associated with different levels of government capabilities. For each U.S. agency, we selected a country that the World Bank has classified as lower income (Cambodia and Liberia), lower-middle income (Honduras and Kosovo), and upper-middle income (Peru and South Africa). For one PFM-related project in each country we summarized the project's background, and, for selected objectives, we summarized some of USAID's expected results and some of Treasury's activities. These examples are meant to be illustrative and not generalizable. Selected USAID PFM-Related Projects: Tables 3 through 5 summarize selected USAID PFM-related projects in Kosovo, Liberia, and Peru. Kosovo Growth and Fiscal Stability Initiative: Time period: July 2010-July 2013: Award amount: $1,051,208: Description: Since 1999, USAID's Kosovo economic policy and institutional strengthening programs have focused on establishing key central economic institutions and an enabling environment for private sector growth. USAID is to adjust the focus of technical assistance as the Growth and Fiscal Stability Initiative builds upon the experience and lessons learned from the creation of reliable financial institutions in the central government to address the fiscal stewardship challenges faced by subnational governments. The initiative is to work with municipalities in areas that are directly linked with the Ministry of Finance and Economy, such as budget, treasury, property tax, and public private partnerships. Table 3: USAID's Kosovo Growth and Fiscal Stability Initiative: Selected Objectives and Expected Results: Objective: Build professionalism and expertise in sound PFM to enable municipal institutions to autonomously and efficiently stimulate local economic development while providing a high standard of public services. Expected results: * Identify PFM strengths and weaknesses and develop a PFM reform action plan; * Share best practices and lessons learned with other municipalities through outreach and training; * New municipalities use the main Kosovo financial management systems daily. Objective: Strengthen the Office of the Auditor General's ability to effectively review and assess municipalities' fiscal operations in a more timely manner. Expected results: * Office has adequate budget resources to enable annual audits of all municipalities; * International audit firms perform some municipal audits and train Auditor General staff; * Municipalities address audit findings and improve fiscal operations. Objective: Assist the Ministry of Finance and Economy to (1) develop and implement a plan to enable data sharing among information systems; (2) assess priority capacity-building needs in budget and treasury functions; and (3) ensure that the property tax department and supporting systems can meet municipalities' need to generate revenue. Expected results: * Install a single, shared physical network with all ministry organizations linked to it; * Ministry budget and treasury departments function effectively and transparently without external assistance; * Municipalities administer property tax as a source of revenue. Source: USAID Growth and Fiscal Stability Initiative Assistance Agreement. [End of table] Liberia: Governance and Economic Management Support Program (GEMS): Time period: July 2011-June 2016: Contract amount: $44,902,679: Description: USAID-GEMS is to address significant governance challenges remaining after USAID completed its previous capacity- building program in 2010. The program is to strengthen human and institutional capacity within selected ministries, agencies, and commissions. USAID-GEMS is to develop and maintain systems that increase transparency and accountability, increase efficiency, reduce expenditures, increase revenue, and limit corruption. Table 4: USAID's Liberia GEMS: Selected Objectives, Subobjectives, and Expected Results: Objective: Broadened capacity building for ministries and agencies; Subobjective: Management systems of targeted government institutions meet international good practices and standards. Expected results: * Improve management systems at selected state-owned enterprises; * Participating ministries and agencies conform with international good practice standards; * Targeted ministries and agencies ready to manage U.S. government and other donor funds. Objective: Provide financial management and information technology training; Subobjective: Liberia Institute of Public Administration provides sustainable, quality training, and advisory services to its clients in financial management and information technology. Expected results: * Design, deliver, and monitor training that responds to the clients' needs; * Provide training and testing for selected international certifications in programs such as computer technician. Objective: Effective concessions monitoring and management; Subobjective: Government of Liberia manages mining, agriculture and forestry concessions effectively. Expected results: * Establish transparent legal and regulatory framework; * Use information and communications technology-based systems for management and oversight. Objective: Assistance to the Central Bank of Liberia; Subobjective: National payment system meets West African Monetary Zone requirements. Expected results: * Implement payment and accounting systems effectively; * Complete transition to direct deposit payments. Source: USAID Governance and Economic Management Support Program Assistance Agreement and first-year work plan. [End of table] Peru: USAID ProDecentralization 2: Time period: August 2008-July 2012: Award amount: $10,644,432: Description: The USAID/Peru ProDecentralization project is in the second phase. The first phase targeted national, regional, and municipal institutions responsible for implementing the decentralization process. The second phase seeks to continue to improve the policy framework at the national level and strengthen the institutional capabilities of regional and municipal governments. The national project aims to improve the legal and policy framework for decentralization, including fiscal policies that support a more equitable distribution of public resources. Regional and local-level activities are to target technical assistance and training to the diverse needs of Peru's regional and municipal governments in effectively administering resources and responding to citizens' increasing expectations. Under this project, existing services offered to subnational governments in planning, budgeting, accountability, and institutional strengthening are supplemented by new training and technical assistance programs. Table 5: USAID's Peru ProDecentralization 2: Selected Objectives and Expected Results: Objective: Improved legal and policy framework. Expected results: * Assess legal and policy framework; * Design a legal and policy framework tool for progress that identifies legislation and policies necessary to further the decentralization process; * Provide technical assistance on critical legal and policy issues; * Produce an annual report on the status of decentralization. Objective: Strengthened local government capacity to effectively govern. Expected results: * Reinforce and reward good governance practices; * Improve implementation of national administrative systems; * Improve efficiency of subnational governments; * Innovate strategies to collect taxes at municipal level. Objective: Strengthened good governance in context of regional and local elections. Expected results: * Discuss at least two priority subjects of the legal and policy framework for decentralization at public hearings; * Provide all municipalities in the project with clear and updated guidance on the transfer of managerial competencies. Source: USAID ProDecentralization 2 Performance Management Plan and second-year work plan. [End of table] Selected Treasury PFM-Related Projects: Tables 6 through 8 summarize selected Treasury Office of Technology Assessment (OTA) PFM-related projects in Cambodia, Honduras, and South Africa. Cambodia: Technical Advisory Services on Budgeting: Time period: October 2009-September 2010: Description: The Cambodian government has been implementing a plan to create a credible budget and improve accountability. In the current phase of the plan, OTA and Cambodian government actions are focused on decentralizing the reform effort to the line ministries and subnational levels. These actions include expanding the use of strategic and program budgeting, implementing a PFM information system, improving macroeconomic forecasting, enhancing the linkage between the capital and operating budgets, and improving overall financial accountability. The overall aim is to improve the ability of the budget to be an instrument for policy delivery and to support effective and efficient service delivery. Table 6: Treasury's Technical Advisory Services to Cambodia on Budgeting: Selected Objectives and Activities: Objective: Improve and expand program budgeting: Action plan activities: * Review and improve the implementation of program budgeting, including integrating poverty reduction into the Budget Strategy Plan and Program Budget guidelines; * Develop strategy to expand program budgeting; * Continue capacity development through the further training on program budgeting. Objective: Further improve budget comprehensiveness: Action plan activity: * Further incorporate off-budget expenditures and revenues into the annual budget, including increasing information on donor funds. Objective: Improve budget integration. Action plan activities: * Develop a policy on the integration of the budget. Continue to improve integration of the recurrent and capital budgets through ongoing initiatives such as the Medium Term Expenditure Framework, Budget Strategy Plan, and Program Budget. Objective: Provide advisory support as necessary and requested by budget entities. Action plan activity: * Improve lines of accountability by clarifying roles, functions, and responsibilities. Objective: Improve instruments for encouraging responsible financial management and accountability. Action plan activity: * Establish a list of flexibilities to be given to those budget entities that meet agreed criteria on satisfactory resource management. Objective: Improve budget classification. Action plan activity: * Further improve the implementation of new budget classifications and chart of accounts. Source: OTA Terms of Reference and 2010 work plans for Cambodia. [End of table] Honduras: Technical Advisory Services on Budgeting: Time Period: January 2011-December 2013: Description: OTA's technical assistance focuses on implementing international public sector accounting standards. Table 7: Treasury's Technical Advisory Services on Honduran Budgeting: Selected Objectives and Action Plan Activities: Objective: Assist in the preparation of year-end financial statements: Action plan activities: * Review financial statements sent to Congress in April 2011; * Assist with the creation of a consolidated balance sheet; * Prepare a Consolidated Statement of Cash Flows for the entire government in accordance with international accounting standards. Objective: Review laws and regulations and recommend changes to the legislative framework. Action plan activities: * Review Organic Budget Law, Organic Law of the Supreme Court of Accounts, Public Administrative Law, and other relevant regulations; * Evaluate current application of laws and regulations; * Provide written recommendations to finance minister. Objective: Identify government business enterprises as defined by the international public sector accounting standards and evaluate the impact of implementing the required standards. Action plan activities: * Identify government business enterprises and determine which ones are using appropriate standards; * Provide assistance to integrate these enterprises into the financial management information system. Objective: Develop regulations and manuals including procedures to identify the data needed and entity responsible for gathering and recording data and templates for financial statements. Action plan activities: * Review existing regulations and manuals for the central government and recommend changes; * Review and provide assistance with the development of manuals and regulations for relevant government entities as they are integrated into the financial management information system. Objective: Provide training and materials to Accountant General and staff. Action plan activities; * Evaluate skill level of accounting staff; * Develop training materials for accounting staff and assist with training; * Develop specialized training materials for central administration and assist with training; * Review financial information provided by other relevant government entities to determine training needs. Objective: Review functionality of the country's financial management information system. Identify changes needed for the system to serve as the primary source of financial information throughout the finance ministry; Action plan activities; * Review current reporting requirements of the finance ministry; * Evaluate the financial management information system's ability to meet reporting requirements of finance ministry directorates. Source: Treasury OTA 2011 Work Plan for Honduras. [End of table] South Africa: Technical Advisory Services on Budgeting: Time Period: February 2007--January 2009: Description: Treasury OTA has conducted technical assistance projects in South Africa in the areas of budget formulation, intergovernmental finance, infrastructure budgeting, public finance training, and others since 1997. In 2006, the National Treasury of South Africa requested a new OTA budget project to focus on the organization of an expenditure program performance evaluation unit, performance information and cost analysis, and support for the Collaborative Africa Budget Reform Initiative. The central mission of the resident advisor was to establish an expenditure program performance review system. Table 8: Treasury's Technical Advisory Services to South Africa on Budgeting: Selected Objectives and Activities: Objective: Assist in the development of a performance information management framework and support the implementation of performance- based budgeting practices: Work plan activities: * Facilitate agreement among different staff at the National Treasury on the core principles of performance information management; * Gather and provide input with the aim to publish a Performance Information Management Framework; * Identify opportunities for performance-based budgeting practices (outlined in the framework) to be integrated into existing budgeting systems in conjunction with Public Finance and Budget Office; * Facilitate the implementation of a pilot for performance-based budgeting into the structure of the 2008-2009 budget. Objective: Develop performance review capacity at the National Treasury. Work plan activities: * Identify the goal, role and primary objectives of a performance review function in conjunction with the Public Finance and Budget Offices; * Assess and make recommendations on the organizational and technical capacity of the National Treasury to conduct performance reviews. Work with the Public Finance and Budget Offices to select a performance review(s) related to the following potential issues: - Reform the budget process for interdepartmental information technology systems; - Review the efficiency and effectiveness of labor-based programs such as public works; - Review the efficiency and effectiveness of infrastructure maintenance and rehabilitation program; - Review the efficiency and effectiveness of training and skills development and other government capacity-building projects; - Review state-owned enterprises with large accumulated surpluses; - Assist in conducting a performance review(s) to identify critical interventions, as support for the establishment of a performance review function; - Assist in the establishment of a permanent performance review function. Objective: Support the oversight role of parliament through the provision of more timely and actionable in-year financial and performance information. Work plan activities: * Clarify and obtain agreement on the information needs and expectations of the Joint Budget Committee in Parliament; * Identify and map the in-year financial and performance information on national departments currently available to the National Treasury; * Develop strategies to collect and report necessary information to the Joint Budget Committee and design report format and processes; * Pilot report format and processes with the Joint Budget Committee; * Improve and finalize report format and processes. Objective: Support efforts to enhance professionalism and sharing of effective practices between the countries of Africa via the Collaborative Africa Budget Reform Initiative. Work plan activity: * Identify possible opportunities for assistance. Source: Treasury OTA 2007 Resident Advisor Work Plan. [End of table] [End of section] Appendix V: Comments from the U.S. Agency for International Development: USAID: From The American People: Thomas Melito: Director, International Affairs & Trade: U.S. Government Accountability Office: Washington, DC 20548: Dear Mr. Melito: I am pleased to provide the U.S. Agency for International Development's formal response to the GAO draft report entitled "Public Financial Management: USAID and Treasury Design programs to Address Developing Countries' Weaknesses, but Improvements Needed" (Engagement Code 320870). The enclosed USAID comments are provided for incorporation with this letter as an appendix to the final report. Thank you for the opportunity to respond to the GAO draft report and for the courtesies extended by your staff in the conduct of this audit review. Sincerely, Signed by: [Illegible] for: Angelique M. Crumbly: Acting Assistant Administrator/Management Bureau: U.S. Agency for International Development: Enclosure: a/s. [End of letter] USAID Comments On GAO Draft Report Code 320870: The U.S. Agency for International Development (USAID) is currently implementing a range of Agency reforms under the rubric of USAID Forward. One of the key components of USAID Forward targets six improvements for Implementation and Procurement Reform (IPR), with Objective #1 focused on "Strengthening partner country capacity to improve aid effectiveness and sustainability." In developing the USAID Reform agenda the Agency adopted a series of targets to guide the reforms and in the case of IPR a target of 30% was established. The 30% target is more specifically defined as obligating 30% of USAID mission budgets by the end of FY 2015 into local partner implementation mechanisms, which include government-2- government (G2G), local non-governmental and civil society organizations, or through Development Credit Authority (DCA) agreements, all of which would occur in the host country of our program. Inherent in the G2G reforms is the understanding that assessments of local partners will require capacity building and a key target of effective G2G implementation will need to focus on Public Financial Management (PFM) and Public Procurement. As we move forward on our 1PR reforms, USAID remains committed to assuring that all funds under the auspices of USAID are effectively implemented, monitored and evaluated for development impact. As a result, USAID appreciates GAO's report. In response, we provide the following responses to GAO Recommendations: Recommendation #1: To monitor progress towards its target to obligate 30 percent of its annual assistance through local systems by 2015, we recommend that the Administrator of USAID direct the appropriate offices to develop a process to reliably identify and track the agency's use of local systems in all countries receiving assistance. Response: USAID concurs with the recommendation to develop and improve the tracking of assistance through local systems and implementers. To effectively monitor progress of USAID forward initiatives, the USAID Administrator has directed the Agency Counselor to lead efforts to incorporate the reforms across the Agency and to monitor progress. The Counselor has convened regular USAID Forward team monitoring meetings that are attended by senior managers from all bureaus of the Agency, all of which are responsible for effective integration of USAID Forward into our business practices. Relative to the 30% target for IPR, the USAID Forward team believes that early progress towards the 30% target will most likely be slow for three reasons: * The necessary analytical requirements for obligation through local systems and implementers will require more extensive pre-obligation analyses, such as the PFMRAF for government-to-government assistance and a deeper application of pre-award surveys for local NGOs. * Much of USAID's staff is unfamiliar with the design and implementation of government to government assistance. * Project implementation training and guidance for working with country partners is new, or in draft. While USAID is making fast progress in addressing these constraints, training, mentoring and reach-back capacity development are all on- going. Ensuring Agency-wide proficiency with new development delivery tools will be a work in progress as we incorporate new policies and practices into the Agency. The USAID Forward Team has been discussing various approaches to monitor progress. One approach that is already in use relies on USAID's core accounting system, Phoenix, to produce macro level reports on implementation through local implementers, which is the target group for the 30% indicator. This report is also being used to report to Congress on the requirements defined in Section 7031 of our annual FY 2012 appropriation. In addition, the Agency is now in the process of refining clear definitions that will identify and help to measure the assistance that qualifies to meet the 30% target. Again, this involves training Agency staff which has been on-going. Once refined, the Agency will be able to report against a definition with a clear set of criteria. Recommendation #2: To help ensure that USAID conducts effective evaluations of PFM- related programs under its new policy, we recommend that the Administrator of USAID direct the appropriate offices to ensure that they are establishing adequate monitoring practices for PFM-related programs. Such practices may include selecting proper indicators, collecting reliable baseline data, and ensuring the reliability of reported results. Response: USAID concurs with this recommendation to conduct effective evaluations on G2G and PFM-related programs. USAID Forward has already introduced two complementary reforms that will address GAO's recommendation. The first is an effort to reinvigorate strategy development and project design into our normal development assistance programs. For the design of new projects, Agency requirements now include detailed pre-obligation analysis and an indicative plan(s) for monitoring progress and evaluation. This will incorporate defining indicators, collecting baseline data, ensuring reliable results and planning for independent impact evaluations. New guidance also requires that final monitoring and evaluation schedules be planned, including refined indicators and agreement on independent approaches to evaluation. All of the described reforms related to strategy and design, monitoring and evaluation, and implementation and procurement reform are being incorporated into USAID's policy and directives systems for continuing use. [End of section] Appendix VI: Comments from the Department of the Treasury: Department of the Treasury: Washington, DC 20220: August 27, 2012: Mr. Thomas Melito, Director: International Affairs and Trade: Government Accountability Office: 441 G St NW: Washington, DC 20548: Dear Mr. Melito, Thank you for the opportunity to comment on the GAO report on H.S. efforts to strengthen developing countries' public financial management systems (GA0-12-920). This is an important topic. Strengthening the capacity of developing country governments to manage well their public finances is a critical aspect of effective international development work and ultimately reduces these countries' dependence on foreign aid. A key mission of the Treasury Department's Office of Technical Assistance (OTA) is to help developing countries foster secure, stable, and transparent financial sectors by more effectively raising domestic finances to provide important public services. Effective management of public financial resources is a core function of an effective state. Since its creation approximately twenty years ago, OTA has established a record of providing effective assistance in public financial management, an area in which the Treasury Department already has considerable experience and expertise. OTA constantly seeks to improve the management and measurement of its own program. In that spirit, and as discussed in the attached comments, OTA concurs with the two recommendations in the GAO report. The fact that GAO had only two specific recommendations for OTA — one already fully implemented and the other being implemented — speaks to OTA's sound management of this important program. Sincerely, Signed by: W. Larry McDonald: Deputy Assistant Secretary, Technical Assistance Policy: [End of letter] GAO Report GAO-12-920 September 2012: "Public Financial Management: Improvements Needed in USAID and Treasury's Monitoring and Evaluation Efforts" Department of the Treasury Comments on GAO Recommendations: Recommendation 1: OTA should implement additional controls to improve the process for computing OTA-wide annual performance measures. OTA concurs with, and has already implemented, this recommendation. OTA has corrected several errors in transcribing and aggregating the 2011 data, and has taken steps to strengthen data quality controls in order to prevent a similar occurrence in the future. Specifically, OTA has implemented additional quality control reviews, improved the formatting and usability of the data collection instrument, and increased staff resources specifically dedicated to computing performance measures. Having taken steps to implement this recommendation, it is also important to understand the very limited extent to which the errors in transcribing and aggregating 2011 project-specific data affected overall program scores. For example, only one project performance score in more than 150 lines of data was transcribed (listed) under the wrong country. The net effect of several errors in aggregating project-specific data was immaterial. When combined, these errors increased the calculation of overall program impact by only one tenth of one point (on a 5-point scale) and had no impact on the calculation of overall program traction. As noted in GAO's report, there is an inherent difficulty in assigning an aggregated numerical value to n large program that consists of many individual projects implemented over multiple countries with different policy and political environments. OTA recognizes that difficult, and is doing its best to comply with government-wide requirements in this area while creating measures that have meaning and utility to the evaluative process. Recommendation 2: OTA should fully implement its existing requirement for end-of-project evaluations and, consistent with its existing guidance, have an independent party conduct the evaluations. OTA concurs with this recommendation. OTA has already begun to implement this requirement and intends to fully implement independent end-of-project evaluations by the end of 2012. [End of section] Appendix VII: GAO Contact and Staff Acknowledgments: GAO Contact: Thomas Melito, (202) 512-9601 or melitot@gao.gov: Staff Acknowledgments: Cheryl Goodman (Assistant Director), Michael Maslowski, Shirley Min, RG Steinman, Michael Hoffman, Debbie Chung, Grace Lui, David Dayton, and Etana Finkler made key contributions to this report. [End of section] Footnotes: [1] Since USAID is working to commit significantly more to partner country systems in Pakistan and Afghanistan, USAID excludes these two countries from the target as they would significantly distort the agency's overall figures. [2] Assistance agreements include all contracts, cooperative agreements, and grant agreements between USAID and its implementing partners. [3] OECD is an international economic organization of 34 countries, including the United States, founded in 1961 to stimulate economic progress and world trade. It is a forum of countries committed to democracy and the market economy, providing a platform to compare policy experiences, seek answers to common problems, identify good practices, and coordinate the domestic and international policies of its members. [4] The steps and entities involved in the budget cycle may differ between countries. Also, some entities, such as the ministry of finance, may be referred to by different names, such as the budget directorate, depending on the country. [5] Supreme audit institution refers to any organization that sets standards for audit work. GAO is the supreme audit institution for the United States. [6] The PEFA program grew out of a multidonor partnership that formed in 2001 to strengthen the ability of aid recipients and donors to assess and improve country public expenditure, procurement, and financial accountability systems. [7] PEFA Secretariat, PFM Performance Measurement Framework: Monitoring Report 2010 (Washington, D.C.: May 19, 2011). [8] The World Bank conducts Country Policy and Institutional Assessments annually to rate the quality of a country's policy and institutional framework against 16 indicators, including the quality of budgetary and financial management. The World Bank only publishes the indicator scores of countries that are eligible for funding from the World Bank's International Development Association, which provides interest-free loans and grants to governments of the poorest countries. See World Bank's Country Policy and Institutional Assessments 2010 Assessment Questionnaire. [9] The International Budget Partnership is a nongovernmental organization that collaborates with civil society around the world to reduce poverty and improve the quality of governance by reforming government budget systems and influencing budget policies. [10] International Budget Partnership, The Open Budget Survey 2010, (Washington, D.C.: October 2010). [11] Transparency International is a global civil society organization focused on the fight against corruption. Through more than 90 chapters worldwide and an international secretariat in Berlin, the organization raises awareness of the damaging effects of corruption and works with partners in government, business, and civil society to develop and implement effective measures to lessen it. [12] Transparency International defines corruption as the abuse of entrusted power for private gain, and its definition encompasses corrupt practices in both the public and private sectors. See Transparency International, Corruption Perceptions Index 2010 (Berlin, Germany: October 2010). [13] Other U.S. agencies provide some PFM-related assistance. For example, GAO provides capacity-building support to fellow audit institutions through its membership in the International Organization of Supreme Audit Institutions. In addition, the Millennium Challenge Corporation (MCC) has minimum financial accountability requirements for countries to be eligible to enter into assistance compacts. Deficiencies in these requirements may be addressed through MCC's threshold program, which is typically administered by USAID. [14] USAID reports that its DG programs support more legitimate, inclusive, and effective governments that are responsive to the needs of their people. DG's stated goals include strengthening and promoting more transparent and accountable governance. Further, USAID reports that its EG programs work to help developing countries achieve rapid, sustained, and broad-based economic growth. Reported activities include developing well-functioning markets and strengthening the framework of policies, institutions, and public goods, including helping poor countries adopt better financial management systems that allow for stronger oversight of government spending and reduce chances for corruption. [15] Funding totals for 2011 do not include amounts that may have been administered by USAID from the Assistance to Europe, Eurasia, and Central Asia account. [16] Exceptions include country offices implementing a single-sector program and country offices that are phasing out or closing down by fiscal year 2014. [17] USAID country offices are required to follow the new project design guidance starting in July 2012. However, to facilitate transition to the new USAID project design process, full application of the new project design guidance will only apply to a limited number of new, priority projects for fiscal years 2012 and 2013 for USAID country offices with an approved country strategy, or an approved Feed the Future strategy for Feed the Future focus countries. Priority projects include those that intend to use government systems. [18] According to USAID guidance, a performance indicator is a particular characteristic or dimension used to measure actual results compared to expected results to determine progress toward the objective at three levels: output, outcome, and impact. Output is an immediate and intended product or consequence of an activity within USAID's control, such as number of personnel trained or new technologies developed. Outcome is a higher-level or end result, or purpose, such as the development objective to develop local-level capacity to create reliable financial institutions. Impact generally refers to higher-level medium-and long-term impacts, which can be intended or unintended, positive or negative, such as growth in national income or the size of the budget deficit. [19] Our three USAID case-study countries' agreements required annual work plans. [20] GAO, International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts, [hyperlink, http://www.gao.gov/products/GAO-09-980] (Washington, D.C.: September 28, 2009). [21] GAO, Foreign Assistance: The United States Provides Wide-ranging Trade Capacity Building Assistance, but Better Reporting and Evaluation Are Needed, [hyperlink, http://www.gao.gov/products/GAO-11-727] (Washington, D.C.: July 29, 2011). [22] GAO, Foreign Assistance: The United States Provides Wide-ranging Trade Capacity Building Assistance, but Better Reporting and Evaluation Are Needed, [hyperlink, http://www.gao.gov/products/GAO-11-727] (Washington, D.C.: July 29, 2011). [23] GAO, Burma: UN and U.S. Agencies Assisted Cyclone Victims in Difficult Environment, but Improved U.S. Monitoring Needed, [hyperlink, http://www.gao.gov/products/GAO-11-700] (Washington, D.C.: July 26, 2011). [24] GAO, Afghanistan Governance: Performance-Data Gaps Hinder Overall Assessment of U.S. Efforts to Build Financial Management Capacity, [hyperlink, http://www.gao.gov/products/GAO-11-907] (Washington, D.C.: Sept. 20, 2011). [25] USAID's DG and EG offices are the primary offices providing PFM- related assistance. [26] The Liberia program's performance management plan indicates that it will be updated in the fourth quarter of fiscal year 2012. [27] The lack of an approved work plan was due to the ongoing negotiations between the Government of Liberia and the USAID/Liberia country-office staff, according to a program progress report. [28] According to a senior USAID official, USAID directed 3 percent of its funds through country systems in 2007. [29] In experimental methods research, subjects are randomly assigned to either a treatment or control group, data are collected both before and after the intervention, and results from the treatment group are benchmarked against results from the control group. [30] The Inter-American Development Bank, an international financial institution composed of 48 member governments, has reported that techniques for using impact evaluations to assess institutional capacity building have not been fully developed. Furthermore, if a program targets a ministry, such as the finance ministry, it may be difficult to establish a control or comparison group equivalent to the ministry (i.e., a counterfactual). Capacity building at the subnational (regional or municipal) level may be more amenable to experimental or quasi-experimental methods if program assistance is provided to some regions or municipalities but not others. [31] Democracy International Inc., Kosovo Effective Municipalities Initiative Mid-Term Evaluation, a special report prepared at the request of the U.S. Agency for International Development, May 2009; and ARD Inc., Prodecentralization Project External Mid Term Review, a special report prepared at the request of the U.S. Agency for International Development, September 2010. [32] U.S. Agency for International Development,USAID Evaluation Policy: Year One. First Annual Report and Plan for 2012 and 2013, PD- ACS-999 (Washington, D.C.: February 2012). [33] GAO, Foreign Assistance: The United States Provides Wide-ranging Trade Capacity Building Assistance, but Better Reporting and Evaluation Are Needed, [hyperlink, http://www.gao.gov/products/GAO-11-727] (Washington, D.C.: July 29, 2011); Afghanistan Development: Enhancements to Performance Management and Evaluation Efforts Could Improve USAID's Agricultural Programs, [hyperlink, http://www.gao.gov/products/GAO-10-368] (Washington, D.C.: July 14, 2010); and International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts, [hyperlink, http://www.gao.gov/products/GAO-09-980] (Washington, D.C.: Sept. 28, 2009). [34] These evaluations were not subject to USAID's new evaluation policy, which was issued in January 2011. [35] Sibley International LLC, Final Evaluation of USAID GEMAP Activities (Governance and Economic Management Assistance Program), a special report prepared at the request of the U.S. Agency for International Development, June 2010. [36] AECOM International Development, Evaluation of the Economic Management for Stability and Growth Program, a special report prepared at the request of the U.S. Agency for International Development, April 2009. [37] The four main categories for project assessment are (1) international standards, processes, and participation; (2) progress toward country goals; (3) building human and systems capacity; and (4) whole-of-government and private sector effectiveness. [38] GAO, Performance Budgeting: Observations on the Use of OMB's Performance Assessment Rating Tool for the Fiscal Year 2004 Budget, [hyperlink, http://www.gao.gov/products/GAO-04-174] (Washington, D.C.: Jan. 30, 2004). [39] See, for example, Paolo de Renzio, Matt Andrews, and Zac Mills, "Does Donor Support to Public Financial Management Reform in Developing Countries Work? An Analytical Study of Quantitative Cross- Country Evidence." Overseas Development Institute Working Paper # 329, April 2011; or Matt Andrews, "How Far Have Public Financial Management Reforms Come in Africa?" Harvard Kennedy School Faculty Working Paper Series, RWP10-018, May 2010. [40] The partnership included the World Bank, the European Commission, the United Kingdom's Department for International Development, the Swiss State Secretariat for Economic Affairs, the French Ministry of Foreign Affairs, the Royal Norwegian Ministry of Foreign Affairs, and the International Monetary Fund. [41] Highest or improved scores are combined because it is not possible to improve on an A score; lowest or worsened scores are combined because it is not possible to be worse than a D score. Five of the eight countries were small island states with challenges not typical of the larger sample, including high vulnerability to external shocks such as hurricanes and volatile tourism earnings, severe capacity constraints with high emigration of skilled human resources, and political instability. The other three were small African states with many of the same challenges. [42] The World Bank does not publish the indicator scores of countries that are not eligible for funding from the World Bank's International Development Association, which provides interest-free loans and grants to governments of the poorest countries. An average of 76 countries received a rating for this indicator from 2005 through 2010. [43] The World Bank rates each country on a scale of 1 (low) to 6 (high) in half-point increments (0.5). A score of 1 corresponds to a very weak performance; a score of 6, to a very strong performance. Mid- range includes scores of 3 and 3.5. [44] The Open Budget Index is an external view of transparency, not of a country's systems and processes. [45] The 2010 Open Budget Survey did not include Kosovo. [46] Transparency International defines corruption as the abuse of entrusted power for private gain; its definition encompasses corrupt practices in both the public and private sectors. [47] Countries showing improvement include Bhutan, Chile, Ecuador, former Yugoslav Republic of Macedonia, Gambia, Haiti, Jamaica, Kuwait, and Qatar. Countries showing deterioration include the Czech Republic, Greece, Hungary, Italy, Madagascar, Niger, and the United States. [48] Since 2008, the annual Department of State, Foreign Operations, and Related Programs Appropriations Acts have prohibited certain U.S. foreign assistance to the central government of any country that fails to make its national budget publicly available on an annual basis. The law permits the Secretary of State to waive this restriction on individual countries by reporting to Congress that a waiver is important to the national interest of the United States. See Pub. L. No. 110-161. [49] Countries that are not transparent with their budgets are ineligible to receive certain U.S. assistance, without a State issued waiver, in accordance with Pub. L. No. 112-74, § 7031(b). [50] The IMF defines fiscal transparency as "openness toward the public at large about government structure and functions, fiscal policy intentions, public sector accounts, and projections. It involves ready access to reliable, comprehensive, timely, understandable, and internationally comparable information on government activities so that the electorate and financial markets can accurately assess the government's financial position and the true costs and benefits of government activities, including their present and future economic and social implications." [51] Consolidated Appropriations Act, 2012, Pub. L. No. 112-74, § 7031(b). [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. [End of document] Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.