President's Emergency Plan for AIDS Relief:

Agencies Can Enhance Evaluation Quality, Planning, and Dissemination

GAO-12-673: Published: May 31, 2012. Publicly Released: May 31, 2012.

Additional Materials:

Contact:

David B. Gootnick
(202) 512-3149
GootnickD@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of State’s (State) Office of the U.S. Global AIDS Coordinator (OGAC), the Department of Health and Human Services’ (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID) have evaluated a wide variety of President’s Emergency Plan for AIDS Relief (PEPFAR) program activities, demonstrating a clear commitment to evaluation. However, GAO found that the findings, conclusions, and recommendations were not fully supported in many PEPFAR evaluations. Agency officials provided nearly 500 evaluations addressing activities ongoing in fiscal years 2008 through 2010 in all program areas relating to HIV/AIDS treatment, prevention, and care. GAO’s assessment of a selected sample of seven OGAC-managed evaluations found that they generally adhered to common evaluation standards, as did most of a selected sample of 15 evaluations managed by CDC and USAID headquarters. Based on this assessment, GAO determined that these evaluations generally contained fully supported findings, conclusions, and recommendations. However, based on a similar assessment of a randomly selected sample taken from 436 evaluations provided by PEPFAR country and regional teams, GAO estimated that 41 percent contained fully supported findings, conclusions, and recommendations, while 44 percent contained partial support and 15 percent were not supported.

State, OGAC, CDC, and USAID have established detailed evaluation policies, as recommended by the American Evaluation Association (AEA). However, PEPFAR does not fully adhere to AEA principles relating to evaluation planning, independence and qualifications of evaluators, and public dissemination of evaluation results. Specifically, OGAC does not require country and regional teams to include evaluation plans in their annual operational plans, limiting its ability to ensure that evaluation resources are appropriately targeted. Further, although OGAC, CDC, and USAID evaluation policies and procedures provide some guidance on how to ensure evaluator independence and qualifications, they do not require documentation of these issues. GAO found that most PEPFAR program evaluations did not fully address whether evaluators had conflicts of interest and some did not include detailed information on the identity and makeup of evaluation teams. Finally, although OGAC, CDC, and USAID use a variety of means to share evaluation findings, not all evaluation reports are available online, limiting their accessibility to the public and their usefulness for PEPFAR decision makers, program managers, and other stakeholders.

Why GAO Did This Study

PEPFAR, reauthorized by Congress in fiscal year 2008, supports HIV/AIDS prevention, treatment, and care overseas. The reauthorizing legislation, as well as other U.S. law and government policy, stresses the importance of evaluation for improving program performance, strengthening accountability, and informing decision making. OGAC leads the PEPFAR effort by providing funding and guidance to implementing agencies, primarily CDC and USAID. Responding to legislative mandates, GAO (1) identified PEPFAR evaluation activities and examined the extent to which evaluation findings, conclusions, and recommendations were supported and (2) examined the extent to which PEPFAR policies and procedures adhere to established general evaluation principles. GAO reviewed these principles as well as agencies’ policies and guidance; surveyed CDC and USAID officials in 31 PEFAR countries and 3 regions; and analyzed evaluations provided by OGAC, CDC, and USAID.

What GAO Recommends

GAO recommends that State work with CDC and USAID to (1) improve adherence to common evaluation standards, (2) develop PEPFAR evaluation plans, (3) provide guidance for assessing and documenting evaluators’ independence and qualifications, and (4) increase online accessibility of evaluation results. Commenting jointly with HHS’s CDC and USAID, State agreed with these recommendations and noted steps it will take to implement them.

For more information, contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: State agreed with our recommendation and in response, OGAC developed PEPFAR's Evaluation Standards of Practice in 2014 (updated in 2015), to be used by all of PEFAR's implementing agencies. OGAC specifically states in its guidance that while evaluation has been an important and integral component of program improvement, it has not been sufficiently prevalent or rigorous, "as highlighted by GAO". In the updated 2015 guidance, OGAC states that the goal of the guidance is to articulate an interagency consensus of standards of evaluation practice deemed most relevant to conducting evaluations within PEPFAR and to inform improved decision-making. The guidance is intended to improve evaluation planning, implementation, oversight, and quality across PEPFAR programs while also responding to recommendations by GAO. OGAC explains that while application of the standards may vary slightly according to specific implementing agency policies and procedures, it is expected that the guidance and the respective planning, adherence, and reporting tools will be applied across all PEPFAR funded evaluations to ensure a consistent, coordinated, and collaborative effort that will produce high-quality evaluations.

    Recommendation: The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to develop a strategy to improve PEPFAR implementing agencies' and country and regional teams' adherence to common evaluation standards.

    Agency Affected: Department of State

  2. Status: Closed - Implemented

    Comments: State agreed with our recommendation and in both the 2014 and updated 2015 Evaluation Standards of Practice, and the PEPFAR Country Operational Plan (COP) Guidance for fiscal year 2014, OGAC required that operating units submit information relevant to the joint evaluation agenda and to the actual evaluation studies in-progress, planned, and completed in their Semi-Annual Performance Results (SAPR). In the COP Guidance, PEPFAR attributes this new requirement to GAO, stating that providing the evaluation plans will allow for improved monitoring of evaluation activities across all of PEPFAR, consistent with recommendations from GAO. The updated 2015 Evaluation Standards of Practice states that as part of future COPs, operating units will be asked to submit operating unit evaluation plans and describe associated development processes. The guidance states that OGAC's Evaluation Working Group will provide specific guidance regarding the content and structure of these evaluation plans to operating units; however, OGAC references the AEA and the Joint United Nations Programme on HIV/AIDS (UNAIDS) as two resources for preliminary guidance. The FY 2016 submission may be in the form of priority lists, evaluation calendars, evaluation budgets, and evaluations questions, among other related formats.

    Recommendation: The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to require implementing agency headquarters and country and regional teams to include evaluation plans in their annual operational plans.

    Agency Affected: Department of State

  3. Status: Closed - Implemented

    Comments: State agreed with our recommendation and included guidance described as ensuring appropriate evaluator qualifications and evaluation independence in the 2014 Evaluation Standards of Practice guidance and the updated version of this guidance in 2015. In the 2014 and 2015 guidance, OGAC specifically states that the guidance responds to recommendations by GAO. Regarding appropriate evaluator qualifications, OGAC states in the guidance that only evaluation teams with the appropriate education, capabilities, skills, and experience should conduct evaluations. OGAC provides examples of the advanced training and subject area expertise that professional evaluators typically hold. In addition, OGAC states that the resumes of evaluation team members should be examined for appropriate education and previous experiences to perform the specific evaluation being considered. Regarding evaluator independence, OGAC requires that in advance of conducting an evaluation, everyone on the evaluation team must disclose any personal, financial, or other relationships they have that might pose a conflict of interest in their role as evaluators. OGAC explains that this can be accomplished through the use of conflict of interest statements prior to conducting the evaluation. Additionally, OGAC provides examples of strategies teams can use to improve evaluator independence to meet this requirement, including but not limited to, having evaluation units that are separate from program units, using external evaluations and evaluators, or establishing formal conflict of interest procedures and declarations for internal evaluators.

    Recommendation: The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to provide detailed guidance for implementing agencies and country and regional teams on assessing, ensuring, and documenting the independence and competence of PEPFAR program evaluators.

    Agency Affected: Department of State

  4. Status: Open

    Comments: PEPFAR program evaluations are accessible via the www.pepfar.gov website; however, they are difficult to find. A PEPFAR official noted that there is room for improvement in the area of disseminating evaluation results and that follow up actions have been taken with PEPFAR implementing agencies. He stated that in response to legislative and GAO requirements, the next annual report, to be issued in FY2016, will contain a complete inventory of new, ongoing, and complete evaluations conducted in each country, in addition to the standards of practice adherence checklist for all completed evaluations. He further stated that posting this information on pepfar.gov will also be more structured, allowing for a specific page (or pages) for evaluation reporting that is more readily apparent to visitors to the site.

    Recommendation: The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to increase the online accessibility of PEPFAR program evaluation results.

    Agency Affected: Department of State

 

Explore the full database of GAO's Open Recommendations »

Apr 18, 2016

Apr 5, 2016

Feb 16, 2016

Feb 2, 2016

Jan 20, 2016

Jan 13, 2016

Jan 11, 2016

Dec 3, 2015

Nov 17, 2015

Looking for more? Browse all our products here