President's Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination
Highlights
What GAO Found
The Department of State’s (State) Office of the U.S. Global AIDS Coordinator (OGAC), the Department of Health and Human Services’ (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID) have evaluated a wide variety of President’s Emergency Plan for AIDS Relief (PEPFAR) program activities, demonstrating a clear commitment to evaluation. However, GAO found that the findings, conclusions, and recommendations were not fully supported in many PEPFAR evaluations. Agency officials provided nearly 500 evaluations addressing activities ongoing in fiscal years 2008 through 2010 in all program areas relating to HIV/AIDS treatment, prevention, and care. GAO’s assessment of a selected sample of seven OGAC-managed evaluations found that they generally adhered to common evaluation standards, as did most of a selected sample of 15 evaluations managed by CDC and USAID headquarters. Based on this assessment, GAO determined that these evaluations generally contained fully supported findings, conclusions, and recommendations. However, based on a similar assessment of a randomly selected sample taken from 436 evaluations provided by PEPFAR country and regional teams, GAO estimated that 41 percent contained fully supported findings, conclusions, and recommendations, while 44 percent contained partial support and 15 percent were not supported.
State, OGAC, CDC, and USAID have established detailed evaluation policies, as recommended by the American Evaluation Association (AEA). However, PEPFAR does not fully adhere to AEA principles relating to evaluation planning, independence and qualifications of evaluators, and public dissemination of evaluation results. Specifically, OGAC does not require country and regional teams to include evaluation plans in their annual operational plans, limiting its ability to ensure that evaluation resources are appropriately targeted. Further, although OGAC, CDC, and USAID evaluation policies and procedures provide some guidance on how to ensure evaluator independence and qualifications, they do not require documentation of these issues. GAO found that most PEPFAR program evaluations did not fully address whether evaluators had conflicts of interest and some did not include detailed information on the identity and makeup of evaluation teams. Finally, although OGAC, CDC, and USAID use a variety of means to share evaluation findings, not all evaluation reports are available online, limiting their accessibility to the public and their usefulness for PEPFAR decision makers, program managers, and other stakeholders.
Why GAO Did This Study
PEPFAR, reauthorized by Congress in fiscal year 2008, supports HIV/AIDS prevention, treatment, and care overseas. The reauthorizing legislation, as well as other U.S. law and government policy, stresses the importance of evaluation for improving program performance, strengthening accountability, and informing decision making. OGAC leads the PEPFAR effort by providing funding and guidance to implementing agencies, primarily CDC and USAID. Responding to legislative mandates, GAO (1) identified PEPFAR evaluation activities and examined the extent to which evaluation findings, conclusions, and recommendations were supported and (2) examined the extent to which PEPFAR policies and procedures adhere to established general evaluation principles. GAO reviewed these principles as well as agencies’ policies and guidance; surveyed CDC and USAID officials in 31 PEFAR countries and 3 regions; and analyzed evaluations provided by OGAC, CDC, and USAID.
Recommendations
GAO recommends that State work with CDC and USAID to (1) improve adherence to common evaluation standards, (2) develop PEPFAR evaluation plans, (3) provide guidance for assessing and documenting evaluators independence and qualifications, and (4) increase online accessibility of evaluation results. Commenting jointly with HHSs CDC and USAID, State agreed with these recommendations and noted steps it will take to implement them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of State | The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to develop a strategy to improve PEPFAR implementing agencies' and country and regional teams' adherence to common evaluation standards. |
State agreed with our recommendation and in response, OGAC developed PEPFAR's Evaluation Standards of Practice in 2014 (updated in 2015), to be used by all of PEFAR's implementing agencies. OGAC specifically states in its guidance that while evaluation has been an important and integral component of program improvement, it has not been sufficiently prevalent or rigorous, "as highlighted by GAO". In the updated 2015 guidance, OGAC states that the goal of the guidance is to articulate an interagency consensus of standards of evaluation practice deemed most relevant to conducting evaluations within PEPFAR and to inform improved decision-making. The guidance is intended to improve evaluation planning, implementation, oversight, and quality across PEPFAR programs while also responding to recommendations by GAO. OGAC explains that while application of the standards may vary slightly according to specific implementing agency policies and procedures, it is expected that the guidance and the respective planning, adherence, and reporting tools will be applied across all PEPFAR funded evaluations to ensure a consistent, coordinated, and collaborative effort that will produce high-quality evaluations.
|
| Department of State | The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to require implementing agency headquarters and country and regional teams to include evaluation plans in their annual operational plans. |
State agreed with our recommendation and in both the 2014 and updated 2015 Evaluation Standards of Practice, and the PEPFAR Country Operational Plan (COP) Guidance for fiscal year 2014, OGAC required that operating units submit information relevant to the joint evaluation agenda and to the actual evaluation studies in-progress, planned, and completed in their Semi-Annual Performance Results (SAPR). In the COP Guidance, PEPFAR attributes this new requirement to GAO, stating that providing the evaluation plans will allow for improved monitoring of evaluation activities across all of PEPFAR, consistent with recommendations from GAO. The updated 2015 Evaluation Standards of Practice states that as part of future COPs, operating units will be asked to submit operating unit evaluation plans and describe associated development processes. The guidance states that OGAC's Evaluation Working Group will provide specific guidance regarding the content and structure of these evaluation plans to operating units; however, OGAC references the AEA and the Joint United Nations Programme on HIV/AIDS (UNAIDS) as two resources for preliminary guidance. The FY 2016 submission may be in the form of priority lists, evaluation calendars, evaluation budgets, and evaluations questions, among other related formats.
|
| Department of State | The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to provide detailed guidance for implementing agencies and country and regional teams on assessing, ensuring, and documenting the independence and competence of PEPFAR program evaluators. |
State agreed with our recommendation and included guidance described as ensuring appropriate evaluator qualifications and evaluation independence in the 2014 Evaluation Standards of Practice guidance and the updated version of this guidance in 2015. In the 2014 and 2015 guidance, OGAC specifically states that the guidance responds to recommendations by GAO. Regarding appropriate evaluator qualifications, OGAC states in the guidance that only evaluation teams with the appropriate education, capabilities, skills, and experience should conduct evaluations. OGAC provides examples of the advanced training and subject area expertise that professional evaluators typically hold. In addition, OGAC states that the resumes of evaluation team members should be examined for appropriate education and previous experiences to perform the specific evaluation being considered. Regarding evaluator independence, OGAC requires that in advance of conducting an evaluation, everyone on the evaluation team must disclose any personal, financial, or other relationships they have that might pose a conflict of interest in their role as evaluators. OGAC explains that this can be accomplished through the use of conflict of interest statements prior to conducting the evaluation. Additionally, OGAC provides examples of strategies teams can use to improve evaluator independence to meet this requirement, including but not limited to, having evaluation units that are separate from program units, using external evaluations and evaluators, or establishing formal conflict of interest procedures and declarations for internal evaluators.
|
| Department of State | The Secretary of State should direct the U.S. Global AIDS Coordinator, in collaboration with CDC and USAID to enhance PEPFAR evaluations, to increase the online accessibility of PEPFAR program evaluation results. |
State agreed with this recommendation and affirmed, in comments on the report, that OGAC would collaborate with PEPFAR implementing agencies to develop strategies for improving dissemination of evaluation results and use PEPFAR's public website to link to agencies' online resources. In September 2015, we found that PEPFAR evaluations were available online at www.pepfar.gov but were very hard to find. We discussed this issue with a key OGAC official and suggested that there be a more direct link to PEPFAR evaluations to make them easier to find. He agreed, and in July 2016, OGAC improved the PEPFAR website so that PEPFAR evaluations are readily accessible via a link to "PEPFAR Dashboards" on the landing page. He said GAO's interest and suggestions have been instrumental in this change.
|