Efforts to Assess the Accuracy and Effectiveness of Federally Funded Programs
GAO-07-87: Published: Oct 3, 2006. Publicly Released: Nov 16, 2006.
Reducing the incidence of sexually transmitted diseases and unintended pregnancies is one objective of the Department of Health and Human Services (HHS). HHS provides funding to states and organizations that provide abstinence-until-marriage education as one approach to address this objective. GAO was asked to describe the oversight of federally funded abstinence-until-marriage education programs. GAO is reporting on (1) efforts by HHS and states to assess the scientific accuracy of materials used in these programs and (2) efforts by HHS, states, and researchers to assess the effectiveness of these programs. GAO reviewed documents and interviewed HHS officials in the Administration for Children and Families (ACF) and the Office of Population Affairs (OPA) that award grants for these programs.
Efforts by HHS and states to assess the scientific accuracy of materials used in abstinence-until-marriage education programs have been limited. This is because HHS's ACF--which awards grants to two programs that account for the largest portion of federal spending on abstinence-until-marriage education--does not review its grantees' education materials for scientific accuracy and does not require grantees of either program to review their own materials for scientific accuracy. In contrast, OPA does review the scientific accuracy of grantees' proposed educational materials. In addition, not all states that receive funding from ACF have chosen to review their program materials for scientific accuracy. In particular, 5 of the 10 states that GAO contacted conduct such reviews. Officials from these states reported using a variety of approaches in their reviews. While the extent to which federally funded abstinence-until-marriage education materials are inaccurate is not known, in the course of their reviews OPA and some states reported that they have found inaccuracies in abstinence-until-marriage education materials. For example, one state official described an instance in which abstinence-until-marriage materials incorrectly suggested that HIV can pass through condoms because the latex used in condoms is porous. HHS, states, and researchers have made a variety of efforts to assess the effectiveness of abstinence-until-marriage education programs; however, a number of factors limit the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs. ACF and OPA have required their grantees to report on various outcomes that the agencies use to measure the effectiveness of grantees' abstinence-until-marriage education programs. In addition, 6 of the 10 states in GAO's review have worked with third-party evaluators to assess the effectiveness of abstinence-until-marriage education programs in their states. Several factors, however, limit the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs. Most of the efforts to evaluate the effectiveness of abstinence-until-marriage education programs included in GAO's review have not met certain minimum scientific criteria--such as random assignment of participants and sufficient follow-up periods and sample sizes--that experts have concluded are necessary in order for assessments of program effectiveness to be scientifically valid, in part because such designs can be expensive and time-consuming to carry out. In addition, the results of efforts that meet the criteria of a scientifically valid assessment have varied and two key studies funded by HHS that meet these criteria have not yet been completed. When completed, these HHS-funded studies may add substantively to the body of research on the effectiveness of abstinence-until-marriage education programs.
Recommendation for Executive Action
Status: Closed - Implemented
Comments: In April 2008, an Administration for Children and Families (ACF) official reported that, in response to our recommendation, ACF began requiring in fiscal year 2007 that community-based grantees sign written assurances that the materials they propose using are accurate. This official also reported that grantees of the State Program were required to submit written assurances of the accuracy of the materials they planned to use starting in fiscal year 2008. ACF's fiscal year 2007 Community-Based Program announcement included the following assurance that applicants were required to sign, "I hereby attest and certify that all medical materials proposed in this application and funded during the project period of this grant are medically accurate". ACF's fiscal year 2009 State Program announcement stated that applicants were required to provide a plan to ACF on how they would ensure that program materials would be reviewed for medical accuracy. The program announcement also stated that, "should ACF find medically inaccurate information during the review process, or at any time during the grant project period, grantees will be required to correct the inaccuracies".
Recommendation: To address concerns about the scientific accuracy of materials used in abstinence-until-marriage education programs, we recommend that the Secretary of HHS develop procedures to help assure the accuracy of such materials used in the State and Community-Based Programs. To help provide such assurances, the Secretary could consider alternatives such as (1) extending the approach currently used by OPA to review the scientific accuracy of the factual statements included in abstinence-until-marriage education to materials used by grantees of ACF's Community-Based Program and requiring grantees of ACF's State Program to conduct such reviews or (2) requiring grantees of both programs to sign written assurances in their grant applications that the materials they propose using are accurate.
Agency Affected: Department of Health and Human Services