Program Evaluation:

OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results

GAO-06-67: Published: Oct 28, 2005. Publicly Released: Oct 28, 2005.

Additional Materials:

Contact:

Nancy R. Kingsbury
(202) 512-2700
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Office of Management and Budget (OMB) designed the Program Assessment Rating Tool (PART) as a diagnostic tool to draw on program performance and evaluation information for forming conclusions about program benefits and recommending adjustments to improve results. To assess progress in improving the evidence base for PART assessments, GAO was requested to examine (1) agencies' progress in responding to OMB's recommendations to evaluate programs, (2) factors facilitating or impeding agencies' progress, and (3) whether agencies' evaluations appear to be designed to yield the information on program results that OMB expects.

GAO examined agency progress on 20 of the 40 evaluations OMB recommended in its PART reviews at four federal agencies: the Department of Energy, Department of Health and Human Services, Department of Labor, and Small Business Administration. About half the programs GAO reviewed had completed an evaluation in the 2 years since those PART reviews were published; 4 more were in progress and 3 were still being planned. Program restructuring canceled plans for the remaining 2 evaluations. Several agencies struggled to identify appropriate outcome measures and credible data sources before they could evaluate program effectiveness. Evaluation typically competed with other program activities for funds, so managers may be reluctant to reallocate funds to evaluation. Some agency officials thought that evaluations should be targeted to areas of policy significance or uncertainty. However, all four agencies indicated that the visibility of an OMB recommendation brought agency management attention--and sometimes funds--to get the evaluations done. Moreover, by coordinating their evaluation activities, agencies met these challenges by leveraging their evaluation expertise and strategically prioritizing their evaluation resources to the studies that they considered most important. Because the OMB recommendations were fairly general, agencies had flexibility in interpreting the kind of information OMB expected. Some program managers disagreed with OMB on the purpose of their evaluations, their quality, and the usefulness of "independent" evaluations by third parties unfamiliar with their programs. Agency officials concerned about an increased focus on process said that they were more interested in learning how to improve program results than in meeting an OMB checklist. Since a few programs did not discuss their evaluation plans with OMB, it is not certain whether OMB will find their ongoing evaluations useful during the programs' next PART review. GAO concludes that the PART review process stimulated agencies to increase their evaluation capacity and available information on program results. Further, agencies are likely to design evaluations to meet their own needs--that is, in-depth analyses that inform program improvement. If OMB wants evaluations with a broader scope, such as information that helps determine a program's relevance or value, it will need to take steps to shape both evaluation design and execution. Finally, because agency evaluation resources tend to be limited, they are most usefully focused on important areas of uncertainty. Regular performance reporting is key to good management, but requiring all federal programs to conduct frequent evaluation studies is likely to result in superficial reviews of little utility and to overwhelm agency evaluation capacity.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: OMB should encourage agencies to discuss their plans for program evaluations--especially those in response to an OMB recommendation--with OMB and with congressional and other program stakeholders to ensure that their findings will be timely, relevant, and credible and that they will be used to inform policy and management decisions.

    Agency Affected: Executive Office of the President: Office of Management and Budget

    Status: Closed - Implemented

    Comments: OMB reports that agencies are now required to brief their appropriators and authorizers on past, completed, and planned PART assessments, including planned program evaluations. This represents partial implementation of our recommendation because it does not clearly distinguish OMB PART assessments from program evaluation studies, nor recognize the need to discuss evaluation plans and criteria with congressional stakeholders to ensure that their findings will be timely, relevant, credible, and used to inform decisions.

    Recommendation: OMB should engage in dialogue with agencies and congressional stakeholders on a risk-based allocation of scarce evaluation resources among programs, based on size, importance, or uncertain effectiveness, and on the timing of such evaluations.

    Agency Affected: Executive Office of the President: Office of Management and Budget

    Status: Closed - Not Implemented

    Comments: OMB reports that it has taken no formal actions in this area but that discussion of agency evaluation resources largely occur as part of agency budget discussions. OMB reports continuing discussions with agencies and outside stakeholders on the need to invest in rigorous evaluations, has offered agencies (in 2009 and 2010) additional funds for rigorous evaluations, but does not plan to address how agencies should allocate evaluation resources among its programs.

    Recommendation: OMB should continue to improve its PART guidance and training of examiners on evaluation to acknowledge a wide range of appropriate methods.

    Agency Affected: Executive Office of the President: Office of Management and Budget

    Status: Closed - Implemented

    Comments: OMB sponsored workshops for OMB and agency staff that clarified that experimental research designs involving randomization were not the only acceptable method for evaluating federal program effectiveness and might not be feasible in some cases. OMB also sponsored a workshop where agency evaluators discussed the applicability of alternative evaluation designs.

    Jul 22, 2014

    Jun 17, 2014

    Jun 11, 2014

    Jun 10, 2014

    May 28, 2014

    May 21, 2014

    May 12, 2014

    May 7, 2014

    Apr 30, 2014

    Looking for more? Browse all our products here