Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget

To improve OMB's ability to judge the efficacy of the Program Assessment Rating Tool (PART), GAO recommended that the Director of OMB centrally monitor agency implementation and progress on PART recommendations and report such progress in OMB's budget submission to Congress. In response to this recommendation, in the PART Summary Worksheets submitted with the FY 2006 President's Budget Request, OMB reported on the status of each program's recommendations. In April 2005, OMB also implemented PARTWeb, a web-based data collection tool to, among other things, centrally track the implementation and status of PART recommendations.

OMB's 2006 PART guidance significantly expanded the discussion in this area. For example, almost two full pages of guidance offer criteria for combining individual programs to make a single PART "program". The guidance also offers suggestions on handling programs that are independent of each other as well as programs that are interdependent. In addition to these improvements, OMB created and posted on its website a PART "FAQ" that addresses a host of related issues. The FAQ addresses common questions and concerns about PART and makes "conventional wisdom" readily available to interested parties. Both of these steps should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations. The FY 2006 guidance also expanded on its discussion of evaluation quality and included references to GAO reports on evaluation, as well as other evaluation resources from a list GAO has circulated to federal evaluation officials including OMB staff. For example, the new section explains what, ideally, evaluations should measure (impact), the types of evaluations considered to be of the highest quality, and when and how other types of evaluations might be acceptable forms of evidence. Clarifying what constitutes quality evaluation evidence should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations. Finally, the 2006 PART guidance significantly expanded the discussion on output vs. outcome measures, resulting in more and clearer information agencies can rely upon when completing PART assessments. For example, OMB added several examples of output vs. outcome measures and how and when to create proxy outcome measures when true outcome measures are unavailable. The FAQ addresses common questions and concerns about PART and makes "conventional wisdom" readily available to interested parties. Both of these steps should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations.

In its Program Assessment Rating Tool (PART) guidance OMB removed the requirement that program evaluations be conducted every 2-5 years, instead tying the timing and frequency to the needs and resources of each program. OMB also clarified the evaluation strategies it wants for PART. The guidance notes that while certain types of evaluations (such as randomized controlled trials) are particularly well suited to measuring impacts, these studies are not suitable or feasible for every program, and a variety of evaluation methods may need to be considered. It recognizes that well-designed quasi-experimental studies may provide useful information about the impact of a program and/or can help address how or why a program is effective (or ineffective) and that evaluations must be appropriate to the type and size of the program. The guidance further notes that agencies and OMB should consult evaluation experts, in-house and/or external, as appropriate, when choosing or vetting rigorous evaluations. It also still generally prohibits evaluations conducted by the program itself from being considered "independent," but now recognizes that evaluations conducted by an agency's Inspector General or program-evaluation office would be considered independent, and notes that evaluations contracted out to a third party may, on a case-by-case basis, qualify as sufficiently independent.

To review the relative contributions of similar programs to common or crosscutting goals and outcomes and to facilitate comparisons and trade-offs between such programs, GAO recommended that OMB conduct Program Assessment Rating Tool (PART) assessments of similar programs and activities in the same year. In response to these recommendations, for the FY06 budget, OMB used PART in conducting two crosscutting analyses to inform the President's Budget request: 1) Community and Economic Development programs, and 2) Rural Water programs. The Community and Economic Development (CED) analysis formed the basis for the administration's proposal to consolidate various CED programs into a $3.7 billion program at the Department of Commerce. The Budget also indicated the Administration's intention to develop recommendations to consolidate and reform Rural Water programs based on the Rural Water cross-cut analysis mentioned above.

As we said in our follow-up October 2005 report on the Program Assessment Rating Tool (PART), most Congressional staff we spoke with still reported a lack of up-front consultation about PART. Consistent with the spirit of our recommendation, in June 2005, the Administration has introduced The Government Reorganization and Program Performance Improvement Act of 2005 (GRPPI). Under GRPPI, a sunset commission would submit to the Congress for its approval a proposed schedule for reviewing the performance of, and need for, Executive Branch agencies and programs at least once every 10 years. More recently, OMB required all executive branch agencies to schedule meetings with the appropriate Congressional committees to explore ways that PART information could prove valuable to them.