Performance Budgeting:

Observations on the Use of OMB's Program Assessment Rating Tool for the Fiscal Year 2004 Budget

GAO-04-174: Published: Jan 30, 2004. Publicly Released: Jan 30, 2004.

Additional Materials:

Contact:

Susan J. Irving
(202) 512-9142
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART) is meant to provide a consistent approach to evaluating federal programs during budget formulation. To better understand its potential, congressional requesters asked GAO to examine (1) how PART changed OMB's fiscal year 2004 budget decisionmaking process, (2) PART's relationship to the Government Performance and Results Act of 1993 (GPRA), and (3) PART's strengths and weaknesses as an evaluation tool.

PART helped structure OMB's use of performance information for its internal program and budget analysis, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. OMB and agency staff said this helped OMB staff with varying levels of experience focus on similar issues. Our analysis confirmed that one of PART's major impacts was its ability to highlight OMB's recommended changes in program management and design. Much of PART's potential value lies in the related program recommendations, but realizing these benefits requires sustained attention to implementation and oversight to determine if desired results are achieved. OMB needs to be cognizant of this as it considers capacity and workload issues in PART. There are inherent challenges in assigning a single rating to programs having multiple purposes and goals. OMB devoted considerable effort to promoting consistent ratings, but challenges remain in addressing inconsistencies among OMB staff, such as interpreting PART guidance and defining acceptable measures. Limited credible evidence on results also constrained OMB's ability to rate program effectiveness, as evidenced by the almost 50 percent of programs rated "results not demonstrated." PART is not well integrated with GPRA--the current statutory framework for strategic planning and reporting. By using the PART process to review and sometimes replace GPRA goals and measures, OMB is substituting its judgment for a wide range of stakeholder interests. The PART/GPRA tension was further highlighted by challenges in defining a unit of analysis useful for both program-level budget analysis and agency planning purposes. Although PART can stimulate discussion on program-specific measurement issues, it cannot substitute for GPRA's focus on thematic goals and department- and governmentwide crosscutting comparisons. Moreover, PART does not currently evaluate similar programs together to facilitate trade-offs or make relative comparisons. PART clearly must serve the President's interests. However, the many actors whose input is critical to decisions will not likely use performance information unless they feel it is credible and reflects a consensus on goals. It will be important for OMB to discuss timely with Congress the focus of PART assessments and clarify the results and limitations of PART and the underlying performance information. A more systematic congressional approach to providing its perspective on performance issues and goals could facilitate OMB's understanding of congressional priorities and thus increase PART's usefulness in budget deliberations.

Matter for Congressional Consideration

  1. Status: Closed - Implemented

    Comments: In response to GAO's work suggesting the need for a more robust Congressional strategy for indicating its performance priorities and using performance information in its deliberations, the House Budget Committee included in the FY 2009 Congressional Budget Resolution a requirement that all committees include recommendations for improved governmental performance in their annual "views and estimates." In response, Committees indeed indicated their performance priorities and have held hearings on these areas.

    Matter: In order to facilitate an understanding of congressional priorities and concerns, Congress may wish to consider the need for a strategy that could include (1) establishing a vehicle for communicating performance goals and measures for key congressional priorities and concerns; (2) developing a more structured oversight agenda to permit a more coordinated congressional perspective on crosscutting programs and policies; and (3) using such an agenda to inform its authorization, oversight, and appropriations processes.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: As we said in our follow-up October 2005 report on the Program Assessment Rating Tool (PART), most Congressional staff we spoke with still reported a lack of up-front consultation about PART. Consistent with the spirit of our recommendation, in June 2005, the Administration has introduced The Government Reorganization and Program Performance Improvement Act of 2005 (GRPPI). Under GRPPI, a sunset commission would submit to the Congress for its approval a proposed schedule for reviewing the performance of, and need for, Executive Branch agencies and programs at least once every 10 years. More recently, OMB required all executive branch agencies to schedule meetings with the appropriate Congressional committees to explore ways that PART information could prove valuable to them.

    Recommendation: The Director of OMB should attempt to generate, early in the PART process, an ongoing, meaningful dialogue with congressional appropriations, authorization, and oversight committees about what they consider to be the most important performance issues and program areas warranting review.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  2. Status: Closed - Implemented

    Comments: To review the relative contributions of similar programs to common or crosscutting goals and outcomes and to facilitate comparisons and trade-offs between such programs, GAO recommended that OMB conduct Program Assessment Rating Tool (PART) assessments of similar programs and activities in the same year. In response to these recommendations, for the FY06 budget, OMB used PART in conducting two crosscutting analyses to inform the President's Budget request: 1) Community and Economic Development programs, and 2) Rural Water programs. The Community and Economic Development (CED) analysis formed the basis for the administration's proposal to consolidate various CED programs into a $3.7 billion program at the Department of Commerce. The Budget also indicated the Administration's intention to develop recommendations to consolidate and reform Rural Water programs based on the Rural Water cross-cut analysis mentioned above.

    Recommendation: The Director of OMB should maximize the opportunity to review similar programs or activities in the same year to facilitate comparisons and trade-offs.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  3. Status: Closed - Not Implemented

    Comments: In our January 2004 report on PART, OMB said that they "are committed to assessing 100 percent of programs using the PART except for programs of insignificant size or impact." It has already assessed 60% of federal programs and is in the process of assessing the next 20%. By February 2006, OMB will have assessed 80% of all federal programs. GAO has followed up with OMB over the past 18 months and their views on assessing 100 percent of federal programs have not changed. Therefore, GAO is closing this recommendation as "not implemented."

    Recommendation: The Director of OMB should reconsider plans for 100 percent coverage of federal programs and, instead, target for review a significant percentage of major and meaningful government programs based on such factors as the relative priorities, costs, and risks associated with related clusters of programs and activities.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  4. Status: Closed - Implemented

    Comments: In its Program Assessment Rating Tool (PART) guidance OMB removed the requirement that program evaluations be conducted every 2-5 years, instead tying the timing and frequency to the needs and resources of each program. OMB also clarified the evaluation strategies it wants for PART. The guidance notes that while certain types of evaluations (such as randomized controlled trials) are particularly well suited to measuring impacts, these studies are not suitable or feasible for every program, and a variety of evaluation methods may need to be considered. It recognizes that well-designed quasi-experimental studies may provide useful information about the impact of a program and/or can help address how or why a program is effective (or ineffective) and that evaluations must be appropriate to the type and size of the program. The guidance further notes that agencies and OMB should consult evaluation experts, in-house and/or external, as appropriate, when choosing or vetting rigorous evaluations. It also still generally prohibits evaluations conducted by the program itself from being considered "independent," but now recognizes that evaluations conducted by an agency's Inspector General or program-evaluation office would be considered independent, and notes that evaluations contracted out to a third party may, on a case-by-case basis, qualify as sufficiently independent.

    Recommendation: The Director of OMB should clarify OMB's expectations to agencies regarding the allocation of scarce evaluation resources among programs, the timing of such evaluations, as well as the evaluation strategies it wants for the purposes of the PART, and consider using internal agency evaluations as evidence on a case-by-case basis--whether conducted by agencies, contractors, or other parties.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  5. Status: Closed - Implemented

    Comments: OMB's 2006 PART guidance significantly expanded the discussion in this area. For example, almost two full pages of guidance offer criteria for combining individual programs to make a single PART "program". The guidance also offers suggestions on handling programs that are independent of each other as well as programs that are interdependent. In addition to these improvements, OMB created and posted on its website a PART "FAQ" that addresses a host of related issues. The FAQ addresses common questions and concerns about PART and makes "conventional wisdom" readily available to interested parties. Both of these steps should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations. The FY 2006 guidance also expanded on its discussion of evaluation quality and included references to GAO reports on evaluation, as well as other evaluation resources from a list GAO has circulated to federal evaluation officials including OMB staff. For example, the new section explains what, ideally, evaluations should measure (impact), the types of evaluations considered to be of the highest quality, and when and how other types of evaluations might be acceptable forms of evidence. Clarifying what constitutes quality evaluation evidence should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations. Finally, the 2006 PART guidance significantly expanded the discussion on output vs. outcome measures, resulting in more and clearer information agencies can rely upon when completing PART assessments. For example, OMB added several examples of output vs. outcome measures and how and when to create proxy outcome measures when true outcome measures are unavailable. The FAQ addresses common questions and concerns about PART and makes "conventional wisdom" readily available to interested parties. Both of these steps should help increase the credibility and acceptance of OMB's program ratings and associated program recommendations.

    Recommendation: The Director of OMB should continue to improve the PART guidance by (1) expanding the discussion of how the unit of analysis is to be determined to include trade-offs made when defining a unit of analysis, implications of how the unit of analysis is defined, or both; (2) clarifying when output versus outcome measures are acceptable; and (3) better defining an "independent, quality evaluation."

    Agency Affected: Executive Office of the President: Office of Management and Budget

  6. Status: Closed - Implemented

    Comments: To improve OMB's ability to judge the efficacy of the Program Assessment Rating Tool (PART), GAO recommended that the Director of OMB centrally monitor agency implementation and progress on PART recommendations and report such progress in OMB's budget submission to Congress. In response to this recommendation, in the PART Summary Worksheets submitted with the FY 2006 President's Budget Request, OMB reported on the status of each program's recommendations. In April 2005, OMB also implemented PARTWeb, a web-based data collection tool to, among other things, centrally track the implementation and status of PART recommendations.

    Recommendation: The Director of OMB should centrally monitor agency implementation and progress on PART recommendations and report such progress in OMB's budget submission to Congress. Governmentwide councils may be effective vehicles for assisting OMB in these efforts.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  7. Status: Closed - Not Implemented

    Comments: Since 2005, the PART guidance has contained language indicating that "OMB and agencies must agree on appropriate measures early to allow for review with relevant stakeholders, if needed", noting that GPRA requires stakeholders be consulted if strategic goals are revised. GAO's October 2005 report on PART recognized that OMB has taken some steps to further clarify the PART-GPRA relationship, but noted that many agencies still struggle to balance the differing needs of the budget and planning processes and their various stakeholders. To date, unresolved tensions between PART and GPRA appear to continue to contribute to a lack of consensus about what to measure and how to measure it. We are closing this recommendation as not implemented.

    Recommendation: The Director of OMB should seek to achieve the greatest benefit from both GPRA and PART by articulating and implementing an integrated, complementary relationship between the two.

    Agency Affected: Executive Office of the President: Office of Management and Budget

 

Explore the full database of GAO's Open Recommendations »

Jul 29, 2014

Jul 18, 2014

Jun 23, 2014

Jun 17, 2014

Jun 12, 2014

Jun 10, 2014

May 28, 2014

Mar 27, 2014

Mar 6, 2014

Looking for more? Browse all our products here