Improvements Needed for Measuring Awardee Performance in Meeting Medical and Public Health Preparedness Goals
GAO-13-278: Published: Mar 22, 2013. Publicly Released: Mar 22, 2013.
What GAO Found
Available measures and awardee data provide some evidence that Hospital Preparedness Program (HPP) awardees have generally made progress in carrying out activities to achieve medical preparedness goals; however, the Department of Health and Human Services (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) lacked a comprehensive performance management system to fully assess awardee progress. According to prior GAO work and the GPRA Modernization Act of 2010 (GPRAMA), successful performance measurement systems should include a select set of performance measures tied to realistically achievable targets with clearly defined milestones. GAO's analysis of ASPR data showed general progress. For example, the percentage of all 62 awardees' participating hospitals with medical evacuation and shelter-in-place plans increased from 79.9 percent to 88.3 percent from fiscal year 2007 to fiscal year 2011. However, while ASPR collected data on a range of activities, it did not have consistent performance measures and targets in place across this entire period. Beginning with fiscal year 2012, ASPR developed new provisional performance measures for the eight new capabilities that awardees are to use for HPP planning for the next 5 years and set 5-year targets for these measures. However, it did not develop annual milestones, which may make it difficult for ASPR and awardees to assess incremental progress toward meeting HPP goals.
Although Public Health Emergency Preparedness (PHEP) program awardees are improving in their ability to carry out preparedness activities, HHS's Centers for Disease Control and Prevention (CDC) lacked a consistent set of performance measures and targets to adequately assess the degree of awardee progress toward meeting the PHEP goal. For example, from fiscal years 2007 through 2011, the number of measures ranged from 5 to 30 in any one year, and CDC had only four targets for any of them. GAO's analysis of CDC data showed general progress on the measures. For example, for one measure, the average time it took the 62 awardees to assemble appropriate response staff decreased from 35 minutes in 2007 to 31 minutes in 2011, although the 50 state awardees did not always meet the 60-minute target that CDC set for them starting in 2009. Beginning with fiscal year 2012, CDC released 47 provisional performance measures for 14 of the 15 new PHEP capabilities but developed only four associated targets. Without consistent performance measures and associated targets, in accordance with prior GAO work and GPRAMA, CDC may not be able to assess how awardees are making progress toward meeting the PHEP goal.
HHS uses internal databases, site visits, and audit reports to help awardees meet HPP and PHEP application, financial, and reporting requirements. ASPR and CDC use internal databases to generate reports on awardee progress in meeting application renewal and reporting deadlines, to assess application completeness, and to periodically query databases in order to review financial information. They also conduct regular site visits and review state and federal audit reports to help awardees meet program requirements and assess awardees' use of funds. ASPR and CDC require awardees that have problems managing their HPP or PHEP funds to complete corrective action plans, and they restrict awardees' access to funds in more serious cases.
Why GAO Did This Study
HHS provides funding to the 50 states and 12 municipalities, territories, and freely associated states, primarily through ASPR's HPP and CDC's PHEP cooperative agreements, to help them build their capability to respond to emergencies such as hurricanes, pandemics, or terrorist events. The 62 awardees are to use this funding to help achieve the HPP goals of strengthening hospital preparedness and medical surge capacity and the PHEP goal of strengthening public health preparedness, and they must meet certain application, financial, and reporting requirements. GAO was asked about the effects of federal support on state and local response capabilities. GAO (1) assessed awardee progress in meeting HPP goals and how ASPR measures that progress, (2) assessed awardee progress in meeting the PHEP goal and how CDC measures that progress, and (3) identified the mechanisms HHS uses to ensure that awardees are meeting application, financial, and reporting requirements. GAO reviewed HPP and PHEP guidance, performance measures, and other documents; interviewed HHS officials; and analyzed HPP and PHEP data for fiscal years 2007 through 2011.
What GAO Recommends
GAO recommends that (1) HHS develop objective and quantifiable performance targets and incremental milestones tied to HPP and PHEP performance measures and (2) ensure that measures remain consistent and comparable to sufficiently measure progress. HHS generally agreed with GAO's recommendations but indicated that it would not be able to fully implement them for several years.
For more information, contact Marcia Crosse at (202) 512-7114 or email@example.com.
Recommendations for Executive Action
Comments: In June 2013, ASPR published the HPP measures and guidance for the second year of the project period, which began in 2012. Based on analysis of mid- and end-of-year data from the first year, ASPR reduced the number of measures and indicators for which it collects awardee data. It developed two overall measures, which it calls program measures, each with seven indicators that generally serve as a performance measurement function and reflect activites for each of the eight HPP capabilities. However, HPP performance targets remain set at 100 percent of awardees and health care coalitions achieving each measure or indicator by the end of the 5-year project cycle, and ASPR has not yet developed interim milestones for these targets to help gauge incremental progress. CDC also issued the PHEP measures and guidance for the second year of the project period in June 2013. CDC reduced the number of measures for which it collects awardee data and developed new evaluation tools. However, CDC has not developed any additional targets other than those we described in the 2012 report. In a July 2014 update, HHS officials noted that for the HPP program, agency officials expect to develop new targets and milestones in fiscal years 2015-2016. For the PHEP program, officials stated that CDC plans to introduce evidence-based targets for several key performance measures through 2014 and possibly later. The agency also noted that, as retained measures continue to stabilize, CDC can set interim targets for some measures, along with targets based on individual awardee benchmarks. While these changes continue for both programs, the agencies are not fully able to conduct consistent review across the project cycle to determine progress.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should direct ASPR and CDC to develop objective and quantifiable performance targets and incremental milestones that correspond to the new HPP and PHEP performance measures, against which HHS can gauge progress toward the medical and public health preparedness goals of the cooperative agreements and direct technical assistance, as needed.
Agency Affected: Department of Health and Human Services
Comments: While ASPR plans to maintain its existing two program measures to help ensure consistency across the current project period, it plans to use the remaining 3 years to further hone, revise, and finalize HPP indicators for each HPP capability, which may result in additional changes, and collect baseline data to develop incremental targets and milestones. CDC anticipates that "a substantial number" of the current performance measures will remain stable for the remainder of the project period. However, the agency also plans to collect baseline data for several measures that underwent changes from 2012 to 2013, which could result in further changes. CDC also plans to introduce new targets for certain performance measures over the next few years and is considering developing interim targets as well. CDC did not specify when in the current project period it planned to develop or finalize any new targets. In addition, CDC plans to refine existing measures for future budget periods, including introducing new measures, but did not provide any information on how it plans to maintain consistency or comparability across project years. In a July 2014 update, HHS officials stated that the HPP program measures established in the program?s budget period 2 (2012-2013) will remain stable throughout the remainder of the five-year project period in order to monitor progress over the years. However, targets may change through 2016, as noted in the update to GAO?s related finding. For the PHEP program, HHS stated that FY 2014 (budget period 3) changes included a reduction from 28 to 15 performance measures and 3 to 1 evaluation tool. Two of the 15 performance measures will be in place for 2014 only, and in FY 2015 the program will have 13 performance measures and 1 evaluation tool, which are expected to remain in place for the remaining years of the cooperative agreement project period. PHEP targets, as noted in the response to the related finding, are expected to continue to evolve, and CDC is also developing a broader PHEP evaluation framework to include other methods, outside of performance measures, to monitor and evaluate awardee programs. While these changes continue for both programs, the agencies are not fully able to conduct consistent review across the project cycle to determine progress.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should ensure that performance measures and targets remain consistent across the 5-year project cycle and that any future measures be comparable to determine whether awardees are making progress toward meeting short- and long-term medical and public health preparedness goals of the cooperative agreements.
Agency Affected: Department of Health and Human Services