Improvements Needed for Measuring Awardee Performance in Meeting Medical and Public Health Preparedness Goals
GAO-13-278: Published: Mar 22, 2013. Publicly Released: Mar 22, 2013.
What GAO Found
Available measures and awardee data provide some evidence that Hospital Preparedness Program (HPP) awardees have generally made progress in carrying out activities to achieve medical preparedness goals; however, the Department of Health and Human Services (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) lacked a comprehensive performance management system to fully assess awardee progress. According to prior GAO work and the GPRA Modernization Act of 2010 (GPRAMA), successful performance measurement systems should include a select set of performance measures tied to realistically achievable targets with clearly defined milestones. GAO's analysis of ASPR data showed general progress. For example, the percentage of all 62 awardees' participating hospitals with medical evacuation and shelter-in-place plans increased from 79.9 percent to 88.3 percent from fiscal year 2007 to fiscal year 2011. However, while ASPR collected data on a range of activities, it did not have consistent performance measures and targets in place across this entire period. Beginning with fiscal year 2012, ASPR developed new provisional performance measures for the eight new capabilities that awardees are to use for HPP planning for the next 5 years and set 5-year targets for these measures. However, it did not develop annual milestones, which may make it difficult for ASPR and awardees to assess incremental progress toward meeting HPP goals.
Although Public Health Emergency Preparedness (PHEP) program awardees are improving in their ability to carry out preparedness activities, HHS's Centers for Disease Control and Prevention (CDC) lacked a consistent set of performance measures and targets to adequately assess the degree of awardee progress toward meeting the PHEP goal. For example, from fiscal years 2007 through 2011, the number of measures ranged from 5 to 30 in any one year, and CDC had only four targets for any of them. GAO's analysis of CDC data showed general progress on the measures. For example, for one measure, the average time it took the 62 awardees to assemble appropriate response staff decreased from 35 minutes in 2007 to 31 minutes in 2011, although the 50 state awardees did not always meet the 60-minute target that CDC set for them starting in 2009. Beginning with fiscal year 2012, CDC released 47 provisional performance measures for 14 of the 15 new PHEP capabilities but developed only four associated targets. Without consistent performance measures and associated targets, in accordance with prior GAO work and GPRAMA, CDC may not be able to assess how awardees are making progress toward meeting the PHEP goal.
HHS uses internal databases, site visits, and audit reports to help awardees meet HPP and PHEP application, financial, and reporting requirements. ASPR and CDC use internal databases to generate reports on awardee progress in meeting application renewal and reporting deadlines, to assess application completeness, and to periodically query databases in order to review financial information. They also conduct regular site visits and review state and federal audit reports to help awardees meet program requirements and assess awardees' use of funds. ASPR and CDC require awardees that have problems managing their HPP or PHEP funds to complete corrective action plans, and they restrict awardees' access to funds in more serious cases.
Why GAO Did This Study
HHS provides funding to the 50 states and 12 municipalities, territories, and freely associated states, primarily through ASPR's HPP and CDC's PHEP cooperative agreements, to help them build their capability to respond to emergencies such as hurricanes, pandemics, or terrorist events. The 62 awardees are to use this funding to help achieve the HPP goals of strengthening hospital preparedness and medical surge capacity and the PHEP goal of strengthening public health preparedness, and they must meet certain application, financial, and reporting requirements. GAO was asked about the effects of federal support on state and local response capabilities. GAO (1) assessed awardee progress in meeting HPP goals and how ASPR measures that progress, (2) assessed awardee progress in meeting the PHEP goal and how CDC measures that progress, and (3) identified the mechanisms HHS uses to ensure that awardees are meeting application, financial, and reporting requirements. GAO reviewed HPP and PHEP guidance, performance measures, and other documents; interviewed HHS officials; and analyzed HPP and PHEP data for fiscal years 2007 through 2011.
What GAO Recommends
GAO recommends that (1) HHS develop objective and quantifiable performance targets and incremental milestones tied to HPP and PHEP performance measures and (2) ensure that measures remain consistent and comparable to sufficiently measure progress. HHS generally agreed with GAO's recommendations but indicated that it would not be able to fully implement them for several years.
For more information, contact Marcia Crosse at (202) 512-7114 or firstname.lastname@example.org.
Recommendations for Executive Action
Comments: Since we examined the HPP and PHEP cooperative agreements in 2012, ASPR and CDC have reported plans to develop performance targets and incremental milestones that correspond to HPP and PHEP performance measures. In a July 2014 update, HHS officials noted that for the HPP program, ASPR expected to develop new targets and milestones in fiscal years 2015-2016. In June 2016, HHS officials told us that ASPR will establish targets and milestones for budget period 4 (July 1, 2015 through June 30, 2016) and 5 (July 1, 2016 through June 30, 2017. As of August 2016, the most recent HPP measures and guidance document provided by HHS for budget period 4 (2015-2016) did not include performance targets or incremental milestones for the 2 overall HPP program measures or 14 associated indicators. The guidance includes Likert-type scales for most, if not all, of the 14 indicators through which awardees are to rate their levels of attainment for each of the indicators. The guidance provides criteria for each of the numbers in the scale; however, it does not set a minimum number that ASPR may expect an awardee to have achieved. In addition, as of August 29, 2016, the guidance for budget period 5 (2016-2017) is not posted on ASPR's website. For the PHEP program, officials stated in July 2014 that CDC planned to introduce evidence-based targets for several key performance measures through 2014 and possibly later. The agency also noted that, as retained measures continued to stabilize, CDC could set interim targets for some measures, along with targets based on individual awardee benchmarks. In July 2016, CDC provided its performance measures guidance for budget periods four and 5. CDC's PHEP performance measures guidance for budget period 5 contains targets for less than half (6) of the 14 PHEP performance measures. CDC did not indicate when it anticipated developing performance targets for the remaining measures.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should direct ASPR and CDC to develop objective and quantifiable performance targets and incremental milestones that correspond to the new HPP and PHEP performance measures, against which HHS can gauge progress toward the medical and public health preparedness goals of the cooperative agreements and direct technical assistance, as needed.
Agency Affected: Department of Health and Human Services
Comments: Since we examined the HPPS and PHEP cooperative agreements in 2012, ASPR and CDC have continued to make changes to the performance evaluation programs. For example, while ASPR planned to maintain its existing two HPP program measures to help ensure consistency across the current project period, it planned to use the remaining budget years in the project period (through 2017) to further hone, revise, and finalize HPP indicators for each HPP capability, which could result in additional changes, and to collect baseline data to develop incremental targets and milestones. In a July 2014 update, HHS stated that PHEP FY 2014 changes included a reduction from 28 to 15 performance measures and from 3 to a single evaluation tool. Two of the 15 performance measures would be in place for 2014 only, and for FY 2015 the program would have 13 performance measures and 1 evaluation tool, which were expected to remain in place for the remaining years of the project period. In August 2015, HHS provided us an HPP measures and guidance document that shows that, as planned, the refined measures remained constant from the previous budget period at two overall measures, called program measures, each with seven indicators. HHS did not provide a corresponding PHEP measures and guidance document for 2015. In June 2016, HHS reaffirmed that ASPR will keep the existing two HPP program measures and the seven indicators under each measure through the end of the 5-year project cycle, which terminates June 30, 2017. HHS provided the HPP performance measure guidance for budget period 4 (July 1, 2015 through June 30, 2016), which contains the 2 program measures and 14 indicators. As of August 29, 2016, the HPP budget period 5 (July 1, 2016 through June 30, 2017) performance measure guidance is not posted on ASPR's website, so we cannot assess it for the consistency of performance measures with those of prior budget periods. CDC's PHEP performance measures have remained consistent across budget periods 4 and 5. ASPR and CDC have committed to the consistency of the performance measures in the last 3 years of the current project cycle. Because part of our recommendation includes consistency of performance measures into future project cycles, this recommendation will remain open as HPP and PHEP enter into a new 5-year project cycle July 1, 2017. GAO anticipates keeping this recommendation open at least through the first budget period of the next project cycle.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should ensure that performance measures and targets remain consistent across the 5-year project cycle and that any future measures be comparable to determine whether awardees are making progress toward meeting short- and long-term medical and public health preparedness goals of the cooperative agreements.
Agency Affected: Department of Health and Human Services