National Preparedness: Improvements Needed for Measuring Awardee Performance in Meeting Medical and Public Health Preparedness Goals
Highlights
What GAO Found
Available measures and awardee data provide some evidence that Hospital Preparedness Program (HPP) awardees have generally made progress in carrying out activities to achieve medical preparedness goals; however, the Department of Health and Human Services (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) lacked a comprehensive performance management system to fully assess awardee progress. According to prior GAO work and the GPRA Modernization Act of 2010 (GPRAMA), successful performance measurement systems should include a select set of performance measures tied to realistically achievable targets with clearly defined milestones. GAO's analysis of ASPR data showed general progress. For example, the percentage of all 62 awardees' participating hospitals with medical evacuation and shelter-in-place plans increased from 79.9 percent to 88.3 percent from fiscal year 2007 to fiscal year 2011. However, while ASPR collected data on a range of activities, it did not have consistent performance measures and targets in place across this entire period. Beginning with fiscal year 2012, ASPR developed new provisional performance measures for the eight new capabilities that awardees are to use for HPP planning for the next 5 years and set 5-year targets for these measures. However, it did not develop annual milestones, which may make it difficult for ASPR and awardees to assess incremental progress toward meeting HPP goals.
Although Public Health Emergency Preparedness (PHEP) program awardees are improving in their ability to carry out preparedness activities, HHS's Centers for Disease Control and Prevention (CDC) lacked a consistent set of performance measures and targets to adequately assess the degree of awardee progress toward meeting the PHEP goal. For example, from fiscal years 2007 through 2011, the number of measures ranged from 5 to 30 in any one year, and CDC had only four targets for any of them. GAO's analysis of CDC data showed general progress on the measures. For example, for one measure, the average time it took the 62 awardees to assemble appropriate response staff decreased from 35 minutes in 2007 to 31 minutes in 2011, although the 50 state awardees did not always meet the 60-minute target that CDC set for them starting in 2009. Beginning with fiscal year 2012, CDC released 47 provisional performance measures for 14 of the 15 new PHEP capabilities but developed only four associated targets. Without consistent performance measures and associated targets, in accordance with prior GAO work and GPRAMA, CDC may not be able to assess how awardees are making progress toward meeting the PHEP goal.
HHS uses internal databases, site visits, and audit reports to help awardees meet HPP and PHEP application, financial, and reporting requirements. ASPR and CDC use internal databases to generate reports on awardee progress in meeting application renewal and reporting deadlines, to assess application completeness, and to periodically query databases in order to review financial information. They also conduct regular site visits and review state and federal audit reports to help awardees meet program requirements and assess awardees' use of funds. ASPR and CDC require awardees that have problems managing their HPP or PHEP funds to complete corrective action plans, and they restrict awardees' access to funds in more serious cases.
Why GAO Did This Study
HHS provides funding to the 50 states and 12 municipalities, territories, and freely associated states, primarily through ASPR's HPP and CDC's PHEP cooperative agreements, to help them build their capability to respond to emergencies such as hurricanes, pandemics, or terrorist events. The 62 awardees are to use this funding to help achieve the HPP goals of strengthening hospital preparedness and medical surge capacity and the PHEP goal of strengthening public health preparedness, and they must meet certain application, financial, and reporting requirements. GAO was asked about the effects of federal support on state and local response capabilities. GAO (1) assessed awardee progress in meeting HPP goals and how ASPR measures that progress, (2) assessed awardee progress in meeting the PHEP goal and how CDC measures that progress, and (3) identified the mechanisms HHS uses to ensure that awardees are meeting application, financial, and reporting requirements. GAO reviewed HPP and PHEP guidance, performance measures, and other documents; interviewed HHS officials; and analyzed HPP and PHEP data for fiscal years 2007 through 2011.
Recommendations
GAO recommends that (1) HHS develop objective and quantifiable performance targets and incremental milestones tied to HPP and PHEP performance measures and (2) ensure that measures remain consistent and comparable to sufficiently measure progress. HHS generally agreed with GAO's recommendations but indicated that it would not be able to fully implement them for several years.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Health and Human Services | To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should direct ASPR and CDC to develop objective and quantifiable performance targets and incremental milestones that correspond to the new HPP and PHEP performance measures, against which HHS can gauge progress toward the medical and public health preparedness goals of the cooperative agreements and direct technical assistance, as needed. |
Since we examined the HPP and PHEP cooperative agreements in 2012, ASPR had developed few targets for the HPP program measures or their corresponding indicators that were contained in the HPP performance measurement guidance documents issued for Budget Periods (BP) 2-5, ending June 30, 2017. Additionally, the new HPP performance measure implementation guidance for the 5-year project cycle from 2017-2022 introduces 28 performance measures, with few having targets; the guidance notes that corresponding goals or targets may be set at a later date after data from the first budget period of this new project cycle has been reviewed. Regarding PHEP, CDC had developed performance targets for about half of the performance measures as of the PHEP BP5 performance measurement guidance (BP5 ended June 30, 2017). These performance measures generally remain the same, with existing targets, for BP1 (July 1, 2017-June 30, 2018) of the new 5-year budget cycle. GAO recognizes that it may not be appropriate to develop performance targets for every performance measure depending on the desired process or outcome; however, both agencies still have work to do in this area. In November 2017, both ASPR and CDC officials noted that they could not commit to setting consistent targets with incremental milestones over a budget cycle and therefore could not implement the recommendation. As of August 2018, there was no change from the agencies' position that they do not have plans to fully implement the recommendation at this time. ASPR and CDC continue to revise performance measures and have not established targets and milestones for more than half of any measures new or old. Based on various changes to HPP and PHEP, these programs have evolved beyond our original recommendations.
|
| Department of Health and Human Services | To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should ensure that performance measures and targets remain consistent across the 5-year project cycle and that any future measures be comparable to determine whether awardees are making progress toward meeting short- and long-term medical and public health preparedness goals of the cooperative agreements. |
Since we first examined the HPP and PHEP cooperative agreements in 2012, ASPR and CDC had made efforts to maintain consistency in their performance measures, particularly in the last 3 years of the prior project cycle which ended June 30, 2017. However, because part of the recommendation includes consistency of performance measures into future project cycles, we also examined whether both cooperative agreements continued to use basically the same performance measures into the current 5-year cycle, which began July 1, 2017. ASPR's HPP has made a significant change in its performance measures, introducing a new set of 28 performance measures for this new 5-year cycle. CDC's PHEP performance measures generally remained consistent in the last two budget periods of the prior 5-year cycle, and remained generally the same for the first year of the new 5-year cycle (some measures were "retired," though key components from a measure may continue to be used by CDC in other types of reviews). Additionally, in November 2017, both CDC and ASPR officials noted that they may need to continue to adjust the performance measures during the new 5-year cycle. As of August 2018, as a result of the change to HPP's measures and the agency statements in November 2017, GAO anticipates keeping this recommendation open at least for the next few budget periods, in order to determine whether the agencies maintain consistency with the performance measures during the new project cycle. ASPR and CDC continue to revise performance measures and stated that they will continue to do so as necessary. Based on various changes to HPP and PHEP, these programs have evolved beyond our original recommendations.
|