Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA's Risk-Based Grant Assessment Model
Fast Facts
The Federal Emergency Management Agency distributes money to states, territories, and urban areas to help them get ready for potential terrorist attacks. The amounts provided are based in part on a model FEMA uses to rank the risk posed to each area. Locations deemed at higher risk of an attack with major consequences, for example, may receive more money.
We looked at FEMA's main grant programs and what it has done to improve its risk model. We recommended FEMA take additional steps to bolster the model's quality and credibility, including fully documenting the assumptions on which the model is based and subjecting it to external review.
This photo shows FEMA headquarters.
Highlights
What GAO Found
GAO found that various factors affected Federal Emergency Management Agency (FEMA) State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grant awards from fiscal year 2008 through 2018. SHSP grant awards to states were based on two factors—(1) minimum amounts set in law each year, and (2) FEMA's risk model. For example, in fiscal year 2012, each state was to receive a minimum of approximately $2.74 million, with each state receiving additional funds based on its relative risk score. Conversely, UASI grant awards are made based on its FEMA's risk-based grant assessment model, which ranks each urban area relative to others in that year, and Department of Homeland Security (DHS) leadership decisions on how funding should be allocated. From fiscal year 2008 through 2018, the number of USAI grantees varied from year to year (see figure below).
Annual Number of Urban Area Security Initiative (UASI) Grantees for Fiscal Years 2008 through 2018
Since 2008, FEMA has taken steps to strengthen its risk-based grant assessment model, but has not incorporated additional scientific practices into its model. For example, in 2011 FEMA included more information in its model on potential targets and their vulnerability in each state and urban area, addressing a prior GAO recommendation. More recently in 2018, FEMA added additional factors to better assess vulnerability in each state and urban area, such as the number of special events where large crowds gather and soft targets susceptible to lone wolf attacks, among other things. However, GAO found that FEMA does not fully utilize scientific practices recognized by the National Research Council and the Office of Management and Budget as best practices. Specifically, FEMA did not fully document its model's underlying assumptions, such as the weights in its model or the justification for changes to these weights. FEMA also did not perform the level of analysis needed to determine how changes to its model could affect the resulting risk scores. Finally, FEMA has not coordinated an independent external peer review of its model. Applying such scientific practices could assist FEMA in further strengthening its model.
Why GAO Did This Study
FEMA, a component of DHS, provides preparedness grants to state, local, tribal, and territorial governments to help prepare for, prevent, protect against, respond to, recover from and mitigate terrorist attacks or other disasters. SHSP grants fund the nation's 56 states and territories, while UASI grants fund eligible urban areas. Grant allocations have been based, in part, on FEMA's risk-based grant assessment model, with states and urban areas deemed to be at higher risk receiving more grant dollars than those deemed at lower risk. Since 2008, GAO and others have assessed the model and made recommendations to strengthen it.
This report 1) describes SHSP and UASI grant awards during fiscal years 2008 through 2018, and factors affecting grant distributions; and 2) examines the steps that FEMA has taken to strengthen its risk assessment model for allocating preparedness grants, and any additional opportunities to improve the model. GAO analyzed the information in FEMA's model, and data on SHSP and UASI grant awards for fiscal years 2008 through 2018. GAO also interviewed FEMA and DHS officials and collected documents.
Recommendations
GAO is making three recommendations to FEMA to further strengthen its risk-based grant assessment model by (1) fully documenting the model's assumptions and justifications, (2) performing additional in-depth analyses, and (3) coordinating an external peer review. FEMA concurred with our recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Federal Emergency Management Agency | The FEMA Administrator should fully document the underlying assumptions and justifications that form the basis of the risk-based grant assessment model, such as the size of the weights assigned to Threat, Vulnerability, and Consequence, or the justification for changes to these weights from one year to the next. |
FEMA agreed that internal and external stakeholders should be provided a justification on why changes were made to the risk methodology as well as the distribution percentage weights for each of the overall components and individual elements. In July 2019, FEMA provided GAO with records of the agency's additional efforts documenting changes to their risk allocation formula. As part of the letter to stakeholders documenting changes made to the data sources or model weights, FEMA provided the reasoning behind those changes. Further, FEMA performed a sensitivity analysis for the Fiscal Year 2019 risk allocation model, and commissioned an external review of the allocation model. According to FEMA, the agency will continue to provide detailed information on the data and risk assessment model makeup as well as reasoning behind any changes to the model and/or data sources. This will continue to provide jurisdictions with the full reasoning behind any changes. In addition, these actions should increase the transparency of the allocation model to the benefit internal decision makers in any future efforts to improve the allocation model's risk methodology. As a result of these actions this recommendation is considered closed as implemented.
|
| Federal Emergency Management Agency | The FEMA Administrator should perform sensitivity analyses to verify how changes to the risk-based grant assessment model could affect the resulting risk scores, and document the results. |
FEMA concurred with our recommendation, stating that the agency will expand the use of sensitivity analysis to review the entire risk methodology, and will also document these results for leadership review, as appropriate. In July 2019, FEMA provided us with their sensitivity analysis of the Fiscal Year 2019 risk allocation model. According to FEMA, their analysis focused on the sensitivity of computed risk to changes in variable values. Specifically, FEMA's analysis considered each jurisdiction individually, and varied each of the model's variable (e.g. down by 10% of its value, or by one level for discrete inputs). FEMA then compared the impact on the variation on the risk score for each jurisdiction and ranked the variables by the magnitude of this impact. By performing this analysis, FEMA is developing a greater understanding of the how uncertainty affects its risk-based grant assessment model's results. In addition, this action should increase the transparency of the allocation model to the benefit internal decision makers in any future efforts to improve the allocation model's risk methodology. As a result of this action, this recommendation is closed as implemented.
|
| Federal Emergency Management Agency | The FEMA Administrator should take steps to coordinate an independent, external peer review of its risk-based grant assessment model. |
FEMA concurred with our report recommendation, stating that they would coordinate an independent external peer review and develop a detailed written response to leadership for further appropriate action. In July 2019, FEMA transmitted to GAO an external subject matter expert's review of the risk allocation model. This outside expert in the field risk assessment performed an external evaluation of agency's risk model and grant allocation approach for the State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grants. This evaluation included a review of FEMA documents that summarized their allocation model and the history of FEMA's risk assessment process, along with discussions with FEMA officials. By conducting an independent external peer review of the methodology of its risk-based grant assessment model, FEMA bolsters its ability to provide reasonable assurance that the agency's risk model and grant allocation approach are reasonable, of high-quality, and credible. As a result of this action, this recommendation closed as implemented.
|