Skip to main content

Quadrennial Homeland Security Review: Improved Risk Analysis and Stakeholder Consultations Could Enhance Future Reviews

GAO-16-371 Published: Apr 15, 2016. Publicly Released: Apr 15, 2016.
Jump To:
Skip to Highlights

Highlights

What GAO Found

The Department of Homeland Security (DHS) assessed risk for the second Quadrennial Homeland Security Review (QHSR) and considered threats, vulnerabilities, and consequences; however, DHS did not document how its various analyses were synthesized to generate results, thus limiting the reproducibility and defensibility of the results. Without sufficient documentation, the QHSR risk results cannot easily be validated or the assumptions tested, hindering DHS's ability to improve future assessments. In addition, the QHSR describes homeland security hazards, but does not rank those hazards or provide prioritized strategies to address them. Comparing and prioritizing risks helps identify where risk mitigation is most needed and helps justify cost-effective risk management options. Without determining prioritized risk outcomes, DHS is missing an opportunity to more efficiently mitigate risk or identify the resources required for addressing different levels and types of risks.

The President's fiscal years 2015 and 2016 budget requests for DHS were generally presented in alignment with the QHSR. However, DHS has faced challenges accounting for its spending by mission, which it is taking actions to address, such as developing a new common appropriation structure to better link the department's funding request to the execution of its missions. DHS also developed performance measures for all of the QHSR mission areas.

DHS expanded its stakeholder outreach efforts, but 43 of 61 stakeholders who provided narrative responses to one question in GAO's survey stated that collaboration with stakeholders could be improved. For example, one respondent reported that stakeholders were asked to react to information provided by DHS rather than assist in formulating the QHSR approach and execution. DHS officials reported being limited by staff, time, and other constraints, and thus directed stakeholders to provide feedback via various web-based forums. Although the online forums allowed DHS to reach 2,000 representatives during its 2014 QHSR development process, DHS's QHSR After Action Report noted that the tools were used to validate study findings instead of informing them. Without fostering interactive communication, DHS may miss opportunities to incorporate stakeholder perspectives from the entire homeland security enterprise and thereby may not have fully informed the QHSR effort.

2014 Quadrennial Homeland Security Review Development Timeline

2014 Quadrennial Homeland Security Review Development Timeline

Why GAO Did This Study

Homeland security threats continue to evolve and include challenges ranging from terrorist attacks to natural disasters, emphasizing the need for DHS to periodically examine and strengthen the nation's homeland security strategy. Further, the Implementing Recommendations of the 9/11 Commission Act of 2007 requires DHS to conduct such a review every 4 years. GAO reported in 2011 on DHS's first QHSR and recommended DHS provide sufficient time for stakeholder consultations and examine how risk information could be used to prioritize mission efforts. DHS issued its second QHSR in 2014.

GAO was asked to assess the QHSR. This report addresses the extent to which DHS (1) examined and used risk information to inform the QHSR and its implementation, (2) aligned its budget and performance measures to QHSR mission goals, and (3) collaborated with stakeholders to develop the QHSR. GAO analyzed relevant statutes and QHSR documentation; conducted a nongeneralizable survey of DHS-identified federal and nonfederal QHSR stakeholders, receiving responses from 93 of the 182 QHSR stakeholders it contacted; and interviewed officials from DHS, federal, and nonfederal entities.

Recommendations

GAO is making four recommendations, including that for future QHSRs DHS improve its risk assessment documentation, prioritize risks, and ensure stakeholder meetings are interactive. DHS concurred with our recommendations.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Homeland Security To ensure the quality of the risk assessments used to inform its future QHSR processes, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to ensure future QHSR risk assessment methodologies reflect key elements of successful risk assessment methodologies, such as being: (1) Documented, which includes documenting how risk information was integrated to arrive at the assessment results, (2) Reproducible, which includes producing comparable, repeatable results, and (3) Defensible, which includes communicating any implications of uncertainty to users of the risk results.
Closed – Not Implemented
In April 2016, the Office of Policy's Office of Strategy, Plans, Analysis and Risks completed initial meetings with government and non-government subject matter experts to refine risk analyses for the 2018 QHSR. Representatives from the department's component and headquarters staff were to take part in the Department's Risk Modeling and Analysis Steering Committee by reviewing, documenting and approving proposed new methodologies planned to help identify and prioritize threats and hazards. This effort was intended to lead to a documented, reproducible, and defensible assessment, according to the DHS officials. However, as of September 2020, the QHSR was not published by DHS, which limited our ability to verify that this recommendation was fully addressed. Given the passage of time since we made the recommendation in 2016, and the remaining uncertainty about when and how DHS will address it, we are closing this recommendation as not implemented.
Department of Homeland Security To enable the use of risk information in supporting resource allocation decisions, guiding investments, and highlighting the measures that offer the greatest return on investment, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to refine its risk assessment methodology so that in future QHSRs it can compare and prioritize homeland security risks and risk mitigation strategies.
Closed – Not Implemented
As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk, with support from the RAND Corporation, proposed a methodology to assess threats, hazards, and vulnerabilities impacting U.S. homeland security. In addition, the department's Risk Modeling and Analysis Executive Steering Committee was to review and approve the proposed methodology. The methodology was intended to enable the Department of Homeland Security to compare and prioritize homeland security risks and risk mitigation strategies, according to DHS officials. However, as of September 2020, the QHSR was not published by DHS, which limited our ability to verify that this recommendation was fully addressed. Given the passage of time since we made the recommendation in 2016, and the remaining uncertainty about when and how DHS will address it, we are closing this recommendation as not implemented.
Department of Homeland Security To ensure proper management of the QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to identify and implement stakeholder meeting processes to ensure that communication is interactive when project planning for the next QHSR.
Closed – Not Implemented
As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk finalized a draft stakeholder outreach plan to include use of the Office of Management and Budget's Max electronic collaboration website to engage with federal, state, and local stakeholders. The OMB-MAX website was available to government and non-government offices and allowed the posting of documents, articles, and links, as well as facilitating collaborative editing of documents and participant interaction threads, according to DHS officials. In addition, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk was exploring the use of different tools to facilitate more interactive stakeholder engagement. For example, DHS's Office of Partnerships and Engagement was to facilitate additional engagement with external subject matter experts, arrange interagency coordination, and organize review and approval with parties of the homeland security enterprise in order to coordinate and approve the development of the 2018 QHSR. However, as of September 2020, the QHSR was not published by DHS, which limited our ability to verify that this recommendation was fully addressed. Given the passage of time since we made the recommendation in 2016, and the remaining uncertainty about when and how DHS will address it, we are closing this recommendation as not implemented.
Department of Homeland Security To ensure proper management of the internal QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to clarify component detailee roles and responsibilities when project planning for the next QHSR.
Closed – Not Implemented
As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk (SPAR) drafted a memorandum for the Deputy Secretary to solicit Component subject matter experts to serve as detailees for the QHSR. The memorandum specified component detailee roles and responsibilities, to include serving in an advisory, consultation, and coordination role, according to DHS officials. SPAR was to lead an integrated group of analysts augmented by the detailees. The detailees were to serve as members of study teams analyzing key threats, trends, and strategy and policy alternatives associated with issues and challenges relating to DHS's mission and objectives. A second memorandum requesting additional detailee support was to be issued in November 2016, prior to the formal review phase of the new QHSR which was to begin in January 2017. However, as of September 2020, the QHSR was not published by DHS, which limited our ability to verify that this recommendation was fully addressed. Given the passage of time since we made the recommendation in 2016, and the remaining uncertainty about when and how DHS will address it, we are closing this recommendation as not implemented.

Full Report

Office of Public Affairs

Topics

BudgetsHomeland securityInvestmentsNatural disastersPerformance measuresPrioritizingRisk assessmentsSecurity threatsStrategic planning