National Transportation Safety Board: More Detail Could Increase the Understanding of Selections for the Most Wanted List of Transportation Safety Improvements [Reissued with revisions on Mar. 19, 2020.]
Fast Facts
The National Transportation Safety Board issued its 2019 “Top 10 Most Wanted Safety Improvements” to help prevent accidents and save lives. But how did NTSB choose these particular issues?
NTSB had a systematic method to determine what makes the list, but in some cases, it’s unclear how they used their criteria to make the selections. If advocacy groups, industry associations and others don’t know why NTSB is advocating for these particular safety issues and recommendations, that could affect their attention to and support for tackling the issues on the list.
We recommended that NTSB better document and communicate its decisions for the list.
Two NTSB workers viewing smoking wreckage
Reissued with Revisions Mar 23, 2020This report was revised on March 19, 2020 to include the National Transportation Safety Board’s letter commenting on the draft report as appendix III, on page 32.
Highlights
What GAO Found
The National Transportation Safety Board (NTSB) developed a multiphase approach to select issues for its 2019–2020 Most Wanted List of Transportation Safety Improvements (see figure). NTSB designed the selection process to encourage collaboration and to consider the expertise of NTSB's four modal offices (Aviation Safety, Highway Safety, Marine Safety, and Railroad, Pipeline, and Hazardous Materials) and its Office of Research and Engineering. Like past lists, each issue must be supported by one or more open safety recommendations. The process also allowed NTSB's board members and others discretion in suggesting changes to the issues proposed for inclusion on the Most Wanted List. NTSB uses the list to raise awareness of its recommendations and to advocate their adoption since the NTSB cannot require implementation of its recommendations.
National Transportation Safety Board's Selection Process for Its 2019–2020 Most Wanted List of Transportation Safety Improvements
NTSB published a methodology report, in response to the requirements in the National Transportation Safety Board Reauthorization Act of 2018, that detailed the methodology to evaluate and select issues for the list. GAO found that the design of NTSB's methodology met the essential components for designing a sytematic deicision-making framework. When implementing that methodology, however, NTSB did not fully document how staff, when evaluating the issues, considered its own established criteria nor fully communicated the rationale for why its selected issues were “ripe for action” now—a key component of the list . While GAO was able to determine the rationale for NTSB's evaluation and selection decisions, NTSB's guidance does not require NTSB to fully document or communicate its decision, and NTSB does not do so. Greater transparency in how issues are evaluated and selected could enhance users' understanding of the list and help ensure the list continues to rally the support and resources needed to tackle difficult and long-standing transportation safety challenges.
Why GAO Did This Study
NTSB investigates accidents across all modes of transportation and issues recommendations intended to prevent similar accidents. In 2019, NTSB published its latest Most Wanted List of Transportation Safety Improvements. It identified 10 transportation safety issues with 268 underlying safety recommendations.
The National Transportation Safety Board Reauthorization Act of 2018 included a provision for GAO to examine NTSB's methodology for evaluating and selecting recommendations for inclusion in the Most Wanted List. This report discusses (1) NTSB's methodology for developing its Most Wanted List and (2) how NTSB addressed statutory requirements and how its methodology aligned with components for systematic decision-making, among other objectives. GAO reviewed NTSB documentation for its process of selecting issues for the Most Wanted List. GAO also interviewed NTSB officials to understand the rationale behind the selection methodology and how the process was applied. GAO compared the methodology to essential components for systematic decision-making and the statutory requirement that NTSB publish a publicly available methodology report that describes NTSB's consideration of key elements.
Reissued with revisions on Mar. 19, 2020.
This report was revised on March 19, 2020 to include the National Transportation Safety Board’s letter commenting on the draft report as appendix III, on page 32.Recommendations
GAO recommends NTSB improve how it documents and communicates decisions for its Most Wanted List. NTSB agreed with both recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| National Transportation Safety Board | The Chairman of the Board should require the Safety Recommendation and Communication team to fully document its evaluations when assessing items to propose for Most Wanted List consideration (Recommendation 1). |
The National Transportation Safety Board (NTSB) created its Most Wanted List of Transportation Safety Improvements (the List) to help raise awareness of its open recommendations and encourage action to improve transportation safety further. In 2020, GAO reported that NTSB published a report, in response to the requirements in the National Transportation Safety Board Reauthorization Act of 2018, that detailed the methodology to evaluate and select issues and recommendations for the 2019-2020 Most Wanted List. In previous work, GAO developed a framework for making risk-informed, systematic decisions. GAO found that when designing the methodology, NTSB met all seven essential components for the design phase of the framework. However, NTSB did not fully document how well the different selections performed with respect to the established objectives and did not fully communicate how its selections achieved an acceptable balance of performance across the objectives of the Most Wanted List program. For example, the documentation that NTSB's modal office staff provided to the Board on draft selections for the List provided little insight into how the selected issues aligned with NTSB's stated criteria or why the proposed issues would be better suited for the List than the others. NTSB officials told GAO their internal guidance does not require this level of documentation but stated they understood how it could be helpful for someone not part of the process to understand the evolution of the issues considered and why certain issues were forwarded through the process. According to internal control standards, it is important to communicate quality information internally, including for the purpose of informing key decisions. Although NSTB's methodology, as designed, intended its modal offices and Board to have flexibility to propose and select issues for the List, such discretion and reliance on these internal groups also increases the importance of documenting decisions. GAO has also previously reported on the importance of such documentation for decision-making. To be useful for decision makers, these results should be documented in a way that facilitates consistent comparison of the options. With internal guidance to document key decisions, NTSB would enhance the transparency of its process. Therefore, GAO recommended that NSTB fully document its evaluations when assessing issues to propose for consideration in the Most Wanted List. In April 2022, NTSB updated the internal Board order that establishes guidelines for selecting safety improvements to be included in the List. NTSB now requires that staff responsible for managing the development of the Most Wanted List provide written justification for each issue proposed for inclusion as well as the scoring worksheets used to assess the issues. This information is to be provided to the Board for its consideration in approving the final Most Wanted List. By better documenting the evaluations used to determine safety improvements for the Most Wanted List and providing this information to the Board, NTSB is better positioned to provide transparency to enhance stakeholders' and other interested parties' confidence in the List and help NTSB achieve continued improvement in transportation safety.
|
| National Transportation Safety Board | The Chairman of the Board should take steps to publicly and fully communicate the selection rationale, such as including why NTSB believes an issue is "ripe for action" to its documentation on its website (Recommendation 2). |
The National Transportation Safety Board (NTSB) created its Most Wanted List of Transportation Safety Improvements (the List) to help raise awareness of its open recommendations and encourage action to further improve transportation safety. The List is intended to reflect the current concerns of NTSB and include issues it believed were "ripe for action" and merit increased attention. In 2020, GAO reported that NTSB did not fully communicate its rationale for selecting the issues it included on the List. Specifically, once NSTB's Board selected its preferred issues, it did not fully communicate its rationale. In accordance with the Board order, NTSB publicly communicated its final selections, including across multiple platforms, such as an online press conference and on its website. As part of this communication, NTSB also clearly detailed why each issue was a risk to transportation safety and what NTSB recommends to address that risk. Despite clearly detailing an issue's risk to the travelling public, NTSB's communication provided limited insight into why selected issues were chosen for heightened attention. NTSB did not detail why these issues were "ripe for action" now-a key component of the List-including, why the Board believed the selected issues, with additional attention, could be successfully acted upon during the next 2 years. According to internal control standards, management should communicate quality information externally through external lines so that those parties can achieve their objectives. Not fully communicating the Board's selection decisions can call into question the merit of the selected issues, potentially affecting stakeholders' level of focus and action to address them. Accordingly, GAO recommended that NSTB take steps to publicly and fully communicate the selection rationale. In 2022, GAO confirmed that NTSB developed a link on its website entitled "Making the MWL" (i.e. Making the Most Wanted List) that addressed GAO's recommendation. On this page, NTSB provides an explanation of the methodology it used to select the issues for the List along with a recording of the April 2021 meeting where the Board members publicly presented the current List. As part of this presentation, NTSB Board Members and staff explained the rationale regarding how each issue met the criteria used to assess issues for inclusion on the List. This included why NTSB felt a high level of urgency was needed (i.e. why an issue is ripe for action). By clearly communicating the rationale for selecting certain issues over others, NTSB provided more transparency of the Most Wanted List program that could enhance users' understanding of the List, which is vital to rally the support and resources needed to tackle these issues.
|