Transportation Research:

Opportunities for Improving the Oversight of DOT's Research Programs and User Satisfaction with Transportation Statistics

GAO-06-917: Published: Aug 15, 2006. Publicly Released: Aug 15, 2006.

Additional Materials:

Contact:

Katherine A. Siggerud
(202) 512-6794
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Transportation's (DOT) research, development, and technology (RD&T) budget totaled $1.1 billion in fiscal year 2005. DOT's Research and Innovative Technology Administration (RITA)--which includes the Bureau of Transportation Statistics (BTS)--oversees DOT's RD&T activities. GAO examined (1) how RITA's responsibilities for overseeing DOT's RD&T activities differ from those of its predecessor, the Research and Special Programs Administration (RSPA); (2) RITA's practices for coordinating, facilitating, and reviewing RD&T activities; (3) the progress DOT has made in implementing GAO's 2003 recommendations on how to improve the coordination and evaluation of RD&T activities; and (4) how BTS identifies and monitors how well it serves its users. To address these issues, GAO reviewed relevant documentation and interviewed officials from RITA, BTS, and three operating administrations.

In 2005, RITA took over RSPA's responsibilities for overseeing DOT's RD&T activities when RSPA was dissolved. While RITA's mission and strategic objectives are similar to those RSPA had, RITA differs from RSPA in a number of ways. For example, RITA proposed a $2 million increase in its fiscal year 2007 budget request for the oversight of DOT's RD&T activities through its proposed Transportation Futures and Applied Technology Program, which, among other things, would provide access to technical experts to RITA on a contract basis. Additionally, RITA's responsibility for evaluation is less clearly defined than RSPA's. RITA, unlike RSPA, is not required to measure the results or evaluate the effectiveness of RD&T activities. However, RITA is not explicitly prevented from evaluating such activities. RITA coordinates, facilitates, and reviews DOT's RD&T activities through various practices. For example, RITA has two coordinating bodies--the RD&T Planning Council and the RD&T Planning Team--and conducts budget reviews, among other practices. RITA has not, however, established performance goals, a clear implementing strategy, or an evaluation plan that delineates how its coordination, facilitation, and review practices will further DOT's mission or ensure the effectiveness of its RD&T investment. Without such a strategic approach, it is difficult for RITA to ensure that DOT is making the most of its approximately $1 billion annual RD&T investment. RITA has partially implemented four of our recommendations and has not implemented the other. For example, while RITA, through its two coordinating bodies, has taken some action to review RD&T activities for duplication and opportunities for joint efforts, RITA has not established the scope of RD&T activities to be reviewed, the methodology of the review, or how the results will be used to make decisions about future RD&T activities. BTS does not have a systematic process for identifying its primary users, soliciting ongoing feedback from those users, and determining whether or how that feedback should be incorporated. For example, rather than identify specific users of BTS data products and services, BTS considers its users to be those broad categories of intended users described in federal legislation. Further, rather than routinely soliciting user feedback on all data products and services, such as through a customer satisfaction survey, BTS only solicits limited feedback from some users of specific products through conferences, workshops, or other meetings. Finally, BTS relies on its program managers to evaluate and determine how best to address feedback from its users; however, the managers are not required to--and often do not--report the results of whether or how they considered user feedback. Without a systematic process for identifying its users, soliciting ongoing feedback, and determining whether or how that feedback should be incorporated, BTS is limited in its ability to consider feedback and use it to make improvements to data products.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a description of RITA's process for systematically evaluating the results of its own multimodal research programs and how this process will be applied to future multimodal research programs that RITA conducts into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA has established a systematic process for evaluating all of DOT's research, including RITA's multi-modal research programs. This process consists of reviews to identify, share, and apply best practices, identify opportunities for leveraging resources, and advance cross-modal research. RITA oversees DOT's Intermodal Advanced Research Working Groups, which have identified eight cross-modal research areas for collaboration with operating administrations including: congestion reduction; energy efficiency/alternative fuels; enhanced safety data; human factors; nanotechnology; positioning, navigation and timing; safe transport in an aging society, and system resilience/global logistics. RITA has developed an evaluation strategy to ensure that these cross-modal activities, as well as other DOT research activities, are evaluated according to best practices, including systematic application of OMB investment criteria, external stakeholder involvement, merit review of competitive proposals, independent expert review, development of research performance measures, and external research coordination. RITA officials told us that they are continuing to pursue the development and implementation of a more consistent process to be applied across all program areas, which they expect to complete by the end of fiscal year 2008.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a summary of all of DOT's RD&T program evaluations conducted by the department for the past 3 years, including ongoing and completed evaluations, and a schedule of future evaluations into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA and its predecessor, RSPA, have together published summaries of DOT's RD&T evaluations for the past three years. First, RSPA published a summary of all research program evaluations conducted in fiscal year 2004 in its fiscal year 2005 annual Research, Development, and Technology plan. Secondly, the results of RITA's fiscal years 2005 and 2006 reviews, and a schedule of RITA's planned and ongoing fiscal year 2007 reviews, was included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. This report to Congress also includes summaries of research program evaluations conducted by modal research advisory committees, the Transportation Research Board, and key modal stakeholders in fiscal years 2006 and 2007. At the time of this followup, a schedule for fiscal year 2008 reviews had not yet been developed.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a DOT-wide database of all of DOT's RD&T projects that will support RITA's coordination, facilitation, and review efforts and will assist in the implementation of the strategies discussed above into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan. Information on the status of these efforts should be included in the upcoming RD&T strategic plan to be issued in the fall of 2006.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA has designed and is now in the process of developing two database systems to inventory and track all of DOT's research activities and provide tools for querying and searching individual projects to identify potential duplication and to identify areas where operating administrations could collaborate. The first database, called the RITA Research Notification System (R2NS) captures research investments at the transactional level, allowing users to search by activity, contracts and grants, and contractor names, enabling identification of funded programs for coordination, collaboration and review. The second database, which is part of the Research Planning and Investment Coordination process, captures research at the budget request level, allowing for Department-wide transparency and coordination of proposed programs and projects. According to a RITA official, eventual combination of the two databases will be a part of the annual Research Planning and Investment Coordination process that will offer a mechanism for measuring and tracking investments from request, through funding and execution. The databases will enable RITA to adjust and leverage research activities across the Department supporting its RITA's coordination, facilitation, and review efforts.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan, a strategy to ensure that the results of all of DOT's RD&T activities are evaluated according to established best practices. This strategy should include (1) which RD&T activities of the operating administrations RITA will ensure were evaluated according to best practices, (2) the methodology for how RITA will ensure evaluation of RD&T activities took place according to established best practices, (3) a timeline for when the RD&T evaluations should occur, and (4) how the results of the RD&T evaluations will inform future research.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA has developed a strategy to ensure that the results of all DOT's research activities are evaluated according to established best practices on an annual basis. RITA's strategy includes three primary mechanisms, (1) ensuring systematic application of the Office of Management and Budget's (OMB) Research and Development Investment Criteria (relevance, quality, and performance) and the Program Assessment Rating Tool by the operating administrations; (2) annual internal program reviews with self-reporting by the operating administrations; and (3) documenting the operating administration's external stakeholder coordination and review. The results of the fiscal year 2005 and 2006 reviews and a schedule for fiscal year 2007 reviews are included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. According to RITA, the fiscal year 2007 reviews have been focused on how well the operating administrations are implementing best practices, including external stakeholder involvement, merit review of competitive proposals, independent expert review, research performance measures, and external research coordination. RITA will report the results of the reviews to the Planning Council and will publish the results in the next Research, Development and Technology Annual Funding Report to Congress. This strategy, documented in the 2006-2010 Research, Development, and Technology Strategic Plan, upholds the technical quality of DOT's research activities and helps to ensure that research activities are addressing critical needs.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate a strategy for identifying and reviewing all of DOT's RD&T projects to determine areas of unnecessary duplication, overlap, and opportunities for joint efforts into RITA's fiscal year 2008 budget process, the annual budget process thereafter, and the upcoming RD&T strategic plan. The strategy should address (1) the scope of the RD&T projects or programs that will be reviewed for duplication or joint efforts, (2) the methodology for how all RD&T projects will be reviewed or how duplication or joint efforts will be identified, (3) a timeline and the frequency for reviews to occur, and (4) how the results of the reviews--the identification of duplication or an opportunity for joint effort--will be reported and used to make decisions about future RD&T activities.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA has developed a strategy for identifying and reviewing all of DOT's RD&T projects to determine areas of unnecessary duplication, overlap, and opportunities for joint efforts. RITA's strategy consists of on-going internal reviews of all of DOT's research programs conducted by the Program Review Working Group. The objectives of these internal reviews are to continuously improve transportation research management and performance; identify, share, and learn best practices; identify opportunities for leveraging resources and for crossmodal research initiatives; and prevent unnecessary research duplication. RITA's Program Review Working Group conducts these reviews by (1) convening meetings where officials from each of the operating administrations share information about areas of on-going or planned research, looking for opportunities for joint effort, and (2) conducting annual reviews of each operating administrations research plans, looking for research duplication, among other things related to the budget process. Results from the coordination efforts are discussed during Planning Team and Planning Council meetings and have been reported in the 2006-2010 Research, Development, and Technology Strategic Plan. According to the Strategic Plan, in fiscal year 2005 and again in fiscal year 2006, the Program Review Working Group found no research duplication and identified a number of areas for crossmodal collaboration, including geospatial technologies, freight capacity, security, alternative energy technologies, and advanced materials and sensors. A RITA official told us that eight working groups have been formed with representation from the relevant operating administrations to foster collaboration in these different areas. In addition to the Strategic Plan, the results of the coordination efforts are also included in the "Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress", expected to be delivered to Congress in mid-September 2007. The results of RITA's internal reviews are used to ensure that research programs are wise public investments that address critical needs.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate into RITA's fiscal year 2008 budget process, and the annual budget process thereafter, common performance measures related to DOT's RD&T activities, which should be developed in consultation with the operating administrations.

    Agency Affected: Department of Transportation

    Status: Closed - Not Implemented

    Comments: According to RITA officials, the Research Planning and Investment Coordination (RPIC) process that the agency had been working on for several years in response to our recommendations was not endorsed by DOT's Planning Council (composed of all modal administrators). As the Planning Council's direction, RITA is now pursuing several initiatives--that may eventually lead to the development of common RD&T performance measures across modes--which involve developing (1) a programmatic review process requiring each mode with research, development, and technology (RD&T) activities to provide an annual briefing on steps taken to successfully apply best practices and identify and meet performance metrics, (2) an RD&T Strategic Plan outlining a departmental strategy for addressing DOT-wide goals and measurable outcomes related to RD&T activities, (3) a Knowledge Management System listing all RD&T projects in a searchable, web-based database, and (4) research "clusters" which are forums for bringing DOT, university, and AASHTO researchers together periodically to discuss topics of interest. According to RITA officials, these initiatives may eventually position RITA to consider common performance measures, but the agency has not set a timeframe for developing such measures.

    Recommendation: To enhance RITA's ability to manage and ensure the effectiveness of RD&T activities in furthering the department's mission, the Secretary of Transportation should direct the RITA Administrator to develop and incorporate performance goals and an overall implementing strategy that delineate how the activities and results of its coordination, facilitation, and review practices will further DOT's mission and ensure the effectiveness of the department's RD&T investment into RITA's fiscal year 2008 budget process, and the annual budget process thereafter. The strategy should include an evaluation plan for monitoring and evaluating its performance against set goals to assist RITA in better allocating its resources to improve performance.

    Agency Affected: Department of Transportation

    Status: Closed - Not Implemented

    Comments: According to RITA officials, the Research Planning and Investment Coordination (RPIC) process that the agency had been working on for several years in response to our recommendations was not endorsed by DOT's Planning Council (composed of all modal administrators). As the Planning Council's direction, RITA is now pursuing four initiatives, which involve developing (1) a programmatic review process requiring each mode with research, development, and technology (RD&T) activities to provide an annual briefing on steps taken to successfully apply best practices and identify and meet performance metrics, (2) an RD&T Strategic Plan outlining a departmental strategy for addressing DOT-wide goals related to RD&T activities, (3) a Knowledge Management System listing all RD&T projects in a searchable, web-based database, and (4) research "clusters" which are forums for bringing DOT, university, and AASHTO researchers together periodically to discuss topics of interest. RITA officials estimate that the draft strategic plan will be published for public comment in the winter of 2010, but they have not set a timeframe for developing performance measures for RD&T activities or an evaluation plan for monitoring performance against goals. They noted that such measures would only be developed after the strategic plan is published.

    Recommendation: To help ensure that BTS's data products meet the needs of its users, the Secretary of Transportation should direct the RITA Administrator and BTS Director to develop and implement a systematic process for BTS to identify its primary users, solicit and incorporate feedback from those users, and measure the satisfaction of its users. This process should contain the following elements: (1) that primary users of BTS's data products and services are identified and documented in a comprehensive manner; (2) that feedback on user satisfaction is solicited on a periodic basis from those users; (3) that user feedback is documented and evaluated at BTS's agencywide level and against established criteria, to ensure consistency in decisions about what improvements should be made to data products; and (4) that performance indicators that measure data users' satisfaction are developed and applied.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: RITA's Bureau of Transportation Statistics (BTS) has taken several steps to identify its primary users, solicit and incorporate feedback from those users, and measure the satisfaction of its users. First, BTS created a detailed Customer Feedback System to capture customer feedback on its statistical products and activities; provide the feedback data to BTS program managers and executives and senior RITA leadership for assessment and potential action; and identify relative priorities for action on the feedback. The system captures and prioritizes feedback from 9 major stakeholder groups identified by BTS, including Congress, White House, DOT modal administrations, other federal agencies, state DOTs, academia, research organizations, students, the the general public. Secondly, BTS held listening sessions with 24 of its primary stakeholders from December 2009 through May 2010, identifying key feedback themes and issues for BTS leadership to consider, e.g., stakeholder communication and customer service, data collection and clearinghouse, information formats and accessibility, data timeliness, and other issues. Finally, BTS implemented a web-based survey tool--the American Customer Satisfaction Index (ACSI)--to proactively gather user feedback on satisfaction with its web products and services. BTS program managers receive quarterly reports of survey responses from ACSI, beginning in March, 2010. During the first six months of its implementation, the ACSI provided BTS with an overall rating of its user satisfaction. The second quarterly report is expected to address many of themes identified in the stakeholder listening sessions held in 2009 and 2010.

    Jul 31, 2014

    Jul 23, 2014

    Jun 25, 2014

    Jun 24, 2014

    Jun 18, 2014

    Jun 11, 2014

    May 30, 2014

    Looking for more? Browse all our products here