DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes
GAO-03-640T, Apr 10, 2003
Improvement and innovation based on highway research have long been important to the highway system. The Federal Highway Administration (FHWA) is the primary federal agency involved in highway research. Throughout the past decade, FHWA received hundreds of millions of dollars for its surface transportation research program, including nearly half of the Department of Transportation's approximate $1 billion budget for research in fiscal year 2002. Given the expectations of highway research and the level of resources dedicated to it, it is important to know that FHWA is conducting high quality research that is relevant and useful. In May 2002, GAO issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts. GAO was asked to testify on (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) how FHWA's processes for developing research agendas align with these best practices; and (3) how FHWA's processes for evaluating research outcomes align with these best practices.
Leading organizations, federal agencies, and experts that conduct scientific and engineering research use best practices designed to ensure that research objectives are related to the areas of greatest interest to research users and that research is evaluated according to these objectives. Of the specific best practices recommended by experts--such as the Committee on Science, Engineering, and Public Policy and the National Science Foundation--GAO identified the following practices as particularly relevant for FHWA: (1) developing research agendas in consultation with external stakeholders to identify high-value research and (2) using a systematic approach to evaluate research through such techniques as peer review. FHWA's processes for developing its research agendas do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is to be used by others that manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a systematic process to ensure that external stakeholders are involved. In response to GAO's recommendation, FHWA has drafted plans that take the necessary steps toward developing a systematic process for involving external stakeholders. While the plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented. FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research outcomes. Instead, the agency primarily uses a "success story" approach to communicate about those research projects that have positive impacts. As a result, it is unclear the extent to which all research projects have achieved their objectives. FHWA acknowledges that it must do more to measure the performance of its research program, however, it is still in the process of developing a framework for this purpose. While FHWA's initial plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented.