Highway Research: Systematic Selection and Evaluation Processes Needed for Research Program
Highlights
The Federal Highway Administration (FHWA) has received hundreds of millions of dollars for its surface transportation research and technology program during the past decade. For example, in 1998 the Transportation Equity Act for the 21st Century, included over $447 million for fiscal year 2002 for FHWA's transportation research and technology efforts for six-year period of 1998 through 2003. FHWA's research and technology program is complex because each of the program offices within the agency are responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals that support the agency's strategic goals. One business unit at FHWA's research laboratory provides support for administering the overall program and conducts some of the research. The agency's leadership team provides periodic oversight of the overall program. FHWA's processes for managing the research and technology program, and in particular for developing research agendas and evaluating research outcomes against intended results, do not always align with the best practices for similar federal research and technology programs. FHWA acknowledges that its approach for developing research agenda and involving external stakeholders in determining the direction of the program's research lacks a consistent, transparent, and systematic process. Instead, most external stakeholder involvement is ad hoc through technical committees and professional societies. The agency primarily uses a "success story" approach to evaluate its research outcomes. While this approach shows some benefits, it cannot be used as the primary method to evaluate the outcomes of the research because these stories represent only a fraction of the program's completed research projects. As a result of its relatively varied processes, it is unclear whether the organization is selecting research projects that have the highest potential value, or what is the extent to which these projects have achieved their objectives.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Transportation | To help ensure that FHWA's research agenda and approach to evaluation are identifying research with the highest value to the surface transportation community and monitoring the outcomes of that research, the Secretary of Transportation should direct FHWA's Administrator to develop a systematic approach for obtaining input from external stakeholders in determining the research and technology program's agendas. |
On May 14, 2003, FHWA released the "FHWA Corporate Master Plan for Research and Deployment of Technology & Innovation." This Plan, which acknowledges that it was propelled in part by the recommendations within GAO's report, lays out guiding principles and commitments for developing a systematic approach for obtaining input from external stakeholders in determining the agency's research and technology program agendas. Under the Plan, stakeholders will be invited to assist FHWA with setting research and technology program agendas and priorities. The Plan notes that FHWA will engage representatives from state and local agencies, resource agencies, metropolitan planning organizations, academia, and industry as appropriate. Finally, the Plan notes that "FHWA is committed to engaging stakeholders throughout the R&T process."
|
| Department of Transportation | To help ensure that FHWA's research agenda and approach to evaluation are identifying research with the highest value to the surface transportation community and monitoring the outcomes of that research, the Secretary of Transportation should direct FHWA's Administrator to develop a systematic process for evaluating significant ongoing and completed research that incorporates peer review or other best practices in use at federal agencies that conduct research. |
On May 14, 2003, FHWA released the "FHWA Corporate Master Plan for Research and Deployment of Technology & Innovation." This Plan, which acknowledges that it was propelled in part by the recommendations within GAO's report, lays out guiding principles and commitments for including stakeholders in merit review during various phases of the research and technology process. The Plan notes that FHWA considers merit review by stakeholders to be "vital to the continued success of the Research and Technology Program, and is committed to using merit review" during the research and technology process. Under the Plan, a merit review panel will conduct evaluations and reviews by collaborating with FHWA personnel, technical experts, peers, special interest groups, senior management, and contracting officers. Project-level evaluations will be conducted at a time determined by established milestones. Program evaluations will be conducted by merit review on a periodic basis, such as yearly, and agency evaluations will be performed by merit review and will be conducted on a periodic basis, such as every 3 years.
|
| Department of Transportation | To help ensure that FHWA's research agenda and approach to evaluation are identifying research with the highest value to the surface transportation community and monitoring the outcomes of that research, the Secretary of Transportation should direct FHWA's Administrator to develop specific plans for implementing these recommendations, including time frames and estimates of their cost. |
On May 14, 2003, FHWA released the "FHWA Corporate Master Plan for Research and Deployment of Technology & Innovation." This Plan, which acknowledges that it was propelled in part by the recommendations within GAO's report, lays out milestones for implementing GAO's recommendations, including time frames and estimates of their cost. The Plan notes that activities such as agenda setting will apply to the next Research and Technology Program programming and budget cycle. Principles for using merit reviews will be implemented as part of key processes related to agenda setting. Regarding cost estimates, the Plan notes that a very rough estimate of overall agency resource requirements was completed as part of developing the Plan. However, staff in each functional area will include self-assessments against FHWA commitments in the Plan--including resources needed--by December 31, 2003.
|