This is the accessible text file for GAO report number GAO-03-746 
entitled 'Pipeline Safety: Systematic Process Needed to Evaluate 
Outcomes of Research and Development Program' which was released on 
June 30, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Subcommittees:

June 2003:

Pipeline Safety:

Systematic Process Needed to Evaluate Outcomes of Research and 
Development Program:

GAO-03-746:

GAO Highlights:

Highlights of GAO-03-746, a report to congressional subcommittees 

Why GAO Did This Study:

From 1998 through 2002, a total of 1,770 pipeline accidents occurred, 
resulting in 100 fatalities and $621 million in property damage. The 
Office of Pipeline Safety (OPS) within the Department of 
Transportation operates a research and development (R&D) program aimed 
at advancing the most promising technologies for ensuring the safe 
operation of pipelines. In fiscal year 2003, OPS received $8.7 million 
for its R&D program, a sevenfold increase since fiscal year 1998. In 
response to a directive from the House Committee on Appropriations, 
GAO (1) assessed OPS’s distribution of funding among various areas of 
R&D and the alignment of this funding with its mission and goals, (2) 
surveyed experts to obtain their views on R&D priorities, and (3) 
determined how OPS evaluates R&D outcomes.

What GAO Found:

OPS distributes its R&D budget among four main areas. For example, in 
fiscal year 2003, the office plans to allocate its $8.7 million budget 
as follows:

* 46 percent ($4.0 million) to developing new technologies to prevent 
damage to pipelines and prevent leaks;

* 21 percent ($1.9 million) to improving technologies for operating, 
controlling, and monitoring the condition of pipelines;

* 19 percent ($1.7 million) to improved pipeline materials, such as 
materials that are resistant to damage and defects; and

* 14 percent ($1.2 million) to efforts to improve data on the location 
and safety performance of pipelines.

On the basis of our work, we believe that OPS’s R&D funding is 
generally aligned with its mission and pipeline safety goals. OPS has 
taken a number of steps to ensure this alignment. For example, it 
obtained the views of a variety of experts and stakeholders in 
deciding on its R&D priorities and has described in various plans how 
its R&D efforts can lead to new and improved technologies that can 
help achieve its safety performance goals, such as reducing the 
impacts of pipeline accidents.

The pipeline safety R&D priorities of the experts we surveyed are 
generally consistent with OPS’s R&D priorities. For example, most 
assigned a high priority to the two areas of R&D that receive the 
highest amount of funding from OPS.

OPS’s efforts to evaluate the outcomes of its R&D have been limited. 
The agency has taken some preliminary steps toward developing an 
evaluation process for its R&D program, such as identifying possible 
measures of program results. Leading research organizations, the 
Office of Management and Budget, and GAO have identified a number of 
best practices for systematically evaluating the outcomes of federal 
R&D programs, such as setting clear R&D goals, measuring progress 
toward goals, and reporting periodically on evaluation results. These 
best practices can help OPS to determine the effectiveness of its R&D 
program in achieving desired outcomes, such as the development and use 
of new and improved technologies that can enhance pipeline safety. 

What GAO Recommends:

To better determine the effectiveness of its R&D program, GAO 
recommends that OPS develop a systematic process for evaluating 
program outcomes, using recognized best practices, and include the 
results of R&D evaluations in its annual reports to Congress. 

OPS officials told us that they generally agreed with the report’s 
findings and will follow our recommendations as they continue to 
develop an evaluation process for their R&D program. 

www.gao.gov/cgi-bin/getrpt?GAO-03-746.

To view the full product, including the scope and methodology, click 
on the link above. For more information, contact Peter Guerrero at 
(202) 512-2834 or guerrerop@gao.gov. 

[End of section]

Letter:

Results in Brief:

Background:

OPS's R&D Funding Is Aligned with Its Mission and Pipeline Safety 
Goals:

Experts Generally Support OPS's R&D Priorities:

OPS Lacks a Systematic Process for Evaluating R&D Outcomes:

Conclusions:

Recommendations for Executive Action:

Agency Comments:

Appendixes:

Appendix I: Experts’ Views on R&D Priorities and OPS’s R&D Funding, by
Type of R&D: 

Appendix II: Scope and Methodology:

Appendix III: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Acknowledgments:

Tables: 

Table 1: Major Categories of R&D Related to Pipeline Safety:

Table 2: Views of Experts from Three Subgroups on Pipeline Safety
R&D Priorities:

Figures: 

Figure 1: OPS’s R&D Budget, Fiscal Years 1998-2003:

Figure 2: OPS’s Planned Allocation of R&D Funding for Fiscal Year 2003:

Figure 3: OPS’s R&D Funding by Area of R&D, Fiscal Years 2001-03:

Figure 4: Expert Ratings of Categories of Pipeline Safety R&D:

Abbreviations: 

DOE: Department of Energy:

DOT: Department of Transportation:

MMS: Minerals Management Service:

NIH: National Institutes of Health:

OPS: Office of Pipeline Safety:

R&D: research and development:

RSPA: Research and Special Programs Administration:

Letter June 30, 2003:

The Honorable Ernest Istook, Jr. 
Chairman 
The Honorable John Olver 
Ranking Minority Member 
Subcommittee on Transportation, Treasury, and Independent Agencies 
Committee on Appropriations 
House of Representatives:

The Honorable Richard C. Shelby 
Chairman 
The Honorable Patty Murray 
Ranking Minority Member 
Subcommittee on Transportation, Treasury, and General Government 
Committee on Appropriations 
United States Senate:

Pipelines transport nearly all of the natural gas and nearly two-thirds 
of the crude oil and refined oil products in the United States. 
Although pipelines have a better safety record than other modes of 
freight transportation, their cargo is dangerous and leaks or ruptures 
can have serious consequences, including fatalities, harm to the 
environment, and property damage. For example, pipeline ruptures in 
Bellingham, Washington, in 1999 and in Carlsbad, New Mexico, in 2000 
together resulted in a total of 15 deaths and property and other 
damages totaling about $46 million. Investigators have determined that 
one of the probable causes of the Bellingham accident was excavation 
damage and that the cause of the Carlsbad accident was severe internal 
corrosion.

The Office of Pipeline Safety (OPS), within the Department of 
Transportation's (DOT) Research and Special Programs Administration, is 
responsible for pipeline safety regulation and research. The agency 
operates a research and development (R&D) program aimed at enhancing 
the safety and reducing the potential environmental impacts of 
transporting natural gas and hazardous liquids through pipelines. 
Specifically, the program seeks to advance the most promising 
technological solutions to problems that impede pipeline safety, such 
as damage to pipelines from excavation or corrosion. From fiscal years 
2001 through 2003, the budget of OPS's R&D program more than tripled, 
from $2.8 million to $8.7 million, partly as a result of congressional 
interest in achieving technological advances that can improve pipeline 
safety.

In House Report 107-722, which accompanied the Department of 
Transportation and Related Agencies Appropriations Bill for fiscal year 
2003, the House Appropriations Committee raised concerns regarding the 
effective management and utilization of these significant increases in 
funding for the department's pipeline safety R&D program. The committee 
directed GAO to review the effectiveness of the program. In subsequent 
discussions with staff of the Subcommittee on Transportation, Treasury, 
and Independent Agencies of the House Appropriations Committee, we 
agreed to determine (1) OPS's distribution of funding among various 
areas of pipeline safety R&D since fiscal year 2001 and the extent to 
which this funding is aligned with the agency's mission and pipeline 
safety goals, (2) the views of experts on pipeline safety R&D 
priorities, and (3) how OPS evaluates the outcomes of the pipeline 
safety R&D it funds.

To carry out this work, we reviewed legislation and agency documents 
pertaining to the R&D program and interviewed agency officials 
responsible for this program. We also interviewed key experts and 
stakeholders regarding their views on R&D priorities and gaps and on 
OPS's management of its R&D program, including the alignment of the 
agency's research agenda with its mission and goals. We identified best 
practices for evaluating the outcomes of R&D through a review of 
relevant literature. In addition, we sent a questionnaire to selected 
experts to obtain their views on pipeline safety R&D priorities. We 
selected experts who are informed about pipeline safety or the 
development of new pipeline safety technologies, including 
representatives of federal and state agencies, pipeline safety advocacy 
groups, industry associations, pipeline companies, technical and 
consulting organizations, and research institutes. We received 
responses from 49 of 55 experts we contacted, for a response rate of 89 
percent. Our results pertaining to experts' views on R&D priorities 
represent the views of only the experts who responded to our 
questionnaire and cannot be generalized to a broader population. (See 
app. II for additional details on our scope and methodology.):

Results in Brief:

OPS distributes its R&D budget to three major areas involving the 
research and development of pipeline safety technologies as well as to 
a fourth area--efforts to improve the agency's pipeline mapping and 
information systems. For example, in fiscal year 2003, OPS plans to 
allocate its $8.7 million R&D budget as follows:

* $4.0 million (46 percent) to developing new technologies for 
preventing damage to pipelines and detecting leaks,

* $1.9 million (21 percent) to improving technologies for operating, 
controlling, and monitoring the condition of pipelines,

* $1.7 million (19 percent) to improving pipeline materials, and:

* $1.2 million (14 percent) on efforts to improve pipeline mapping data 
and data on the safety performance of pipelines.[Footnote 1]

On the basis of our work, we believe that OPS's R&D funding is 
generally aligned with its mission and pipeline safety goals. The 
agency has obtained the views of external experts and stakeholders in 
determining what types of R&D are aligned with its mission of ensuring 
the safe, reliable, and environmentally sound operation of the nation's 
pipeline transportation system. OPS has also recently improved 
coordination with other federal agencies that fund pipeline R&D in 
order to avoid overlap between their R&D programs. Both expert review 
and coordination among agencies are recognized as best practices that 
help ensure that federal agencies' R&D activities are relevant to their 
missions and do not overlap. OPS has also described, in various plans, 
how its R&D efforts can lead to new and improved technologies that can 
help achieve its performance goals of reducing the impacts of pipeline 
incidents, including fatalities and injuries, and reducing spills of 
hazardous material. Key experts and stakeholders we contacted generally 
told us that, in their view, the agency has chosen appropriate R&D 
areas to fund.

The pipeline safety R&D priorities of the experts who completed our 
questionnaire are generally consistent with OPS's R&D priorities. The 
ranking of the major R&D areas based on the responses to our 
questionnaire is similar to the relative levels of funding OPS has 
assigned to these areas:

* 92 percent (45 of 49) of the experts assigned a high priority to the 
development of new technologies for preventing damage to pipelines and 
detecting leaks,

* 80 percent (39 of 49) assigned a high priority to improvements in 
technologies for operating, controlling, and monitoring the condition 
of pipelines, and:

* 31 percent (15 of 49) assigned a high priority to improvements in 
pipeline materials.

However, the experts' level of support for improvements in pipeline 
materials was much lower than that for the other two main R&D areas 
that OPS is funding and this level of support differed across different 
groups of experts. Although 70 percent (7 of 10) of experts from 
research organizations indicated that this area should receive high 
priority, only 21 percent (8 of 39) of the remaining experts--from 
government, public interest, industry, and technical and consulting 
organizations--indicated that it should receive high priority. OPS 
officials told us that they are currently updating their research 
agenda, using the input of experts and stakeholders, and that they will 
consider our questionnaire results in this process.

Despite the significant growth in its R&D budget since fiscal year 
2001, OPS has not developed a systematic process for evaluating the 
outcomes of the R&D it funds. For example, the agency tracks and 
disseminates information on the progress of individual R&D projects but 
has not developed a process for assessing and reporting on the results 
of its R&D program as a whole. Without such a process, OPS cannot 
determine and demonstrate the progress of its R&D program in achieving 
intended results, such as the development and use of new and improved 
technologies that can enhance pipeline safety. The agency has taken 
some preliminary steps toward developing an evaluation process for its 
R&D program, such as identifying possible measures of program results, 
and could benefit from adopting identified best practices for 
systematically evaluating the outcomes of federal R&D programs. Leading 
research organizations, the Office of Management and Budget, and GAO 
have identified a number of such practices, including setting clear R&D 
goals and measuring progress toward these goals, using expert review to 
evaluate the quality of research outcomes, and reporting periodically 
on evaluation results. The results of evaluations can be used to 
refocus R&D priorities periodically, as necessary, to ensure that 
program resources are most effectively utilized. The Pipeline Safety 
Improvement Act of 2002 requires that, starting in December 2003, DOT, 
the Department of Energy (DOE), and the National Institute of Standards 
and Technology jointly provide annual reports to Congress on their 
pipeline R&D efforts but does not fully specify what types of 
information should be included in these reports. This requirement 
provides an opportunity for OPS to keep Congress informed about the 
results of evaluations of its R&D program.

To improve OPS's ability to demonstrate the effectiveness of its R&D 
program and to make the most effective use of program resources, we are 
recommending that the agency develop a systematic process for 
evaluating program outcomes, using identified best practices, and 
include the results of R&D evaluations in the required annual reports 
to Congress on pipeline R&D. We provided DOT with a draft of this 
report for its review and comment. DOT officials generally agreed with 
the report's findings and conclusions. They emphasized that they have 
started to develop a framework for evaluating the effectiveness of 
their pipeline safety R&D program and that they intend to follow our 
recommendations as they move forward in developing and implementing 
this framework.

Background:

Three primary types of pipelines form a 2.2 million-mile network across 
the nation.

* Natural gas transmission pipelines transport natural gas over long 
distances from sources to communities.

* Natural gas distribution pipelines continue to transport natural gas 
from transmission lines to consumers.

* Hazardous liquid pipelines transport crude oil to refineries and 
refined oil products, such as gasoline, to product terminals.

OPS, within DOT's Research and Special Programs Administration (RSPA), 
is responsible for enhancing the safety of and reducing the potential 
environmental impacts of transporting natural gas and hazardous liquids 
through pipelines. The agency primarily carries out this responsibility 
through regulation, oversight, enforcement, and R&D. OPS sets and 
enforces regulations that pipeline operators must follow in designing, 
constructing, maintaining, and operating pipelines. State agencies 
responsible for overseeing pipeline safety help OPS to enforce its 
regulations.[Footnote 2] In December 2000, it began implementing a new 
risk-based regulatory approach, called "integrity management." Under 
this approach, operators are required, in addition to meeting minimum 
safety standards, to better protect pipeline segments where a leak or 
rupture could have significant consequences, such as near highly 
populated areas, by conducting new tests of these segments, completing 
repairs according to specified schedules, and developing comprehensive 
plans for addressing the range of risks facing these segments.[Footnote 
3] The agency's R&D program is aimed at advancing the most promising 
technologies for ensuring the safe operations of pipelines. For 
example, current R&D projects seek to develop new and improved 
techniques for assessing the condition of pipelines and detecting 
anomalies--such as leaks, corrosion, and damage from excavators--that 
can lead to pipeline accidents. From 1998 through 2002, a total of 
1,770 pipeline accidents occurred, resulting in 100 fatalities and $621 
million in property damage.[Footnote 4]

OPS's R&D program has undergone major changes in the last several 
years. In particular, the agency has developed a new agenda for its R&D 
program, using the input of key experts and stakeholders, and has 
received significant increases in funding for this program.

* Until 2001, most of the research funded by OPS was aimed at helping 
the agency perform its regulatory function or was in response to an 
accident investigation or congressional direction. In November 2001, 
the agency held an R&D planning workshop to gain the perspectives of a 
variety of experts and stakeholders on areas of R&D that have the most 
potential for enhancing pipeline safety. Attendees included 
representatives of federal and state agencies, research organizations, 
industry groups, pipeline companies, and technical organizations that 
set industry safety standards. OPS used the R&D priorities identified 
in this workshop to develop a new agenda for its R&D program, focusing 
on three main areas: (1) developing new technologies for preventing 
damage and detecting leaks, (2) improving technologies for operating, 
controlling, and monitoring the condition of pipelines, and (3) 
improving pipeline materials. From March through December 2002, the 
agency issued announcements requesting project proposals in these 
areas, asking that prospective funding recipients provide at least 50 
percent of the proposed project's cost. As of May 2003, it had funded 
10 R&D proposals it received in response to these 
announcements.[Footnote 5] In addition, after its November 2001 R&D 
workshop, OPS established a Web site on its R&D program in order to 
improve communications with experts, stakeholders, and the public about 
its R&D agenda and activities.

* OPS's budget for its R&D program has risen more than sevenfold since 
fiscal year 1998, with the most significant increases occurring since 
fiscal year 2001. Figure 1 shows the agency's budgeted amounts for R&D 
from fiscal years 1998 through 2003.[Footnote 6] OPS's budget for R&D 
rose steadily from fiscal year 1998 to fiscal year 2001, from $1.3 
million to $2.8 million. In fiscal year 2002, the agency received $4.8 
million for its R&D program, which was $2 million more than RSPA had 
requested for the program. Agency officials attribute this funding 
increase to increased concerns for pipeline safety within Congress 
following the tragic pipeline accidents in Bellingham, Washington 
(1999), and Carlsbad, New Mexico (2000), which together caused 15 
fatalities. For fiscal year 2003, RSPA requested and received about $4 
million in additional funding for the program, for a total of $8.7 
million. OPS officials told us that this requested increase was a 
response to heightened congressional interest in achieving 
technological solutions to pipeline safety, as evidenced by legislative 
proposals that called for increased attention to this area.[Footnote 7] 
RSPA is proposing funding for OPS's R&D program of $9.2 million in 
fiscal year 2004, an increase of about $0.5 million above the fiscal 
year 2003 amount. OPS officials explained that they intend to use most 
of this increase for a study, required by the Pipeline Safety 
Improvement Act of 2002, to assess the performance of controllers who 
monitor pipeline operations. Overall, agency officials also attribute 
recent increases in funding for OPS's pipeline safety R&D program to a 
recognition of the challenges posed by the agency's new integrity 
management regulatory approach and the criticality of the nation's 
pipeline infrastructure, in the aftermath of the terrorist attacks of 
September 11, 2001.

Figure 1: OPS's R&D Budget, Fiscal Years 1998-2003:

[See PDF for image]

Note: Figures are in constant fiscal year 2003 dollars.

[End of figure]

OPS's pipeline safety R&D program is continuing to evolve in response 
to new directives in the Pipeline Safety Improvement Act of 2002 for 
the planning and reporting of federal pipeline R&D efforts. The act, 
which became law in December 2002, assigned the Secretary of 
Transportation responsibility for developing a 5-year plan for pipeline 
R&D and transmitting the plan to Congress by December 2003, in 
coordination with DOE and the National Institute of Standards and 
Technology. (OPS officials told us that the Secretary has delegated 
this responsibility to OPS.):

DOE operates an R&D program that is focused on developing future 
technologies to improve the integrity, reliability, and security of the 
natural gas infrastructure, including pipelines and storage facilities. 
In comparison with OPS's R&D program, which focuses on the development 
of quick-to-market technologies that could become available in the 
short term (1-3 years) or midterm (3-5 years), DOE's program focuses on 
technologies that could become available in the midterm (3-5 years) or 
longer term (5-8 years). The National Institute of Standards and 
Technology does not operate an R&D program focused on pipelines, but, 
reflecting its expertise in materials research, the act assigns it a 
key role in planning future pipeline R&D.

The Department of the Interior's Minerals Management Service (MMS), 
although not assigned an R&D planning role in the act, funds pipeline 
R&D, including research on offshore pipeline safety. Consequently, OPS 
plans to include that agency in efforts to develop a 5-year plan for 
pipeline R&D. The act requires the heads of DOT, DOE, and the National 
Institute of Standards and Technology to jointly report annually to 
Congress, beginning in December 2003, on the status and results of 
implementation of the plan.

OPS's R&D Funding Is Aligned with Its Mission and Pipeline Safety 
Goals:

Since fiscal year 2001, OPS has allocated its rising R&D funding to 
three main areas of pipeline safety R&D that were identified at its 
2001 workshop: (1) developing new technologies for preventing damage to 
pipelines and detecting leaks, (2) improving technologies for 
operating, controlling, and monitoring the condition of pipelines, and 
(3) improving the performance of pipeline materials. The agency has 
also allocated some R&D funding to a fourth area, efforts to improve 
the agency's mapping and information systems.

On the basis of our work, we believe that the agency's R&D funding is 
generally aligned with its mission and pipeline safety goals. The 
agency has obtained the views of external experts and stakeholders in 
determining what types of R&D are aligned with its mission of ensuring 
the safe, reliable, and environmentally sound operation of the nation's 
pipeline transportation system. The agency has also recently improved 
coordination with other federal agencies that fund pipeline R&D in 
order to avoid overlap between their R&D programs. Both of these 
practices have been recommended by leading organizations that conduct 
scientific and engineering research. OPS has also linked its R&D 
efforts with its performance goals of reducing the impacts of pipeline 
incidents, including fatalities and injuries, and reducing spills of 
hazardous material. In its plans, the agency has described how new and 
improved technologies resulting from its R&D funding can help achieve 
these performance goals. Finally, a number of key experts and 
stakeholders told us that, in their view, the agency has chosen 
appropriate R&D areas to fund.

OPS Allocates Pipeline Safety R&D Funding to Four Major Areas:

OPS allocates its R&D budget to three major areas involving the 
research and development of pipeline safety technologies as well as to 
a fourth area--efforts to improve the agency's pipeline mapping and 
information systems--that does not involve such research and 
development. Figure 2 shows how the agency plans to distribute its 
fiscal year 2003 R&D budget of $8.7 million among these areas.[Footnote 
8]

OPS plans to spend the largest share of its R&D budget, 46 percent, or 
$4.0 million, on the area of Damage Prevention and Leak Detection, 
which includes the development of new technologies to prevent damage to 
pipelines, detect pipeline defects, and quickly and accurately locate 
and control pipeline leaks. Damage to pipelines from "third parties," 
such as companies performing excavation work, is the leading cause of 
pipeline failures and can lead to property damage and injuries or 
fatalities.[Footnote 9]

OPS plans to allocate 21 percent of its R&D budget, $1.9 million, to 
the area of Enhanced Operations, Controls, and Monitoring, which 
includes improvements in technologies for operating, controlling, and 
monitoring the integrity of pipelines to help identify and prioritize 
pipeline safety problems and solutions.

The agency intends to spend a slightly lesser amount, 19 percent of its 
R&D budget, or $1.7 million, on the area of Improved Materials 
Performance, which includes improvements in pipeline materials in order 
to extend the integrity and lifetime of installed pipelines and their 
various components.

Finally, the agency plans to allocate the smallest portion of its R&D 
budget, 14 percent, or $1.2 million, to the area of Mapping and 
Information Systems, which includes efforts to improve the collection, 
integration, and analysis of data on the location and safety 
performance of pipelines. These efforts make pipeline mapping 
information available to federal, state, and local officials and 
support pipeline inspection activities of OPS and its state partners.

Figure 2: OPS's Planned Allocation of R&D Funding for Fiscal Year 2003:

[See PDF for image]

Note: Shaded areas represent the major pipeline safety R&D areas funded 
by OPS. Dollar figures have been rounded.

[End of figure]

Since fiscal year 2001, OPS's allocation of funding to each of the 
three main areas of pipeline safety R&D--Damage Prevention and Leak 
Detection; Enhanced Operations, Controls, and Monitoring; and Improved 
Materials Performance--has risen significantly, while its allocation to 
Mapping and Information Systems efforts has remained level. The 
tripling of the agency's R&D budget--from $2.8 million in fiscal year 
2001 to $8.7 million in fiscal year 2003--has enabled it to increase 
funding for these three R&D areas. Specifically, OPS has increased 
funding for R&D efforts in Damage Prevention and Leak Detection from 
$1.3 million in fiscal year 2001 to $4.0 million in fiscal year 2003, 
an increase of over 200 percent. The agency has increased funding for 
Enhanced Operations, Controls, and Monitoring from $309,000 in fiscal 
year 2001 to $1.9 million in fiscal year 2003, an increase of more than 
500 percent. OPS started funding Improved Materials Performance 
research in fiscal year 2002, increasing funding in this area to a 
level of $1.7 million in fiscal year 2003.

Agency officials explained to us that they allocated funding to these 
three R&D areas in fiscal years 2002 and 2003 based on the results of 
their 2001 R&D planning workshop.[Footnote 10] For example, they added 
Improved Materials Performance to their R&D agenda because it was 
identified as a priority area at the workshop. They have also 
considered other factors in deciding how to allocate funding. For 
example, the agency significantly increased funding for R&D in the 
areas of Damage Prevention and Leak Detection and Enhanced Operations, 
Controls, and Monitoring because of a great need for improved 
performance in these areas. OPS officials explained that, because the 
agency's new risk-based regulatory approach requires pipeline operators 
to assess and mitigate risks to pipeline segments where a leak or 
rupture could have significant consequences, these operators need 
better tools and methods for monitoring pipelines and making necessary 
repairs. They also noted that OPS's R&D results assist in the creation 
of industry standards on the appropriate use of new technologies. In 
addition, officials explained that they decided to allocate a 
significant portion of their R&D budget to the area of Improved 
Materials Performance because, on the basis of current information on 
the development of pipeline technologies, they believed that advances 
in this area held much promise for improving pipeline safety.

Finally, OPS has allocated about $1.2 million per year to the Mapping 
and Information Systems area since fiscal year 2001 in order to 
maintain efforts to improve these systems.[Footnote 11] (See fig. 3.):

Figure 3: OPS's R&D Funding by Area of R&D, Fiscal Years 2001-03:

[See PDF for image]

Note: Figures are in constant fiscal year 2003 dollars and represent 
the agency's budgeted amounts for each area. In some cases, OPS spends 
less than the budgeted amount in a fiscal year. The agency has been 
allowed up to 3 years to spend amounts appropriated for R&D.

[End of figure]

OPS has provided $3.0 million in funding to 10 projects related to 
Damage Prevention and Leak Detection since fiscal year 2001. Examples 
of funded projects include the following:

* OPS provided $0.6 million in funding to five projects focused on 
improving in-line inspection techniques, including "smart pigs" and 
other technologies, for detecting damage and defects in pipe 
walls.[Footnote 12] Such improved techniques can help to prevent 
pipeline leaks or ruptures by making possible the early detection and 
repair of damage and defects.

* In partnership with the U.S. Air Force, OPS provided $1.2 million in 
funding to a project focused on developing an approach for detecting 
pipeline leaks using an airborne laser system that measures levels of 
chemicals in the atmosphere just above the earth's surface.

OPS has provided $0.9 million in funding to six projects related to 
Enhanced Pipeline Operations, Controls, and Monitoring since fiscal 
year 2001. Most of this funding--$0.6 million--has been allocated to 
two projects to improve alternative inspection techniques, called 
direct assessment, for identifying internal and external corrosion and 
other defects in pipelines that cannot accommodate smart pigs.[Footnote 
13] This is a significant issue for natural gas pipelines. One industry 
association estimates that only about 35 percent of the total natural 
gas pipeline mileage can accommodate smart pigs, which are typically 
used to assess the condition of liquid pipelines. OPS officials told us 
that they are planning to fund three additional R&D projects in this 
area in June 2003.

As of May 2003, OPS has provided $0.1 million in funding to one project 
in the area of Improved Materials Performance. This project seeks to 
develop a "smart" composite pipe that will allow for real-time 
monitoring of the condition of the pipe through a remote monitoring 
system. The agency requested proposals in this area in December 2002 
and expects to start funding some of these proposals in the summer of 
2003. Among the types of proposals that OPS has requested are proposals 
to develop:

* materials that better withstand third-party damage, corrosion, and 
cracking;

* higher grade/strength steels; and:

* materials that facilitate the operation of pipelines at higher design 
pressures.

Finally, of the roughly $1.2 million that OPS has allocated each year 
since fiscal year 2001 to the Mapping and Information Systems area, it 
spent or plans to spend:

* about $800,000 each year for efforts to improve the National Pipeline 
Mapping System, which depicts the location of pipelines in relation to 
areas that are populated or environmentally sensitive, and:

* about $400,000 each year for efforts to integrate information systems 
the agency uses in overseeing pipeline safety in cooperation with the 
states.

The agency expects to continue funding this area at this level for the 
foreseeable future in order to improve and update these systems 
continually. OPS officials explained that these mapping and information 
systems assist OPS inspectors and state and local officials in their 
efforts to oversee pipelines and protect the community and environment 
from pipeline leaks or ruptures.

Expert Review and Coordination Help OPS Align Its R&D Funding with Its 
Mission and Goals:

OPS's mission is to ensure the safe, reliable, and environmentally 
sound operation of the nation's pipeline transportation system. It has 
indicated in its budget and plans that its R&D program supports this 
broad mission as well as the following more specific performance goals: 
(1) to reduce deaths, injuries, property damage, and economic 
disruptions resulting from pipeline incidents and (2) to reduce the 
amount of oil and other hazardous liquids spilled from pipelines. The 
agency has described how new and improved technologies resulting from 
its R&D funding can help achieve these performance goals. For example, 
the number of pipeline incidents and the amount of hazardous material 
spilled could be reduced through the use of improved technologies for 
detecting third-party damage, corrosion, and defects and the use of 
improved pipeline materials that can better withstand damage and 
corrosion.

The Committee on Science, Engineering, and Public Policy--a joint 
committee of the National Academy of Sciences, the National Academy of 
Engineering, and the Institute of Medicine--has recommended the use of 
expert review to determine whether a research program is focused on the 
subjects most relevant to an agency's mission.[Footnote 14] Under this 
form of review, experts in related fields as well as potential users of 
the research evaluate the relevance of research to an agency's mission 
and goals and its potential value to intended users.

OPS has used expert review to help it develop a research agenda that is 
aligned with its mission and goals. At its November 2001 R&D planning 
workshop, it asked a variety of experts as well as potential users of 
research to identify the types of R&D that would be most likely to 
enhance pipeline safety. Participants included representatives from 
federal and state agencies with pipeline responsibilities, pipeline 
companies and their associations, research groups, and technical 
organizations that set industry safety standards for pipelines. The 
agency subsequently used the results of this workshop in developing its 
research agenda, guided by an R&D planning panel composed of key 
experts from such groups.

OPS has also used peer review, a form of expert review, in deciding 
which R&D proposals to fund, a practice that is recommended by the 
Committee on Science, Engineering, and Public Policy. OPS's review 
panels have included representatives from other federal agencies that 
conduct pipeline R&D, industry associations, and associations of state 
agencies with pipeline safety responsibilities.

The Pipeline Safety Improvement Act, enacted in December 2002, requires 
that the Secretary of Transportation consult with a variety of groups 
in preparing a 5-year plan for pipeline safety R&D, which must be 
provided to Congress by December 2003. In response, OPS is continuing 
to involve various experts and stakeholders in its R&D planning. Agency 
officials have told us that, in preparation for developing this 5-year 
plan, they are in the process of obtaining updated external views in 
order to reassess research priorities. This has involved participating 
in the pipeline R&D planning efforts of industry associations and 
research organizations, discussing R&D priorities with state agency 
officials, and reconvening their R&D planning panel of outside experts. 
In developing the plan, agency officials also plan to consult with 
OPS's two technical advisory committees. Finally, OPS plans to hold 
another R&D workshop during the winter of 2003-04.

Coordination among federal agencies that conduct related research helps 
to avoid duplication and ensure that each agency performs research that 
is aligned with its particular mission and goals. The Committee on 
Science, Engineering, and Public Policy has recommended that agencies 
establish a formal process for coordinating similar fields of research, 
in order to improve collaboration, help keep important questions from 
being overlooked, and avoid duplication of effort.[Footnote 15] Since 
2001, OPS has increased efforts to coordinate pipeline R&D with DOE and 
the Department of the Interior's MMS, both of which also conduct 
research related to pipelines. This increased coordination has taken 
the form of mutual participation in panels that review R&D proposals 
and workshops to plan R&D activities. According to OPS officials, 
officials of these agencies have used these opportunities to 
communicate about their respective pipeline R&D efforts and avoid 
duplication. However, these agencies have not had a formal mechanism in 
place that defines each agency's responsibilities for pipeline R&D.

The Pipeline Safety Improvement Act requires that the heads of DOT, 
DOE, and the National Institute of Standards and Technology develop a 
memorandum of understanding to formally coordinate pipeline R&D 
efforts. (Although the institute does not operate an R&D program 
focused on pipelines, the act assigned it a key role in pipeline R&D 
based on its expertise in materials research.) In response, OPS, DOE, 
and the institute have developed such a memorandum and are in the 
process of finalizing it.[Footnote 16] The Pipeline Safety Improvement 
Act also requires that DOT coordinate with DOE and the National 
Institute of Standards and Technology in developing a 5-year plan for 
pipeline R&D. In response, OPS is involving DOE and the institute, as 
well as MMS, in efforts to develop such a plan. These agencies are also 
considering holding joint workshops on pipeline R&D in the future. In 
addition, OPS and the National Institute of Standards and Technology 
have started to participate in each others' proposal review panels and 
are discussing entering into an agreement to have the institute conduct 
some research on pipeline materials.

We asked a number of key experts and stakeholders for their views on 
the extent to which OPS's R&D agenda is aligned with its mission and 
goals. These individuals included officials in DOE and MMS, 
representatives of four industry associations, a former head of a state 
agency that regulates gas pipelines, the heads of two leading pipeline 
R&D organizations, two foremost technical experts in pipeline safety, 
and an environmentalist active in pipeline safety. Six of these 
individuals have been or are members of OPS advisory committees or R&D 
planning or review panels. They generally told us that, in their view, 
the agency has chosen to fund appropriate areas.

Experts Generally Support OPS's R&D Priorities:

The pipeline safety R&D priorities of the experts who completed our 
questionnaire are generally consistent with OPS's R&D priorities. Of 
the three main R&D areas that OPS is currently funding, Damage 
Prevention and Leak Detection received the most scores of high or very 
high funding priority; Enhanced Operations, Controls, and Monitoring 
received the second highest number of such scores; and Improved 
Materials Performance received the third highest number. This ranking 
corresponds to the relative levels of funding OPS has assigned to these 
areas, as described in the previous section. However, the experts' 
level of support for Improved Materials Performance was much lower than 
that for the other two main R&D areas that OPS is funding. OPS 
officials told us that they are currently updating their research 
agenda, using the input of experts and stakeholders, and that they will 
consider our questionnaire results in this process.

To obtain the views of experts on pipeline safety R&D priorities, we 
asked 55 experts to complete a questionnaire indicating the funding 
priority they would assign to various types of pipeline safety R&D, 
using categories identified as part of OPS's 2001 R&D planning 
workshop. Table 1 provides a description of the main categories of R&D 
we asked experts to prioritize. The first three categories correspond 
to the main areas of R&D that OPS is currently funding. Although the 
fourth category--Arctic and Offshore Technologies--was identified as a 
main area of pipeline R&D at its workshop, OPS decided not to include 
it as a main area in its R&D agenda. Agency officials told us that they 
made this decision because R&D related to Arctic and Offshore 
Technologies was not considered to be a high priority by participants 
at its workshop and because MMS funds some R&D in this area and is the 
primary offshore regulator. We did not include Mapping and Information 
Systems--an area that OPS is currently funding from its R&D budget--as 
a category for the experts to rate because it was not identified as a 
main category of R&D at the 2001 workshop.

Table 1: Major Categories of R&D Related to Pipeline Safety:

Category of R&D: Damage Prevention and Leak Detection; Description: 
Develop new technologies to prevent third-party damage, detect pipeline 
defects, and quickly and accurately locate and control pipeline leaks.

Category of R&D: Enhanced Operations, Controls, and Monitoring; 
Description: Improve technology for operating, controlling, and 
monitoring the integrity of pipelines to help identify and prioritize 
pipeline safety problems and solutions.

Category of R&D: Improved Materials Performance; Description: Improve 
pipeline materials to extend the integrity and lifetime of installed 
pipelines and their various components.

Category of R&D: Arctic and Offshore Technologies; Description: Develop 
safer, more cost-effective materials and procedures to support Arctic 
and offshore pipeline applications.

Sources: Materials from OPS's 2001 R&D planning workshop and other OPS 
documents related to pipeline safety R&D.

[End of table]

Figure 4 shows how the 49 experts who completed our questionnaire rated 
the four categories of pipeline safety R&D. We also asked experts to 
rate specific types of R&D within each category. (See app. I for how 
the experts rated specific types of R&D within these main categories 
and for information on the agency's funding of these specific types of 
R&D. See app. II for information on our methodology for selecting 
experts and obtaining their views.):

The experts who completed our questionnaire strongly supported the 
Damage Prevention and Leak Detection and Enhanced Operations, Controls, 
and Monitoring categories of R&D as important areas for OPS to fund. 
Ninety-two percent of the experts (45 of 49) indicated that the Damage 
Prevention and Leak Detection category should receive high or very high 
funding priority.[Footnote 17] Within this category, experts assigned 
the most scores of high or very high funding priority to the following 
types of R&D: improvements in the ability of in-line inspection tools, 
such as "smart pigs," to detect damage and defects (39 of 49), and the 
development of new technologies, such as the innovative application of 
ultrasonics, that can be used for inspecting pipelines (38 of 49). 
Several experts we interviewed highlighted the need to improve methods 
for detecting damage to pipelines, citing the fact that third-party 
damage is the leading cause of pipeline accidents. According to both 
liquid and gas pipeline associations, current inspection tools cannot 
reliably detect such damage to pipelines.

Figure 4: Expert Ratings of Categories of Pipeline Safety R&D:

[See PDF for image]

Note: Percentages are based on 49 respondents.

[End of figure]

Eighty percent of the experts (39 of 49) indicated that the Enhanced 
Operations, Controls, and Monitoring category should receive high or 
very high funding priority. Within this category, the type of R&D that 
received the most scores of high or very high funding priority (37 of 
49) was the improvement of alternative inspection techniques, called 
direct assessment, to identify corrosion and other defects in pipelines 
that cannot accommodate in-line inspection devices known as smart pigs. 
This is a significant issue for natural gas pipelines because the 
majority of these pipelines cannot currently accommodate smart pigs, 
which are typically used to assess the condition of liquid pipelines.

In contrast to the experts' views on the importance of these first two 
categories, less than one-third of the experts considered the remaining 
two categories of R&D, Improved Materials Performance and Arctic and 
Offshore Technologies, to be a high priority for OPS to fund. Thirty-
one percent of the experts (15 of 49) assigned scores of high or very 
high funding priority to the Improved Materials Performance category, 
and 20 percent (10 of 49) assigned such scores to the Arctic and 
Offshore Technologies category. However, within the category of 
Improved Materials Performance, about half (25 of 49) of the experts 
indicated that the type of R&D aimed at developing damage-and defect-
resistant materials should receive high or very high funding priority. 
Such materials could be used in the replacement of existing pipe or in 
the installation of new pipe. One researcher we interviewed noted that 
such materials are particularly important for the gas pipeline 
industry, which is expanding its infrastructure in response to 
increased demands for natural gas. One industry association estimates 
that the natural gas industry will need to install about 49,500 miles 
of transmission pipeline from 2001 through 2015 to meet increased 
demand in the United States.

Some differences exist in the views of experts from the following three 
subgroups: (1) federal and state government and public interest 
organizations, (2) pipeline industry and technical and consulting 
organizations, and (3) research organizations.[Footnote 18] As shown in 
table 2, experts from all three subgroups generally gave the category 
of Damage Prevention and Leak Detection the highest ranking, followed 
by the category of Enhanced Operations, Controls, and Monitoring. 
However, experts from research organizations considered the categories 
of Improved Materials Performance and Arctic and Offshore Technologies 
to be more important for OPS to fund than did experts from the other 
two subgroups. For example, 70 percent of experts from research 
organizations (7 of 10) rated Improved Materials Performance as a high 
or very high priority compared with 19 percent of experts from 
government and public interest organizations (3 of 16) and 22 percent 
of experts from pipeline industry and technical and consulting 
organizations (5 of 23). In addition, 60 percent of the researchers (6 
of 10) rated Arctic and Offshore Technologies as a high or very high 
priority for OPS compared with 19 percent of experts from government 
and public interest organizations (3 of 16) and only 4 percent of 
experts from pipeline industry and technical and consulting 
organizations (1 of 23).

Table 2: Views of Experts from Three Subgroups on Pipeline Safety R&D 
Priorities:

Category of R&D: Damage Prevention and Leak Detection; Number of 
experts who assigned a high or very high funding priority to category: 
Experts from government and public interest organizations (16): 16; 
Number of experts who assigned a high or very high funding priority to 
category: Experts from pipeline industry and technical and consulting 
organizations (23): 20; Number of experts who assigned a high or very 
high funding priority to category: Experts from research organizations 
(10): 9.

Category of R&D: Enhanced Operations, Controls, and Monitoring; Number 
of experts who assigned a high or very high funding priority to 
category: Experts from government and public interest organizations 
(16): 13; Number of experts who assigned a high or very high funding 
priority to category: Experts from pipeline industry and technical and 
consulting organizations (23): 18; Number of experts who assigned a 
high or very high funding priority to category: Experts from research 
organizations (10): 8.

Category of R&D: Improved Materials Performance; Number of experts who 
assigned a high or very high funding priority to category: Experts from 
government and public interest organizations (16): 3; Number of experts 
who assigned a high or very high funding priority to category: Experts 
from pipeline industry and technical and consulting organizations (23): 
5; Number of experts who assigned a high or very high funding priority 
to category: Experts from research organizations (10): 7.

Category of R&D: Arctic and Offshore Technologies; Number of experts 
who assigned a high or very high funding priority to category: Experts 
from government and public interest organizations (16): 3; Number of 
experts who assigned a high or very high funding priority to category: 
Experts from pipeline industry and technical and consulting 
organizations (23): 1; Number of experts who assigned a high or very 
high funding priority to category: Experts from research organizations 
(10): 6.

Source: GAO.

[End of table]

An OPS official told us that he believed that researchers rated the 
Improved Materials Performance category more highly than other experts 
did because researchers have the best and most current information 
about the "state of the art" in technology development and are more 
aware of opportunities in this area. A leading expert from a pipeline 
research organization noted that the foundation of pipeline R&D has 
been the development of defect-resistant steels and that, as a 
consequence, researchers in this area are very interested in R&D that 
will lead to further improvements in the performance of pipeline 
materials. He also explained that researchers may have rated the Arctic 
and Offshore Technologies category more highly than the other types of 
experts who completed our questionnaire because researchers may be more 
aware of the need for such R&D to support the construction of new 
pipelines in these areas in order to reach new energy supplies.

OPS Lacks a Systematic Process for Evaluating R&D Outcomes:

Although OPS has received significant increases in funding for its R&D 
program in recent years, the agency has not developed a systematic 
process for evaluating the effectiveness of its R&D program. For 
example, the agency tracks and disseminates information on the progress 
of individual R&D projects but has not developed a process for 
assessing and reporting on the results of its R&D program as a whole. 
Such a process is needed to demonstrate the program's progress toward 
achieving its objectives, such as the development and use of new 
technologies that can improve pipeline safety. OPS has taken some 
preliminary steps toward developing an evaluation process for its R&D 
program and could benefit from adopting identified best practices for 
systematically evaluating the outcomes of federal R&D programs. Leading 
research organizations, the Office of Management and Budget, and GAO 
have identified a number of such practices, including setting clear R&D 
goals and measuring progress toward these goals, using expert review to 
evaluate the quality of research outcomes, and reporting periodically 
on evaluation results. The results of evaluations can be used to 
refocus the direction of R&D programs periodically, as necessary, to 
ensure that resources are most effectively utilized.

OPS's Efforts to Evaluate Research Outcomes Have Been Limited:

Although OPS has funded R&D to develop pipeline safety technologies 
since the mid-1990s, the agency's efforts to evaluate the outcomes of 
this R&D have been limited and have focused on individual 
projects.[Footnote 19] OPS's R&D contracts define project goals and 
require research performers to meet specific milestones for the 
development of a technology. Contracts also require research performers 
to report quarterly and at the end of the project on results, including 
milestones achieved and patents applied for and received. OPS has made 
some efforts to disseminate the results to date of individual R&D 
projects. For example, it has started to put "success stories" on its 
Web site that describe achievements in ongoing projects, such as the 
development of product prototypes. These success stories help to 
communicate the results of individual projects to industry and other 
interested parties.

At the program level, OPS has not yet established specific quantifiable 
goals for its R&D program or a method for measuring progress toward 
these goals. OPS has indicated, in various planning documents, that its 
R&D program will help achieve its performance goals of reducing the 
impacts of pipeline incidents, including fatalities and injuries, and 
reducing spills of hazardous material. However, agency officials have 
acknowledged that it is difficult to show the effect of the R&D program 
on these performance goals. A more immediate objective of the program, 
according to agency plans, is to promote the transfer of new and 
improved pipeline safety technologies to the market in the near term.

In deciding which R&D proposals to fund, OPS gives preference to those 
that plan to bring a new product to market within 5 years. In addition, 
agency officials told us that OPS plans to promote the use of new 
technologies by providing information to potential users and its state 
partners about them and, when appropriate, by encouraging their use 
through regulation.[Footnote 20] Agency officials told us that the R&D 
program aims to have 80 percent of its projects result in products on 
the market within 5 years. Such an objective is specific and 
measurable, but OPS has not formally established it as a goal in any 
plan or developed a method for measuring progress toward achieving it. 
Furthermore, since the agency has not yet established specific goals or 
outcome measures for its R&D program, it does not have a process for 
documenting and reporting on the extent to which this program is 
achieving its goals.

OPS officials explained that they have not yet developed a process for 
evaluating the outcomes of the agency's R&D program because, prior to 
2001, the program's budget was relatively low and, since restructuring 
the program in 2001, they have focused program efforts on building a 
process for setting research priorities. However, officials do 
recognize the need for evaluating R&D outcomes and have taken some 
preliminary steps toward developing an evaluation process for their R&D 
program.

OPS is considering some possible measures of the outcomes of its R&D 
program as a whole, such as the number of new patents resulting from 
R&D efforts. In addition, agency officials told us that, although 
tracking the transfer to the market of new pipeline safety technologies 
can be challenging, OPS intends to track the use of new technologies in 
the future through its process for inspecting operators' "integrity 
management" programs.[Footnote 21] For example, OPS inspectors could 
document the use of new or improved technologies by companies to 
evaluate the condition of their pipelines. Agency officials noted that 
the agency will develop inspection protocols that require inspectors to 
collect data on the use of new technologies after their proposed 
integrity management rule for natural gas transmission pipelines is 
finalized.

OPS is also considering the number of documented R&D "success stories"-
-summaries of the accomplishments of individual R&D projects--as a 
possible measure of program results. However, in previous reviews of 
R&D programs operated by other federal agencies, we have found that the 
success story approach is selective and does not adequately assess 
programwide performance.[Footnote 22]

In early June 2003, OPS presented a potential set of performance 
measures for its R&D program to its R&D planning panel of outside 
experts in order to obtain their views on these measures. This panel 
includes representatives of DOE, MMS, the National Institute of 
Standards and Technology, pipeline industry associations, state 
agencies with pipeline responsibilities, and a key pipeline research 
organization. OPS intends to refine its set of measures based on 
comments received from this panel and to continue obtaining the views 
of this panel as it moves forward in developing an evaluation process 
for its R&D program.

Finally, OPS officials also told us that the agency intends to obtain 
the views of experts on its R&D outcomes as well as on its future R&D 
priorities at its next R&D workshop, scheduled for the winter of 2003-
04. However, OPS is in the beginning stages of planning this workshop 
and has not defined a process for using experts' views to evaluate the 
outcomes of its R&D program.

OPS officials told us that they are considering including information 
on the effectiveness of the agency's R&D program in the annual reports 
to Congress on pipeline R&D that the agency is required to submit, 
starting in December 2003. The Pipeline Safety Improvement Act requires 
that DOT, DOE, and the National Institute of Standards and Technology 
jointly provide these annual reports to Congress, but does not fully 
specify what types of information should be included in these reports.

Best Practices Help Agencies Systematically Evaluate Research Outcomes:

Since OPS is in the beginning stages of developing an evaluation 
process for its R&D program, it could benefit from adopting best 
practices for systematically evaluating federal R&D programs. Leading 
organizations that conduct scientific and engineering research, the 
Office of Management and Budget, and GAO have identified a number of 
these best practices. Although the uncertain nature of research 
outcomes over time can make it challenging to demonstrate the results 
of such R&D programs, these practices are designed to enable agencies 
to systematically assess and report on these results regularly in 
accordance with the Government Performance and Results Act of 
1993.[Footnote 23] These assessments can be used to refocus the 
direction of R&D programs periodically, as necessary, to ensure that 
resources are most effectively utilized. Identified best practices are 
discussed in the following sections.

Setting Clear, Quantifiable Goals and Measuring Progress toward These 
Goals:

We have previously reported that, to be effective, any R&D program must 
be directed toward a clear, measurable goal.[Footnote 24] Such goals 
help ensure a direct linkage between R&D program efforts and an 
agency's overall performance goals and mission. Applied research 
programs, such as OPS's R&D program, are directed toward achieving 
specific useful outcomes, such as the development of new technologies, 
which can help accomplish agency performance goals. The Committee on 
Science, Engineering, and Public Policy recommended in a 1999 report 
that agencies operating applied research programs measure progress 
toward practical outcomes and noted that such measurement can usually 
be performed annually using milestones.[Footnote 25]

Similarly, in May 2002 the Office of Management and Budget established 
investment criteria for federal R&D programs that require these 
programs to clearly define goals and track progress toward these goals 
using appropriate outcome measures and interim milestones. Indicators 
that have been used to measure the outcomes of R&D include the 
achievement of specific targets for developing new or improved 
technologies and patent applications filed and granted.[Footnote 26] 
However, measuring research outcomes can be challenging. For example, 
outcomes may not occur for a number of years and may be difficult to 
track.

Using Expert Review to Evaluate the Quality of Research Outcomes:

In its 1999 report and again in 2001, the Committee on Science, 
Engineering, and Public Policy recommended the use of expert review, 
supplemented by quantitative methods, to evaluate research 
regularly.[Footnote 27] Expert review can be a useful addition to 
performance measures because of the value of the reviewers' deep 
knowledge of the field. Such review can be performed on a somewhat 
longer term basis, rather than annually, and does not require that the 
final impact of the research be known. Peer review, a form of expert 
review, includes an independent assessment of the technical and 
scientific merit or quality of research by peers with essential subject 
matter expertise and perspective equal to that of the researchers. In 
1999, we reported that some federal agencies, such as the Department of 
Agriculture, the National Institutes of Health (NIH), and DOE, use peer 
review to help them evaluate the performance of programs and determine 
whether to continue or renew research projects.[Footnote 28]

The Committee on Science, Engineering, and Public Policy reported in 
2001 on the use of expert review, including peer review, by NIH, DOE, 
the National Science Foundation, the Department of Defense, and the 
National Aeronautics and Space Administration to evaluate the quality 
of their research programs. These agencies used varying methods for 
carrying out this review, including convening panels of experts who use 
defined evaluation processes and obtaining the views of external 
advisory committees. The Committee on Science, Engineering, and Public 
Policy has also noted that expert evaluation of applied research 
programs requires the input of potential users of the results of the 
research, since the ultimate usability of these results is an important 
factor in determining the worth of the research. Similarly, key experts 
and stakeholders we interviewed noted that the degree to which new 
technologies are actually used would be a good indication of the 
effectiveness of OPS's R&D program. One industry association 
representative we interviewed noted that a "constant theme" raised by 
pipeline companies is the need for R&D efforts to produce new 
technologies that they can actually use in operating their pipelines.

Reporting Periodically on Evaluation Results:

Periodic reporting by applied research programs on results can help 
keep key stakeholders--including oversight organizations and potential 
users of new technologies--up-to-date on program accomplishments. 
According to the Committee on Science, Engineering, and Public Policy, 
applied research programs can usually report annually on progress in 
meeting milestones. In addition, a retrospective analysis over several 
years is needed to evaluate outcomes that take more than 1 year to 
emerge. The committee also has recommended that agencies demonstrate 
the value of their review processes by publicly describing them to 
oversight groups, the potential users of research results, and the 
general public. One expert we interviewed stressed the importance of 
periodic public reporting by OPS on research goals and outcomes and on 
the method for evaluating outcomes, in order to disseminate research 
results and build support for its R&D program.

Conclusions:

OPS has made significant progress in establishing a pipeline safety 
research agenda that is aligned with its mission and goals and that 
incorporates the views of experts and stakeholders. However, without a 
systematic process for evaluating the outcomes of its R&D program, the 
agency is not able to demonstrate that it is effectively using its 
increased resources for R&D to foster new and improved technologies 
that can enhance pipeline safety. Identified best practices for 
evaluating federal R&D programs--including setting clear quantifiable 
R&D goals and measuring progress toward these goals, using expert 
review to evaluate the quality of research outcomes, and reporting 
periodically on evaluation results--can guide OPS as it moves forward 
in developing an evaluation process for its program. By following such 
practices, the agency can help ensure that it develops a systematic 
evaluation process that will enable it to determine and demonstrate the 
results of its investment in pipeline safety R&D. OPS could use such an 
evaluation process to periodically refocus the direction of its program 
in order to make the most effective use of resources.

Furthermore, although the Pipeline Safety Improvement Act's requirement 
for annual reports on pipeline R&D, starting in December 2003, does not 
specify in detail what information should be included in these reports, 
this requirement provides an opportunity for the agency to keep 
Congress informed about the results of evaluations of its R&D program. 
In addition, such reporting, along with other communication methods 
already in use by the agency, can keep other interested parties--
including the pipeline industry, state pipeline safety agencies, 
pipeline safety advocates, and researchers--up-to-date on the program's 
progress in advancing the most promising pipeline safety technologies.

Recommendations for Executive Action:

To improve OPS's ability to demonstrate the effectiveness of its R&D 
program and make the most effective use of program funds, we recommend 
that the Secretary of Transportation direct OPS to:

* develop a systematic process for evaluating the outcomes of its R&D 
program that incorporates identified best practices and:

* include in the annual reports to Congress, which are required by the 
Pipeline Safety Improvement Act, information on the results of R&D 
evaluations.

Agency Comments:

We provided DOT with a draft of this report for review and comment. DOT 
officials, including OPS's Director of Program Development, provided 
oral comments on the draft on June 13, 2003. The officials generally 
agreed with the report's findings and conclusions. They emphasized that 
they are starting to develop a framework for evaluating the 
effectiveness of their pipeline safety R&D program and that they intend 
to finalize this framework by December 2003 by documenting it in the 5-
year plan and first annual report on pipeline R&D that DOT is required 
to submit to Congress, jointly with DOE and the National Institute of 
Standards and Technology. They also noted that they agree with and 
intend to implement our recommendations and provided some technical 
clarifications, which we have incorporated as appropriate.

We are sending copies of this report to the Secretary of 
Transportation, the Administrator of RSPA, RSPA's Associate 
Administrator for Pipeline Safety, the Director of the Office of 
Management and Budget, and appropriate congressional committees. We 
will make copies available to others upon request. In addition, this 
report will be available at no charge on the GAO Web site at http://
www.gao.gov.

If you or your staffs have any questions about this report, please 
contact me at (202) 512-2834 or guerrerop@gao.gov. Individuals making 
key contributions to this report are listed in appendix III.

Signed by:

Peter Guerrero 
Director, 
Physical Infrastructure Issues:

[End of section]

Appendixes:


Appendix I: Experts' Views on R&D Priorities and OPS's R&D Funding, by 
Type of R&D:

We asked selected experts to review the following descriptions of 
specific types of pipeline safety research and development (R&D) and 
assign a funding priority to each, based on its importance in achieving 
the Office of Pipeline Safety's (OPS) mission of ensuring the safe, 
reliable, and environmentally sound operation of the nation's pipeline 
transportation system. Experts used the following scale: 1=little or no 
funding, 2=some funding priority, 3=moderate funding priority, 4=high 
funding priority, and 5=very high funding priority. Experts could also 
indicate that they did not know or had no basis to judge the funding 
priority for a particular type of R&D. The following table shows, for 
each type of R&D, the number of experts who assigned it a high or very 
high funding priority and OPS's current and planned allocation of 
funding to it. A total of 49 experts completed our questionnaire.

Type of R&D: Damage Prevention and Leak Detection:

Type of R&D: In-line inspection for damage and defects: Improve in-line 
inspection techniques, including "smart pigs" and other technologies, 
for detecting and measuring damage, cracking, and defects in pipe 
walls; Number of experts who assigned it a high or very high funding 
priority: 39; OPS's current and planned allocation of 
funding to this type of R&D[A]: 
Allocated $592,500 to five projects in November 2002 for periods of 9 
to 24 months.

Type of R&D: Nondestructive evaluation: Develop new approaches or 
technologies, such as the innovative application of ultrasonics, that 
can be used for the nondestructive evaluation of operational pipelines; 
Number of experts who assigned it a high or very high funding priority: 
38; OPS's current and planned allocation of funding to 
this type of R&D[A]: Allocated 
$500,000 to one project in November 2002 for a period of 24 months.

Type of R&D: Real-time monitoring using sensors attached to pipe: 
Develop and test real-time sensors applied or attached to the pipe that 
can detect possible third-party contact, leaks, or other signs of 
damage; Number of experts who assigned it a high or very high funding 
priority: 27; OPS's current and planned allocation of 
funding to this type of R&D[A]: 
Allocated $182,000 to one project in April 2001 for period of 12 
months. Requested proposals in March 2002 but did not fund any of those 
received. Requested additional proposals in December 2002 and plans to 
make funding decisions in July 2003.

Type of R&D: Small leak detection: Improve technologies for quickly 
detecting small pipeline leaks; Number of experts who assigned it a 
high or very high funding priority: 22; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Requested proposals in December 
2002 and plans to make funding decisions in July 2003.

Type of R&D: Pipe location: Develop better techniques or materials to 
locate steel and plastic pipelines, including determining their depth; 
Number of experts who assigned it a high or very high funding priority: 
17; OPS's current and planned allocation of funding to 
this type of R&D[A]: Allocated 
$534,521 to two projects in July 2002 for periods of 23 to 24 months.

Type of R&D: Encroachment monitoring using satellites: Develop 
satellite monitoring for encroachment and ground movement; Number of 
experts who assigned it a high or very high funding priority: 16; 
OPS's current and planned allocation of funding to 
this type of R&D[A]: Requested 
proposals in March 2002 but did not fund any of those received. 
Requested additional proposals in December 2002 and plans to make 
funding decisions in July 2003.

Type of R&D: Improved directional drilling: Improve directional 
drilling techniques to avoid accidental damage to other underground 
utilities; Number of experts who assigned it a high or very high 
funding priority: 14; OPS's current and planned 
allocation of funding to this type of R&D[A]: Requested proposals in 
March 2002 but did not fund any 
of those received. Requested additional proposals in December 2002 and 
plans to make funding decisions in July 2003.

Type of R&D: Real-time right-of-way monitoring without pipe contact: 
Develop fiber optic lines that can be buried above or alongside 
pipeline to detect nearby movement; Number of experts who assigned it 
a high or very high funding priority: 12; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Requested proposals in March 2002 
but did not fund any of those received. Requested additional proposals 
in December 2002 and plans to make funding decisions in July 2003.

Type of R&D: Airborne chemical mapping: Develop approaches using aerial 
surveillance with optical technologies for right-of-way monitoring or 
other pipeline safety concerns; Number of experts who assigned it a 
high or very high funding priority: 11; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Allocated $600,000 to one project 
in April 2001 for a period of 12 months. Allocated an additional 
$600,000 to this project in April 2002 for an additional 12 months. 
Plans to allocate an additional $600,000 to this project in May 2003 
for an additional 12 months.[B].

Type of R&D: Enhanced Operations, Controls, and Monitoring:

Type of R&D: Direct assessment: Improve alternative inspection 
techniques for "unpiggable" pipelines to identify internal and 
external corrosion, third-party damage, and other pipe defects; Number 
of experts who assigned it a high or very high funding priority: 
Damage Prevention and Leak Detection: 37; OPS's current and planned 
allocation of funding to this type of R&D[A]: Allocated 
$572,000 to two projects in January 2003 for periods of 12 to 26 
months.

Type of R&D: External corrosion control: Improve techniques for 
characterizing, detecting, and preventing external corrosion damage; 
Number of experts who assigned it a high or very high funding priority: 
30; OPS's current and planned allocation of funding to 
this type of R&D[A]: Allocated 
$297,000 to one project in January 2003 for a period of 26 months.

Type of R&D: Internal corrosion control: Improve techniques for 
characterizing, detecting, and preventing internal corrosion damage; 
Number of experts who assigned it a high or very high funding priority: 
30; OPS's current and planned allocation of funding to 
this type of R&D[A]: Allocated 
$275,000 to one project in January 2003 for a period of 12 months.

Type of R&D: Stress corrosion cracking detection: Improve techniques 
for characterizing, detecting, and preventing stress corrosion 
cracking; Number of experts who assigned it a high or very high funding 
priority: 25; OPS's current and planned allocation of 
funding to this type of R&D[A]: 
Allocated $675,281 to four projects in May and July 2002 for periods of 
12 to 24 months.

Type of R&D: Enhanced repair techniques: Develop enhanced repair 
techniques that can be implemented without shutdown of pipeline; Number 
of experts who assigned it a high or very high funding priority: 25; 
Detection: OPS's current and planned allocation of funding to 
this type of R&D[A]: [Empty].

Type of R&D: Risk assessment: Enhance techniques to integrate and 
evaluate risk data to define pipe susceptibility to various threats; 
Number of experts who assigned it a high or very high funding priority: 
25; OPS's current and planned allocation of funding to 
this type of R&D[A]: Allocated 
$97,737 to three projects in May 2002 for a period of 12 months. 
Allocated $70,000 to an additional project in January 2003 for a period 
of 24 months. Requested additional proposals in December 2002 and plans 
to make funding decisions in July 2003.

Type of R&D: Pipe strength: Improve methods for characterizing 
remaining pipe strength; Number of experts who assigned it a high or 
very high funding priority: 17; 
OPS's current and 
planned allocation of funding to this type of R&D[A]: Requested 
proposals in December 2002 and plans to make funding decisions in 
July 2003.

Type of R&D: Human factors: Study human factors, such as operator 
fatigue, that influence pipeline integrity and develop technologies or 
procedures to minimize operator error; Number of experts who assigned 
it a high or very high funding priority: 7; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Requested proposals in June 2002 
but did not fund any of those received. Requested additional proposals 
in December 2002 and plans to make funding decisions in July 2003.

Type of R&D: Trenchless pipe installation: Develop trenchless pipe 
installation and replacement techniques, including techniques that use 
directional drilling or robotics; Number of experts who assigned it a 
high or very high funding priority: 5; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
[Empty].

Type of R&D: Improved Materials Performance:

Type of R&D: Damage-and defect-resistant materials: Develop materials 
that better withstand third-party damage, corrosion, and cracking; 
Number of experts who assigned it a high or very high funding priority: 
25; OPS's current and planned allocation of funding to 
this type of R&D[A]: Requested 
proposals in December 2002 and plans to make funding decisions in July 
2003.

Type of R&D: Pipe coatings: Develop enhanced field-and factory-applied 
coatings, methods for testing coatings, and methods to improve coating 
choices; Number of experts who assigned it a high or very high funding 
priority: 15; OPS's current and planned allocation of 
funding to this type of R&D[A]: Requested proposals in December 2002 
and plans to make funding decisions in July 2003.

Type of R&D: Higher grade/strength steels: Develop higher grade/
strength steels, evaluate their performance, and develop methods for 
determining when to use them; Number of experts who assigned it a high 
or very high funding priority: 14; OPS's current and 
planned allocation of funding to this type of R&D[A]: Requested 
proposals in December 2002 and plans to make funding decisions in 
July 2003.

Type of R&D: Welding and joining: Develop enhanced welding and joining 
techniques and improved methods for assessing performance of welds and 
joints; Number of experts who assigned it a high or very high funding 
priority: 13; OPS's current and planned allocation of 
funding to this type of R&D[A]: 
Requested proposals in December 2002 and plans to make funding 
decisions in July 2003.

Type of R&D: Higher design pressure: Develop materials that facilitate 
pipelines operating at higher design pressures and methods for 
determining when to use higher pressure designs; Number of experts who 
assigned it a high or very high funding priority: 11; 
OPS's current and planned allocation of funding to this type of R&D[A]: 
Requested proposals in December 
2002 and plans to make funding decisions in July 2003.

Type of R&D: Composite pipe: Develop pipe made of, or layered with, 
materials other than steel that may exceed current performance 
standards or allow greater flexibility or lower cost in challenging 
installation conditions; Number of experts who assigned it a high or 
very high funding priority: 11; 
OPS's current and 
planned allocation of funding to this type of R&D[A]: Allocated $98,680 
to one project in November 2002 
for a period of 6 months. Requested additional proposals in December 
2002 and plans to make funding decisions in July 2003.

Type of R&D: Plastic pipe: Develop new or improved plastic pipe for 
local distribution company systems; Number of experts who assigned it 
a high or very high funding priority: 11; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Requested proposals in December 
2002 and plans to make funding decisions in July 2003.

Type of R&D: Arctic and Offshore Technologies.

Type of R&D: Leak detection: Develop approaches to detect, verify, and 
respond to leaks; Number of experts who assigned it a high or very high 
funding priority: 17; OPS's current and planned 
allocation of funding to this type of R&D[A]: Allocated $7,781 to one 
project in May 2002 for a period of 12 months.

Type of R&D: Inspection and maintenance procedures: Develop alternative 
inspection and maintenance technologies and procedures; Number of 
experts who assigned it a high or very high funding priority: 8; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
Allocated $50,000 to one project in May 2001 for a period of 12 months.

Type of R&D: Enhanced performance: Develop materials and fabrication 
techniques to enhance low temperature performance; Number of experts 
who assigned it a high or very high funding priority: 8; 
OPS's current and planned allocation of funding to this type of R&D[A]: 
Allocated $59,955 to one project 
in May 2002 for a period of 12 months.

Type of R&D: Site evaluation: Develop improved techniques for site 
evaluation; Number of experts who assigned it a high or very high 
funding priority: 5; OPS's current and planned 
allocation of funding to this type of R&D[A]: [Empty].

Type of R&D: Other Pipeline Safety Improvements.

Type of R&D: Inspection tools: Evaluate and quantify, where possible, 
the strengths, limits, and performance of current inspection tools; 
Number of experts who assigned it a high or very high funding priority: 
28; OPS's current and planned allocation of funding to 
this type of R&D[A]: [Empty].

Type of R&D: Pipeline modeling enhancements: Develop better 
mathematical or computational modeling techniques to improve ability to 
detect defects, including growth defects and small leaks; Number of 
experts who assigned it a high or very high funding priority: 24; 
Detection: OPS's current and planned allocation of funding to 
this type of R&D[A]: Requested 
proposals in December 2002 and plans to make funding decisions in July 
2003.

Type of R&D: Impact of multiple utilities: Characterize impact of 
multiple utilities in common right-of-way on integrity management 
practices, such as cathodic protection; Number of experts who assigned 
it a high or very high funding priority: 18; OPS's 
current and planned allocation of funding to this type of R&D[A]: 
[Empty].

Type of R&D: Higher stress levels: Evaluate potential for current 
piping to operate at higher stress levels; Number of experts who 
assigned it a high or very high funding priority: 17; 
OPS's current and planned allocation of funding to this type of R&D[A]: 
[Empty].

Type of R&D: Reduction of rupture impact: Explore means to reduce the 
impact of a pipeline rupture and explosion, for example, through 
additives to gas/liquid or enhanced shutoff capability; Number of 
experts who assigned it a high or very high funding priority: 15;  
Detection: OPS's current and planned allocation of funding to 
this type of R&D[A]: [Empty].

Type of R&D: Impact of past releases: Research the impact of past 
pipeline releases on their surrounding areas and provide information 
that could be used to support local zoning decisions; Number of experts 
who assigned it a high or very high funding priority: 7; 
OPS's current and planned allocation of funding to this type of R&D[A]: 
[Empty].

Sources: OPS data; GAO analysis.

Note: The pipeline safety R&D categories were identified as part of 
OPS's 2001 R&D Planning Workshop. The descriptions of the categories 
are based on materials from this workshop as well as OPS's 2002 and 
2003 announcements soliciting R&D proposals. The information on OPS's 
funding of each category is based on GAO's analysis of information 
provided by OPS.

[A] This column describes OPS's funding of R&D projects, by category, 
in fiscal years 2001 and 2002 and planned allocation of funding in 
fiscal year 2003. Some projects that are applicable to more than one 
category of R&D appear more than once.

[B] In conference reports accompanying appropriations bills for fiscal 
years 2001, 2002, and 2003, Congress expressed its intent that the 
Research and Special Programs Administration devote $600,000 of its 
pipeline safety R&D budget to this project in each of these fiscal 
years.

[End of table]

[End of section]

Appendix II: Scope and Methodology:

To perform our work, we reviewed Office of Pipeline Safety (OPS) 
documentation on its research and development (R&D) funding and 
analyzed this information to identify trends; reviewed pertinent 
legislation and agency documents pertaining to the R&D program; and 
interviewed OPS officials regarding their R&D funding, agenda-setting 
processes, and processes for evaluating the outcomes of their R&D 
program. We also interviewed key experts and stakeholders concerning 
OPS's management of its R&D program, including the alignment of the 
agency's research agenda with its mission and goals, and their views on 
R&D priorities and gaps. These individuals included officials of the 
Department of Energy (DOE) and the Department of the Interior's 
Minerals Management Service (MMS) who are responsible for pipeline R&D; 
representatives of pipeline industry associations and leading pipeline 
research organizations; and several key experts in pipeline safety. 
Also, we identified best practices for evaluating the outcomes of 
federal R&D through a review of relevant literature and compared the 
agency's processes with these practices.

To determine the views of experts on pipeline safety R&D priorities, we 
sought to identify experts considered to be very knowledgeable about 
the development of new pipeline safety technologies or pipeline safety 
issues. To identify appropriate experts, we obtained recommendations on 
individuals to contact from key organizations, contacted those 
individuals, and obtained further recommendations from them on 
additional individuals to contact. We identified initial individuals to 
contact through prior work on pipeline safety issues or through 
recommendations from OPS. These initial contacts included officials in 
DOE and MMS, representatives of four industry associations, a former 
head of a state agency that regulates gas pipelines, the heads of two 
leading pipeline R&D organizations, two technical experts in pipeline 
safety, and an environmentalist active in pipeline safety. Six of these 
individuals have been or are members of OPS advisory committees or R&D 
planning or review panels. We obtained recommendations from these 
individuals on experts who could provide us with views on pipeline 
safety R&D priorities.

We based our final selection of experts on the criteria of knowledge, 
balance, and independence. We considered indications of their extent of 
knowledge of pipeline safety R&D, as evidenced by the number of times 
they had been recommended, their participation in OPS's R&D planning 
and review activities, or other relevant factors. We included 
individuals from a variety of groups in order to achieve a balanced 
representation of experts, including some who are relatively 
independent of OPS and the pipeline industry. We included individuals 
from federal and state agencies, pipeline safety advocacy groups, 
industry associations, pipeline companies, technical and consulting 
organizations, and research organizations. We also provided our list of 
identified experts to the National Academy of Sciences and OPS for 
their review and comment.

We contacted 55 individuals whom we had identified as appropriate 
experts for our review and asked them to complete a questionnaire 
indicating their views on pipeline safety R&D priorities. Forty-nine 
individuals responded, for an 89 percent response rate. Our results 
pertaining to experts' views on R&D priorities represent the views of 
only the experts who responded to our questionnaire. In a number of 
cases, these individuals collaborated with others in their 
organizations in completing their questionnaires. Listed below are the 
organizational affiliations of experts who completed our 
questionnaire.[Footnote 29]

Government and Public Interest Organizations:

Federal Agencies Federal Energy Regulatory Commission Minerals 
Management Service, Department of the Interior National Institute of 
Standards and Technology, Department of Commerce National 
Transportation Safety Board Office of Fossil Energy, Department of 
Energy:

State Agencies and Associations National Association of Pipeline Safety 
Representatives National Association of Regulatory Utility 
Commissioners New York State Department of Public Service Railroad 
Commission of Texas Virginia State Corporation Commission Washington 
Utilities and Transportation Commission:

Pipeline Safety Advocacy Groups Common Ground Alliance Cook Inlet 
Keeper Safe Bellingham:

Pipeline Industry and Technical/Consulting Organizations:

Industry Associations American Gas Association American Petroleum 
Institute Association of Oil Pipelines Interstate Natural Gas 
Association of America Offshore Operators Committee:

Pipeline Companies BP Pipelines, North America ConocoPhillips CMS 
Panhandle Companies Duke Energy El Paso Corporation Enbridge Pipelines 
Enron Explorer Pipeline Company ExxonMobil Pipeline Company KeySpan 
Energy Peoples Energy Shell Pipeline Company:

Technical/Consulting Organizations Accufacts, Inc. Batten and 
Associates, Inc. Duckworth Pipeline Integrity Services, Inc. HSB 
Solomon Kiefner and Associates, Inc. National Association of Corrosion 
Engineers:

Research Organizations:

Advantica, Inc. Battelle CFER Technologies Edison Welding Institute Gas 
Technology Institute Ohio State University, Fontana Corrosion Center 
Pipeline Research Council International, Inc. Southwest Research 
Institute Texas A&M University, Department of Mechanical Engineering 
University of Florida, Department of Chemical Engineering:

In the questionnaire, we asked respondents to review descriptions of 
various main categories of pipeline safety R&D as well as specific 
types of R&D within these main categories and indicate what funding 
priority they would assign to each.[Footnote 30] (See table 1 for 
descriptions of the main R&D categories. See app. I for descriptions of 
the types of R&D within these main categories.) We based the R&D 
categories and descriptions on materials prepared as part of an R&D 
planning workshop held by OPS in 2001, in which a variety of experts 
and stakeholders participated; on announcements the agency subsequently 
issued soliciting proposals for R&D in various areas; and on other OPS 
documents related to pipeline safety R&D.

We compiled the scores obtained from the questionnaires to produce a 
ranking of R&D priorities representing the views of the experts who 
completed our survey. We also analyzed our results to determine whether 
any differences existed in the responses of experts from the three 
subgroups: government and public interest organizations, industry and 
technical and consulting organizations, and research organizations. In 
addition, we identified organizations that had bid on R&D funding from 
OPS in fiscal year 2002 and conducted a separate analysis of the 
responses of experts from these organizations to determine how they 
compared with those of other experts who completed our questionnaire. 
Seven of the experts who completed our questionnaire are from 
organizations that had bid on OPS R&D funding within this time 
frame.[Footnote 31]

We conducted our work from January 2003 through June 2003 in accordance 
with generally accepted government auditing standards.

[End of section]

Appendix III: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

Peter Guerrero, (202) 512-2834 Susan Fleming, (202) 512-4431:

Acknowledgments:

In addition to those named above, Sharon Dyer, Etana Finkler, Judy 
Guilliams-Tapia, Brandon Haller, Bert Japikse, Nancy Kingsbury, Donna 
Leiss, Gary Stofko, Ron Stouffer, and Stacey Thompson made key 
contributions to this report.

(545025):

FOOTNOTES

[1] Figures do not add to total due to rounding.

[2] In general, OPS retains full responsibility for inspecting and 
enforcing regulations on interstate pipelines but certifies states to 
perform these functions for intrastate pipelines. In 2003, 49 state 
agencies, the District of Columbia, and Puerto Rico were certified for 
inspecting and enforcing regulations on intrastate pipelines. In 
addition, OPS has agreements with 11 states to inspect segments of 
interstate pipelines within their boundaries.

[3] We have previously reported on OPS's implementation of this new 
regulatory approach. See U.S. General Accounting Office, Pipeline 
Safety and Security: Improved Workforce Planning and Communication 
Needed, GAO-02-785 (Washington, D.C.: Aug. 26, 2002), and Pipeline 
Safety: The Office of Pipeline Safety Is Changing How It Oversees the 
Pipeline Industry, GAO/RCED-00-128 (Washington, D.C.: May 15, 2000). 

[4] These figures are based on accidents reported to OPS. For hazardous 
liquid pipelines, they include accidents involving any fatality or 
injury, a fire or explosion, total costs of $50,000 or more, or 
releases of 50 or more barrels of hazardous liquids or 5 or more 
barrels of highly volatile liquids. For natural gas pipelines, they 
include accidents involving any fatality or injury, total costs of 
$50,000 or more, or the emergency shutdown of a liquified natural gas 
facility, as well as any accidents considered to be significant by the 
pipeline operator.

[5] In March 2002, OPS requested proposals related to damage prevention 
and leak detection. It received 82 proposals in response and, in 
November 2002, funded 7 of them. In June 2002, the agency requested 
proposals related to enhanced pipeline operations, controls, and 
monitoring. It received 57 proposals in response and, in February 2003, 
funded 3 of them, based on the availability of funding. OPS intends to 
fund 3 more of these proposals in June 2003. OPS has provided 
approximately 50 percent of the cost of the projects to awardees. In 
December 2002, the agency requested proposals related to improved 
performance of pipeline materials and other pipeline safety 
improvements. It expects to make funding decisions about these 
proposals in summer 2003. 

[6] These figures have been adjusted to account for inflation. They are 
in constant fiscal year 2003 dollars.

[7] In addition, RSPA's budget submission for fiscal year 2003 noted 
that the proposed pipeline safety R&D budget would consolidate into 
RSPA a pipeline infrastructure R&D program operated by DOE. However, 
according to DOE and OPS officials, no transfer of funding or projects 
between the two programs actually took place.

[8] These amounts represent OPS's planned expenditures in each area. 
However, the agency's actual expenditures in an area depend on the 
approval of R&D proposals received and may therefore differ from 
planned expenditures. Figures do not add to total due to rounding.

[9] "Third parties" are people or companies not associated with a 
pipeline company or its contractors. Damage to pipelines can result 
from such people or companies digging in the vicinity of buried 
pipelines without realizing that the pipelines are there. For example, 
excavating equipment can accidentally strike a pipeline and cause a 
leak or rupture, either immediately or over time, which poses a hazard 
to life and property.

[10] Another area of pipeline R&D--the development of technologies to 
support Arctic and offshore pipeline operations--was identified as a 
main area of R&D at OPS's 2001 workshop. However, the agency did not 
include this as a main area of funding in its R&D agenda because it was 
not identified as a high-priority area at the workshop and because the 
Department of the Interior's MMS funds some R&D in this area. OPS has 
recently cofunded with MMS several projects and a workshop in this 
area, at a cost of almost $148,000.

[11] Figures have been adjusted to account for inflation. They are in 
constant fiscal year 2003 dollars.

[12] Smart pigs are devices that run inside a pipeline to detect 
anomalies, such as corrosion, metal loss, or damage from excavation.

[13] Direct assessment involves several steps, including digging holes 
at intervals along a pipeline to examine suspected problem areas. In a 
notice of proposed rulemaking, OPS has proposed integrity management 
regulations for gas transmission pipelines that would allow operators 
to use direct assessment techniques. See 68 Fed. Reg. 4278, 4318 (Jan. 
28, 2003). We have previously reported on the challenges faced by OPS 
in ensuring that operators use these techniques appropriately. See GAO-
02-785. 

[14] Committee on Science, Engineering, and Public Policy, Evaluating 
Federal Research Programs: Research and the Government Performance and 
Results Act (Washington, D.C.: National Academy of Sciences, February 
1999). 

[15] Committee on Science, Engineering, and Public Policy, Evaluating 
Federal Research Programs: Research and the Government Performance and 
Results Act, 11.

[16] DOT and the Department of the Interior have a memorandum of 
understanding in place to coordinate their regulatory efforts regarding 
outer continental shelf pipelines; this memorandum states that that the 
two departments will coordinate their respective R&D projects 
concerning these pipelines. In addition, OPS and MMS have an 
interagency agreement to jointly fund R&D projects related to offshore 
pipelines. 

[17] Experts assigned a funding priority to each category and specific 
type of R&D using the following scale: 1=little or no funding, 2=some 
funding priority, 3=moderate funding priority, 4=high funding priority, 
and 5=very high funding priority. Experts could also indicate that they 
did not know or had no basis to judge the funding priority for a 
particular R&D category.

[18] We also examined results for experts from those organizations that 
had bid on OPS R&D funding in fiscal year 2002 to see how they compared 
to those of other experts who completed our questionnaire. Seven of the 
experts who completed our questionnaire are from organizations that had 
bid on OPS R&D funding within this time frame. Of these, all seven 
assigned scores of high or very high funding priority to the Damage 
Prevention and Leak Detection category; six assigned such scores to the 
Enhanced Operations, Controls, and Monitoring category; three assigned 
such scores to the Improved Materials Performance category; and four 
assigned such scores to the Arctic and Offshore Technologies category. 

[19] We have recently reported that RSPA has not fulfilled a DOT 
requirement for overseeing and developing ways to improve research 
evaluation efforts throughout the department. See U.S. General 
Accounting Office, Transportation Research: Action Needed to Improve 
Coordination and Evaluation of Research, GAO-03-500 (Washington, D.C.: 
May 1, 2003).

[20] For example, in a notice of proposed rulemaking, OPS proposed 
integrity management regulations for natural gas transmission pipelines 
that would allow pipeline operators to assess the integrity (structural 
soundness) of their pipelines using a new technique called direct 
assessment. See 68 Fed. Reg. 4278, 4318 (Jan. 28, 2003). OPS has funded 
and is currently funding R&D to develop and validate this assessment 
method. 

[21] OPS has issued requirements for hazardous liquid pipeline 
operators to develop such programs, which are aimed at assessing the 
integrity (structural soundness) of pipelines and identifying and 
addressing risks to segments where a leak or rupture could have 
significant consequences, such as near highly populated areas. See 49 
CFR § 195.452. In a notice of proposed rulemaking, the agency has 
proposed such requirements for operators of natural gas transmission 
pipelines. See 68 Fed. Reg. 4278 (Jan. 28, 2003).

[22] U.S. General Accounting Office, Highway Research: DOT's Actions to 
Implement Best Practices for Setting Research Agendas and Evaluating 
Outcomes, GAO-03-640T (Washington, D.C.: Apr. 10, 2003); Highway 
Research: Systematic Selection and Evaluation Processes Needed for 
Research Program, GAO-02-573 (Washington, D.C.: May 24, 2002); and 
DOE's Success Stories Report, GAO/RCED-96-120R (Washington, D.C.: Apr. 
15, 1996).

[23] The Government Performance and Results Act requires all federal 
agencies to measure and report on the results of their activities 
annually.

[24] U.S. General Accounting Office, Research and Development: Lessons 
Learned from Previous Research Could Benefit FreedomCAR Initiative, 
GAO-02-810T (Washington, D.C.: June 6, 2002).

[25] Committee on Science, Engineering, and Public Policy. Evaluating 
Federal Research Programs: Research and the Government Performance and 
Results Act. 

[26] See U.S. General Accounting Office, Measuring Performance: 
Strengths and Limitations of Performance Indicators, GAO/RCED-97-91 
(Washington, D.C.: Mar. 21, 1997), and Intellectual Property: Federal 
Agency Efforts in Transferring and Reporting New Technology, GAO-03-47 
(Washington, D.C.: Oct. 31, 2002). Also, see Committee on Science, 
Engineering, and Public Policy, Evaluating Federal Research Programs: 
Research and the Government Performance and Results Act. 

[27] Committee on Science, Engineering, and Public Policy, Evaluating 
Federal Research Programs: Research and the Government Performance and 
Results Act, and Implementing the Government Performance and Results 
Act: A Status Report (Washington, D.C.: National Academy of Sciences, 
2001).

[28] U.S. General Accounting Office, Federal Research: Peer Review 
Practices at Federal Science Agencies Vary, GAO/RCED-99-99 (Washington, 
D.C.: Mar. 17, 1999).

[29] Two of the individuals who responded are former officials of these 
organizations.

[30] Experts assigned a funding priority to each category and 
subcategory of R&D using the following scale: 1=little or no funding, 
2=some funding priority, 3=moderate funding priority, 4=high funding 
priority, and 5=very high funding priority. We also asked respondents 
if they wished to identify any additional R&D categories and, if so, 
what score they would assign to these categories. 

[31] Of the 49 experts who completed our questionnaire, we identified 7 
from organizations that had submitted R&D proposals in response to 
announcements issued by OPS in March and June 2002, based on 
information provided by OPS. Of these 7, 5 were from organizations that 
received funding from OPS. In December 2002, OPS issued another 
announcement soliciting R&D proposals. However, because OPS has not yet 
made funding decisions about proposals received in response to this 
announcement, officials preferred not to provide us with information 
about these proposals. 

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: