This is the accessible text file for GAO report number GAO-06-623 
entitled 'United States Coast Guard: Improvements Needed in Management 
and Oversight of Rescue System Acquisition' which was released on June 
1, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 

GAO: 

May 2006: 

United States Coast Guard: 

Improvements Needed in Management and Oversight of Rescue System 
Acquisition: 

Coast Guard's Rescue System: 

GAO-06-623: 

GAO Highlights: 

Highlights of GAO-06-623, a report to congressional committees. 

Why GAO Did This Study: 

Search and rescue—one of the United States Coast Guard’s (USCG) oldest 
missions and highest priorities—involves minimizing loss of life, 
injury, and property damage by aiding people and boats in distress. In 
September 2002, USCG contracted to replace its search and rescue 
communications system—installed in the 1970s—with a new system known as 
Rescue 21. However, the acquisition and initial implementation of 
Rescue 21 has resulted in significant cost overruns and schedule 
delays. 

GAO’s objectives in reviewing the Rescue 21 program were to (1) assess 
the reasons for the significant cost overruns and implementation 
delays; (2) evaluate the viability of the revised cost and schedule 
estimates; and (3) evaluate the impact of the implementation delays. 

What GAO Found: 

Key factors that contributed to Rescue 21 cost overruns and schedule 
delays were inadequacies in requirements management, project 
monitoring, risk management, contractor cost and schedule estimation 
and delivery, and executive-level oversight (see table). Accordingly, 
the estimated total acquisition cost for Rescue 21 has increased from 
$250 million in 1999 to $710.5 million in 2005, and the timeline for 
achieving full operating capability has been delayed from 2006 until 
2011. USCG officials agreed that improvements need to be made to the 
management of the Rescue 21 project, and they are taking steps to 
address some of these issues. 

USCG’s current acquisition cost estimate of $710.5 million is not 
viable. Our analysis of contractor performance trends indicates that 
additional overruns will likely bring the total acquisition cost to 
$872 million, unless critical changes are made. Additionally, USCG’s 
schedule is uncertain due to on-going contract item renegotiations and 
pending decisions regarding vessel asset tracking functionality. 
Finally, further cost increases may result from the variability of 
costs for tower preparation and construction. To improve its current 
cost and schedule estimates, USCG plans to complete an integrated 
baseline review after contract items are renegotiated; however, a date 
for completing this review has not been established. 

The delays in deploying Rescue 21 could affect sites awaiting 
modernization. For example, key functionality, such as improved 
direction finding and improved coverage, will not be available for a 
significant period of time, and legacy equipment may fail, requiring 
costly repairs and upgrades to address coverage gaps and other 
operational issues. 

Table: Key Factors in Rescue 21 Cost Overruns and Schedule Delays: 

Requirements management: 
USCG did not follow a rigorous requirements management process and 
testing revealed incomplete and poorly defined requirements. 

Project monitoring: 
USCG did not effectively use earned value management data to measure 
performance and take corrective action on negative trends. 

Risk management: 
USCG did not always effectively mitigate and communicate risks. 

Contractor cost and schedule estimation and delivery: The contractor 
created a schedule that underestimated the time required to complete 
key tasks, and development took longer than planned, which led to 
delays in testing. 

Executive-level oversight: 
USCG stated that it had an executive-level oversight process that 
included semiannual and key decision point reviews. However, there is 
no evidence that these reviews of Rescue 21 occurred before 2005. 

[End of Table] 

What GAO Recommends: 

GAO recommends that the Commandant of USCG ensure that executive-level 
management oversees Rescue 21’s progress toward cost and schedule 
milestones and manages risks; establishes milestones to complete an 
integrated baseline review; and develops revised cost and schedule 
estimates. The Department of Homeland Security agreed with the 
recommendations and has begun to implement them; however, the 
department expressed concerns with selected aspects of the report. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-623]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact David A. Powner at (202) 
512-9286 or pownerd@gao.gov or Stephen L. Caldwell at (202) 512-9610 or 
caldwells@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Briefing Slides: 

Appendix II: Comments from the Department of Homeland Security: 

Appendix III: GAO Contacts and Staff Acknowledgments: 

Abbreviations: 

DHS: Department of Homeland Security: 
EVM: earned value management: 
NDRS: National Distress and Response System: 
OMB: Office of Management and Budget: 
USCG: United States Coast Guard: 

United States Government Accountability Office: 

Washington, DC 20548: 

May 31, 2006: 

The Honorable Harold Rogers: 
Chairman: 
The Honorable Martin Sabo: 
Ranking Member: 
Subcommittee on Homeland Security, 
Committee on Appropriations: 
House of Representatives: 

The Honorable Olympia Snowe: 
Chairman: 
The Honorable Maria Cantwell: 
Ranking Member: 
Subcommittee on Fisheries and Coast Guard, 
Committee on Commerce, Science and Transportation: 
United States Senate: 

The United States Coast Guard (USCG) is currently replacing its 30- 
year-old search and rescue communications systems with a new system 
known as Rescue 21. Rescue 21 will be used not only for search and 
rescue, but it will support other USCG missions, including those 
involving homeland security. However, Rescue 21 has had significant 
cost overruns and schedule delays. As a result of these delays, 
completion of Rescue 21 deployment nationwide--originally scheduled for 
fiscal year 2006--has been delayed until fiscal year 2011, and its 
total acquisition cost has risen from $250 million in 1999 to its 
current estimate of $710.5 million.[Footnote 1] In addition to the cost 
overruns and schedule delays, Rescue 21 may not be deployed with all of 
the functionality originally promised. For example, functionality for 
vessel asset tracking was not deployed as part of the initial operating 
capability due to the system's inability to meet original requirements, 
and USCG is now considering alternative strategies for this 
functionality. Also, there has been a reduction in promised 
improvements to limit communications gaps; originally, Rescue 21 was 
intended to limit communications gaps to 2 percent--now the target is 
less than 10 percent. 

This report responds to your request that we (1) assess the reasons for 
the significant implementation delay and cost overruns against the 
original Rescue 21 proposal, (2) evaluate the viability of the USCG's 
revised cost and implementation schedule, and (3) evaluate the impact 
of Rescue 21's implementation delay on USCG's field units that are 
awaiting the modernization of antiquated communications equipment. 

On March 27, 2006, we provided your staff with a briefing on the 
results of our study. The slides from that briefing are included in 
this report as appendix I. The purpose of this report is to formally 
publish the briefing slides and officially transmit our recommendations 
to the Commandant of USCG. 

To assess the reasons for the implementation delay and cost overruns of 
Rescue 21, we analyzed current and previous program documents, 
including current and previous acquisition plans, cost estimates, 
status reports, schedules, promised functionality, earned value 
management (EVM)[Footnote 2] data, and cost analyses. We also 
interviewed key officials and contractors. To evaluate the viability of 
the USCG's revised cost estimate and implementation schedule, we 
analyzed program cost and schedule documents, cost estimating 
methodologies, and changes to development and deployment plans. To 
identify factors that could affect the program baseline in the future, 
we assessed the prime contractor's performance related to cost and 
schedule. We applied established earned value analysis techniques to 
data from contractor cost performance reports that spanned a 26-month 
period from October 2003 to December 2005 to show trends in cost and 
schedule performance and to estimate the likely costs at the completion 
of the prime contract. To evaluate the impact of Rescue 21's 
implementation delay, we reviewed reports on legacy system failures and 
anticipated failures and interviewed field officials responsible for 
overseeing legacy system operations at 11 field units, the Maintenance 
and Logistic Commands for the Pacific and Atlantic Areas, and Rescue 21 
users at the two initial operating regions. 

We performed our work at USCG headquarters and at selected field units 
and communications centers from December 2005 through March 2006, in 
accordance with generally accepted auditing standards. 

In summary, we made the following major points in our briefing: 

* Key factors that contributed to Rescue 21 cost overruns and schedule 
delays were shortcomings in requirements management, project 
monitoring, risk management, contractor cost and schedule estimation 
and delivery, and executive-level oversight. Among the points we made 
were that USCG did not effectively use EVM data to measure performance 
and take corrective actions; USCG, while identifying, analyzing, and 
prioritizing risks, did not always mitigate and communicate risks; and 
USCG stated that it had an executive oversight process that included 
semiannual and key decision point reviews. However, there is no 
evidence that these reviews of Rescue 21 occurred before 2005. USCG 
agreed that improvements need to be made in the management of the 
Rescue 21 project, and it is taking steps to address some of these 
issues. 

* The USCG's current acquisition cost estimate of $710.5 million is not 
viable: based on our analysis of contractor performance trends and the 
renegotiation of 91 contract items for site deployment, we forecast a 
cost overrun of approximately $161 million, which will likely bring 
total acquisition costs to $872 million, unless critical changes are 
made. Also, the USCG's schedule is uncertain because of the continuing 
contract item renegotiations and decisions regarding vessel asset 
tracking functionality. Finally, these additional factors may increase 
costs and delay schedules: (1) the variability of costs for tower 
preparation and construction and (2) site reorganization.[Footnote 3] 
USCG realizes that it needs to improve its cost and schedule management 
and has plans to complete an integrated baseline review for the 
contract items as they are renegotiated; however, a date for completing 
this review has not been established. 

* The delays in deploying Rescue 21 could have several effects: (1) key 
functionality, such as improved direction finding and improved 
coverage, will not be available for a significant period of time; (2) 
legacy equipment may fail, requiring costly repairs; and (3) legacy 
equipment may require upgrades to address coverage gaps and other 
operational issues. For example, USCG instituted a moratorium on 
upgrades to the legacy system because Rescue 21 was to replace the 
equipment. However, despite the moratorium, according to USCG 
officials, some field units have upgraded their legacy systems to 
address operational challenges. Upgrades such as these would result in 
higher costs, especially at field units scheduled for implementation at 
a later date. 

Recommendations for Executive Action: 

To more effectively manage the remaining development and deployment of 
Rescue 21, we recommend that the Commandant of USCG ensure that USCG 
executive-level management implements the following three 
recommendations: 

* Oversee the project's progress toward cost and schedule milestones 
and management of risks. 

* Establish a milestone to complete Rescue 21's integrated baseline 
review, to include all renegotiated contract items. 

* Use the results of this baseline review to complete a revised cost 
and schedule estimate. 

Agency Comments and Our Evaluation: 

In providing written comments on a draft to this report, the Department 
of Homeland Security's (DHS) Director of the Departmental GAO/Office of 
the Inspector General Liaison Office concurred with our recommendations 
and stated that USCG would continue to aggressively pursue the 
recommendations in the GAO report in order to ensure that Rescue 21 
provides critical life-saving capabilities and that the acquisition is 
effectively managed. DHS stated that it was currently addressing our 
recommendations and provided examples of efforts to date. Specifically, 
DHS and USCG increased oversight with quarterly flag officer/vice 
president-level program reviews and periodic DHS Joint Requirements 
Council and Investment Review Board reviews in October 2005 and plan to 
initiate monthly progress reviews of the project schedule beginning in 
June 2006. 

Regarding the cost of Rescue 21, DHS stated that it had recently 
revised its acquisition program baseline. The revised total acquisition 
cost for Rescue 21 is $730.2 million, which was achieved by adjusting 
several performance parameters and terminating the contractor's option 
to provide vessel subsystem components. DHS plans to achieve this 
capability through other means. However, we continue to stress the 
importance of our recommendation that calls for the completion of an 
integrated baseline review. If performed correctly, an integrated 
baseline review will provide insight into the risks associated with the 
$730.2 million estimate and the extent to which the estimate is 
achievable. 

DHS agreed that USCG has responsibilities for managing schedule delays 
and cost overruns in major acquisition projects and with our findings 
regarding the contractor's inability to manage cost and schedule. DHS 
stated that the root cause for the systems deployment delays and costs 
overruns was poor estimates and failed performance by the contractor. 
The department, however, raised concerns with four of our findings 
regarding USCG management and oversight and provided us with some 
additional information regarding these findings that did not fully 
address our concerns. Therefore, we continue to believe that 
shortcomings in USCG's management and oversight of Rescue 21 
contributed to cost overruns and schedule delays. USGC concerns and our 
response associated with each of the four areas follows. 

First, DHS stated that our findings on requirements management are not 
consistent with our previously reported 2003 findings that USCG did 
have a process for managing requirements and it cited various 
requirements management documents. Our current finding states that USCG 
did not follow rigorous requirements management processes for Rescue 21 
not that the USCG did not have a process and is therefore consistent 
with our previous report. DHS also stated that testing revealed that 
requirements were clearly defined. However, as stated in our briefing, 
a USCG usability and operability assessment of Rescue 21 stated that 
most of the operational advancements envisioned for the project were 
not achieved, and these problems could have been avoided by including 
user requirements in the contract. DHS's response also cites software 
and hardware defects that can be attributed to shortcomings in 
requirements management, providing further support for our finding that 
USCG did not follow a rigorous requirements management process. 

Second, DHS disagreed with our finding that they did not effectively 
use EVM data to measure performance and take corrective actions. 
Evidence showing USCG actions to address schedule delays from EVM data 
was not provided to us during our review. On May 8, 2006, 6 weeks after 
the briefing on March 27, 2006, DHS provided documentation showing that 
some actions were taken by USCG in 2004. However, this is nearly a year 
after the EVM data began showing schedule slippages in March 2003 and 
after the Rescue 21 project missed its initial operating capability 
date of September 2003. Further, DHS stated that the contractor 
undertook a significant reorganization of its management structure in 
response to its concerns related to cost and schedule performance. 
Because this information was not mentioned to us during our discussions 
with either USCG or the contractor during our review, on May 15, 2006, 
we contacted both the contractor's Rescue 21 project manager and 
contract manager to clarify the reorganization statement. These 
individuals told us that changes in contracting personnel were due to 
internal movement between ongoing contracts and did not constitute a 
reorganization. 

Third, DHS disagreed with our finding that risks were not always 
mitigated and communicated. DHS stated that there were various reports 
and meetings where risks were reported and discussed. Documentation for 
some of these reports and meetings was provided to us on May 8, 2006. 
Furthermore, as stated in the briefing, there were examples of risks 
that were not effectively mitigated or communicated. The Human System 
Interface (HSI) work group referred to in the agency's response was 
established only in 2005, after Rescue 21 had shown significant HSI 
issues in its final operational testing and evaluation phase. Effective 
risk management would have identified and addressed risks earlier in 
the system life cycle. 

Finally, DHS disagreed with our finding that there was no evidence to 
support its statement that there was executive-level management 
oversight prior to 2005. During our review, DHS cited a list of 
semiannual and key decision point reviews, dating from 2002 to 2005; 
however, as stated in our briefing, it did not provide evidence of the 
pre-2005 briefings. On May 8, 2006, DHS provided some evidence that 
these meetings occurred; however, based on our review of the 
documentation, it was unclear which executives attended these meetings 
and, therefore, whether the appropriate executives were in attendance. 
Furthermore, the department did not provide evidence showing executive 
direction to mitigate risks or address problems. Finally, the 
department stated that it had increased oversight in October 2005 by 
establishing quarterly flag officer/vice president-level program 
reviews and periodic DHS Joint Requirements Council and Investment 
Review Board reviews. This additional oversight provides further 
support for our finding on the lack of executive level oversight prior 
to 2005. DHS also challenged our finding that executive-level 
monitoring of the risk mitigation activities that resulted from the 
2005 meetings did not occur, citing various monitoring activities that 
occurred. However, it did not provide evidence of the monitoring 
activities. 

DHS also provided technical comments that were incorporated in this 
report where appropriate. DHS's written comments are reprinted in 
appendix II. 

We are sending copies of this report to the Secretary of Homeland 
Security, the Commandant of the United States Coast Guard, and other 
interested parties. We will make copies available to others on request. 
In addition, the report will be available at no charge on the GAO Web 
site at [Hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact David A. Powner at (202) 512-9286 or by e-mail at 
pownerd@gao.gov or Stephen L. Caldwell at (202) 512-9610 or by e-mail 
at caldwells@gao.gov. 

Signed by: 

David A. Powner: 
Director, Information Technology Management Issues: 

Signed by: 

Stephen L. Caldwell: 
Acting Director, Homeland Security and Justice Issue: 

[End of section] 

Appendix I: Briefing Slides: 

Management and Oversight Improvements Needed for Coast Guard's Rescue 
21 Acquisition: 

Briefing for: 

the House Committee on Appropriations, Subcommittee on Homeland 
Security: 

and: 

the Senate Committee on Commerce, Science and Transportation, 
Subcommittee on Fisheries and Coast Guard: 

March 27, 2006: 

Briefing Overview: 

Introduction: 

Objectives, Scope, and Methodology Results in Brief: 

Background: 

Objective 1: Reasons for Cost Overruns and Schedule Delays Objective 2: 
Revised Cost and Schedule Estimates: 

Objective 3: Delay in Implementation Conclusions: 

Recommendations: 

Agency Comments: 

Introduction: 

Search and rescue is one of the United States Coast Guard's (USCG) 
oldest missions and highest priorities. The search and rescue mission 
includes minimizing the loss of life, injury, and property damage by 
aiding people and boats in distress. 

The National Distress and Response System (NDRS) is the legacy 
communications component of USCG's search and rescue program. However, 
the 30-year-old system has several deficiencies and is difficult to 
maintain, according to USCG officials. In September 2002, the USCG 
contracted with General Dynamics Decision Systems (General Dynamics) to 
modernize and replace the NDRS with a system called Rescue 21. 

In October 2005, due to significant cost increases and schedule delays 
associated with the Rescue 21 program, the Chairman and Ranking Member 
of the House Appropriations Subcommittee on Homeland Security requested 
that we evaluate the Rescue 21 program. 

Objectives, Scope, and Methodology: 

As agreed, our objectives were to: 

* assess the reasons for the significant implementation delay and cost 
overruns against the original Rescue 21 proposal; 

* evaluate the viability of the USCG's revised cost and implementation 
schedule; and: 

* evaluate the impact of Rescue 21's implementation delay on the USCG's 
field units that are awaiting the modernization of antiquated 
communications equipment. 

To assess the reasons for the implementation delay and cost overruns of 
Rescue 21, we analyzed Rescue 21 documents, including current and 
previous acquisition plans, cost estimates, status reports, schedules, 
promised functionality, earned value management (EVM)[Footnote 4] data, 
and cost analyses. We also interviewed key officials and contractors 
involved in the development and deployment of Rescue 21. 

To evaluate the viability of USCG's revised cost estimate and 
implementation schedule, we analyzed current program cost and schedule 
estimates, cost estimating methodologies, and changes to development 
and deployment plans. To identify factors that could affect the program 
baseline in the future, we assessed the prime contractor's performance 
related to cost and schedule. We applied established earned value 
analysis techniques to data from contractor cost performance reports 
that spanned a 26-month period, from October 2003 to December 2005, to 
show trends in cost and schedule performance and to estimate the likely 
costs at the completion of the prime contract. 

To evaluate the impact of Rescue 21's implementation delay, we reviewed 
reports on legacy system failures and anticipated failures as well as 
documentation from USCG sites regarding problems associated with the 
system. We also interviewed field officials responsible for overseeing 
legacy system operations at 11 field units. We visited five of these 
field units (Humboldt, California; San Francisco, California; New 
Orleans, Louisiana; Mobile, Alabama; and Miami, Florida) and 
interviewed officials by telephone at six other locations (Honolulu, 
Hawaii; Guam, Territory of Guam; Detroit, Michigan; Valdez, Alaska; 
Kodiak, Alaska; and Juneau, Alaska). These locations were selected 
based on coverage gaps previously identified and their significance to 
search and rescue and homeland security operations. 

We also interviewed Maintenance and Logistic Commands for the Pacific 
and Atlantic Areas and Rescue 21 users at the initial operating sites 
at the Atlantic City and Eastern Shore communications centers. We 
compared planned Rescue 21 functionality with what was actually 
delivered. 

We performed our work at USCG headquarters and selected field units and 
communications centers from December 2005 through March 2006, in 
accordance with generally accepted auditing standards. 

Results in Brief: 

Key factors that contributed to Rescue 21 cost overruns and schedule 
delays were shortcomings in requirements management, project 
monitoring, risk management, contractor cost and schedule estimation 
and delivery, and executive-level oversight. USCG agreed that 
improvements need to be made in these areas and is taking steps to 
address these issues. 

The Coast Guard's current acquisition cost estimate of $710.5 million 
is not viable: our analysis of contractor performance trends indicates 
that additional cost overruns will likely bring total acquisition costs 
to $872 million unless critical changes are made. This includes a 
significant number of contract items that have not been completed as 
planned and must now be renegotiated. Also, USCG's schedule, which now 
shows full operating capability in 2011--5 years behind the originally 
scheduled date of 2006 is not finalized because of the continuing 
contract item renegotiations and decisions regarding vessel asset 
tracking functionality. Finally, these additional factors may increase 
costs: (1) the variability of costs for tower preparation and 
construction and (2) site reorganization.[Footnote 5] USCG realizes 
that it needs to be more effective in managing costs and schedule and 
has plans to complete an integrated baseline review for the contract 
items as these are renegotiated; however, a date for completing this 
review has not been established. In addition, USCG plans to begin 
monthly reviews of the project schedule and to incrementally deploy to 
future Rescue 21 sites. 

The delays in deploying Rescue 21 could have several effects: 

* key functionality, such as improved direction finding and improved 
coverage, will not be available for a significant period of time; 

* legacy equipment may fail, requiring costly repairs; and: 

* legacy equipment may require upgrades to address coverage gaps and 
other operational issues. 

We are recommending that the Secretary of Homeland Security direct the 
Commandant of the United States Coast Guard to ensure that executive- 
level management oversees the project's progress toward cost and 
schedule milestones and manages risks. In addition, we recommend that 
executive-level management establish a milestone to complete its 
integrated baseline review and revise its cost and schedule estimates. 

In providing oral comments on a draft of this briefing, the Coast Guard 
Assistant Commandant for Acquisition stated that the briefing fairly 
represented Rescue 21 risks and ongoing activities to address program 
management and other issues. He also stated that this briefing would be 
used as input to take corrective actions to improve program management. 

Background: 

Rescue 21 Description: 

The NDRS is USCG's primary VHF communications system, important to the 
USCG's search and rescue program as well as its homeland security 
mission. However, the USCG has reported that NDRS-much of which was 
installed in the 1970s--has several critical deficiencies: 

* numerous gaps in communications coverage-in some areas the Coast 
Guard cannot hear calls from mariners in distress or communicate with 
other USCG vessels; 

* limited direction-finding capabilities and no digital selective 
calling capabilities;[Footnote 6] no effective way to track USCG assets 
that conduct search and rescue; 

* limited interoperability with other federal and state communications 
systems; and: 

* no means for protected communication. 

Rescue 21's primary functions are to provide USCG with communication 
for maritime search-and-rescue functions-such as monitoring distress 
calls from boaters and coordinating the response or rescue. It is 
intended to replace NDRS, and will be a short-range communications 
system reaching about 20 nautical miles. The system will include VHF-FM 
radios, communications towers, and hardware and software at USCG sites 
as well as on Coast Guard vessels. This system also, in addition to 
search and rescue, provides support to homeland security missions. 
USCG's acquisition directorate, under the Assistant Commandant for 
Acquisition, is responsible for managing the Rescue 21 acquisition. 

In September 2002, USCG awarded an indefinite delivery, indefinite 
quantity[Footnote 7] contract to General Dynamics for developing and 
implementing Rescue 21. The contract and original schedule called for 
the following: 

* two initial Coast Guard regions to operate the new system with full 
functionality-referred to as initial operating capability-scheduled for 
September 2003 and: 

* continued deployment to additional locations through 2006. 

According to the Coast Guard contract with General Dynamics, the Rescue 
21 system is to: 

* allow continuous, uninterrupted communications on Channel 
16;[Footnote 8]: 

* limit the number of communication coverage gaps to less than 10 
percent in the United States (originally this was to less than 2 
percent); 

* provide direction finding and digital selective calling to locate 
boaters; * allow USCG to track its mobile assets, such as vessels; 

* allow communication with other federal and state systems; 

* allow protected communication of sensitive information; and: 

* archive voice and data communications and make them instantly 
accessible. 

A table showing the capabilities of NDRS and Rescue 21 follows. 

Table: Comparison of Rescue 21 Capabilities with Legacy System: 

[See PDF for Image] 

Source: U.S. Coast Guard. 

[End of table] 

Background: 

Rescue 21 Components: 

Key components of Rescue 21-some of which appear in the following 
diagram-are: 

* communications centers (46) that contain operator workstations and 
electronic equipment; 

* antenna towers (322); 

* other equipment and/or infrastructure that could be used on vessels 
or remote facilities, including communication lines and radio 
equipment; and: 

* disaster recovery assets (not pictured). A disaster recovery asset is 
a smaller antenna tower and equipment similar to a remote facility. It 
is portable so that it can be moved to locations as needed. 

Group communications center processes distress signal and directs 
rescue: 

High level site towers determine direction of signal: 

Vessel sends distress signal: 

[See PDF for Image] 

Source: GAO analysis of U.S. Coast Guard data. 

[End of figure] 

Background: 
Previous GAO Work: 

In September 2003, we reported that USCG's original schedule for 
reaching initial operating capability by September 2003 had been 
postponed and that it had yet to finalize a new schedule.[Footnote 9] 
This postponement was due, in part, to the development of the system 
taking longer than planned. 

We also determined that USCG postponed key tests and the initial 
operating capability because of the delays in software development, and 
we highlighted risks associated with its plan to compress and overlap 
key tests as a result of the delays. In addition, schedules for key 
tests and deliverables, including test plans, were still not complete. 

We recommended that the Secretary of the Department of Homeland 
Security direct the Commandant of the United States Coast Guard to: 

* establish a new schedule for critical testing phases and initial 
operating capability and * ensure that milestones were established for 
completing test plans for formal qualifications testing, system 
integration testing, and operational testing and evaluation and that 
test plans address all requirements for the system. 

Following our report, key test schedules were revised and milestones 
and test plans were developed. 

Rescue 21 Schedule Delays: 

However, further delays occurred during testing. [Footnote 10] For 
example, 

Formal qualification testing, originally expected to begin in February 
2003 and end in March 2003, actually began in January 2004 and ended in 
June 2004--15 months behind schedule. At that time, only 4 of the 235 
allocated requirements were validated and performance issues in several 
areas, including fault management, archive/restore, and channel 
performance, were found that required additional work. Formal 
qualification testing was repeated in June 2004, at which time the 
remaining 231 allocated requirements were validated. 

System integration testing, originally expected to begin in April 2003 
and end in May 2003, actually began in August 2004 and ended in 
December 2004--19 months behind schedule. Additional software defects 
were found during this time and USCG decided to have the contractor 
concurrently fix the system integration testing problems while going 
into the operational test and evaluation phase. 

Operational test and evaluation, originally intended to begin in July 
2003 and to end in August 2003, actually began in January 2005 and 
ended March 2005-19 months behind schedule. Further software defects 
were found, and USCG also requested additional changes to the human 
systems interface based on operator comments. The changes to the human 
systems interface were prioritized into three builds, the first of 
which was required by USCG before it would approve initial operating 
capability. The remaining two builds are being negotiated with the 
contractor and represent additional development. Finally, it was 
determined that the vessel portion of Rescue 21 did not meet 
performance requirements. 

Because of the delays in development and testing, the date for initial 
operating capability was delayed from September 2003 to December 2005. 
Full operating capability, which was originally scheduled for fiscal 
year 2006, was delayed to fiscal year 2011. 

Figure: Development and Implementation Schedule: 

[See PDF for Image] 

[End of Figure] 

Rescue 21 Cost Growth: 

Rescue 21's total acquisition cost estimate has increased from $250 
million to $710.5 million over a 6-year period, as shown in the 
following chart. 

[See PDF for image] 

Source: U.S. Coast Guard Acquisition Project baseline data. 

[End of figure] 

Rescue 21 Appropriations: 

Rescue 21 has been appropriated a total of $382.9 million for 
development since award of the contract in fiscal year 2002. 

[See PDF for image] 

Source: U.S. Coast Guard data. 

[End of figure] 

Rescue 21 Current Status: 

Rescue 21 was deployed on December 20, 2005, with an initial operating 
capability[Footnote 11] and is being used at two regions (Atlantic 
City, New Jersey, and Chincoteague, Virginia). The deployment included 
all the station functionality and equipment that: 

* allowed continuous, uninterrupted communications on Channel 16; 

* limited the number of communication coverage gaps; 

* provided direction finding and digital selective calling to locate 
boaters; 

* allowed radio communication with other federal and state officials; 

* allowed protected communication of sensitive information; and: 

* archived voice and data communications and made them accessible in 
real time. 

However, functionality for vessel asset tracking and data transfer plus 
an expanded vessel communications suite was not delivered as planned. 
This was deferred because the contractor could not meet the original 
contract requirements for simultaneous communications. In January 2006, 
a stop work order was issued on the vessel tracking and data transfer 
functionality for 90 days. USCG and the Department of Homeland Security 
(DHS) are currently determining whether Rescue 21 should complete 
development of the vessel functionality or consider other sources for 
that functionality. 

The current schedule calls for the deployment of Rescue 21 to four low- 
rate initial production sites during 2006.[Footnote 12] In addition, 
according to USCG officials, they are currently revalidating 
implementation plans for the first 15 of the remaining 40 sites. These 
first 15 of the remaining 40 sites are approved for deployment and are 
currently scheduled to be completed by fiscal year 2009. Completion of 
the remaining 25 sites with full operating capability is currently 
scheduled for fiscal year 2011. 

However, some development activity remains. First, there are two 
remaining builds of enhancements to the human systems interface, with 
the second build scheduled for completion in October 2006 and the third 
in March 2007. Second, a decision is pending on how to implement vessel 
asset tracking and data transfer functionality. 

Objective 1: 

Reasons for Cost Overruns and Schedule Delays: 

Key factors that contributed to Rescue 21 cost overruns and schedule 
delays were shortcomings in requirements management, project 
monitoring, risk management, contractor cost and schedule estimation 
and delivery, and executive-level oversight. USCG agreed that 
improvements need to be made in these areas and is taking steps to 
address these issues. 

To effectively manage major IT programs, organizations should use sound 
acquisition and management practices to minimize risks and thereby 
maximize a project's ability to be delivered on time, within budget, 
and with promised functionality. Such practices have been identified by 
leading organizations, such as the Software Engineering Institute, the 
Chief Information Officer's Council, and in our prior work analyzing 
best practices in industry and government. 

Requirements management: 

Requirements management includes processes for eliciting, documenting, 
verifying and validating, and managing requirements through the system 
life cycle. 

The Coast Guard did not have a rigorous requirements management 
process. Although the USCG developed high-level requirements for Rescue 
21, it relied solely on the contractor to manage these requirements. 
According to USCG acquisition officials, they took this approach 
because of the performance-based contract vehicle. However, our 
experience in reviewing major systems acquisitions has shown that it is 
important for government organizations to exercise strong leadership in 
managing requirements, regardless of the contracting vehicle. 

In addition to not effectively managing requirements, testing revealed 
numerous problems linked back to incomplete and poorly defined user 
requirements. For example: 

* A USCG usability and operability assessment of Rescue 21 stated that 
most of the operational advancements envisioned for the system were not 
achieved and concluded that these problems could have been avoided by 
including user requirements in the contract. 

Requirements management (continued): 

* A key requirement stated, "Provide a consolidated regional geographic 
display." The contractor provided a capability based on that statement, 
but during testing the USCG operators felt the maps did not display 
sufficient detail. Such discrepancies led to an additional statement of 
work defining required enhancements to the system interface (e.g., 
screen displays). 

After these initial problems, USCG recognized the need for an orderly 
process to review changes that could have a significant impact to cost, 
schedule, and performance. Specifically, in October 2005, USCG 
established a Configuration Control Board to control changes to Rescue 
21 requirements. 

Project monitoring: 

A key project monitoring activity uses earned value management (EVM) 
techniques to track the progress of work completed against project cost 
and schedule estimates. 

USCG did not effectively use EVM data in order to measure performance 
and take corrective action. The Coast Guard has received EVM cost 
performance reports that identified problems on a regular basis from 
Booz Allen Hamilton.[Footnote 13] However, USCG did not take action on 
these reports. For example, the March 2003 report indicated that, given 
the number of slipped tasks and small remaining time, it was unlikely 
that the contractor would be able to recover the schedule deadlines; 
the September and October 2003 reports described the contractor's 
performance as rapidly worsening. Nevertheless, USCG did not take 
action in response to these reports. 

Risk management: 

An effective risk management process identifies potential problems 
before they occur, so that risk-handling activities may be planned and 
invoked as needed across the life of the product and project in order 
to mitigate adverse impacts on achieving objectives. Key activities 
include identifying and analyzing risks, assigning resources, 
developing risk mitigation plans and milestones for key mitigation 
deliverables, briefing senior-level managers on high-priority risks, 
and tracking risks to closure. 

USCG’s management process includes identifying, analyzing, and 
prioritizing risks; establishing mitigation strategies; and assigning 
responsibility for addressing risks. However, USCG did not always 
mitigate and communicate risks. For example: 

In 2003, USCG identified software development as a high risk to Rescue 
21’s schedule. However, despite early acknowledgment of this risk, it 
was not mitigated and ultimately became one of the primary drivers for 
the increase in costs. 

USCG did not communicate risks to partners when applicable. During our 
review, the contractor stated that it was unaware of the items in the 
USCG risk list that related to the contractor’s performance.

Contractor cost and schedule estimation and delivery: 

The contractor underestimated the time for development and testing of 
Rescue 21 and established an integrated project schedule that it could 
not meet. 

* Development took longer than planned, which led to delays in testing. 
Subsequently, testing phases were not completed on schedule. 

* Additionally, the initial estimate for completing a regional site 
deployment was 12 months. The new estimate of 24 months is considered 
to be more realistic for executing the same tasks. 

The contractor did not always provide personnel in a timely manner or 
with the knowledge and skills required to complete the project. 

* According to USCG information, it took longer than expected to obtain 
developers and subcontractors for Rescue 21. 

* According to USCG documentation, support personnel involved in 
initial site surveys were minimally qualified and had limited civil 
engineering and environmental analysis skills. 

Executive oversight and governance: 

Office of Management and Budget and GAO guidance call for agencies to 
establish IT investment management boards comprised of key executives 
to regularly track the progress of major systems acquisitions. These 
boards should have documented policies and procedures for management 
oversight of IT projects and systems and should be able to adequately 
oversee the project's progress toward cost and schedule milestones and 
their risks. The board should also employ early warning systems that 
enable it to take corrective actions at the first sign of cost, 
schedule, and performance slippages. 

USCG told us that it had an executive oversight process that included 
semi-annual and key decision point reviews. However, there is no 
evidence that these reviews of Rescue 21 occurred before 2005. 

Rescue 21 oversight meetings commenced in 2005, but executive 
management did not take action in response to the risks and problems 
presented. For example, briefings given to the USCG Acquisition Review 
Council and the Department of Homeland Security (DHS) Investment Review 
Board for key decision point 3 approval identified numerous high-level 
and moderate risks and DHS officials requested additional actions, but 
executive-level monitoring to ensure the adequacy and completion of 
risk mitigation activities did not occur. 

Recent Efforts by USCG to Improve Rescue 21 Management: 

USCG has recently begun to address several of the key factors that led 
to cost overruns and schedule slippages. 

According to the Assistant Commandant for Acquisition, the USCG plans 
to improve the capabilities of the Acquisition Office by obtaining more 
technical assistance to help manage requirements, monitor project 
performance, and mitigate risks, among other things. For example, USCG 
intends to begin monthly reviews of the project schedule in June 2006 
to track actual against planned milestones and to use earned value 
management data to improve its oversight of cost and schedule 
performance. 

Regarding contractor performance, USCG stated that it was difficult to 
manage the project effectively from headquarters and that the distance 
between USCG and the contractor delayed decisions. As a result, in 
November 2005, USCG established a Project Resident Office adjacent to 
the contractor's facility. This office's responsibilities include 
overseeing nationwide regional site implementation; monitoring 
contractor performance; managing risks; establishing closer 
communication and coordination with the contractor to minimize contract 
changes and control cost growth; and monitoring shipping and delivery 
of Rescue 21 equipment. 

USCG has also increased the visibility of Rescue 21 at the executive 
level by establishing quarterly meetings between the USCG Assistant 
Commandant for Acquisition and the vice president of the contractor. 
These meetings may facilitate greater executive-level involvement in 
Rescue 21. 

Revised Cost and Schedule Estimates: 

Coast Guard's current acquisition cost estimate of $710.5 million is 
not viable. Our analysis of contractor performance trends indicates 
that there will likely be additional cost overruns that will bring the 
total acquisition cost to $872 million unless critical changes are 
made. This includes a significant number of contract items that have 
not been completed as planned and must now be renegotiated. Also, 
USCG's schedule is uncertain and will likely change due to the 
continuing contract item renegotiations and decisions regarding vessel 
functionality. Negotiations for contract items for the first 15 full- 
rate production regions continues and is expected to be completed by 
June 2006. Subsequently, contract items to deploy Rescue 21 to the 
remaining 25 full-rate production regions must also be renegotiated. 

Finally, additional factors may increase costs, such as the variability 
of costs for tower preparation and construction and site 
reorganization.[Footnote 14] 

As of June 2005, the current acquisition cost estimate to reach full 
operational capability in 2011 for Rescue 21 is $710.5[Footnote 15] 
million. 

Based on our analysis of contractor performance trends (using 
contractor EVM data [Footnote 16]), we forecast a cost overrun of 
approximately $161 million. The total acquisition costs could be as 
high as $872 million. 

The current cost estimate does not take into account contract items 
that are being renegotiated. There are 91 contract items for site 
deployment that have expired and have to be renegotiated. According to 
USCG, based on their experience with the cost overruns at the IOC 
sites, costs for each of the contract item listed below will increase: 

* regional implementation plans: 

* regional infrastructure preparation: 

* regional system installation: 

Coast Guard officials agreed with our analysis of projected cost 
overruns. 

USCG's schedule is also uncertain and will likely change. 

The current project schedule is uncertain because of the 91 contract 
items that must be renegotiated. USCG estimates that this renegotiation 
will be completed for its first 15 full-rate production regions in June 
2006. Contract items for the remaining full-rate production regions- 
intended to be addressed in two additional groups-are to be negotiated 
at a later date. 

There is also uncertainty regarding Rescue 21 vessel functionality. 
Initial plans for Rescue 21 included functionality to allow USCG to 
track its vessels and to allow data transfer between vessels and group 
communications centers plus an expanded vessel communications suite. 
This functionality is now being deferred while USCG and DHS decide 
whether to complete development of its contractor-developed solution or 
use a different solution such as the Automatic Identification 
System[Footnote 17] for asset tracking and data transfer on vessels. 
This decision is currently expected by the end of April 2006. 

Additional issues may also contribute to further cost overruns and 
schedule delays. 

Variability in infrastructure preparation costs. According to USCG 
officials, there is significant variation in total costs for antenna 
towers. Additional costs can be incurred in building a tower and 
significant schedule delays can occur. Further, environmental concerns 
may further delay tower construction. For these reasons, there is 
significant potential for cost overruns and schedule slippages in tower 
construction. 

Site reorganization. New site command centers are to be established 
throughout USCG, based on a new standard organization model. There is 
no guarantee that current sites will have the same functions that they 
have now. For example, the Eastern Shore group will become a site with 
the communication center at Hampton Roads. The equipment at the Eastern 
Shore GCC will be moved to Hampton Roads and sites will be reconfigured 
to the new arrangement. Therefore, rework may be needed when the site 
centers are finalized. 

Cost and Schedule Management: 

USCG has plans to improve its cost and schedule management in the 
future. For example: 

USCG intends to conduct an integrated baseline review of the 
contractor's cost and schedule estimates after it completes 
renegotiation of expired contract items for the deployment to the first 
15 full-rate production regions. This baseline review is currently 
expected to begin in the June 2006 time frame and is supposed to 
validate the accuracy of the Rescue 21 schedule. However, there is no 
completion date for the baseline review. USCG also intends to complete 
another baseline review of the schedule once the contracts for the 
remaining 25 full-rate production sites are renegotiated. 

USCG intends to conduct full-rate production site deployment 
incrementally as it expects this gradual deployment to improve its 
ability to manage cost and schedule. The contracts for the first group 
of 15 regions are currently being negotiated; contracts for the second 
set of 11 regions and third set of 14 regions will be addressed 
afterwards. 

Delay in Implementation: 

The effects of the delays in deploying Rescue 21 include (1) key 
functionality, such as improved direction finding and improved 
coverage, will not be available; (2) potential failure of legacy 
equipment requiring costly repairs; and (3) potential for further 
upgrades to the legacy system and resulting costs, despite a moratorium 
on upgrades. 

Key functionality to production sites will be delayed. Originally, 
Rescue 21 was to be deployed nationwide by fiscal year 2006. The 
current schedule shows full deployment in fiscal year 2011, five years 
behind schedule. This will result in a significant implementation delay 
for many field units. Personnel at the two initial operating capability 
stations acknowledged that Rescue 21 provides significantly improved 
functionality compared to the legacy system. For example, according to 
these personnel, Rescue 21's direction finding capability-which allows 
USCG to locate boaters in distress-is significantly more accurate as 
well as more reliable than the legacy system. As a result, until Rescue 
21 is deployed, field units with the legacy system will continue to be 
at risk of performing larger and potentially more costly searches due 
to limited direction finding capabilities. 

Until Rescue 21 is deployed, field units will likely continue to 
experience communication coverage gaps, which limits their ability to 
hear boaters in distress. However, the impact of coverage gaps varies 
depending on the availability of local partners to conduct search and 
rescue, or vessels to relay distress calls. Personnel at one field unit 
explained that they rely on a marine patrol unit to conduct searches in 
a coverage gap area, whereas at another site, the USCG field unit is 
the only resource available to conduct search and rescue operations. 
Nevertheless, until Rescue 21 is fully deployed, missed mayday calls- 
and the potential for lives lost-creates uncertainty about the legacy 
system's reliability and reduces operators' confidence in the system. 

Potential for failure of legacy equipment requiring costly repairs. The 
legacy system is about 30 years old and is difficult to maintain. Until 
Rescue 21 is deployed, legacy equipment will continue to be at high 
risk for failure; these failures could result in costly repairs. Some 
USCG maintenance officials questioned whether legacy equipment will 
continue to function until Rescue 21 is fully deployed, especially in 
field units scheduled for later implementation. Field units could also 
experience higher costs resulting from the need to replace equipment 
and reengineer parts that are no longer manufactured. In addition, 
maintenance expertise needed to repair the antiquated equipment is 
limited and, as a result, repairs are not always timely. 

Further upgrades to the legacy system to address operational issues 
could occur, despite a moratorium on upgrades. The USCG instituted a 
moratorium on upgrades to the legacy system because Rescue 21 was to 
replace the equipment. However, with units facing delays of up to 5 
years, USCG may decide that the risk to its operations is sufficiently 
high that it should upgrade or install new legacy equipment rather than 
wait to receive the new system. In certain cases, this has already 
occurred, despite the moratorium. For example, USCG officials reported 
that they upgraded a console at one unit to mitigate operational 
challenges, and installed a new antenna at a second unit to address 
coverage gaps. Additional upgrades such as these would result in higher 
costs, especially at field units scheduled for later implementation. 

Conclusions: 

Although USCG plans to improve its capabilities in a number of areas to 
better manage the remaining development and deployment of Rescue 21 by 
establishing executive-level meetings with the prime contractor, it is 
imperative that key executives aggressively monitor costs and schedule 
performance and mitigate risks so the remaining development and 
deployment of Rescue 21 is effectively managed and delivered. 

USCG's current acquisition cost estimate is not viable and its schedule 
is uncertain and likely to change. USCG has plans to complete an 
integrated baseline review, but it has not established a target 
completion date. Without the results of this review, the cost and 
schedule estimate cannot be completed. 

If deployment of Rescue 21 continues to be delayed, Coast Guard sites 
and services will be impacted in several ways. First, key 
functionality, such as improved direction finding and improved coverage 
of coastal areas, will not be available as planned. Coast Guard 
personnel at these sites will continue to use outdated legacy 
communications systems for search and rescue operations and coverage of 
coastal regions will remain limited. In addition, these delays could 
result in costly upgrades to the legacy system to address 
communications coverage gaps as well as other operational concerns. 

Recommendations: 

To more effectively manage the remaining development and deployment of 
Rescue 21, we recommend that the Commandant of the United States Coast 
Guard ensure that executive-level management implements the following 
three recommendations: 

* oversee the project's progress toward cost and schedule milestones 
and manage risks, 

* establish a milestone to complete Rescue 21's integrated baseline 
review, to include all renegotiated contract items, and: 

* use the results of this review to complete a revised cost and 
schedule estimate. 

Agency Comments: 

In providing oral comments on a draft of this briefing, the Coast Guard 
Assistant Commandant for Acquisition stated that the briefing fairly 
represented Rescue 21 risks and ongoing activities to address program 
management and other issues. He also stated that this briefing would be 
used as input for them to take corrective actions to improve program 
management. 

Regarding the status of the vessel asset tracking and data transfer 
functionality that was deferred, the Assistant Commandant stated that 
he expects this functionality to be addressed with the Automatic 
Identification System, rather than by Rescue 21. This decision is 
pending Department of Homeland Security concurrence. 

Coast Guard did not concur with our statements regarding executive 
oversight. They stated that, prior to 2005, there were briefings to 
executive management, such as the key decision point and semi-annual 
meetings. However, as previously stated, there is no evidence that 
these executive-level meetings occurred. 

The Coast Guard also provided specific technical comments, which we 
have incorporated into the briefing as appropriate. 

[End of Slide Protection] 

[End of section] 

Appendix II: Comments from the Department of Homeland Security: 

U.S. Department of Homeland Security: 
Washington, DC 20528: 

Homeland Security: 

May 12, 2006: 

Mr. David A. Powner: 
Director: 
Information Technology Management Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Powner: 

Thank you for the opportunity to review and comment on the Government 
Accountability Office's (GAO's) draft report entitled UNITED STATES 
COAST GUARD: Improvements Needed in Management and Oversight of Rescue 
System Acquisition (GAO-06-632). This draft report includes briefing 
slides presented on March 27, 2006 to the House Committee on 
Appropriations, Subcommittee on Homeland Security and the Senate 
Committee on Commerce, Science and Transportation, Subcommittee on 
Fisheries and Coast Guard. 

While we agree with GAO's three recommendations contained therein, and 
are in the process of implementing each recommendation, we do have 
substantial concerns with several parts of the report and the briefing. 

We agree that the Coast Guard is responsible for managing schedule 
delays and cost overruns in major acquisition projects. The following 
program management improvements to control cost and schedule have been 
initiated by the Department of Homeland Security (DHS) and the Coast 
Guard: 

* Established an on-site Coast Guard Project Resident Office for 
increased contractor oversight. (Aug 2005). 

* Ordered full rate production regions incrementally to improve control 
(Sep 2005). * Implemented Alpha Contracting for improved communications 
between the government and contractor to streamline the process (Sep 
2005). * Implemented fixed price maintenance contracts after initial 
year (Sep 2005). 

* Increased oversight with quarterly Flag Officer/Vice President level 
program reviews and periodic DHS Joint Requirements Council (JRC) and 
Investment Review Board (IRB) reviews (Oct 2005). 

* Issued a Terminate for Convenience of the Government letter to the 
contractor on all vessel initiatives as a cost mitigating strategy 
(April 2006): 

* Will conduct Integrated Baseline Reviews (IBR) for Full Rate 
Production to better identify and control cost beginning Aug 2006. 

* Will conduct monthly reviews of project schedule with the contractor 
beginning June 2006. 

As reflected above, many of the cost and schedule concerns raised by 
the report have since been addressed. Accordingly, the project's APB 
(acquisition program baseline) for cost, schedule, and performance have 
been revised. For instance, the total acquisition cost parameter is 
$730.2M (vice $872M, as estimated by GAO) which has been achieved by 
adjusting several performance parameters. Additionally, on April 21, 
2006, the Coast Guard issued a notice of termination for convenience, 
canceling General Dynamics' option to provide all vessel sub-system 
(VSS) components. The Coast Guard anticipates that this work will be 
handled organically or by another contractor or a combination of both. 

The shortcomings cited in the GAO report, however, do not adequately 
emphasize the root cause of the RESCUE 21 system deployment delays and 
overruns. As confirmed by GAO's findings on page 27 of the briefing, 
the contractor: 

* underestimated the time for development and testing of RESCUE 21 and 
could not meet its own established integrated project schedule, and: 

* failed to consistently provide personnel in a timely manner or with 
the knowledge and skills required to complete the project. 

General Dynamics' inability to meet the technical performance within 
its proposed schedule was due to challenges in software development. 
The Coast Guard was fully aware of the problem the contractor was 
experiencing and made a conscious decision during the software 
development phase to offset cost and schedule to ensure technical 
performance was not compromised for this critical life-saving system. 

The GAO draft report highlights five key factors that contributed to 
RESCUE 21 cost overruns and schedule delays. As noted above, the Coast 
Guard agrees with GAO's fourth factor regarding "Contractor Cost and 
Schedule Estimation and Delivery." However, the remaining key factors 
are not accurate in important aspects and the Coast Guard provides the 
following facts and rationale (the following citations reflect the 
revisions GAO made to the draft report as a result of subsequent 
discussions with the Coast Guard): 

Requirements Management: GAO "Coast Guard did not effectively manage 
requirements and testing revealed incomplete and poorly defined 
requirements." 

This draft report does not recognize well-established Coast Guard 
processes and is not consistent with the GAO's previous acknowledgement 
of these processes (Report GAO-03-1111). The Coast Guard's requirements 
management process is outlined in the Major Systems Acquisition Manual 
(MSAM COMDINST M4150.2) Chapter 3 "Requirements Management Process," 
which includes development of the: 

* Mission Capability Gap Analysis documented in the Mission Analysis 
Report (MAR) dated 19 May 1994. 

* Broad Mission and Capability Need documented in the Mission Needs 
Statement (MNS) dated 18 Apr 1995, updated 14 May 1998, and most 
recently revalidated 12 Apr 2005. 

* Mission Oriented Capabilities documented in the Preliminary 
Operational Requirements Documents (PORD) dated 19 Nov 1997, and the 
Operational Requirements Document (ORD) 1 dated 19 Jul 1999, updated 20 
Aug 2001, and most recently revalidated 12 Apr 2005. 

* System Specific Capabilities documented in the Performance 
Specification (P-Spec) dated 8 Feb 2002. 

The Coast Guard maintains that such requirements as, "receive a 1 watt 
transmission from 2 meters above the water 20NM offshore" and 
"Direction finding (DF) to within +/-2 degrees", demonstrate the 
specificity of the RESCUE 21 system requirements, as documented in the 
Performance Specification. 

Moreover, field testing demonstrated that the requirements were clearly 
defined and that such testing is the appropriate mechanism to verify 
whether General Dynamics' design met the contractual requirements of 
the Performance Specifications. 

* General Dynamic's software design (and to a certain degree the 
contractor's hardware configuration design) included over 200 defects 
that required correction and re-testing for conformance to requirements 
prior to acceptance by the Coast Guard. These defects were documented 
in the Formal Qualification Testing (FQT) and System Integration 
Testing (SIT) plans; both of which were provided to GAO. 

* While General Dynamics ultimately met the system requirements as a 
result of their efforts to correct defects and retest performance, the 
Coast Guard determined that the functionality of the system could be 
improved by incorporating human systems interface (HSI) enhancements 
proposed by the users. Given the life-saving nature of the RESCUE 21 
system, the Coast Guard believes that the decision to offset schedule 
for greater HSI performance was prudent, and in the end ensures the 
Coast Guard can adequately serve the American public. 

Project Monitoring: GAO "Coast Guard did not effectively use earned 
value management data to measure performance and take corrective action 
on negative trends." 

Since award of the Phase II contract to General Dynamics in September 
2002, the Coast Guard established an Earned Value Management (EVM) work 
group comprised of Coast Guard officials with support from Subject 
Matter Experts from Booz Allen Hamilton. Together, this group was 
responsible for reviewing the monthly Cost Performance Reports (CPR) 
submitted by General Dynamics, conducting analyses to determine 
performance trends and developing monthly Performance Measurement 
Reports. 

In accordance with the performance-based, performance-specification 
nature of the Phase II contract, the RESCUE 21 Project Manager (PM) 
employed the recommendations of the Coast Guard EVM work group to 
highlight to General Dynamics' upper management their poor performance 
(e.g., inability to maintain cost and schedule performance within the 
variance thresholds). The PM further requested that General Dynamics 
institute corrective actions to reverse the negative performance 
trends. General Dynamics clearly acknowledged the challenges they were 
facing in designing the system as documented in their Nov 2002 CPR: 

From General Dynamics Source Document ID: PIA 16276/A008-002A (Nov 
2002): 

* "The schedule variance was primarily attributable to the Software 
Development account" 

* "The program's critical path continues to be driven by the software 
development effort." 

The following excerpts are from formal correspondence the Coast Guard 
submitted to General Dynamics and this information was provided to GAO 
as evidence of concrete actions taken by the RESCUE 21 PM in light of 
the EVM data: 

From Coast Guard Source Document ID: PIA 16276/AO08-017 (Feb 2004): 

* "The CG is extremely concerned about the current and cumulative cost/ 
schedule performance trends and status related to the cost plus CLINs. 
There are several key issues that must be addressed in order to provide 
insight into the contractor's commitment to control cost overruns and 
recover from schedule delays." 

* "The variance reporting must be expanded to provide more insight into 
ongoing and future risk mitigation and recovery plans related to cost 
control and schedule recovery." 

From Coast Guard Source Document ID PIA 16276/AO08-022 (Jul 2004): 

* "The USCG is very concerned that the information reflected on the CPR 
is not being used by GDDS as a management tool to make informed 
decisions and manage risk." 

In response to Coast Guard concerns, in June 2004 General Dynamics 
undertook a significant re-organization of its management structure as 
a corrective action in an attempt to better manage cost and schedule 
performance. 

Risk Management: GAO "Coast Guard did not effectively mitigate and 
communicate risks." 

In addition to the process for managing risks as documented in RESCUE 
21's Risk Management Plan, risks are discussed and reported during the 
Project Management Reviews (PMR), Semi-Annual briefs, Performance 
Measurement Reports, Monthly Acquisition Reports, Coast Guard 
Acquisition Review Council (CGARC) and IRB meetings, and various 
weekly/monthly workgroup meetings. During each PMR, risks affecting 
cost, schedule, logistics, and technical performance are communicated 
to all General Dynamics and Coast Guard teams and follow on discussions 
are held to validate the mitigation actions, ownership, and planned 
dates for closure. 

For example, the risk associated with software development was 
communicated, and actions were taken to mitigate "TECHNICAL 
PERFORMANCE" risk. This risk was assigned top priority, given the life- 
saving aspect of RESCUE 21. Mitigation actions included (i) 
establishment of an HSI workgroup comprised of Coast Guard and General 
Dynamics personnel to discuss and resolve issues, (ii) assignment of a 
member of the Coast Guard team to the General Dynamics' facility for 6 
months to facilitate issue resolution, (iii) establishment of a 
contract modification to include enhancements to the system proposed by 
the Coast Guard operational community, (iv) development of a plan of 
action and milestones to implement the HSI enhancements. 

Executive-Level Oversight: GAO "Coast Guard stated that it had an 
executive oversight process that included semi-annual and key decision 
point reviews. However, there is no evidence that these reviews of 
Rescue 21 occurred before 2005." 

Early in the audit, the Coast Guard provided oral evidence as to the 
existence of senior leader executive oversight, upon which GAO 
requested documentary evidence. The Coast Guard later provided GAO 
staff a CD containing the semi-annual and key decision reviews dating 
from 2002 thru 2005. These reviews included: 

Table: 

Date: Jan 2002: 
Executive Governance & Oversight Presentation: Key Decision Point 2 
(milestone decision): 

Date: Oct 2002: 
Executive Governance & Oversight Presentation: Semi-Annual Presentation 
to Coast Guard Acquisition Review Council (CGARC): 

Date: Aug 2003: 
Executive Governance & Oversight Presentation: Semi-Annual Presentation 
to CGARC: 

Date: Feb 2004: 
Executive Governance & Oversight Presentation: Semi-Annual Presentation 
to CGARC: 

Date: July 2004: 
Executive Governance & Oversight Presentation: Semi-Annual Presentation 
to CGARC: 

Date: Apr 2005: 
Executive Governance & Oversight Presentation: Key Decision Point 3 
CGARC: 

Date: Jun 2005: 
Executive Governance & Oversight Presentation: Key Decision Point 3 DHS 
Investment Review Board: 

[End of Table] 

Additionally, throughout this period the RESCUE 21 Project Manager 
provided status updates on cost, schedule, technical performance and 
risks to executive management for oversight via established weekly 
reports to the Acquisition Director; Monthly Acquisition Reports to the 
Commandant; Quarterly Acquisition Reports to Congress (as approved by 
DHS and OMB); annual reports to Congress and Business Case Exhibit 300 
to the Office of Management and Budget (OMB) as well as periodic 
milestone decision reviews to the CGARC and DHS Joint Requirements 
Council (JRC)/Investment Review Board (IRB). 

Page 28 of the GAO briefing states that "executive management did not 
take action in response to the risks presented" at the DHS IRB in Jun 
2005. To the contrary, the Top Ten project risks were presented to the 
CGARC in Apr 2005 in preparation for the DHS IRB. The Vice Commandant 
(in his role as the Coast Guard Acquisition Executive) led the 
discussion regarding ongoing mitigation actions for the risks 
identified. A number of these risks were successfully mitigated and 
closed, and the remaining risks were highlighted in the DHS JRC/IRB 
investment review process. The DHS Deputy Secretary specifically 
addressed the cost, environmental, real estate and vessel subsystem 
risks in his Acquisition Decision Memorandum of 23 Jun 2005. 

The Department greatly appreciates the efforts of the GAO audit team 
and acknowledges that the scope of the evaluation was broad, requiring 
the review and analysis of a myriad of project plans and documentation 
within an extremely compressed timeline to meet Congressional 
deadlines. The Coast Guard will continue to aggressively pursue the 
recommendations of the GAO report in order to ensure RESCUE 21 provides 
critical life-saving capabilities, and that the acquisition itself is 
effectively managed. 

Thank you again for the opportunity to comment on this draft report and 
we look forward to working with you on future homeland security issues. 

Sincerely, 

Signed by: 

Steven J. Pecinovsky: 
Director: 
Departmental GAO/OIG Liaison Office: 

[1] With respect to the ORD, the GAO draft report (Page 2) states, 
"Also, originally, Rescue 21 was intended to limit communications gaps. 
However, recent information indicates that while the Coast Guard has a 
goal of achieving 2 percent gaps to the extent possible, constraints 
may limit their ability to meet this goal. Coast Guard further stated 
that limiting gaps to 2 percent was only a goal, its threshold gap 
level was 10 percent or less." The Coast Guard has not made any 
reduction in coverage requirement since the ORD (Operational 
Requirements Document) was approved in 2001. The ORD specifies a 
communications coverage goal of 98% and threshold of 90%. The goal of 
the Coast Guard is to achieve 98% communications coverage to the 
maximum extent possible. In discrete cases, where environmental, 
historical preservation, geography or other external factors may 
constrain communication coverage to something less than 98%, the 
established threshold of 90% as documented in the ORD will be met. In 
no case will the project deploy less than 90% communications coverage 
unless a waiver is approved (only considered in rare situations where 
the cost of covering a particular area is grossly disproportionate to 
the potential benefits). 

[End of section] 

Appendix III GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

David A. Powner, (202) 512-9286 Stephen L. Caldwell, (202) 512-9610: 

Staff Acknowledgments: 

In addition to the contacts named above, Joel Aldape, Lisa Canini, 
Carol Cha, Nancy Glover, Dawn Hoff, Tonia Johnson, George Kovachick, 
Freda Paintsil, Karl Seifert, and Kim Zelonis made key contributions to 
this report. 

FOOTNOTES 

[1] In April 2006, the Department of Homeland Security approved a new 
acquisition program baseline for Rescue 21 with a total acquisition 
cost of $730.2 million after decreasing certain functionality. 

[2] EVM is a project management tool that integrates the investment 
scope of work with schedule and cost elements for investment planning 
and control. This method compares the value of work accomplished during 
a given period with that of the work expected in the period. 
Differences in expectations are measured in both cost and schedule 
variances. The Office of Management and Budget (OMB) requires agencies 
to use EVM as part of their performance-based management system for any 
investment under development or with system improvements under way. 

[3] USCG is currently reorganizing its field units into new, 
integrated, and standardized organizations referred to as sectors. 

[4]EVM is a project management tool that integrates the investment 
scope of work with schedule and cost elements for investment planning 
and control. This method compares the value of work accomplished during 
a given period with that of the work expected in the period. 
Differences in expectations are measured in both cost and schedule 
variances. OMB requires agencies to use EVM as part of their 
performance-based management system for any investment under 
development or with system improvements under way. 

[5] [5] USCG is in the process of reorganizing its field units into new 
integrated and standardized organizations, referred to as sectors. 

[6] A digital selective calling radio has the ability to send a mayday 
signal that identifies the vessel and, when connected to a Global 
Positioning System, sends the vessel's location. This occurs on channel 
70. 

[7] An indefinite-delivery, indefinite-quantity contract provides for 
an indefinite quantity, within stated limits, of supplies or services 
during a fixed period of time. The government schedules deliveries or 
performance by placing orders with the contractor. 48 C.F.R. §16.504. 

[8]According to USCG officials, Channel 16, which is analogous to 911, 
is the channel designated for the use of distressed boaters to gain 
assistance in an emergency. 

[9] GAO, Coast Guard: New Communications System to Support Search and 
Rescue Faces Challenges, GAO-03-1111 (Washington, D.C.: Sept. 30, 
2003). 

[10] Rescue 21 tests include: formal qualification testing, which 
demonstrates that the system meets contract performance specifications; 
system integration testing, which demonstrates that the system is fully 
functional at USCG stations; and operational testing and evaluation, 
which is conducted by the USCG and ensures that the system satisfies 
contract performance requirements and is ready to be deployed at other 
USCG locations. 

[11]USCG also uses the term "ground sub-system" for the functionality 
and equipment deployed to the stations in the initial operating 
capability regions. The functionality and equipment that was to be 
deployed to the vessels associated with these stations, which was not 
delivered as scheduled, is referred to as the "vessel sub-system". 

[12] USCG intends to use these low-rate initial production sites to 
test Rescue 21 capabilities in various geographical climates. 

[13] USCG contracted Booz Allen Hamilton to assist with program 
management for Rescue 21. 

[14] USCG is in the process of reorganizing its field units into new 
integrated and standardized organizations, referred to as sectors. 

[15] This estimate reflects only the acquisition component and does not 
include operations & maintenance. The total life cycle cost estimate is 
approximately $1 .6 billion. 

[16] We used contractor-provided data that reflected work from October 
2003 to December 2005. 

[17] The Automatic Identification System is a Coast Guard project to 
develop a new system that will be used to monitor and track vessels 
traveling to and through US waters. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: