This is the accessible text file for GAO report number GAO-05-858 
entitled 'DOD Business Systems Modernization: Navy ERP Adherence to 
Best Business Practices Critical to Avoid Past Failures' which was 
released on October 31, 2005. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

September 2005: 

DOD Business Systems Modernization: 

Navy ERP Adherence to Best Business Practices Critical to Avoid Past 
Failures: 

GAO-05-858: 

GAO Highlights: 

Highlights of GAO-05-858, a report to congressional requesters: 

Why GAO Did This Study: 

The Department of Defense’s (DOD) difficulty in implementing business 
systems that are efficient and effective continues despite the billions 
of dollars that it invests each year. For a decade now—since 1995—we 
have designated DOD’s business systems modernization as “high-risk.” 
GAO was asked to (1) provide a historical perspective on the planning 
and costs of the Navy’s four Enterprise Resource Planning (ERP) pilot 
projects, and the decision to merge them into one program; (2) 
determine if the Navy has identified lessons from the pilots, how the 
lessons are being used, and challenges that remain; and (3) determine 
if there are additional best business practices that could be used to 
improve management oversight of the Navy ERP. 

What GAO Found: 

The Navy invested approximately $1 billion in four ERP pilots without 
marked improvement in its day-to-day operations. The planning for the 
pilots started in 1998, with implementation beginning in fiscal year 
2000. The four pilots were limited in scope and were not intended to be 
corporate solutions for any of the Navy’s long-standing financial and 
business management problems. Furthermore, because of the various 
inconsistencies in the design and implementation of the pilots, they 
were not interoperable, even though they performed many of the same 
business functions. In short, the efforts were failures and $1 billion 
was largely wasted. 

Because the pilots would not meet its overall requirements, the Navy 
decided to start over and develop a new ERP system, under the 
leadership of a central program office. Using the lessons learned from 
the pilots, the current Navy ERP program office has so far been 
committed to the disciplined processes necessary to manage this effort. 
GAO found that, unlike other systems projects it has reviewed at DOD 
and other agencies, Navy ERP management is following an effective 
process for identifying and documenting requirements. The strong 
emphasis on requirements management, which was lacking in the previous 
efforts, is critical since requirements represent the essential 
blueprint that system developers and program managers use to design, 
develop, test, and implement a system and are key factors in projects 
that are considered successful. 

While the Navy ERP has the potential to address some of the Navy’s 
financial management weaknesses, as currently planned, it will not 
provide an all-inclusive end-to-end corporate solution for the Navy. 
For example, the current scope of the ERP does not include the 
activities of the aviation and shipyard depots. Further, there are 
still significant challenges and risks ahead as the project moves 
forward, such as developing and implementing 44 system interfaces with 
other Navy and DOD systems and converting data from legacy systems into 
the ERP system. The project is in its early phases, with a current 
estimated completion date of 2011 at an estimated cost of $800 million. 
These estimates are subject to, and very likely will, change. Broader 
challenges, such as alignment with DOD’s business enterprise 
architecture, which is not fully defined, also present a significant 
risk. Given DOD’s past inability to implement business systems that 
provide the promised capability, continued close management 
oversight—by the Navy and DOD—will be critical. In this regard, the 
Navy does not have in place the structure to capture quantitative data 
that can be used to assess the effectiveness of the overall effort. 
Also, the Navy has not established an IV&V function. Rather, the Navy 
is using in-house subject matter experts and others within the project. 
Industry best practices indicate that the IV&V activity should be 
independent of the project and report directly to agency management in 
order to provide added assurance that reported results on the project’s 
status are unbiased. 

What GAO Recommends: 

GAO makes three recommendations to DOD: (1) develop and implement the 
quantitative metrics needed to evaluate project performance and risks 
and use the metrics to assess progress and compliance with disciplined 
processes, (2) establish an independent verification and validation 
(IV&V) function and direct that all IV&V reports be provided to Navy 
management and the appropriate DOD investment review board, and (3) 
institute semiannual reviews of the program. DOD generally agreed with 
our recommendations. 

www.gao.gov/cgi-bin/getrpt?GAO-05-858. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Gregory Kutz at (202) 512-
9505 or Keith Rhodes at (202) 512-6412. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Navy's Pilot Projects Lacked Coordinated Management Oversight: 

Requirements Management Process Effective, but Implementation 
Challenges and Risks Remain: 

Additional Actions Can be Taken to Improve Management Oversight of the 
Navy ERP Effort: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Scope and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: Identification of the Navy and Defense Systems That Must 
Interface with the ERP: 

Appendix IV: GAO Contacts and Acknowledgments: 

Tables: 

Table 1: Navy ERP Pilot Projects: 

Table 2: Functions Performed by the Pilot Projects: 

Table 3: Documentation for Detailed Requirements: 

Figures: 

Figure 1: Percent of Effort Associated with Undisciplined Projects: 

Figure 2: Relationship between Requirements Development and Testing: 

Figure 3: Navy ERP Required System Interfaces: 

Letter September 29, 2005: 

Congressional Requesters: 

The Department of Defense (DOD) has historically been unable to develop 
and implement business systems on time, within budget, and with the 
promised capability. As noted in our recent report,[Footnote 1] this 
difficulty continues despite the billions of dollars that DOD spends 
each year to operate, maintain, and modernize its currently reported 
4,150 business systems. For fiscal year 2005, the department requested 
$13 billion for its existing business systems environment. For a decade 
now--since 1995--we have designated DOD's financial management and 
business systems modernization as "high-risk." In fact, of the 25 areas 
on GAO's governmentwide "high-risk" list, 8 are DOD specific program 
areas, and the department shares responsibility for 6 other high-risk 
areas that are governmentwide in scope.[Footnote 2] 

DOD has recognized the importance of transforming its business 
operations and systems to make them more efficient and effective in 
support of the department's defense mission. A critical aspect of the 
department's transformation effort will be its ability to effectively 
implement business systems on time, within budget, and with the 
promised capability. 

This report responds to your request for information on DOD's 
management of selected facets of its business modernization efforts 
that are intended to enhance the department's reporting on its results 
of operation. As agreed with your offices, we selected the Department 
of the Navy's (Navy) Enterprise Resource Planning (ERP) 
program,[Footnote 3] initially created as four pilot projects and now 
being pursued as a consolidated effort, to review and determine if it 
will help to resolve some of the Navy's long-standing financial and 
business management problems. The Navy expects the ERP to be fully 
operational by fiscal year 2011. It is currently estimated that for 
fiscal years 2004-2011, the program will cost approximately $800 
million. When fully operational, the Navy reports that the ERP will 
manage an estimated 80 percent of its appropriated funds.[Footnote 4] 
Our objectives were to (1) provide a historical perspective on the 
planning and costs of the Navy's four ERP pilot projects, and the 
decision to merge them into one program; (2) determine if the Navy has 
identified lessons from the pilots, how the lessons are being used, and 
the challenges that remain; and (3) determine if there are additional 
best business practices that could be used to improve management 
oversight of the Navy ERP. 

To obtain a historical perspective on the planning and costs of the 
Navy's four ERP pilot projects, and the decision to merge them into one 
program, we reviewed DOD's budget justification materials and other 
background information on the four pilot projects and met with program 
management and DOD Chief Information Officer (CIO) officials. 
Additionally, we reviewed project documentation provided by DOD and 
held discussions with program management officials related to two key 
processes--requirements management and testing--to determine if these 
key aspects of program management were being performed and if the 
system is being designed to help address some of Navy's long-standing 
management problems. Further, we reviewed relevant industry standards 
and best practices, and interviewed and requested documentation from 
the Navy ERP to determine whether there are additional best business 
practices that could appropriately be used to improve management 
oversight of the ERP. Given that this effort is still in the early 
stages of development, we did not evaluate all best practices. Rather, 
we concentrated on those that could have an immediate impact in 
improving management's oversight. 

Our work was performed from August 2004 through June 2005 in accordance 
with U.S. generally accepted government auditing standards. Details on 
our scope and methodology are included in appendix I. We requested 
comments on a draft of this report from the Secretary of Defense or his 
designee. Written comments from the Deputy Under Secretary of Defense 
(Financial Management) and the Deputy Under Secretary of Defense 
(Business Transformation) are reprinted in appendix II. 

Results in Brief: 

The Navy has invested approximately $1 billion in its four pilot ERP 
efforts, without marked improvement in its day-to-day operations. The 
four pilots were limited in scope and were not intended to be a 
corporate solution for resolving any of the Navy's long-standing 
financial and business management problems. Because of the various 
inconsistencies in the design and implementation, the pilots were not 
interoperable, even though they performed many of the same business 
functions. The lack of a coordinated effort among the pilots led to a 
duplication of efforts in implementing many business functions and 
resulted in ERP solutions that carry out similar functions in different 
ways from one another. In essence, the pilots resulted in four more DOD 
stovepiped systems that did not enhance DOD's overall efficiency and 
resulted in $1 billion being largely wasted. 

Because the pilots were stovepiped, limited within the scope of their 
respective commands, and not interoperable, they did not transform the 
Navy's business operations. As a result, under the leadership of a 
central program office--something that was lacking in the pilots--the 
Navy decided to start over and undertake the development and 
implementation of a single ERP system. Using the lessons learned from 
the pilots, the current Navy ERP program office has so far demonstrated 
a commitment to the disciplined processes necessary to manage this 
effort and reduce the risks associated with system implementation. 
Specifically, we found that, unlike other reviews we have performed at 
DOD and other agencies, the Navy ERP program office is following an 
effective process for identifying and documenting 
requirements.[Footnote 5] Requirements represent the essential 
blueprint that system developers and program managers use to design, 
develop, test, and implement a system. 

While the Navy ERP has the potential to address some of the Navy's 
financial management weaknesses, its current planned functionality will 
not provide an all-inclusive end-to-end corporate solution for the 
Navy. For example, the current scope of the ERP does not provide for 
real-time asset visibility of shipboard inventory. Asset visibility has 
been and continues to be a long-standing problem within the department. 
The scope of the current effort is much larger than the pilots. The 
system is intended to manage an estimated 80 percent of the Navy's 
appropriated funds, according to the Navy ERP program office. 

The project has a long way to go, with a current estimated completion 
date of 2011, at an estimated cost of $800 million. These estimates are 
very likely to change, given (1) the Navy ERP's relatively early phase 
of development and (2) DOD's past history of not implementing systems 
on time and within budget. The project faces numerous significant 
challenges and risks that must be dealt with as the project moves 
forward. For example, 44 system interfaces[Footnote 6] with other Navy 
and DOD systems must be developed and implemented. Long-standing 
problems regarding the lack of integrated systems and use of 
nonstandard data within DOD pose significant challenges and risks to a 
successful Navy ERP interface with these systems. Testing the system 
interfaces in an end-to-end manner is necessary in order for the Navy 
to have reasonable assurance that the ERP will be capable of providing 
the intended functionality. Another challenge and risk is the Navy's 
ability to convert the necessary data from its legacy systems into the 
ERP system. Data conversion is one of the critical tasks necessary to 
successfully implement a new financial system. If data conversion is 
done right, the new system has a much greater opportunity for success. 
On the other hand, converting data incorrectly or entering unreliable 
data from a legacy system has lengthy and long-term repercussions. The 
adage "garbage in, garbage out" best describes the adverse impact of 
inadequate data conversion efforts. 

A broader challenge and risk that is out of the Navy ERP project's 
control, but could significantly affect it, is DOD's development of a 
business enterprise architecture (BEA).[Footnote 7] An enterprise 
architecture consists of snapshots of the enterprise's current 
environment and its target environment, as well as a capital investment 
road map for transitioning from the current to the target environment. 
The real value of an enterprise architecture is that it provides the 
necessary content for guiding and constraining system investments in a 
way that promotes interoperability and minimizes overlap and 
duplication. As we have recently reported,[Footnote 8] DOD's BEA still 
lacks many of the key elements of a well-defined architecture, and no 
basis exists for evaluating whether the Navy ERP will be aligned with 
the BEA and whether it would be a corporate solution for DOD in its "To 
Be" or target environment. At this time, it is unknown what the target 
environment will be. Therefore, it is unknown what business processes, 
data standards, and technological standards the Navy ERP must align to, 
as well as what legacy systems will be transitioned into the target 
environment. Developing a large-scale systems modernization program, 
such as the Navy ERP, without the context of an enterprise architecture 
poses risks of rework to comply with the BEA once it is fully defined. 

While we are encouraged by the Navy's management of the requirements 
process, other actions are needed to enhance management's oversight, 
both Navy and DOD, of the ERP effort. The Navy does not have in place 
the structure to capture quantitative data that can be used to assess 
the effectiveness of the overall effort. This information is necessary 
to understand the risk being assumed and whether the project will 
provide the desired functionality. Additionally, the Navy has not 
established an independent verification and validation (IV&V) function 
to provide an assessment of the Navy ERP to DOD and Navy management. 
Rather, the Navy is using in-house subject matter experts and others 
who report to the Quality Assurance team leader. Industry best 
practices indicate that IV&V activities should be conducted by 
individuals independent of the project and who report directly to 
agency management in order to provide added assurance that reported 
results on the project's status are unbiased. Since effective 
implementation of the disciplined processes has been shown to be a key 
indicator of whether a project has reduced its risk to acceptable 
levels, these management actions would provide increased confidence 
that the Navy ERP project is "on track." 

Considering that the project is still in its early stages of 
development and that there are significant challenges and high risks 
ahead, it is critical that mechanisms be in place to critically review 
the project at all stages to ensure it will result in improvements to 
the Navy's operations and alert the Navy and DOD if the project does 
not remain on schedule and within budget. Given the department's past 
history of not being able to successfully implement business systems on 
time and within budget, accountability is imperative. Thus, we are 
making three recommendations to the Secretary of Defense directed 
towards improving oversight over the Navy ERP. In its written comments 
on a draft of this report, DOD generally agreed with our 
recommendations. DOD agreed to develop quantitative metrics that can be 
used to evaluate the program and stated that the metrics would be 
developed by December 2005. While the department agreed with the 
recommendation to establish an IV&V, it noted that the IV&V would be 
within the project team and will report directly to the Navy ERP 
program manager. Additionally, while the department agreed with the 
intent of our recommendation that the Defense Business System 
Management Committee review the status of the Navy ERP on a semiannual 
basis, DOD noted that its tiered accountability structure, recently put 
in place to improve the control and accountability of business system 
investments, would provide the necessary oversight. In regard to the 
last two points, we continue to believe that providing the IV&V reports 
to the appropriate investment review board and instituting semiannual 
reviews by the Defense Business System Management Committee are the 
appropriate courses of action. Given (1) the Navy's initial estimate 
that the new ERP will cost at least $800 million and (2) the 
department's past difficulties in effectively developing and 
implementing business systems, substantive reviews that are focused 
just on the Navy ERP by the highest levels of management within the 
department are warranted so that management can act quicker rather than 
later if problems emerge. 

In its comments, the department took exception to our characterization 
of the pilots as failures and identified what it believes were 
achievements of the pilot programs. However, as discussed in the 
report, the pilots were limited in scope. Although they used the same 
software, inconsistencies in the design and implementation resulted in 
them not being interoperable. In essence, the department had four more 
stovepiped systems. The department's comments did not disagree with any 
of the above points. We characterized the pilots as failures because 
the department spent $1 billion on systems that did not result in 
marked improvement in the Navy's day-to-day operations. Furthermore, if 
there had been effective management oversight of the pilot programs at 
the outset, there would not have been a need to, in essence, start over 
with the Navy ERP. See the Agency Comments and Our Evaluation section 
of this report for a more detailed discussion of the agency comments. 
We have reprinted DOD's written comments in appendix II. 

Background: 

The Navy, with reported assets totaling $321 billion in fiscal year 
2004,[Footnote 9] would be ranked among the largest corporations in the 
world if it were a private sector entity. According to the Navy, based 
upon the reported value of its assets, it would be ranked among the 15 
largest corporations on the Fortune 500 list. Additionally, in fiscal 
year 2004 the Navy reported that its inventory was valued at almost $73 
billion and that it held property, plant, and equipment with a reported 
value of almost $156 billion. Furthermore, the Navy reported for fiscal 
year 2004 that its operations involved total liabilities of $38 
billion, that its operations had a net cost of $130 billion, and that 
it employed approximately 870,000 military and civilian personnel-- 
including reserve components.[Footnote 10] 

The primary mission of the Navy is to control and maintain freedom of 
the seas, performing an assortment of interrelated and interdependent 
business functions to support its military mission with service members 
and civilian personnel in geographically dispersed locations throughout 
the world. To support its military mission and perform its business 
functions, the Navy requested for fiscal year 2005 almost $3.5 billion 
for the operation, maintenance, and modernization of its business 
systems and related infrastructure--the most of all the DOD components-
-or about 27 percent of the total $13 billion DOD fiscal year 2005 
business systems budget request. Of the 4,150 reported DOD business 
systems, the Navy holds the largest inventory of business systems--with 
2,353 reported systems or 57 percent of DOD's reported inventory of 
business systems. 

The Secretary of Defense recognized that the department's business 
operations and systems have not effectively worked together to provide 
reliable information to make the most effective business decisions. He 
challenged each military service to transform its business operations 
to support DOD's warfighting capabilities and initiated the Business 
Management Modernization Program (BMMP) in July 2001. Further, the 
Assistant Secretary of the Navy for Financial Management and 
Comptroller (Navy Comptroller) testified that transforming the Navy's 
business processes, while concurrently supporting the Global War on 
Terrorism, is a formidable but essential task.[Footnote 11] He stated 
that the goal of the transformation is to "establish a culture and 
sound business processes that produce high-quality financial 
information for decision making." One of the primary elements of the 
Navy's business transformation strategy is the Navy ERP. 

Recently Reported Business and Financial Weaknesses at the Navy: 

The need for business processes and systems transformation to provide 
management with timely information to make important business decisions 
is clear. However, none of the military services, including the Navy, 
have passed the scrutiny of an independent financial audit. Obtaining a 
clean (unqualified) financial audit opinion is a basic prescription for 
any well-managed organization, as recognized by the President's 
Management Agenda. For fiscal year 2004, the DOD Inspector General 
issued a disclaimer on the Navy's financial statements--Navy's General 
Fund and Working Capital Fund--citing eight material 
weaknesses[Footnote 12] and six material weaknesses[Footnote 13] 
respectively, in internal control and noncompliance with the Federal 
Financial Management Integrity Act of 1996 (FFMIA).[Footnote 14] The 
inability to obtain a clean financial audit opinion is the result of 
weaknesses in the Navy's financial management and related business 
processes and systems. Most importantly, the Navy's pervasive 
weaknesses have (1) resulted in a lack of reliable information to make 
sound decisions and report on the status of activities, including 
accountability of assets, through financial and other reports to the 
Navy and DOD management and the Congress; (2) hindered its operational 
efficiency; (3) adversely affected mission performance; and (4) left 
the Navy and DOD vulnerable to fraud, waste, and abuse, as the 
following examples illustrate. 

* The Navy's lack of detailed cost information hinders its ability to 
monitor programs and analyze the cost of its activities. We 
reported[Footnote 15] that the Navy lacked the detailed cost and 
inventory data needed to assess its needs, evaluate spending patterns, 
and leverage its telecommunications buying power. As a result, at the 
sites we reviewed, the Navy paid for telecommunications services it no 
longer required, paid too much for services it used, and paid for 
potentially fraudulent or abusive long-distance charges. In one 
instance, we found that DOD paid over $5,000 in charges for one card 
that was used to place 189 calls in one 24-hour period from 12 
different cities to 12 different countries. 

* Ineffective controls over Navy foreign military sales using blanket 
purchase orders placed classified and controlled spare parts at risk of 
being shipped to foreign countries that may not be eligible to receive 
them. For example, we identified instances in which Navy country 
managers (1) overrode the system to release classified parts under 
blanket purchase orders without filing required documentation 
justifying the release; and (2) substituted classified parts for parts 
ordered under blanket purchase orders, bypassing the control-edit 
function of the system designed to check a country's eligibility to 
receive the parts.[Footnote 16] 

* The Naval Inventory Control Point and its repair contractors have not 
followed DOD and Navy procedures intended to provide the accountability 
for and visibility of inventory shipped to Navy repair contractors. 
Specifically, Navy repair contractors are not routinely acknowledging 
receipt of government-furnished material received from the Navy. A DOD 
procedure requires repair contractors to acknowledge receipt of 
government-furnished material that has been shipped to them from the 
Navy's supply system. However, Naval Inventory Control Point officials 
are not requiring repair contractors to acknowledge receipt of these 
materials. By not requiring repair contractors to acknowledge receipt 
of government-furnished material, the Naval Inventory Control Point has 
also departed from the procedure to follow up with the contractor 
within 45 days when the contractor fails to confirm receipt for an 
item. Without material receipt notification, the Naval Inventory 
Control Point cannot be assured that its repair contractors have 
received the shipped material. This failure to acknowledge receipt of 
material shipped to repair contractors can potentially impair the 
Navy's ability to account for shipments leading to possible fraud, 
waste, or abuse.[Footnote 17] 

* A limited Naval Audit Service audit revealed that 53 of 118 erroneous 
payment transactions, valued at more than $990,000, occurred because 
Navy certifying officials did not ensure accurate information was 
submitted to the Defense Finance and Accounting Service (DFAS) prior to 
authorizing payment. In addition, certifying officials submitted 
invoices to DFAS authorizing payment more than once for the same 
transaction.[Footnote 18] 

Brief Overview of Navy ERP: 

To address the need for business operations reform, in fiscal year 1998 
the Navy established an executive committee responsible for creating a 
"Revolution in Business Affairs" to begin looking at transforming 
business affairs and identifying areas of opportunity for change. This 
committee, in turn, set up a number of working groups, including one 
called the Commercial Business Practices (CBP) Working Group,[Footnote 
19] which consisted of representatives from financial management 
organizations across the Navy. This working group recommended that the 
Navy should use ERP as a foundation for change and identified various 
ERP initiatives that were already being developed or under 
consideration within the Navy. Ultimately, the Navy approved the 
continuation of four of these initiatives, using funds from existing 
resources from each of the sponsors (i.e., commands) to test the 
feasibility of ERP solutions within the Navy. From 1998 to 2003, four 
different Navy commands began planning, developing, and implementing 
four separate ERP pilot programs to address specific business areas. A 
CBP Executive Steering Group was created in December 1998 to monitor 
the pilot activities. 

As the pilots progressed in their development and implementation, the 
Navy identified issues that had to be addressed at a higher level than 
the individual pilots, such as the integration between the pilots as 
well as with other DOD systems, and decided that one program would 
provide a more enterprisewide solution for the Navy. In August 2002, 
the Assistant Secretary of the Navy for Research, Development, and 
Acquisition established a Navy-wide ERP program to "converge" the four 
ongoing pilots into a single program. This Navy-wide program is 
expected to replace all four pilots by fiscal year 2008 and to be 
"fully operational" by fiscal year 2011. The Navy estimates that the 
ERP will manage about 80 percent of the Navy's estimated appropriated 
funds--after excluding appropriated funds for the Marine Corps and 
military personnel and pay. Based on the Navy's fiscal years 2006 to 
2011 defense planning budget, the Navy ERP will manage approximately 
$74 billion annually.[Footnote 20] 

According to a Navy ERP official, while the Navy ERP would account for 
the total appropriated amount, once transactions occur at the depots, 
such as when a work order is prepared for the repair of an airplane 
part, the respective systems at the depots will execute and maintain 
the detailed transactions. This accounts for about 2 percent, or 
approximately $1.6 billion, being executed and maintained in detail by 
the respective systems at the aviation and shipyard depots--not by the 
Navy ERP. The remaining 20 percent that the ERP will not manage 
comprises funds for the Navy Installations Command, field support 
activity, and others. 

Navy's Pilot Projects Lacked Coordinated Management Oversight: 

Each of the Navy's four ERP pilot projects was managed and funded by 
different major commands within the Navy. The pilots, costing over $1 
billion in total, were limited in scope and were not intended to 
provide corporate solutions to any of the Navy's long-standing 
financial and business management problems. The lack of centralized 
management oversight and control over all four pilots allowed the 
pilots to be developed independently. This resulted in four more DOD 
stovepiped systems that could not operate with each other, even though 
each carried out many of the same functions and were based on the same 
ERP commercial-off-the-shelf (COTS) software. Moreover, due to the lack 
of high-level departmentwide oversight from the start, the pilots were 
not required to go through the same review process as other acquisition 
projects of similar magnitude. 

Pilots Developed Independently of Each Other: 

Four separate Navy organizations began their ERP pilot programs 
independently of each other, at different times, and with separate 
funding. All of the pilots implemented the same ERP COTS software, and 
each pilot was small in scale--relative to the entire Navy. For 
example, one of the pilots, SMART, was responsible for managing the 
inventory items and repair work associated with one type of engine, 
although the organization that implemented SMART--the Naval Supply 
Systems Command--managed the inventory for several types of engines. As 
of September 2004, the Navy estimated that the total investment in 
these four pilots was approximately $1 billion. Table 1 summarizes each 
of the pilots, the cognizant Navy organization, the business areas they 
address, and their reported costs through September 2004. 

Table 1: Navy ERP Pilot Projects: 

Dollars in millions. 

ERP pilot: CABRILLO; 
Organization: Space and Naval Warfare Systems Command; 
Area of pilot's focus: Financial Management; 
* Navy Working Capital Fund; 
Initial start: June 2000; 
Costs through FY 2004[A]: $67.4. 

ERP pilot: SMART; 
Organization: Naval Supply Systems Command; 
Area of pilot's focus: Supply Management; 
* Intermediate-Level Maintenance Management; 
* Interface to Aviation Depots; 
Initial start: August 2000; 
Costs through FY 2004[A]: $346.4. 

ERP pilot: NEMAIS; 
Organization: Naval Sea Systems Command; 
Area of pilot's focus: Regional Maintenance; 
* Intermediate-Level Maintenance Management; 
* Project Systems; 
* Workforce Management; 
Initial start: June 2000; 
Costs through FY 2004[A]: $414.6. 

ERP pilot: SIGMA; 
Organization: Naval Air Systems Command; 
Area of pilot's focus: Program Management with linkage among: 
* Contract Management; 
* Financial Management; 
* Workforce Management; 
Initial start: May 2001; 
Costs through FY 2004[A]: $215.9. 

Total; 
Costs through FY 2004[A]: $1,044.3. 

Source: GAO analysis of Navy data. 

[A] The costs reflect amounts disbursed through September 30, 2004, as 
reported by the Navy ERP program. 

[End of table] 

Even after the pilots came under the purview of the CBP Executive 
Steering Group in December 1998, they continued to be funded and 
controlled by their respective organizations. We have previously 
reported[Footnote 21] that allowing systems to be funded and controlled 
by component organizations has led to the proliferation of DOD's 
business systems. These four pilots are prime examples. While there was 
an attempt made to coordinate the pilots, ultimately each organization 
designed its ERP pilot to accommodate its specific business needs. The 
Navy recognized the need for a working group that would focus on 
integration issues among the pilots, especially because of the desire 
to eventually extend the pilot programs beyond the pilot organizations 
to the entire Navy. In this regard, the Navy established the Horizontal 
Integration Team in June 1999, consisting of representatives from all 
of the pilots to address this matter. However, one Navy official 
described this team as more of a "loose confederation" that had limited 
authority. As a result, significant resources have been invested that 
have not and will not result in corporate solutions to any of the 
Navy's long-standing business and financial management problems. This 
is evident as noted in the DOD Inspector General's audit reports of the 
Navy's financial statements discussed above. 

In addition to the lack of centralized funding and control, each of the 
pilots configured the software differently, which, according to Navy 
ERP program officials, caused integration and interoperability 
problems. While each pilot used the same COTS software package, the 
software offers a high degree of flexibility in how similar business 
functions can be processed by providing numerous configuration 
points.[Footnote 22] According to the Navy, over 2.4 million 
configuration points exist within the software. The pilots configured 
the software differently from each other to accommodate differences in 
the way they wanted to manage their functional area focus. These 
differences were allowed even though they perform many of the same 
types of business functions, such as financial management. These 
configuration differences include the levels of complexity in workflow 
activities and the establishment of the organizational structure. For 
example, the primary work order managed by the NEMAIS pilot is an 
intricate ship repair job, with numerous tasks and workers at many 
levels. Other pilots had much simpler work order definitions, such as 
preparing a budget document or procuring a single part for an engine. 

Because of the various inconsistencies in the design and 
implementation, the pilots were stovepiped and could not operate with 
each other, even though they performed many of the same business 
functions. Table 2 illustrates the similar business functions that are 
performed by more than one pilot. 

Table 2: Functions Performed by the Pilot Projects: 

Type of functions to be performed: 

Type of functions to be performed: Materials management: 
* Sales and distribution; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: Yes. 

Type of functions to be performed: Materials management: 
* Procurement; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: No. 

Type of functions to be performed: Financial management: 
* Financial accounting; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: Yes. 

Type of functions to be performed: Financial management: 
* Revenue and cost controlling; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: Yes. 

Type of functions to be performed: Financial management: 
* Asset accounting; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: Yes. 

Type of functions to be performed: Financial management: 
* Budgeting and funds management; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: No. 

Type of functions to be performed: Program management: 
* Project management; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: No. 

Type of functions to be performed: Program management: 
* Program planning, budgeting, control; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: No. 

Type of functions to be performed: Workforce management: 
* Time and attendance; 
NEMAIS: Yes; 
Cabrillo: Yes; 
SIGMA: Yes; 
SMART: No. 

Source: GAO, based upon information provided by the Navy. 

[End of table] 

By definition, an ERP solution should integrate the financial and 
business operations of an organization. However, the lack of a 
coordinated effort among the pilots led to a duplication of efforts and 
problems in implementing many business functions and resulted in ERP 
solutions that carry out redundant functions in different ways from one 
another. 

The end result of all of the differences was a "system" that could not 
successfully process transactions associated with the normal Navy 
practices of moving ships and aircraft between fleets. Another 
configuration problem occurred because the pilots generally developed 
custom roles[Footnote 23] for systems users. Problems arose after the 
systems began operating. Some roles did not have the correct 
transactions assigned to enable the users with that role to do their 
entire job correctly. Further, other roles violated the segregation-of- 
duties principle due to the inappropriateness of roles assigned to 
individual users. 

The pilots experienced other difficulties with respect to controlling 
the scope and performance schedules due to the lack of disciplined 
processes,[Footnote 24] such as requirements management. For example, 
the pilots did not identify in a disciplined manner the amount of work 
necessary to achieve the originally specified capabilities--even as the 
end of testing approached. There were repeated contract cost-growth 
adjustments, delays in delivery of many planned capabilities, and 
initial periods of systems' instabilities after the systems began 
operating. All of these problems have been shown as typical of the 
adverse effects normally associated with projects that have not 
effectively implemented disciplined processes. 

Pilots Lacked Departmentwide Oversight: 

The Navy circumvented departmentwide policy by not designating the 
pilots as major automated information systems acquisition programs. DOD 
policy in effect at the time[Footnote 25] stipulated that a system 
acquisition should be designated as a major program if the estimated 
cost of the system exceeds $32 million in a single year, $126 million 
in total program costs, or $378 million in total life-cycle costs, or 
if deemed of special interest by the DOD Chief Information Officer 
(CIO). According to the Naval Audit Service,[Footnote 26] all four of 
the pilots should have been designated as major programs based on their 
costs--which were estimated to be about $2.5 billion at the time--and 
their significance to Navy's operations. More specifically, at the time 
of its review, SMART's total estimated costs for development, 
implementation, and sustainment was over $1.3 billion--far exceeding 
the $378 million life-cycle cost threshold. However, because Navy 
management considered each of its ERP programs to be "pilots," it did 
not designate the efforts as major automated information systems 
acquisitions, thereby limiting departmental oversight.[Footnote 27] 

Consistent with the Clinger-Cohen Act of 1996, DOD acquisition 
guidance[Footnote 28] requires that certain documentation be prepared 
at each milestone within the system life cycle. This documentation is 
intended to provide relevant information for management oversight and 
in making decisions as to whether the investment of resources is cost 
beneficial. The Naval Audit Service reported[Footnote 29] that a key 
missing document that should have been prepared for each of the pilots 
was a mission needs statement.[Footnote 30] A mission needs statement 
was required early on in the acquisition process to describe the 
projected mission needs of the user in the context of the business need 
to be met. The mission needs statement should also address 
interoperability needs. As noted by the Naval Audit Service, the result 
of not designating the four ERP pilots as major programs was that 
program managers did not prepare and obtain approval of this required 
document before proceeding into the next acquisition phase. In 
addition, the pilots did not undergo mandatory integrated reviews that 
assess where to spend limited resources departmentwide. The DOD CIO is 
responsible for overseeing major automated information systems and a 
program executive office is required to be dedicated to executive 
management and not have other command responsibilities. However, 
because the pilots were not designated major programs, the oversight 
was at the organizational level that funded the pilots (i.e., command 
level). Navy ERP officials stated that at the beginning of the pilots, 
investment authority was dispersed throughout the Navy and there was no 
established overall requirement within the Navy to address systems from 
a centralized Navy enterprise level. The Navy ERP is now designated a 
major program under the oversight of the DOD CIO. 

Experience Has Shown the Effects of Not Effectively Implementing 
Disciplined Processes: 

The problems identified in the failed implementation of the four pilots 
are indicative of a system program that did not adhere to the 
disciplined processes. The successful development and implementation of 
systems is dependent on an organization's ability to effectively 
implement best practices, commonly referred to as disciplined 
processes, which are essential to reduce the risks associated with 
these projects to acceptable levels.[Footnote 31] However, the 
inability to effectively implement the disciplined processes necessary 
to reduce risks to acceptable levels does not mean that an entity 
cannot put in place a viable system that is capable of meeting its 
needs. Nevertheless, history shows that the failure to effectively 
implement disciplined processes and the necessary metrics to understand 
the effectiveness of processes implemented increases the risk that a 
given system will not meet its cost, schedule, and performance 
objectives. 

In past reports we have highlighted the impact of not effectively 
implementing the disciplined processes. These results are consistent 
with those experienced by the private sector. More specifically: 

* In April 2003, we reported[Footnote 32] that NASA had not implemented 
an effective requirements management process and that these requirement 
management problems adversely affected its testing activities. We also 
noted that because of the testing inadequacies, significant defects 
later surfaced in the production system. In May 2004, we 
reported[Footnote 33] that NASA's new financial management system, 
which was fully deployed in June 2003 as called for in the project 
schedule, still did not address many of the agency's most challenging 
external reporting issues, such as external reporting problems related 
to property accounting and budgetary accounting. The system continues 
to be unable to produce reliable financial statements. 

* In May 2004, we reported[Footnote 34] that the Army's initial 
deployments for its Logistics Modernization Program (LMP) did not 
operate as intended and experienced significant operational 
difficulties. In large part, these operational problems were due to the 
Army not effectively implementing the disciplined processes that are 
necessary to manage the development and implementation of the systems 
in the areas of requirements management and testing. The Army program 
officials have acknowledged that the problems experienced in the 
initial deployment of LMP could be attributed to requirements and 
testing. Subsequently, in June 2005,[Footnote 35] we reported that the 
Army still had not put into place effective management control and 
processes to help ensure that the problems that have been identified 
since LMP became operational in July 2003 are resolved in an efficient 
and effective manner. The Army's inability to effectively implement the 
disciplined processes provides it with little assurance that (1) system 
problems experienced during the initial deployment that caused the 
delay of future deployments have been corrected and (2) LMP is capable 
of providing the promised system functionality. The failure to resolve 
these problems will continue to impede operations at Tobyhanna Army 
Depot, and future deployment locations can expect to experience similar 
significant disruptions in their operations, as well as having a system 
that is unable to produce reliable and accurate financial and logistics 
data. 

* We reported in February 2005[Footnote 36] that DOD had not 
effectively managed important aspects of the requirements for the 
Defense Integrated Military Human Resources System, which is to be an 
integrated personnel and pay system standardized across all military 
components. For example, DOD had not obtained user acceptance of the 
detailed requirements nor had it ensured that the detailed requirements 
were complete and understandable. Based on GAO's review of a random 
sample of the requirements documentation, about 77 percent of the 
detailed requirements were difficult to understand. 

The problems experienced by DOD and other agencies are illustrative of 
the types of problems that can result when disciplined processes are 
not properly implemented. The four Navy pilots provide yet another 
example. As discussed previously, because the pilots were four 
stovepiped efforts, lacking centralized management and oversight, the 
Navy had to start over when it decided to proceed with the current ERP 
effort after investing about $1 billion. Figure 1 shows how 
organizations that do not effectively implement disciplined processes 
lose the productive benefits of their efforts as a project continues 
through its development and implementation cycle. Although 
undisciplined projects show a great deal of productive work at the 
beginning of the project, the rework associated with defects begins to 
consume more and more resources. In response, processes are adopted in 
the hopes of managing what later turns out to be, in reality, 
unproductive work. However, generally these processes are "too little, 
too late," and rework begins to consume more and more resources because 
the adequate foundations for building the systems were not done or not 
done adequately. In essence, experience shows that projects that fail 
to implement disciplined processes at the beginning are forced to 
implement them later, when it takes more time and they are less 
effective. 

As can be seen in figure 1, a major consumer of project resources in 
undisciplined efforts is rework (also known as thrashing). 

Figure 1: Percent of Effort Associated with Undisciplined Projects: 

[See PDF for image] 

[End of figure] 

Rework occurs when the original work has defects or is no longer needed 
because of changes in project direction. Disciplined organizations 
focus their efforts on reducing the amount of rework because it is 
expensive. Studies have shown that fixing a defect during testing is 
anywhere from 10 to 100 times more expensive than fixing it during the 
design or requirements phase.[Footnote 37] 

Requirements Management Process Effective, but Implementation 
Challenges and Risks Remain: 

To date, Navy ERP management has followed a comprehensive and 
disciplined requirements management process, as well as leveraged 
lessons learned from the implementation of the four ERP pilot programs 
to avoid repeating past mistakes. Assuming that the project continues 
to effectively implement the processes it has adopted, the planned 
functionality of the Navy ERP has the potential to address at least 
some of the weaknesses identified in the Navy's financial improvement 
plan. However, the project faces numerous challenges and risks. Since 
the program is still in a relatively early phase--it will not be fully 
operational until fiscal year 2011, at a currently estimated cost of 
$800 million--the project team must be continually vigilant and held 
accountable for ensuring that the disciplined processes are followed in 
all phases to help achieve overall success. For example, the project 
management office will need to ensure that it effectively oversees the 
challenges and risks associated with developing interfaces with 44 Navy 
and DOD systems and data conversion--areas that were troublesome in 
other DOD efforts we have audited. Considering that the project is in a 
relatively early phase and DOD's history of not implementing systems on 
time and within budget, the projected schedule and costs estimates are 
subject to, and very likely will, change. Furthermore, a far broader 
challenge, which lies outside the immediate control of the Navy ERP 
program office, is that the ERP is proceeding without DOD having 
clearly defined its BEA. As we have recently reported,[Footnote 38] 
DOD's BEA still lacks many of the key elements of a well-defined 
architecture. The real value of a BEA is that it provides the necessary 
content for guiding and constraining system investments in a way that 
promotes interoperability and minimizes overlap and duplication. 
Without it, rework will likely be needed to achieve those outcomes. 

Navy ERP Built on Lessons Learned from Pilot Projects: 

Although the four pilot projects were under the control of different 
entities and had different functional focuses, a pattern of issues 
emerged that the Navy recognized as being critical for effective 
development of future projects. The Navy determined that the pilots 
would not meet its overall requirements and concluded that the best 
alternative was to develop a new ERP system--under the leadership of a 
central program office--and use efforts from the pilots as starting 
points by performing a review of their functionality and lessons 
learned, eliminating redundancies, and developing new functionalities 
that were not addressed by the pilots. The lessons learned from the 
pilots cover technical, organizational, and managerial issues and 
reinforce the Navy's belief that it must effectively implement the 
processes that are necessary to effectively oversee and manage the ERP 
efforts. Navy ERP project management recognizes that the failure to do 
so would, in all likelihood, result in this ERP effort experiencing the 
same problems as those resulting in the failure of the four earlier 
pilots. 

One of the most important lessons learned from the earlier experiences 
by the Navy ERP project management is the need for following 
disciplined processes to identify and manage requirements. As discussed 
later in this report, the ERP program is following best practices in 
managing the system's requirements. A key part of requirements 
identification is to have system users involved in the process to 
ensure that the system will meet their needs. Additionally, the 
inclusion of system users in the detailed requirement development 
process creates a sense of ownership in the system, and prepares system 
users for upcoming changes to the way they conduct their business. 
Moreover, the experience from the pilots demonstrated that the working- 
level reviews must be cross functional. For example, the end-to-end 
process walkthroughs, discussed later, reinforce the overall business 
effect of a transaction throughout the enterprise, and help to avoid a 
stovepiped view of an entity's operations. 

Another lesson learned is the need to adopt business processes to 
conform with the types of business practices on which the standard COTS 
packages are based, along with the associated transaction 
formats.[Footnote 39] Just the opposite approach was pursued for the 
pilots, during which the Navy customized many portions of the COTS 
software to match the existing business process environment. However, 
the current Navy ERP management is restraining customization to the 
core COTS software to allow modifications only where legal or 
regulatory demands require. Obviously, minimizing the amount of 
customization reduces the complexity and costs of development. Perhaps 
more importantly, holding customization to a minimum helps an entity 
take advantage of two valuable benefits of COTS software. First, COTS 
software provides a mature, industry-proven "best practices" approach 
to doing business. The core elements of work-flow management, 
logistics, financial management, and other components have been 
optimized for efficiency and standardization in private industry over 
many years. According to program officials, the Navy ERP will adhere to 
the fundamental concepts of using a COTS package and thus take 
advantage of this efficiency benefit by modifying their business 
practices to match the COTS software rather than vice versa as was done 
in the four pilots. Having the software dictate processes is a 
difficult transition for users to accept, and Navy ERP officials 
recognize the challenge in obtaining buy-in from system users. To meet 
this challenge, they are getting users involved early in requirements 
definition, planning for extensive training, and ensuring that senior 
level leadership emphasize the importance of process change, so the 
entire chain of command understands and accepts its role in the new 
environment. In effect, the Navy is taking the adopted COTS process and 
then presenting it to the users. As a result, the Navy is attempting to 
limit the amount of customization of the software package. One 
important consideration in doing this is that if the standard COTS 
components are adopted, the maintenance burden of upgrades remains with 
the COTS vendor. 

Finally, the Navy learned from the pilots that it needed to manage its 
system integrators[Footnote 40] better. The ERP officials also found 
that they could significantly reduce their risk by using the 
implementation methodology of the COTS vendor rather than the specific 
approach of a system integrator. Each of the pilots had separate system 
integrators with their own particular methodology for implementing the 
COTS software. According to Navy ERP officials, using the 
implementation methodology and tool set of the COTS vendor maintains a 
closer link to the underlying software, and provides more robust 
requirements management by easily linking requirements from the highest 
level down to the COTS transaction level. Navy ERP is focused on 
staying as close as possible to the delivered COTS package, both in its 
avoidance of customization and its use of tools provided by the COTS 
vendor. In contrast, with the pilots, the Navy allowed the system 
integrators more latitude in the development process, relying on their 
expertise and experience with other ERP efforts to guide the projects. 
Navy ERP management realized they needed to maintain much better 
control over the integrators' work. As a result, the Navy established 
the Strategy, Architecture, and Standards Group to structure and guide 
the effort across the Navy. 

Requirements Management Process Following Best Practices: 

Our review found that the ERP development team has so far followed an 
effective process for managing its requirements development. 
Documentation was readily available for us to trace selected 
requirements from the highest level to the lowest, detailed transaction 
level. This traceability allows the user to follow the life of the 
requirement both forward and backward through the documentation, and 
from origin through implementation. Traceability is also critical to 
understanding the parentage, interconnections, and dependencies among 
the individual requirements. This information in turn is critical to 
understanding the impact when a requirement is changed or deleted. 

Requirements represent the blueprint that system developers and program 
managers use to design, develop, test, and implement a system. 
Improperly defined or incomplete requirements have been commonly 
identified as a cause of system failure and systems that do not meet 
their cost, schedule, or performance goals. Without adequately defined 
requirements that have been properly reviewed and tested, significant 
risk exists that the system will need extensive and costly changes 
before it will achieve its intended capability. 

Because requirements provide the foundation for system testing, 
specificity and traceability defects in system requirements preclude an 
entity from implementing a disciplined testing process. That is, 
requirements must be complete, clear, and well documented to design and 
implement an effective testing program. Absent this, an organization is 
taking a significant risk that its testing efforts will not detect 
significant defects until after the system is placed into production. 
Industry experience indicates that the sooner a defect is recognized 
and corrected, the cheaper it is to fix. As shown in figure 2, there is 
a direct relationship between requirements and testing. 

Figure 2: Relationship between Requirements Development and Testing: 

[See PDF for image] 

[End of figure] 

Although the actual testing activities occur late in the development 
cycle, test planning can help disciplined activities reduce 
requirements-related defects. For example, developing conceptual test 
cases based on the requirements derived from the concept of operations 
and functional requirements stages can identify errors, omissions, and 
ambiguities long before any code is written or a system is configured. 
Disciplined organizations also recognize that planning testing 
activities in coordination with the requirements development process 
has major benefits. As we have previously reported,[Footnote 41] 
failure to effectively manage requirements and testing activities has 
posed operational problems for other system development efforts. 

The Navy ERP requirements identification process began with formal 
agreement among the major stakeholders on the scope of the system, 
followed by detailed, working-level business needs from user groups and 
legacy systems. The high-level business or functional requirements 
identified initially are documented in the Operational Requirements 
Document (ORD). The ORD incorporates requirements from numerous major 
DOD framework documents[Footnote 42] and defines the capabilities that 
the system must support, including business operation needs such as 
acquisition, finance, and logistics. In addition, the ORD also 
identifies the numerous policy directives to which the Navy ERP must 
conform, such as numerous DOD infrastructure systems, initiatives, and 
policies. The ORD was distributed to over 150 Navy and DOD reviewers. 
It went through seven major revisions to incorporate the comments and 
suggestions provided by the reviewers before being finalized in April 
2004. According to Navy ERP program officials, any requested role for 
the Navy ERP to perform that was not included in the ORD will not be 
supported. This is a critical decision that reduces the project's risks 
since "requirements creep" is another cause of projects that do not 
meet their cost, schedule, and performance objectives. 

We selected seven requirements[Footnote 43] from the ORD that related 
to specific Navy problem areas, such as financial reporting and asset 
management, and found that the requirements had the expected 
attributes, including the necessary detail one would normally expect to 
find for the requirement being reviewed. For example, a requirement 
stated that the ERP will provide reports of funds expended versus funds 
allocated. We found this requirement was described in a low-level 
requirement document called a Customer Input Template, which included a 
series of questions that must be addressed. The documentation further 
detailed the standard reports that were available based on the 
selection of configuration options. Further, the documentation of the 
detailed requirements identified the specific COTS screen number that 
would be used and described the screen settings that would be used when 
a screen was "activated." 

While the ORD specifies the overall capabilities of the system at a 
high level, more specific, working-level requirements also had to be 
developed to achieve a usable blueprint for configuration and testing 
of the system. To develop these lower-level requirements, the Navy ERP 
project held detailed working sessions where requirements and design 
specifications were discussed, refined, formalized, and documented. 
Each high-level requirement was broken down into its corresponding 
business processes, which in turn drove the selection of transactions 
(COTS functions) to be used for configuration of the software. For each 
selected transaction, comprehensive documentation was created to 
capture the source information that defines how and why a transaction 
must be configured. This documentation is critical for ensuring 
accurate configuration of the software, as well as for testing the 
functionality of the software after configuration. Table 3 describes 
the kinds of documentation used to maintain these lower-level detailed 
requirements. 

Table 3: Documentation for Detailed Requirements: 

Documentation name: Customer Input Templates; 
Documentation description: A series of questions completed for every 
requirement. It enforces a comprehensive review of the requirement and 
documents the reasoning behind the answers. 

Documentation name: Functional Design Specification; 
Documentation description: A detailed description for each interface. 

Documentation name: Technical Functional Script; 
Documentation description: A description of any customization that had 
to be made to a transaction. 

Documentation name: Implementation Guide; 
Documentation description: Automatically documents the actual 
configuration choices made by the developer for each transaction. 

Source: GAO, based upon information provided by the Navy. 

[End of table] 

Additionally, the Navy ERP program is using a requirements management 
tool containing a database that links each requirement from the highest 
to the lowest level and maintains the relationship between the 
requirements. This tool helps to automate the linkage between 
requirements and helps provide the project staff reasonable assurance 
that its stated processes have been effectively implemented. This 
linkage is critical to understanding the scope of any potential change. 
For example, the users can utilize the tool to (1) determine the number 
of transactions affected by a proposed change and (2) identify the 
detailed documentation necessary for understanding how this change will 
affect each business process. To further ensure that the individual 
transactions ultimately support the adopted business process, Navy ERP 
officials conducted master business scenarios[Footnote 44] or end-to- 
end process walkthroughs. This end-to-end view of the business process 
ensures that the business functionality works across the various 
subsystems of the COTS package. For instance, the requirements for a 
purchase order could be viewed simply from the vantage point of a 
logistics person or the acquisition community. However, a purchase 
order also has financial ramifications, and therefore must be posted to 
financial records, such as the general ledger. The master business 
scenarios provide a holistic review of the business process surrounding 
each transaction. 

ERP Has the Potential to Address Some of the Navy's Reported Financial 
Management Weaknesses: 

The Navy expects the new ERP project to address a number of the 
weaknesses cited in the Department of the Navy Financial Improvement 
Plan--a course of action directed towards achieving better financial 
management and an unqualified audit opinion for the Department of the 
Navy annual financial statements. According to ERP officials, the COTS 
software used for the ERP program will improve the Navy's current 
financial controls in the areas of asset visibility, financial 
reporting, and full cost accounting. However, the currently planned ERP 
is not intended to provide an all-inclusive end-to-end corporate 
solution for the Navy. 

The COTS software offers the potential for real-time asset visibility 
for the Navy, limited by two factors beyond the project's scope. First, 
items in transit fall under the authority of the U.S. Transportation 
Command (TRANSCOM). Once the Navy hands off an item to TRANSCOM, it 
does not retain visibility of that asset until it arrives at another 
Navy location. The second factor is the limited ability for 
communication with ships at sea. Once the currently planned ERP is 
fully implemented, it will cover all inventories, including inventory 
on ships. However, the data for shipboard inventory will be current 
only as of when the ship leaves port. Those data will typically not be 
updated until the ship docks in another port and can transmit updated 
information to the ERP system. This lag time for some ships could be as 
much as 3 to 4 months. While the ERP has the capability to maintain 
real-time shipboard inventory, the Navy has yet to decide whether to 
expand the scope of the ERP and build an interface with the ships, 
which could be extensive and costly, or install the ERP on the ships. 
Both options present additional challenges that necessitate thorough 
analysis of all alternatives before a decision is made. According to 
the program office, a time frame for making this critical decision has 
not been established. 

The COTS software is also intended to provide standardized government 
and proprietary financial reporting at any level within the defined 
organization. According to Navy ERP officials, full cost accounting 
will be facilitated by a software component integrated with the ERP. 
For example, the Navy expects that this component will provide up-to- 
date cost information--including labor, materials, and overhead--for 
its numerous, and often complicated, maintenance jobs. Full cost 
information is necessary for effective management of production, 
maintenance, and other activities. 

According to Navy ERP program officials, when fully operational in 
fiscal year 2011, the Navy ERP will be used by organizations comprising 
approximately 80 percent of Navy's estimated appropriated funds--after 
excluding the Marine Corps and military pay and personnel.[Footnote 45] 
Based on fiscal years' 2006 through 2011 defense planning budget, the 
Navy ERP will manage approximately $74 billion annually. The 
organizations that will use Navy ERP include the Naval Air Systems, the 
Naval Sea Systems, the Naval Supply Systems, the Space and Naval 
Warfare Systems, and the Navy Facilities Engineering Commands, as well 
as the Office of Naval Research, the Atlantic and Pacific Fleets, and 
the Strategic Systems Programs. However, the Navy ERP will not manage 
in detail all of the 80 percent. About 2 percent, or approximately $1.6 
billion, will be executed and maintained in detail by respective 
financial management systems at the aviation and shipyard depots. For 
example, when a work order for a repair of an airplane part is 
prepared, the respective financial management system at the depot will 
execute and maintain the detailed transactions. The remaining 20 
percent that the Navy ERP will not manage comprises the Navy 
Installations Command, field support activities, and others. Navy ERP 
officials have indicated that it is the Navy's intent to further expand 
the system in the future to include the aviation and shipyard depots, 
but definite plans have not yet been made. According to Navy ERP 
officials, the software has the capability to be used at the aviation 
and shipyard depots, but additional work would be necessary. For 
example, the desired functionality and related requirements--which as 
discussed above, are critical to the success of any project--would have 
to be defined for the aviation and shipyard depots. 

System Interfaces and Data Conversion Will Be Challenges: 

While the Navy's requirements management process is following 
disciplined processes and comprises one critical aspect of the overall 
project development and implementation, by itself, it is not sufficient 
to provide reasonable assurance of the ERP's success. Going forward, 
the Navy faces very difficult challenges and risks in the areas of 
developing and implementing 44 system interfaces with other Navy and 
DOD systems, and accurately converting data from the existing legacy 
systems to the ERP. As previously noted, financial management is a high-
risk area in the department and has been designated as such since 1995. 
One of the contributing factors has been DOD's inability to develop 
integrated systems. As a result, the Navy is dependent upon the 
numerous interfaces to help improve the accuracy of its financial 
management data. Navy ERP program managers have recognized the issues 
of system interfaces and data conversion in their current list of key 
risks. They have identified some actions that need to be taken to 
mitigate the risks; however, they have not yet developed the 
memorandums of agreement with the owners of the systems which the Navy 
ERP will interface. According to the Navy ERP program office, they plan 
to complete these memorandums of agreement by October 2005. 

Integrated Systems: 

One of the long-standing problems within DOD has been the lack of 
integrated systems. This is evident in the many duplicative, stovepiped 
business systems among the 4,150 that DOD reported as belonging to its 
systems environment. Lacking integrated systems, DOD has a difficult 
time obtaining accurate and reliable information on the results of its 
business operations and continues to rely on either manual reentry of 
data into multiple systems, convoluted system interfaces, or both. 
These system interfaces provide data that are critical to day-to-day 
operations, such as obligations, disbursements, purchase orders, 
requisitions, and other procurement activities. Testing the system 
interfaces in an end-to-end manner is necessary in order for the Navy 
to have reasonable assurance that the ERP will be capable of providing 
the intended functionality. 

The testing process begins with the initial requirements development 
process. Furthermore, test planning can help disciplined activities 
reduce requirements-related defects. For example, developing conceptual 
test cases based on the requirements can identify errors, omissions, 
and ambiguities long before any code is written or a system is 
configured. The challenge now before Navy ERP is to be sure its testing 
scenarios accurately reflect the activities of the "real users," and 
the dependencies of external systems. 

We previously reported[Footnote 46] that Sears and Wal-Mart, recognized 
as leading-edge inventory management companies, have automated systems 
that electronically receive and exchange standard data throughout the 
entire inventory management process, thereby reducing the need for 
manual data entry. As a result, information moves through the data 
systems with automated ordering of inventory from suppliers; receiving 
and shipping at distribution centers; and receiving, selling, and 
reordering at retail stores. Unlike DOD, which has a proliferation of 
nonintegrated systems using nonstandard data, Sears and Wal-Mart 
require all components and subsidiaries to operate within a standard 
systems framework that results in an integrated system and does not 
allow individual systems development. 

For the first deployment, the Navy has to develop interfaces that 
permit the ERP to communicate with 44 systems--27 that are Navy 
specific and 17 systems belonging to other DOD entities. Figure 3 
illustrates the numerous required system interfaces. 

Figure 3: Navy ERP Required System Interfaces: 

[See PDF for image] 

Note: See app. III for the definitions of the system acronyms. 

[End of figure] 

Long-standing problems regarding the lack of integrated systems and use 
of nonstandard data within DOD pose significant risks for the Navy ERP 
to successfully interface with these systems. Even if integration is 
successful, if the information within the 44 systems is not accurate 
and reliable, the overall information on Navy's operation provided by 
the ERP to Navy management and the Congress will not be useful in the 
decision-making process. While the Navy ERP project office is working 
to develop agreements with system owners for the interfaces and has 
been developing the functional specifications for each system, 
officials acknowledged that, as of May 2005, they are behind schedule 
in completing the interface agreements due to other tasks. The Navy ERP 
is dependent on the system owners to achieve their time frames for 
implementation. For example, the Defense Travel System (DTS)[Footnote 
47] is one of the DOD systems with which the Navy ERP is to interface 
and exchange data. DTS is currently being implemented, and any problems 
that result in a DTS schedule slippage will, in turn, affect Navy ERP's 
interface testing. 

We have previously reported that the lack of system interface testing 
has seriously impaired the operation of other system implementation 
efforts. For example, in May 2004, we reported[Footnote 48] that 
because the system interfaces for the Defense Logistics Agency's 
Business Systems Modernization (BSM) program and the Army's LMP were 
not properly tested prior to deployment, severe operational problems 
were experienced. Such problems have led BSM, LMP, and organizations 
with which they interface--such as DFAS--to perform costly manual 
reentry of transactions, which can cause additional data integrity 
problems. For example: 

* BSM's functional capabilities were adversely affected because a 
significant number of interfaces were still in development or were 
being executed manually once the system became operational. Since the 
design of system interfaces had not been fully developed and tested, 
BSM experienced problems with receipts being rejected, customer orders 
being canceled, and vendors not being paid in a timely manner. At one 
point, DFAS suspended all vendor payments for about 2 months, thereby 
increasing the risk of late payments to contractors and violations of 
the Prompt Payment Act.[Footnote 49] 

* In January 2004, the Army reported that due to an interface failure, 
LMP had been unable to communicate with the Work Ordering and Reporting 
Communications System (WORCS) since September 2003. WORCS is the means 
by which LMP communicates with customers on the status of items that 
have been sent to the depot for repair and initiates procurement 
actions for inventory items. The Army has acknowledged that the failure 
of WORCS has resulted in duplicative shipments and billings and 
inventory items being delivered to the wrong locations. Additionally, 
the LMP program office has stated that it has not yet identified the 
specific cause of the interface failure. The Army is currently entering 
the information manually, which, as noted above, can cause additional 
data integrity errors. 

Besides the challenge of developing the 44 interfaces, the Navy ERP 
must also develop the means to be compliant with DOD's efforts to 
standardize the way that various systems exchange data with each other. 
As discussed in our July 2004 report,[Footnote 50] DOD is undertaking a 
huge and complex task (commonly referred to as the Global Information 
Grid or GIG) that is intended to integrate virtually all of DOD's 
information systems, services, applications, and data into one 
seamless, reliable, and secure network. The GIG initiative is focused 
on promoting interoperability throughout DOD by building an Internet- 
like network for DOD-related operations based on common standards and 
protocols rather than on trying to establish interoperability after 
individual systems become operational. DOD envisions that this type of 
network would help ensure systems can easily and quickly exchange data 
and change how military operations are planned and executed since much 
more information would be dynamically available to users. 

DOD's plans for realizing the GIG involve building a new core network 
and information capability and successfully integrating the majority of 
its weapon systems; command, control, and communications systems; and 
business systems with the new network. The effort to build the GIG will 
require DOD to make a substantial investment in a new set of core 
enterprise programs and initiatives. To integrate systems such as the 
Navy ERP into the GIG, DOD has developed (1) an initial blueprint or 
architecture for the GIG and (2) new policies, guidance, and standards 
to guide implementation. According to project officials, the Navy ERP 
system will be designed to support the GIG. However, they face 
challenges that can result in significant cost and schedule risks 
depending on the decisions reached. One challenge is the extent to 
which other DOD applications with which the Navy ERP must exchange data 
are compliant with the GIG. While traditional interfaces with systems 
that are not GIG compliant can be developed, these interfaces may 
suboptimize the benefits expected from the Navy ERP. The following is 
one example of the difficulties faced by the Navy ERP project. 

As mentioned previously, one system that will need to exchange data 
with the Navy ERP system is DTS. However, the DTS program office and 
the Navy ERP project office hold different views of how data should be 
exchanged between the two systems. The travel authorization process 
exemplifies these differences. DTS requires that funding information 
and the associated funds be provided to DTS in advance of a travel 
authorization being processed. In effect, DTS requires that the 
financial management systems set aside the funds necessary for DTS 
operations. Once a travel authorization is approved, DTS notifies the 
appropriate financial management system that an obligation has been 
incurred. The Navy ERP system, on the other hand, only envisions 
providing basic funding information to DTS in advance, and would delay 
providing the actual funds to DTS until they are needed in order to (1) 
maintain adequate funds control, (2) ensure that the funds under its 
control are not tied up by other systems, and (3) ensure that the 
proper accounting data are provided when an entry is made into its 
system. 

According to the Software Engineering Institute (SEI), a widely 
recognized model evaluating a system of systems interoperability is the 
Levels of Information System Interoperability. This model focuses on 
the increasing levels of sophistication of system interoperability. 
According to Navy ERP officials, the GIG and the ERP effort are 
expected to accomplish the highest level of this model--enterprise- 
based interoperability. In essence, systems that achieve this level of 
interoperability can provide multiple users access to complex data 
simultaneously, data and applications are fully shared and distributed, 
and data have a common interpretation regardless of format. This is in 
contrast to traditional interface strategies, such as the one used by 
DTS. The traditional approach is more aligned with the lowest level of 
the SEI model. Data exchanged at this level rely on electronic links 
that result in a simple electronic exchange of data. 

Alignment with DOD's BEA Is a Significant Risk Factor: 

A broader challenge and risk that is out of the Navy ERP project's 
control, but could significantly affect it, is DOD's development of a 
BEA. As we recently reported,[Footnote 51] DOD's BEA still lacks many 
of the key elements of a well-defined architecture and no basis exists 
for evaluating whether the Navy ERP will be aligned with the BEA and 
whether it would be a corporate solution for DOD in its "To Be" or 
target environment. 

An enterprise architecture consists of snapshots of the enterprise's 
current environment and its target environment, as well as a capital 
investment road map for transitioning from the current to the target 
environment. The real value of an enterprise architecture is that it 
provides the necessary content for guiding and constraining system 
investments in a way that promotes interoperability and minimizes 
overlap and duplication. At this time, it is unknown what the target 
environment will be. Therefore, it is unknown what business processes, 
data standards, and technological standards the Navy ERP must align to, 
as well as what legacy systems will be transitioned into the target 
environment. 

The Navy ERP project team is cognizant of the BEA development and has 
attempted to align to prior versions of it. The project team analyzed 
the BEA requirements and architectural elements to assess Navy ERP's 
compliance. The project team mapped the BEA requirements to the Navy 
ERP functional areas and the BEA operational activities to the Navy 
ERP's business processes. The Navy ERP project team recognizes that 
architectures evolve over time, and analysis and assessments will 
continue as requirements are further developed and refined. The scope 
of the BEA and the development approach are being revised. As a result 
of the new focus, DOD is determining which products from prior releases 
of the BEA could be salvaged and used. 

Since the Navy ERP is being developed absent the benefit of an 
enterprise architecture, there is limited, if any, assurance that the 
Navy ERP will be compliant with the architecture once it becomes more 
robust in the future. Given this scenario, it is conceivable that the 
Navy ERP will be faced with rework in order to be compliant with the 
architecture, once it is defined, and as noted earlier, rework is 
expensive. At the extreme, the project could fail as the four pilots 
did. If rework is needed, the overall cost of the Navy ERP could exceed 
the Navy's current estimate of $800 million. 

Accuracy of Data Conversion Is Critical: 

The ability of the Navy to effectively address its data conversion 
challenges will also be critical to the ultimate success of the ERP 
effort. A Joint Financial Management Improvement Program 
(JFMIP)[Footnote 52] white paper on financial system data 
conversion[Footnote 53] noted that data conversion (that is, converting 
data in a legacy system to a new system) was one of the critical tasks 
necessary to successfully implement a new financial system. The paper 
further pointed out that data conversion is one of the most frequently 
underestimated tasks. 

If data conversion is done right, the new system has a much greater 
opportunity for success. On the other hand, converting data incorrectly 
or entering unreliable data from a legacy system can have lengthy and 
long-term repercussions. The adage "garbage in, garbage out" best 
describes the adverse impact. Accurately converting data, such as 
account balances, from the pilots, as well as other systems that the 
Navy ERP is to replace, will be critical to the success of the Navy 
ERP. While data conversion is identified in the Navy ERP's list of key 
risks, it is too early in the ERP system life cycle for the development 
of specific testing plans. 

However, our previous audits have shown that if data conversion is not 
done properly, it can negatively impact system efficiency. For example, 
the Army's LMP data conversion effort has proven to be troublesome and 
continues to affect business operations. As noted in our recent 
report,[Footnote 54] when the Tobyhanna Army Depot converted ending 
balances from its legacy finance and accounting system--the Standard 
Depot System (SDS)--to LMP in July 2003, the June 30, 2003, ending 
account balances in SDS did not reconcile to the beginning account 
balances in LMP. Accurate account balances are important for producing 
reliable financial reports. Another example is LMP's inability to 
transfer accurate unit-of-issue data--quantity of an item, such as each 
number, dozen, or gallon--from its legacy system to LMP. This resulted 
in excess amounts of material ordered. Similar problems could occur 
with the Navy ERP if data conversion issues are not adequately 
addressed. The agreements between the Navy ERP and the other systems 
owners, discussed previously, will be critical to effectively support 
Navy's ERP data conversion efforts. 

Additional Actions Can be Taken to Improve Management Oversight of the 
Navy ERP Effort: 

Navy officials could take additional actions to improve management 
oversight of the Navy ERP effort. For example, we found that the Navy 
does not have a mechanism in place to capture the data that can be used 
to effectively assess the project management processes. Best business 
practices indicate that a key facet of project management and oversight 
is the ability to effectively monitor and evaluate a project's actual 
performance, cost, and schedule against what was planned.[Footnote 55] 
Performing this critical task requires the accumulation of quantitative 
data or metrics that can be used to evaluate a project's performance. 
This information is necessary to understand the risk being assumed and 
whether the project will provide the desired functionality. Lacking 
such data, the ERP program management team can only focus on the 
project schedule and whether activities have occurred as planned, not 
whether the activities achieved their objectives. 

Additionally, although the Navy ERP program has a verification and 
validation function, it relies on in-house subject matter experts and 
others who are not independent to provide an assessment of the Navy ERP 
to DOD and Navy management. The use of an IV&V function is recognized 
as a best business practice and can help provide reasonable assurance 
that the system satisfies its intended use and user needs. Further, an 
independent assessment of the Navy ERP would provide information to DOD 
and Navy management on the overall status of the project, including the 
effectiveness of the management processes being utilized and 
identification of any potential risks that could affect the project 
with respect to cost, schedule, and performance. Given DOD's long- 
standing inability to implement business systems that provide users 
with the promised capabilities, an independent assessment of the ERP's 
performance is warranted. 

Quantitative Data Necessary for Assessing Whether the System Will 
Provide the Needed Functionality: 

The Navy's ability to understand the impact of the weaknesses in its 
processes will be limited because it has not determined the 
quantitative data or metrics that can be used to assess the 
effectiveness of its project management processes. This information is 
necessary to understand the risk being assumed and whether the project 
will provide the desired functionality. The Navy has yet to establish 
the metrics that would allow it to fully understand (1) its capability 
to manage the entire ERP effort; (2) how its process problems will 
affect the ERP cost, schedule, and performance objectives; and (3) the 
corrective actions needed to reduce the risks associated with the 
problems identified. Experience has shown that such an approach leads 
to rework and thrashing instead of making real progress on the project. 

SEI has found that metrics identifying important events and trends are 
invaluable in guiding software organizations to informed decisions. Key 
SEI findings relating to metrics include the following. 

* The success of any software organization depends on its ability to 
make predictions and commitments relative to the products it produces. 

* Effective measurement processes help software groups succeed by 
enabling them to understand their capabilities so that they can develop 
achievable plans for producing and delivering products and services. 

* Measurements enable people to detect trends and anticipate problems, 
thus providing better control of costs, reducing risks, improving 
quality, and ensuring that business objectives are achieved.[Footnote 
56] 

The lack of quantitative data to assess a project has been a key 
concern in other projects we have reviewed. Without such a process, 
management can only focus on the project schedule and whether 
activities have occurred as planned, not whether the activities 
achieved their objectives. Further, such quantitative data can be used 
to hold the project team accountable for providing the promised 
capability. 

Defect-tracking systems are one means of capturing quantitative data 
that can be used to evaluate project efforts. Although HHS had a system 
that captured the reported defects, we found that the system was not 
updated in a timely manner with this critical information.[Footnote 57] 
More specifically, one of the users identified a process weakness 
related to grant accounting as a problem that will affect the 
deployment of HHS's system--commonly referred to as a "showstopper." 
However, this weakness did not appear in the defect-tracking system 
until about 1 month later. As a result, during this interval the HHS 
defect-tracking system did not accurately reflect the potential 
problems identified by users, and HHS management was unable to 
determine (1) how well the system was working and (2) the amount of 
work necessary to correct known defects. Such information is critical 
when assessing a project's status. 

We have also reported[Footnote 58] that while NASA had a system that 
captured the defects that have been identified during testing, an 
analysis was not performed to determine the root causes of reported 
defects. A critical element in helping to ensure that a project meets 
its cost, schedule, and performance goals is to ensure that defects are 
minimized and corrected as early in the process as possible. 
Understanding the root cause of a defect is critical to evaluating the 
effectiveness of a process. For example, if a significant number of 
defects are caused by inadequate requirements definition, then the 
organization knows that the requirements management process it has 
adopted is not effectively reducing risks to acceptable levels. 
Analysis of the root causes of identified defects allows an 
organization to determine whether the requirements management approach 
it has adopted sufficiently reduces the risks of the system not meeting 
cost, schedule, and functionality goals to acceptable levels. Root- 
cause analysis would also help to quantify the risks inherent in the 
testing process that has been selected. 

Further, the Navy has not yet implemented an earned value management 
system, which is another metric that can be employed to better manage 
and oversee a system project. Both OMB[Footnote 59] and DOD[Footnote 
60] require the use of an earned value management system. The earned 
value management system attempts to compare the value of work 
accomplished during a given period with the work scheduled for that 
period. By using the value of completed work as a basis for estimating 
the cost and time needed to complete the program, management can be 
alerted to potential problems early in the program. For example, if a 
task is expected to take 100 hours to complete and it is 50 percent 
complete, the earned value management system would compare the number 
of hours actually spent to complete the task to the number of hours 
expected for the amount of work performed. In this example, if the 
actual hours spent equaled 50 percent of the hours expected, then the 
earned value would show that the project's resources were consistent 
with the estimate. Without an effective earned value management system, 
the Navy and DOD management have little assurance that they know the 
status of the various project deliverables in the context of progress 
and the cost incurred in completing each of the deliverables. In other 
words, an effective earned value management system would be able to 
provide quantitative data on the status of a given project deliverable, 
such as a data conversion program. Based on this information, Navy 
management would be able to determine whether the progress of the data 
conversion effort was within the expected parameters for completion. 
Management could then use this information to determine actions to take 
to mitigate risk and manage cost and schedule performance. According to 
Navy ERP officials, they intend to implement the earned value 
management system as part of the contract for the next phase of the 
project. 

Independent Assessment of Navy ERP Could Enhance DOD and Navy 
Management Oversight of the Project: 

The Navy has not established an IV&V function to provide an assessment 
of the Navy ERP to DOD and Navy management. Best business practices 
indicate that use of an IV&V function is a viable means to provide 
management reasonable assurance that the planned system satisfies its 
planned use and users. An effective IV&V review process would provide 
independent information to DOD and Navy management on the overall 
status of the project, including a discussion of any impacts or 
potential impacts to the project with respect to cost, schedule, and 
performance. These assessments involve reviewing project documentation, 
participating in meetings at all levels within the project, and 
providing periodic reports and recommendations, if deemed warranted, to 
senior management. The IV&V function[Footnote 61] should report on 
every facet of a system project such as: 

* Testing program adequacy. Testing activities would be evaluated to 
ensure they are properly defined and developed in accordance with 
industry standard and best practices. 

* Critical-path analysis. A critical path defines the series of tasks 
that must be finished in time for the entire project to finish on 
schedule. Each task on the critical path is a critical task. A critical-
path analysis helps to identify the impact of various project events, 
such as delays in project deliverables, and ensures that the impact of 
such delays is clearly understood by all parties involved with the 
project. 

* System strategy documents. Numerous system strategy documents that 
provide the foundation for the system development and operations are 
critical aspects of an effective system project. These documents are 
used for guidance in developing documents for articulating the plans 
and procedures used to implement a system. Examples of such documents 
include the Life-cycle Test Strategy, Interface Strategy, and 
Conversion Strategy. 

The IV&V reports should identify the project management weaknesses that 
increase the risks associated with the project to senior management so 
that they can be promptly addressed. The Navy ERP program's approach to 
the verification and validation of its project management activities 
relies on in-house subject matter experts and others who work for the 
project team's Quality Assurance leader. The results of these efforts 
are reported to the project manager. While various approaches can be 
used to perform this function, such as using the Navy's approach or 
hiring a contractor to perform these activities, independence is a key 
component to successful verification and validation activities. The 
system developer and project management office may have vested 
interests and may not be objective in their self-assessments. 
Accordingly, performing verification and validation activities 
independently of the development and management functions helps to 
ensure that verification and validation activities are unbiased and 
based on objective evidence. The Navy's adoption of verification and 
validation processes is a key component of its efforts to implement the 
disciplined processes necessary to manage this project. However, Navy 
and DOD management cannot obtain reasonable assurance that the 
processes have been effectively implemented since the present 
verification and validation efforts are not conducted by an independent 
party. 

In response to the Ronald W. Reagan National Defense Authorization Act 
for Fiscal Year 2005,[Footnote 62] DOD has established a hierarchy of 
investment review boards from across the department to improve the 
control and accountability over business system investments.[Footnote 
63] The boards are responsible for reviewing and approving investments 
to develop, operate, maintain, and modernize business systems for their 
respective business areas.[Footnote 64] The various boards are to 
report to the Defense Business Systems Management Committee (DBSMC), 
which is ultimately responsible for the review and approval of the 
department's investments in its business systems. To help facilitate 
this oversight responsibility, the reports prepared by the IV&V 
function should be provided to the appropriate investment review board 
and the DBSMC to assist them in the decision-making process regarding 
the continued investment in the Navy ERP. The information in the 
reports should provide reasonable assurance that an appropriate rate of 
return is received on the hundreds of millions of dollars that will be 
invested over the next several years and the Navy ERP provides the 
promised capabilities. 

To help ensure that the Navy ERP achieves its cost, schedule, and 
performance goals, the investment review should employ an early warning 
system that enables it to take corrective action at the first sign of 
slippages. Effective project oversight requires having regular reviews 
of the project's performance against stated expectations and ensuring 
that corrective actions for each underperforming project are 
documented, agreed to, implemented, and tracked until the desired 
outcome is achieved. 

Conclusions: 

The lack of management control and oversight and a poorly conceived 
concept resulted in the Navy largely wasting about $1 billion on four 
ERP system projects that had only a limited positive impact on the 
Navy's ability to produce reliable, useful, and timely information to 
aid in its day-to-day operations. The Navy recognizes that it must have 
the appropriate management controls and processes in place to have 
reasonable assurance that the current effort will be successful. While 
the current requirements management effort is adhering to the 
disciplined processes, the overall effort is still in the early stages 
and numerous challenges and significant risks remain, such as 
validating data conversion efforts and developing numerous systems 
interfaces. Given that the current effort is not scheduled to be 
complete until 2011 and is currently estimated by the Navy to cost 
about $800 million, it is incumbent upon Navy and DOD management to 
provide the vigilant oversight that was lacking in the four pilots. 
Absent this oversight, the Navy and DOD run a higher risk than 
necessary of finding, as has been the case with many other DOD business 
systems efforts, that the system may cost more than anticipated, take 
longer to develop and implement, and does not provide the promised 
capabilities. In addition, attempting large-scale systems modernization 
programs without a well-defined architecture to guide and constrain 
business systems investments, which is the current DOD state, presents 
the risk of costly rework or even system failure once the enterprise 
architecture is fully defined. Considering (1) the large investment of 
time and money essentially wasted on the pilots and (2) the size, 
complexity, and estimated costs of the current ERP effort, the Navy can 
ill afford another business system failure. 

Recommendations for Executive Action: 

To improve the Navy's and DOD's oversight of the Navy ERP effort, we 
recommend that the Secretary of Defense direct the Secretary of the 
Navy to require that the Navy ERP Program Management Office (1) develop 
and implement the quantitative metrics needed to evaluate project 
performance and risks and use the quantitative metrics to assess 
progress and compliance with disciplined processes and (2) establish an 
IV&V function and direct that all IV&V reports be provided to Navy 
management and to the appropriate DOD investment review board, as well 
as the project management. 

Furthermore, given the uncertainty of the DOD business enterprise 
architecture, we recommend that the Secretary of Defense direct the 
DBSMC to institute semiannual reviews of the Navy ERP to ensure that 
the project continues to follows the disciplined processes and meets 
its intended costs, schedule, and performance goals. Particular 
attention should be directed towards system testing, data conversion, 
and development of the numerous system interfaces with the other Navy 
and DOD systems. 

Agency Comments and Our Evaluation: 

We received written comments on a draft of this report from the Deputy 
Under Secretary of Defense (Financial Management) and the Deputy Under 
Secretary of Defense (Business Transformation), which are reprinted in 
appendix II. While DOD generally concurred with our recommendations, it 
took exception to our characterization that the pilots were failures 
and a waste of $1 billion. 

Regarding the recommendations, DOD agreed that it should develop and 
implement quantitative metrics that can be used to evaluate the Navy 
ERP and noted that it intends to have such metrics developed by 
December 2005. The department also agreed that the Navy ERP program 
management office should establish an IV&V function and noted that the 
IV&V team will report directly to the program manager. We continue to 
reiterate the need for the IV&V to be completely independent of the 
project. As noted in the report, performing IV&V activities 
independently of the development and management functions helps to 
ensure that the results are unbiased and based on objective evidence. 
Further, rather than having the IV&V reports provided directly to the 
appropriate DOD investment review boards as we recommended, DOD stated 
that the Navy management and/or the project management office shall 
inform the Office of the Under Secretary of Defense for Business 
Transformation of any significant IV&V results. We reiterate our 
support for the recommendation that the IV&V reports be provided to the 
appropriate investment review board so that it can determine whether 
any of the IV&V results are significant. Again, by providing the 
reports directly to the appropriate investment review board, we believe 
there would be added assurances that the results were objective and 
that the managers who will be responsible for authorizing future 
investments in the Navy ERP will have the information needed to make 
the most informed decision. 

With regard to the reviews by the DBSMC, DOD partially agreed. Rather 
than semiannual reviews by the DBSMC as we recommended, the department 
noted that the components (e.g., the Navy) would provide briefings on 
their overall efforts, initiatives, and systems during meetings with 
the DBSMC. Given the significance of the Navy ERP, in terms of dollars 
and its importance to the overall transformation of the department's 
business operations, and the failure of the four ERP pilots, we 
continue to support more proactive semiannual reviews by the DBSMC. As 
noted in the report, the Navy's initial estimate is that the ERP will 
cost at least $800 million, and given the department's past 
difficulties in effectively developing and implementing business 
systems, substantive reviews by individuals outside of the program 
office that are focused just on the Navy ERP by the highest levels of 
management within the department are warranted. Further, we are 
concerned that the briefings contemplated to the DBSMC may not 
necessarily discuss the Navy ERP, nor provide the necessary detailed 
discussions to offer the requisite level of confidence and assurance 
that the project continues to follow disciplined processes with 
particular attention to numerous challenges, such as system interfaces 
and system testing. 

In commenting on the report, the department depicted the pilots in a 
much more positive light than we believe is merited. DOD pointed out 
that it viewed the pilots as successful, exceeding initial 
expectations, and forming the foundation upon which to build a Navy 
enterprise solution, and took exception to our characterization that 
the pilots were failures and largely a waste of $1 billion. As 
discussed in the report, the four pilots were narrow in scope, and were 
never intended to be a corporate solution for resolving any of the 
Navy's long-standing financial and business management problems. We 
characterized the pilots as failures because the department spent $1 
billion on systems that did not result in marked improvement in the 
Navy's day-to-day operations. While there may have been marginal 
improvements, it is difficult to ascertain the sustained, long-term 
benefits that will be derived by the American taxpayers for the $1 
billion. 

Additionally, the pilots present an excellent case study as to why the 
centralization of the business systems funding would be an appropriate 
course of action for the department, as we have previously 
recommended.[Footnote 65] Each Navy command was allowed to develop an 
independent solution that focused on its own parochial interest. There 
was no consideration as to how the separate efforts fit within an 
overall departmental framework, or, for that matter, even a Navy 
framework. As noted in table 2, the pilots performed many of the same 
functions and used the same software, but yet were not interoperable 
because of the various inconsistencies in the design and 
implementation. 

Because the department followed the status quo, the pilots, at best, 
provided the department with four more stovepiped systems that perform 
duplicate functions. Such investments are one reason why the department 
reported in February 2005[Footnote 66] that it had 4,150 business 
systems. Further, in its comments the department noted one of the 
benefits of the pilots was that they "proved that the Navy could 
exploit commercial ERP tools without significant customization." Based 
upon our review and during discussions with the program office, just 
the opposite occurred in the pilots. Many portions of the pilots' COTS 
software were customized to accommodate the existing business 
processes, which negated the advantages of procuring a COTS package. 
Additionally, the department noted that one of the pilots--SMART, on 
which, as noted in our report, the Navy spent approximately $346 
million through September 30, 2004--has already been retired. We 
continue to question the overall benefit that the Navy and the 
department derived from these four pilots and the $1 billion it spent. 

As agreed with your offices, unless you announce the contents of this 
report earlier, we will not distribute it until 30 days after its 
issuance date. At that time, we will send copies to the Chairmen and 
Ranking Minority Members, Senate Committee on Armed Services; Senate 
Committee on Homeland Security and Governmental Affairs; Subcommittee 
on Defense, Senate Committee on Appropriations; House Committee on 
Armed Services; House Committee on Government Reform; and Subcommittee 
on Defense, House Committee on Appropriations. We are also sending 
copies to the Under Secretary of Defense (Comptroller); the Under 
Secretary of Defense (Acquisition, Technology and Logistics); the Under 
Secretary of Defense (Personnel and Readiness); the Assistant Secretary 
of Defense (Networks and Information Integration); and the Director, 
Office of Management and Budget. Copies of this report will be made 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at [Hyperlink, 
http://www.gao.gov]. If you or your staff have any questions on matters 
discussed in this report, please contact Gregory D. Kutz at (202) 512-
9505 or [Hyperlink, kutzg@gao.gov] or Keith A. Rhodes at (202) 512-6412 
or [Hyperlink, rhodesk@gao.gov]. Key contributors to this report are 
listed in appendix IV. Contact points for the Offices of Congressional 
Relations and Public Affairs are shown on the last page of the report. 

Signed by: 

Gregory D. Kutz: 
Managing Director: 
Forensic Audits and Special Investigations: 

Signed by: 

Keith A. Rhodes: 
Chief Technologist: 
Applied Research and Methodology: 
Center for Engineering and Technology: 

List of Requesters: 

The Honorable Tom Davis: 
Chairman: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Christopher H. Shays: 
Chairman: 
Subcommittee on National Security, Emerging Threats, and International 
Relations: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Todd R. Platts: 
Chairman: 
Subcommittee on Government Management, Finance and Accountability: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Adam H. Putnam: 
House of Representatives: 

[End of section] 

Appendixes: 

Appendix I: Scope and Methodology: 

To obtain a historical perspective on the planning and costs of the 
Navy's four Enterprise Resource Planning (ERP) pilot projects, and the 
decision to merge them into one program, we reviewed the Department of 
Defense's (DOD) budget justification materials and other background 
information on the four pilot projects. We also reviewed Naval Audit 
Service reports on the pilots. In addition, we interviewed Navy ERP 
program management and DOD Chief Information Officer (CIO) officials 
and obtained informational briefings on the pilots. 

To determine if the Navy has identified lessons learned from the 
pilots, how they are being used, and the challenges that remain, we 
reviewed program documentation and interviewed Navy ERP program 
officials. Program documentation that we reviewed included concept of 
operations documentation, requirements documents, the testing strategy, 
and the test plan. In order to determine whether the stated 
requirements management processes were effectively implemented, we 
performed an in-depth review and analysis of seven requirements that 
relate to the Navy's problem areas, such as financial reporting and 
asset management, and traced them through the various requirements 
documents. These requirements were selected in a manner that ensured 
that the requirements selected were included in the Navy's Financial 
Improvement Plan. 

Our approach to validating the effectiveness of the requirements 
management process relied on a selection of seven requirements from 
different functional areas. From the finance area, we selected the 
requirement to provide reports of funds expended versus funds 
allocated. From the intermediate-level maintenance management area, we 
selected the requirement related to direct cost per job and forecasting 
accuracy. From the procurement area, we selected the requirement to 
enable monitoring and management of cost versus plan. In the plant 
supply functions area, we reviewed the requirement related to total 
material visibility and access of material held by the activity and the 
enterprise. From the wholesale supply functions area, we selected the 
requirements of in-transit losses/in-transit write-offs and total 
material visibility and access of material held by the activity and the 
enterprise. Additionally, we reviewed the requirement that the ERP be 
compliant with federal mandates and requirements and the U.S. Standard 
General Ledger. 

In order to provide reasonable assurance that our test results for the 
selected requirements reflected the same processes used to document all 
requirements, we did not notify the project office of the specific 
requirements we had chosen until the tests were conducted. Accordingly, 
the project office had to be able to respond to a large number of 
potential requests rather than prepare for the selected requirements in 
advance. Additionally, we obtained the list of systems the Navy ERP 
will interface with and interviewed selected officials responsible for 
these systems to determine what activities the Navy ERP program office 
is working with them on and what challenges remain. 

To determine if there are additional business practices that could be 
used to improve management oversight of the Navy ERP, we reviewed 
industry standards and best practices from the Institute of Electrical 
and Electronics Engineers, the Software Engineering Institute, the 
Joint Financial Management Improvement Program, GAO executive guides, 
and prior GAO reports. Given that the Navy ERP effort is still in the 
early stages of development, we did not evaluate all best practices. 
Rather, we concentrated on those that could have an immediate impact in 
improving management's oversight. We interviewed Navy ERP program 
officials and requested program documentation to determine if the Navy 
ERP had addressed or had plans for addressing these industry standards 
and best practices. 

We did not verify the accuracy and completeness of the cost information 
provided by DOD for the four pilots or the Navy ERP effort. We 
conducted our work from August 2004 through June 2005 in accordance 
with U.S. generally accepted government auditing standards. 

We requested comments on a draft of this report from the Secretary of 
Defense or his designee. We received written comments on a draft of the 
report from the Deputy Under Secretary of Defense (Financial 
Management) and the Deputy Under Secretary of Defense (Business 
Transformation), which are reprinted in appendix II. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

OFFICE OF THE UNDER SECRETARY OF DEFENSE: 
ACQUISITION TECHNOLOGY AND LOGISTICS: 
3000 DEFENSE PENTAGON: 
WASHINGTON, DC 20301-3000: 

SEP 13 2005: 

Mr. Gregory D. Kutz: 
Managing Director, 
Forensic Audits and Special Investigations: 

Mr. Keith A. Rhodes: 
Chief Technologist, 
Applied Research And Methodology: 
Center For Engineering And Technology: 
United States Government Accountability Office: 
441 G Street, N.W.: 
WASHINGTON, D.C. 20548: 

Dear Sirs: 

This is the Department of Defense (DoD) response to the Government 
Accountability Office (GAO) Draft Report, "DOD BUSINESS SYSTEMS 
MODERNIZATION: Navy ERP Adherence to Best Business Practices Critical 
to Avoid Past Failures" dated 11 August 2005, (GAO Code 192171/GAO-05- 
858). While the Department generally concurs with the two 
recommendations in the report, we take strong exception to the 
characterization in the executive summary and throughout the report of 
the Navy pilots as being failures and a waste of $1 B. 

The Department views the pilots as successful, exceeding initial 
expectations, and forming the foundation upon which to build a Navy 
Enterprise solution. Your report recognizes that the pilots were 
intended to be limited in scope and not designed to be corporate 
solutions yet inconsistently concludes that they were failures and a 
waste of $1B. 

The SIGMA pilot, now operational at the Naval Air Systems Command, has 
over 15,000 users; processes over 4,700 funding documents and 139 
contract actions each week; and to date, has accounted for $165B in 
obligations and $129B in expenditures. It is the financial system of 
record for this major command. Additionally, SIGMA received the 2005 
Americas' SAP Users' Group (ASUG) Impact Award for achieving strategic 
business results, the first time such and award has been given to a 
government agency. 

The NEMAIS pilot, also operational, has resulted in standardized 
planning and monitoring of Intermediate level maintenance across 
multiple global regions and the entire non-nuclear Navy ship and 
submarine community. These standardized business practices permit the 
analyses and correction of inefficiencies not possible prior to the 
implementation of NEMAIS throughout the Regional Maintenance Centers 
(RMC). NEMAIS provided the tool necessary for the successful 
unification of the global RMC's providing homogeneous maintenance 
processes throughout the previously disparate Commands. 

NEMAIS has unified the efforts of approximately 11,000 personnel 
involved in intermediate level ships' maintenance and has significantly 
assisted in Navy surging 75% of the Fleet in support of Operation Iraqi 
Freedom and will be crucial in rapid reconstitution of the Fleet at the 
conclusion of Iraqi Freedom. NEMAIS will provide the methodology to 
easily and concisely extract the FY05 fiscal position in Navy ships' 
intermediate repair at a level of precision and accuracy not previously 
attainable through the various stovepipe legacy systems. NEMAIS 
provides the solution for sensitive data within the Department of Navy 
and is the foundation for Navy's maintenance solution. 

The CABRILLO pilot was developed and deployed as the financial system 
of record for the Space and Naval Warfare Systems Center, San Diego, 18 
months after initiation. It currently serves over 3,700 people and is 
used to manage an annual operating budget of over $1.4B. 

While the final pilot, SMART has been retired, it produced the 
foundation for supply chain management processes that have been 
incorporated into the blueprint for the converged Enterprise Resource 
Planning (ERP) program. The four pilot projects have provided 
invaluable insight into processes and costs for maintenance, supply, 
financial and work-force management. This converged solution will be 
the cornerstone of the Department of the Navy's financial improvement 
roadmap, supporting a better integrated and auditable business 
processing environment that will ultimately allow us to comply with the 
intent of the Chief Financial Officer's act. 

For clarification, 35% or $350M of the $1B referred to in the draft 
report was for investment in development of the pilots, while 55% was 
for operating and support expenses, and the remaining 10% for the early 
definition of the converged Navy ERP solution. The sustainment expenses 
would have been incurred on legacy systems in either case. 

To characterize these investments as failures and a waste of $1B is 
inconsistent with the facts presented. The success of the Navy pilots 
proved that the Navy could exploit commercial ERP tools without 
significant customization, and that further business process 
improvements and return on investment were feasible - making a Navy 
wide enterprise solution an achievable goal. These investments and the 
lessons learned have significantly reduced the risks and uncertainty of 
undertaking such an ambitious program. 

It should also be noted that the Navy did reduce the scope and 
functionality originally planned for the NEMAIS solution as the Navy 
moved toward the integrated Navy ERP solution. PBD-022 (Program Budget 
Decision 22 from Program Objective Memorandum 2004 (POM04)) and the 
NEMAIS Acquisition Decision Memorandum (ADM) have restricted any 
further development and deployments. It is noted that the GAO audit has 
no mention of PBD-022 in the report. Footnote 27 of the report should 
indicate that PBD-022 was the decision point for NEMAIS to become an 
Acquisition Category (ACAT) I AM program even with the reduced scope. 

We appreciate the opportunity to provide comments on the draft report 
and look forward to on-going engagement and discussion with the GAO as 
we continue the development and deployment of the Navy ERP program and 
the business transformation it will bring to Navy. 

Signed by: 

Thomas Modly: 
Deputy Under Secretary of Defense (Financial Management)

Paul Brinkley: 
Deputy Under Secretary of Defense (Business Transformation): 

Enclosure: As stated: 

GAO Code 192171/GAO-05-858: 

"DOD BUSINESS SYSTEMS MODERNIZATION: Navy ERP Adherence to Best 
Practices Critical to Avoid Past Failures" 

DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS: 

RECOMMENDATION 1: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to require that the Navy ERP Program 
Management Office develop and implement the quantitative metrics needed 
to evaluate project performance and risks and use the quantitative 
measures to assess progress and compliance with disciplined processes. 

DOD RESPONSE: Concur. The Navy ERP Direct Reporting Program Manager 
(DPRM) is activating quantitative metrics related to configuration 
control, development progress, earned value, and quality. These metrics 
are targeted to be operational by December 2005. Moreover the DRPM is 
preparing for robust tracking to prevent repetitive occurrences of 
similar defects. 

RECOMMENDATION 2: The GAO recommended that the Secretary of Defense 
direct the Secretary of the Navy to require that the Navy ERP Program 
Management Office [a] establish an independent verification and 
validation function and [b] direct that all IV&V reports be provided to 
Navy management and the appropriate DOD investment review board as well 
as the project management. The GAO also recommended [c] that the 
Defense Business Systems Management Committee institute semiannual 
reviews of the Navy ERP to ensure that the project continues to follows 
the discipline processes and meets its intended costs, schedule and 
performance goals. Particular attention should be directed towards the 
system testing, data conversion, and development of the numerous system 
interfaces with the other Navy and DOD systems. 

DOD RESPONSE: Part [a], Concur. The DRPM is establishing an independent 
verification and validation (IV&V) team within its organization which 
will report directly to the Program Manager (PM). This is expected to 
be in place and operational in September 2005. Part [b], Partially 
concur. The Department endorses tiered accountability and is holding 
the Components accountable for the management and execution of business 
system transformation. IV&V reports will be provided directly to the 
Navy ERP PM and the Navy Component Acquisition Executive. The Component 
Executive and/or the Navy ERP PM shall inform the OSD Business 
Transformation Office of any significant IV&V results which it believes 
will result in a breach of the established program baseline. Part [c], 
Partially concur. The Defense Business Systems Management Committee 
(DBSMC) meetings include Transformation briefings from each of the 
Components that contain their overall efforts, initiatives and systems. 
The DBSMC is also conducting reviews of DoD enterprise level business 
programs. 

[End of section] 

Appendix III: Identification of the Navy and Defense Systems That Must 
Interface with the ERP: 

Navy systems: 

Acronym: AIT; 
System name: Automated Information Technology. 

Acronym: CAV II; 
System name: Commercial Asset Visibility System. 

Acronym: CDF; 
System name: Consolidated Data File. 

Acronym: CDMD-OA; 
System name: Configuration Data Manager's Database - Open Architecture. 

Acronym: CRCS-CADS; 
System name: Common Rates Computation System/Common Allowance 
Development System. 

Acronym: DONIBIS; 
System name: Department of the Navy Industrial Budget Information 
System. 

Acronym: G02APU; 
System name: G02 Annual Pricing Update. 

Acronym: ITIMP; 
System name: Integrated Technical Item Management & Procurement. 

Acronym: Manugistics; 
System name: Manugistics. 

Acronym: MSWP; 
System name: Maintenance and Ship Work Planning. 

Acronym: NALCOMIS OMA & OOMA; 
System name: Naval Aviation Logistic Command Management Information 
System (2 different versions). 

Acronym: NALDA; 
System name: Naval Aviation Logistics Data Analysis. 

Acronym: Navy Data Mart; 
System name: Defense Civilian Personnel Data System. 

Acronym: NDMS; 
System name: NAVAIR Depot Maintenance System. 

Acronym: NES; 
System name: Navy Enlisted Personnel System. 

Acronym: One Touch; 
System name: One Touch Supply System. 

Acronym: OPINS; 
System name: Officer Personnel Information System. 

Acronym: PBIS; 
System name: Program Budget Information System. 

Acronym: RMAIS; 
System name: Regional Maintenance Automated Information System. 

Acronym: SAS; 
System name: SAS Activity-Based Management. 

Acronym: SDRS; 
System name: Supply Discrepancy Reporting System. 

Acronym: SKED; 
System name: Preventative Maintenance Scheduling Program. 

Acronym: SLDP; 
System name: Standard Logistics Data Procedures. 

Acronym: TDSA; 
System name: Technical Directive Status Accounting System. 

Acronym: TFMMS; 
System name: Total Force Manpower Management System. 

Acronym: UADPS-SP/U2; 
System name: Uniform Automated Data Processing System - Stock Points. 

DOD Systems: 

Acronym: ADS; 
System name: DFAS Corporate Database. 

Acronym: APVM; 
System name: Accounting Pre-Validation Module. 

Acronym: CCR; 
System name: Central Contractor Registration System. 

Acronym: DAAS; 
System name: Defense Automatic Addressing System. 

Acronym: DCAS; 
System name: Defense Cash Accountability System. 

Acronym: DCPS; 
System name: Defense Civilian Pay System. 

Acronym: DDRS; 
System name: Defense Departmental Reporting System. 

Acronym: DTS; 
System name: Defense Travel System. 

Acronym: FLIS; 
System name: Federal Logistics Information System. 

Acronym: FRS; 
System name: Financial Reporting System - Accounting. 

Acronym: ICAPS; 
System name: Interactive Computer Aided Provisioning System. 

Acronym: MISIL; 
System name: Management Information System for International Logistics. 

Acronym: PPVM; 
System name: Payment Pre-Validation Module. 

Acronym: SPS; 
System name: Standard Procurement System. 

Acronym: VPIS; 
System name: Vendor Pay Inquiry System. 

Acronym: WAWF; 
System name: Wide Area Workflow. 

Acronym: WINS; 
System name: Web Based Invoicing System. 

Source: Navy ERP program office. 

[End of table] 

[End of section] 

Appendix IV: GAO Contacts and Acknowledgments: 

GAO Contacts: 

Gregory D. Kutz (202) 512-9505 or [Hyperlink, kutzg@gao.gov]: 

Keith A. Rhodes (202) 512-6412 or [Hyperlink, rhodesk@gao.gov]: 

Acknowledgments: 

In addition to the contacts above, Darby Smith, Assistant Director; J. 
Christopher Martin, Senior Level Technologist; Francine DelVecchio; 
Kristi Karls; Jason Kelly; Mai Nguyen; and Philip Reiff made key 
contributions to this report. 

(192171): 

FOOTNOTES 

[1] GAO, DOD Business Systems Modernization: Billions Being Invested 
without Adequate Oversight, GAO-05-381 (Washington, D.C.: Apr. 29, 
2005). 

[2] GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: 
January 2005). The eight specific DOD high-risk areas are (1) approach 
to business transformation, (2) business systems modernization, (3) 
contract management, (4) financial management, (5) personnel security 
clearance program, (6) supply chain management, (7) support 
infrastructure management, and (8) weapon systems acquisition. The six 
governmentwide high-risk areas that include DOD are: (1) disability 
programs, (2) interagency contracting, (3) information systems and 
critical infrastructure, (4) information sharing for homeland security, 
(5) human capital, and (6) real property. 

[3] An ERP solution is an automated system using commercial off-the- 
shelf (COTS) software consisting of multiple, integrated functional 
modules that perform a variety of business-related tasks such as 
payroll, general ledger accounting, and supply chain management. 

[4] The 80 percent is calculated after excluding estimated appropriated 
funding for the Marine Corps and military personnel and pay. Of the 80 
percent, about 2 percent of the appropriated funds will be executed and 
maintained in detail by the financial management systems at the 
aviation and shipyard depots. 

[5] According to the Software Engineering Institute, requirements 
management is a process that establishes a common understanding between 
the customer and the software project manager regarding the customer's 
business needs that will be addressed by a project. A critical part of 
this process is to ensure that the requirements development portion of 
the effort documents, at a sufficient level of detail, the problems 
that need to be solved and the objectives that need to be achieved. 

[6] An interface is a connection between two devices, applications, or 
networks or a boundary across which two systems communicate. 

[7] In July 2001, the Secretary of Defense established the Business 
Management Modernization Program to improve the efficiency and 
effectiveness of DOD business operations and to provide the 
department's leaders with accurate and timely information through the 
development and implementation of a business enterprise architecture. 

[8] GAO, DOD Business Systems Modernization: Long-standing Weaknesses 
in Enterprise Architecture Development Need to Be Addressed, GAO-05-702 
(Washington, D.C.: July 22, 2005). 

[9] Department of the Navy, Fiscal Year 2004 Annual Financial Report 
(November 2004). Numbers are combined from the fiscal year 2004 General 
Fund and Working Capital Fund financial reports. 

[10] This includes the Marine Corps. 

[11] Status of Financial Management Reform Within the Department of 
Defense and the Individual Services: Hearing Before the Subcommittee on 
Readiness and Management Support, Senate Armed Services Committee, 
108TH Cong. 2 (Nov. 18, 2004) (statement by the Assistant Secretary of 
the Navy (Financial Management and Comptroller), Richard Greco, Jr.) 

[12] The eight material weaknesses for the Navy General Fund are 
related to: (1) accounting and financial management systems; (2) fund 
balance with Treasury; (3) accounts receivable; (4) inventory and 
related property; (5) general property, plant, and equipment; (6) 
accounts payable; (7) environmental liabilities; and (8) problem 
disbursements. 

[13] The six material weaknesses for the Navy Working Capital Fund are 
related to: (1) accounting and financial management systems; (2) fund 
balance with Treasury; (3) accounts receivable; (4) inventory and 
related property; (5) general property, plant, and equipment; and (6) 
accounts payable. 

[14] FFMIA, Pub. L. No. 104-208, div. A., §101(f), title VIII, 110 
Stat. 3009, 3009-389 (Sept. 30, 1996), requires the 24 major 
departments and agencies covered by the Chief Financial Officers Act of 
1990, Pub. L. No. 101-576, 104 Stat. 2838 (Nov. 15, 1990) (31 U.S.C. § 
901(b), as amended), to implement and maintain financial management 
systems that comply substantially with (1) federal financial management 
systems requirements, (2) applicable federal accounting standards, and 
(3) the U.S. Standard General Ledger at the transaction level. 

[15] GAO, Vendor Payments: Inadequate Management Oversight Hampers the 
Navy's Ability to Effectively Manage Its Telecommunication Program, GAO-
04-671 (Washington, D.C.: June 14, 2004). 

[16] GAO, Foreign Military Sales: Improved Navy Controls Could Prevent 
Unauthorized Shipment of Classified and Controlled Spare Parts to 
Foreign Countries, GAO-04-507 (Washington, D.C.: June 25, 2004). 

[17] GAO, Defense Inventory: Navy Needs to Improve the Management Over 
Government-Furnished Material Shipped to Its Repair Contractors, GAO- 
04-779 (Washington, D.C.: July 23, 2004). 

[18] Naval Audit Service, Erroneous Payments Made to Navy Vendors, 
N2005-0011 (Washington, D.C.: Dec. 2, 2004). 

[19] Initially the focus of the committee was on financial practices, 
and it was named the Commercial Financial Practices Working Group. The 
committee changed its name to reflect a broader focus from financial 
practices to business practices. 

[20] The amount is based on DOD's estimated planning budget for fiscal 
years 2006 through 2011. 

[21] GAO, Department of Defense: Financial and Business Management 
Transformation Hindered by Long-standing Problems, GAO-04-941T 
(Washington, D.C.: July 8, 2004). 

[22] A configuration point is a place at which the system developer 
must define the business flows with inputs, conditions, and criteria 
that will be used in the application. 

[23] Roles are the actions and activities assigned to or required or 
expected of a person or group. A user's access to the transactions, 
reports, and applications is controlled by the roles assigned to the 
user. 

[24] Disciplined processes include a wide range of activities, 
including project planning and oversight, requirements management, risk 
management, and testing. 

[25] Department of Defense Directive 5000.1, The Defense Acquisition 
System, DOD Instruction 5000.2, Operation of the Defense Acquisition 
System (Apr. 5, 2002) and DOD Regulation 5000.2-R, Mandatory Procedures 
for Major Defense Acquisition Programs and Major Automated Information 
System Acquisition Programs (Apr. 5, 2002). The DOD policy also 
specifies that the DOD CIO is the milestone decision authority, 
responsible for program approval, for all major automated information 
systems. 

[26] Naval Audit Service, Department of the Navy Implementation of 
Enterprise Resource Planning Solutions, N2002-0024 (Washington, D.C.: 
Jan. 25, 2002). 

[27] Subsequent to the Naval Audit Service's report, on July 21, 2003, 
NEMAIS was designated a major automated information system. This 
memorandum also allowed the fielding of NEMAIS to four Navy regions. 

[28] DOD, Operation of the Defense Acquisition System, DOD Instruction 
5000.2, (Apr. 5, 2002); and DOD Regulation 5000.2-R, Mandatory 
Procedures for Major Defense Acquisition Programs and Major Automated 
Information System Acquisition Programs (Apr. 5, 2002). The acquisition 
controls described in this guidance apply to all DOD acquisition 
programs. 

[29] Naval Audit Service N2002-0024. 

[30] The mission needs statement has been replaced by the requirement 
for an Initial Capabilities Document. 

[31] Acceptable levels refer to the fact that any systems acquisition 
effort will have risks and will suffer the adverse consequences 
associated with defects in the processes. However, effective 
implementation of the disciplined processes reduces the potential of 
risks actually occurring and prevents significant defects from 
materially affecting the cost, timeliness, and performance of the 
project. 

[32] GAO, Business Modernization: Improvements Needed in Management of 
NASA's Integrated Financial Management Program, GAO-03-507 (Washington, 
D.C.: Apr. 30, 2003). 

[33] GAO, National Aeronautics and Space Administration: Significant 
Actions Needed to Address Long-standing Financial Management Problems, 
GAO-04-754T (Washington, D.C.: May 19, 2004). 

[34] GAO, DOD Business Systems Modernization: Billions Continue to Be 
Invested with Inadequate Management Oversight and Accountability, GAO- 
04-615 (Washington, D.C.: May 27, 2004). 

[35] GAO, Army Depot Maintenance: Ineffective Oversight of Depot 
Maintenance Operations and System Implementation Efforts, GAO-05-441 
(Washington, D.C.: June 30, 2005). 

[36] GAO, DOD Systems Modernization: Management of Integrated Military 
Human Capital Program Needs Additional Improvements, GAO-05-189 
(Washington, D.C.: Feb. 11, 2005). 

[37] Steve McConnell, Rapid Development: Taming Wild Software Schedules 
(Redmond, Wash.: Microsoft Press, 1996). 

[38] GAO-05-702. 

[39] A transaction format is a logical process, as defined by the ERP. 
From the user's point of view, a transaction is a self-contained unit, 
such as changing an address for a customer or executing a program. 

[40] A system integrator is a company that specializes in enabling an 
organization to use off-the-shelf hardware and software packages to 
meet the organization's computing needs. 

[41] See, for example, GAO-04-615 and GAO, Financial Management 
Systems: Lack of Disciplined Processes Puts Implementation of HHS' 
Financial System at Risk, GAO-04-1008 (Washington, D.C.: Sept. 23, 
2004). 

[42] The documents include: DOD Financial Bluebook; Global Information 
Grid Capstone Requirements Document (CRD) Crosswalk; Global Combat 
Support System CRD Crosswalk; Joint Deployment Systems CRD Crosswalk; 
DoD Joint Technical Architecture; DOD 8500.1, Information Assurance; 
DOD 8510.1-M, Information Technology Security Certification and 
Accreditation Process; and DOD 5000.1, The Defense Acquisition System. 

[43] As discussed in appendix I, our approach to validating the 
effectiveness of the requirements management process relied on a 
selection of seven requirements from different functional areas. From 
the finance area, we selected the requirement to provide reports of 
funds expended versus funds allocated. From the intermediate-level 
maintenance management area, we selected the requirement related to 
direct cost per job and forecasting accuracy. From the procurement 
area, we selected the requirement to enable monitoring and management 
of cost versus plan. In the plant supply functions area, we reviewed 
the requirement related to total material visibility and access of 
material held by the activity and the enterprise. From the wholesale 
supply functions area, we selected the requirements of in-transit 
losses/in-transit write-offs and total material visibility and access 
of material held by the activity and the enterprise. Additionally, we 
reviewed the requirement that the ERP be compliant with federal 
mandates and requirements and the U.S. Standard General Ledger. 

[44] A business scenario is a description of the flow of business 
processes that runs within a particular area of a company process. 

[45] Military pay and personnel will not be included in the Navy ERP 
because DOD plans to use a new system--the Defense Integrated Military 
Human Resources System (DIMHRS)--to process these functions for all DOD 
components. 

[46] GAO, DOD Management: Examples of Inefficient and Ineffective 
Business Processes, GAO-02-873T (Washington, D.C.: June 25, 2002). 

[47] According to DOD, DTS is expected to reengineer defense travel to 
a seamless, paperless, automated system that meets the needs of 
individual travelers, force commanders, and process owners (such as 
finance and accounting services). DTS represents a whole new way of 
doing business for government and it is expected to make the travel 
process faster, easier, and better than ever before. During fiscal 
years 2004-2006, DTS is expected to be fielded to more than 250 high- 
volume sites across the country that serve over 80 percent of all DOD 
travelers. 

[48] GAO-04-615. 

[49] The Prompt Payment Act, codified at 31 U.S.C. §§ 3901 - 3907, and 
as implemented at 5 C.F.R. pt. 1315 (2005), provides for agencies, 
among other things, to pay interest and penalties under various 
circumstances for late payments, generally when payments are not made 
within 30 days of the payment due date. 

[50] GAO, Defense Acquisitions: The Global Information Grid and 
Challenges Facing Its Implementation, GAO-04-858 (Washington, D.C.: 
July 28, 2004). 

[51] GAO-05-702. 

[52] JFMIP was a joint and cooperative undertaking of the Department of 
the Treasury, GAO, the Office of Management and Budget (OMB), and the 
Office of Personnel Management (OPM), working in cooperation with each 
other and other federal agencies to improve financial management 
practices in the federal government. Leadership and program guidance 
were provided by the four Principals of JFMIP--the Secretary of the 
Treasury, the Comptroller General of the United States, and the 
Directors of OMB and OPM. Although JFMIP ceased to exist as a stand- 
alone organization as of December 1, 2004, the JFMIP Principals will 
continue to meet at their discretion. 

[53] JFMIP, White Paper: Financial Systems Data Conversion- 
Considerations (Washington, D.C.: Dec. 20, 2002). 

[54] GAO-05-441. 

[55] GAO, Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
GAO-04-722 (Washington, D.C.: July 30, 2004). 

[56] William A. Florac, Robert E. Park, and Anita D. Carleton, 
Practical Software Measurement: Measuring for Process Management and 
Improvement (Pittsburgh, Pa.: Software Engineering Institute, Carnegie 
Mellon University, 1997). 

[57] GAO-04-1008. 

[58] GAO-03-507. 

[59] OMB Circular No. A-11, Part 7, Planning, Budgeting, Acquisition, 
and Management of Capital Assets (June 21, 2005) and the supplement to 
Part 7, the Capital Programming Guide (July 22, 1997). 

[60] Under Secretary of Defense (Acquisition, Technology and 
Logistics), Department of Defense, Revision to DOD Earned Value 
Management Policy, March 7, 2005. 

[61] According to Institute of Electrical and Electronics Engineers 
(IEEE), verification and validation processes for projects such as Navy 
ERP can be used to determine whether (1) the products of a given 
activity conform to the requirements of that activity and (2) the 
software satisfies its intended use and user needs. This determination 
may include analyzing, evaluating, reviewing, inspecting, assessing, 
and testing software products and processes. The verification and 
validation processes should assess the software in the context of the 
system, including the operational environment, hardware, interfacing 
software, operators, and users. 

[62] Ronald W. Reagan National Defense Authorization Act for Fiscal 
Year 2005, Pub. L. No. 108-375, § 332, 118 Stat. 1811, 1851-56 (Oct. 
28, 2004) (codified, in part, at 10 U.S.C. §§ 186, 2222). 

[63] The act requires the use of procedures for ensuring consistency 
with the guidance issued by the Secretary of Defense and the Defense 
Business Systems Management Committee and incorporation of common 
decision criteria, including standards, requirements, and priorities 
that result in the integration of defense business systems. 

[64] Approval authorities include (1) the Under Secretary of Defense 
for Acquisition, Technology and Logistics; (2) the Under Secretary of 
Defense (Comptroller); (3) the Under Secretary of Defense for Personnel 
and Readiness; (4) the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer of the Department of 
Defense; and (5) the Deputy Secretary of Defense or an Under Secretary 
of Defense, as designated by the Secretary of Defense. These approval 
authorities are responsible for the review, approval, and oversight of 
business systems and must establish investment review processes for 
systems under their cognizance. 

[65] GAO-04-615. 

[66] GAO-05-381. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: