This is the accessible text file for GAO report number GAO-02-345 
entitled 'Information Technology: Greater Use of Best Practices Can 
Reduce Risks in Acquiring Defense Health Care System' which was 
released on September 26, 2002. 

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States General Accounting Office: 
GAO: 

Report to the Chairman and Ranking Minority Member, Subcommittee on
Readiness and Management Support, Committee on Armed Services, U.S. 
Senate: 

September 2002: 

Information Technology: 

Greater Use of Best Practices Can Reduce Risks in Acquiring Defense 
Health Care System: 

GAO-02-345: 

GAO Highlights: 

Highlights of GAO-02-345, a report to the Chairman and Ranking Minority 
Member, Subcommittee on Readiness and Management Support, Committee on 
Armed Services, U.S. Senate. 

Why GAO Did This Study: 

This report examines the acquisition of the Composite Health Care 
System (CHCS) II. It is one in a series of reports reviewing the 
Department of Defense’s use of best practices in acquiring information 
technology systems. CHCS II is expected to cost about $1 billion to 
deliver full capability to almost 1,100 health facilities worldwide by
2008. GAO’s objectives were to determine (1) what progress has been 
made against project commitments, (2) whether the system has been 
economically justified, and (3) whether effective technical and 
management controls are in place. 

What GAO Found: 

CHCS II is envisioned as a state-of-the-art automated medical 
information system allowing improved health-care decisions and lower 
medical and system costs. An expected highlight is computer-based 
patient records that doctors and other health service providers will be 
able to access from any military treatment facility, irrespective of 
location (see figure). While early CHCS II progress was limited, clear 
improvement has been evident over the past 2 years, as Defense has 
begun to embrace industry best practices. The first incremental release 
of the system, with a deployment decision scheduled for September 2002, 
is set to contain more capability than was originally planned. The 
current schedule is nevertheless 3 years beyond the initial estimate, 
due in part to major program changes and a system redesign; benefits 
are in question since measurement has not yet begun; and costs to date 
are about 2½ times the 1998 estimate for deploying the first increment 
to one region. 

Until recently, Defense’s basis for investing in this system has been an
outdated cost/benefit analysis that did not reflect important changes in
assumptions and, further, justified all system increments beyond the 
first solely on an economic analysis of the entire system. Officials 
now, however, are finalizing an updated analysis and have stated they 
plan to adopt an incremental approach to justifying investment in CHCS 
II, a best practice that successful organizations have been following 
for years. 

The department is moving forward in ensuring that effective acquisition
controls are in place, but further progress can be made. Technical and
management controls are largely in effect in several areas, including
testing and risk management. But performance-based contracting is not
being followed, resulting in the risk that CHCS II will take longer to
acquire and cost more than necessary. 

Figure: Creating a Computer-based Patient Record: 

[See PDF for image] 

This figure is an illustration of the creation of a computer-based 
patient record. The illustration depicts the following information: 

Patient arrives: 
* Diagnostics (blood pressure, heart rate, temperature, etc.); 
* Triage (nurse asks about reason for visit, history of illness); 
* Physical exam (doctor sees patient). 

CHCS II workstation: 
Date from Diagnostics, Triage, and Physical exam is entered, producing 
a computer-based patient record, which is stored in a server. 

Source: GAO, based on DOD data. 

[End of figure] 

What GAO Recommends: 

GAO is making several recommendations to the Secretary of Defense, 
including ensuring expanded use of best practices in managing CHCS II by
(1) modifying the project’s investment strategy to justify investment 
in each system release before beginning development, and measuring 
return on investment; and (2) employing performance-based contracting
practices where possible on all future delivery orders. Defense 
generally agreed with our recommendations. 

This is a test for developing highlights for a GAO report. The full 
report, including GAO's objectives, scope, methodology, and analysis, 
is available at [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-
45]. For additional information about the report, contact Randolph C. 
Hite (202-512-3439). To provide comments on this test highlights, 
contact Keith Fultz (202-512-3200) or email HighlightsTest@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

DOD Has Made Recent Progress Since Missing Early Project Commitments: 

DOD Has Not Adequately Justified Investments in CHCS II Releases, but 
Improvements Are Under Way and Planned: 

Important Technical and Management Controls Now Being Followed, but 
Opportunities Exist For Further Use of Best Practices: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comparison of CHCS and CHCS II Capabilities: 

Appendix III: Detailed Explanation of How CHCS II Will Operate: 

Appendix IV: CHCS II Release 1 Software Packages: 

Appendix V: CHCS II Capabilities Planned for Releases 3-7, and Expected 
Release Dates: 

Appendix VI: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Acknowledgments: 

Tables: 

Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment 
Dates: 

Table 2: Division of CHCS II Management Responsibilities and Functions 
Among DOD Units: 

Table 3: MHS Enterprise Architecture Satisfaction of DOD Architecture 
Framework Essential Products: 

Table 4: Simplified Part of Information Exchange Matrix: 

Table 5: Examples of MHS and CHCS II Mapping: 

Table 6: CHCS II Release 1 Software Packages by Location: 

Figures: 

Figure 1: Simplified DOD Health Affairs Organization Chart: 

Figure 2: Release 1 Interfaces with Four Existing Systems: 

Figure 3: Time Line of CHCS II’s Initial and Current Schedules: 

Figure 4: CHCS II Expenditures to Date: 

Figure 5: Number of Priority 1, 2, and 3 Defects Opened June 2001 
through July 2002: 

Figure 6: Unresolved Priority 1, 2, and 3 Defects Opened April 2001 
through July 2002: 

Figure 7: A Simplified Operational High-Level Graphic: 

Figure 8: Creating a Computer-based Patient Record: 

Figure 9: CHCS II Tri-level Architecture: 

Figure 10: Example of CHCS II Regional Communications: 

Abbreviations: 

ASD(C3I): Assistant Secretary of Defense for Command, Control, 
Communications, and Intelligence: 

CIO: chief information officer: 

CDR: clinical data repository: 

COTS: commercial-off-the-shelf: 

CHCS: Composite Health Care System: 

CPR: computer-based patient record: 

DISA: Defense Information Systems Agency: 

DOD: Department of Defense: 

GOTS: government-off-the-shelf: 

IT: information technology: 

MHS: Military Health System: 

MTF: military treatment facilities: 

SEI: Software Engineering Institute: 

[End of section] 

United States General Accounting Office: 
Washington, DC 20548: 

September 26, 2002: 

The Honorable Daniel K. Akaka: 
Chairman: 
The Honorable James M. Inhofe: 
Ranking Minority Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

This report responds to your request that we review the Department of
Defense’s (DOD) acquisition of the Composite Health Care System (CHCS)
II, an automated medical information system to be deployed to about 
1,100 health facilities at 142 military installations worldwide. 
According to the department, CHCS II will facilitate the worldwide 
delivery of health care to active-duty service members, their 
dependents, and other eligible beneficiaries, when care is received in 
military facilities. DOD has invested about $320 million to acquire an 
initial version of CHCS II, and expects to invest about $1 billion to 
acquire and deploy the full system and an additional $3.2 billion to 
operate and maintain the system over its useful life. DOD began the 
last phase of testing of the initial version of the system in May 2002, 
plans to make a deployment decision for this version in September 2002, 
and plans to deploy it to all sites by October 2005. Over the next 6 
years, DOD plans to acquire and deploy a series of more capable 
versions of the system, delivering full capability to all health 
facilities at all installations by 2008. 

This report is one in a series examining DOD’s use of best practices in
acquiring information technology (IT) systems. [Footnote 1] As agreed 
with your offices, the objectives of our review were to determine (1) 
what progress DOD has made against CHCS II project commitments, 
including required system capabilities, expected system benefits, and 
estimated project costs and milestones; (2) whether DOD has 
economically justified its investment in the system; and (3) whether 
DOD has effective technical and management controls in place for 
acquiring the system. The controls reviewed were requirements 
management, test management, architecture development and alignment, 
risk management, and contract management. 

Results in Brief: 

Details of our objectives, scope, and methodology are included in
appendix I. Concerning progress to date, DOD did not meet its May 1998 
commitment to deliver initial CHCS II system capabilities and 
associated mission benefits in April 1999; and, since it did not 
estimate the cost of delivering the initial system capabilities 
(release 1), it does not have a cost commitment against which to 
measure progress. Reasons for missing delivery of the capabilities 
include initial use of a Web-based architecture that did not meet 
system performance requirements, initial requirements that were ill-
defined, a later influx of requirements changes, and unexpected budget 
cuts that forced changes in the project’s scope and approach. By July 
2000, 26 months later, the department had redefined its plans for CHCS 
II to include adopting a new technical architecture, establishing a 
means for controlling changes to requirements, committing to 
incremental releases [Footnote 2] of system capabilities, and delaying 
the decision date for deploying release 1 to January 2001, which was 21 
months later than its May 1998 commitment. It did not, however, 
similarly redefine benefits, or provide a cost commitment for the 
initial version. 

Since July 2000, DOD has made progress in delivering release 1’s system
capabilities, although precisely which capabilities will be included in 
this release have continued to change, and the deployment decision date 
was delayed another 20 months, to September 2002. Progress against CHCS 
II’s benefits commitment is not yet known, because the project has yet 
to measure the accrual of actual benefits even though versions of 
release 1 have been used at test sites for about 2 years. Similarly, 
progress against a cost commitment has not been measured because DOD 
did not develop a cost estimate for release 1 until April 2002, only 5 
months before this release is to be deployed. Available measures of 
whether release 1 meets revised commitments [Footnote 3] indicate that 
the system is maturing, but questions still exist as to the system’s 
operational efficiency. For example, while DOD has corrected all of the 
most serious defects, about 50 defects affecting system efficiency 
remain unresolved. 

Second, DOD has not economically justified its investment in CHCS II
releases, but it plans to do so from this point forward. It initially 
developed a single economic justification for CHCS II in 1998, which 
has been used as the basis for its past and ongoing investment in 
releases 1, 2, and 3. However, this justification had known limitations 
in the cost estimate and was not updated to reflect material changes in 
planned system capabilities, costs, and milestones. Further, although 
DOD has adopted an incremental approach to acquiring and deploying 
system releases, it did not treat investment in releases 2 and 3 as 
separate investment decisions. This approach is not consistent with 
best practices and federal requirements. As such, it has invested 
hundreds of millions of dollars in the system over 4 years without 
knowing whether the cost of a particular release is outweighed by its 
benefits. Recently, DOD updated its analysis and reported that releases 
1 and 2 are economically justified. CHCS II project officials have 
acknowledged that incremental investment is a best practice that should 
be followed, and stated that they will change their strategy for future 
releases to include economically justifying each release before 
investing in it and verifying each release’s benefits and costs after
deployment. 

Finally, DOD has appropriately given attention and priority to employing
certain acquisition best practices, including those related to managing
system requirements and test events, aligning the system architecture 
with the DOD military health system enterprise architecture, [Footnote 
4] and proactively managing project risks. Such practices better 
position the department for successfully acquiring CHCS II. 
Nevertheless, management can be improved upon. For example, performance-
based contracting methods are not being used to ensure contractor 
accountability. Until this is changed, acquisition risks will remain 
higher than they need to be. 

DOD’s recent progress on CHCS II is due in part to its attention and
commitment to adopting certain acquisition management best practices,
such as those governing requirements management. We are making
recommendations to the Secretary of Defense for similarly pursuing 
improvements in CHCS II investment and other acquisition management
controls. 

DOD provided what it termed “official oral comments” on a draft of this
report. In its comments, DOD generally agreed with the report’s message
and recommendations. Its comments are discussed more fully in the
Agency Comments and Our Evaluation section of this report. 

Background: 

DOD operates a nationwide health care program, including overseas 
facilities, for its health care beneficiaries. This program is just one 
of the responsibilities of the Office of the Under Secretary of Defense 
for Personnel and Readiness, along with recruiting, training, 
educating, and a range of morale, welfare, and recreation services. 
Within the Office of the Under Secretary is the Office of the Assistant 
Secretary for Health Affairs, as well as assistant secretaries or 
deputy undersecretaries for force management, reserve affairs, 
readiness, and program integration. 

The Assistant Secretary for Health Affairs is responsible for the 
military health system (MHS) program (see fig. 1). The MHS program has 
two missions: wartime readiness (maintaining the health of service 
members and treating wartime casualties), and peacetime care (providing 
for the health care needs of the families of active-duty members, 
retirees, their families, and survivors). The Assistant Secretary for 
Health Affairs establishes policy regarding health care for all DOD 
beneficiaries and also plans and budgets for health care operations and 
maintenance. At the same time, each military service has its own 
medical department, headed by a surgeon general, which operates medical 
facilities and recruits and funds military medical personnel. Health 
Affairs, through the surgeons general, provides policy direction, 
oversight, and resource support to these medical facilities, referred 
to as military treatment facilities. 

Figure 1: Simplified DOD Health Affairs Organization Chart: 

[See PDF for image] 

This figure is an organizational chart, depicting the following 
relationships: 

Top level: 
Deputy Secretary of Defense; 

Second level: 
Under Secretary of Defense (Personnel and Readiness); 
* Assistant Secretary of Defense (Health Affairs); 
- Military Health System Program. 

Assistant Secretary of Defense (Command, Control, Communications, and 
Intelligence). 

Secretary of the Army; 
* Army Medical Service; 
* Army Surgeon General (interfaces with the Military Health System 
Program); 
- Army Medical Facilities. 

Secretary of the Navy; 
* Bureau of Medicine and Surgery; 
* Navy Surgeon General (interfaces with the Military Health System 
Program); 
- Navy Medical facilities. 

Secretary of the Air Force; 
* Air Force Medical Service; 
* Air Force Surgeon General (interfaces with the Military Health System 
Program); 
- Air Force Medical facilities. 

Source: GAO based on DOD data. 

[End of figure] 

MHS: A Large Health Care Provider: 

Today, over 8 million people are eligible to receive care from MHS
facilities—about 80 percent retirees and dependents and about 20 percent
active-duty personnel. Reflecting the magnitude of this patient 
population, MHS’s worldwide operations are significant: about 130,000 
personnel at about 1,000 Army, Navy, and Air Force medical facilities 
divided into 12 domestic plus European, Pacific, and Latin American 
regions. The fiscal year 2002 MHS budget is almost $24 billion, with 
about $5 billion of this amount funded by the three services’ budgets 
for military medical personnel. 

DOD provides about 75 percent of MHS services through these military
facilities, supplementing this by contracting for health services with
civilian providers. Active-duty members are required to obtain care at
military treatment facilities if such are available; in contrast, 
retirees and dependents may obtain care at either military facilities, 
on a space-available basis, or through civilian contract providers. 

Since 1968, DOD has pursued the goal of providing computer support to 
its hospitals and clinics. During fiscal years 1976 to 1984, DOD spent 
about $222 million to acquire, implement, and operate various health-
care computer systems. The original CHCS, begun by DOD in 1988 and first
deployed in 1993, is the primary DOD medical information system now
used in all MHS facilities worldwide. This system supports DOD hospitals
and clinics in registering patients, documenting inpatient activity, and
providing laboratory, radiology, pharmacy, drug-interaction, and other
functions. Other medical information systems include the Ambulatory
Data System, Preventive Health Care Application, and the Nutrition
Management Information System. [Footnote 5] According to a 1996 
analysis of the MHS mission, the existing medical information systems 
cannot meet DOD’s needs. The department thus initiated CHCS II in 1997 
with the goal of assisting clinicians in making health-care decisions 
and lowering medical and systems costs through, for example, 
replacement of each of the above-cited systems. (Appendix II shows a 
comparison of the capabilities provided by CHCS and CHCS II.) 

CHCS II Design: A Brief Description: 

CHCS II is designed to be a multi-level, open system based on a client-
server model. [Footnote 6] (Appendix III describes the system in 
greater detail.) The system is to consist of user [Footnote 7] 
workstations and application and security network computing devices, 
located at military treatment facilities. DOD plans to divide the CHCS 
II acquisition into seven releases. The decision to deploy release 1 is 
scheduled for September 2002; release 2 for July 2003. Remaining 
deployments are scheduled to begin at about 1-year intervals. 

DOD’s description of release 1 shows users creating and storing 
computer-based patient records (CPRs) using 30 CHCS II workstation- and
computer-based software packages (21 commercial, 9 government-owned;
see app. IV for more information on these packages). DOD plans to
connect each facility’s workstations and servers via each installation’s
local or wide area networks. Each installation is also to be connected
through a wide area network [Footnote 8] to a defense computing center, 
where the CPRs will be stored in a database known as the clinical data 
repository. As release 1 is deployed to more installations, DOD intends 
that medical providers will be able to access a patient’s CPR from any 
military treatment facility, no matter where the patient is being or 
has been treated. 

Release 1 is not intended to meet all needs, and thus additional system
capabilities are to be added over several years, beginning in July 2003.
DOD plans for release 1 to provide system security; user and system
interface controls; and various functions such as appointment 
management, order-entry/management, reporting, preventive health care,
immunization tracking, and CPR management. Release 1 will target
ambulatory beneficiaries (out-patients). Other service categories, such 
as inpatient support, are planned for future releases. Release 1 will 
also replace the Ambulatory Data System and the Preventive Health Care
Application. 

Release 1 is planned to interface with four systems: CHCS; the Defense
Enrollment Eligibility Reporting System, which receives and responds to
CHCS II requests for eligibility and immunization information; the 
Third-party Outpatient Collection System, which receives third-party 
billing data from CHCS II; and the Corporate Executive Information 
System, which receives patient information from CHCS II that is used 
for executive reporting (see fig. 2). Two additional interfaces are 
planned for release 2, one to a new patient eligibility system and one 
to a new primary care manager system. [Footnote 9] 

Figure 2: Release 1 Interfaces with Four Existing Systems: 

[See PDF for image] 

This figure illustrates the Release 1 interface with four existing 
systems as follows: 

CHCS II: 

Interface with CHCS: Global data files and order entry requests and 
responses; 
 
Interface with Defense Enrollment Eligibility Reporting System: 
Eligibility and immunization requests and responses; 

Interface with Corporate Executive Information System: Patient data for 
data mining and reporting goes to CEIS; 

Interface with Third-part Outpatient Collection System: Billing data 
goes to third-party payers. 

Source: GAO based on DOD data. 

[End of figure] 

The acquisition plan calls for release 1 to rely on certain existing 
CHCS capabilities, including patient appointment/scheduling, radiology,
pharmacy, and laboratory; plans call for future releases to gradually
replace CHCS, allowing it to be shut down by 2008. Table 1 provides
details on releases 1 and 2 deployment dates and capabilities. Appendix 
V provides details on capabilities and deployment dates for later 
releases. 

Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment 
Dates: 

Release number and expected deployment date: 1, September 2002; 
Capabilities included: 
* Automated health evaluation/assessment questionnaire; 
* Beneficiary population health reports; 
* Centralized health record repository; 
* CHCS, eligibility, executive information, and third-party collection 
system interfaces; 
* Clinical outpatient graphical user interface; 
* CPR; 
* Enrollment and eligibility data; 
* Immunization data; 
* New ambulatory data and preventive health systems; 
* Patient data transcription; 
* Patient encounter documentation; 
* Patient encounter results codes; 
* Patient health history; 
* Patient satisfaction reports; 
* Patient self-assessment data entry; 
* Provider profile data; 
* Specialist consults results reports; 
* User alerts and reminders; 
* User role-based access to system. 

Release number and expected deployment date: 2, July 2003; 
Capabilities included: 
* Automated clinical practice guidelines; 
* Dental charting and documentation; 
* Enhanced clinical decision support; 
* Global health record repository; 
* New eligibility and primary care manager systems interfaces; 
* Optometric documentation and order entry; 
* Theater (deployed) functions. 

Source: GAO based on DOD data. 

[End of table] 

CHCS II Project Management: 

Several DOD units within the Office of the Assistant Secretary for 
Health Affairs, supported by other DOD components, are involved in 
acquiring and deploying CHCS II. The MHS chief information officer 
(CIO) under the Assistant Secretary is responsible for several MHS IT 
projects, of which CHCS II is but one. MHS’s clinical information 
technology program office provides direct management of the project. 
The Assistant Secretary of Defense for Command, Control, 
Communications, and Intelligence (ASD(C3I)) is responsible for 
overseeing the project and deciding whether it can proceed to the next 
acquisition cycle milestone. Table 2 shows how management 
responsibility for CHCS II is divided among DOD units. 

Table 2: Division of CHCS II Management Responsibilities and Functions 
Among DOD Units: 

Entity: MHS CIO; 
Responsibility/function: Oversees the MHS information management and 
technology program. 

Entity: Program Executive Office; 
Responsibility/function: Directs all centrally managed MHS IT system 
acquisitions, including oversight of procurement, development, 
implementation, deployment, maintenance, and operations. 

Entity: Clinical IT Program Office (includes the CHCS II project team); 
Responsibility/function: Acquires and deploys CHCS II; monitors 
contract performance; performs testing and training on use of the 
product delivered by the contractor. 

Entity: Other DOD organizations; 
Responsibility/function: Service military treatment facilities upgrade 
power needs, validate data conversions, and uninstall and remove older 
system hardware. Another MHS program office provides communications 
systems infrastructure. The Office of Program Analysis and Evaluation 
is responsible for verifying and validating the reliability of economic 
analyses for major projects such as CHCS II. The Army Test and 
Evaluation Command is responsible for conducting the operational test. 
The Navy Center for Cost Analysis is responsible for validating the 
CHCS II life-cycle cost estimate. 

Entity: ASD (C3I); 
Responsibility/function: Approves the project to proceed through its 
acquisition cycle on the basis of a review of key documents – an 
independently evaluated life-cycle cost/benefit estimate, a component 
cost analysis, and an acquisition strategy and project baseline. (Is 
also the milestone decision authority for CHCS II, which is categorized 
as a major automated information system initiative.)[a}; 

Entity: Joint Requirements Oversight Council; 
Responsibility/function: Approves mission need and operational 
requirements for automated information systems with joint 
(multiservice) interest. 

[A] A project that is either designated as such by the Assistant 
Secretary, or estimated to require (a) project costs in any single year 
in excess of $30 million; (b) total project costs in excess of $120 
million; or (c) total life-cycle costs in excess of $360 million, all 
in fiscal year 1996 constant dollars. 

Source: GAO based on DOD data. 

[End of table] 

DOD Has Made Recent Progress Since Missing Early Project Commitments: 

Best practices and related federal guidance emphasize the need for
disciplined processes and information to help ensure that projects are
implemented at acceptable costs and within reasonable and expected time
frames. [Footnote 10] They also recognize that acquisitions should 
contribute to tangible, observable mission performance enhancements. In 
short, projects are expected to meet the capability, schedule, benefit, 
and cost commitments upon which their approval was justified. 

For the last 4 years, MHS has invested about $320 million in CHCS II but
either does not know if it is meeting its commitments or has largely not
met them. The exception is in its system capability commitment, where it
is delivering more in release 1 than originally envisioned. The reasons 
for missing commitments relate to a flawed initial design, user 
dissatisfaction with the system prototype, additional requirements, 
technical problems, such as slow network performance, and a funding cut 
for the 1999 MHS budget. As a result, the deployment and concomitant 
benefits have been delayed, and costs have risen substantially over the 
initial costs approved for release 1. Nevertheless, release 1 is now in 
the last phase of testing and available data suggest that it is largely 
performing as intended, although some issues surrounding system 
efficiency remain. 

Overall Progress Against Project Commitments Has Been Limited, but 
Recent Events Show Improvement: 

During the first 2 years of the project, MHS made little progress 
against the CHCS II capability, schedule, benefits, and cost 
commitments it made in 1998. Progress has improved since then, however. 

* Capability Commitment. CHCS II release 1 is intended to meet 
foundational system functional, performance, and interface requirements,
allowing medical providers at military treatment facilities to exchange,
display, and place data into beneficiaries’s medical records regardless 
of where the patient is being treated or where past treatments occurred.
Release 1 is also expected to deliver more capabilities than the 
original commitment due to the requirements added in 2001 in response 
to user demands. Examples of added requirements (originally planned for 
future releases) include Windows 2000, pathology order entry, patient 
self-assessment, and certain patient medical charting features. MHS 
reports that these additional capabilities added about $7 million to 
the cost of release 1, and about 6 months to the schedule. 

* Schedule Commitment. MHS plans to make a deployment decision for 
release 1 in September 2002––over 3 years late. When the project was
approved in May 1998, MHS envisioned deploying a prototype system
beginning in October 1998 and beginning deployment of the initial 
version about April 1999. The initial deployment has been delayed for 
several reasons. The prototype architecture, which employed a Web-based 
user interface, [Footnote 11] was found to cause slow system 
performance. Further, medical personnel were not satisfied with other 
system functions, such as the ability to view and print patient 
documentation, causing requirements redefinition and a system redesign; 
and the Joint Requirements Oversight Council added more CHCS II 
requirements in 2000, which added 15 months to the schedule. DOD also 
cut the MHS budget for fiscal year 1999 that MHS reports caused the 
initial deployment to be delayed 6 months. System users demanded a 
number of additional requirements in 2001, and system testing found 
technical problems with the implementation, such as slow network 
performance. This combination of the additional user requirements and 
technical problems added another 20 months to the schedule, for a total 
delay of 41 months past the original April 1999 estimate. Figure 3 
provides a time line of CHCS II’s initial and current schedules through 
2003. 

Figure 3: Time Line of CHCS II’s Initial and Current Schedules: 

[See PDF for image] 

This figure is a time line of CHCS II’s initial and current schedules, 
as follows: 

Initial plan: 

Prototype development starts: early, 1998; 
Prototype operational testing: late, 1998; 
Prototype deployment: late, 1998; 
Release 1 operational testing: mid-1999; 
Release 1 initial operational capability: mid-2000; 
Today: last, 2002. 

Actual/Future plan: 
Prototype development starts: early, 1998; 
Prototype operational testing: early, 1999; 
System assessment/prototype deployment canceled: mid-1999; 
System redesign/release 1 development starts: last, 1999; 
Release 2 development starts: late 1999; 
Release 1 preliminary installation acceptance testing: late, 2000; 
Release 3 development starts: late, 2001; 
Release 1 final installation acceptance testing: early, 2002; 
Release 1 operational testing: mid-2002; 
Release 1 deployment: late 2002; 
Release 1 initial operational capability: early, 2003; 
Release 2 deployment: mid-2003. 

Source: GAO based on DOD data. 

[End of figure] 

* Benefits Commitment. The 1998 economic analysis estimated CHCS II
benefits at about $5.7 billion in 1998 dollars, including $86 million in
benefits between fiscal years 2000 and 2002. A May 2002 update of this
analysis estimates benefits of about $6.6 billion in 1998 dollars, but 
accrual of benefits is not expected to begin until fiscal year 2003. 
Thus, MHS has not met its original benefit commitments. In addition, it 
has yet to begin verifying whether its benefit expectations are being 
met, and only began validating its plan for measuring benefit accrual 
in March 2002, although versions of release 1, each providing more 
capabilities, have been operating at test sites since June 2000. 
According to project officials, they did not begin this process earlier 
because users at the sites would not yet have been in a position to 
determine how CHCS II affected their work. 

* Cost Commitment. MHS did not prepare a baseline cost estimate for
acquiring and deploying release 1 until April 2002. However, on the 
basis of the 1998 economic analysis, DOD did approve project funding of 
about $109 million for acquiring and deploying CHCS II from 1997 
through 1999. This included the acquisition cost for release 1, plus 
costs to deploy the system to the sites in a single MHS region from 
April through September 1999. As of April 2002, MHS reports it has 
spent about $284 million to acquire release 1, which is about two and 
one-half times the amount approved in 1998 to acquire and deploy 
release 1 to a single MHS region. As with the schedule delays, MHS 
reports that these additional costs are due to the problems with the 
prototype, additional requirements, and technical complexity. In 
addition to the release 1, MHS reports that it has spent about $35 
million to date on release 2 and about $700,000 on release 3, as shown 
in figure 4. 

Figure 4: CHCS II Expenditures to Date: 

[See PDF for image] 

This figure is a pie-chart, depicting the following data: 

CHCS II Expenditures to Date: 
Release 1: 88.8%; 
Release 2: 11.0%; 
Release 3: 0.2%. 

Source: GAO based on DOD data. 

[End of figure] 

Recent Test Results and System Defect Rates Show Initial Release Is 
Maturing, But Issues Remain: 

Two indicators of system maturity are the results of system testing and 
the number of system defects remaining. One type of testing that both 
best practices and federal guidance advocate before a system is 
deployed is system acceptance testing, the purpose of which is to 
verify that the system meets key technical and system requirements. 
Defects are system problems that require resolution. As a system 
matures, the number of defects should show a downward trend. The CHCS 
II project categorizes defects by severity, ranging from 1 to 5, with 1 
being the most serious and the highest priority. [Footnote 12] 
According to the test plan, priority 1 and 2 defects, which jeopardize 
patient safety or adversely affect mission-essential capabilities, need 
to be resolved before the system can continue to operational testing 
and deployment. Priority 3, 4, and 5 defects do not need to be resolved 
since these defects have less effect on the system. Both the system 
acceptance test results and trends in priority 1 and 2 defects show 
that release 1 is maturing. Trends in priority 3 defects, which require 
inefficient workarounds to overcome, do not yet show a maturing trend. 

Acceptance Test Results: 

The project team conducted the acceptance test to validate that the
system is effective and suitable when operating in a production-
representative military treatment facility environment, and that the 
system meets all critical operational issues and key performance 
parameters. An outcome of acceptance testing is a determination of the 
system’s maturity and its readiness to proceed to the final phase of 
testing—operational testing and evaluation. [Footnote 13] 

The acceptance test was sufficiently scoped to provide a basis for 
making each of these determinations. For example, the test was 
conducted at 4 sites and covered all 28 categories of release 1 
functionality, [Footnote 14] and all 1,138 release 1 requirements, 
including performance (e.g., system availability) and interface 
requirements. While two functionality categories—telephone consults and 
self-assessments—were not exercised at all test sites, this was 
mitigated by the fact that “telephone consult” was exercised at one 
site using an automated test tool, and “self-assessment” was tested at 
one site and also used by the test team on test patients at another 
site. 

The acceptance test also defined 15 system maturity parameters/
indicators. [Footnote 15] According to the test results, 10 of these 15 
were fully satisfied (judged to be mature) and 5 were partially 
satisfied (judged to be of limited maturity). For example, one 
parameter called “functional completeness” required that 90 percent of 
the system requirements be met. According to the test results, release 
1 exceeded this requirement, meeting 100 percent of the requirements. 
Other examples of parameters that were fully met are “fault closure 
rate” and “fault severity/safety,” meaning that defect density is 
decreasing and no priority 1 and 2 defects exist, respectively. 
According to the test results and our analysis of defects (discussed in 
the next section of this report), both were met. 

For the five parameters that were not fully satisfied and thus rated as
limited maturity, we analyzed the reasons provided for these
determinations, and believe that the parameters were sufficiently 
satisfied to mitigate any concerns about whether the parameter was 
sufficiently satisfied. For example, one of the five is “key 
performance parameters” which actually consists of nine performance 
requirements that must be met. Of these nine, test results show that 
eight were met and one was not. However, the one that was not met is 
actually a performance requirement for release 2 and not release 1. 
Similarly, one of the five is “system availability,” which for CHCS II 
is 99 percent system availability. According to the test results, this 
parameter was not met because a scheduled equipment replacement at the 
Defense Information Systems Agency computing center caused the system 
to not be operational while the replacement occurred. However, as we 
have previously reported, [Footnote 16] system availability is 
generally regarded as the time that a system is operating 
satisfactorily, expressed as percentage of the time that the system is 
required to be operational (i.e., excluding the time that scheduled 
system maintenance is occurring). Thus, the down time associated with 
the scheduled equipment replacement should not have been used in 
measuring system availability during the acceptance test, and according 
to the test report, if the equipment replacement had occurred during 
nonclinic hours, the system would have met the 99 percent requirement. 

On the basis of the 15 maturity parameter/indicator ratings, the project
office determined that release 1 was mature, meaning that the testers 
had high confidence that the system was ready for the final phase of 
testing—operational testing and evaluation. 

Trends in System Defects: 

Another indicator of system maturity and quality is defect density, 
which can be measured in a number of ways, including the trend in the 
number of defects being reported and the trend in the number of 
unresolved (uncorrected) defects. Available data on these measures show 
a generally positive maturity trend, although issues of system 
efficiency and progress relative to defect expectations exist. For 
example, after adding requirements in June 2001, the number of defects 
opened each month began to rise, as shown in figure 5. By January 2002, 
however, this number began to drop and by the end of July 2002 was 
about zero for priority 1, 2, and 3 defects combined. 

Figure 5: Number of Priority 1, 2, and 3 Defects Opened June 2001 
through July 2002: 

[See PDF for image] 

This figure is a multiple line graph depicting the number of priority 
1, 2, and 3 defects opened June 2001 through July 2002. The vertical 
axis of the graph represents number of defects from 0 to 50. The 
horizontal axis represents months from June 2001 through July 2002. 
Lines depict Priority 1 and 2 defects combined and Priority 3 defects. 
Shaded ares indicate the system acceptance testing period and the 
operational testing and evaluation period. 

Source: GAO, based on DOD data. 

[End of figure] 

The number of unresolved defects (or defects open each month) also
began to rise after last June, as shown in figure 6. Since February,
however, this number, especially for priority 1s and 2s, has dropped. 
As of the end of July 2002, all priority 1 and 2 defects had been 
resolved, but 46 priority 3 defects remained open. 

While not as serious as priority 1s and 2s, priority 3 defects are still
detrimental because they require workarounds of some sort, which by
definition decrease system efficiency. For example, one unresolved
priority 3 defect is when CHCS II prevents terminal access because the
user had not employed the workstation within the required time limit.
When this happens, reentering the user password should allow renewed
access, but does not. This means that the workstation is unusable until 
it can be restarted, resulting in lost work time. The test plan only 
requires priority 1 and 2 defects to be resolved before moving on to 
operational testing and deployment. However, according to DOD 
officials, they plan to correct all defects identified prior to July 
31, 2002, before deploying release 1, except for 11 that are embedded 
in vendor commercial-off-the-shelf software packages and thus must be 
corrected by the respective vendors. 

Figure 6: Unresolved Priority 1, 2, and 3 Defects Opened April 2001 
through July 2002: 

[See PDF for image] 

This figure is a multiple line graph depicting the number of unresolved 
priority 1, 2, and 3 defects opened April 2001 through July 2002. The 
vertical axis of the graph represents number of defects from 0 to 90. 
The horizontal axis represents months from June 2001 through July 2002. 
Lines depict Priority 1 and 2 defects combined and Priority 3 defects. 
Shaded ares indicate the system acceptance testing period and the 
operational testing and evaluation period. 

Source: GAO, based on DOD data. 

[End of figure] 

DOD Has Not Adequately Justified Investments in CHCS II Releases, but 
Improvements Are Under Way and Planned: 

Between the project’s start in 1997 and April 2002, MHS invested 
hundreds of millions of dollars in CHCS II without following all IT 
investment best practices and related federal guidance that call for 
separating large systems into smaller increments and making informed 
return-on-investment decisions based on reliable estimates of 
incremental project costs, benefits, and risks. Instead, MHS has 
justified its past and ongoing investment in releases 1, 2, and 3 using 
an outdated analysis of costs and benefits that, until recently, did 
not reflect material changes to cost and benefit assumptions. Moreover, 
MHS did not treat investment in releases 2 and 3 as separate decisions, 
instead viewing these releases as being justified as part of MHS’s 
economic analysis of the entire CHCS II system (all releases). Such a 
monolithic approach to analyzing and justifying a system’s return on 
investment has been abandoned by successful organizations as inherently 
unreliable because it relies on predictions of many variables over many 
years. According to CHCS II project officials, they will adopt an 
incremental approach to investing in releases from this point forward. 

Reliable Economic Analysis and Incremental Investment Are Tenets of 
Effective Investment Management: 

The Clinger-Cohen Act of 1996 and OMB guidance [Footnote 17] provide a 
framework for IT investment management that comports with recognized 
best practices. Together, they set requirements for (1) economically 
justifying proposed projects on the basis of reliable analyses of 
expected life-cycle costs, benefits, and risks; (2) using these 
analyses throughout a project’s life cycle as the basis for investment 
selection, control, and evaluation decision-making; and (3) doing so 
for large projects (to the maximum extent practical) by dividing them 
into a series of smaller, incremental subprojects or releases. By doing 
so, the tremendous risk associated with investing large sums of money 
over many years in anticipation of delivering capabilities and expected 
business value far into the future can be spread across project 
fragments that are smaller, of shorter duration, and capable of being 
more reliably justified and more effectively measured against cost, 
benefit, and risk expectations. DOD policy also reflects these 
investment principles by requiring that investments be justified by an
economic analysis and, more recently, [Footnote 18] that investment 
decisions for major projects such as CHCS II be made incrementally to 
better ensure that each increment delivers measurable benefit, 
independent of future increments. 

DOD Did Not Economically Justify Past, Ongoing Investments in CHCS II, 
But Has Taken Steps to Do So: 

IT investment management best practices and federal guidance advocate
economically justifying proposed investments on the basis of a net 
present value benefit-to-cost ratio that is greater than 1, basing that 
ratio on reliable analyses of expected life-cycle benefits, costs, and 
risks, and updating the analysis when significant system changes occur. 
MHS has not followed this practice. It originally justified its 
investment in CHCS II in 1998, with an economic analysis that showed a 
1.14 to 1 benefit-to-cost ratio for the total program and a 1.09 to 1 
ratio for the initial increment (based on present value adjustments to 
1998 dollars). However, in January 1999 the DOD Inspector General 
reported that the cost estimate was based on assumptions that could be 
flawed and result in estimate inaccuracies, and that the analysis did 
not disclose this or its return-on-investment implications. [Footnote 
19] Moreover, since this analysis was developed, the project has 
undergone a number of changes affecting its cost and benefits profile, 
as previously discussed. 

MHS prepared two draft updates to its 1998 economic analysis (January
and August 2000), but it did not approve an update until May 2002.
According to project officials, the approved version was delayed because
system requirements were in a state of flux and did not become stable
until late 2000. Project officials also stated that the draft updates, 
which had benefit-to-cost ratios greater than 1, provided them with 
sufficient assurance that continued investment in CHCS II (releases 1, 
2, and 3) was justified. However, according to the project office’s 
quarterly reports during 2000, CHCS II was still in a state of change 
during and after the time of these updates. As stated in the October 
2000 quarterly report, the economic analysis required revision to 
reflect recent project developments and a change in the system release 
strategy. 

The May 2002 economic analysis estimates life-cycle benefits and costs 
for the entire CHCS II system, including benefits of about $6.5 billion 
and costs of about $4.3 billion (both in 1998 dollars). [Footnote 20] 
When converted to present values, these estimates are approximately 
$4.6 billion and $2.8 billion, respectively, producing a benefit-to-
cost ratio of 1.63 to 1 and a net present value of about $1.78 billion. 
[Footnote 21] However, this analysis is still undergoing review by DOD 
stakeholders. For example, the analysis’s lifecycle cost estimate, 
which was developed by MHS, is currently being validated by the Naval 
Center for Cost Analysis. Because this analysis is still being 
reviewed, we did not evaluate it. 

Incremental Justification Planned for Future CHCS II Investments: 

Incremental investment management involves three fundamental 
components: (1) acquiring a large system in a series of smaller 
increments; (2) individually justifying investment in each separate 
increment on the basis of costs, benefits, and risks; and (3) 
monitoring actual benefits achieved and costs incurred on ongoing 
increments. MHS has structured CHCS II into a series of seven 
increments (releases). However, until recently it had not followed best 
practices in incrementally justifying investments in all system 
releases. More specifically, it did not treat releases 2 and 3 as 
separate investments. Further, it has yet to determine whether actual 
benefit and cost expectations are being met by the first release. 
According to project officials, this is because DOD policy did not 
require them to do so, and the DOD CIO, who is the project’s milestone
decision authority, approved the 1998 economic analysis and granted
approval to proceed. 

Nevertheless, the May 2002 economic analysis, which was developed
during the course of our review and is currently under review within the
department, does define release 1 and 2 costs and benefits. The release 
1 analysis reports life-cycle benefits and costs of $3.4 billion and 
$2.8 billion, respectively (in 1998 dollars). When converted to present 
values, these estimates are approximately $2.4 billion and $2.1 
billion, respectively, producing a benefit-to-cost ratio of 1.12 to 1 
and a net present value of about $255 million. The release 2 analysis 
reports life-cycle benefits and costs of $509 million and $103 million, 
respectively (in 1998 dollars). When converted to present values, these 
estimates are approximately $359 million and $84 million, producing a 
benefit-to-cost ratio of 4.25 to 1 and a net present value of about 
$275 million. The analysis does not, however, address release 3. 

According to project officials, they have recently decided to 
economically justify all future system releases as separate investments 
based on updates to the CHCS II economic analysis, and they plan to 
determine whether return-on-investment expectations for deployed 
releases are being met. The officials also stated that they would 
analyze the benefits and costs of all future releases during MHS’s 
annual IT investment portfolio reviews and decide whether to fund them. 
Further, they stated that actual benefits and costs from deployed 
releases will be measured for each release, starting in February 2003 
for release 1, and the CHCS II investment strategy will be updated to 
reflect these changes. 

Important Technical and Management Controls Now Being Followed, but
Opportunities Exist for Further Use of Best Practices: 

MHS has generally established technical and management control best
practices in key areas: requirements management, test management,
architecture development and alignment, and risk management. This has
not been the case throughout the project’s life, but project officials 
said that they have devoted considerable management attention and given
priority to applying lessons learned and acting on the results of our 
work, and they have made improvements to apply best practices. 
Nevertheless, the CHCS II project office can improve its risk 
management efforts, and it is still not following performance-based 
contracting best practices. According to project officials, they did 
not know how to apply such contracting practices to a system 
acquisition, but have recently begun to develop this expertise. By not 
following these practices, MHS risks paying more and taking longer to 
acquire CHCS II than necessary. 

Key Requirements Management Controls Have Recently Been Established: 

Effectively managing requirements involves establishing and maintaining
agreement among the project team–including end users and contractors—on 
what the system is to do (functionality), how well it is to do it 
(performance), and how it is to interact with other systems 
(interfaces). Best practices [Footnote 22] for managing requirements 
include (1) adhering to a documented requirements management plan, (2) 
establishing a validated set of requirements that serves as the 
baseline against which changes are made, (3) controlling changes to the 
baseline, (4) maintaining traceability among requirements and related 
project deliverables, and (5) involving end users in developing and 
changing requirements. 

For CHCS II, MHS has defined products and processes that meet each of
these tenets of effective requirements management. First, there is a
documented CHCS II requirements management plan with specific written
steps governing development and maintenance of requirements. Second,
the requirements for release 1 have been validated and approved by CHCS
II users, and a baseline requirements set exists. Third, a formal change
control process has been put in place, which requires change control
board approval for baseline modifications based on the changes’s impact
on the project. Fourth, a procedure exists for tracking requirements to
other work products by allowing changes to the contractor’s requirements
database only from DOD’s requirements database. Last, the process 
provides for end user participation in both developing and approving
changes to existing requirements and addition of new requirements. 

To confirm that established requirements management controls were
being followed, we reviewed requirements for two CHCS II capabilities–
clinical practice guidelines, which were defined in 2000, the first 
year the current management controls were put in place, and patient 
index, which were defined in 2001. In both cases, we determined that 
defined process controls were being followed. For example, in both 
cases, users were involved in developing and changing requirements and 
requirements were baselined and put under change control. Also in both 
cases, changes to requirements were assessed in terms of costs, 
benefits, and risks before being accepted. 

According to project officials, they did not effectively manage 
requirements until 2000. A 1999 DOD Inspector General report affirms
early problems with requirements management, [Footnote 23] as does a 
report in October 1999 from a CHCS II test team concluding that the 
lack of complete requirements caused test problems. In response, MHS 
redefined how requirements would be managed, adopting the current 
process during the 2000-2001 timeframe. By doing so, the risks 
associated with defining complete and correct system requirements, and 
preventing uncontrolled changes, commonly referred to as “requirements 
creep,” have been mitigated. 

Key System Acceptance Testing Management Controls in Place: 

Complete and thorough testing is essential to providing reasonable
assurance that systems perform as intended. Testing a system is not a 
one time event, but rather is a series of test events throughout the 
systems development and maintenance cycle, each of which addressing 
different levels and aspects of the system and building on the results 
of previous tests. Among the types of tests are (1) tests of small 
units of software, known as unit testing; (2) tests of whether an 
integrated version of the system meets defined requirements 
(functional, performance, and interface), referred to as acceptance 
testing; and (3) tests of whether the system meets the needs of users 
in an operational setting, called operational testing. 

Best practices including our own guidance [Footnote 24] recommend that 
organizations implement a structured and disciplined approach to 
managing each type of tests. DOD policy requires a similar approach. In 
general, effectively managing a test event, such as system acceptance 
testing, entails (1) developing a test plan that defines, for example, 
test objectives, scope, schedules, resource needs, locations, and 
documentation and reporting requirements; (2) preparing test 
procedures, based on the plan, that specify test cases, steps, data, 
inputs, and expected outputs, and that are traceable to requirements; 
(3) defining test exit criteria, which establish the requirements that 
must be met to successfully complete testing; (4) executing tests in 
accordance with plans and procedures; (5) documenting test results in 
accordance with plans and procedures; (6) identifying, prioritizing, 
and correcting defects, and re-testing defect corrections; (7) 
comparing test results with exit criteria to ensure that specified 
requirements are met; and (8) reporting test results to management in 
accordance with plans and procedures. 

Between January and April 2002, MHS conducted system acceptance 
testing. Our analysis of the management of this test event showed that 
key tenets of effective test management were met. For example, MHS 
prepared test plans that addressed test objectives, scope, schedules, 
resource needs, locations, and documentation and reporting 
requirements. Also, MHS developed detailed test procedures that 
included, among other things, test steps, cases, data, inputs, and 
expected outcomes. Further, our analysis showed that test procedures 
were directly linked to functional, performance, and interface 
requirements. Specifically, we analyzed a statistically valid sample of 
59 requirements against the test procedures [Footnote 25] and found 
that 57 were traceable to specific test procedures. With respect
to the two other requirements, one was a duplicate, and the other was 
not a CHCS II requirement but rather a requirement for an interfacing 
system that was inadvertently associated with CHCS II. 

Additionally, the test plan defined acceptance test exit criteria as 
closing all priority 1 and 2 defects, developing workarounds for 
priority 3 defects, freezing the system baseline, and demonstrating 
system interoperability and stability. Further, MHS executed the 
acceptance test between January and April 2002 and documented the 
results in an acceptance report. 

During and after the test, DOD identified, prioritized, and corrected
defects, closing all priority 1 and 2 defects. Finally, in the 
acceptance report, DOD compared test results with exit criteria and 
provided the test results to management. 

CHCS II and MHS Architectural Alignment Is Occurring: 

Developing, maintaining, and using architectures, or blueprints, is a 
best practice in engineering both individual systems and entire 
enterprises. [Footnote 26] Requirements for having and using 
architectures to guide and constrain IT investment decisionmaking are 
also addressed in federal law and guidance, [Footnote 27] and in DOD 
policy. [Footnote 28] When acquiring or developing new IT systems or 
maintaining existing systems, these sources recognize that it is
very important to ensure that systems are built and modified (i.e.,
“architected”) within the context of the architecture for the 
enterprise that the system supports. To do less risks having systems 
that are, for example, duplicative and not integrated. In the case of 
CHCS II, we found that the system and the MHS enterprise architecture 
are generally aligned. 

MHS Enterprise Architecture: 

An enterprise architecture is a systematically derived snapshot—in 
useful models, diagrams, and narrative—of a given entity’s operations 
(business and systems), including how its operations are performed, what
information and technology are used to perform the operations, where the
operations are performed, who performs them, and when and why they
are performed. The architecture describes the entity in both logical 
terms (e.g., interrelated functions, information needs and flows, work 
locations, systems, and applications) and technical terms (e.g., 
hardware, software, data, communications, and security). Enterprise 
architectures provide these perspectives for both the entity’s current, 
or “as is” environment, and for its target, or “to be” environment; 
they also provide a high-level capital investment roadmap for moving 
from one environment to the other. 

Managed properly, an enterprise architecture can clarify and help 
optimize the interdependencies and interrelationships among a given 
entity’s business operations and the underlying systems and technical
infrastructure that support these operations. [Footnote 29] Our 
experience with federal agencies has shown that attempting to define 
system-level architectures (e.g., develop requirements and design 
specifications) and use these systems architectures to build systems 
without having an enterprise architecture and aligning its systems’ 
architectures with the enterprise architecture often results in systems 
that are duplicative, not well integrated, unnecessarily costly to 
maintain, and limited in terms of optimizing mission performance. 

MHS has developed an initial version of an enterprise architecture. 
[Footnote 30] We analyzed this architecture against the requirements 
for an enterprise architecture as defined by the DOD architecture 
framework, [Footnote 31] which specifies the requirements that all DOD 
enterprise architectures are to meet. As we have previously reported, 
this framework is consistent with commercial architecture frameworks. 
[Footnote 32] According to the DOD framework, enterprise architectures 
must have seven essential products, including a high-level operational 
concept graphic, information exchange matrix, and technical 
architecture profile. Our analysis shows that the current version of 
the MHS enterprise architecture has each of these products (see table
3). 

Table 3: MHS Enterprise Architecture Satisfaction of DOD Architecture 
Framework Essential Products: 

Product: Enterprise view and summary information; 
Description: Serves as planning guide and summarizes “who, what, when, 
why, and how”for architecture to be developed; 
MHS architecture satisfies? Yes. 

Product: Integrated dictionary; 
Description: Provides a central source for definitions of all terms 
used in all architecture products; 
MHS architecture satisfies? Yes. 

Product: High-level operational concept graphic; 
Description: Shows a high-level graphic description of operational 
concept, including organizations, missions, and geographic distribution 
of assets; 
MHS architecture satisfies? Yes. 

Product: Operational node connectivity description; 
Description: Identifies organizational elements that produce, process, 
and consume information; need to exchange information between elements; 
and characteristics of information exchanged (content, media, volume
requirements, security classification, timeliness, and interoperability
requirements); 
MHS architecture satisfies? Yes. 

Product: Operational information exchange matrix; 
Description: Provides information exchange requirements, identifying 
who exchanges what information with whom, why information is necessary, 
and how it is needed; 
MHS architecture satisfies? Yes. 

Product: System interface description; 
Description: Links operational and systems architecture views by 
depicting information systems and their interfaces to organizational 
elements that produce, process, and consume information; 
MHS architecture satisfies? Yes. 

Product: Technical architecture profile; 
Description: Establishes a set of rules governing system implementation 
and operation; references existing technical guidance and discusses how 
that guidance has been or needs to be implemented. 
MHS architecture satisfies? Yes. 

Source: GAO, based on DOD data. 

[End of table] 

For example, MHS’s operational high-level graphic illustrates four
business areas: access to care, provision of health services, population
health management, and manage the business. (See fig. 7 for a simplified
version of the operational high-level graphic.) These business areas are
then decomposed into activities. The activities, in part or whole, are
carried out by MHS systems. 

Figure 7: A Simplified Operational High-Level Graphic: 

[See PDF for image] 

This figure is an illustration of a Simplified Operational High-Level 
Graphic. The illustration contains two concentric circles. The core 
circle is the Military Health System. Surrounding it is a circle 
divided into four quadrants, as follows: 

Population health management: 
* Develop population management practices; 
* Evaluate; 
* Implement tools/manage processes; 
* Define/assess population. 

Provision of health services: 
* Plan health services; 
* Deliver health services; 
* Assess beneficiary health status; 
* Coordinate and integrate health services; 
* Ensure quality of health; 
* Manage information/manage documentation. 

Manage the business: 
* Deliver worldwide logistics; 
* Patient financial management; 
* Manage finances; 
* Manage human resources; 
* Review/improve business management; 
* Support managed care contracting; 
* Perform medical management. 

Access to care: 
* Manage patient movement/encounter; 
* Perform assessment and plan for care; 
* Assess effectiveness of access to care; 
* Manage enrollment and eligibility; 
* Support community outreach; 
* Schedule services and check-in; 
* Support beneficiary services. 

Source: GAO, based on DOD data. 

[End of figure] 

As another example, DOD’s framework requires an information exchange
matrix that identifies who exchanges what information with whom, why
the information is necessary, and how it is needed. The MHS enterprise
architecture has this matrix, and for each information exchange, it
identifies the information source and destination, criticality and
timeliness, and frequency and format requirements. (See table 4.) 

Table 4: Simplified Part of Information Exchange Matrix: 

Information exchange: Eligibility determination; 
Source (Who?): Enrollment/eligibility management staff; 
Destination (Who?): Enterprise-wide registration staff; 
Timeliness (Why?): Seconds-hours-days; 
Criticality (Why?): Critical; 
Frequency (How?): Event-driven; 
Media type (How?): Data, text. 

Information exchange: Customer data release agreement; 
Source (Who?): Enrollment/eligibility management staff; 
Destination (Who?): Primary care manager; 
Timeliness (Why?): Seconds-hours-days; 
Criticality (Why?): High; 
Frequency (How?): Event-driven; 
Media type (How?): Data, text. 

Information exchange: Encounter (administrative) data; 
Source (Who?): Enterprise-wide registration staff; 
Destination (Who?): Enterprise-wide scheduling staff
Timeliness (Why?): Seconds-minutes; 
Criticality (Why?): Routine. 
Frequency (How?): Event-driven; 
Media type (How?): Data. 

Source: GAO, based on DOD data. 

[End of table] 

A third example pertains to DOD’s requirement that component
architectures contain a profile of related DOD technical standards. 
[Footnote 33] MHS’s enterprise has such a standards profile, and this 
profile is aligned with the relevant DOD technical standards. For 
example, DOD requires a standard graphic data interchange, specific 
Windows management standards, and use of the federal information 
processing standard for password usage. The MHS architecture specifies 
each of these standard requirements in its standards profile. 

CHCS II System Architecture: 

MHS has also developed a CHCS II architecture to define a level of 
system detail and specificity that is not found in an enterprise 
architecture, but which is needed in order to build the system. The 
system architecture is articulated in a number of system engineering 
documents, such as the system requirements document, the system design 
specifications, a data dictionary and database description, and 
commercial-off-the-shelf application and system software descriptions. 
According to MHS officials, the CHCS II architecture is fully aligned 
with the MHS architecture. They attributed this alignment largely to 
the fact that the two architectures were developed at the same time by 
the same individuals. 

To verify this alignment, we compared the two architectures at both the
logical and the technical levels. Our analysis showed that the CHCS II
system architecture is aligned with the MHS enterprise architecture. At 
the logical level, we identified 52 capabilities that CHCS II is to 
perform in support of the MHS mission. Of these 52 capabilities, we 
judgmentally selected 13, or 25 percent, from 3 of the 4 MHS business 
areas that relate to CHCS II, and compared them with activities defined 
in MHS’s enterprise architecture. All 13 were aligned with the MHS 
enterprise architecture. (See table 5 for three examples.) 

Table 5: Examples of MHS and CHCS II Mapping: 

MHS business area: Access to Care; 
MHS activity: Check-in patient; 
CHCS II capability: Patient registration. 

MHS business area: Population Health Management; 
MHS activity: Educate patients concerning new practices; 
CHCS II capability: Patient education. 

MHS business area: Provision of Health Services; 
MHS activity: Document care plans and delivery; 
CHCS II capability: Clinical documentation. 

Source: GAO based on DOD data. 

[End of table] 

At the technical level, we compared the MHS technical standards profile
and the CHCS II system architecture to determine if they specified the
same standards. MHS’s profile identifies 61 technical standards that are
appropriate for CHCS II. Of the 61 technical standards, 59 are 
specified in the CHCS II system architecture. For instance, DOD 
requires that medical systems conform to the defense information 
infrastructure common operating environment. MHS and CHCS II plans show 
that CHCS II is to be compliant with this standard in release 2. In 
addition, DOD requires that medical systems follow certain file 
transfer standards, and CHCS II architecture documentation specifies 
these standards. Further, DOD requires that a standard database 
language be used for medical systems, and CHCS II architecture 
documentation specifies this standard language. 

The two standards in the MHS standards profile that are not specified in
the CHCS II system architecture relate to document interchange. These
standards are not currently applicable because the interfaces they 
apply to are for future releases. 

Risk Management Controls in Place, but Application of Controls Can Be 
Improved: 

Managing project risk means proactively identifying facts and 
circumstances that increase the probability of failing to meet project
commitments, and taking steps to prevent this from occurring. Best
practices and federal guidance34 advocate risk management. To be
effective, risk management activities should be (1) based on documented
policies and procedures and (2) executed according to a written plan 
that provides for identifying and prioritizing risks, developing and
implementing appropriate risk mitigation strategies, and tracking and
reporting on progress in implementing the strategies. By doing so,
potential problems can be avoided before they manifest themselves into
cost, schedule, and performance shortfalls. 

MHS has largely implemented a process for managing CHCS II risks that
meets risk management best practices. Using a documented risk 
management policy and associated procedures, project officials have
developed and are implementing a written plan for managing CHCS II
risks. Under this plan, identified risks are captured in a database, 
and each risk is assigned a priority of 1 to 5, with 1 being the most 
serious and the highest priority. [Footnote 35] Further, a strategy for 
mitigating each risk is defined and its implementation is managed, 
including assigning accountability for the risk, tracking status of the 
risk, and reporting on progress in implementing the risk mitigation 
strategy. 

Using its risk management approach, MHS has identified 99 risks over the
life of the project, including 27 priority 1 and 2 risks for release 1 
and release 2. To verify that the established risk management approach 
was being followed, we reviewed the history and status of these 
priority 1 and 2 risks and found that they were generally being managed 
in a manner consistent with the process. For example, of the five open 
priority 1 and 2 risks associated with release 1 of the system, all had 
risk mitigation strategies that were being implemented and tracked 
through formal status reporting. However, of the five priority 1 and 2 
risks associated with release 2, four were being reported as active but 
no mitigation actions were being reported over the last year for any of 
the four. According to project officials and documentation, this was 
because the four were actually inactive but the risk database had not 
been updated. The project office has since updated the database. 

In addition, our analysis led us to question why one priority 1 
risk—user acceptance of the system—had been closed and was no longer 
being actively managed since user acceptance is a common risk of 
commercial-off-the-shelf-based system solutions and, given the status 
of CHCS II deployment, user interaction with the system to date has 
been extremely limited (i.e., confined largely to only the 
approximately 100 medical personnel who have participated in system 
acceptance testing). In response, the project office returned this risk 
to active status, and it is now being formally managed using a 
mitigation strategy and tracked via formal reporting. 

Project officials attributed these instances to lapses in updating the 
risk database. The quality of this database is important, as it defines 
where limited resources will be focused to thwart the emergence of real
problems with real consequences. Without an accurate and complete
inventory of risks, their status, and the progress of strategies to 
address them, the chances of system cost, schedule, and performance 
problems occurring are increased. 

Performance-based Contract Management Practices Not Being Used: 

Performance-based contracting is a recognized best practice that federal
procurement policy reflects. [Footnote 36] Specifically, this policy 
requires agencies to use performance-based contracting methods to the 
maximum extent practicable when acquiring services. Moreover, the CHCS 
II acquisition strategy calls for performance-based contracting 
practices. 

To qualify as performance-based under federal regulations, an 
acquisition should have (1) requirements that define the work to be 
performed in measurable terms; (2) standards (quality or timeliness) 
that are tied to performance requirements; (3) a quality assurance plan 
that describes how the contractor’s performance in meeting requirements 
will be measured against standards; and (4) positive and negative 
incentives. 

In the delivery orders that it has executed to date with the CHCS II
integration contractor, the project office has not fully satisfied any 
of these tenets. First, the delivery orders contain performance 
requirements and the requirements are written in measurable, mission-
related terms. However, the terms are allowed to change with each 
delivery order modification without holding the contractor accountable 
for performance up to the point of the modification. Instead, both the 
requirements and performance prior to the modification are supplanted 
by the modified requirements, and contractor performance begins anew. 
In light of the number of modifications that have accompanied CHCS II 
delivery orders, this results in considerable contractor activity not 
being managed on the basis of performance. For example, one delivery 
order was modified 19 times during its 30-month life, and the 
contractor’s performance was started anew five times, although work was 
behind schedule each time. 

Second, the CHCS II delivery orders contain timeliness, but not quality,
performance standards for each requirement. More specifically, each 
order has a defined delivery date for each deliverable, but does not 
specify standards governing the quality of each deliverable. Third, 
CHCS II project management does not have a quality assurance plan 
defining how it will apply standards to measure contractor performance 
in meeting requirements. Fourth, the orders do not specify incentives 
for either positive or negative contractor performance. 

According to project officials, performance-based contracting practices
have not been employed because these practices have traditionally not
been used for IT services contracts, and they did not know how to
successfully do so for a system acquisition like CHCS II, where the
government is acquiring a product rather than a service. The officials 
also stated that the multi-agency contract vehicle they chose to use 
offered them flexibility and was immediately available, and they have 
not viewed performance-based contracting as a priority. 

By not using performance-based contracting practices on CHCS II, the
project office lacks an effective means for ensuring that it pays a 
fair price for what it receives. To illustrate, three times in 2001 the 
project office agreed to a schedule delay for release 2 deliverables 
because the contractor was not able to meet the release 1 schedule. In 
each instance, the release 2 work was also behind schedule. 
Nevertheless, in both instances, release 2 delivery dates were changed 
and the contractor’s performance was shown as being on schedule. 

Project officials stated that they recognize the value of performance-
based contracting. To prepare them for applying these best practices on 
future CHCS II releases, they stated that they have recently awarded a
performance-based contract for help-desk services for a number of MHS
systems, including CHCS II, and that another MHS project is currently
negotiating a performance-based contract for software development. They
said that they intend to use these experiences to assist them in 
negotiating a performance-based contract for CHCS II release 3. 

Conclusions: 

Owing largely to the absence of the kind of management and technical
controls that are hallmark qualities of system investment and 
acquisition best practices, CHCS II’s early years produced little more 
than lessons learned. Since then, the project’s management team has 
recognized the need to change and given priority attention to doing so. 
As a result, they introduced key missing best practices and made other 
improvements to the project, some of which have occurred during the 
course of our review. These needed practices and improvements have 
contributed to where MHS stands today: in the latter stages of having 
an initial version of CHCS II that shows signs of maturation and 
operational readiness, although questions about operational efficiency 
due to unresolved defects remain a concern. 

A larger concern, however, are unanswered questions about CHCS II’s
investment value. These questions exist because the project’s management
and oversight teams, to include both the MHS and DOD CIOs, have not
given implementation of incremental investment management practices
adequate priority and attention. Consequently, MHS has continued to
invest in CHCS II without sufficient economic justification for doing 
so. The prospects for this changing are encouraging, given statements by
project officials, but until defining and implementing processes for
incremental investment, the risk of DOD’s spending more on the system
than it will return in measurable benefits is increased. Opportunities 
also exist for increasing the effectiveness of ongoing risk management
activities by ensuring that the risk database is complete and correct,
thereby heading off problems before they occur. Further, opportunities
exist for adopting performance-based contracting best practices to 
better ensure that DOD pays a fair price for contractor deliverables. 

Greater use of best practices in the areas of investment, risk, and 
contract management will better position CHCS II management to ensure 
that it will not only be investing in the right vehicle but that it 
will be investing the right way, meaning that it will be following the 
kind of proven management practices that increase the probability that 
required system capabilities and expected benefits will be delivered on 
time and within budget. 

Recommendations for Executive Action: 

To strengthen CHCS II investment, risk, and contract management
practices, and thereby increase the chances of the department’s
investment in CHCS II’s producing mission value commensurate with
system costs, we recommend that the Secretary of Defense, through the
Assistant Secretary of Defense for Health Affairs, direct the MHS CIO to
give expanded use of best practices in managing CHCS II the attention 
and priority they deserve. At a minimum, the Assistant Secretary should 
direct the MHS CIO to take the following actions: 

* As part of CHCS II deployment decisions, including any request to the
DOD CIO for deployment approval, consider the aggregate impact on
defense health affairs mission performance caused by the workarounds
needed to compensate for all unresolved defects affecting the system’s
operational efficiency. 

* Define and implement incremental investment management processes to
include (1) modifying the CHCS II investment strategy to define how this
approach will be implemented; (2) justifying investment in each system
release before beginning detailed design and development of the release;
(3) requiring that such justification be based on reliable estimates of 
costs, benefits, and risks; (4) measuring whether actual return-on-
investment for each deployed release is in line with justification 
forecasts; and (5) using actual return-on-investment results in 
deciding whether to begin detailed design and development of the next 
system release. 

* Verify that the CHCS II inventory of risks is complete and correct, 
and report this to the Assistant Secretary for Health Affairs every 6 
months, along with a report on the status of all top priority risks, 
including each risk’s probability of occurrence and impact on mission. 

* Employ performance-based contracting practices on all future CHCS II
delivery orders to the maximum extent possible, including (1) defining
performance standards against which deliverables can be judged, (2)
developing and using quality assurance plans that describes how
contractor performance against the standards will be measured, and (3)
defining and using contractor incentives and penalties tied to the 
quality plan. 

Additionally, we recommend that the Secretary of Defense direct the
Assistant Secretary of Defense for Command, Control, Communications,
and Intelligence, who is the designated approval authority for CHCS II, 
to monitor the project’s use of best practices, including 
implementation of each of the above recommendations, and use this 
information to oversee the project’s movement through its acquisition 
cycle. To this end, we also recommend that the Assistant Secretary, or 
other designated CHCS II approval authority, not grant any request for 
deployment approval of any CHCS II release that is not justified by 
reliable analysis of the release’s costs, benefits, and risks. 

Agency Comments and Our Evaluation: 

DOD provided what it termed “official oral comments” from the Assistant
Secretary of Defense for Health Affairs on a draft of this report. In 
its comments, DOD stated that it agreed with our report’s overall 
message of expanding the department’s use of acquisition management 
best practices on CHCS II. Further, it agreed with two of our three 
primary recommendations and associated findings, and it partially 
agreed with the third, taking exception only to two of this 
recommendation’s component elements. Each of these two elements is 
discussed below. 

First, DOD disagreed with the recommendation calling for using actual
return-on-investment results from an implemented system release in
deciding whether to begin detailed design and development of the next
system release, citing the impact on the product delivery cycle (i.e.,
schedule). Instead, DOD stated that it would use “post-deployment
findings and lessons learned for a given release to minimize risks
associated with future release development.” Assuming that DOD’s
reference to “post-deployment findings” includes having at least
preliminary information on cost and benefit accrual, we do not disagree
with DOD’s proposed alternative and do not believe that it is 
inconsistent with our recommendation. If it does not, we do not believe 
that the proposed alternative adequately positions the department to 
make informed investment decisions on future releases because it does 
not provide for knowing, at least preliminarily, whether a prior 
release’s mission value is being realized. Given that producing mission 
value is a paramount reason for investing in technology, such 
information about prior system releases is essential in making prudent 
decisions about further investment in the system. 

Moreover, by not having at least preliminary information about actual
return-on-investment, DOD would not be able to accomplish its stated 
goal of using “post-deployment findings...to minimize risks associated 
with future release development” because it would not have any 
indication about whether release return-on-investment expectations are 
accruing. On the contrary, DOD’s approach would expose the program to 
unnecessary risk in the name of meeting a scheduled milestone. 
Restated, it would increase the chances of doing the wrong thing 
faster. The intent of our recommendation is to prevent this by ensuring 
that DOD has an adequate understanding of returns from prior 
investments before choosing a course of action on subsequent 
investments. Accordingly, we have not modified our recommendation. 

Second, DOD disagreed with the recommendation element calling for 
reporting on CHCS II program risks to the Assistant Secretary for 
Health Affairs every 6 months, instead stating that the timing of its 
reporting would comply with DOD internal milestone review requirements, 
which based on program plans would result in annual reporting. We do not
believe that DOD’s proposed alternative provides for adequate disclosure
of those items that have a high probability of significantly affecting
delivery of promised system capabilities on time and within budget to 
the executive leadership sponsoring CHCS II. As our research of leading
organization IT investment management practices shows, awareness of
program risks, particularly the top priority risks covered by this
recommendation, are fundamental to ongoing investment control activities
and, among other things, should be frequently disclosed to investment
executives. Further, over extended periods of time, such as 1 year, 
these risks can manifest themselves into material program cost, 
schedule, and performance shortfalls. For these reasons, 6-month 
reporting on these risks should be viewed as the minimum acceptable 
frequency. Thus, we have not modified our recommendation. 

DOD also provided a comment on our recommendation for the Assistant
Secretary of Defense for Command, Control, Communications, and
Intelligence to only approve deployment of a CHCS II release if it is
justified by reliable analysis of the release’s cost, benefits, and 
risks. Specifically, DOD stated that while the Assistant Secretary was 
currently the approval authority for CHCS II deployment and related 
milestone decisions, this decision authority could be delegated in the 
future. We have adjusted our recommendation to recognize this 
possibility. 

We are sending copies of this report to the Chairman and Ranking 
Minority Members of the Senate Committee on Armed Services; House 
Committee on Armed Services; Subcommittee on Defense, Senate Committee 
on Appropriations; the Subcommittee on Defense, House Committee on 
Appropriations; the Subcommittee on Military Readiness, House Committee 
on Armed Services; Senate Committee on Government Affairs; and House 
Committee on Government Reform. We are also sending copies to the 
Secretary of Defense and the Director, Office of Management and Budget. 
Copies will be available upon request and will also be available 
without charge at our Web site at [hyperlink, http://www.gao.gov]. 

Should you have any questions on matters discussed in this report, 
please contact me at (202) 512-3439 or by e-mail at HiteR@gao.gov. Key
contributors to this report are listed in appendix VI. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of our review were to determine (1) what progress the
Department of Defense (DOD) has made against the Composite Health
Care System (CHCS) II project commitments, (2) whether DOD has
economically justified its investment in CHCS II, and (3) whether DOD 
has effective technical and management controls in place for acquiring 
the system. Project commitments included in our scope of work were 
required system capabilities, promised system benefits, and estimated 
project costs and milestones; controls included requirements 
management, test management, architecture development and alignment, 
risk management, and contract management. 

To determine the progress made against commitments, we reviewed 
relevant project management documents, such as acquisition strategy and
program baseline documents, acquisition decision memoranda, and 
quarterly Defense Acquisition Executive Summary reports, and we 
interviewed CHCS II project, military health system (MHS) program, and
DOD oversight officials, to determine original and revised system
capabilities, expected benefits, estimated costs, and project 
milestones for both system increments and the entire system. We then 
reviewed program management reports and briefings and system 
documentation (e.g., cost reports, defect reports, and test results), 
and interviewed project, program, and oversight officials, to determine 
actual (as reported) system capabilities, costs, and schedules, as well 
as accrued benefits. Comparing the two, we identified variances and, 
through document review and interviews, identified the causes of the 
variances. We did not independently validate the status information 
obtained. 

To determine whether DOD had economically justified CHCS II, we 
reviewed best practices governing system investment management, 
including those pertaining to incremental investment practices, and we
reviewed relevant legislative requirements and federal guidance. 
[Footnote 37] We then analyzed the available economic justification for 
the project, an April 1998 economic analysis. [Footnote 38] We compared 
this analysis to the best practices and federal guidance, and discussed 
with project, program, and oversight officials, as well as CHCS II 
contractor officials to determine how the analysis was prepared, 
including the basis for cost and benefit estimates and assumptions. We 
also requested updates of this analysis, and reviewed draft updates 
provided to determine whether these updates were approved. Using data 
from approved and updated draft analyses, we also calculated net 
present values for CHCS II. In addition, we reviewed available 
documentation and interviewed CHCS II project officials on its 
portfolio-based investment analysis process and how this process was
applied to CHCS II. 

To determine whether DOD has effective technical and management
controls in place for acquiring the system, we focused on whether best
practices were being employed in five key areas: requirements
management, test management, architecture development and alignment,
risk management, and contract management. We reviewed these areas
because they are critical to successfully acquiring systems. Among other
sources, these practices were derived from the work and research of
Carnegie Mellon University’s Software Engineering Institute (SEI), 
[Footnote 39] as well as our own research. [Footnote 40] Where 
appropriate, we also applied relevant legislative requirements and 
federal guidance that embody these best practices. [Footnote 41] We 
also interviewed MHS officials, CHCS II project officials, and 
contractor staff from Integic, Incorporated, the CHCS II integration
contractor. More detailed description of our scope and methodology in
each control area is provided below. 

Requirements Management: To evaluate requirements management controls, 
we reviewed the current CHCS II requirements management process and 
compared it to best practices. We then evaluated the project’s 
compliance with its established process by analyzing selected sets of
requirements that were managed under the current process. Since the
current process was not in place until 2000, we judgmentally selected 
one of the three requirements sets that were added in 2000, and one of 
the five that were added in 2001. Requirements documentation that we 
reviewed also included requirements descriptions, cost and impact 
estimates, and the related 2000 and 2001 budget analysis documents. We 
also interviewed project and MHS officials responsible for requirements 
management. 

Test management: To evaluate test management controls, we reviewed the 
CHCS II master test plan to determine the overall test approach and 
scope, including the types of testing performed and planned. Because
system acceptance testing was being planned, conducted, and completed
during the course of our review, we focused on this test event. 
Specifically, we analyzed acceptance test planning documents, including
test procedures, and the actual test results, and we compared the 
practices defined and followed to relevant best practices to identify 
whether any variances existed. We also analyzed the scope of the test 
by selecting a statistically valid sample of 59 requirements and 
analyzing test procedures to determine if each requirement was 
addressed. This sample size provided us with a 95 percent confidence 
level. Additionally, we analyzed test results to determine whether 
defined system maturity parameters were met, and if not, analyzed the 
variance to determine the significance of the variance. 

Architecture development and alignment: To evaluate these controls, we
first focused on the MHS enterprise architecture, determining what the
architecture framework is based on and whether the framework used is
consistent with recognized commercial frameworks. We then compared
the MHS enterprise architecture to the framework to determine if 
essential architectural artifacts had been developed, and where 
applicable, whether the MHS architecture was consistent with DOD-wide 
architecture requirements, such as whether the technical standards 
profile in the MHS architecture was consistent with the DOD Joint 
Technical Architecture. [Footnote 42] Next, we obtained relevant CHCS 
II architectural documents, such as requirements documents and design 
specifications, and traced selected CHCS II architecture 
characteristics to the MHS architecture to determine alignment. At the 
business or logical level of the architecture, we compared 13 release 1 
system capabilities (categories of requirements) to the business areas 
and activities in the MHS architecture. We selected 13 of the 52 
release 1 capabilities (25 percent) to ensure that we included 
capabilities covering all MHS business areas relevant to CHCS II. We did
not plan to select more than the 13 capabilities unless we were unable 
to trace any of the 13. At the technical level, we compared all of the 
CHCS II technical standards to the MHS technical standards profile. In 
conducting our analysis, we also interviewed MHS and project officials 
and reviewed architecture development plans to determine how the 
respective architectures were developed, who developed them, and how 
they ensured alignment between the two. 

Risk management: To evaluate risk management controls, we reviewed
the CHCS II risk management process and compared it with best practices.
To determine whether the process was being followed, we analyzed all
priority 1 and 2 risks for release 1 and release 2, including 
determining whether defined steps were performed and criteria for 
reporting on and closing risks were met. Additionally, we used the 
results of review of CHCS II to determine whether additional risks 
should be added to the existing risk inventory. In conducting our 
analysis, we interviewed project officials and reviewed relevant 
documentation, such as the risk management plan, risk mitigation 
strategies, and risk status reports. 

Contract management: In evaluating contract management controls, we
discussed with project officials their criteria and approach for 
defining contract deliverables and measuring contractor performance, 
and whether they had plans for changing from past practices. We also 
reviewed the 11 1999 to 2001 release 1 contract delivery orders with 
the CHCS II integration contractor and compared them with tenets of 
performance-based contracting, which are defined in federal acquisition 
regulations. [Footnote 43] Additionally, we reviewed internal reports, 
such as earned value management system summary reports, which described 
progress against delivery order schedules and budgets to obtain data on 
contractor performance. 

We conducted our work at offices of the Military Health System in Falls
Church, Virginia, and at Integic, Incorporated, offices in Chantilly, 
Virginia, from August 2001 through July 2002, in accordance with 
generally accepted government auditing standards. 

[End of section] 

Appendix II: Comparison of CHCS and CHCS II Capabilities: 

Capability: Access to Care: Bed Availability Reporting; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Case Management; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Enrollment/Eligibility; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Enterprise Member/Patient Index; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Enterprise Registration; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Enterprise Scheduling; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Evacuation Requests; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Global Clinical Data Repository; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Referral Management; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Access to Care: Triage and Demand Management; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Alerts and Reminders 
(including Allergies and Drug Interactions); 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Centralized Health Record 
Repository; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Clinical Decision Support; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Clinical Documentation; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Dental Charting and 
Documentation; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Discharge Planning; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Encounter Coding; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Enterprise Health Record; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Home-based Monitoring; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Inpatient Order 
Entry/Management (Laboratory, Pharmacy, Radiology); 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Operating Room Management; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Optometric 
Documentation/Order Entry; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Outpatient Order 
Entry/Management (Laboratory, Pharmacy, Radiology); 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Patient Education; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Patient Health History; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Pharmaceutical Profiling; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Results Reporting (Ancillary 
Services); 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Role-Based Security; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Telemedicine; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Theater Integration Related 
to Provision of Health Service; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Transcription Services 
Interface; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Provisions of Health Service: Voice Recognition
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Clinical Enterprise 
Member/Patient Index; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Clinical Look-back; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Clinical Outcomes Reporting; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Clinical Registries; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Demand Material; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Health Plan Employer Data and 
Information Set Reporting; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Health Risk Assessment; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Health Surveillance 
Monitoring/Reporting; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Immunization Tracking; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Individual Medical Readiness 
Status Reporting; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Interface to VA Mail Order 
Pharmacy; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Occupational Health 
Monitoring; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Patient Satisfaction 
Reporting; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Patient Self-assessment Data 
Entry; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Population Utilization 
Management; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Provider Profiling; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Radiation Health Monitoring 
and Reporting; 
Provided by CHCS: Yes; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Rules-based Clinical 
Protocols; 
Provided by CHCS: No; 
Provided by CHCS II release 1: No; 
Provided by CHCS II when all releases complete: Yes. 

Capability: Population Health Management: Theater Integration Related 
to Population Health Material; 
Provided by CHCS: No; 
Provided by CHCS II release 1: Yes; 
Provided by CHCS II when all releases complete: Yes. 

Source: GAO, based of DOD data. 

[End of table] 

[End of section] 

Appendix III: Detailed Explanation of How CHCS II Will Operate: 

The initial version (release 1) of CHCS II is a medical information 
system that combines existing MHS systems with commercial software to 
provide a computer-based patient record (CPR) of treatment provided to 
DOD health-care beneficiaries in DOD medical treatment facilities 
(MTF). CHCS II is intended to allow a provider (physician, nurse, 
technician, etc.) to record information gained from a patient’s visit 
to the MTF in the patient’s CPR, as shown in figure 8. 

Figure 8: Creating a Computer-based Patient Record: 

[See PDF for image] 

This figure is an illustration of the creation of a computer-based 
patient record. The illustration depicts the following information: 

Patient arrives: 
* Diagnostics (blood pressure, heart rate, temperature, etc.); 
* Triage (nurse asks about reason for visit, history of illness); 
* Physical exam (doctor sees patient). 

CHCS II workstation: 
Date from Diagnostics, Triage, and Physical exam is entered, producing 
a computer-based patient record, which is stored in a server. 

Source: GAO, based on DOD data. 

[End of figure] 

CHCS II’s architecture is an open system, client-server design of three
levels: the user (client) workstation at various MTF locations, the MTF
computers’ (servers’) operating system and storage hardware and
software, and a clinical data repository at a remote computing center
where the CPR is stored. DOD plans to connect all workstations at an
installation’s hospital or clinics to the servers through the 
installation’s local or wide area network. DOD also plans for each 
installation to be connected, via the Defense Information Systems 
Agency’s (DISA) unclassified wide area network, [Footnote 44] to the 
Montgomery, Alabama, computing center, where the CPRs will be stored in 
a database known as the clinical data repository (CDR). Figure 9 shows 
how the three CHCS II levels are connected. 

Figure 9: CHCS II Tri-level Architecture: 

[See PDF for image] 

This figure is an illustration of the CHCS II Tri-level Architecture, 
depicting the following layout: 

CDR level: 
* CDR; connects to: 
* CDR server. 

Server level: 
* Security server; connects to: DSIA wide area network; 
* Application server; connects to: DSIA wide area network; 
* DSIA wide area network: connects to CDR server at CDR level. 

Client level: 
* CHCS II workstations: connect to MTF local/wide area network; 
* MTF local/wide area network: connects to DSIA wide area network at 
server level. 

Source: GAO, based on DOD data. 

[End of figure] 

DOD plans to deploy release 1 regionally, starting with all sites in MHS
region 2 (Virginia and North Carolina). Communications between the MTFs 
and the CDR go through a single access point in each region (the 
regional hub). For region 2, the access point is planned for Portsmouth,
Virginia, site of the Portsmouth Naval Hospital. The Portsmouth regional
access point plan shows a primary and secondary (for redundancy and 
diversity) connection to the CDR through two separate DISA computing
centers. The communication plan shows each installation in region 2 with
an installation access point connected to Portsmouth, and a backup 
connection to the CDR through a DISA computing center. Figure 10 shows
a typical communications setup for release 1, in this case for region 
2. The figure uses the Ft. Bragg, North Carolina, MTF as an example of 
the nonregional hub. 

Figure 10: Example of CHCS II Regional Communications: 

[See PDF for image] 

This figure is an illustration of the CHCS II Regional Communications 
setup. The following layout is depicted: 

CDR: connects to CDR server at DISA computer center, Montgomery, 
Alabama; 

CDR server: connects to: 
* DISA computer center, Columbus, Georgia; and; 
* DISA computer center, Atlanta, Georgia; 
* Both centers connect to: Regional access point, Portsmouth, Virginia; 
* DISA, Atlanta also has a backup connection to: Ft. Bragg, North 
Carolina; 
* A primary connection exists between Portsmouth and Ft. Bragg; 

Regional access point, Portsmouth, Virginia: contains the Portsmouth, 
VA MTF, local/wide area network. 

Access point, Ft. Bragg: contains Fort Bragg, NC MTF local/wide area 
network. 

Source: GAO, based on DOD data. 

[End of figure] 

[End of section] 

Appendix IV: CHCS II Release 1 Software Packages: 

Release 1 is composed of a number of commercial-off-the-shelf (COTS) 
[Footnote 45] and government-off-the-shelf (GOTS) software packages, as 
shown in table 7. The core CHCS II functionality is provided by the 
existing CHCS application (nine GOTS packages), plus the two 3M 
software packages, MEDCIN charting software, data dictionary, and 
clinical data repository (five COTS packages). The packages are 
integrated by software developed by the CHCS II contractor. In addition 
to the core functionality, CHCS II is also composed of 16 other COTS 
packages that provide operating system, security, data exchange, and 
other system functions. The CHCS II software is located on hardware 
platforms at the 142 military treatment facilities (user workstations 
and application, security, interface, and CHCS servers) and the DISA 
computing center (clinical data repository and server). 

Table 6: CHCS II Release 1 Software Packages by Location: 

Location: Client Workstation; 
Product: MEDCIN; 
Use: Charting; 
Type: Commercial. 

Location: Client Workstation; 
Product: Snareworks; 
Use: Security; 
Type: Commercial. 

Location: Client Workstation; 
Product: Adobe Acrobat Reader; 
Use: Form viewer; 
Type: Freeware. 

Location: Client Workstation; 
Product: Dimension 4; 
Use: Time service; 
Type: Freeware. 

Location: Client Workstation; 
Product: Windows 2000; 
Use: Operating System; 
Type: Commercial. 

Location: Client Workstation; 
Product: Problem Knowledge Couplers; 
Use: Education; 
Type: Commercial. 

Location: Client Workstation; 
Product: McAfee Netshield; 
Use: Virus protection; 
Type: Commercial. 

Location: Client Workstation; 
Product: PARS II Data Collector; 
Use: Data collection; 
Type: Government. 

Location: Administrator Workstation; 
Product: Problem Knowledge Couplers; 
Use: Education; 
Type: Commercial. 

Location: Administrator Workstation; 
Product: Adobe Acrobat Reader; 
Use: Form viewer; 
Type: Freeware. 

Location: Administrator Workstation; 
Product: McAfee Netshield; 
Use: Virus protection; 
Type: Commercial. 

Location: Administrator Workstation; 
Product: 3M Clinical Workstation; 
Use: User Interface; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: 3M Software; 
Use: Data dictionary, repository; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: SNOMED; 
Use: Data dictionary; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: Netscape Unix; 
Use: User registration; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: HP Distributed computing; 
Use: Security; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: Snareworks; 
Use: Security; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: Oracle; 
Use: Database; 
Type: Commercial. 

Location: Clinical Data Repository/Enterprise Security Server; 
Product: Tuxedo; 
Use: On-line transaction monitor; 
Type: Commercial. 

Location: Military Treatment Facility Security Server; 
Product: Enosis Connection Manager; 
Use: Software management; 
Type: Commercial. 

Location: Military Treatment Facility Security Server; 
Product: Tardis; 
Use: Time service; 
Type: Shareware. 

Location: Military Treatment Facility Security Server; 
Product: Windows NT 4; 
Use: Operating system; 
Type: Commercial. 

Location: Military Treatment Facility Security Server; 
Product: Gradient; 
Use: Middleware; 
Type: Commercial. 

Location: Military Treatment Facility Security Server; 
Product: Snareworks; 
Use: Security; 
Type: Commercial. 

Location: Military Treatment Facility Application Server; 
Product: Active X; 
Use: Operating system component; 
Type: Government. 

Location: Military Treatment Facility Application Server; 
Product: Enosis Connection Manager; 
Use: Data exchange; 
Type: Commercial. 

Location: Military Treatment Facility Application Server; 
Product: Snareworks; 
Use: Security; 
Type: Commercial. 

Location: Military Treatment Facility Application Server; 
Product: Tardis; 
Use: Time service; 
Type: Shareware. 

Location: Military Treatment Facility Application Server; 
Product: Oracle; 
Use: Database; 
Type: Commercial. 

Location: Military Treatment Facility Application Server; 
Product: Business Objects; 
Use: Reports; 
Type: Government. 

Location: Interface Engine Server; 
Product: Datagate; 
Use: Operating system; Interface: 3M to CHCS II; 
Type: Commercial. 

Location: CHCS Server; 
Product: CHCS; 
Use: CHCS application; 
Type: Government. 

Location: CHCS Server; 
Product: Fileman; 
Use: CHCS application; 
Type: Government. 

Location: CHCS Server; 
Product: GIS; 
Use: Interface software; 
Type: Government. 

Location: CHCS Server; 
Product: HL 7; 
Use: Message software; 
Type: Government. 

Location: CHCS Server; 
Product: OV MVS; 
Use: Operating system; 
Type: Government. 

Location: CHCS Server; 
Product: MUMPS; 
Use: Programming language; 
Type: Government. 

Location: CHCS Server; 
Product: M Adaptor; 
Use: Data exchange; 
Type: Government. 

Location: CHCS Server; 
Product: M Functions; 
Use: Data exchange; 
Type: Government. 

Source: GAO, based on DOD data. 

[End of table] 

[End of section] 

Appendix V: CHCS II Capabilities Planned for Releases 3-7, and Expected 
Release Dates: 

Release number and expected release date: 3, July 2004; 
Capabilities: 
* Enterprise member/patient index; 
* Government computerized patient record (DOD–Department of Veterans 
Affairs clinical information exchange); 
* Health risk assessment; 
* Health surveillance monitoring and reporting; 
* Individual medical readiness status reporting; 
* New eligibility system interface; 
* Patient self-assessment data entry (additional functionality); 
* Referral management; 
* Replacement ancillary system–pharmacy; 
* Rules-based clinical protocols; 
* Theater integration. 

Release number and expected release date: 4, July 2004; 
* Capabilities: 
* Bed availability reports; 
* Cost accounting and patient accounting interfaces; 
* Dental imaging; 
* Enterprisewide registration; 
* Enterprisewide scheduling (includes operating room); 
* Evacuation requests; 
* Operating room management; 
* Outpatient order entry and management; 
* Pharmaceutical profiling; 
* Replacement laboratory, pathology, radiology systems; 
* Theater integration; 
* Triage and demand management. 

Release number and expected release date: 5, July 2005; 
Capabilities: 
* Assurance/safety; 
* Case management; 
* Inpatient order entry and management; 
* Patient education; 
* Patient safety reporting; 
* Population utilization management and quality; 
* Theater integration; 
* Voice recognition. 

Release number and expected release date: 6, July 2005; 
Capabilities: 
* Clinical look-back; 
* Clinical outcomes reporting; 
* Discharge planning; 
* Enterprise health record; 
* Health plan employer data reporting; 
* Provider database interface; 
* Theater integration. 

Release number and expected release date: 7, July 2006; 
Capabilities: 
* Home-based monitoring; 
* Nonradiation and nonlaboratory ancillary procedures interface; 
* Occupational health monitoring; 
* Radiation health monitoring and reporting; 
* Telemedicine; 
* Theater integration. 

Source: GAO, based on DOD data. 

[End of table] 

[End of section] 

Appendix VI: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Carl L. Higginbotham, (404) 679-1824: 

Acknowledgments: 

In addition to the individual named above, key contributors to this 
report included Nabajyoti Barkakati, Harold J. Brumm Jr., Katherine I. 
Chu-Hickman, Michael P. Fruitman, Chetna Lal, and Keith E. Steck. 

[End of section] 

Footnotes: 

[1] U.S. General Accounting Office, DOD Systems Modernization: 
Continued Investment in the Standard Procurement System Has Not Been 
Justified, GAO-01-682 (Washington, D.C.: July 31, 2001); U.S. General 
Accounting Office, Information Technology: DLA Should Strengthen 
Business Systems Modernization Architecture and Investment Activities,
GAO-01-631 (Washington, D.C.: June 29, 2001). 

[2] DOD plans to divide the CHCS II acquisition into seven releases. 
Release 1 was scheduled for a deployment decision in September 2002; 
release 2 in July 2003; the remaining deployments following at about 1-
year intervals. 

[3] Examples include functional and performance requirements. 

[4] An enterprise architecture is the operational and technical 
blueprint for DOD-wide military health affairs. 

[5] Each of these systems provides certain patient-related information. 
For example, the Ambulatory Data System captures certain outpatient 
information relating to diagnosis and treatment; the Preventive Health 
Care Application contains information on preventive health services; 
and the Nutrition Management Information System supports therapeutic
nutrition therapy and medical food management. 

[6] Open systems conform to industry standards so that commercial 
products can easily be used and support costs can be minimized. A 
client is usually a desktop computing device or program that is 
“served” by one or more networked computing devices. 

[7] CHCS II users are physicians, nurses, other medical personnel, and 
technical or administrative support staff. 

[8] The network is DOD’s Unclassified but Sensitive Internet Protocol 
Router Network. 

[9] The new eligibility system is intended to replace the existing 
Defense Enrollment Eligibility Reporting System and improve MHS sharing 
of health care information and eliminate manual benefits 
determinations. The new Primary Care Manager System is intended to 
allow assignment and change of beneficiaries’s primary care managers. 

[10] Clinger-Cohen Act of 1996, Public Law 104-106 and Office of 
Management and Budget (OMB) Circular A-130, Management of Federal 
Information Resources (Nov. 30, 2000). 

[11] A program used to connect the user to the system. The prototype 
version was based on a program used to connect to the World Wide Web. 

[12] Priority 1 defects prevent the accomplishment of an operational- 
or mission-essential capability or jeopardize personnel safety. 
Priority 2 defects adversely affect the accomplishment of a mission-
essential capability, degrading performance, for which no alternative 
workaround is known. Priority 3 defects are similar to priority 2, but
workarounds are available. Priority 4 and 5 defects are less serious. 

[13] Operational testing is used to independently assess effectiveness 
and suitability for end users. 

[14] Alerts, allergies, appointments/scheduling, assessment plan, 
clinical notes, consult tracking, encounter coding, encounter 
documentation, external interfaces, forms/reports, immunizations, 
medications, order entry, order sets, patient demographics, patient
disposition, patient search, patient self-assessment tools, problem 
list, results retrieval, screening, security, summary of care, 
telephone consults, template management, templates, user preferences, 
and wellness. 

[15] Configuration management, data recovery and restoration, fault 
closure rate, fault severity, functional baseline, functional 
completeness, installation, interchangeability, key performance 
parameters, network design, reliability, software, system availability, 
test environment, and safety. 

[16] See, for example, Weather Forecasting: Radar Ability Requirement 
Not Being Met, GAO/AIMD-95-132 (Washington, D.C.: May 31, 1995) and Air 
Traffic Control: FAA Plans to Replace Its Host Computer System Because 
Future Availability Cannot Be Assured, GAO/AIMD-98-138R (Washington, 
D.C.: May 1, 1998). 

[17] Clinger-Cohen Act of 1996, Public Law 104-106; OMB Circular A-130 
(Nov. 30, 2000). 

[18] DOD Interim Regulation 5000.2-R and DOD Instruction 5000.2, Change 
1, Operation of the Defense Acquisition System (Jan. 4, 2001). 

[19] Office of the Inspector General (OIG), Department of Defense, 
Acquisition Management of the Composite Health Care System II Automated 
Information System, report number 99-068 (Jan. 21, 1999). 

[20] The cost estimate appropriately does not include the about $320 
million already spent on CHCS II because this constitutes a sunk cost 
and thus is not relevant to determining current net present value. 

[21] The life-cycle benefits estimate includes benefits resulting from 
improvements in MHS processes ($4.4 billion) and benefits from 
replacing existing MHS systems with CHCS II ($2.1 billion). 

[22] See, for example, Software Acquisition Capability Maturity Model 
SM (CMU/SEI-99-TR-002, April 1999). 

[23] Office of the Inspector General (OIG), Department of Defense, 
report number 99-068 (Jan. 21, 1999). 

[24] See, for example, Year 2000 Computing Crisis: A Testing Guide 
(GAO/AIMD-10.1.21, November 1998). 

[25] See appendix I for statistical sampling methodology. 

[26] Enterprises can be single organizations or business/mission areas 
that transcend more than one organization (e.g., financial management, 
combat system identification, or medical health care). 

[27] Clinger-Cohen Act of 1996, P.L. 104-106; OMB Circular A-130 (Nov. 
30, 2000); A Practical Guide to Federal Enterprise Architectures, 
Version 1.0, Chief Information Officers Council (February 2001); 
Federal Enterprise Architecture Framework, Version 1.1, Chief 
Information Officers Council (September 1999). 

[28] February 28, 1998, memorandum – jointly signed by the Under 
Secretary of Defense for Acquisition and Technology; the Acting 
Assistant Secretary of Defense for Command, Control, Communications, 
and Intelligence, ASD(C3I); and the Director for Command, Control, 
Communications, and Computer Systems, Joint Chiefs of Staff. 

[29] For additional information on enterprise architectures, see 
Information Technology: Enterprise Architecture Use across the Federal 
Government Can Be Improved, GAO-02-6 (Feb. 19, 2002). 

[30] DOD has recently initiated a program to develop and implement a 
DOD-wide financial management enterprise architecture, which is to 
include each DOD business area that triggers a financial event. MHS is 
participating in this DOD-wide architecture effort, including providing 
MHS architecture products and staff. 

[31] DOD’s architecture framework is called the Command, Control, 
Communications, Computers, Intelligence, Surveillance, and 
Reconnaissance Architecture Framework. 

[32] GAO-01-631. 

[33] Department of Defense, Joint Technical Architecture, version 3.1, 
March 31, 2000. 

[34] See, for example, Software Acquisition Capability Maturity Model 
SM (CMU/SEI-99-TR-002, April 1999); OMB Circular A-130 (Nov. 30, 2000). 

[35] In MHS’s risk management plan, priority 1 risks require immediate 
project changes to eliminate or reduce the risk and management 
attention; priority 2 risks require some project changes, mitigation 
strategies, and careful monitoring; priority 3 risks require mitigation 
strategies and careful monitoring; and priority 4 and 5 risks do not 
require mitigation activities. 

[36] Federal Acquisition Regulation, Part 37. 

[37] Clinger-Cohen Act of 1996, Public Law 104-106, and OMB Circular A-
130 (Nov. 30, 2000). 

[38] CHCS II Milestone I Economic Analysis, April 15, 1998. 

[39] See, for example, Software Acquisition Capability Maturity Model 
SM (CMU/SEI-99-TR-002, April 1999). 

[40] See, for example, Year 2000 Computing Crisis: A Testing Guide 
(GAO/AIMD-10.1.21, November 1998); Information Technology Investment 
Management: A Framework for Assessing and Improving Process Maturity, 
Exposure Draft, GAO/AIMD-10-1.23, version 1 (Washington, D.C.: May 
2000). 

[41] Clinger-Cohen Act of 1996, Public Law 104-106; OMB Circular A-130, 
(Nov. 30, 2000); A Practical Guide to Federal Enterprise Architectures, 
Version 1.0, Chief Information Officers Council (October 2000). 

[42] Department of Defense, Joint Technical Architecture, version 3.1, 
March 31, 2000. 

[43] Federal Acquisition Regulation, Part 37. 

[44] The network is the Unclassified but Sensitive Internet Protocol 
Router Network. 

[End of section] 

GAO’s Mission: 

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO’s commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO’s Web site [hyperlink, 
http://www.gao.gov] contains abstracts and fulltext files of current 
reports and testimony and an expanding archive of older products. The 
Web site features a search engine to help you locate documents using 
key words and phrases. You can print these documents in their entirety, 
including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
[hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail 
alert for newly released products” under the GAO Reports heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to: 

U.S. General Accounting Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 

E-mail: fraudnet@gao.gov: 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 
Jeff Nelligan, managing director, NelliganJ@gao.gov: 
(202) 512-4800: 
U.S. General Accounting Office: 
441 G Street NW, Room 7149:
Washington, D.C. 20548: