This is the accessible text file for GAO report number GAO-11-50 
entitled 'Information Technology: Opportunities Exist to Improve 
Management of DOD's Electronic Health Record Initiative' which was 
released on October 6, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Ranking Member, Committee on the Budget, U.S. Senate: 

United States Government Accountability Office:
GAO: 

October 2010: 

Information Technology: 

Opportunities Exist to Improve Management of DOD's Electronic Health 
Record Initiative: 

GAO-11-50: 

GAO Highlights: 

Highlights of GAO-11-50, a report to the Ranking Member, Committee on 
the Budget, U.S. Senate. 

Why GAO Did This Study: 

The Department of Defense (DOD) provides medical care to 9.6 million 
active duty service members, their families, and other eligible 
beneficiaries worldwide. DOD’s Military Health System has long been 
engaged in efforts to acquire and deploy an electronic health record 
system. The latest version of this initiative-—the Armed Forces Health 
Longitudinal Technology Application (AHLTA)-—was expected to give 
health care providers real-time access to individual and military 
population health information and facilitate clinical support. 
However, the system’s early performance was problematic, and DOD 
recently stated that it intended to acquire a new electronic health 
record system. GAO was asked to (1) determine the status of AHLTA, (2) 
determine DOD’s plans for acquiring its new system, and (3) evaluate 
DOD’s acquisition management of the initiative. To do this, GAO 
reviewed program plans, reports, and other documentation and 
interviewed DOD officials. 

What GAO Found: 

After obligating approximately $2 billion over the 13-year life of its 
initiative to acquire an electronic health record system, as of 
September 2010, DOD had delivered various capabilities for outpatient 
care and dental care documentation. DOD had scaled back other 
capabilities it had originally planned to deliver, such as replacement 
of legacy systems and inpatient care management. In addition, users 
continued to experience significant problems with the performance 
(speed, usability, and availability) of the portions of the system 
that have been deployed. DOD has initiated efforts to improve system 
performance and enhance functionality and plans to continue its 
efforts to stabilize the AHLTA system through 2015, as a “bridge” to 
the new electronic health record system it intends to acquire. 

According to DOD, the planned new electronic health record system—-
known as the EHR Way Ahead—-is to be a comprehensive, real-time health 
record for service members and their families and beneficiaries. The 
system is expected to address performance problems, provide 
unaddressed capabilities such as comprehensive medical documentation, 
capture and share medical data electronically within DOD, and improve 
existing information sharing with the Department of Veterans Affairs. 
As of September 2010, the department had established a planning 
office, and this office had begun an analysis of alternatives for 
meeting the new system requirements. Completion of this analysis is 
currently scheduled for December 2010. Following its completion, DOD 
expects to select a technical solution for the system and release a 
delivery schedule. DOD’s fiscal year 2011 budget request included $302 
million for the EHR Way Ahead initiative. 
Weaknesses in key acquisition management and planning processes 
contributed to AHLTA having fewer capabilities than originally 
expected, experiencing persistent performance problems, and not fully 
meeting the needs of users. 

* A comprehensive project management plan was not established to guide 
the department’s execution of the system acquisition. 

* A tailored systems engineering plan did not exist to guide the 
technical development of the system, an effort that was characterized 
by significant complexity. 

* Requirements were incomplete and did not sufficiently reflect user 
and operational needs. 

* An effective plan was not used to improve users’ satisfaction with 
the system. 

DOD has initiated efforts to bring its processes into alignment with 
industry best practices. However, it has not carried out a planned 
independent evaluation to ensure it has made these improvements. Until 
it ensures that these weaknesses are addressed, DOD risks undermining 
the success of further efforts to acquire electronic health record 
system capabilities. 

What GAO Recommends: 

GAO is recommending that DOD take six actions to help ensure that it 
has disciplined and effective processes in place to manage the 
acquisition of further electronic health record system capabilities. 
In written comments on a draft of this report, DOD concurred with 
GAO’s recommendations and described actions planned to address them. 

View [hyperlink, http://www.gao.gov/products/GAO-11-50] or key 
components. For more information, contact Valerie C. Melvin at (202) 
512-6304 or melvinv@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

AHLTA Has Limited Capabilities and Continues to Experience Performance 
Problems: 

DOD Has Initiated Planning Activities for the EHR Way Ahead: 

AHLTA Performance Was Hindered by Weaknesses in Key Acquisition 
Management and Planning Processes: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Organizations Responsible for Managing and Providing 
Oversight of AHLTA: 

Table 2: Capabilities Planned and Delivered for Blocks 1 and 2: 

Table 3: Top 10 Priorities for EHR Way Ahead: 

Figure: 

Figure 1: Overall AHLTA User Satisfaction Ratings between April 2005 
and July 2007: 

Abbreviations: 

AHLTA: Armed Forces Health Longitudinal Technology Application: 

CHCS: Composite Health Care System: 

CIO: chief information officer: 

CITPO: Clinical Information Technology Program Office: 

CMMI: Capability Maturity Model Integration: 

DHIMS: Defense Health Information Management System: 

DOD: Department of Defense: 

EHR: Electronic Health Record: 

IEEE: Institute of Electrical and Electronics Engineers: 

MHS: Military Health System: 

SEI: Software Engineering Institute: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

October 6, 2010: 

The Honorable Judd Gregg: 
Ranking Member: 
Committee on the Budget: 
United States Senate: 

Dear Senator Gregg: 

This report responds to your request that we examine the Department of 
Defense's (DOD) efforts to implement its military electronic health 
record system known as the Armed Forces Health Longitudinal Technology 
Application (AHLTA). When fully deployed, AHLTA was envisioned to 
provide the department with a modernized health information system 
that would generate and maintain a comprehensive, lifelong, computer-
based patient record for every soldier, sailor, airman, and marine; 
their family members; and others entitled to DOD military health care. 
The electronic health record was expected to give health care 
providers real-time access to individual and military population 
health care information, thus facilitating clinical decision support 
and rationale for care rendered to U.S. service members worldwide. 
However, after more than a decade of effort to deliver this system, 
the department has recently begun planning for a new electronic health 
record system. 

At your request, we conducted a study of DOD's efforts to acquire and 
implement its electronic health record system. Specifically, our 
objectives were to (1) determine DOD's status in implementing AHLTA, 
(2) determine the department's plans for acquiring a new system, and 
(3) evaluate the department's acquisition management for its 
electronic health record system. 

To accomplish the objectives, we reviewed relevant program 
documentation and interviewed appropriate DOD officials. Specifically, 
to determine the status of the AHLTA project, we reviewed project 
plans and status reports. To determine the department's plans for 
acquiring a new electronic health record system, we reviewed relevant 
planning documents, including an initial capabilities document. To 
evaluate the department's management of its electronic health record 
acquisition, we compared the department's activities for project 
management planning, systems engineering management, requirements 
development and management, user satisfaction feedback, and 
acquisition management with DOD guidelines and industry best practices. 

We conducted this performance audit from September 2009 to October 
2010 at DOD offices in Falls Church, Virginia, and Bethesda, Maryland, 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. A more 
complete description of our objectives, scope, and methodology is 
provided in appendix I. 

Background: 

DOD operates a worldwide health care program, through which it 
provides medical care and assistance to 9.6 million active duty 
service members, their families, and other eligible beneficiaries. Its 
health care operations are significant, involving approximately 
135,000 personnel in approximately 700 Army, Navy, and Air Force 
medical facilities in 12 domestic regions, as well as European, 
Pacific, and Latin American regions. The department's fiscal year 2010 
budget for providing health care services was about $49 billion. 

DOD's health care program is a responsibility of the Office of the 
Undersecretary of Defense for Personnel and Readiness. Within the 
Office of the Undersecretary is the Office of the Assistant Secretary 
of Defense for Health Affairs, which is responsible for the 
department's Military Health System (MHS) program. 

MHS has two missions: wartime readiness (maintaining the health of 
service members and treating wartime casualties) and peacetime care 
(providing for the health care needs of the families of active-duty 
members, retirees and their families, and survivors). The Assistant 
Secretary of Defense for Health Affairs establishes policy regarding 
health care for all DOD beneficiaries and also plans and budgets for 
health care operations and maintenance. At the same time, each 
military service has its own medical department that operates medical 
facilities (referred to as military treatment facilities) and recruits 
and funds military medical personnel. Currently, the military 
treatment facilities include 59 military hospitals and 650 medical and 
dental clinics. DOD provides about half of MHS services through these 
military facilities, supplementing this by contracting for health 
services with civilian contract providers. Active-duty members are 
required to obtain care at military treatment facilities if such care 
is available; in contrast, retirees and dependents may obtain care at 
either military facilities or through civilian contract providers. 

History of DOD's Electronic Health Record System: 

To facilitate the delivery of medical services, in 1988, DOD initiated 
the acquisition of an electronic health record system to support all 
of its hospitals and clinics. This system, the Composite Health Care 
System (CHCS), was intended to be the primary medical information 
system deployed worldwide to support the department's hospitals and 
clinics. DOD envisioned that it would provide automated support for 
patient administrative functions (such as registrations, admission, 
and disposition); ordering and retrieving results of laboratory and 
radiology procedures; ordering and recording prescriptions; and 
patient appointment scheduling. 

CHCS was deployed in 1993; however, it was supported by numerous stand-
alone medical information systems, such as the department's Ambulatory 
Data System, Preventive Health Care Application, and Nutrition 
Management Information System,[Footnote 1] and was not designed to 
facilitate the exchange of information from one system or military 
treatment facility to the next. Specifically, CHCS was facility- 
centric, in which each facility stored only its own medical 
information for patients using different data standards. Therefore, if 
a medical provider wanted to obtain complete information about a 
patient, a query would have to be made to each of the CHCS locations--
a time-and resource-intensive activity. Additionally, when a patient 
moved to another region, the electronic records did not transfer 
across the CHCS locations because of the different data standards at 
each location. The lack of an integrated system perpetuated the 
reliance on paper-based records, leading DOD to pursue a comprehensive 
electronic health care record. 

To this end, in 1997, the department initiated the CHCS II program to 
address the need for a comprehensive, lifelong, computer-based health 
care record for every service member and their beneficiaries. The 
vision for CHCS II was to provide access to a patient's health care 
information with a single query by providers in military treatment 
facilities. Specifically, with this system, DOD planned to provide 
worldwide access to outpatient, inpatient, dental, and vision records, 
and to make them available 24 hours a day, 7 days a week. This new 
system was to be accomplished with the use of a centralized repository 
of all health care information derived using common data standards. 
The system was to build on capabilities of existing systems, subsuming 
their functionality over time, while adding new functionality to meet 
mission needs. 

CHCS II's architecture was to be an open system, client-server design 
of three levels: the user (client) workstation at various DOD 
locations, the DOD computers' (servers') operating system and storage 
hardware and software, and a clinical data repository at a remote 
computing center where the information would be stored.[Footnote 2] 
The department had planned to connect all workstations at an 
installation's hospital or clinic to the servers through the 
installation's local or wide area network. It had planned to divide 
the system acquisition into seven software releases to be delivered 
incrementally by June 2006 at an estimated cost of $4.3 billion (in 
1998 dollars). 

The department's original plan had called for deploying a prototype 
system in October 1998 and beginning deployment of the initial version 
in about April 1999. However, the department did not meet its schedule 
to deliver initial CHCS II system capabilities and associated mission 
benefits by April 1999; it reported that the initial deployment was 
delayed by 6 months because of a failure to meet initial performance 
requirements and changes in system requirements. 

In July 2000, the department redefined its plans for the system to 
include adopting a new technical architecture, establishing a means 
for controlling changes to requirements, and committing to the 
incremental release of system capabilities. It also delayed the 
decision date for deploying the initial system capabilities (for 
outpatient documentation) to January 2001--21 months later than its 
original commitment for the system. 

However, the department did not meet this commitment, and subsequently 
established a new plan that called for incrementally deploying 
functionality to achieve the system's full operational capability. 
Delivery of the system was to commence in July 2003 and was to be 
completed by September 2007, yielding four blocks of capabilities that 
would incrementally populate the system's electronic health record at 
a revised estimated life-cycle cost of $3.8 billion through 2017. 

* Block 1 was to make outpatient information available worldwide on a 
continuous basis through the electronic health record system (as 
opposed to CHCS legacy functionality which only made records available 
at a single location), provide encounter documentation, aid in order 
entry/results retrieval, assist in encounter coding support, provide 
alerts and reminders (such as drug interaction alerts and special duty 
status), facilitate role-based security, and establish a health data 
dictionary and a master patient index. 

* Block 2 was to provide automated clinical practice guidelines, 
optometric documentation, and dental documentation. 

* Block 3 was to replace CHCS ancillary functionality for results 
retrieval and order entry for outpatient encounters such as laboratory 
and automatic pathology, pharmacy, and radiology. 

* Block 4 was to provide for inpatient order entry and management, 
including inpatient clinical and critical care documentation. 

When delivered, the system was to allow users to create and store 
computer-based patient records using workstation-and computer-based 
software packages. Each facility's workstations and servers were to be 
connected via each installation's local or wide area networks. 
Further, each installation was to be connected through a wide area 
network to a defense computing center where the patient records would 
be stored in a database known as the clinical data repository. DOD 
intended that medical providers would ultimately be able to access a 
patient's computer-based record from any military treatment facility, 
no matter where the patient was being or had been treated. 

According to program documentation, the department began worldwide 
deployment of Block 1 in January 2004. It completed the deployment of 
this block in December 2006. However, program officials stated that 
users experienced numerous performance problems with the capabilities 
that were delivered, which impacted its usability, speed, and 
availability. Specifically, the department reported experiencing the 
following problems with the delivery of Block 1: 

* Usability. The system did not support varied clinical workflow to 
meet the needs of various types of practitioners, had missing or 
incomplete clinical capabilities (e.g., consult and referrals 
management, ancillaries, specialty workflow support), did not support 
fully unified or user-customizable patient data, and did not have a 
user-friendly interface. 

* Speed. The system did not have the speed or performance to 
efficiently support the clinicians' workflow in certain environments 
and was affected by problems such as coding and infrastructure which 
impacted its speed. 

* Availability. The system was not reliable on a 24-hour-a-day, 7-day- 
a-week basis; it had no backup for disaster recovery; and the data 
repository experienced system shutdowns and functional interruptions. 

As a result of the system problems associated with Block 1, DOD set a 
new date for system completion--September 2011--and increased the 
projected life-cycle cost of the system to approximately $5 billion, 
which it attributed primarily to the need for increased operations and 
maintenance for Block 1. 

The department also took a number of other steps with regard to the 
initiative. Specifically, in May 2005, it terminated plans for 
deploying the Block 4 inpatient functionality with the intent of 
moving this functionality into Block 3.[Footnote 3] However, due to 
continuing performance problems with the functionality that had been 
delivered, and because the Block 3 deployment had exceeded the 
department's 5-year limit for achieving initial operational capability 
by January 2008, DOD terminated Block 3 (laboratory, radiology, and 
pharmacy) as well. This action left only one of the four planned 
blocks--Block 2--for implementation. Although the department reduced 
the scope of the initiative to only two blocks, the estimated life-
cycle costs were revised back to the original $3.8 billion (through 
2021). However, the department encountered performance problems with 
the Block 2 dental module as well and, in December 2009, MHS senior 
leadership implemented a strategic pause in its further deployment. 

Beyond these actions, the department took other steps over the course 
of the initiative. Specifically, in November 2005, the Assistant 
Secretary of Defense for Health Affairs announced a change in the name 
of the system from CHCS II to AHLTA, but did not give a specific 
reason for doing so. Further, as part of its attempt to improve the 
system, DOD awarded several contracts between fiscal year 2006 and 
fiscal year 2009 for a total of approximately $40 million to address 
performance problems and implement software enhancements. The 
contractors began deployment of these software enhancements (which DOD 
referred to as AHLTA 3.3) in December 2008. 

DOD's Acquisition Process for Its Electronic Health Record: 

To acquire its electronic health record system, DOD used several 
contractors and types of contracts.[Footnote 4] These included fixed- 
price, time-and-materials, and cost-plus-fixed-fee contracts, each of 
which involved a different level of cost or performance risk for the 
government.[Footnote 5] The prime developer and lead integrator for 
CHCS II, Integic (acquired by Northrop Grumman in 2005), was awarded a 
time-and-materials contract for about $65.4 million in 1997 and was 
tasked to perform systems engineering, requirements analysis, 
architecture evaluation, software design and development, engineering 
and development testing, test and evaluation, maintenance, site 
installation and implementation, and training. Contracts for system 
development and integration continued through fiscal year 2009. 

DOD also used noncompetitive contracts[Footnote 6] for the development 
of the system. According to the program office, 11 noncompetitive 
contracts and task or delivery orders, totaling approximately $44.6 
million,[Footnote 7] were awarded for the system from fiscal year 2004 
through fiscal year 2012. Program officials stated that the 
noncompetitive contracts were awarded on the basis that (1) DOD's need 
for the supplies or service was so urgent that providing each awardee 
under a multiple award contact a fair opportunity would have resulted 
in unacceptable delays; (2) only one awardee was capable of providing 
the supplies or services required at the level of quality required 
because the supplies or services ordered were unique or highly 
specialized; or (3) an order was a logical follow-on to an order 
already issued under the contract.[Footnote 8] 

According to AHLTA program documentation, the system acquisition was 
guided by the defense acquisition system, which is documented in the 
department's DOD 5000.02 Instructions. The defense acquisition system 
consists of five key program life-cycle phases and three related 
milestone decision points that major acquisitions must meet in order 
to proceed to the next phase of the acquisition.[Footnote 9] At each 
milestone point, the program is reviewed by a milestone decision 
authority to determine whether it can move to the next life-cycle 
phase. 

The five phases of the defense acquisition are as follows: 

1. Materiel solution analysis: The purpose of this phase is to assess, 
through an analysis of alternatives, potential solutions to satisfy an 
approved capability need. 

2. Technology development: The purpose of this phase is to determine 
and mature the appropriate set of technologies to be integrated into 
the investment solution by iteratively assessing the viability of the 
various technologies while simultaneously refining user requirements. 
To enter this phase, a program must have an approved analysis of 
alternatives and pass milestone A. To exit this phase, the acquisition 
must demonstrate affordable technology. 

3. Engineering and manufacturing development: The purpose of this 
phase is to develop a system or an increment of capability, and 
demonstrate integrated system design through developer testing to show 
that the system can function in its target environment. To enter this 
phase, a program must have approved requirements and pass milestone B. 
To exit this phase, the acquisition must meet performance requirements 
in the intended environment. 

4. Production and deployment: The purpose of this phase is to achieve 
an operational capability that satisfies the mission needs, as 
verified through independent operational test and evaluation, and to 
implement the system at all applicable locations. To enter this phase, 
a program must have completed development testing and pass milestone 
C. To exit this phase, the system must be deployed and ready to 
operate for all users. 

5. Operations and support: The purpose of this phase is to 
operationally sustain the system in the most cost-effective manner 
over its life cycle. DOD criteria do not require that the milestone 
decision authority conduct milestone reviews during the period after a 
system has been deployed and stabilized. 

For the purpose of conducting milestone reviews, AHLTA was assigned 
the highest level of oversight for DOD information system 
acquisitions.[Footnote 10] As such, oversight was provided within the 
Office of the Secretary of Defense. 

Management Structure for AHLTA: 

Various DOD units were involved in acquiring and deploying AHLTA. As 
the principal advisor to the Assistant Secretary of Defense for Health 
Affairs and to the DOD medical leaders on all matters related to 
information management and information technology, the MHS chief 
information officer (CIO) has primary responsibility for overseeing 
the acquisition, development, testing, and deployment of AHLTA to the 
military treatment facilities. Key offices within the Office of the 
MHS CIO perform critical information management and information 
technology functions to support AHLTA, including the Joint Medical 
Information Systems Office, which is responsible for the testing, 
implementation, training, fielding of system components, operations, 
maintenance, and ultimate disposal of system components. 

Also within MHS, the Composite Health Care System (CHCS) II Program 
Office was established in January 1997 to provide direct management of 
the project; it had operational responsibility for the acquisition and 
deployment of the electronic health record, as well as the migration 
of the numerous standalone clinical information systems. In fiscal 
year 2000, the CHCS II program office was renamed the Clinical 
Information Technology Program Office (CITPO). In 2008, with the 
merger of CITPO and the MHS Theater Medical Information Program 
Office--Joint, the office is now called the Defense Health Information 
Management System (DHIMS). 

To provide oversight in accordance with DOD's defense acquisition 
system, the Assistant Secretary of Defense for Networks and 
Information Integration, within the Office of the Secretary of 
Defense, was designated the milestone decision authority responsible 
for deciding at each acquisition cycle milestone whether the project 
could proceed to the next milestone. The project also received 
oversight from several other bodies, including the Human Resources 
Management Investment Review Board, headed by the MHS CIO, and the 
Overarching Integrated Project Team, which evaluated project 
performance in accordance with DOD 5000 and approved acquisition 
program baselines and acquisition decision memorandums. 

Table 1 summarizes the assignment of responsibilities for AHLTA among 
the various DOD units. 

Table 1: Organizations Responsible for Managing and Providing 
Oversight of AHLTA: 

Management organizations: 

Organization: Office of the Assistant Secretary for Health Affairs; 
Description: Responsible for the department's military health system 
program. Establishes policy regarding health care operations and 
maintenance. Several units within this office, including MHS, are 
involved in acquiring and deploying AHLTA. 

Organization: MHS CIO; 
Description: Oversees the MHS information management and technology 
program. 

Organization: Joint Requirements Oversight Council; 
Description: Approves mission need and operational requirements for 
automated information systems with joint (i.e., multiservice) interest. 

Organization: Joint Medical Information Systems Office--Deputy CIO; 
Description: Supports health care operations through design, 
development, test, evaluation, and deployment of medical information 
systems. The Program Executive Office is responsible for each of the 
program management offices that oversee this activity. 

Organization: Defense Health Information Management System Program 
Office; 
Description: Manages the acquisition, development, deployment, and 
maintenance of AHLTA and other related systems. The program office 
reports to the Joint Medical Information Systems Deputy CIO. Within 
the program office, the project officer is responsible for ensuring 
successful planning, technical development, and acquisition of 
specific information applications and elements of AHLTA. The office 
was established in June 2008 with the merger of CITPO--the original 
CHCS II program office--and the Theater Medical Information Program 
(the office responsible for acquiring the theater portion of the 
electronic health record). 

Oversight organizations: 

Organization: Office of the Assistant Secretary of Defense, Networks 
and Information Integration; 
Description: Acts as the milestone decision authority that authorizes 
AHLTA's readiness to move into each phase of the acquisition life 
cycle, based on successful completion of the criteria for the 
preceding phase. Conducts milestone reviews and prepares decision 
memorandums. 

Organization: Human Resources Management Investment Review Board; 
Description: This board is responsible for annual certification to 
ensure AHLTA meets specified requirements and should be approved for 
funding. 

Organization: AHLTA Overarching Integrated Product Team; 
Description: Reviews program planning in support of the milestone 
decision authority, including oversight, review, and evaluation of 
project execution performance relative to DOD guidance. 

Source: GAO analysis of DOD data. 

[End of table] 

Previous Reviews of DOD's Electronic Health Record Initiatives 
Highlighted Management Deficiencies and Risks: 

DOD's Inspector General and we have previously reported on the 
department's actions toward acquiring its new health care information 
system and have noted the need for improvement in key management 
areas, such as project management, contract management, and risk 
management. 

In reporting on the department's efforts in January l999,[Footnote 11] 
the Inspector General noted that the project management system for the 
acquisition (called CHCS II at the time of the report) was not 
complete. While finding that DOD had taken positive actions to manage 
the acquisition, the report noted that the department had not 
established a project management control system to evaluate and 
measure the program's performance. In addition, the report stated that 
the program's funding visibility was limited because DOD was combining 
funding for sustaining the system with modernization funding for CHCS 
and other clinical business area automated systems. The Inspector 
General made recommendations related to designing and implementing a 
project management control system, the reporting of funding for the 
system, and providing milestone exit criteria that demonstrated the 
level of performance, accomplishments, and progression. 

Further, in May 2006,[Footnote 12] the Inspector General conducted an 
evaluation of the project's program requirements, the related 
acquisition strategy, and system testing to determine whether the 
system was being implemented to meet cost, schedule, and performance 
requirements. While the report found that the program management 
office was using risk mitigation techniques, such as risk management, 
lessons learned, and performance monitoring, the program remained at 
high risk because of the complexities of integrating commercial, off-
the-shelf software into the existing program. In particular, the 
report noted that the program office had not identified any mitigation 
strategies to reduce and control program risk related to integration 
of commercial, off-the-shelf software for the third block of 
functionality. As a result, the Inspector General concluded that the 
program was vulnerable to continued increases in cost, extended 
schedules for implementation, and unrealized goals in performance from 
underestimating the difficulties of integrating commercial, off-the-
shelf products. Subsequently, the program office developed mitigation 
strategies, but the Inspector General reported that they were 
inadequate and did not follow risk management guidance, including 
identifying significant activities and milestones. Accordingly, the 
Inspector General recommended that the program office develop more 
robust mitigation strategies in accordance with the program office's 
risk management plan. 

We have also reported on DOD's management of the system acquisition, 
noting the need for improvements. For example, in 2002, we reported 
that, because the department had not estimated the cost of delivering 
the initial system capabilities, it had lacked a cost commitment 
against which to measure progress.[Footnote 13] In addition, we noted 
that program benefits were in question since measurements had not yet 
begun and that costs were about two-and-a-half times the l998 
estimate. Further, DOD had initially identified a single economic 
justification for the entire project, which had been used as the basis 
for its system releases, and had not treated the releases as separate 
investment decisions. Finally, DOD had not followed performance-based 
contracting practices, resulting in the risk that the system would 
take longer to acquire and cost more than necessary. 

Accordingly, we recommended that DOD expand its use of best practices 
in managing the system by (1) modifying the project's investment 
strategy to justify investment in each system release before beginning 
development and measuring return on investment and (2) employing 
performance-based contracting practices where possible on all future 
delivery orders. The department agreed with these recommendations and 
took actions to update and validate its life-cycle cost estimate in 
September 2002. This was used by the department to approve the 
deployment of the system release. Also, the department employed 
performance-based contracting practices, such as using performance 
standards, quality assurance plans, and contractor incentives on CHCS 
II delivery orders. 

AHLTA Has Limited Capabilities and Continues to Experience Performance 
Problems: 

Despite having obligated approximately $2 billion over the 13-year 
life of its initiatives to acquire and operate an electronic health 
record system, as of September 2010, DOD continued to experience 
performance problems with the one block of AHLTA functionality (Block 
1) that it had fully deployed and with a second block of functionality 
(Block 2) that it had partially deployed. Further, after having 
terminated its plans for deploying the two other blocks of 
functionality (Block 3 and Block 4) that were intended to be part of 
the system, the department has identified April 2011 as the date by 
which it now expects to achieve full operational capability of the 
scaled-backed AHLTA system. Program officials told us they are taking 
steps to stabilize the existing system capabilities through 2015, as 
the department proceeds with plans to pursue yet another new 
electronic health record system. 

In deploying Block 1, the department reported that it achieved all of 
the planned outpatient capabilities for direct patient care, including 
encounter documentation, order entry and results retrieval, encounter 
coding support, consult tracking, and alerts and reminders. According 
to the department, it deployed the AHLTA outpatient documentation 
capability worldwide, providing 77,000 clinicians with the ability to 
document over 148,000 outpatient encounters daily. The department 
stated that medical providers can access the patient's computer-based 
record from any military treatment facility. Also, DOD currently 
shares a significant amount of patient information with the Department 
of Veterans Affairs, including outpatient pharmacy data, laboratory 
results, and radiology results on shared and separated service members. 

In addition, with the deployment of Block 2, including enhancements to 
Block 1, dental capabilities were provided to 73 of 375 dental 
treatment facilities, allowing graphical dental charting, order and 
entry results retrieval, and automated dental readiness 
classification. In this regard, the capabilities were deployed to 46 
Air Force dental medical facilities, 25 Navy facilities, and 2 Army 
facilities. Further, program officials stated that in October 2009, 
because of technical and functionality upgrades made over time to the 
legacy Spectacle Request Transmission System, funding was ceased for 
optometric capabilities for Block 2. The department stated that it 
plans to achieve full operational capabilities by April 2011. Table 2 
shows the capabilities planned and delivered for Blocks 1 and 2. 

Table 2: Capabilities Planned and Delivered for Blocks 1 and 2: 

Block 1 (outpatient care): 

Capability: Encounter documentation; 
Status: Met. 

Capability: Order entry and results retrieval; 
Status: Met. 

Capability: Encounter coding support; 
Status: Met. 

Capability: Consult tracking; 
Status: Met. 

Capability: Alerts and reminders; 
Status: Met. 

Capability: Health data dictionary; 
Status: Met. 

Capability: Master patient index; 
Status: Met. 

Capability: Role-based security; 
Status: Met. 

Block 2: 

Capability: Dental charting and documentation; 
Status: In progress. 

Capability: Optometric documentation and order entry; 
Status: Not Met. 

Source: GAO analysis of DOD data. 

[End of table] 

Nonetheless, program officials, as well as users of the system, 
acknowledged that problems with the system's performance have 
persisted. During a demonstration of the system's operation in April 
2010, medical providers discussed problems with AHLTA, including 
limitations in its availability and usability. For example, the 
providers participating in the demonstration stated that it is time- 
consuming to document encounters using AHLTA because of the time 
required to enter information and navigate through the application 
screens. Thus, they sometimes must document portions of an outpatient 
encounter after the patient leaves. In their experience, using the 
system at the time of the encounter would take attention away from the 
patient for unacceptable periods of time. Also, they stated that when 
system downtime occurs, providers can neither access patient data nor 
electronically document care; in these instances, medical notes are 
recorded manually and later entered in the system after it returns to 
operation--an inefficient process. 

As noted in the earlier discussion, since fiscal year 2006 the 
department has been taking steps to address performance problems and 
enhance existing system capabilities.[Footnote 14] DOD is proceeding 
with what it refers to as a "stabilization effort" to continue making 
improvements to the system and provide ongoing capabilities until a 
new system is acquired. According to DOD officials, the estimated cost 
of this effort for fiscal year 2010 through fiscal year 2015 is $826.3 
million. The stabilization effort is expected to improve the speed, 
availability, and usability of the system; moreover, according to 
officials in the Office of the Deputy Secretary of Defense, the 
stabilization effort is expected to allow the department to meet its 
near-term needs and implement additional enhancements to support its 
future system. 

DOD Has Initiated Planning Activities for the EHR Way Ahead: 

Because AHLTA has consistently experienced performance problems and 
has not delivered the full operational capabilities intended, DOD has 
initiated plans to develop a new electronic health record system. This 
new initiative is called the Electronic Health Record (EHR) Way Ahead. 
As with AHLTA, department officials stated that the new electronic 
health record system is expected to be a comprehensive, real-time 
health record for active and retired service members, their families, 
and other eligible beneficiaries. They added that the new system is 
being planned to address the capability gaps and performance problems 
of previous iterations, and to improve existing information sharing 
between DOD and the Department of Veterans Affairs and expand 
information sharing to include private sector providers. 

Thus far, the department has taken several steps to launch its 
acquisition of the new system. Specifically, in February 2010 it 
established the EHR Way Ahead Planning Office to identify options for 
the future electronic health record system. The planning office 
currently resides within the MHS Joint Medical Information Systems 
Program Executive Office under the Office of the CIO. 

In May 2010, the department approved plans to assess solutions for the 
new electronic health record system. In this regard, the planning 
office began conducting an analysis of alternatives to provide 
guidance on selecting a technical solution. According to planning 
officials, efforts to develop the analysis of alternatives are being 
supervised by the Office of the Assistant Secretary of Defense for 
Health Affairs, and this analysis is expected to define and evaluate 
reasonable alternatives for meeting the capability requirements. The 
analysis is currently scheduled to be completed by December 2010. 

To facilitate the analysis of alternatives, planning officials stated 
that they had identified system capabilities needed to meet the 
department's medical mission. They added that a list of the "top 10" 
priority capabilities for a new system had been developed based on the 
gaps identified in prior iterations of their electronic health 
systems. (These priorities are summarized in table 3.) 

Table 3: Top 10 Priorities for EHR Way Ahead: 

Priority: 1; 
Capability needed to meet DOD's medical mission: Comprehensive medical 
and dental documentation, including encounter data, medications, 
physical examinations, occupational health (including industrial 
hygiene), environmental exposure information and ancillary service 
data (both inpatient and outpatient), documentation of care plan 
objectives, alternatives, patient education, health care services 
provided, patient disposition instructions (including deaths), and 
disposition of remains. 

Priority: 2; 
Capability needed to meet DOD's medical mission: Global capture and 
exchange of all health data for beneficiaries--direct care, network, 
managed care, Veterans Affairs, active duty components, reserve 
components, etc. 

Priority: 3; 
Capability needed to meet DOD's medical mission: Inpatient and 
outpatient order entry and management (laboratory, pharmacy, 
radiology, consults, health care plans, nutrition management, 
prescription spectacle orders). 

Priority: 4; 
Capability needed to meet DOD's medical mission: Laboratory diagnostic 
services (includes results, retrieval and reporting); 
pharmacy services (includes dispensing, operations, reporting, and 
pharmacy data transaction service); radiology diagnostic services 
(includes imagery capture, results, retrieval, and reporting). 

Priority: 5; 
Capability needed to meet DOD's medical mission: En-route care 
documentation on any transport platform. 

Priority: 6; 
Capability needed to meet DOD's medical mission: Results retrieval 
(ancillary services and consults). 

Priority: 7; 
Capability needed to meet DOD's medical mission: Data collection and 
decision support in austere environments starting at the point of 
injury and continuing through all levels of care. 

Priority: 8; 
Capability needed to meet DOD's medical mission: Consult and referral 
management (includes referrals to the civilian health care sector). 

Priority: 9; 
Capability needed to meet DOD's medical mission: Assessments of 
medical deployability of individual service members. 

Priority: 10; 
Capability needed to meet DOD's medical mission: Patient 
administration (includes who the patient is, what he/she is entitled 
to, where he/she is located, etc.). 

Source: GAO analysis of DOD data. 

[End of table] 

According to planning documents, following completion of the analysis, 
DOD expects to select a technical solution and to develop and release 
a delivery schedule. 

DOD's fiscal year 2011 budget request includes $302 million for the 
EHR Way Ahead initiative.[Footnote 15] For fiscal year 2012, the 
department intends to submit an updated budget request and the 
schedule for delivery of the EHR Way Ahead based on the results of the 
analysis of alternatives. 

AHLTA Performance Was Hindered by Weaknesses in Key Acquisition 
Management and Planning Processes: 

The success of a large information technology project such as AHLTA is 
dependent on an agency possessing capabilities to effectively plan and 
manage acquisitions, design the associated systems, define and manage 
system requirements, and use effective measures to gauge user 
satisfaction. In the case of AHLTA, weaknesses in these key management 
areas contributed to DOD delivering a system that provided fewer 
capabilities than originally expected, experienced persistent 
performance problems, and ultimately, did not fully meet the needs of 
its intended users. Alleviating these areas of weakness will be 
essential to the success of further initiatives, including the AHLTA 
stabilization effort and the EHR Way Ahead, that the department 
undertakes in pursuit of its electronic health record system 
capabilities. 

Project Plan Was Incomplete and Not Maintained: 

Program management principles and best practices emphasize the 
importance of having a project management plan in place that, among 
other things, establishes a complete description that ties together 
all program activities and evolves over time to continuously reflect 
the current status and desired end point of the project.[Footnote 16] 
An effective plan is comprised of a description of the program's 
scope, cost, lines of responsibility and authority, management 
processes, and schedule. Such a plan incorporates all the critical 
areas of system development and is to be used as a means of 
determining what needs to be done, by whom, and when. 

Other guidance, such as our Information Technology Investment 
Management framework,[Footnote 17] states that effective program 
oversight of IT projects and systems, including those in operation and 
maintenance, involves maintaining approved project management plans 
that include expected cost and schedule milestones and measurable 
benefit and risk expectations. 

However, officials did not follow best practices in developing a 
project management plan to guide the department's electronic health 
record system. Although the department established a project 
management plan, it did not include several standard components such 
as the project's scope, a requirements management plan, cost estimates 
and baseline, a schedule, and a staffing management plan. In addition, 
although DOD identified the plan as a keystone document for guiding 
the project, the plan was last revised in 2005 and was not updated 
during subsequent development work and the operations and maintenance 
phase to reflect significant changes to the program. These changes 
included termination and postponement of planned capabilities, and 
revisions to the acquisition processes used to guide the AHLTA 
program. As a result, a plan was not in place to effectively guide the 
program throughout these changes. Moreover, there is no such plan to 
guide current activities associated with the stabilization effort, 
which, as discussed previously, involves attempts to address system 
performance problems and enhance functionality. 

According to program officials, the project management plan was last 
revised in 2005 before their focus shifted to addressing the system 
performance problems that occurred as a result of completing Block 1 
deployment in December 2006. Nevertheless, significant changes 
occurred to the program's scope, cost, and schedule after Block 1 
deployment, and the agency lacked a current and complete plan to guide 
activities and measure program progress. Going forward, developing and 
maintaining a comprehensive project plan will be an essential tool for 
overseeing the AHLTA stabilization effort, which is to provide crucial 
improvements to the system and act as a bridge over the next 5 years 
to the deployment of the EHR Way Ahead system. Further, having a 
comprehensive and current project plan for the EHR Way Ahead program 
will help to guide the project and provide oversight of the project's 
progress. Without a project management plan that reflects the status 
and goals of the project, DOD increases the risk that stakeholders 
will not have the insight into program status that is needed to 
exercise effective oversight of both the AHLTA stabilization effort 
and the EHR Way Ahead acquisition. 

DOD Lacked a Systems Engineering Plan to Guide the Electronic Health 
Record System's Design: 

According to industry best practices,[Footnote 18] systems engineering 
governs the total technical and managerial effort required to 
transform a set of user requirements and expectations into specific 
capabilities and, ultimately, into a system design that will meet 
users' needs. Systems engineering practices include developing 
solutions for achieving system performance requirements such as system 
availability, and ensuring compatibility when integrating multiple 
systems and their components. Further, DOD guidance states that a 
tailored and detailed systems engineering plan is a critical tool for 
guiding systems engineering practices throughout the life of an 
acquisition program. Having such a plan is particularly important for 
a system characterized by significant technical complexities. 

DOD's electronic health record system design reflected numerous 
technical complexities, such as the need to capture, manage, and share 
health information across a worldwide network that must be available 
24 hours a day, 7 days a week, and that is to serve a transient 
patient population. In addition, the system design involved a network 
that had to be integrated with a central patient database and multiple 
nonstandard hardware and software platforms, such as commercial, off- 
the-shelf products at over 800 military treatment facilities. 

Nonetheless, although the program office recognized these types of 
system complexities as being part of the electronic health record 
system design, the office never established a tailored systems 
engineering plan to guide the acquisition, or to facilitate the 
resolution of the many performance problems that have plagued the 
system since its initial deployment. 

In this regard, a particularly troublesome area for the department has 
been in deploying enhancements to the system. For example, following 
Block 1 deployment in 2006, the department implemented local cache 
servers in an attempt to improve the system's operational 
availability. According to the department, the specific purpose of the 
local cache servers had been to mitigate the need to access patient 
medical information in the central data repository during system 
outages. However, after the servers were deployed, DOD realized that 
the placement of the servers within the system architecture did not 
resolve the problem and created a single point of failure. Rather than 
yield operational improvements, department officials acknowledged that 
these actions resulted in additional challenges, including the need 
for a costly local cache server redesign, which was begun in fiscal 
year 2009. Program documentation noted that the local cache server 
effort was probably one of the most difficult engineering challenges 
that the program office had faced so far. Further, as various issues 
were faced, it became increasingly clear that detailed planning in the 
earlier stages was not what it could have been. In April 2010, 
clinicians demonstrating the system at the Bethesda Naval Medical 
Center stated that the servers continued to be a major contributing 
factor to system availability issues.[Footnote 19] 

The lack of a systems engineering plan to guide the program office 
through this type of complexity is particularly notable in light of 
the DOD Inspector General's report of 2006, which stated that 
inadequate planning for technical complexities significantly impacts 
the cost, schedule, and performance of a program. The report further 
stated that the AHLTA program office had underestimated the technical 
complexity of integrating products with the electronic health record 
system and, as a result, remained at high risk for continued cost 
increases, schedule overruns, and unrealized performance goals. 

In discussing this matter, agency officials stated that a tailored 
systems engineering plan had not been developed to guide the design of 
AHLTA because such a plan was not required when the system was 
originally planned. Specifically, the officials stated that, it was 
not until February 2004 that DOD issued a policy requiring that a 
systems engineering plan be in place for acquisition programs' 
milestone reviews; but all milestone reviews for AHLTA had been 
completed prior to this time. 

However, current DOD guidance emphasizes the need for a tailored 
systems engineering plan to guide all systems engineering practices, 
including those that occur after the completion of milestone reviews. 
Without a tailored systems engineering plan to guide the program's 
efforts to address long-standing system performance problems as part 
of the AHLTA stabilization efforts, the department may continue to be 
challenged in achieving the desired results. Further, in planning for 
the acquisition of the new EHR Way Ahead system, it will be essential 
that the department establish early in the process and have in place a 
detailed and tailored plan to avoid encountering technical challenges 
similar to those of the AHLTA program, and thus again failing to meet 
users' needs. 

Weaknesses in DOD's Requirements Processes Impacted AHLTA's Usability: 

According to recognized guidance,[Footnote 20] using disciplined 
processes for developing and managing requirements can help reduce the 
risks of developing a system that does not meet user and operational 
needs. Requirements should serve as the basis for establishing 
agreement between users and developers and a shared understanding of 
the system to be developed. Effective requirements development 
practices include, among other things, involving users in identifying 
requirements throughout the project's life cycle to ensure system 
requirements are complete and accurately reflect their needs. 
Effective requirements management practices include maintaining 
bidirectional traceability of requirements to ensure that system-level 
requirements can be traced both backward to high-level operational 
requirements, and forward to low-level system design specifications. 

For the AHLTA acquisition, program documentation revealed that users 
were not adequately involved throughout the requirements development 
process. According to the documentation, users did not seek 
involvement in the requirements development process and system 
developers did not seek user input when making changes to 
requirements. As a result, requirements were neither complete nor 
sufficiently detailed to guide system development, and did not 
adequately provide a shared understanding between the users and 
developers of how the system was to be developed. Program 
documentation noted that requirements often were not adequately 
specified and did not adequately reflect user needs. In particular, 
the program documentation revealed that, while users were involved in 
developing an initial set of requirements used to make system 
acquisition decisions, they were largely not involved in identifying 
new requirements and making changes to existing ones while the system 
was being developed and deployed. 

In certain instances, because users were involved only at the 
beginning and end of the requirements development process, they were 
only able to determine that capabilities would not meet their needs 
after those capabilities had already been deployed. For example, when 
the dental application was in the process of being deployed to Army, 
Navy, and Air Force sites, the MHS senior leadership voted to halt 
further training and implementation because users reported that the 
capabilities were not complete and did not address their needs. 
Consequently, alternate dental solutions will be explored as part of 
the analysis of alternatives for the EHR Way Ahead, resulting in 
additional costs and delays in deploying dental capabilities that will 
meet users' requirements. 

Since the initial deployment, the department has taken steps to 
increase user involvement in defining requirements. For example, to 
better involve users in the requirements process and identify issues 
with system usability, the program office held conferences in 2006 at 
which users identified over 200 new requirements for inclusion in the 
system. Program officials stated that the requirements identified 
during the conference were used to develop the AHLTA 3.3 software 
release. However, our evaluation of the requirements traceability 
matrix used to develop the AHLTA 3.3 release showed that bidirectional 
traceability had not been fully established; thus, the requirements 
were not always linked to high-level operational requirements or to 
more detailed design specifications. Without adequate traceability, 
the department cannot ensure that all agreed-upon requirements will be 
developed, fully tested, and work as intended. 

In addition, the department has plans for making improvements in the 
requirements management process in its MHS Information Management/ 
Information Technology Strategic Plan 2010-2015 and includes a goal to 
improve the requirements management process to enable greater 
participation of system users. According to the plan, this will 
improve the value, quality, timeliness, and stakeholder ownership of 
the resulting system. However, because the department is in the early 
stages of implementing improvements for greater user participation, it 
is too early to determine their effectiveness. 

As the department proceeds with the AHLTA stabilization effort and the 
new EHR Way Ahead system, ensuring that user needs are met will be 
essential to effective and cost-efficient delivery of system 
capabilities. Until the department ensures that a requirements 
development process with adequate user involvement is in place, it 
will continue to lack a vital tool for ensuring the efficient and 
effective delivery of electronic health record system capabilities 
that will meet the needs of its users. 

Efforts to Improve User Satisfaction Were Not Guided by Effective 
Planning: 

DOD has stated that the success of AHLTA can be gauged by improvements 
in user satisfaction and user acceptance, among other things. In this 
regard, effectively managing program improvement activities to improve 
user satisfaction requires planning and executing such activities in a 
disciplined fashion. The Software Engineering Institute's IDEALSM 
[Footnote 21] model is a recognized approach for managing efforts to 
make system improvements. According to this model, user satisfaction 
improvement efforts should include a written plan that serves as the 
foundation and basis for guiding improvement activities, including 
obtaining management commitment to and funding for the activities, 
establishing a baseline of commitments and expectations against which 
to measure progress, prioritizing and executing activities and 
initiatives, determining success, and identifying and applying lessons 
learned. Through such a structured and disciplined approach, 
improvement resources can be invested in a manner that produces 
optimal results. 

However, DOD has not demonstrated that user satisfaction improvement 
efforts are being guided by a documented plan that defines prioritized 
improvement projects and associated resource requirements, schedules, 
and measurable goals and outcomes. Instead, efforts that the office 
undertook to improve user satisfaction were ad hoc and did not meet 
with success. Specifically, the program office stopped measuring AHLTA 
user satisfaction levels in July 2007 after overall user satisfaction 
had declined to its lowest point in more than 2 years. Between 2005 
and 2007 the program office collected user satisfaction feedback 
through online user surveys, and used the data to identify areas for 
system improvements and to measure progress toward improving 
satisfaction. The results of the surveys showed not only that users 
rated their overall satisfaction level with the system between below 
average and average, but that user satisfaction levels had declined to 
a low point with the results of the final survey report of July 2007. 
Thus, as shown in figure 1, the program office was not able to improve 
user satisfaction during this time period. 

Figure 1: Overall AHLTA User Satisfaction Ratings between April 2005 
and July 2007: 

[Refer to PDF for image: vertical bar graph] 

Date: April 2005; 
Score: 2.61 (below average). 

Date: October 2005; 
Score: 3 (average). 

Date: February 2006; 
Score: 3.17 (average). 

Date: June 2006; 
Score: 3.05 (average). 

Date: October 2006; 
Score: 2.72 (below average). 

Date: January 2007; 
Score: 2.94 (below average). 

Date: July 2007; 
Score: 2.36 (below average). 

Source: GAO analysis of DOD data. 

[End of figure] 

According to program officials, they have implemented a major effort 
toward improving user satisfaction with the AHLTA 3.3 software 
release. The improvements associated with this software release began 
as early as 2006 and include features such as improved medical coding 
support and increased speed of the order entry connection, as well as 
other changes to improve users' satisfaction with the system's 
performance and capabilities. Yet, program officials did not provide 
evidence of a plan to guide these efforts or a schedule for 
implementing these improvements, and it is unclear how specific 
capabilities of the software release will be used to address specific 
user concerns. The lack of a documented plan to guide user 
satisfaction improvement activities is of particular significance 
because users have continued to express their dissatisfaction with the 
system. Program officials stated that additional online user 
satisfaction surveys were not conducted after 2007 because users had 
grown weary of the surveys and efforts to address user feedback from 
the existing survey results are ongoing. The next online survey is 
expected to be conducted after full deployment of AHLTA 3.3, but a 
schedule has not yet been set. 

Given the history of system performance problems and the extent to 
which users have not been able to effectively and efficiently use 
AHLTA, it is critical that the department identify and implement 
system improvements in a disciplined and structured fashion. Without a 
documented improvement plan, efforts to improve user satisfaction, 
including those associated with the ongoing AHLTA stabilization 
effort, may be reduced to trial and error, and the office cannot 
adequately ensure that it is effectively investing program resources 
on improvement efforts that will result in a system that satisfies 
users. Further, since increasing user satisfaction is a key goal for 
the EHR Way Ahead, it is critical that a disciplined approach is 
established and maintained throughout the program's life cycle. 

MHS Lacks Assurance of a Disciplined Acquisition Management Process to 
Guide Its Electronic Health Record Initiative: 

The use of disciplined processes to guide the effort of acquiring and 
implementing a major system has been shown to increase the likelihood 
of achieving intended results and reduce the risks associated with an 
acquisition to acceptable levels. Although there is no standard set of 
practices that will ever guarantee success, several organizations, 
such as Carnegie Mellon University's Software Engineering Institute 
and the Institute of Electrical and Electronics Engineers (IEEE), 
[Footnote 22] as well as individual experts, have identified and 
developed the types of policies, procedures, and practices that have 
been demonstrated to reduce development time and enhance 
effectiveness. The key to having a disciplined system development 
effort is to have disciplined processes in multiple areas, including 
project planning, requirements management, systems engineering, system 
testing, and risk management. Because change in a program is constant, 
effective processes should be implemented in each of these throughout 
the project life cycle. Effectively implementing the disciplined 
processes necessary to reduce project risks to acceptable levels is 
difficult because a project must effectively implement several best 
practices, and inadequate implementation of any one may significantly 
reduce or even eliminate the positive benefits of the others. 

Recognizing weaknesses in its acquisition of systems such as AHLTA, 
MHS has been taking steps to institutionalize more disciplined 
management processes across all of its programs. In March 2008 the MHS 
CIO identified an approach for improving its management processes that 
included aligning MHS processes with best practices outlined in the 
Software Engineering Institute's Capability Maturity Model Integration 
(CMMI) for Acquisition. In support of the approach, certain program 
offices, including DHIMS (the program office responsible for the AHLTA 
acquisition), were selected for an internal evaluation to identify 
areas for improvement in the existing MHS processes. The assessment, 
which was conducted in May 2008, identified weaknesses in processes 
such as project management, requirements development, and project 
monitoring and control, among others. It also identified weaknesses in 
MHS's oversight of the implementation of these processes within 
program offices. Specifically, the assessment identified weaknesses in 
the area of Process and Product Quality Assurance, which is supposed 
to provide staff and management with objective insight into processes 
associated with work products. The assessment found little evidence 
that process evaluations were performed across the organization, 
quality assurance audits were conducted, and noncompliance issues were 
tracked and reported. 

In response to the assessment, officials stated that they established 
a plan for addressing the identified weaknesses. Specifically, their 
goal was to achieve CMMI's "maturity level 2" for processes such as 
project planning and acquisition requirements development. Level 2 
processes are "managed" processes, or processes that are planned and 
executed in accordance with policy; employ skilled people who have 
adequate resources to produce controlled outputs; involve relevant 
stakeholders; are monitored, controlled, and reviewed; and are 
evaluated for adherence to their process description. The department 
planned to conduct a formal external assessment of the maturity of its 
processes by December 2008. 

Program officials stated that they provided guidance and assistance 
for program offices to adopt practices associated with CMMI maturity 
level 2 processes. However, they have yet to perform the planned 
external assessment of their processes, and there is therefore little 
assurance that improvements have been carried out. As the department 
proceeds with the AHLTA stabilization effort, it is critical that it 
have disciplined processes in place to avoid past problems with not 
delivering system improvements as planned. Further, as the department 
is allocating resources to and planning for the EHR Way Ahead 
acquisition, it is critical that it have disciplined management 
processes in place to avoid repeating the mistakes of the past. Until 
the department ensures that these disciplined and managed processes 
are in place, it risks delivering another system with limited 
functionality and performance problems and that does not meet the 
needs of its users. 

Conclusions: 

After over a decade of effort, DOD has not accomplished what it set 
out to achieve in acquiring a comprehensive electronic health record 
system. While it has delivered a number of outpatient capabilities, 
weaknesses in key management areas hindered its ability to deliver the 
full complement of intended capabilities and to ensure that the 
capabilities it has delivered meet required performance parameters. 
The program office did not maintain a comprehensive and current 
project management plan, a critical document that provides 
stakeholders insight into the project's plans and status. Also, 
despite the department's need to deliver a complex, worldwide system, 
it did not develop a systems engineering plan to help address the 
technical aspects of the project, and it continues to experience 
problems with system availability, speed, and usability. Further, the 
system requirements were too general and did not adequately reflect 
user needs. Although the department has collected user feedback, it 
did not establish a comprehensive plan for improving user satisfaction 
with the system. Recognizing weaknesses in acquisition management 
areas, the MHS CIO issued guidance for improving its management 
processes, but it has not performed the planned external assessment 
that it needs to certify that these improvements have been made or 
established a date for doing so. 

As DOD continues to invest significant resources in a stabilization 
effort to address shortcomings of AHLTA and plan for the acquisition 
of a new electronic health record system, it is imperative that the 
department take immediate steps to improve its management of the 
initiative. Until it does so, it risks a continuation of the problems 
it has already experienced, which could again prevent DOD from 
delivering a comprehensive health record system for serving its 
service members and their families. 

Recommendations for Executive Action: 

To help guide and ensure the successful completion of the AHLTA 
stabilization effort, we recommend that the Secretary of Defense, 
through the Assistant Secretary of Defense for Health Affairs, direct 
the MHS CIO to take the following six actions: 

* Develop and maintain a comprehensive project plan that includes key 
elements, such as the project's scope, cost, schedule, and risks and 
update the plan to provide key information for stakeholders on the 
project's plans and status. 

* Develop a systems engineering plan in accordance with DOD guidance 
to address the technical complexities of delivering a worldwide 
electronic health record system. 

* Ensure that its requirements development process involves system 
users throughout the development process, to obtain an understanding 
of what will satisfy their needs. 

* Ensure the establishment of bidirectional traceability for all 
system requirements. 

* Develop and document a plan for improving user satisfaction that 
prioritizes improvement projects; identifies needed resources; 
includes schedules for improvement efforts, including future user 
feedback surveys; and links efforts to measurable outcomes and 
specific user needs. 

* Establish acquisition management processes in accordance with 
industry best practices, including identifying milestones and a 
completion date for the external evaluation that MHS's processes are 
at maturity level 2 of the Capability Maturity Model Integration for 
Acquisition. 

Further, to help ensure that the EHR Way Ahead does not have 
shortfalls similar to those experienced with AHLTA, we recommend that 
the above six management practices be implemented as part of the 
planning for this important initiative. 

Agency Comments and Our Evaluation: 

The Deputy Assistant Secretary of Defense (Force Health Protection and 
Readiness), performing the duties of the Assistant Secretary of 
Defense (Health Affairs), provided written comments on a draft of this 
report. In its comments, the department agreed with our six 
recommendations and described actions planned to address them. 

Specifically, to help guide and ensure the successful completion of 
the AHLTA stabilization effort, DOD stated that it will develop and 
maintain a comprehensive project plan in accordance with our 
recommendation and DOD acquisition program guidelines. It also stated 
that it plans to develop a systems engineering plan to address the 
technical complexities of the project in accordance with current DOD 
requirements. Further, to obtain an understanding of system users' 
needs, the department stated that it plans to engage users and manage 
the requirements development process in accordance with our 
recommendation. The department stated that it will ensure that 
bidirectional traceability is performed for all system requirements. 
Regarding its intent to develop and document a plan for improving user 
satisfaction, including identifying needed resources and a schedule 
for improvement, the department stated that it will augment its 
current user feedback plan to include these and other key elements, 
such as measurable outcomes. Further, in response to the need to 
establish acquisition management processes in accordance with industry 
best practices, at maturity level 2, the department said it plans to 
establish a milestone for completing the external review in accordance 
with Capability Maturity Model guidelines. 

Finally, the department stated that it will ensure that the six 
recommendations are implemented as part of future EHR Way Ahead 
initiative. To the extent that the department follows through on 
implementing the recommendations, it should be better positioned to 
deliver a comprehensive electronic health care record for serving its 
service members and others entitled to military health care. 

DOD also provided technical comments on our draft report. In these 
comments, DOD said it took exception to several inaccurate, 
misleading, and subjective statements provided in the report. The 
department said that GAO's statements conflicted with the extensive 
volume of programmatic documentation, written responses, and 
consistent interview feedback provided during the course of the audit. 
In particular, the department believed that the report did not 
sufficiently reflect AHLTA's operational capabilities and its benefit 
to DOD's worldwide health care operations. While we agree that the 
department provided substantial documentation, we believe that our 
analysis of the information received supports our findings. Where 
appropriate, however, we have made revisions to statements in the 
report to update our discussions of AHLTA's operational capabilities 
and the program's management. 

The department's written comments are reproduced in appendix II. The 
department also provided technical comments, which we have 
incorporated in the report as appropriate: 

We are sending copies of the report to appropriate congressional 
committees, the Secretary of Defense, and other interested parties. In 
addition, the report will be available at no charge on the GAO Web 
site at [hyperlink, http://www.gao.gov]. 

If you or your staff have questions about this report, please contact 
me at (202) 512-6304 or melvinv@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report are 
listed in appendix III. 

Sincerely yours, 

Signed by: 

Valerie C. Melvin: 
Director, Information Management and Human Capital Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to (1) determine the Department of Defense's (DOD) 
status in implementing the Armed Forces Health Longitudinal Technology 
Application (AHLTA) system, (2) determine the department's plans for 
acquiring a new electronic health record system, and (3) evaluate the 
department's acquisition management for its electronic health record 
system. 

To determine the department's status in implementing the AHLTA system, 
we reviewed project status reports, acquisition decision memorandums, 
quarterly defense acquisition executive summaries, monthly in-progress 
review reports, monthly contractor performance reports, and 
overarching integrated project team meeting minutes. We supplemented 
these reviews with interviews of DOD officials in the Defense Health 
Information Management System (DHIMS) Program Office, including the 
DHIMS Program Manager, Deputy Program Manager, and Director of 
Products Branch officials with whom we discussed the project's cost 
and schedule, as well as the planning, development, and deployment of 
the original and current release of AHLTA. We also attended two 
demonstrations of AHLTA: at the program office located in Falls 
Church, Virginia, and at the National Naval Medical Center in 
Bethesda, Maryland. We observed demonstrations of AHLTA system 
functionality and held discussions with system users. We also observed 
a daily technical review meeting with technical staff from the Army, 
Navy, and Air Force in which the discussion largely focused on the 
reporting of issues that caused the system to be unavailable to users 
at various locations for up to 24 hours. The discussion also included 
identification of known root causes of the availability problems 
(e.g., incorrectly configured firewalls, tripped network circuits, and 
problems with virtual private networks) and planned actions to address 
the issues. 

To determine the department's plans for acquiring a new system, we 
reviewed Electronic Health Record (EHR) Way Ahead planning documents. 
Specifically, we reviewed the acquisition decision memorandum issued 
by the milestone decision authority, the Joint Requirements Oversight 
Council-approved Initial Capabilities Document to identify EHR needs, 
and the Capabilities-Based Assessment. We also reviewed the analysis 
of alternatives procedures for guidance on determining a technology 
solution for the new EHR. We also reviewed department briefings issued 
between 2008 and 2010, as well as a prepared statement to Congress 
from 2009 on preliminary plans for the EHR Way Ahead. These documents 
provided a high-level overview of the need and the goals for the new 
system, as well as plans for the system's enterprise architecture and 
expected capabilities. We supplemented our review by interviewing 
officials from the EHR Way Ahead planning office, including the 
department's Acting Chief Information Officer, the DHIMS Program 
Manager, and the DHIMS Deputy Program Manager. 

To evaluate the department's acquisition management for its electronic 
health record system initiative, we evaluated key practices used by 
the agency against best practices. In this regard, we examined 
practices related to project management planning, systems engineering 
planning, system requirements development and management, and user 
satisfaction improvement planning and compared the agency's work with 
agency policy, guidance, and recognized best practices. Specifically: 

* To assess DOD's project planning for AHLTA, we compared the 
program's project management plan against relevant guidance, including 
the Military Health System's project management process area 
description and our Information Technology Investment Management 
framework for assessing and improving process maturity. 

* We assessed the agency's approach to systems engineering by 
comparing program documentation such as acquisition strategies and the 
AHLTA project management plan to systems engineering guidance from the 
Defense Acquisition University on systems engineering. We also 
reviewed relevant agency policies, such as DOD Instruction 5000.02 
which discusses the use of systems engineering across the acquisition 
life cycle and memorandums from the Office of the Under Secretary of 
Defense on a 2004 revision to the policy regarding use of a systems 
engineering plan, to determine whether the AHLTA program was guided by 
appropriate systems engineering planning documents such as a systems 
engineering plan. 

* Regarding requirements development, we reviewed program procedures 
describing the processes for developing requirements and reviewed 
relevant external evaluations of the effectiveness of those processes 
against recognized guidance. Specifically, we reviewed an external 
evaluation of the requirements development processes including the 
2002 Carnegie Mellon External Assessment of the AHLTA program office 
and the process area description or requirements management. We also 
reviewed the 2008 internal assessment of requirements management; a 
2009 concept of operations document for a more integrated, 
departmentwide requirements development process; and the 2010 Joint 
Requirements Oversight Council-approved Initial Capabilities Document, 
which identifies past challenges with the department's requirements 
processes. In addition, we analyzed the requirements traceability 
matrix for the most recent version of AHLTA to determine the extent to 
which bidirectional traceability had been performed. We also reviewed 
program documentation relative to requirements development and user 
community participation. In addition, we interviewed process 
improvement officials, including the cognizant official from the 
Office of the Chief Information Officer (CIO) about internal 
acquisition process evaluations and their results and the status of 
plans for improving acquisition management processes. We then compared 
the department's current approach to requirements development and 
management with best practices identified in the Software Engineering 
Institute's Capability Maturity Model Integration for Acquisition. 

* To assess the department's approach to improving user satisfaction, 
we reviewed and analyzed program documentation pertaining to the 
collection, analysis, and utilization of AHLTA user satisfaction 
feedback such as seven survey reports and a postimplementation review 
that were produced between 2005 and 2007 and compared the agency's 
approach to best practices such as the Office of Management and 
Budget's Capital Programming Guide and Standards and Guidelines for 
Statistical Surveys. We also reviewed lessons learned reports from 
2006 through 2008 and a user conference briefing from 2006 that 
identified areas of user dissatisfaction. In addition, we reviewed 
program office documents that identified improvement initiatives such 
as the AHLTA 3.3 software release and the deployment of local cache 
servers, which were intended to improve user satisfaction. We 
supplemented our review by interviewing program officials, including 
the DHIMS Program Manager and Deputy Program manager, to determine the 
extent to which user satisfaction improvement efforts and initiatives 
have been guided by documented plans. We then compared the 
department's approach to improving user satisfaction with the Software 
Engineering Institute's IDEALSM[Footnote 23] model, which is a 
recognized approach for managing process improvement efforts such as 
managing improvements to user satisfaction. 

* To assess DOD's plans to improve acquisition management processes, 
we reviewed documentation and interviewed officials from the DHIMS 
Program Office and the Office of Process Improvement on their plans to 
improve the processes based on the Software Engineering Institute's 
Capability Maturity Model Integration for Acquisition. We also 
reviewed DOD's 2008 internal assessment related to acquisition 
management processes, action plans, and tasks planned for process 
improvement. 

We supplemented our analysis with interviews with officials in the 
DHIMS Office, including, the Program Manager, Deputy Program Manager, 
Director of Products Branch and Engineering and Resources offices. We 
also obtained written responses from the responsible program manager 
or subject matter expert for areas of our review. These responses were 
approved by the MHS CIO or the Program Executive Officer, Joint 
Medical Information Systems/Deputy MHS CIO. 

We did not conduct an independent validation of the life-cycle costs 
and obligations provided to us by DOD. 

We conducted this performance audit at the DHIMS Program Office in 
Falls Church, Virginia, and the National Naval Medical Center, in 
Bethesda, Maryland, from September 2009 through October 2010 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our objectives. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Assistant Secretary Of Defense: 
Health Affairs: 
TRICARE Management Activity: 
Skyline Five, Suite 810, 5111 Leesburg Pike: 
Falls Church, Virginia 22041-3206: 

September 27, 2010: 

Ms. Valerie C. Melvin: 
Director, Information Management and Human Capital Issues: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Ms. Melvin: 

This is the Department of Defense (DoD) response to the 
recommendations in the Government Accountability Office (GAO) Draft 
Report GAO-11-50, "Information Technology — Opportunities Exist to 
Improve Management of DoD's Electronic Health Record Initiative," 
October 2010 (Engagement Code 310944). 

DoD acknowledges receipt of the draft report and will address each of 
the recommendations and ensure appropriate measures are carried out 
effectively. DoD takes exception to several inaccurate, misleading, 
and subjective statements provided in the draft report. GAO's 
statements conflict with the extensive volume of programmatic 
documentation, written responses, and consistent interview feedback 
provided during the course of this audit. Enclosed are suggested 
technical comments and corrections to GAO's draft report. 

Thank you for the opportunity to review and comment on the draft 
report. The points of contact for additional information are Ms. Lois 
Kellett, Lois.Kellett@tma.osd.mil, or (703) 681-8836; and Mr. Gunther 
Zimmerman, Gunther.Zimmerman@tma.osd.mil, or (703) 681-4360. 

Sincerely, 

Signed by: 
George Peach Taylor, Jr., M.D.
Deputy Assistant Secretary of Defense (Force Health Protection and 
Readiness): 
Performing the Duties of the Assistant Secretary of Defense (Health 
Affairs): 

Attachments: As stated: 

[End of letter] 

GAO Draft Report-Dated October 2010: 
GAO-11-50 (Engagement Code 310944): 

"Opportunities Exist to Improve Management of Doll's Electronic Health 
Record Initiative" 

Department of Defense Comments to GAO Recommendations: 

Recommendation: Develop and maintain a comprehensive project plan that 
includes key elements, such as the project's scope, cost, schedule and 
risks and update the plan to provide key information for stakeholders 
on the project's plans and status. 

DoD Response: Concur. DoD will develop and maintain a comprehensive 
project plan in accordance with this recommendation and DoD 
acquisition program guidelines. 

Recommendation: Develop a systems engineering plan in accordance with 
DoD guidance to address the technical complexities of delivering a 
worldwide electronic health record system. 

DoD Response: Concur. Since implementing a requirement to develop a 
Systems Engineering Plan (SEP) in February 2004, DoD will continue to 
develop and maintain a SEP in accordance with this recommendation. 

Recommendation: Ensure that the requirements development process 
involves system users throughout the development process, to obtain an 
understanding of what will satisfy their needs. 

DoD Response: Concur. DoD will continue to engage system users and 
manage the requirements development process in accordance with this 
recommendation. 

Recommendation: Ensure the establishment of bidirectional traceability 
for all system requirements. 

DoD Response: Concur. DoD will ensure the bidirectional traceability 
for requirements in accordance with this recommendation. 

Recommendation: Develop and document a plan for improving user 
satisfaction that prioritizes improvement projects; identifies needed 
resources; includes schedules for improvement efforts, including 
future user feedback survey; and links efforts to measureable outcomes 
and specific user needs. 

DoD Response: Concur. DoD will augment its current user feedback plan 
for improved user satisfaction in accordance with this recommendation. 

Recommendation: Establish acquisition management processes in 
accordance with industry best practices, including indentifying 
milestones and a completion date for the external evaluation that 
MHS's processes are at maturity Level 2 of the Capability Management 
Maturity Model Integrated for Acquisition. 

DoD Response: Concur. DoD continues to follow DoD acquisition program 
guidelines and will establish a milestone for an external review in 
accordance with the Capability Management Maturity Model guidelines. 

Recommendation: Further, to help ensure that the EHR Way Ahead does 
not have shortfalls similar to those experienced with AHLTA, we 
recommend that the above management practices be implemented as part 
of the planning for this important initiative. 

DoD Response: Concur. DoD will ensure that these recommendations are 
implemented as part of the EIIR Way Ahead initiative. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Cynthia J. Scott (Assistant 
Director); Harold Brumm, Jr.; Neil Doherty; Ronalynn Espedido; Rebecca 
Eyler; Nancy Glover; Joel Grossman; Linda Kochersberger; Lee 
McCracken; Madhav Panwar; Donald Sebers; Sylvia Shanks; Adam Vodraska; 
Daniel Wexler; and Robert Williams, Jr. made key contributions to this 
report. 

[End of section] 

Footnotes: 

[1] Each of these systems provided certain patient-related 
information. For example, the Ambulatory Data System captured certain 
outpatient information relating to diagnosis and treatment; the 
Preventive Health Care Application contained information on preventive 
health services; and the Nutrition Management Information System 
supported therapeutic nutrition therapy and medical food management. 

[2] Open systems conform to industry standards so that commercial 
products can easily be used and support costs can be minimized. A 
client is usually a desktop computing device or program that is 
"served" by one or more networked computing devices. 

[3] Military hospitals currently use Essentris, a commercial-off-the- 
shelf product, to document inpatient encounters that were originally 
planned for Block 4. As of March 2010, inpatient functionality was 
deployed at 29 sites, representing 62 percent of the Military Health 
System's inpatient beds. 

[4] We have identified DOD contracting in our high-risk list since 
1992, and DOD business systems modernization as high risk since 1995; 
however, we did not explicitly identify DOD's health care information 
technology procurement processes as high-risk areas. See GAO, High-
Risk Series: An Update, [hyperlink, 
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: Jan. 22, 
2009). 

[5] A fixed-price contract provides for a firm price or, in 
appropriate cases, a ceiling or adjustable price. A time-and-materials 
contract provides for acquiring supplies or services on the basis of 
direct labor hours at specified fixed hourly rates that include wages, 
overhead, general and administrative expenses, and profit and actual 
cost of materials. A cost-reimbursement contract provides for payment 
of allowable incurred costs, to the extent prescribed in the contract. 

[6] The Federal Acquisition Regulation allows for contracts awarded 
without full and open competition under certain circumstances and 
requires written justification that addresses these circumstances. 

[7] The noncompetitive contracts' costs are about 2 percent of 
obligations of approximately $2 billion. 

[8] Federal Acquisition Regulation, Part 16.505 (b) (2) i-iii. 

[9] The defense acquisition system is a framework-based approach that 
is intended to translate mission needs and requirements into stable, 
affordable, and well-managed acquisition programs. 

[10] AHLTA is assigned acquisition category IAM, which is the highest 
information system acquisition category for IT Systems and is assigned 
to acquisitions with at least $126 million in fiscal year 2000 
constant dollars in development and deployment costs or at least $378 
million in fiscal year 2000 constant dollars for all system costs. 

[11] DOD Office of the Inspector General, Acquisition Management of 
the Composite Health Care II Automated Information System, report 
number 99-068 (January 21, 1999). 

[12] DOD Office of the Inspector General, Information Technology 
Management: Acquisition of the Armed Forces Health Longitudinal 
Technology Application, report number D-2006-089 (May 18, 2006). 

[13] GAO, Information Technology: Greater Use of Best Practices Can 
Reduce Risks in Acquiring Defense Health Care System, [hyperlink, 
http://www.gao.gov/products/GAO-02-345] (Washington, D.C.: Sept. 26, 
2002). 

[14] This effort included the AHLTA 3.3 software release discussed 
above. 

[15] In addition, DOD plans to spend $40 million on a related effort 
to test the exchange of electronic health records with the Department 
of Veterans Affairs and private health care providers and to work 
toward a goal announced by President Obama on April 9, 2009, that the 
departments would cooperate to create a joint virtual lifetime 
electronic health record for service members and veterans. 

[16] See Institute of Electrical and Electronics Engineers (IEEE), 
IEEE/EIA Guide for Information Technology, IEEE/EIA 12207.1-1997 
(April 1998) and Carnegie Mellon Software Engineering Institute, 
Capability Maturity Model Integration for Acquisition, Version 1.2 
(Pittsburgh, Pa, November 2007). 

[17] GAO, Information Technology Investment Management: A Framework 
for Assessing and Improving Process Maturity, [hyperlink, 
http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March 
2004). 

[18] Carnegie Mellon Software Engineering Institute, Capability 
Maturity Model Integration for Acquisition, Version 1.2. 

[19] According to DOD, the desirable target for AHLTA system 
availability is 100 percent, meaning that the system is available to 
users whenever it is needed, and the performance threshold is 99 
percent, meaning that if availability falls below 99 percent, 
performance is considered to be unacceptable. Further, a system 
performance report for the time period October 2008 to February 2010 
did not show any months with availability at the desired level of 100 
percent, and only 1 month when it was available at the acceptable 
level between 99 and 100 percent, and then only at the Army and Navy 
facilities. The Air Force experienced the lowest levels of 
availability, with 7 months that were between 93 and 97 percent 
availability. The report showed that system downtime included some 
system maintenance, but the primary cause of downtime was 
implementation of improvements to address performance problems. 

[20] See Federal Acquisition Regulation 39.102 and Carnegie Mellon, 
Software Engineering Institute, Capability Maturity Model-Integration 
for Development, Version 1.2 (Pittsburgh, Pa., August 2006) and 
Software Acquisition Capability Maturity Model, Version 1.03, CMU/SEI- 
2002-TR-010 (Pittsburgh, Pa., March 2002). 

[21] The Software Engineering Institute is a federally funded research 
and development center established at Carnegie Mellon University to 
address software engineering practices. IDEALSM is a service mark of 
Carnegie Mellon University and stands for initiating, diagnosing, 
establishing, acting, and leveraging. For more information on this 
model, see IDEALSM: A User's Guide for Software Process Improvement 
(CMU/SEI-96-HB-001). 

[22] The IEEE is a nonprofit, technical professional association that 
develops standards for a broad range of global industries, including 
the IT and information assurance industries and is a leading source 
for defining best practices. 

[23] The Software Engineering Institute is a federally funded research 
and development center established at Carnegie Mellon University to 
address software engineering practices. IDEAL is a service mark of 
Carnegie Mellon University and stands for initiating, diagnosing, 
establishing, acting, and leveraging. For more information on this 
model, see IDEALSM: A User's Guide for Software Process Improvement 
(CMU/SEI-96-HB-001). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: