This is the accessible text file for GAO report number GAO-09-888 
entitled 'Information Technology: DOD Needs to Strengthen Management of 
Its Statutorily Mandated Software and System Process Improvement 
Efforts' which was released on September 8, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

September 2009: 

Information Technology: 

DOD Needs to Strengthen Management of Its Statutorily Mandated Software 
and System Process Improvement Efforts: 

GAO-09-888: 

GAO Highlights: 

Highlights of GAO-09-888, a report to congressional requesters. 

Why GAO Did This Study: 

The Department of Defense’s (DOD) acquisition of weapon systems and 
modernization of business systems have both been on GAO’s list of high-
risk areas since 1995. To assist DOD in managing software-intensive 
systems, Section 804 of the Bob Stump National Defense Authorization 
Act for Fiscal Year 2003 required the Office of the Secretary of 
Defense (OSD) and DOD component organizations, including the military 
departments, to undertake certain software and systems process 
improvement (SSPI) actions. As requested, GAO assessed (1) the extent 
to which DOD has implemented the process improvement provisions of the 
act, and (2) the impact of DOD’s process improvement efforts. To do so, 
GAO analyzed relevant plans, policies, guidance, performance measures, 
and reports against statutory requirements and relevant guidance, and 
interviewed DOD officials. 

What GAO Found: 

OSD and the military departments have implemented a number of statutory 
requirements aimed at improving their processes for acquiring software-
intensive systems. However, they have not satisfied all of their 
respective statutory requirements, or key aspects of relevant SSPI 
guidance. In particular, 

* OSD has issued guidance calling for military departments and defense 
agencies to implement process improvement programs, revised guidance to 
emphasize contractor past performance in source selection decisions, 
and established a clearinghouse for software and system acquisition and 
development best practices, all of which are required by the statute. 
However, it has not implemented a requirement in the statute related to 
overseeing DOD component organization process improvement programs to 
ensure compliance with its guidance, and it has not satisfied a key 
aspect of relevant guidance pertaining to monitoring organizationwide 
process improvement efforts. According to OSD, process improvement is a 
component responsibility and thus it does not view oversight of 
component SSPI efforts as necessary. Without strong, central leadership 
over DOD’s improvement efforts, OSD is not fulfilling key tenets of 
section 804 and relevant guidance associated with well-managed software 
process improvement programs, and has increased the risk that component 
process improvement efforts and their impacts are not being maximized. 

* The military departments have established process improvement 
programs, although two did not do so within the time frame specified in 
the statute. Also, each has documented processes that address the four 
key software process areas cited in the statute, and have taken steps 
to ensure that key personnel have the appropriate level of 
software/system-related experience or training, and to develop process 
improvement performance metrics, as required by the statute. However, 
none is using these performance metrics for continuous process 
improvement, as provided for in the statute and relevant guidance. 
Also, while each has a process governing implementation of key 
acquisition requirements, these processes do not fully reflect the 
range of verification steps advocated in relevant guidance. Reasons 
cited for the state of the department’s respective efforts include 
senior leadership turnover and not viewing all the statutory 
requirements as necessary. By not having fully implemented the statute 
and relevant guidance, the military departments are not positioned to 
maximize the potential of their process improvement efforts. 

Neither OSD nor the military departments have measured the impact of 
their collective or separate process improvement efforts. However, 
studies by GAO and others continue to identify system and software 
acquisition and development process weaknesses, as well as cost, 
schedule, and performance shortfalls, across a range of DOD software-
intensive programs, thus suggesting that the potential value of these 
efforts has yet to be fully realized. 

What GAO Recommends: 

GAO is making recommendations to the Secretary of Defense aimed at OSD 
and DOD components adopting the kind of strategic approach to process 
improvement embodied in section 804 and relevant guidance, and 
reporting to congressional defense committees on the progress and 
impacts of doing so. DOD partially agreed with GAO’s recommendations 
and described actions to address each. While GAO supports DOD’s 
actions, it does not believe they are sufficient to effectively manage 
a departmentwide SSPI program. 

View GAO-09-888 or key components. For more information, contact 
Randolph C. Hite at (202) 512-3439 or hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DOD Has Not Fully Implemented Statutory Requirements and Guidance for 
Improving Software and System Processes: 

DOD Does Not Know Impact of SSPI Efforts but Studies of Large-Scale 
Acquisition Programs Show that Performance Shortfalls and Process 
Weaknesses Continue: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Table: 

Table 1: Extent to which Military Departments Have Documented Processes 
for Each Key Software Area: 

Figures: 

Figure 1: View of DOD Web-based Portal: 

Figure 2: Timeline of Establishment of SSPI Programs: 

Abbreviations: 

2003 NDAA: Bob Stump National Defense Authorization Act for fiscal year 
2003: 

ASD(C3I): Assistant Secretary of Defense for Command, Control, 
Communications, and Intelligence: 

ASD(NII)/CIO: Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer: 

CMMI: Capability Maturity Model Integration: 

DOD: Department of Defense: 

EVM: earned value management: 

GCSS-MC: Global Combat Support System - Marine Corps: 

Navy ERP: Navy Enterprise Resource Planning: 

OASD(C3I): Office of the Assistant Secretary of Defense for Command, 
Control, Communications, and Intelligence: 

OASD(NII)/CIO: Office of the Assistant Secretary of Defense for 
Networks and Information Integration/Chief Information Officer: 

OSD: Office of the Secretary of Defense: 

OUSD(AT&L): Office of the Undersecretary of Defense for Acquisition, 
Technology, and Logistics: 

PEO: program executive officer: 

PSR: Program Support Review: 

SEI: Software Engineering Institute: 

SSPI: software and systems process improvement: 

USD(AT&L): Under Secretary of Defense for Acquisition, Technology, and 
Logistics: 

[End of section] 

United States Government Accountability Office:
Washington, DC 20548: 

September 8, 2009: 

The Honorable Evan Bayh: 
Chairman: 
The Honorable Richard Burr: 
Ranking Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

The Honorable Daniel K. Akaka: 
United States Senate: 

The Honorable John Ensign: 
United States Senate: 

The Department of Defense (DOD) relies heavily on software-intensive 
systems to support military operations and associated business 
functions, such as logistics, personnel, and financial management. One 
important determinant of the quality of these systems, and thus DOD's 
mission performance, is the quality of the processes used to develop 
and acquire them. Recognizing the importance of these processes in 
producing systems that perform as intended and meet cost and schedule 
goals, successful public and private organizations have adopted and 
implemented software and systems process improvement (SSPI) programs. 
[Footnote 1] 

Section 804 of the Bob Stump National Defense Authorization Act for 
fiscal year 2003 (2003 NDAA)[Footnote 2] requires military departments, 
and defense agencies that manage major acquisition programs, to 
establish process improvement programs.[Footnote 3] This report 
responds to your request to review the department's implementation of 
this requirement. As agreed, our objectives were to determine: (1) the 
extent to which DOD has implemented the process improvement provisions 
of the act, and (2) the impact of DOD's process improvement efforts. To 
accomplish these objectives, we reviewed SSPI policies, guidance, 
plans, oversight controls, and performance measures, and compared these 
to the statutory requirements and relevant guidance; we interviewed 
responsible officials of the Office of the Assistant Secretary of 
Defense for Networks and Information Integration/Chief Information 
Officer (OASD(NII)/CIO), the Office of the Under Secretary of Defense 
for Acquisition, Technology, and Logistics (OUSD(AT&L)), and the 
military departments; and reviewed and analyzed related studies and 
reports on the impact of DOD's SSPI efforts. 

We conducted this performance audit at DOD and military department 
offices in Arlington, Virginia, from December 2008 through September 
2009 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. Appendix I 
further discusses our scope and methodology. 

Background: 

DOD is a massive and complex organization. To meet its missions, the 
department relies on a complex array of computer-dependent and mutually 
supportive organizational components, including the military 
departments and defense agencies. It also relies on a broad array of 
systems to support operations related to intelligence, surveillance, 
security, and sophisticated weaponry--as well as financial management 
and other business functions. 

DOD's investment in major acquisition programs, including largely 
software-intensive weapons systems, is expected to be about $357 
billion over the next 5 years.[Footnote 4] We have designated DOD's 
business systems modernization and its acquisition of weapons systems 
as high-risk areas.[Footnote 5] 

The quality of the processes involved in developing and acquiring 
software and systems has a significant effect on the quality of the 
resulting products. Public and private organizations have reported 
significant returns on investment through improvements to these 
processes. For example, the Software Engineering Institute[Footnote 6] 
(SEI) reported in 2006[Footnote 7] that a major defense contractor 
implemented a process improvement program and improved its system 
development earned value management cost and schedule performance by 5 
percent and 8 percent, while reducing cost and schedule variability by 
34 percent and 50 percent, respectively. It also reported that the 
contractor reduced system defects by about 44 percent. Further, SEI 
reported that a defense software maintenance group decreased the cost 
of its services by an average of 27 percent and reduced its effort 
required to deliver test programs by 25 percent. 

Summary of GAO's Prior Review on DOD's SSPI Programs: 

In 2001, we reported that DOD lacked a corporate approach to guiding 
and overseeing the military department and defense agency SSPI 
activities, and as a result, the scope and nature of these activities 
varied.[Footnote 8] For example, the Air Force, Army, and certain Navy 
units had established programs that generally satisfied the tasks of 
SEI's IDEALSM[Footnote 9] process improvement model, while the Marine 
Corp and other Navy units did not. Further, the military departments 
were using different management strategies for directing and 
controlling their respective SSPI activities. In light of these 
variations and the opportunity for DOD's component organizations to 
learn from and leverage each other's experiences and best practices, we 
concluded that the Office of the Secretary of Defense (OSD) had an 
important leadership role to play in expanding SSPI across the 
department. Accordingly, we recommended that the Secretary of Defense: 

* direct DOD component organizations to begin software process 
improvement efforts where our report showed none existed, and that 
these organizations consider following the best practices embodied in 
the SEI IDEALSM model and drawn from the experiences of other component 
organizations that have successfully implemented SSPI programs; 

* direct the Assistant Secretary of Defense for Command, Control, 
Communications and Intelligence (ASD(C3I)),[Footnote 10] in 
collaboration with the Under Secretary of Defense for Acquisition, 
Technology, and Logistics (USD(AT&L)), to (1) issue a policy requiring 
DOD components that are responsible for software-intensive systems 
development, acquisition, or engineering to implement SSPI programs, 
and (2) develop and issue SSPI guidance, and, in doing so, consider 
basing this guidance on the SEI IDEALSM model and the positive examples 
within the military departments and defense agencies cited in our 
report; and: 

* direct the ASD(C3I) to (1) annually determine the components' 
compliance with the SSPI policy and (2) establish and promote a means 
for sharing SSPI lessons learned and best practices knowledge through 
DOD. 

In response, DOD agreed that SSPI practices should be pursued and 
encouraged, and that information about SSPI practices should be shared 
among DOD components. 

Summary of Section 804 of the 2003 National Defense Authorization Act: 

Congress included several provisions in section 804 of the 2003 NDAA 
related to SSPI that are consistent with the SSPI recommendations we 
provided to DOD in 2001.[Footnote 11] In general, these provisions 
provide for a strategic and corporate approach to SSPI in the 
department by placing certain requirements on organizations within OSD 
as well as other requirements on the military departments and defense 
agencies. The provisions are as follows: 

Office of the Secretary of Defense: 

The ASD(C3I), in collaboration with the USD(AT&L), shall: 

* Prescribe uniformly applicable guidance for the administration of all 
the software process improvement programs established by this mandate. 

* Take such actions as are necessary to ensure that the military 
departments and defense agencies comply with this guidance. 

* Assist the secretaries of the military departments and heads of 
defense agencies to carry out such programs effectively by: (1) 
ensuring that criteria applicable to the selection of sources provide 
added emphasis on past performance of potential sources, as well as on 
the maturity[Footnote 12] of the software products offered by the 
potential sources; and (2) identifying, and serving as a clearinghouse 
for, information regarding best practices in software development in 
both the public and private sectors. 

Military Departments and Defense Agencies: 

The secretary of each military department and head of each defense 
agency that manages a major defense acquisition program with a 
substantial software component shall: 

* Establish a program to improve the software acquisition processes of 
that military department or defense agency within 120 days after the 
act's enactment. 

* Ensure that a program to improve software acquisition processes 
includes, at a minimum, the following: (1) a documented process for 
software acquisition planning, requirements development and management, 
project management and oversight, and risk management; (2) efforts to 
develop appropriate metrics for performance measurement and continual 
process improvement; (3) a process to ensure that key program personnel 
have an appropriate level of expertise or training in software 
acquisition; and (4) a process to ensure implementation and adherence 
to established processes and requirements relating to the acquisition 
of software. 

DOD Has Not Fully Implemented Statutory Requirements and Guidance for 
Improving Software and System Processes: 

OSD and the military departments have met some, but not all of their 
respective statutory requirements[Footnote 13] aimed at adopting a 
corporate and strategic approach to improving DOD's processes for 
developing and acquiring software-intensive systems, although the 
military departments vary in how and the extent to which they have and 
have not met their requirements. In addition, neither OSD nor the 
military departments have established programs that fully utilize SSPI 
guidance. 

OSD officials responsible for implementing applicable provisions of the 
statute cited various reasons for not meeting all the requirements, 
including requirements being inconsistent with OSD's role. Reasons 
cited by military department officials included changes in senior 
leadership and not viewing all requirements as necessary. Regardless, 
this means that neither OSD nor the military departments fully complied 
with requirements of section 804, and as a result, have increased the 
risk that the billions of dollars being spent each year on DOD software-
intensive system acquisitions will not benefit from an effectively and 
efficiently managed corporate approach to SSPI. 

OSD Has Partially Satisfied Statutory SSPI Requirements: 

OSD has partially implemented the statutory SSPI requirements that 
apply to it, and relevant process improvement guidance, which are aimed 
at providing a corporate and strategic approach to SSPI efforts. To 
their credit, the Assistant Secretary of Defense for Networks and 
Information Integration/Chief Information Officer (ASD(NII)/CIO) and 
USD(AT&L) jointly issued a memorandum that, among other things, 
established DOD's Software Acquisition Process Improvement Program and 
provided guidelines and expectations for OSD and component 
organizations relative to developing and implementing their respective 
SSPI programs. OUSD(AT&L) also has revised existing guidance on source 
selection to emphasize contractor past performance, and established a 
clearinghouse for information on SSPI-related best practices and 
lessons learned. In addition, OSD has taken other steps to assist the 
military departments in implementing their respective SSPI efforts. 

However, OSD has not taken steps to ensure that component organizations 
comply with its SSPI guidance and expectations, and it has not 
emphasized the maturity of software products in existing source 
selection guidance. Without such oversight and guidance, DOD cannot 
adequately ensure that its organizational components are effectively 
and efficiently implementing SSPI activities, and thus that the risks 
associated with acquiring software-intensive systems are being 
minimized. 

OSD Has Issued Guidance and Expectations for Developing and 
Implementing SSPI Programs: 

The statute[Footnote 14] states that the ASD(C3I), in collaboration 
with the USD(AT&L), shall prescribe uniformly applicable guidance to 
the military departments and defense agencies for the administration of 
their software process improvement programs. Relatedly, SEI's IDEALSM 
[Footnote 15] model emphasizes the importance of establishing an SSPI 
management structure that includes setting goals and performance 
measures, assigning roles and responsibilities, having a plan to guide 
process improvement activities, and allocating and securing resources 
needed to execute the plan. 

Consistent with the statute, the ASD(NII)/CIO and USD(AT&L) issued a 
joint memorandum in March 2003 that created the department's Software 
Acquisition Process Improvement Program, referenced section 804, and 
established the OSD Software-Intensive Systems Steering Group to lead 
and facilitate component SSPI efforts and to recommend uniformly 
applicable guidance for the administration of these SSPI programs. 
Further, the memorandum includes guidelines and expectations for 
component organizations' SSPI programs relative to identifying specific 
goals, milestones, performance measures, and resource needs, and 
ensuring that SSPI personnel have an appropriate level of training and 
experience. The guidelines also direct DOD components to identify an 
approach and evaluation criteria to be used in guiding and assessing 
their SSPI activities and goals. According to officials in the military 
departments that are responsible for SSPI, they have used these 
guidelines and expectations in developing and implementing their 
respective programs. 

Beyond meeting the statute's requirement for prescription of guidance, 
the joint memorandum and relevant guidelines and expectations also 
satisfy key aspects of SEI's IDEALSM model. For example, the memorandum 
states that the components should, among other things, set goals and 
performance measures, assign roles and responsibilities, develop an 
approach and evaluation criteria to guide process improvement and to 
assess achievement of goals, and allocate and secure resources needed 
to execute the plan. Further, the memorandum stated that components' 
respective SSPI programs should address eight key process areas, 
including the four they are required to address under the act (see 
later section of this report for a discussion of these four process 
areas). By issuing these guidelines and expectations, OSD complied with 
the statute and took foundational steps towards DOD-wide implementation 
of SSPI. 

OSD Is Not Overseeing Component SSPI Programs to Ensure Compliance with 
Guidelines and Expectations: 

According to the statute,[Footnote 16] the ASD(C3I), in collaboration 
with the USD(AT&L), is to take necessary actions to ensure that the 
military departments and defense agencies comply with guidance for the 
administration of their software process improvement programs. 
Relatedly, SEI's IDEALSM model recognizes the importance of overseeing 
and monitoring an organization's SSPI activities. For example, it 
states that a steering group representing senior management should be 
responsible for oversight of organizational SSPI efforts. 

Neither OASD(NII)/CIO nor OUSD(AT&L) has overseen the military 
department and defense agency efforts to comply with OSD-issued SSPI 
guidelines and expectations, nor have they developed accountability 
mechanisms to ensure that the military departments and defense agencies 
comply. According to officials from both organizations, SSPI 
implementation is a component responsibility, and therefore they 
believe that DOD's role should be collaborative and facilitative, and 
should not focus on requiring compliance and accountability. Further, 
they stated that OSD visibility into component SSPI efforts does occur 
indirectly through the system acquisition Program Support 
Reviews[Footnote 17] (PSR) at major acquisition milestones, as well as 
through technical working groups, such as DOD's Software Working Group 
and Systems Engineering Forum. However, the PSRs do not address 
component compliance with the OSD-issued SSPI guidelines and 
expectations. Specifically, they do not verify that the military 
departments have developed appropriate metrics for continual process 
improvement, as required by the statute. They also do not ensure that 
the military departments followed key aspects of relevant SSPI 
guidance, such as developing and implementing strategic action plans. 
Further, while the technical working groups permit visibility into 
component SSPI efforts, these groups do not have the authority to 
ensure compliance with OSD guidelines and expectations. 

Without effective oversight and accountability mechanisms to ensure 
that components comply with the SSPI requirements, OSD has not met a 
key requirement of section 804 and relevant guidance, and it has 
increased the risk of DOD components not implementing SSPI in an 
effective and efficient manner and not maximizing DOD-wide process 
improvement outcomes. 

OSD Has Partially Addressed Other Statutory Requirements and Has Taken 
Steps to Otherwise Assist Military Departments in Developing and 
Implementing SSPI Programs: 

The statute[Footnote 18] requires specific officials within OSD to 
assist the military departments and defense agencies in implementing 
SSPI programs by (1) ensuring that criteria applicable to the selection 
of sources provides added emphasis on potential sources' past 
performance and software product maturity; and (2) identifying, and 
serving as a clearinghouse for, information regarding best practices in 
software development in both the public and private sectors. OSD has 
partially met the first statutory requirement and fully met the second. 
In addition, it has provided additional assistance to the military 
departments, including aiding them in establishing their process 
improvement efforts. 

OSD Guidance Emphasizes Importance of Contractor Past Performance 
during Source Selection, but not Maturity of Contractor Products: 

OUSD(AT&L) has issued source selection guidance that describes the 
criteria and key techniques that should be used by program offices. 
Among other things, this guidance provides for considering a 
contractor's past performance by selecting and reviewing similar 
efforts involving the contractor that are still ongoing or have just 
been completed, addressing performance expectations in the government 
and contractor's initial postaward meeting; and using presolicitation 
meetings with industry to obtain past performance information. 

However, this source selection guidance does not emphasize considering 
the maturity of contractor products as part of source selection. 
According to OUSD(AT&L) officials, steps have been taken to begin to 
address this gap in guidance but significant work remains. For example, 
they cited a 2007 USD(AT&L) memorandum that encourages programs to 
implement a competitive prototyping approach, which they described as 
one means of establishing the maturity of an acquisition program's key 
technologies during its early phases. They also said that OSD has 
recently begun working with the Navy to identify the key product 
maturity information needed prior to source selection. Notwithstanding 
these steps, source selection guidance specifically addressing product 
maturity has yet to be issued. 

Until OSD source selection guidance also emphasizes contractor product 
maturity, the department will not be in full compliance with section 
804, and it will increase the risk of acquisitions falling short of 
expectations due to the use of immature hardware and software products. 

OSD Has Established a Clearinghouse for Software and System Acquisition 
and Development Best Practices: 

OSD has established a clearinghouse of best practices information 
relative to software and system acquisition and development processes 
and SSPI. Specifically, OUSD(AT&L) has partnered with the Defense 
Acquisition University to provide both instructor-led training and a 
Web-based portal to share knowledge about software process-related 
methodologies, such as the SEI's Capability Maturity Model Integration 
(CMMI®),[Footnote 19] SEI's IDEALSM model, and Lean Six Sigma.[Footnote 
20] For example, the portal features content areas such as software 
acquisition management, the SEI CMMI Acquisition model, system 
engineering planning, and system acquisition career fields. Further, 
for each content area, relevant best practices information is provided. 
To illustrate, the CMMI Acquisition model content area features best 
practices on 22 acquisition process areas, such as requirements 
development and management, project monitoring and control, and risk 
management. The primary information sources of the portal's content are 
conference publications, journal and trade magazines, and individuals' 
best practices submissions that are vetted through a content advisory 
group. (See figure 1 for a top-level view of this portal.) 

Figure 1: View of DOD Web-based Portal: 

[Refer to PDF for image: illustration] 

Best Practices Clearinghouse, Defense Acquisition University web-based 
portal page. 

Source: [hyperlink, https://bpch.dau.mil/Pages/default.aspx]. 

[End of figure] 

By establishing a clearinghouse of information on software and system 
acquisition and development process best practices, OSD has provided 
the military departments and defense agencies with an important enabler 
for departmentwide process improvement. 

OSD Has Taken Additional Measures to Assist Components in Implementing 
SSPI Programs: 

To OSD's credit, it established the Software-Intensive Systems Steering 
Group in March 2003 to assist the military departments and defense 
agencies in setting up their process improvement programs. More 
specifically, this steering group worked with the military departments 
and defense agencies to ensure that each had established an SSPI 
effort, after which the group was disestablished and its 
responsibilities subsumed into the Systems Engineering Forum. According 
to DOD officials, this forum is responsible for facilitating 
information sharing and discussion among DOD components about their 
respective SSPI efforts. 

OSD has also established several software technical working groups that 
meet at least yearly to facilitate discussion among components on steps 
and actions needed to address identified software and system 
acquisition and development weaknesses. For example, OSD hosted a 
conference in 2006 to identify, among other things, SSPI issues, 
barriers, and recommendations. According to an OUSD(AT&L) official, 
this conference raised DOD-wide awareness on significant software and 
system engineering and management issues, such as ineffective 
requirements management, system engineering decisions being made 
without full participation of software engineers, and insufficient 
quantity and quality of software engineering expertise within the 
department. 

Military Departments Have Partially Satisfied Statutory SSPI 
Requirements and Relevant Guidance: 

The military departments have largely satisfied the SSPI statutory 
requirements that apply to them, although in doing so they have not 
always followed key aspects of SEI guidance that provide for adopting a 
corporate and strategic approach to process improvement. Specifically, 
each military department has established an SSPI program or 
organization, although only one, the Army, did so within the 
statutorily specified time frame. Further, all have largely met the 
requirements relating to ensuring that key program personnel have an 
appropriate level of experience or training in software acquisition, 
documenting certain software and system process areas, and ensuring 
that acquisition programs adhere to such documented process areas. 
However, even though each has taken steps to provide for defining and 
collecting process improvement performance measures, none are actually 
using the related metrics for continuous improvement, as provided for 
in the law. Reasons cited by military department officials varied from 
changes in senior leadership impeding program continuity to some of the 
statutory requirements not being viewed as necessary. This means that 
the military departments have not fully implemented section 804, and 
have thus limited the potential value to be derived from their SSPI 
efforts. 

Military Departments Have Established SSPI Programs, although Two Did 
Not Meet Statutory Deadline for Doing So: 

The statute required the military departments to establish process 
improvement programs within 120 days of the statute's enactment. 
[Footnote 21] When establishing an SSPI program, relevant guidance such 
as SEI's IDEALSM[Footnote 22] model advocates having a program 
management structure that includes, among other things, defined program 
goals and performance measures and a strategic action plan to guide the 
program's implementation. According to the model, such a corporate and 
strategic approach is critical to implementing SSPI effectively and 
consistently across an organization. 

Each of the military departments have established SSPI programs. 
However, only the Army did so within the statutorily required time 
frame (see figure 2). Further, the respective program management 
structures being employed vary in the extent to which they reflect 
SEI's IDEALSM model. The military department's respective programs are 
described below. 

Figure 2: Timeline of Establishment of SSPI Programs: 

[Refer to PDF for image: timeline] 

August 18, 2002: Army established the Strategic Software Improvement 
Program. 

December 2, 2002: Enactment of section 804 fiscal year 2003 NDAA. 

March 31, 2003: Statutory deadline for military department creation of 
a software and systems process improvement program. 

January 13, 2004: Air Force established the Software-Intensive Systems 
Strategic Improvement Program. 

May 15, 2006: Navy established the Software Process Improvement 
Initiative. 

Sources: Military departments and section 804 of fiscal year 2003 NDAA. 

[End of figure] 

* The Army Strategic Software Improvement Program was established in 
August 2002, which is prior to the statute's enactment. The goal of 
this program is to institutionalize improved development and business 
processes across the Army, and to lower costs, reduce cycle times, and 
enhance the performance of software-intensive systems. The program is 
overseen by the Army Strategic Software Improvement Program Senior 
Steering Group, which includes the Assistant Secretary of the Army for 
Acquisition, Logistics, and Technology Military Deputy and program 
executive officers (PEO).[Footnote 23] Shortly after the Army's program 
was created, it assessed the state of the Army's software-intensive 
system acquisitions (fiscal years 2003 and 2004), and based on these 
assessments, it developed a Strategic Software Improvement Master Plan. 
Among other things, this strategic action plan provided an Army-wide 
roadmap for adopting and institutionalizing software-related best 
practices, training, continuous improvement, and research. It is 
updated annually to establish specific program objectives and 
commitments for that year. The Army has also required its PEOs to 
develop and implement program-specific SSPI plans that are linked to 
the goals of the Strategic Software Improvement Master Plan. According 
to Army officials, about 45 percent of the PEOs have approved program-
specific plans, and implementation of the plans is regularly tracked by 
PEOs and the Army senior software acquisition manager. 

* The Air Force Software-Intensive Systems Strategic Improvement 
Program was established in January 2004, about 9 months later than the 
statute required. The goal of this program is to evaluate the 
effectiveness and progress of software management and processes for 
software-intensive system acquisitions and to form strategies and 
recommendations for improving Air Force software processes. The program 
is overseen by a working group that consists of Air Force software 
experts and engineers. Since it was established, the program issued 
guidance in September 2004 for improving software acquisition processes 
by having PEOs address 10 software process focus areas, including 
requirements development and management and risk management. To assist 
PEOs and program managers in improving software-related acquisitions, 
the program has also issued a guidebook on Weapons System Software 
Management. In contrast to the Army, the Air Force has not developed a 
departmentwide strategic action plan to guide process improvement 
implementation, as advocated by SEI. 

* The Navy established its Software Process Improvement Initiative 
program in May 2006, which is over 3 years after the date specified in 
statute for doing so. According to Navy officials, changes in senior 
leadership impeded efforts to establish the program sooner. The purpose 
of the program is to evaluate existing software-related policies and 
procedures and to implement process improvements. The program is led by 
a steering group, which is headed by the Assistant Secretary of the 
Navy for Research, Development and Acquisition Chief Engineer and 
composed of senior engineers. It also consists of five focus teams, 
each of which is led by senior engineers. The teams' respective areas 
of focus are software acquisition management, software systems 
engineering, software development techniques, business implications, 
and human resources. These teams are to, among other things, draft 
software policy documents; develop software metrics; research and 
evaluate software development methodologies; examine business, 
acquisition, and contracting strategies and practices; and refine the 
required skills and capabilities needed by government software 
acquisition and engineering professionals. Through these Software 
Process Improvement Initiative steering group and team efforts, the 
Navy has issued a range of guidance and policies aimed at improving its 
software and system development and acquisition activities, including 
guidance and policy pertaining to software metrics, organizational 
staffing, and contract language. Similar to the Air Force, however, the 
Navy has yet to develop a departmentwide strategic action plan to guide 
its process improvement activities. 

Notwithstanding that each military department now has an SSPI program, 
two departments were late in doing so, which in turn delayed their 
opportunities to exploit the benefits that the programs can produce. 
Moreover, the Air Force and Navy programs do not have the kind of 
strategic and corporate management structures that reflect recognized 
guidance, such as having a strategic plan to guide program 
implementation, thus increasing the risk that their SSPI efforts will 
not produce optimal results. 

Military Departments Have Largely Met the Remaining Statutory 
Requirements: 

The statute[Footnote 24] also required each military department to 
ensure that its software acquisition process improvement program 
include, at a minimum, (1) documented processes for software 
acquisition planning, requirements development and management, project 
management and oversight, and risk management; (2) efforts to develop 
appropriate metrics for performance measurement and continual process 
improvement; (3) a process to ensure that key program personnel have an 
appropriate level of expertise or training in software acquisition; and 
(4) a process to ensure implementation and adherence to established 
processes and requirements relating to the acquisition of software. The 
extent to which the military departments have satisfied these four 
requirements varies, but all have met most of them. Reasons that 
department officials cited for not fully meeting certain requirements 
vary. To the extent that a military department has not fully met a 
given requirement, it is not in compliance with section 804 and, as a 
result, has increased the risk of its acquisition programs experiencing 
cost overruns and schedule delays due to software-related process 
weaknesses. 

Military Departments Have Documented Processes Addressing Four Software 
Process Areas: 

The statute[Footnote 25] requires the military departments to document 
processes[Footnote 26] addressing four key software areas--software 
acquisition planning, requirements development and management, project 
management and oversight, and risk management. These four process areas 
are recognized as important in relevant guidance, such as SEI's CMMI 
model, DOD's system acquisition policy and guidance,[Footnote 27] and 
OSD's SSPI guidelines and expectations. 

Each of the military departments has issued guidance that recognizes 
the importance of software acquisition planning, requirements 
development and management, project management and oversight, and risk 
management. Moreover, while their respective guidance documents do not 
define the full range of practices associated with each process area 
that SEI's CMMI does, they do reference and are linked to DOD 
acquisition policy and guidance, which we reported in 2004 do address 
the acquisition planning and the requirements development process 
areas, and partially address the project management and oversight and 
the risk management areas.[Footnote 28] Further, since our 2004 report, 
DOD has issued supplemental guidance pertaining to risk management 
[Footnote 29] that addresses the range of key risk management practices 
advocated by SEI. In addition, the Air Force and the Navy have also 
issued risk management guidance that describe these SEI CMMI key 
practices, such as identifying program risks, analyzing the likelihood 
of each risk actually occurring and its impact, and developing and 
implementing risk mitigation strategies. (See table 1 for a summary of 
the extent to which each military department has addressed the four 
software areas.) 

Table 1: Extent to which Military Departments Have Documented Processes 
for Each Key Software Area: 

Military department: Air Force; 
Software acquisition planning: Fully; 
Requirements development and management: Fully; 
Project management and oversight: Partially; 
Risk management: Fully. 

Military department: Army; 
Software acquisition planning: Fully; 
Requirements development and management: Fully; 
Project management and oversight: Partially; 
Risk management: Fully. 

Military department: Navy; 
Software acquisition planning: Fully; 
Requirements development and management: Fully; 
Project management and oversight: Partially; 
Risk management: Fully. 

Source: GAO analysis based on DOD data. 

[End of table] 

By largely complying with section 804 and SEI guidance, the military 
departments are better positioned to effectively carry out these four 
key software process areas, thereby increasing their chances of 
successfully delivering promised program capabilities on time and 
within budget. 

Military Departments Have Established Efforts to Allow Software 
Performance Measurement and Improvement, but Related Metrics Are Not 
Being Collected and Used: 

The statute[Footnote 30] requires the military departments to establish 
efforts to develop appropriate metrics for performance measurement and 
continual process improvement. Relevant guidance, such as SEI's IDEALSM 
model and OSD SSPI guidelines and expectations, also recognizes the 
importance of adopting a corporate approach to such measurement. For 
example, the SEI model states that process improvement activities 
should have metrics for monitoring progress against organizational 
goals, and the OSD guidelines state that appropriate measures should be 
collected and used to determine the success or failure of SSPI efforts. 

The military departments have undertaken efforts to develop SSPI- 
related performance metrics, as required by the statute. However, they 
are not currently using the metrics for continuous process improvement, 
as also required by the statute. Each of the military department's 
efforts is described below. 

* According to Army officials, the Army began a departmentwide effort 
to determine and implement metrics for software acquisition process 
improvement in 2004. However, they described the effort as not 
successful because the data that were collected for these metrics were 
not sufficiently consistent to permit meaningful departmentwide 
analysis. As a result, the Army ended these efforts, and instead 
provided its commands discretion in how each developed and collected 
process improvement metrics. 

* The Air Force began efforts to develop and collect software core 
metrics in 2004. However, similar to the Army, it left development and 
collection of process improvement metrics to the discretion of each 
PEO, rather than adopting a more corporate approach. According to Air 
Force guidance, devolving this to the PEOs was due to the fact that the 
section 804 requirement was considered subjective and not addressed in 
detail in OSD's guidance. As a result, the metrics that were 
established and used have varied across the program offices. 

* The Navy began an effort to develop and collect process improvement 
measures in 2008. Specifically, it issued a policy in July 2008 that 
directs all programs that have a software component to define and 
collect a set of core metrics covering, for example, software quality, 
and to report these metrics at program milestone reviews. According to 
Navy Software Process Improvement Initiative officials, programs have 
begun to report these metrics. They added that the data reported will 
be used to, among other things, assess the effectiveness of the Navy's 
process improvement program. 

While each of the military departments has satisfied the statute's 
requirement to establish efforts to permit software-related measurement 
and improvement, none of these efforts has progressed to the point that 
it is actually being used to understand how well the departments' 
respective SSPI programs are achieving expected outcomes and having an 
impact. Further, the Army and Air Force efforts are diffused across its 
commands and PEOs, respectively, and thus do not reflect the kind of 
corporate approach to process improvement measurement advocated by 
relevant guidance. 

Until each military department has established appropriate metrics and 
used them to understand the organizational impact of their respective 
SSPI programs, each will be challenged in its ability to better focus 
its SSPI efforts and thereby maximize their potential value. 

Military Departments Have Established Processes for Ensuring Program 
Personnel Are Trained and Experienced: 

The statute[Footnote 31] requires each military department to establish 
a process to ensure that key program personnel have an appropriate 
level of expertise or training in software acquisition. Relevant 
guidance, such as SEI's CMMI model, DOD's acquisition policy and 
guidance, and OSD's SSPI memo, also address the importance of ensuring 
that key acquisition personnel are trained. 

All three military departments require that each program manager have 
processes aimed at ensuring that program personnel have a certain level 
of experience and/or training in relevant software disciplines. For 
example, the Assistant Secretary of the Navy for Research, Development, 
and Acquisition issued a memorandum in September 2008 that requires 
that the Navy's acquisition workforce be trained in Defense Acquisition 
University-defined software-related core competencies, and that its 
program managers ensure that program personnel meet certification 
requirements. Similarly, the Army and Air Force require their PEOs to 
ensure that acquisition personnel are certified in the relevant 
acquisition career field and at the required level. 

Moreover, each department has process steps aimed at verifying that 
these personnel have met requisite training or experience requirements. 
For example, each has established guidance for system acquisition 
program milestone reviews that requires, among other things, 
determining whether personnel with the needed skills, experience, and 
certifications are available.[Footnote 32] Furthermore, each captures 
and maintains certification information on all its personnel, which are 
used for workforce planning and assignment. 

By having processes for ensuring that key personnel have appropriate 
levels of expertise or training, the military departments have not only 
satisfied the statute, but have also increased the chances that their 
acquisition programs can be successful. 

Military Departments Have Processes to Ensure that Acquisition Programs 
Implement Defined Software Processes and Requirements, but They Do Not 
Incorporate Key SEI Verification Steps: 

The statute requires[Footnote 33] each military department to establish 
a process to ensure that its acquisition programs implement and adhere 
to defined software processes and requirements. Relevant guidance, such 
as SEI's CMMI model and OSD's SSPI guidance, also stresses the 
importance of evaluating both individual program and institutional 
adherence to defined software processes and requirements. 

Each of the military departments has processes for ensuring that its 
programs follow defined software and system development and acquisition 
processes, as required by the statute. Specifically, each has issued 
guidance that provides for, among other things, reporting 
implementation progress and problems in such key process areas as 
acquisition planning, requirements development and management, program 
management and oversight, and risk management. Among other things, this 
reporting includes the status of a program's technology development 
strategy, key requirements documents, cost estimates, and risk 
mitigation activities. 

However, the department's guidance does not include the range of 
verification steps that are provided for in SEI's CMMI model, which is 
referenced in OSD's SSPI guidance. Specifically, the CMMI model 
provides for conducting detailed and documented evaluations of the 
extent to which key software processes are being followed. Moreover, 
these evaluations are typically performed by an organization external 
to the program, and include analysis of the range of key practices that 
comprise a given process area. Officials with each of the military 
departments told us they do not conduct such evaluation activities to 
verify that programs conform to established software processes and 
requirements, and do not believe that such verification activities are 
appropriate. According to these officials, the combination of requiring 
programs to adhere to established software-related processes and 
requirements and the attention to this adherence at program reviews is 
sufficient to reasonably ensure that programs adhere. 

Without the range of process adherence verification advocated by SEI, 
the military departments lack a level of assurance that SEI's research 
has shown is needed. As discussed in the next section, our reviews of 
software-intensive DOD acquisition programs show that many fail to 
fully implement and thus adhere to key software-related processes. This 
means that the level of process adherence assurance that occurs under 
the military departments' current program review guidance is not always 
sufficient, and therefore may allow for process weaknesses that in turn 
increase the chances of program cost, schedule, and performance 
shortfalls. 

DOD Does Not Know Impact of SSPI Efforts but Studies of Large-Scale 
Acquisition Programs Show that Performance Shortfalls and Process 
Weaknesses Continue: 

As previously noted, neither OSD nor the military departments are using 
performance measures to understand how well the department's collective 
SSPI efforts are meeting goals and producing expected outcomes, and 
thus DOD officials told us they do not know the corporate impact of 
these efforts. This void in SSPI-related measurement is not consistent 
with either the statute, which requires that each military department 
develop SSPI performance measures, or SEI's IDEALSM and CMMI models, 
which state that measurements are key to effectively monitoring and 
evaluating the impact of process improvement. OSD and military 
department officials cited various reasons for not measuring the impact 
of their SSPI efforts, including difficulties in obtaining accurate 
information from their respective programs and the subjective nature of 
performance measurements that have been defined. By not knowing the 
impacts accruing from these efforts, the department is not positioned 
to maximize the potential of its SSPI efforts. 

In the absence of DOD knowledge of the impact of its SSPI efforts, 
studies by us and others of the department's major system acquisition 
programs suggest that these programs have yet to fully realize the 
benefits from the kind of SSPI required by section 804 and advocated by 
SEI. In particular, we recently reported that as of March 2009, DOD's 
large-scale, software-intensive system acquisitions continue to fall 
short of cost, schedule, and performance expectations. Specifically, in 
2008, DOD's portfolio of major defense acquisition programs experienced 
average delays in delivering initial operational capabilities of 22 
months, which is a 4-month increase in delays compared to 5 years ago. 
Moreover, these programs collectively overran their cost estimates by 
about $296 billion,[Footnote 34] which is greater than comparable 
overruns experienced 5 years ago. Relatedly, we continue to report on 
weaknesses in the extent to which a range of programs has implemented 
system and software acquisition and development processes, including 
weaknesses in the four key process areas that the act specifically 
referenced as SSPI focus areas---acquisition planning, requirements 
development and management, program management and oversight practices, 
and risk management. These process implementation weaknesses are 
described below. 

Software Acquisition Planning: 

One of the purposes of acquisition planning is to establish and 
maintain the plans that govern the execution of acquisition programs. 
Among other things, it includes defining work products and tasks as 
well as estimating needed resources and schedules, negotiating 
contractor and stakeholder commitments, and identifying and analyzing 
risks associated with delivering work products and executing tasks. 
Acquisition planning is important because it results in plans and 
estimates that provide the basis for executing programs and measuring 
performance. For example, estimates of program cost and effort, which 
are generally based on results of analysis using models or historical 
data applied to size, activities, and other planning parameters, are 
used to develop budgets and control costs and schedules during program 
execution. 

Our work continues to show that the department is challenged in its 
efforts to effectively perform the key practices associated with the 
acquisition planning key process area. For example, we recently 
reported that the Navy Enterprise Resource Planning (Navy ERP) program 
did not develop a life cycle cost estimate in accordance with relevant 
guidance.[Footnote 35] More specifically, we reported that Navy ERP's 
cost estimate was not grounded in a historical record of comparable 
data from similar programs and was not based on a reliable schedule 
baseline, both of which are necessary to having a cost estimate that 
can be considered credible and accurate. We concluded that these 
acquisition planning limitations could result in actual program costs 
continuing to exceed the estimates, and made recommendations to address 
each limitation. DOD largely agreed with our recommendations. 

Similarly, we recently reported that the Global Combat Support System- 
Marine Corps (GCSS-MC) also had not developed a life cycle cost 
estimate that was grounded in a historical record of comparable data 
from similar programs, and it did not account for significant risks 
associated with the program's aggressive schedule, both of which 
limited the estimate's credibility and accuracy.[Footnote 36] As a 
result, we concluded that the Marine Corps did not have an adequate 
basis for informed investment decision making, and that actual program 
costs would likely not be consistent with estimates. Accordingly, we 
made recommendations aimed at addressing each of these weaknesses. DOD 
agreed with our recommendations. 

Requirements Development and Management: 

Well-defined and managed requirements are recognized by DOD directives 
and guidance and other relevant guidance as essential.[Footnote 37] 
Effective requirements development and management includes, among other 
things, (1) developing detailed system requirements; (2) establishing 
policies and plans for managing changes to requirements, including 
defining roles and responsibilities, and identifying how the integrity 
of a baseline set of requirements will be maintained; and (3) 
maintaining bidirectional requirements traceability, meaning that 
system-level requirements can be traced both backward to higher-level 
business or operational requirements, and forward to system design 
specifications and test plans. Effective requirements development and 
management is important because requirements provide the cornerstone of 
any new system development or acquisition program. 

Our work continues to show that the department is challenged in its 
efforts to effectively perform key practices associated with the 
requirements development and management key process area. For example, 
we recently reported that the Navy's program for creating cashless 
environment on ships, known as Navy Cash, had not defined and 
implemented key practices for developing and managing requirements, and 
thus was without basic requirements documentation needed to inform 
program cost and schedule estimates and accomplish work associated with 
delivery of needed system capabilities.[Footnote 38] In particular, the 
program had not defined how system requirements were to be managed and 
who would be responsible for managing them. Instead, the program had 
adopted a reactive approach to developing and managing requirements 
that consisted of considering requests for requirements changes 
proposed as part of the program's change control process. As a result, 
we concluded that Navy Cash could not develop and measure performance 
against meaningful cost, schedule, and capability baselines, and could 
not reliably ensure that the system would meet expectations or that 
those responsible for it could be held accountable for results. 
Accordingly, we made recommendations to address these limitations, 
which DOD agreed to implement. 

Similarly, we recently reported that the Army Future Combat System 
program had not adequately developed and managed requirements.[Footnote 
39] Specifically, during the system's initial stages of development, 
the program did not establish firm requirements and preliminary designs 
to meet requirements. Consequently, after more than 5 years, the Army 
was still seeking to stabilize system designs at a time when the 
program was already past the midpoint of the development phase, which 
is the point when a program would normally be demonstrating a stable 
design capable of meeting performance requirements. We concluded that 
the program would likely need to relax system design requirements in 
order to stay within schedule commitments. Accordingly, we made 
recommendations to address each limitation. DOD agreed with the 
recommendations. 

Recent DOD studies have also identified deficiencies in requirements 
development and management practices that have resulted in cost 
overruns and schedule delays. For example, a September 2008 DOD 
briefing stated the cost of these system acquisitions has grown by 33 
percent over the past 10 years due to, among other things, insufficient 
requirements analysis. 

Program Management and Oversight: 

The purpose of program management and oversight is to understand a 
program's progress against commitments so that any needed corrective 
actions can be taken. Among other things, program management and 
oversight consists of comparing the actual cost, schedule, and quality 
of work products and tasks to plans and estimates; determining whether 
significant deviations exist; and developing corrective actions, if 
necessary, to address the deviations. One accepted technique for 
managing and overseeing program performance is earned value management 
(EVM), which is a means for determining and disclosing actual work 
completed in comparison with cost and schedule estimates. 

Our work continues to show that the department is challenged in 
performing key practices related to the program management and 
oversight key process area. For example, we recently reported that the 
Navy ERP program had not performed basic EVM activities, such as 
conducting integrated baseline reviews of its cost and schedule 
estimates,[Footnote 40] resulting in actual program costs and schedules 
that did not track to estimates. Similarly, we recently reported that 
GCSS-MC's was not effectively performing EVM because its schedule 
baseline was not reflective of important practices, such as conducting 
a schedule risk assessment or allocating schedule reserve.[Footnote 41] 
As a result, we concluded that actual program cost and schedule figures 
will not be consistent with estimates, and will not provide an adequate 
basis for informed investment decision making. Accordingly, we made 
recommendations to address each limitation, which DOD agreed to 
implement. 

DOD has also found that program management and oversight process 
deficiencies were impacting major acquisitions. For example, a 
September 2008 DOD briefing[Footnote 42] on system acquisition software 
problems states that over the past 11 years, the department incurred 
system estimation changes totaling approximately $201 billion, systems 
engineering changes totaling approximately $147 billion, and systems 
schedule changes totaling approximately $70 billion. To understand the 
cause of these shortfalls, DOD performed related analysis that 
identified problems with management and oversight of programs' 
integrated master plans, integrated master schedules, EVM data, and 
risk management. 

Risk Management: 

DOD and relevant guidance[Footnote 43] recognize the importance of 
performing effective program risk management. Among other things, 
effective risk management includes (1) establishing and implementing a 
written plan and defined process for risk identification, analysis, and 
mitigation; (2) assigning responsibility for managing risks to key 
stakeholders; (3) encouraging program-wide participation in risk 
management; and (4) examining the status of identified risks during 
program milestone reviews. Risk management is important because it 
focuses on avoiding problems and their associated cost and schedule 
impacts before they occur. 

Our work continues to show that the department is challenged in its 
efforts to effectively manage risk. For example, we recently reported 
that the Navy Cash program had not developed plans, processes, or 
procedures to identify, mitigate, and disclose risks, nor had it 
assigned risk management roles and responsibilities to key 
stakeholders.[Footnote 44] As a result, the program was not proactively 
attempting to avoid the occurrence of cost, schedule, and performance 
problems, but rather was reacting to the consequences of actual 
problems. To address these limitations, we made a number of 
recommendations, which DOD agreed to implement. 

Similarly, we recently reported that while the Navy ERP program had 
defined and established a process for proactively avoiding problems, it 
had not effectively implemented risk mitigation strategies for some 
significant risks, such as those associated with data conversion and 
organizational change management.[Footnote 45] As a result, operational 
testing of the system at the first user organization revealed 
significant problems that required additional resources and time to 
correct. In addition, we reported that not all known risks had been 
included in the risk inventory, such as not having adequately 
implemented program-level EVM. We concluded that not having effectively 
addressed such risks had not only contributed to existing schedule 
delays but would also likely contribute to future cost and schedule 
shortfalls. To address these limitations, we made a number of 
recommendations, which DOD agreed to implement. 

Conclusions: 

An SSPI program, if properly defined and implemented, can have a 
positive impact on the quality and cost of software-intensive system 
acquisition and development programs. In the case of DOD, such impacts 
could be significant given the enormous size and mission significance 
of its many programs. Congress has recognized this tremendous potential 
by directing OSD and the military departments and defense agencies to 
take a range of statutorily defined SSPI-related steps. To their 
credit, OSD and the military departments have satisfied a number of 
these statutory requirements, but they have not satisfied them all. 
Moreover, they also have not fully satisfied key aspects of relevant 
guidance associated with well-managed SSPI programs to include 
measuring the impact of their collective efforts. As a result, DOD does 
not know whether it is meeting organizational SSPI goals and achieving 
desired institutional outcomes, and thus whether program changes are 
warranted. Given that studies by us and others continue to identify 
weaknesses in DOD's implementation of key software and systems 
development and acquisition process areas, the department has yet to 
realize the full potential of an effectively and efficiently managed 
corporate approach to SSPI. 

While the reasons cited by OSD organizations and the military 
departments for not fully satisfying SSPI statutory requirements and 
relevant guidance vary, they can be traced to a lack of a DOD-wide 
strategic approach to SSPI that includes strong central leadership and 
strategic planning. Such an approach is reflected in the statute and 
relevant guidance, and includes departmentally assigned and endorsed 
responsibilities and authorities, clearly defined strategic plans and 
performance measures, and rigorously enforced accountability 
mechanisms. Until DOD adopts such an approach, it will continue to fall 
short of statutory requirements and relevant guidance, and will thus 
not be positioned to reap the potential benefits promised by properly 
defined and implemented SSPI efforts. 

Recommendations for Executive Action: 

To strengthen DOD's management of its SSPI efforts, we recommend that 
the Secretary of Defense direct the ASD(NII)/CIO and the USD(AT&L), in 
concert with the military departments and defense agencies, to: 

* jointly develop a DOD-wide strategic plan, and supporting 
organizational component plans, for ensuring that all of the 
requirements in section 804 of the 2003 NDAA, as well as relevant SEI 
guidance, are fully implemented; and: 

* jointly and periodically report to the congressional defense 
committees on their progress in implementing the plan and the impacts 
of doing so. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Acting 
Director of Systems Engineering and reprinted in appendix II, the 
department described our report as important and insightful, adding 
that it provides useful feedback on DOD's software acquisition 
practices that may require improvement. In addition, it committed to 
promoting and applying process improvement across the department in 
support of the fiscal year 2003 NDAA section 804 process improvement 
provisions and DOD's system engineering efforts. To this end, it stated 
that it will consider our findings and that it partially agreed with 
our two recommendations. 

In partially agreeing with our first recommendation, DOD stated that it 
agrees that it needs to formalize the DOD-wide process improvement 
activities that its Software Working Group has under way, and to do so, 
it will do an analysis comparing these activities to the section 804 
requirements, and it will develop objectives and a plan to respond to 
our recommendations. However, DOD commented that it does not believe 
that a DOD-wide strategic plan is needed to meet the section 804 
requirements. Further, it said that the SEI guidance that our report 
references should not be seen as a panacea, noting that other 
techniques exist and are used within the department and that section 
804 does not require any specific techniques or methodologies. In 
response, we support DOD's planned actions and agree with most of its 
comments, as they are consistent with the intent of our recommendation. 
Specifically, we agree that section 804 does not require a specific 
technique or methodology. Further, we agree that the SEI guidance that 
our recommendation references is neither a panacea nor the only 
available technique or methodology. Accordingly, our recommendation 
specifically provides for using relevant SEI guidance, and it does not 
preclude using other available guidance. Further, while we agree that 
section 804 does not require a strategic plan, we would note that our 
report identifies the root cause of DOD's currently decentralized and 
inconsistently pursued range of SSPI efforts as being a lack of a 
corporate and strategically driven approach to process improvement, 
that includes clearly assigned responsibility and accountability for 
achieving strategic outcomes and results across the heterogeneous, 
component-based process improvement activities. As a result, the full 
intent of our recommendation is aimed at having the department adopt a 
more strategic and coordinated approach to SSPI, which is consistent 
with section 804. While the department's planned actions to address our 
recommendation would be part of such an approach, they alone are not 
sufficient. Therefore, we would encourage DOD to fully implement our 
recommendation. 

In partially agreeing with our second recommendation, DOD stated that 
it would periodically report to its congressional committees on its 
progress in implementing the section 804 requirements, as we 
recommended, and would do so through a new report that DOD is to 
provide under a recently enacted public law. However, it stated that 
this reporting will be limited to its plans for addressing our first 
recommendation. Moreover, it added that its efforts to respond to both 
of our recommendations will not extend beyond software acquisition 
process improvement, and thus will not include system-related process 
improvement, which it stated our report includes. In this regard, the 
department stated that section 804 only calls for software acquisition 
process improvement. We support DOD's use of this existing reporting 
mechanism, but again encourage the department to implement the full 
scope of our first recommendation for the reasons discussed above, and 
to also report in line with the full scope of our recommendation. 
Further, we encourage the department to not limit its process 
improvement activities to software acquisition. As our report states, 
section 804 is consistent with the SSPI recommendations that we made to 
DOD in 2001,[Footnote 46] and these recommendations extend to software 
and system development and acquisition. Further, section 804 
specifically uses the terms software acquisition and development, and 
it specifically refers to defense programs with a substantial software 
component, which are in fact defense systems that are intensively 
dependent on software. 

DOD also provided technical comments on a draft of this report that we 
have incorporated throughout the report, as appropriate. 

We are sending copies of this report to interested congressional 
committees. We will also send copies to the Secretary of Defense and 
the Director of the Office of Management and Budget. Copies of this 
report will be available at no charge on the GAO Web site at 
[hyperlink, http://www.gao.gov]. 

Should you or your offices have any questions on matters discussed in 
this report, please contact me at (202) 512-3439 or at hiter@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. Key contributors 
to this report are listed in appendix III. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Section 804 of the Bob Stump National Defense Authorization Act for 
fiscal year 2003[Footnote 47] required the Department of Defense (DOD) 
to establish process improvement programs for its military departments 
and defense agencies that manage major acquisition programs. Our 
objectives were to assess: (1) the extent to which DOD has implemented 
the process improvement provisions of the act, and (2) the impact of 
DOD's process improvement efforts. The scope of work focused on the 
efforts and activities of the Office of the Assistant Secretary of 
Defense for Networks and Information Integration/Chief Information 
Officer (OASD(NII)/CIO) and the Office of the Under Secretary of 
Defense for Acquisition, Technology, and Logistics (OUSD(AT&L)), 
because both have responsibility for, among other things, software and 
system process improvement, oversight of major acquisition programs, 
and establishment of policies and procedures related to software and 
system acquisition.[Footnote 48] Our scope also focused on the 
Departments of the Army, Air Force, and Navy, and did not include any 
defense agencies, because as of October 2008 about 99 percent of DOD's 
major programs were being managed by either the military departments or 
DOD (only one major acquisition program was being managed by a defense 
agency). 

To address the first objective, we reviewed relevant ASD(NII)/CIO and 
USD(AT&L) and military department process improvement and system 
acquisition policies, guidance, plans, oversight controls, and 
performance measures. We also interviewed responsible officials within 
each of these organizations concerning their respective efforts to 
address relevant aspects of the act. We then compared their respective 
efforts to the provisions in the act as well as other relevant 
guidance, such as the Software Engineering Institute's IDEALSM and 
Capability Maturity Model Integration models. Where we found 
deviations, we interviewed responsible officials to determine reasons 
for the deviations. 

To address the second objective, we reviewed available documentation 
and interviewed officials from OASD(NII)/CIO, OUSD(AT&L), and the 
military departments concerning efforts to determine the impact of 
their respective and collective process improvement efforts. In 
addition, we reviewed recently issued GAO reports and DOD assessments 
addressing software-related process and practice weaknesses and program 
cost and schedule outcome changes. 

We conducted this performance audit at DOD and military department 
offices in Arlington, Virginia, from December 2008 through September 
2009 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Director Of Defense Research And Engineering: 
3040 Defense Pentagon: 
Washington, DC 20301-3040: 

August 26, 2009: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, N.W. 
Washington, DC 20548: 

Dear Mr. Hite: 

This is the Department of Defense (DoD) response to the GAO Draft 
Report, GAO-09-888, "Information Technology: DoD Needs to Strengthen 
Management of its Statutorily Mandated Software and System Process 
Improvement Efforts," dated July 22, 2009 (GAO Code 310664). Detailed 
comments on the report recommendations are enclosed.
The Department thanks the GAO for this insightful draft report and the 
recommendations, which we can continue to fold into our systems 
engineering revitalization efforts. We believe it is an important 
report, as it gives Congress more visibility into our continuing 
efforts to improve systems engineering across the defense acquisition 
community. The draft report provides useful feedback about software 
acquisition areas that may still require improvement, and we will take 
the findings from this report in consideration as we continue to 
monitor our software acquisition practices. 

It is important to note that while the Software Engineering Institute's 
specific process improvement techniques highlighted may improve insight 
and impose better controls over software development; they should not 
be seen as a panacea. Significant challenges associated with 
acquisition and sustainment of complex defense systems are not always 
well served by generalizing software development and process 
improvement across programs; it is important to take into account a 
program's acquisition characteristics (size, complexity, stage of 
development, technology precedent, team makeup, length of service). The 
Department has conducted a number of root-cause analyses to show that 
many problems attributed to software were not caused by software 
processes, but to other factors outside the realm of Section 804. We 
will continue to address software issues in our program support 
reviews, and as part of our acquisition oversight activities. 

In addition, DoD's organization-wide Continuous Process Improvement 
program, codified in DoDI 50I0.43, Implementation and Management of the 
DoD-Wide Continuous Process Improvement/Lean Six Sigma (CPI/LSS) 
Program, cites process improvement techniques such as lean six sigma, 
theory of constraints, and business process reengineering. Although 
many such improvement initiatives have been implemented across DoD, we 
have learned that imposing a specific process improvement methodology 
across the board does not well serve the needs of the wide diversity of 
complex acquisition programs. Further, the Section 804 legislation did 
not require any specific software acquisition process improvement 
techniques or methodologies, as the report implies. 

We will continue to promote and apply process improvement across our 
DoD acquisition programs as warranted and to improve software 
acquisition in support of the 2003 NDAA Section 804 and our systems 
engineering efforts. 

Sincerely, 

Signed by: 

Terry Jaggers: 
Acting Director: 
Systems Engineering: 

Enclosure: As stated: 

[End of letter] 

GAO Draft Report Dated July 22, 2009: 
GAO-09-888 (GAO Code 310664): 

"Information Technology: DoD Needs to Strengthen Management of its 
Statutorily Mandated Software and System Process Improvement Efforts" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommends that the Secretary of Defense 
direct the Assistant Secretary Defense for Networks and Information 
Integration/Chief Information Officer (ASD(NII)/CIO) and the Under 
Secretary of Defense for Acquisition, Technology, and Logistics 
(USD(AT&L)), in concert with the military departments and defense 
agencies, to jointly develop a DoD-wide strategic plan, and supporting 
organizational component plans, for ensuring that all of the 
requirements in Section 804 of the Bob Stump National Defense 
Authorization Act for FY 2003, as well as relevant Software Engineering 
Institute guidance, are fully implemented. 

DOD Response: Partial Concur. The DoD agrees with continuing to track 
the progress of the implementation of all the requirements of Section 
804 and plans to do so by formalizing many of the activities ongoing in 
the DoD-wide Software Working Group (SWWG). The DoD will do a gap 
analysis on our implementation of Section 804 to develop objectives and 
a plan of action to respond to the recommendations. The Department does 
not believe that a DOD-wide strategic plan is warranted to meet the 
Section 804 requirements. 

Software Engineering Institute (SEI) guidance was not a requirement of 
the Section 804 legislation. While the DoD has been a sponsor of many 
of the SEI products, such as the CMMI Product Suite, and continues to 
support its prudent use, the results of its implementation have not 
always been as we had hoped when we began its development. In 2007 the 
Defense Contract Management Agency (DCMA) conducted a study relating 
Earned Value Management (EVM) data on program performance and the 
reported CMMI maturity level of the respective programs that reported 
less than positive results. SEI guidance will simply continue to 
provide guidance for our efforts, not govern them. 

DoD Proposed Revision of Recommendation I: The GAO recommends that the 
Secretary of Defense direct the Under Secretary of Defense for 
Acquisition, Technology, and Logistics (USD(AT&L)), and the Assistant 
Secretary Defense for Networks and Information Integration/Chief 
Information Officer (ASD(NII)/CIO), in concert with the military 
departments and defense agencies, to jointly conduct a gap analysis to 
prepare objectives and a plan of action focused on implementation of 
the specific requirements of Section 804 of the FY 2003 National 
Defense Authorization Act. 

Recommendation 2: The GAO recommends that the Secretary of Defense 
direct the ASD(NII)/CIO and the USD(AT&L), in concert with the military 
departments and defense agencies, to jointly and periodically report to 
the congressional defense committees on its progress in implementing 
the plan and the impacts of doing so. 

DOD Response: Partial Concur. DoD will address this reporting in 
accordance with our recommended revision of Recommendation I and the 
Section 804 requirements. Such reporting will be through the new Joint 
Developmental Test/Systems Engineering (DT/SE) Capabilities Report that 
DoD is to provide to Congress on a periodic basis per the Weapon 
Systems Acquisition Reform Act of 2009 (Public Law 111-23). 

Section 804 calls for software acquisition process improvement, which 
is what we will be judging in our reporting. The GAO report frequently 
addresses software and systems process improvement (SSPI). No inference 
should he made that DoD will perform oversight of SE capability the 
same way we oversee improvement of the software acquisition processes 
for these reporting requirements. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439, or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, Tonia Johnson (Assistant 
Director), Sher'rie Bacon, Mathew Bader, Elena Epps, Rebecca Eyler, 
Franklin Jackson, Freda Paintsil, and Madhav Panwar made key 
contributions to this report. 

[End of section] 

Footnotes: 

[1] As used in this report, SSPI refers to improvements in the 
processes associated with developing, acquiring, and engineering 
software and systems. 

[2] Pub. L. No. 107-314, § 804, 116 Stat 2458, 2604-2605 (Dec. 2, 
2002). 

[3] DOD major defense acquisition programs are those estimated to 
require total research, development, test, and evaluation expenditures 
of more than $365 million or procurement expenditures of more than 
$2.19 billion in fiscal year 2000 constant dollars. They also include 
programs otherwise designated by the department as major defense 
acquisition programs. 

[4] GAO, High-Risk Series: An Update [hyperlink, 
http://www.gao.gov/products/GAO-09-271] (Washington, D.C.: January 
2009). 

[5] [hyperlink, http://www.gao.gov/products/GAO-09-271]. 

[6] SEI is a federally funded research and development center 
established at Carnegie Mellon University to address software 
engineering practices. 

[7] SEI, Technical Report CMU/SEI-2006-TR-004, August 2006. 

[8] GAO, DOD Information Technology: Software and Systems Process 
Improvement Programs Vary in Use of Best Practices, [hyperlink, 
http://www.gao.gov/products/GAO-01-116] (Washington, D.C.: Mar. 30, 
2001). 

[9] IDEALSM is a service mark of Carnegie Mellon University and stands 
for initiating, diagnosing, establishing, acting, and learning. IDEALSM 
is the SEI methodology for organizational software process improvement. 

[10] This position has since been renamed as the Assistant Secretary of 
Defense for Networks and Information Integration/Chief Information 
Officer (ASD(NII)/CIO). 

[11] Pub. L. No. 107-314 (Dec. 2, 2002). 

[12] According to DOD's May 2005 Technology Readiness Assessment 
Deskbook, technology should be "mature" before system development 
begins. Normally, for technology to be considered mature, it must have 
been tested in a relevant or operational environment, and found to have 
performed adequately for the intended application. 

[13] Pub. L. No. 107-314, § 804 (Dec. 2, 2002). 

[14] Pub. L. No. 107-314, § 804(c)(1) (Dec. 2, 2002). 

[15] Software Engineering Institute, IDEALSM: A User's Guide for 
Software Process Improvement, CMU/SEI-96-HB-001 (Pittsburgh, Pa., 
February 1996). 

[16] Pub. L. No. 107-314, § 804(c)(1) (Dec. 2, 2002). 

[17] Program Support Reviews are the department's milestone reviews for 
each system acquisition. According to DOD's Instruction 5000.02, 
Operation of the Defense Acquisition System, OSD performs these reviews 
only on those acquisitions that meet certain dollar thresholds or that 
are otherwise designated as special interest. 

[18] Pub. L. No. 107-314, § 804(c)(2) (Dec. 2, 2002). 

[19] The CMMI is a model used to examine an organization's software 
engineering process maturity. It combines earlier SEI models for 
software development and acquisition into one model for enterprise-wide 
process improvement. 

[20] Lean Six Sigma is a systematic, rigorous methodology that uses 
metrics and analysis to drive continuous improvement of an 
organization's processes. 

[21] Pub. L. No. 107-314 was enacted on December 2, 2002. Section 
804(a) required, among other things, that the military departments 
establish SSPI programs within 120 days, or by March 31, 2003. 

[22] CMU/SEI-96-HB-001. 

[23] PEOs are responsible for overseeing a related group of major 
system acquisitions. 

[24] Pub. L. No. 107-314, § 804(b) (Dec. 2, 2002). 

[25] Pub. L. No. 107-314, § 804(b)(1) (Dec. 2, 2002). 

[26] As defined by SEI, a process area represents a cluster of related 
practices that, when implemented correctly, satisfies a set of goals 
considered important for making improvement in that area. For example, 
the requirements development and management process area includes 
clusters of practices for the elicitation, refinement, documentation, 
and management of requirements for a system or software product. 

[27] DOD Directive 5000.1, The Defense Acquisition System, DOD 
Instruction 5000.02, Operation of the Defense Acquisition System. 

[28] GAO, Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls, 
[hyperlink, http://www.gao.gov/products/GAO-04-722] (Washington, D.C.: 
July 30, 2004). 

[29] DOD, Risk Management Guide for DOD Acquisition, Sixth Edition 
(Version 1.0), August 2006. 

[30] Pub. L. No. 107-314, § 804(b)(2) (Dec. 2, 2002). 

[31] Pub. L. No. 107-314, § 804(b)(3) (Dec. 2, 2002). 

[32] This guidance employs the Probability of Program Success model, 
which is intended to determine the ability of a program to deliver a 
specified capability, within approved cost and schedule limits, that 
meets the performance levels mandated by the warfighter. 

[33] Pub. L. No. 107-314, § 804(b)(4) (Dec. 2, 2002). 

[34] GAO, Defense Acquisitions: Assessments of Selected Weapon 
Programs, [hyperlink, http://www.gao.gov/products/GAO-09-326SP] 
(Washington, D.C.: Mar. 30, 2009). 

[35] GAO, DOD Business Systems Modernization: Important Management 
Controls Being Implemented on Major Navy Program, but Improvements 
Needed in Key Areas, [hyperlink, 
http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8, 
2008). 

[36] GAO, DOD Business Systems Modernization: Key Marine Corps System 
Acquisition Needs to Be Better Justified, Defined, and Managed, 
[hyperlink, http://www.gao.gov/products/GAO-08-822] (Washington, D.C.: 
July 28, 2008). 

[37] DOD, Defense Acquisition Guidebook, Version 1.0 (Oct. 17, 2004). 
Software Engineering Institute, Software Acquisition Capability 
Maturity Model® (SA-CMM®) version 1.03, CMU/SEI-2002-TR-010 
(Pittsburgh, Pa., March 2002). 

[38] GAO, DOD Business Systems Modernization: Planned Investment in 
Navy Program to Create Cashless Shipboard Environment Needs to Be 
Justified and Better Managed, [hyperlink, 
http://www.gao.gov/products/GAO-08-922] (Washington, D.C.: Sept. 8, 
2008). 

[39] GAO, Defense Acquisitions: Decisions Needed to Shape Army's Combat 
Systems for the Future, [hyperlink, 
http://www.gao.gov/products/GAO-09-288] (Washington, D.C.: Mar. 12, 
2009). 

[40] [hyperlink, http://www.gao.gov/products/GAO-08-896]. 

[41] [hyperlink, http://www.gao.gov/products/GAO-08-822]. 

[42] DOD, Software Problems Found on DOD Acquisition Programs, 
September 2008. 

[43] DOD, Risk Management Guide for DOD Acquisition, 6th Edition, 
Version 1.0, and Software Engineering Institute, CMMI for Acquisition, 
Version 1.2 CMU/SEI-2007-TR-017 (Pittsburgh, Pa., November 2007). 

[44] [hyperlink, http://www.gao.gov/products/GAO-08-922]. 

[45] [hyperlink, http://www.gao.gov/products/GAO-08-896]. 

[46] [hyperlink, http://www.gao.gov/products/GAO-01-116]. 

[47] Pub. L. No. 107-314, § 804 (Dec. 2, 2002). 

[48] The ASD(NII/CIO) was formerly known as the Assistant Secretary of 
Defense for Command, Control, Communications and Intelligence 
(ASD(C3I)). The act specifically cites ASD(C3I). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: