DOD Major Automated Information Systems: Improvements Can Be Made in Applying Leading Practices for Managing Risk and Testing

What GAO Found

Most of the 18 selected Department of Defense (DOD) major automated information system (MAIS) programs that GAO reviewed had experienced changes in their planned cost and schedule estimates and half of the programs had met their technical performance targets. Specifically, 16 programs experienced changes in their cost estimates ranging from a 39 percent decrease ($1.47 billion) to a 469 percent increase ($1.63 billion). The average cost increase was $457.2 million among the 11 programs reporting an increase. Fourteen programs experienced schedule delays, which ranged from 2 months to over 13 years. Finally, half of the MAIS programs fully met all of their technical performance targets. Of the remaining nine programs, 4 four had partially met their target because each was still conducting tests. The other five programs were in the early stages of system development and had not begun testing.

In addition, for the five MAIS programs GAO selected for in-depth review, all had either fully or partially applied leading practices for managing requirements, risks, and for conducting systems testing and integration.

• Managing requirements. Three of the five programs had fully implemented the practices for managing requirements, while the other two had partially implemented some practices. Leading practices in this area include establishing requirements and ensuring traceability between requirements and work products.

• Managing risks. Three of the five programs had fully implemented the risk management practices, while two had partially implemented some practices. An effective risk management process identifies potential problems before they occur. For example, one Army program did not have standard operating procedures for defining thresholds or bounds to manage risk. Unless such procedures are defined, the program will not have the tools needed to define risk management activities, including whether and how certain risks are prioritized. Further, programs should include practices to identify potential problems so that risk-handling activities may be planned and invoked across the project to mitigate the potential for adverse impacts. However, one Air Force program did not develop an overall risk mitigation plan to guide the implementation of individual risk mitigation activities. Without an overall risk plan to guide individual development efforts, those efforts cannot be managed cohesively.

• Testing and integration. Four of the five programs had fully implemented practices for systems testing and integration. Programs should, among other activities, establish roles and responsibilities to manage testing and integration activities, including a chief developmental tester to oversee testing activities. However, one Air Force program reported difficulty in hiring a qualified individual to perform these duties. Until this position is filled, the program may not effectively manage risks and verify compliance with system acquisition and operational requirements.

Why GAO Did This Study

DOD's MAIS programs include systems that are intended to help the department sustain its key operations. The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on the department's MAIS programs annually through March 2018. This is GAO's fifth report and (1) describes the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance targets and (2) assesses the extent to which selected MAIS programs have used leading IT acquisition practices, including risk management.

GAO selected and reviewed cost, schedule, and performance data for 18 of DOD's MAIS programs that were non-classified and had an acquisition performance baseline. In addition, GAO performed an in-depth review of 5 of the programs, comparing selected IT management practices used by them to leading practices for requirements and risk management and systems testing and integration. The five selected programs were from at least two military services and had not been assessed by GAO in the past year. GAO also interviewed relevant program officials.