DOD Major Automated Information Systems: Adherence to Best Practices Is Needed to Better Manage and Oversee Business Programs
Highlights
What GAO Found
The strength of Department of Defense's (DOD) policies for managing and overseeing major automated information system (MAIS) programs varies. Specifically, the policy for managing 24 non-business MAIS programs adheres to leading information technology (IT) management practices, but the policy for managing 10 MAIS business programs does not always do so (see table).
Analysis of Department of Defense (DOD) Policies for Managing Major Automated Information System (MAIS) Programs
|
Leading information technology management practice |
Adherence to policy for non-business MAIS programs |
Adherence to policy for MAIS business programs |
|
Instituting an investment board as a process for creating and defining the membership, guiding principles, operations, roles, responsibilities, and authorities |
Yes |
Yes |
|
Identifying decision authorities for making important executive-level acquisition decisions |
Yes |
Yes |
|
Providing oversight whereby the organization monitors programs on their performance progress; such information includes (1) baseline estimates on cost and schedule goals and (2) thresholds to identify high risk on cost and schedule |
Yes |
No |
|
Capturing and providing performance information about a particular investment (program) to decision makers at regular intervals (e.g., quarterly and annually) |
Yes |
No |
Source: GAO analysis of agency documentation. | GAO-18-326
When DOD categorized 10 of the 34 MAIS programs as MAIS business programs, it also directed these programs to adhere to DOD's business systems policy (DOD Instruction 5000.75). However, the department directed those programs to use a policy for the management and oversight of MAIS business programs that was not fully comprehensive. Until DOD updates its business systems policy to address gaps in establishing performance information such as baseline estimates on program cost and schedule goals, identifying thresholds to identify high risk, and requiring periodic reports to be provided to stakeholders at regular intervals, stakeholders will likely not have all the information they need to manage and oversee MAIS business programs.
While all 15 business and non-business MAIS programs had either increased or decreased their planned cost estimates and the majority had delays in their planned schedule estimates, the majority of the 9 programs that had performance targets met those performance goals. Specifically, the decreases and increases in cost estimates ranged from a decrease of $1.6 billion (-41 percent) to an increase of $1.5 billion (163 percent). The decreases in planned cost were largely due to scope reduction, while cost increases were due to underestimating levels of effort and contracting issues. The slippages in schedule estimates ranged from a delay of 5 years to 5 months; these delays were caused by unrealistic expectations or unplanned changes. Six of the 9 programs that had performance targets met all of them, while the other 3 met several but not all of their performance targets. The other 6 programs were in the early stages of system development and had not begun performance testing.
Why GAO Did This Study
DOD's MAIS programs are intended to help the agency sustain its key operations. In April 2017, recognizing that MAIS programs met different mission needs, DOD categorized its MAIS programs into business and non-business systems.
The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on DOD's MAIS programs annually through March 2018. GAO's objectives, among others, were to (1) assess DOD's policies for managing and overseeing MAIS programs and (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance goals. To address these objectives, GAO compared DOD's policies for managing and overseeing all 34 MAIS programs (24 non-business programs and 10 business programs) to leading IT management practices. GAO also compared 15 selected programs' initial cost, schedule, and performance baselines to their current acquisition program estimates.
Recommendations
GAO is making three recommendations, including that DOD update its policy for managing MAIS business programs to include baseline estimates. DOD partially concurred with this recommendation, and fully concurred with the other two recommendations. GAO continues to believe that all the recommendations are warranted.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment to update the policy or guidance for MAIS business programs. Specifically, the update should include the following elements: (1) establishment of initial and current baseline cost and schedule estimates, (2) predetermined threshold cost and schedule estimates to identify the point when programs may be at high risk, and (3) quarterly and annual reports on the performance of programs to stakeholders. (Recommendation 1) |
As of November 2024, the department has made progress in implementing this recommendation, but more work remains. Specifically, in January 2020, the Under Secretary of Defense for Acquisition and Sustainment issued an updated instruction on defense business systems requirements and acquisition, which included guidance on establishing baseline cost and schedule estimates and considering progress against the baselines at key decision points. However, the instruction does not make a distinction between initial and current baselines. Further, according to DOD, the office considers specifying predetermined threshold cost and schedule estimates and frequency for status reporting to be matters for implementation guidance issued by department components or determined by a program decision authority. In August 2023, DOD reiterated that its position regarding specifying predetermined threshold cost and schedule estimates and frequency for status reporting remains unchanged. GAO disagrees with the department's position. As of November 2024, the department has not indicated that it has changed its position on this recommendation. We will continue to monitor the department's implementation of this recommendation.
|
| Department of Defense | The Secretary of Defense should direct the Director of the Defense Health Agency to direct the program manager for the Defense Healthcare Management System Modernization program to: (1) finalize and approve its requirements management plan, (2) identify and document changes that should be made to plans and work products resulting from changes to the requirements baseline, and (3) quantify costs and benefits of risk mitigation within its program-level risk mitigation plans. (Recommendation 2) |
In May 2022, DOD demonstrated that it has implemented this recommendation. Specifically, in March 2019, the DHMSM program manager approved a requirements management plan that included identifying and documenting changes that should be made to plans and work products resulting from changes to the baseline requirements. Specifically, it included forward and backward configuration and change management of the baselined requirements and managing traceability of requirements to design artifacts, test cases, defects, and change requests. As a result, the program is better positioned to identify inconsistencies between requirement changes and project plans and work products and to initiate corrective actions to resolve them. In addition, DOD provided the program's acquisition strategy, which describes how the program reduced its costs related to three risk areas.
|
| Department of Defense | The Secretary of Defense should direct the Secretary of the Navy to direct the program manager for the Navy Consolidated Afloat Networks and Enterprise Services program to identify and document, in the failover/recovery plan, all potential external environmental issues, such as hazards, threats, and vulnerabilities that could negatively affect work efforts. (Recommendation 3) |
The Department of Defense agreed with our recommendation. In a March 29, 2018 memo, the department reported that the Secretary of the Navy plans to direct the program manager to identify and document all potential external environmental issues in its plan. As of August, 2019, the department has addressed GAO's recommendation. The program indicated that within the CANES Interactive Electronic Technical Manual (IETM), the CANES system has a disaster recovery plan. Based on GAO's recommendation, the program office has updated the IETM to state that CANES inherits environmental effects risks and mitigations from the platform and that CANES should be considered in all platform disaster recovery plans. The updated IETM was completed on December 30, 2018.
|