Software Development: DOD Faces Risks and Challenges in Implementing Modern Approaches and Addressing Cybersecurity Practices
Fast Facts
The Department of Defense plans to spend $12 billion on its 29 largest business information technology systems during FYs 2019-2022. DOD's efforts to modernize business systems has been a topic on our High Risk List since 1995.
We found:
- 22 programs that were actively developing software reported using approaches that reduced risk of cost and schedule overruns, such as early cybersecurity testing
- DOD may be underestimating the risks for some of its acquisitions
- DOD has taken steps to improve the sharing and transparency of data it uses to monitor acquisitions but needs to do more
We made 2 recommendations to improve acquisition monitoring.
Highlights
What GAO Found
According to the Department of Defense's (DOD) fiscal year (FY) 2021 budget request, DOD spent $2.8 billion on the 29 selected major business information technology (IT) programs in FY 2019. The department also reported that it planned to invest over $9.7 billion on these programs between FY 2020 and FY 2022. In addition, 20 of the 29 programs reported experiencing cost or schedule changes since January 2019. Program officials attributed cost and schedule changes to a variety of reasons, including modernization changes and requirements changes or delays. Seventeen of the 29 programs also reported experiencing challenges associated with the early impacts of the COVID-19 pandemic, including the slowdown of contractors' software development efforts.
DOD and GAO's assessments of program risk identified a range of program risk levels and indicated that some programs could be underreporting risks. Specifically, of the 22 programs that were actively using a register to manage program risks, DOD rated nine programs as low risk, 12 as medium risk, and one as high risk. In contrast, GAO rated seven as low risk, 12 as medium risk, and three as high risk. In total, GAO found 10 programs for which its numerical assessments of program risk reflected greater risk than reported by DOD, while DOD had three programs with greater reported risk than GAO. DOD officials noted that differences in risk levels might be associated with a variety of factors, including different risk assessment approaches. However, the differences in risk level GAO identified highlight the need for DOD to ensure that it is accurately reporting program risks. Until the department does so, oversight of some programs could be limited by overly optimistic risk perspectives.
As of December 2020, program officials for the 22 major DOD business IT programs that were actively developing software reported using approaches that may help to limit cost and schedule risks. (See table.)
Selected Software Development and Cybersecurity Approaches That May Limit Risks and Number of Major DOD Business IT Programs That Reported Using the Approach
|
Software development and cybersecurity approaches that may limit risk |
Number of programs that reported using the approach |
|
Using off-the-shelf software |
19 of 22 |
|
Implementing continuous iterative software development |
18 of 22 |
|
Delivering software at least every 6 monthsa |
16 of 22 |
|
Developing or planning to develop a cybersecurity strategy |
21 of 22 |
|
Conducting developmental cybersecurity testing |
16 of 22 |
|
Conducting operational cybersecurity testing |
15 of 22 |
Source: GAO analysis of Department of Defense questionnaire responses. | GAO-21-351
aThe Defense Innovation Board encourages more frequent delivery of working software to users for Agile and DevOps practices.
Program officials also reported facing a variety of software development challenges while implementing these approaches. These included difficulties finding and hiring staff, transitioning from waterfall to Agile software development, and managing technical environments. DOD's continued efforts to address these challenges will be critical to the department's implementation of modern software development approaches.
DOD has also made organizational and policy changes intended to improve the management of its IT acquisitions, such as taking steps to implement Agile software development and improve data transparency. In addition, to address statutory requirements, DOD has taken steps to remove the department's chief management officer (CMO) position. However, the department had not yet sufficiently implemented these changes. Officials from many of the 18 programs GAO assessed that reported using Agile development reported that DOD had implemented activities associated with Agile transition best practices to only some or little to no extent, indicating that the department had not sufficiently implemented best practices. For example, 12 of the 18 programs reported that DOD's life-cycle activities only supported Agile methods to some or little to no extent. Program officials also reported challenges associated with implementing Agile software development. The department has a variety of efforts underway to help with its implementation of Agile software development. DOD officials stated that the department's transition to Agile will take years and will require sustained engagement throughout DOD.
In addition, DOD has taken steps aimed at improving the sharing and transparency of data it uses to monitor its acquisitions. According to a November 2020 proposal from the Office of the Under Secretary for Acquisition and Sustainment, DOD officials are to develop data strategies and metrics to assess performance for the department's acquisition pathways. However, as of February 2021, DOD did not have data strategies and had not finalized metrics for the two pathways associated with the programs discussed in this report. Officials said they were working with DOD programs and components to finalize initial pathway metrics. They stated that they plan to implement them in fiscal year 2021 and continue to refine and adjust them over the coming years. Without important data from acquistion pathways and systems, DOD risks not having timely quantitative insight into program performance, including its acquisition reform efforts.
Finally, DOD's CMO position was eliminated by a statute enacted in January 2021. This position was responsible for key efforts associated with the department's business systems modernization, which has been on GAO's High Risk List since 1995. DOD plans to take steps to address the uncertainty associated with the recent elimination of the position.
Why GAO Did This Study
For fiscal year 2021, DOD requested approximately $37.7 billion for IT investments. These investments included major business IT programs, which are intended to help the department carry out key business functions, such as financial management and health care.
The National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to assess selected IT programs annually through March 2023. GAO's objectives for this review were to (1) summarize DOD's reported performance of its portfolio of IT acquisition programs and the reasons for this performance; (2) evaluate DOD's assessments of program risks; (3) summarize DOD's approaches to software development and cybersecurity and identify associated challenges; and (4) evaluate how selected organizational and policy changes could affect IT acquisitions.
To address these objectives, GAO selected 29 major business IT programs that DOD reported to the federal IT Dashboard (a public website that includes information on the performance of major IT investments) as of September 2020. GAO reviewed planned expenditures for these programs, from fiscal years 2019 through 2022, as reported in the department's FY 2021 budget request. It also aggregated program office responses to a GAO questionnaire that requested information about cost and schedule changes that occurred since January 2019 and the early impacts of COVID-19.
GAO also analyzed the risks of the 22 programs that were actively using central repositories known as risk registers to manage program risks. GAO used these registers to create program risk ratings, and then compared its ratings to those of the DOD chief information officer (CIO).
In addition, GAO aggregated DOD program office responses to the questionnaire that requested information about the software and cybersecurity practices used by 22 of the 29 IT programs that were actively developing software. GAO compared the responses to relevant guidance and leading practices.
GAO reviewed selected IT-related organizational and policy changes and reviewed reports and documentation related to the effects of these changes on IT acquisitions. GAO also aggregated program office responses to the questionnaire that requested information about DOD's implementation of these changes. This included information on DOD's implementation of best practices as part of its efforts to implement Agile software development. GAO met with relevant DOD officials to discuss each of the topics addressed in this report.
Recommendations
GAO is making two recommendations to DOD related to revisiting the department's CIO risk ratings and improving data strategies and automated data collection efforts for the business system and software acquisition pathways necessary for stakeholders to monitor acquisitions and critical to the department's ability to assess acquisition performance.
DOD concurred with GAO's recommendations and described actions it planned to take, or had begun taking, to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | The Secretary of Defense should direct the Chief Information Officer to revisit program risk ratings for its next submission to the federal IT Dashboard for the programs where the DOD CIO's program risk ratings indicated less risk than GAO's assessments of program risk. (Recommendation 1) |
In October 2021, the Department of Defense (DOD) described steps it had taken and planned to take to address this recommendation. Specifically, the department stated that it tasked components to review risk ratings for programs that reported having less risk than the GAO risk ratings reported in GAO-21-351, and to report findings to inform their component Chief Information Officers' (CIO) next risk ratings submission to DOD CIO. In addition, the department stated that it would assess component CIO risk ratings before posting final risk ratings to the federal IT Dashboard by November 2021. In July 2023, DOD indicated that DOD CIO received confirmation from components that GAO's recommendation was considered by programs that reported less risk than the GAO risk ratings as part of their fiscal year 2022 submissions to DOD CIO prior to submitting their risk ratings. In addition, DOD reported that DOD CIO assessed the component risk ratings prior to their fiscal year 2022 submissions to the federal IT Dashboard. Based on our analysis comparing DOD's reported fiscal year 2021 and 2022 program risk ratings, one of these programs (the Air Force's Defense Enterprise Accounting and Management System) raised its risk rating from medium risk to moderately high risk, reflecting greater consistency with GAO's rating for the program. In addition, in DOD's fiscal year 2023 submission to the federal IT Dashboard, another program (The Defense Medical Logistics - Enterprise Solution) increased its risk rating from low risk to moderately low risk, consistent with GAO's rating. As a result, we are closing this recommendation as implemented.
|
| Department of Defense | The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment, in consultation with appropriate internal and external stakeholders, to ensure the data strategies and data collection efforts for the business system and software acquisition pathways define, collect, automate, and share, with the appropriate level of visibility, the metrics necessary for stakeholders to monitor acquisitions and that are critical to the department's ability to assess acquisition performance. (Recommendation 2) |
In October 2021, DOD described steps it has taken and plans to take to address this recommendation. Specifically, related to the business systems pathway, the department stated that it plans to determine, review, and analyze existing defense business systems data reported to the Office of the Secretary of Defense and military services; define reporting thresholds and identify metrics; and document required defense business system data elements by the fourth quarter of fiscal year 2022. Related to the software pathway, the department stated that it has established a software acquisition pathway (SWP) data collection strategy and socialized it with component headquarters and relevant program offices. The department also stated that it plans to prepare a SWP semi-annual reporting template and conduct trial submissions with early adopter programs to gain insights, implement suggestions, and improve the template. Further, the department planned to collect the first iteration of the SWP program metrics data via a MS Excel template in October 2021 and then transition to automated transmission of metrics during calendar year 2022. In February 2024, DOD provided an update and documentation associated with these pathways. The documentation included reports that showed information such as expenditures and number of systems. However, the department did not provide a data strategy associated with the business systems pathway. For the software pathway, the department provided documentation such as a data standards promulgation memo and a data governance document describing the metrics data to be collected about programs. However, it did not provide the data actually being collected and show how it is being automated with the appropriate level of visibility. Additionally, for both pathways, DOD did not provide sufficient information about how it coordinated with stakeholders to ensure it developed and shared the metrics necessary to monitor acquisitions and that are critical to the department's ability to assess acquisition performance. We will follow up with the department to obtain additional information as part of our efforts to continue to monitor the status of this recommendation.
|