Artificial Intelligence: DOD Should Improve Strategies, Inventory Process, and Collaboration Guidance
Fast Facts
The Defense Department believes that artificial intelligence will transform warfare, and failure to adopt AI technology could hinder national security. DOD is making organizational changes and investing billions of dollars to incorporate AI technology.
We found that DOD's AI-related strategies could be more comprehensive, such as by including full descriptions of the resources needed for developing AI-enabled technologies. In addition, DOD has not yet issued guidance that clearly defines the roles and responsibilities of components that participate in AI activities.
Our 7 recommendations address these and other issues.
Highlights
What GAO Found
The 2018 Department of Defense (DOD) Artificial Intelligence (AI) Strategy defines AI as the ability of machines to perform tasks that normally require human intelligence. The strategy and associated plans include some, but not all, characteristics of a comprehensive strategy. For example, DOD's nine AI-related strategies and plans do not include full descriptions of resources and investments and risk associated with adoption of AI-enabled technologies (See fig.). Issuing guidance to include all characteristics of a comprehensive strategy in future AI-related strategies could help DOD be better positioned to help managers ensure accountability and responsible use of AI.
Assessment of DOD Artificial Intelligence-Related Strategies and Plans
DOD has begun to identify and report on its AI activities, but limitations exist in its AI baseline inventory, such as the exclusion of classified activities. DOD officials said these limitations will be addressed in subsequent phases of the AI inventory identification process. However, DOD has not yet developed a high-level plan or roadmap that captures all requirements and milestones. Such a plan would provide DOD with a high-level, end-to-end view of all the features necessary to accomplish the program's goal to provide a complete and accurate inventory of AI activities to Congress and to DOD decision makers.
DOD organizations collaborate on AI activities, but can more fully incorporate leading collaboration practices. DOD uses a variety of formal and informal collaborative mechanisms that GAO's prior work has identified, such as interagency groups. DOD has partially incorporated leading collaboration practices, such as identifying leadership. However, DOD officials told us they are in the process of developing guidance and agreements that clearly define the roles and responsibilities of DOD components that participate in AI activities. By finalizing and issuing such guidance, DOD could help ensure all participants agree upon responsibilities and decision making on AI efforts across the department.
Why GAO Did This Study
DOD strategies state that AI will transform the character of warfare, and failure to adopt AI technology could hinder the capability of warfighters to defend our nation. DOD is making organizational changes and investing billions of dollars to incorporate AI technology, such as establishing the Joint AI Center to accelerate the delivery of AI-enabled capabilities across DOD.
House Report 116-442 accompanying the National Defense Authorization Act for Fiscal Year 2021, includes a provision for GAO to assess DOD's resources, capabilities, and plans for AI technology. This report evaluates the extent to which (1) DOD's AI Strategy and associated plans include characteristics of a comprehensive strategy; (2) DOD has identified and reported AI activities across the department; and (3) DOD collaborates on its AI activities. GAO reviewed relevant laws and DOD strategies that outline plans and processes to manage AI across the department, interviewed officials, and conducted a department-wide survey. This is a public version of a sensitive report that GAO issued in February 2022. Information that DOD deemed sensitive has been omitted.
Recommendations
GAO is making seven recommendations, including for DOD to issue guidance to include all characteristics of a comprehensive strategy; develop a high-level plan or roadmap for its AI inventory process; and finalize and issue guidance and agreements that define roles and responsibilities for AI collaboration. DOD concurred with all seven of these recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | The Secretary of Defense should ensure that the Deputy Secretary of Defense issues guidance to the Chief Digital and AI Officer (once established) and the JAIC, military services, and relevant DOD organizations to include all characteristics of a comprehensive strategy in future AI strategies and associated plans and to consider the key governance practices outlined in the GAO AI accountability framework. (Recommendation 1) |
DOD concurred with this recommendation. In November 2023, the Chief Digital and AI Officer (CDAO) issued the Data, Analytics, and Artificial Intelligence Adoption Strategy (2023 AI Strategy), which builds upon and supersedes the 2018 AI Strategy and the 2020 Data Strategy and includes the four principles of the GAO AI Accountability Framework: data, governance, performance, and monitoring, which overlap with the characteristics of a comprehensive strategy. The 2023 AI Strategy states that successful implementation requires components to continue to follow responsible security procedures, review current policies and processes on data aggregation and classification, revise issuances as needed, and comply with updated guidance. By including the principles of the GAO AI Accountability Framework in its 2023 DOD AI Strategy and direction to components in its implementation guidance, DOD is better positioned to help managers ensure accountability and responsible use of AI in government programs and processes and ensure future AI strategies and plans include key elements of a comprehensive strategy. These actions meet the intent of GAO's recommendation.
|
| Department of Defense | The Secretary of Defense should ensure that the Deputy Secretary of Defense, in consultation with the Chief Digital and AI Officer (once established), the JAIC, and military departments, establish documented procedures, including timelines, for the periodic review of the DOD AI Strategy and associated military service annexes to assess the implementation of the strategy and whether any revision is necessary. (Recommendation 2) |
DOD concurred with this recommendation. In November 2023, the Chief Digital and AI Officer (CDAO) issued Data, Analytics, and Artificial Intelligence Adoption Strategy (2023 AI Strategy), which builds upon and supersedes the 2018 AI Strategy and the 2020 Data Strategy. In alignment with our recommendation, the CDAO developed and shared in March 2024 the DOD Data, Analytics, and Artificial Intelligence (AI) Adoption Strategy Implementation Guidance, which describes a process to annually review whether updates are necessary. By establishing a process to annually review whether revisions to the 2023 Strategy are necessary, DOD will be better positioned to help managers ensure accountability and responsible use of AI in government programs and processes. These actions meet the intent of GAO's recommendation.
|
| Department of Defense | The Secretary of Defense should ensure that the Chief Digital and AI Officer (once established) and the Director of the JAIC in collaboration with the Office of the Under Secretary of Defense Comptroller and other entities, as appropriate, develop a high-level plan or roadmap—aligned with the best practices of an integrated master schedule—that captures all requirements, activities, and milestones that supports the preparation of the department's AI portfolio inventory and budget data. (Recommendation 3) |
DOD concurred with this recommendation. In response to our recommendation CDAO officials told us that by the end of 2022 they would develop a high-level roadmap that captures all requirements, activities, and milestones that supports the preparation of DOD's AI portfolio inventory and budget data, as well as strategy implementation more generally. In October 2023, the White House issued an Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. In follow up to the Executive Order, the Office of Management and Budget (OMB) requested DOD develop a plan to capture AI investments. In response, DOD provided information, including a high-level roadmap that captures requirements, activities, and milestones, including duration and estimated completion dates as we recommended. We believe DOD's actions in response to OMB's request also meet the intent of our recommendation.
|
| Department of Defense | The Secretary of Defense should ensure that the Deputy Secretary of Defense issues guidance that defines outcomes and monitors accountability for AI-related activities and includes AI key performance indicators. (Recommendation 4) |
DOD concurred with this recommendation. In June 2023, DOD officials told us that the Chief Digital and Artificial Intelligence Officer (CDAO) will include high-level outcomes in the DOD Data, Analytics, and AI Strategy and help components translate that guidance to specific efforts and projects through efforts in AI test and evaluation, and monitor accountability for AI-related activities through CDAO governance processes. Additionally, officials noted that Section 1513 of the FY23 NDAA also tasked DOD to address this matter. In May 2024, DOD officials told us that the CDAO expects to brief Congress on the progress made on this front by end of December 2024, and that they have a working group that is developing metrics and performance indicators to assess progress on strategy implementation. When we confirm the actions the department has taken in response to this recommendation, we will provide updated information.
|
| Department of Defense | The Secretary of Defense should ensure that the Chief Digital and AI Officer (once established) and the Director of the JAIC issue a roadmap or a high-level plan that captures all requirements and milestones for developing and onboarding users to the Joint Common Foundation. (Recommendation 5) |
DOD concurred with this recommendation. In January 2023, DOD officials told us they completed the merge of key areas of the Joint Common Foundation (JCF) and Advana and provided us an integrated roadmap for current and planned AI development capabilities. DOD officials stated that the roadmap documents the key areas of JCF that merged and migrated to Advana, including improvements to the customer intake and onboarding processes. We reviewed the integrated roadmap and requested additional documentation to ensure its sufficiency in addressing our recommendation. In January 2024, CDAO officials told us that they pivoted from JCF investment toward Advana EDGE as well as an initiative known as Alpha-1. These officials said in September 2024 that they would issue a plan that considers requirements and milestones for incentivizing use of Alpha-1 across DOD. When we confirm the actions the department has taken in response to this recommendation, we will provide updated information.
|
| Department of Defense | The Secretary of Defense should ensure that the Deputy Secretary of Defense develops a timeline and guidance that directs Chief Digital and AI Officer (once established), and the JAIC, military services, and other relevant entities to establish common terminology for AI related activities. (Recommendation 6) |
DOD concurred with this recommendation. On September 1, 2023, in response to our recommendation, the Chief Digital and AI Officer issued a memo, Data Analytics and Artificial Intelligence Lexicon, that stated that the use of common terminology is key to DOD's ability to collectively realize the goals of the Data, Analytics, and AI Adoption Strategy. The memo provided that to promote consistency within the national security community, DOD aligns data management terminology to the Office of the Director of National Intelligence's Data Management Lexicon. DOD officials also stated that they will leverage the National Institute of Science and Technology, The Language of Trustworthy AI: An In-Depth Glossary of Terms (NIST AI 100-3, issued March 2023), as the basis for terminology regarding AI. By developing and issuing the Data Analytics and Artificial Intelligence Lexicon Memorandum, DOD has provided the guidance for establishing common terminology for DOD AI related activities, which could further bridge organizational culture and better align DOD-wide AI activities, such as improving processes for sharing classified data. This action meets the intent of GAO's recommendation.
|
| Department of Defense | The Secretary of Defense should ensure that the Deputy Secretary of Defense, in coordination with the Chief Digital and AI Officer (once established) and the JAIC, finalize and issue guidance and agreements that define the roles and responsibilities of the military services and other DOD organizations for leadership and relevant participants collaborating on AI activities. (Recommendation 7) |
DOD concurred with this recommendation. In June 2022, DOD officials stated that by the end of December 2022, the Chief Digital and Artificial Intelligence Officer (CDAO) will coordinate with the Deputy Secretary of Defense to ensure the necessary guidance and agreements are established for the Services and other DOD organizations. In September 2024, DOD officials stated that the CDAO expects to publish a CDAO Charter by October 2024, and corresponding Capstone Directive to follow, that will define the roles and responsibilities of DOD components as they pertain to collaborating on AI activities. When we confirm the actions the department has taken in response to this recommendation, we will provide updated information.
|