Missile Defense: The Warfighter and Decision Makers Would Benefit from Better Communication about the System's Capabilities and Limitations
Fast Facts
The Missile Defense Agency is developing a system to track and destroy enemy missiles. In our annual review of this system, we found that, as in prior years, MDA has made progress developing, testing, and delivering some parts of this system but did not always complete its goals. For example, while MDA delivered some kinds of missiles to the military as planned, other missiles were delayed.
We also found some of these assets are not delivered with information users need, like limitations about how they operate or how they were tested. Our six recommendations include actions MDA can take to better communicate this and other information.
Components of the Ballistic Missile Defense System
Photo collage showing sea-based, land-based, and ground-based missile defense systems, and a command and control station.
Highlights
What GAO Found
In fiscal year 2017, the Missile Defense Agency (MDA) made mixed progress in achieving its delivery and testing goals.
MDA continued to deliver assets to the military services. However, system-level integrated capabilities, such as some discrimination and integrated cyber defense improvements, were delayed and delivered with performance limitations.
Several programs achieved notable firsts, including the first intercept of an Intercontinental Ballistic Missile. However, one program experienced a failure, and other tests were delayed or deleted.
Moreover, GAO found challenges in MDA's processes for communicating the extent and limitations of integrated capabilities when they are delivered. As a result, warfighters do not have full insight into the capabilities MDA delivers.
GAO found that the average length of the undefinitized period and the not-to-exceed price of MDA's undefinitized contract actions, which authorize contractors to begin work before an agreement on terms, specifications, or price have been agreed upon, have increased over the past 5 years. While MDA policy permits use of undefinitized contracts on a limited basis, GAO and others have found that they can place unnecessary cost risks on the government.
MDA does not completely assess BMDS performance using traditional flight tests. Instead, MDA relies on models, some of which produce data with limited credibility. According to Department of Defense and MDA policy, models used to operationally assess weapons systems must be accredited to ensure they reflect the real-world system. In addition, using unaccredited models increases the risk that test results could be distorted, and leaves decision makers without key information on how the system will perform. While MDA has taken steps to improve its models, it has used many models in system operational ground tests that were not certified for that use (see figure). Additionally, MDA does not communicate model limitations to some decision makers.
Percentage of Accredited Models Used in Operational Assessments of Ballistic Missile Defense System Capability Deliveries, 2015 through 2017
Why GAO Did This Study
Since 2002, MDA has been developing a Ballistic Missile Defense System that can identify and intercept enemy threats. MDA has received approximately $132 billion and is planning to spend an additional $47.8 billion through fiscal year 2022 to continue its efforts.
The National Defense Authorization Act for Fiscal Year 2012 included a provision that GAO annually assess and report on the extent to which MDA has achieved its acquisition goals and objectives. This report addresses (1) the progress MDA made in achieving fiscal year 2017 goals; (2) the extent to which MDA uses contracting vehicles known as undefinitized contract actions; and (3) the extent to which models provide credible information about the system's operational performance. To do this work, GAO reviewed planned fiscal year 2017 baselines and other documentation and assessed them against baseline reviews and GAO's acquisition best practices guides. In addition, GAO interviewed relevant officials.
Recommendations
GAO is making six recommendations to, among other things, improve the way MDA communicates capability deliveries; better report information about MDA's use of undefinitized contract actions; and address the challenges MDA has encountered with certifying its test models and communicating limitations of those models. DOD partially concurred with the first recommendation and concurred with the other five. GAO continues to believe the recommendations are valid as discussed in the report.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA revises MDA policies to require that all integrated capabilities--capabilities that require integration of two or more elements--be included in a Technical Capability Declaration. (Recommendation 1) |
DOD partially concurred with this recommendation. While DOD agreed with the intent of this recommendation, DOD stated that the MDA Director will determine which major integrated capabilities should be delivered via the Technical Capability Declaration (TCD) process. MDA noted that it has updated MDA Directive 5000.17 in March 2021, which addresses this recommendation. GAO has reviewed the revised directive and the related manual (MDA Manual 5000.17-M date June 2021) that outlines the TCD roles and procedures. Neither the updated directive nor the related manual identify which major integrated capabilities will be delivered via the TCD process. While MDA has outlined the procedures for initiating and completing TCD assessments, it has not established clear TCD requirements. In order for the agency to meet the full intent of our recommendation, it should establish in policy, a clear, definitive standard for which capabilities require a TCD delivery. DOD considers this recommendation closed. GAO will continue to monitor the implementation process of the new manual in order to determine whether related actions are meeting the intent of our recommendation.
|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA clarifies, in written policy, the exact requirements, process, and key milestones necessary to issue a Technical Capability Declaration, including a requirement that the Assessment Requirements Review be held in such a time frame that it can provide meaningful input to MDA's test plans. (Recommendation 2) |
DOD concurred with this recommendation and contends that its new Policy Memorandum 90 meets the intent of our recommendation. However, this new policy memorandum leaves open the possibility of continued inconsistent application of the Technical Capability Declaration (TCD) process. MDA officials noted that revisions to MDA Directive 5000.17, issued in March 2021, address the recommendation. We have reviewed the updated directive and the related MDA Manual 5000.17-M (issued February 2021 and updated June 2021) that discusses the TCD process and found that it lacks a clear, definitive standard for which capabilities require a TCD. While the manual lays out the TCD process and roles, it does not mention the Assessment Requirement Review, or similar. In order for the agency to meet the full intent of our recommendation, MDA should establish in policy, a clear, definitive standard for which capabilities require a TCD for delivery that should be tracked over several years to ensure that the policy is followed. DOD considers this recommendation closed. GAO will continue to track how MDA implements the recent policies as well as any other new policies on TCDs and whether the information is transparent.
|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA includes information on current undefinitized contract actions in the BMDS Accountability Report, including the not-to-exceed price, the definitization schedule, the amount of funds obligated for the undefinitized period, and any changes since the contract action was entered into. (Recommendation 3) |
DOD concurred with and has implemented this recommendation. Since Fiscal year 2019, MDA has reported a summary of undefinitized contract actions and unpriced change orders in Appendix E of the 2019 and 2020 BMDS Accountability Reports and the 2021 Missile Defense Accountability Report in a way that satisfies this recommendation.
|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA, ensures that models used for operational tests are validated and accredited for such assessments. To help achieve this, MDA should provide the BMDS Operational Test Agency all evidence previously agreed to and needed to accredit models before ground testing events, as specified in the BMDS OTA accreditation plan. (Recommendation 4) |
DOD concurred with this recommendation and stated that MDA is actively working with the BMDS Operational Test Agency (BMDS OTA) to resolve any issues associated with, and the reporting of, modeling limitations. However, according to BMDS OTA officials, while the cooperation with MDA has improved and a number of accredited models has risen in recent years, BMDS OTA is still experiencing information access issues for data needed for accreditation. The BMDS OTA team continues to address these concerns with MDA. In October 2021, BMDS OTA team reported improved cooperation and collaboration regarding validation evidence in fiscal years 2020 and 2021. In addition, in fiscal year 2021, MDA made updates to their Modeling and Simulation Accreditation Plan, that among other things, establish a closer linkage between BMDS OTA assessment objectives and modeling and simulation performance parameters. More importantly, the updated plan changes the accreditation process and timeline to a continuous approach, which should allow for more opportunities to submit validation evidence and receive positive accreditation. At the same time, a number of models remain unaccredited due to insufficient evidence, in part caused by delays in flight testing and inability to conduct validation activities. Despite improvements in communication and collaboration regarding validation evidence, BMDS OTA continues to express general concern over missing validation data and over the numbers of unaccredited models. DOD considers this recommendation closed. GAO will continue to track MDA's progress on this subject.
|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA includes in capability delivery packages, such as the Technical Capability Declaration memos and Operational Capability Baseline change packages, the following: a. The verification, validation, and accreditation status of the models used in operational ground tests; and b. Modeling and simulation limitations that affect operational ground test results. (Recommendation 5) |
DOD concurred with this recommendation and stated that MDA has made significant progress in these efforts. MDA will provide the Technical Capability Declaration (TCD) and Operational Capability Baseline (OCB) packages for us to assess. Until we are able to do so, we believe it is premature to closeout this recommendation. We will continue to monitor MDA's progress in taking the necessary steps.
|
| Office of the Under Secretary of Defense for Research and Engineering | The Under Secretary of Defense for Research and Engineering should ensure that the Director, MDA and the Secretaries of the Armed Services responsible for operating BMDS elements develop written agreements as soon as feasible for modeling and simulations technical and funding requirements for any BMDS elements that are service-operated but represented in BMDS performance assessments. (Recommendation 6) |
DOD concurred with and are taking steps to implement this recommendation. We will continue to monitor the efforts to finalize and execute written agreements between MDA and any BMDS elements that are Service-operated but represented in BMDS performance assessments.
|