Missile Defense: Cost Estimating Practices Have Improved, and Continued Evaluation Will Determine Effectiveness

What GAO Found

The Department of Defense's Missile Defense Agency (MDA) has taken steps in-line with GAO best practices to improve its cost estimating practices including:

• releasing an internal cost estimating handbook,
• performing internal cost reviews,
• assembling an independent cost assessment team, and
• receiving independent cost estimates from the Office of the Secretary of Defense for Cost Assessment and Program Evaluation (CAPE).

However, continued evaluation over time by MDA's internal cost review and its independent cost assessment teams will provide more confidence in the agency's ability to identify areas where the cost estimate follows its guide and where correction is needed. Making these corrections will do the real work to improve the quality of the estimates.

GAO's evaluation of MDA's Cost Estimating and Analysis Handbook shows that it is well aligned with all the GAO best practices in GAO's Cost Estimating and Assessment Guide. MDA released this handbook in June 2012 to standardize the agency's cost analysis requirements and procedures for preparing, documenting, and presenting MDA's cost estimates.

GAO's examination of MDA's cost review scores indicates that the agency can effectively assess the comprehensiveness of its estimates. In the past few years, MDA created an internal process to evaluate its program cost estimates. GAO's cost guide states that relying on a standard process that, among other things, communicates the estimate's basis, identifies the quality of the data, and thoroughly documents the effort should result in cost estimates that are trustworthy. As part of MDA's cost review process, MDA cost estimators meet annually to compare program estimates to characteristics associated with a high quality estimate, resulting in scores ranging from "not met" to "met." For 11 estimates, GAO determined whether it concurred or did not concur with MDA's scores for one of the four GAO characteristics of a high quality estimate--comprehensiveness. Based on its analysis, GAO concurred with about 90 percent of MDA's scores.

MDA also stood up an independent cost team in January 2013 to obtain an additional assessment of the quality of its cost estimates; however, it is too early to determine the team's effectiveness. GAO's guide states an independent cost assessment team may be used to evaluate the program estimate's quality and accuracy to ensure it captures all requirements. While the purpose of MDA's team aligns with this GAO best practice, the team was still drafting a guide detailing its procedures and reporting format planned for release by January 2015, and had only reported on one program estimate with plans to review all program estimates by the end of 2015.

Lastly, the CAPE has conducted independent cost assessments for MDA programs in the past few years and provided some credibility for MDA's estimates, but about half of the cost baselines still remain unverified. MDA concurred with GAO's 2010 recommendation to obtain CAPE independent cost estimates and has done so for about half of MDA program baselines presented in the 2014 baseline report. One of GAO's criteria for a credible cost estimate is having an independent organization conduct an independent cost estimate. If the results are close to the program's estimate, one can be more confident of the accuracy of the program's estimate. CAPE's comparisons of their independent cost estimate results to MDA's estimates are close and appear to validate many areas of MDA's estimates. However, the agency is on track to receive validation for about half of its programs, representing less than 25 percent of the total baselined costs in MDA's 2014 baseline report. 

Why GAO Did This Study

The National Defense Authorization Act for Fiscal Year 2014 mandated that MDA submit a report to the congressional defense committees by December 26, 2014, which includes, among other things, the actions it has taken to improve the quality of cost estimates that support cost baselines presented in MDA's annual baseline report, how those actions compare to GAO best practices, and GAO's views on its contents. This report includes GAO's assessment of four key actions outlined in MDA's October 2014 draft report including MDA's: (1) June 2012 Cost Estimating Handbook, (2) internal cost reviews, (3) independent cost assessment team, and (4) CAPE independent cost estimates. To examine the effectiveness of these actions, GAO compared MDA's cost handbook to best practices in GAO's cost estimating guide, determined whether we concurred or did not concur with MDA cost review scores for one of GAO's four characteristics of a high quality cost estimate, met with estimators in MDA's independent cost assessment team to discuss its process and products, and reviewed CAPE's independent cost assessment results. This report was published prior to MDA's report for inclusion, as mandated, in MDA's final report to congressional defense committees.