Independent Review Is Needed to Ensure DOD's Use of Cost Estimating Tool for Contingency Operations Follows Best Practices
GAO-08-982: Published: Sep 15, 2008. Publicly Released: Sep 15, 2008.
Since the September 2001 terrorist attacks, Congress has provided about $800 billion as of July 2008 to the Department of Defense (DOD) for military operations in support of the Global War on Terrorism (GWOT). GWOT budget requests have grown in scope and the amount requested has increased every year. DOD uses various processes and the Contingency Operations Support Tool (COST) to estimate costs for these operations and to develop budget requests. GAO assessed (1) how DOD uses COST and other processes to develop GWOT budget requests and (2) what actions DOD has taken to ensure COST adheres to best practices for cost estimation. GAO interviewed DOD officials and others to determine how the services develop GWOT budget requests using COST and other processes. GAO also used its Cost Assessment Guide as criteria for best practices for cost estimation.
The services use COST as part of their process to develop a GWOT budget request. While the Army relies more on the estimate resulting from COST, the other services adjust the results of COST to reflect estimates they generate outside of COST, based on historical obligation data and other information. DOD's financial management regulation and other guidance require components to use COST to develop an estimate for the deployment and sustainment of military personnel and equipment for ongoing operations in support of GWOT. While all services use COST to develop an initial estimate, Air Force, Marine Corps, and Navy budget officials alter the results of the tool to match information provided by lower level commands and historical obligation data that they believe are more accurate than the COST-generated estimate. These officials stated that the tool routinely overestimates some costs and therefore most changes made are decreases in the amount estimated by COST. These officials believe that the requirement to use COST to develop a GWOT budget request is a duplicative process to their preferred method of using historical obligation data and other information better suited to their specific service. For example, they stated that COST better represents the needs of Army ground forces and the tool has not been refined to be as effective for estimating needs for their service's mission. These officials also mentioned that COST is better suited for developing estimates for smaller-scale contingency operations than for the lengthy deployments and sustainment phases associated with a large campaign such as GWOT. To develop estimates for items that are outside the scope of COST, such as procurement and certain contracts, the military services rely primarily on needs assessments developed by commanders and historical obligation data. DOD has taken steps to improve the performance and reliability of COST; however, COST could benefit from an independent review of the tool's adherence to best practices for high-quality cost estimation as described in GAO's Cost Assessment Guide. COST has been refined many times and cost factors are routinely updated in an effort to use the most current information available to develop an estimate. DOD officials stated they are confident in the tool's ability to provide reasonable estimates because COST is frequently updated. However, COST has not been assessed against best practices for cost estimation to determine whether COST can provide high-quality estimates that are well documented, comprehensive, accurate, and credible. While GAO did not undertake a full assessment of COST against best practices, it determined that some features of the tool meet best practices while other features would benefit from further review. For example, the tool adheres to several best practices for a comprehensive and accurate cost estimate, such as frequent updates to the structure of COST and the data the tool uses to generate estimates. However, COST relies on GWOT obligation data that GAO has identified as being of questionable reliability. A thorough, independent review of COST against best practices could provide decision makers with information about whether the tool creates cost estimates for GWOT expenses that are well documented, comprehensive, accurate, and credible.
Recommendations for Executive Action
Status: Closed - Not Implemented
Comments: While DOD concurred with our recommendation, the agency comments did not say that an independent review of COST would be performed and that a prior review by the Air Force was sufficient. However, DOD subsequently performed a review of the COST model. The department reviewed the cost estimates generated by the COST model and the actual expenditures for Operation Unified Protector. The study, completed in October 2011, found that COST model estimation was accurate within a 5% range of the actual execution for this operation. However, the review was not independent as it was completed by department officials and others responsible for the creating and maintaining the model. Officials stated that an independent review against best practices for cost estimation was not planned, therefore we consider this recommendation closed not implemented.
Recommendation: To ensure that DOD budget requests for GWOT are based on a sound cost estimation process, the Secretary of Defense should direct the Under Secretary of Defense (Comptroller) to arrange for an independent review of COST against best practices for cost estimation.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: The Department uses every opportunity to refine and improve the Contingency Operations Support Tool (COST) model for comprehensiveness and accuracy. For example, during a supplemental budget build OSD Comptroller, the Joint Staff, Institute for Defense Analyses, and Service personnel review the model results and critique assumptions, factors, and outputs of the COST model. The results of the review are used to adjust the Supplemental amounts and for initiating improvements to the COST model. In addition, annual training involving all COST users is conducted before supplemental requests are developed. The feedback and exchange between the attendees (Services, OUSD(C), the Joint Staff and on occasion the Office of Management and Budget) has resulted in changes to the model that made it more useful and accurate. The COST model has proven itself to be a dynamic and adaptable tool that is used successfully by different organizations in DOD to create estimates for a variety of force deployments. For supplemental requirements that are not calculated in the COST model, the Department has developed an extensive review process for each non-COST line item. Each Service must submit all non-COST model items to OUSD(C) at the line items level of detail which are then compared to historical execution and/or anticipated operational needs. Only those line items that are fully justified are included in the supplemental request.
Recommendation: Based on that review, and taking into consideration how each service uses COST, the Secretary of Defense should direct the Under Secretary of Defense (Comptroller) to consider options for refining COST, determine the appropriate items or types of costs for which COST should be applied, and identify methods to be used when COST is not appropriate.
Agency Affected: Department of Defense