DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews
Highlights
The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide.
The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the analysis of alternatives (AOA), which occurs between the JROC's review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program's CDD. As a result, the JROC does not have an opportunity to provide military advice on trade-offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC's efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD's joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC's requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis. GAO recommends that the JROC establish a mechanism to review AOA results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO's recommendations and generally agreed with their intent, but differed with GAO on how to implement them.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Joint Chiefs of Staff | To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority. |
DOD has revised both the Joint Capabilities Integration and Development System (JCIDS) instruction and the associated guidance to address this recommendation. Specifically, in January 2012, the Joint Requirements Oversight Council (JROC) updated its operating manual and required AOA briefings that will allow the JROC or its subordinate body - the Joint Capabilities Board - to advise the milestone decision authority on the best approach to satisfying capability requirements.
|
| Joint Chiefs of Staff | To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should require that capability sponsors present resource estimates that have been reviewed by a military service's cost analysis organization to ensure best practices are being followed. |
The Joint Staff implemented the recommendation through an update to the Joint Capabilities Integration and Development System (JCIDS) Manual. The JCIDS manual now states that sponsors of Capability Development Documents (CDD) are to ensure that the resource estimates contained in those documents have been reviewed by the sponsors' cost analysis organization to ensure that best practices are being followed.
|
| Joint Chiefs of Staff | To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program's current level of knowledge. |
DOD partially concurred with this recommendation, and the Joint Staff addressed it through an update to the Joint Capabilities Integration and Development System (JCIDS) Manual. The JCIDS manual now states that resource estimates are to present key results from sensitivity and uncertainty analyses, including confidence levels, based on the program's current level of knowledge.
|
| Joint Chiefs of Staff | To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should assign priority levels to the CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes. |
DOD partially concurred with this recommendation, and revised both the Joint Capabilities Integration and Development System (JCIDS) instruction and the associated guidance to address this recommendation. Specifically, in January 2012, the JROC updated its operating manual and required that its subordinate bodies - the Functional Capabilities Boards - develop joint priorities for validated capability requirements, which could be used to inform DOD's annual program and budget review. The updated manual also created distinct processes for urgent (near-term) and emergent (mid-term) operational needs. Specifically, it established that urgent operational needs - related to ongoing contingency operations - are expected to be addressed within two years, and that emergent operational needs - related to anticipated contingency operations - are expected to be addressed within five years. The updated manual also established that urgent operational needs shall be staffed for validation within 15 calendar days, and that emergent operational needs shall be staffed for validation within 31 calendar days.
|
| Joint Chiefs of Staff | To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should modify the Joint Capabilities Integration and Development System (JCIDS) operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements. |
In January 2012, DOD revised the Joint Capabilities Integration and Development System (JCIDS) operation manual, and the updated JCIDS manual states that the chairs of Functional Capabilities Boards should determine if proposed programs are unnecessarily redundant to existing capabilities before recommending a Capability Development Document (CDD) for Joint Requirement Oversight Council (JROC) validation. The Functional Capabilities Boards can also recommend non-materiel changes and/or collaboration opportunities between groups with similar needs in lieu of new acquisition programs if a submitted CDD includes an unnecessary redundancy.
|