DOD Weapon Systems:

Missed Trade-off Opportunities During Requirements Reviews

GAO-11-502, Jun 16, 2011

Additional Materials:

Contact:

Michael J. Sullivan
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide.

The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the analysis of alternatives (AOA), which occurs between the JROC's review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program's CDD. As a result, the JROC does not have an opportunity to provide military advice on trade-offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC's efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD's joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC's requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis. GAO recommends that the JROC establish a mechanism to review AOA results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO's recommendations and generally agreed with their intent, but differed with GAO on how to implement them.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should assign priority levels to the CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes.

    Agency Affected: Department of Defense: Joint Chiefs of Staff

    Status: Closed - Implemented

    Comments: DOD partially concurred with this recommendation, and revised both the Joint Capabilities Integration and Development System (JCIDS) instruction and the associated guidance to address this recommendation. Specifically, in January 2012, the JROC updated its operating manual and required that its subordinate bodies - the Functional Capabilities Boards - develop joint priorities for validated capability requirements, which could be used to inform DOD's annual program and budget review. The updated manual also created distinct processes for urgent (near-term) and emergent (mid-term) operational needs. Specifically, it established that urgent operational needs - related to ongoing contingency operations - are expected to be addressed within two years, and that emergent operational needs - related to anticipated contingency operations - are expected to be addressed within five years. The updated manual also established that urgent operational needs shall be staffed for validation within 15 calendar days, and that emergent operational needs shall be staffed for validation within 31 calendar days.

    Recommendation: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program's current level of knowledge.

    Agency Affected: Department of Defense: Joint Chiefs of Staff

    Status: Open

    Comments: DOD partially concurred with this recommendation, but has not taken action to address it. In DOD's January 2012 revision to the Joint Capabilities Integration and Development System (JCIDS) operation manual, there is still no requirement to conduct sensitivity or uncertainty analyses, or disclose confidence levels for resource estimates. The only requirement is to "cite applicable cost analyses conducted to date," the same as in 2009.

    Recommendation: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should require that capability sponsors present resource estimates that have been reviewed by a military service's cost analysis organization to ensure best practices are being followed.

    Agency Affected: Department of Defense: Joint Chiefs of Staff

    Status: Open

    Comments: DOD partially concurred with this recommendation, but has not taken action to address it. In DOD's January 2012 revision to the Joint Capabilities Integration and Development System (JCIDS) operation manual, there is still no requirement to have resource estimates reviewed by a military service's cost analysis organization. The only requirement is to "cite applicable cost analyses to date," the same as in 2009.

    Recommendation: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority.

    Agency Affected: Department of Defense: Joint Chiefs of Staff

    Status: Closed - Implemented

    Comments: DOD has revised both the Joint Capabilities Integration and Development System (JCIDS) instruction and the associated guidance to address this recommendation. Specifically, in January 2012, the Joint Requirements Oversight Council (JROC) updated its operating manual and required AOA briefings that will allow the JROC or its subordinate body - the Joint Capabilities Board - to advise the milestone decision authority on the best approach to satisfying capability requirements.

    Recommendation: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, should modify the Joint Capabilities Integration and Development System (JCIDS) operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements.

    Agency Affected: Department of Defense: Joint Chiefs of Staff

    Status: Open

    Comments: DOD partially concurred with this recommendation, but has not taken action to address it. In DOD's January 2012 revision to the Joint Capabilities Integration and Development System (JCIDS) operation manual, there is still no requirement for Capability Development Documents (CDDs) to discuss or address potential redundancies across proposed or existing programs. The manual similarly lacked this requirement in 2009.