This is the accessible text file for GAO report number GAO-11-502 entitled 'DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews' which was released on June 16, 2011. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Committees: June 2011: DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews: GAO-11-502: GAO Highlights: Highlights of GAO-11-502, a report to congressional committees. Why GAO Did This Study: The Weapon Systems Acquisition Reform Act of 2009 (WSARA) directed the Joint Requirements Oversight Council (JROC) to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its requirements review process. WSARA also directed GAO to assess the implementation of these requirements. This report addresses (1) the extent to which the JROC has considered trade-offs within programs, (2) the quality of resource estimates presented to the JROC, and (3) the extent to which the JROC is prioritizing requirements and capability gaps. To do so, GAO analyzed requirement documents reviewed by the JROC in fiscal year 2010, which identified capability gaps or performance requirements for new major defense acquisition programs. GAO also assessed resource estimates presented to the JROC against best practices criteria in the GAO Cost Estimating and Assessment Guide. What GAO Found: The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. According to DOD officials, the most significant trade-offs are made by the military services during the AOA, which occurs between the JROC’s review of an Initial Capabilities Document (ICD) and its review of a Capability Development Document (CDD). The AOA is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. The JROC does not formally review the trade-off decisions made as a result of an AOA until it reviews a proposed program’s CDD. As a result, the JROC does not have an opportunity to provide military advice on trade- offs and the proposed solution before it is selected, and a significant amount of time and resources can be expended in technology development before the JROC gets to formally weigh in. Figure: AOA’s Relationship to JROC Requirements Reviews: [Refer to PDF for image: illustration] ICD developed: No cost, schedule, performance objectives. (Requirements development) JROC review: Too early for trade-offs. Analysis of alternatives: Most significant trade-offs made. (Acquisition activity) Milestone A. (Acquisition milestone) Technology development: Requirements refined and vetted. (Requirements development) CDD developed: Cost, schedule, performance objectives identified. (Acquisition activity) JROC review: Requirements rarely change. Milestone B. (Acquisition milestone) Source: GAO analysis of DOD policy. [End of figure] The military services did not consistently provide high-quality resource estimates to the JROC for proposed programs in fiscal year 2010. GAO found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses or examined the effects of changing assumptions and ground rules, all of which could further the JROC’s efforts to ensure that programs are fully funded and provide a sound basis for making cost, schedule, and performance trade-offs. The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve the management of DOD’s joint portfolio of weapon programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JROC’s requirements review process is considering changes that would address the prioritization of requirements on a departmentwide basis. What GAO Recommends: GAO recommends that the JROC establish a mechanism to review analysis of alternatives (AOA) results earlier in the acquisition process, require higher quality resource estimates from requirements sponsors, prioritize requirements across proposed programs, and address potential redundancies during requirements reviews. The Joint Staff partially concurred with GAO’s recommendations and generally agreed with their intent, but differed with GAO on how to implement them. View [hyperlink, http://www.gao.gov/products/GAO-11-502] or key components. For more information, contact Michael J. Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Background: JROC Did Not Always Consider Trade-offs or Influence Trade-off Decisions: Military Services Did Not Consistently Provide High-Quality Resource Estimates to JROC: JROC Did Not Consistently Prioritize Requirements and Capability Gaps: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010: Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010: Appendix IV: Best Practice Criteria for Resource Estimates: Appendix V: Comments from the Department of Defense: Appendix VI: GAO Contact and Acknowledgments: Tables: Table 1: JROC Consideration of Trade-offs Among Cost, Schedule, and Performance Objectives for Seven Proposed Programs in Fiscal Year 2010: Table 2: JROC Cost Breach Reviews in Fiscal Year 2010: Table 3: Best Practice Criteria for Resource Estimates: Figures: Figure 1: JCIDS Reviews and the DOD Acquisition Process: Figure 2: AOA's Relationship to JROC Reviews: Figure 3: GAO Assessment of Resource Estimates Presented to the JROC During CDD Reviews in Fiscal Year 2010: Figure 4: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010: Abbreviations: AIAMD SOS: Army Integrated Air and Missile Defense System of Systems: AOA: Analysis of Alternatives: CAPE: Cost Assessment and Program Evaluation: CDD: Capability Development Document: CPD: Capability Production Document: CSAR-X: Combat Search and Rescue Replacement Vehicle: CVLSP: Common Vertical Lift Support Platform: DOD: Department of Defense: FCB: Functional Capabilities Board: ICD: Initial Capabilities Document: JCB: Joint Capabilities Board: JCIDS: Joint Capabilities Integration and Development System: JPALS: Joint Precision Approach and Landing System: JROC: Joint Requirements Oversight Council: KPP: Key Performance Parameter: KSA: Key System Attribute: MDAP: Major Defense Acquisition Program: USD AT&L: Under Secretary of Defense for Acquisition, Technology and Logistics: WSARA: The Weapon Systems Acquisition Reform Act of 2009: [End of section] United States Government Accountability Office: Washington, DC 20548: June 16, 2011: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Howard P. McKeon: Chairman: The Honorable Adam Smith: Ranking Member: Committee on Armed Services: House of Representatives: With the prospect of slowly growing or flat defense budgets for years to come, the Department of Defense (DOD) must get better returns on its weapon system investments and find ways to deliver more capability to the warfighter for less than it has in the past. In this environment, DOD's capacity to make effective trade-offs among cost, schedule, and performance objectives when developing and validating weapon system requirements and acquiring those systems will be important to achieving a balance between DOD's weapon system investments and resources available to it. The Weapon Systems Acquisition Reform Act of 2009 (WSARA) took several steps to encourage DOD to engage in a more robust discussion of trade- offs among cost, schedule, and performance objectives before beginning a new weapon system program. First, WSARA directed DOD's Joint Requirements Oversight Council (JROC), which validates joint military requirements, to ensure trade-offs among cost, schedule, and performance objectives are considered as part of its process for assessing and prioritizing requirements.[Footnote 1] Additionally, WSARA stated that the requirements development process must be structured to enable incremental, evolutionary, or spiral acquisition approaches,[Footnote 2] and that acquisition, budget, and cost estimating officials should be provided an opportunity to develop resource estimates and raise cost and schedule issues before performance objectives are established.[Footnote 3] Finally, WSARA amended the U.S. Code to require the JROC to seek and consider, among other things, the input of combatant commanders when assisting the Chairman of the Joint Chiefs of Staff in identifying, assessing, and approving joint military requirements; considering trade-offs; and establishing and assigning priority levels, among other areas. [Footnote 4] WSARA also directed GAO to assess the implementation of these requirements.[Footnote 5] This report addresses (1) the extent to which the JROC has considered trade-offs among cost, schedule, and performance objectives within programs; (2) the quality and effectiveness of efforts to estimate the level of resources needed to fulfill joint military requirements; and (3) the extent to which the JROC is prioritizing requirements and capability gaps. In addition, the Ike Skelton National Defense Authorization Act for Fiscal Year 2011 requires GAO to conduct a comprehensive review of the JROC's requirements validation process and report its results in 2012. [Footnote 6] To conduct our work, we focused on JROC activities in fiscal year 2010. We chose this time period to allow for any changes the JROC would implement as a result of the enactment of WSARA in May 2009. To determine the extent to which the JROC has considered cost, schedule, and performance trade-offs within programs, we reviewed the seven Capability Development Documents (CDD) submitted to the JROC in fiscal year 2010, JROC decision memos related to the CDDs, and analyses of alternatives (AOA) conducted by the military services prior to JROC reviews. We focused on CDDs because they are the first requirements documents that contain cost, schedule, and performance objectives. Additionally, we reviewed documentation from 15 JROC reviews of programs that incurred substantial cost growth after program start to determine if cost, schedule, and performance trade-offs were made. To determine the quality and effectiveness of efforts to estimate the level of resources needed to fulfill joint military requirements, we assessed the resource estimates used to support the 7 CDDs presented to the JROC for approval. To do so, we applied GAO's best practice criteria for cost estimates and reviewed supporting documentation. [Footnote 7] Each program was also provided with a copy of our assessment of their resource estimates for review and comment. To determine the extent to which the JROC prioritized requirements and capability gaps, we reviewed the 13 Initial Capabilities Documents (ICD) and 7 CDDs submitted to the JROC in fiscal year 2010, and any discussions of priorities and redundancies contained in each document. We also interviewed officials from DOD, the Joint Staff, and military service headquarters about the extent to which the JROC and its supporting bodies have addressed prioritization issues. We conducted this performance audit from June 2010 to June 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives. Appendix I contains detailed information on our scope and methodology. Background: DOD uses three interrelated processes to deliver capabilities to the U.S. military: the Joint Capabilities Integration and Development System (JCIDS), which validates gaps in joint warfighting capabilities and requirements that resolve those gaps; the Defense Acquisition System, which develops and fields weapon systems to meet these requirements; and the Planning, Programming, Budgeting and Execution process, which allocates the funding needed to develop, acquire, and field these weapon systems. The JCIDS process is overseen by the JROC, which supports the Chairman of the Joint Chiefs of Staff in advising the Secretary of Defense on joint military capability needs. The JROC is chaired by the Vice Chairman of the Joint Chiefs of Staff, and includes one senior leader from each of the military services, such as the Vice Chief of Staff of the Army or the Vice Chief of Naval Operations. The JROC has a number of statutory responsibilities related to the identification, validation, and prioritization of joint military requirements. The JROC assists the Chairman of the Joint Chiefs of Staff with a number of tasks, including (1) identifying, assessing, and approving joint military requirements; (2) establishing and assigning priority levels for joint military requirements; and (3) reviewing the estimated level of resources required to fulfill each requirement and ensuring that the resource level is consistent with the requirement's priority. The JROC also assists acquisition officials in identifying alternatives to any acquisition programs that experience significant cost growth. Since 2008, Congress has added to the JROC's statutory responsibilities and increased the number of JROC members and advisors who provide input to it. The National Defense Authorization Act for Fiscal Year 2008 amended the U.S. Code to require that the Under Secretary of Defense for Acquisition, Technology and Logistics (USD AT&L), the Under Secretary of Defense (Comptroller), and the Director of the Office of Program Analysis and Evaluation serve as advisors to the JROC on matters within their authority and expertise.[Footnote 8] In 2009, WSARA expanded the role of the JROC by directing it to assist the Chairman of the Joint Chiefs of Staff in (1) ensuring that trade- offs among cost, schedule, and performance objectives are considered for joint military requirements; and (2) establishing an objective period of time within which an initial operational capability should be delivered. WSARA also stated that the newly constituted Director of Cost Assessment and Program Evaluation (CAPE) would advise the JROC. [Footnote 9] The Ike Skelton National Defense Authorization Act for Fiscal Year 2011 allowed the Vice Chairman of the Joint Chiefs of Staff to direct senior leaders from combatant commands to serve as members of the JROC when matters related to the area of responsibility or functions of that command are under consideration. It also added the Under Secretary of Defense for Policy, the Director of Operational Test and Evaluation, and other civilian officials designated by the Secretary of Defense as advisors to the JROC on issues within their authority and expertise.[Footnote 10] The JROC is supported in the JCIDS process by two Joint Capabilities Boards (JCB) and seven Functional Capabilities Boards (FCB), each of which is chaired by a general/flag officer or civilian equivalent. JCBs and FCBs are responsible for specific Joint Capability Areas, such as Force Protection, Logistics, or Battlespace Awareness. The JCBs, FCBs, and associated FCB Working Groups review requirements documents prior to JROC reviews. The JCB also serves as the validation authority for requirements documents that are not associated with major defense acquisition programs (MDAP).[Footnote 11] In some instances, the JROC will not meet in person to approve requirements documents if there are no outstanding issues to discuss. The JROC and its supporting organizations review requirements documents related to capability gaps and the MDAPs intended to fill those gaps prior to key acquisition milestones. These requirements documents--the Initial Capabilities Documents (ICD), Capability Development Documents (CDD), and Capability Production Documents (CPD)--are submitted by capability sponsors, which are generally the military services, but can also be other DOD agencies or combatant commands. Figure 1 depicts how JCIDS reviews align with the acquisition process. Figure 1: JCIDS Reviews and the DOD Acquisition Process: [Refer to PDF for image: process illustration] Capabilities assessment: JCIDS reviews: Initial capabilities document (focus of this report). Analysis of alternatives: Acquisition milestone reviews: Materiel Development Decision. Milestone A: Validate selected alternatives. Technology development: JCIDS reviews: Capability development document (focus of this report). Acquisition milestone reviews: Milestone B: Program start. Engineering and manufacturing development: JCIDS reviews: Capability production document. Acquisition milestone reviews: Milestone C: Production start. Production: Source: GAO analysis of DOD policy. Note: Our audit primarily focused on JROC efforts to ensure programs were initiated on a sound basis, i.e., requirements validation prior to program start. [End of figure] The ICD is the first requirements document reviewed in JCIDS. It is intended to identify a specific capability gap, or set of gaps, in joint military capabilities that are determined to require a materiel solution as a result of a capabilities-based assessment. DOD policy requires that the JROC validate the ICD prior to a Materiel Development Decision, which is the formal entry point into the acquisition process. The ICD does not contain specific cost, schedule, or performance objectives. Once the JROC validates an ICD, the Milestone Decision Authority, working with appropriate stakeholders, shall determine whether to proceed to a Materiel Development Decision.[Footnote 12] After the Materiel Development Decision, the capability sponsor initiates an AOA to consider alternative solutions to fulfilling the capability need described in an ICD, and possible trade-offs among cost, schedule, and performance for each alternative are considered. The CDD is the second requirements document reviewed in JCIDS. It can address capability gaps presented in one or more ICDs. The CDD is intended to define a proposed program's Key Performance Parameters (KPP), Key System Attributes (KSA), and other performance attributes. KPPs are the system characteristics that the CDD sponsor considers most critical to delivering that military capability, while KSAs are system attributes the CDD sponsor considers essential for an effective military capability, but a lower priority than the KPPs. DOD policy calls for the JROC to validate the CDD to inform the Milestone B decision, which marks the official start of an acquisition program and entry into the engineering and manufacturing development phase. The CDD is the first requirements document that contains cost, schedule, and performance objectives. The CPD is the third and final requirements document reviewed in JCIDS. It is intended to refine the KPPs, KSAs, and performance attributes validated in the CDD. DOD policy calls for the JROC to validate the CPD to inform the Milestone C decision, which marks a program's entry into production. Appendix II identifies the ICDs, CDDs, and CPDs reviewed by the JCB or JROC in fiscal year 2010. In addition to JCIDS reviews, the JROC reviews MDAP requirements after a program experiences cost growth beyond JROC-specified amounts, triggering what is called a tripwire review. The JROC's tripwire policy directs military services to brief the JROC when unit cost increases by 10 percent over the current baseline or 25 percent over the original baseline. DOD officials explained this allows the JROC to consider relaxing, deferring, or deleting a program's KPPs if they are found to be unachievable or of lesser priority than reducing program cost. The JROC also participates in what are commonly known as critical Nunn-McCurdy[Footnote 13] reviews by providing an assessment of whether the program is essential to national security. A critical Nunn-McCurdy breach occurs when the program acquisition unit cost or the average procurement unit cost increases by at least 25 percent over the current baseline estimate or at least 50 percent over the original baseline estimate. Appendix III lists the JROC's 15 fiscal year 2010 Nunn-McCurdy and tripwire reviews. JROC Did Not Always Consider Trade-offs or Influence Trade-off Decisions: The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010, and provided input to the military services on the cost, schedule, and performance objectives for two of the seven programs. The JROC's requirements review was the final step in a long requirements vetting process, with most trade-offs being made by the military services earlier in the process. Key stakeholders from the offices of the Under Secretary of Defense (Comptroller), USD AT&L, Director of CAPE, and the combatant commands were all satisfied with their opportunities to provide input to the JROC; but they provided limited input on trade-offs among cost, schedule, and performance objectives, and used other means to influence trade-offs. Perhaps most importantly, none of the JROC's requirements reviews align with the AOA, which is where the military services reported making the most significant trade-offs. As a result, a program can spend significant time in technology development before the JROC gets to formally weigh in on these trade-offs through the JCIDS process. The JROC also reviews MDAP requirements after a program enters development and experiences substantial cost growth. DOD and the JROC stated that requirements were not the primary causes of cost growth for the 15 programs reviewed for this purpose in fiscal year 2010 and the JROC did not change any KPPs to mitigate the reported cost growth. JROC Did Not Always Consider Trade-offs when Validating Requirements for Proposed Programs: The JROC considered trade-offs made by the military services before validating requirements for four of the seven proposed programs it reviewed in fiscal year 2010. On three programs, the JROC did not receive information on the potential cost and schedule implications of each of the alternatives considered. Table 1 summarizes the JROC's consideration of cost, schedule, and performance objectives for the seven proposed MDAPs it reviewed in fiscal year 2010. Table 1: JROC Consideration of Trade-offs Among Cost, Schedule, and Performance Objectives for Seven Proposed Programs in Fiscal Year 2010: Proposed program: Common Vertical Lift Support Platform (CVLSP); Capability description: The CVLSP program is expected to provide a helicopter that will support nuclear security and passenger transport missions, and improve carrying capacity, range, speed, survivability, and battlespace awareness among other capabilities; Did JROC consider cost, schedule, and performance trade-offs? Yes. The JROC established rapid fielding as the priority and instructed the Air Force to discuss trade-offs with the acquisition and testing communities if the system's requirements were adversely affecting its rapid fielding. Proposed program: HH-60 Recapitalization; Capability description: The HH-60 Recapitalization program is expected to provide a helicopter that will support personnel recovery missions, and be capable of operating day or night, in adverse weather, and amongst a variety of threats; Did JROC consider cost, schedule, and performance trade-offs? Yes. The JROC approved the decision to change combat radius, survivability, cabin space, payload, and airspeed requirements to decrease cost. Proposed program: Army Integrated Air and Missile Defense System of Systems (AIAMD SOS), Increment 2; Capability description: The AIAMD SOS, Increment 2 program is expected to provide an air and missile defense system that will integrate sensors and weapons across an integrated fire control network; Did JROC consider cost, schedule, and performance trade-offs? Yes. The JROC approved the decision to accept greater cost and schedule risk in order to meet performance objectives. Proposed program: Ground Soldier System, Increment 1[A]; Capability description: The Ground Soldier System, Increment 1 program is expected to provide a communications system that will improve ground combat leaders' battle command and situational awareness capabilities; Did JROC consider cost, schedule, and performance trade-offs? Yes. The JROC emphasized the importance of weight and mobility and directed the Army to seek JROC approval if the operating weight exceeded a specific threshold. Proposed program: Joint Precision Approach and Landing System (JPALS), Increment 2; Capability description: The JPALS, Increment 2 program is expected to provide avionics systems and mobile and fixed ground stations that will increase the efficiency of approach and landing operations; Did JROC consider cost, schedule, and performance trade-offs? No. The JROC received brief descriptions of seven potential solutions, but the cost and schedule implications of each alternative were not identified. Proposed program: P-8A, Increment 3[B]; Capability description: The P-8A, Increment 3 program is expected to provide an antisurface warfare weapon and improved communications capabilities; Did JROC consider cost, schedule, and performance trade-offs? No. The JROC received brief descriptions of seven analyses supporting the P-8A program, but they did not explicitly discuss Increment 3 cost, schedule, and performance objectives. Proposed program: Ship to Shore Connector; Capability description: The Ship to Shore Connector program is expected to provide an air cushion vehicle for transporting vehicles, cargo, and personnel from ship to shore, and will replace the current Landing Craft Air Cushion; Did JROC consider cost, schedule, and performance trade-offs? No. The JROC received brief descriptions of three potential solutions, but the cost and schedule implications of each alternative were not identified. Source: GAO analysis of JCIDS and acquisition documents. [A] The Ground Soldier System, Increment 1 program has been renamed Nett Warrior. [B] The proposed P-8A, Increment 3 program has not yet reached Milestone A, but at the direction of the Joint Staff, its requirements were approved in the same document as the proposed P-8A, Increment 2 program. [End of table] The JROC's review of the CDD for a proposed program is the final step in a long requirements vetting process, and DOD officials reported that trade-offs typically occur earlier in the process. Each military service conducts its own internal requirements reviews for its proposed programs, which are used to refine requirements documents before they are submitted into JCIDS. Military service officials reported that they make significant trade-offs during these internal reviews, and that KPPs and technical requirements rarely change after requirements documents are submitted into JCIDS because extensive analysis has already been conducted. For the seven proposed MDAPs we reviewed, the military services generally submitted requirements to the JROC that would be fully funded, provide initial capability within 6 years, utilize critical technologies that were nearing maturity, and be acquired using an incremental approach. These characteristics are consistent with provisions in the Weapon Systems Acquisition Reform Act (WSARA) related to how the requirements process should be structured[Footnote 14] and aspects of GAO's best practices for weapon system acquisitions.[Footnote 15] Two of the proposed program requirements presented to the JROC included major trade-offs among cost, schedule, and performance objectives and revisions to their acquisition approaches that had been made after predecessor programs were canceled over affordability concerns. The Air Force initiated the HH-60 Recapitalization program after the Combat Search and Rescue Replacement Vehicle (CSAR-X) program was canceled, and the HH-60 Recapitalization program is expected to decrease cost by changing cabin space, velocity, and range from the CSAR-X requirements. In 2007, the Army, with input from a Functional Capabilities Board, decided to use an incremental acquisition approach for the Ground Soldier System in order to reduce costs, meet schedule demands, and avoid some of the mistakes made during the Land Warrior program, which was canceled because of funding and cost issues. Key Stakeholders Provided Limited Input into JCIDS, but Use Other Means to Influence Trade-offs: The JROC received limited input on trade-offs among cost, schedule, and performance objectives from key stakeholders when validating requirements for the seven proposed MDAPs we reviewed from fiscal year 2010. Both WSARA and the National Defense Authorization Act for Fiscal Year 2008 directed the JROC to consult with the Under Secretary of Defense (Comptroller), the USD AT&L, and the Director of CAPE. [Footnote 16] Additionally, WSARA instructed the JROC to consult with the combatant commands. Officials from these organizations reported that they had ample opportunity to participate in JROC requirements reviews, and Joint Staff officials said efforts to involve these stakeholders preceded WSARA. However, officials from the offices of the Under Secretary of Defense (Comptroller), USD AT&L, and the Director of CAPE also reported that the acquisition and budgeting/funding processes are the primary mechanisms through which they influence programs, rather than JCIDS. For example, CAPE oversees AOAs for MDAPs and has an opportunity to provide input and guidance on AOA considerations. Further, the combatant commands reported that they most often submit prioritized lists of capability gaps directly to the Chairman of the Joint Chiefs of Staff as part of the resource allocation process, which is separate from JCIDS.[Footnote 17] Nonetheless, joint stakeholders did provide some significant input during the JROC's reviews of the seven proposed programs in fiscal year 2010. For example, the Army more fully defined a Ground Soldier System, Increment 1 KPP in response to input from DOD's Joint Interoperability Test Command, and in another instance, the Army added a KSA to the AIAMD SOS, Increment 2 CDD due to input from the office of the USD AT&L, the Defense Information Systems Agency, and the Joint Staff. Neither of these changes involved trade-offs among cost, schedule, and performance objectives. JCIDS Reviews Are Not Aligned with the Most Significant Trade-off Decisions: The JROC does not formally review the trade-off decisions made as a result of an AOA until a proposed program's CDD enters the JCIDS process. According to DOD officials, the most significant trade-offs are made by the military services between ICD and CDD reviews during the AOA, which is intended to compare the operational effectiveness, cost, and risks of a number of alternative potential solutions. For example, during the CVLSP AOA, the Air Force decided to decrease troop transport capacity in order to reduce cost. Alternatively, during the AIAMD SOS AOA, the Army decided to pursue the most costly option reviewed because it provided greater capability. A significant amount of time and resources can be expended before the JROC gets to weigh in on these trade-offs during CDD reviews. For example, the JROC reviewed the AOA summary for JPALS, Increment 2, 4 years after the conclusion of the AOA. During the time between the AOA and the CDD review, the technology intended to enable the chosen alternative is developed. Figure 2 shows the AOA's relationship to both the requirements and acquisition processes. Figure 2: AOA's Relationship to JROC Reviews: [Refer to PDF for image: illustration] ICD developed: No cost, schedule, performance objectives. (Requirements development) JROC review: Too early for trade-offs. Analysis of alternatives: Most significant trade-offs made. (Acquisition activity) Milestone A. (Acquisition milestone) Technology development: Requirements refined and vetted. (Requirements development) CDD developed: Cost, schedule, performance objectives identified. (Acquisition activity) JROC review: Requirements rarely change. Milestone B. (Acquisition milestone) Source: GAO analysis of DOD policy. [End of figure] Joint Staff officials have stated that establishing a JROC review of the AOA would allow it to provide military advice on trade-offs and the proposed materiel solution before Milestone A, and an ongoing Joint Staff review of JCIDS is considering an increased role for the JROC at this point.[Footnote 18] According to the Joint Staff, increased JROC engagement at these early stages of the acquisition process is warranted to align it with other elements of recent acquisition reforms. For example, WSARA emphasized that the AOA should fully consider possible trade-offs among cost, schedule, and performance objectives for each alternative considered,[Footnote 19] and in September 2010, USD AT&L issued a memorandum that emphasized the need for trade-offs from a program's inception.[Footnote 20] The memorandum also dictated that affordability targets shall be established at the conclusion of the AOA and that these targets will be treated like KPPs, even though they will be set and managed by the acquisition, not requirements, community. JROC Did Not Change KPPs when Programs Incurred Substantial Cost Growth: The JROC did not change any KPPs during 15 reviews of programs that reported substantial cost growth in fiscal year 2010. According to the Joint Staff, by holding requirements firm and accepting increased cost and schedule delays, the JROC essentially traded cost and possibly schedule for performance. In fiscal year 2010, the JROC reviewed six programs after they experienced a critical Nunn-McCurdy breach and nine programs as part of the tripwire process. During all 15 reviews, DOD and the JROC stated that requirements were not the primary causes of cost growth. For all six programs that experienced a critical Nunn- McCurdy cost breach, the JROC validated the system's capabilities as being essential to national security and did not make any changes to their KPPs. For all nine programs that were approaching Nunn-McCurdy thresholds, the JROC did not identify opportunities to mitigate cost growth by modifying requirements. Most of these programs were in production in fiscal year 2010, and changing requirements at this late stage might not have mitigated the reported cost growth. When the JROC reviewed the Family of Advanced Beyond Line-of-Sight Terminals program, which was still in development, it concluded that the program's requirements could not be met in an affordable manner. The JROC did not immediately defer any of the program's requirements, but instead requested that USD AT&L identify potential alternatives for the program, including reviewing whether adjustments to performance requirements would be appropriate. Military Services Did Not Consistently Provide High-Quality Resource Estimates to JROC: The military services did not consistently provide high-quality resource estimates to the JROC to support its review of requirements for 7 proposed programs in fiscal year 2010. We found the estimates presented to the JROC were often unreliable when assessed against best practices criteria. The type of resource estimates the military services presented to the JROC varied from ones that had been validated by the military services' cost analysis agencies to less rigorous rough-orders-of-magnitude estimates. In most cases, the military services had not effectively conducted uncertainty and sensitivity analyses, which establish confidence levels for resource estimates, based on the knowledge available, and examine the effects of changing assumptions and ground rules. Lacking risk and uncertainty analysis, the JROC cannot evaluate the range of resources that might be necessary to cover increased costs resulting from unexpected design complexity, technology uncertainty, and other issues. The lack of this information affects the JROC's efforts to ensure that programs are fully funded and its ability to consider the resource implications of cost, schedule, and performance trade-offs. JROC Received Resource Estimates That Did Not Meet Best Practices: The JROC first receives resource estimates for proposed programs when it reviews CDDs, and when we reviewed the CDD resource estimates presented to the JROC in fiscal year 2010, we found that they were generally unreliable when assessed against our best practices criteria.[Footnote 21] While most of the resource estimates substantially met our criteria for a comprehensive resource estimate, they generally were not very accurate, credible, or well-documented. Appendix IV includes a list of the best practices against which we assessed these resource estimates. The type of resource estimates the military services presented to the JROC varied from ones that had been validated by the military services' cost analysis agencies to less rigorous rough-orders-of- magnitude estimates.[Footnote 22] According to Joint Staff officials, military services can initiate CDD reviews at any point in the acquisition process prior to program start, even if good resource estimates are not available. For example, the JROC validated the P-8A, Increment 3 CDD more than 2 years before the program was expected to start, before an AOA had been completed, and with a rough-order-of- magnitude estimate. Joint Staff officials reported that they depend on CAPE to review the quality of resource estimates during the JCIDS process, but CAPE cost assessment officials told us that they rarely participate in JCIDS reviews. Regardless of the type of resource estimate, uncertainty and sensitivity analysis can establish confidence levels for resource estimates, based on the knowledge available at the time, and examine the effects of changing assumptions and ground rules, including those related to trade-offs among cost, schedule, and performance objectives. The military services sponsoring the requirements generally did not effectively meet best practices for uncertainty and sensitivity analyses using the knowledge they had available to them for any of the seven resource estimates we reviewed. Figure 3 summarizes our assessment of the resource estimates presented to the JROC against our best practices criteria. Figure 3: GAO Assessment of Resource Estimates Presented to the JROC During CDD Reviews in Fiscal Year 2010: [Refer to PDF for image: illustrated table] CDDs: Ship to Shore Connector; Type of resource estimate: Naval Center for Cost Analysis validated estimate; Characteristics of a reliable resource estimate: Comprehensive: Substantially met – evidence satisfied a large portion of the criterion; Accurate: Substantially met – evidence satisfied a large portion of the criterion; Credible: Substantially met – evidence satisfied a large portion of the criterion; Well-documented: Partially met – evidence satisfied about half of the criterion. CDDs: JPALS, Increment 2; Type of resource estimate: Air Force cost estimate; Characteristics of a reliable resource estimate: Comprehensive: Substantially met – evidence satisfied a large portion of the criterion; Accurate: Substantially met – evidence satisfied a large portion of the criterion; Credible: Minimally met – evidence satisfied a small portion of the criterion; Well-documented: Substantially met – evidence satisfied a large portion of the criterion. CDDs: CVLSP; Type of resource estimate: Air Force Cost Analysis Agency validated estimate[A]; Characteristics of a reliable resource estimate: Comprehensive: Substantially met – evidence satisfied a large portion of the criterion; Accurate: Partially met – evidence satisfied about half of the criterion; Credible: Partially met – evidence satisfied about half of the criterion; Well-documented: Partially met – evidence satisfied about half of the criterion. CDDs: AIAMD SOS, Increment 2; Type of resource estimate: Army cost position; Characteristics of a reliable resource estimate: Comprehensive: Substantially met – evidence satisfied a large portion of the criterion; Accurate: Partially met – evidence satisfied about half of the criterion; Credible: Partially met – evidence satisfied about half of the criterion; Well-documented: Partially met – evidence satisfied about half of the criterion. CDDs: Ground Soldier System, Increment 1; Type of resource estimate: Program office estimate; Characteristics of a reliable resource estimate: Comprehensive: Substantially met – evidence satisfied a large portion of the criterion; Accurate: Partially met – evidence satisfied about half of the criterion; Credible: Minimally met – evidence satisfied a small portion of the criterion; Well-documented: Partially met – evidence satisfied about half of the criterion. CDDs: HH-60 Recapitalization; Type of resource estimate: Rough-order-of-magnitude estimate; Characteristics of a reliable resource estimate: Comprehensive: Minimally met – evidence satisfied a small portion of the criterion; Accurate: Minimally met – evidence satisfied a small portion of the criterion; Credible: Not met – no evidence satisfied any of the criterion; Well-documented: Minimally met – evidence satisfied a small portion of the criterion. CDDs: P-8, Increment 3; Type of resource estimate: Rough-order-of-magnitude estimate; Characteristics of a reliable resource estimate: Comprehensive: Minimally met – evidence satisfied a small portion of the criterion; Accurate: Not met – no evidence satisfied any of the criterion; Credible: Not met – no evidence satisfied any of the criterion; Well-documented: Not met – no evidence satisfied any of the criterion. Source: GAO analysis of DOD data. [A] Air Force Cost Analysis Agency found the CVLSP estimate adequate for an AOA, but not for budgeting purposes. [End of figure] Most Resource Estimates Substantially Met Comprehensiveness Criteria: Five of the seven CDD resource estimates substantially met our criteria for a comprehensive resource estimate. The resource estimates generally completely defined their respective programs, and included most, if not all, life-cycle costs. The Ship to Shore Connector, CVLSP, and JPALS, Increment 2 resource estimates also effectively documented all cost-influencing ground rules and assumptions, although the other resource estimates did not. Additionally, only the Ship to Shore Connector's work breakdown structure effectively met our criteria, which require that work breakdown structures are product- oriented and at an appropriate level of detail. If a resource estimate does not specifically break out common costs, such as government- furnished equipment costs, or does not include an associated work breakdown structure dictionary, cost estimators cannot ensure that the estimate includes all relevant costs. The HH-60 Recapitalization and P-8A, Increment 3 resource estimates did not effectively meet any of our best practices for a comprehensive resource estimate. Unless resource estimates account for all costs, they cannot enhance decision making by allowing for design trade-off studies to be evaluated on a total cost, technical, and performance basis. Additionally, unless ground rules and assumptions are clearly documented, the resource estimate will not have a basis for resolving areas of potential risk. Most Resource Estimates Did Not Substantially Meet Accuracy Criteria: Only two of the seven CDD resource estimates substantially met our criteria for an accurate resource estimate, while three partially met the criteria, and two did not meet or minimally met the criteria. We found that the Ship to Shore Connector, CVLSP, AIAMD SOS, Increment 2, and the Ground Soldier System, Increment 1 resource estimates contained few, if any, minor mistakes, and that the Ship to Shore Connector, CVLSP, and JPALS, Increment 2 resource estimates were appropriately adjusted for inflation. Additionally, we found that the Ship to Shore Connector and JPALS, Increment 2 resource estimates were based on historical records of actual experiences from other comparable programs. However, we generally found that the resource estimates were not consistent with our best practices. Accurate resource estimates are rooted in historical data, which provide cost estimators with insight into actual costs of similar programs, and can be used to challenge optimistic assumptions and bring more realism to a resource estimate. Unless an estimate is based on an assessment of the most likely costs, and reflects the degree of uncertainty given all of the risks considered, management will not be able to make well-informed decisions. Most Resource Estimates Were Not Credible: Four of the seven CDD resource estimates did not meet or minimally met our criteria for a credible resource estimate, and only the Ship to Shore Connector resource estimate substantially met the criteria. The Ship to Shore Connector and AIAMD SOS, Increment 2 resource estimates included sensitivity analyses that identified a range of possible costs based on varying assumptions, parameters, and data inputs, but none of the other resource estimates included this analysis. As a best practice, sensitivity analysis should be included in all resource estimates because it examines the effects of changing assumptions and ground rules. Since uncertainty cannot be avoided, it is necessary to identify the cost elements that represent the most risk and, if possible, cost estimators should quantify that risk. When an agency fails to conduct sensitivity analysis to identify the effect of uncertainties associated with different assumptions, this increases the chance that decisions will be made without a clear understanding of the impact on cost. Additionally, only the Ship to Shore Connector resource estimate effectively met our best practices for risk and uncertainty analysis. For management to make good decisions, the program estimate must reflect the degree of uncertainty so that a level of confidence can be given about the estimate. An estimate without risk and uncertainty analysis is unrealistic because it does not assess the variability in the resource estimate from effects such as schedules slipping, missions changing, and proposed solutions not meeting users' needs. Lacking risk and uncertainty analysis, management cannot determine a defensible level of contingency reserves that is necessary to cover increased costs resulting from unexpected design complexity, technology uncertainty, and other issues. Further, none of the planned programs effectively met our criteria for an independent cost estimate when they were reviewed by the JROC. An independent cost estimate is considered one of the best and most reliable resource estimate validation methods because it provides an independent view of expected program costs that tests the program office and service estimates for reasonableness. Without an independent cost estimate, decision makers lack insight into a program's potential costs because these estimates frequently use different methods and are less burdened with organizational bias. Moreover, independent cost estimates tend to incorporate adequate risk, and therefore tend to be more conservative by forecasting higher costs than the program office. A program estimate that has not been reconciled with an independent cost estimate has an increased risk of proceeding underfunded because an independent cost estimate provides an objective and unbiased assessment of whether the program estimate can be achieved. Alternatively, programs can reinforce the credibility of their resource estimates through cross-checking, which determines whether alternative cost estimating methods produce similar results. However, only the Ship to Shore Connector resource estimate effectively met our best practices for cross-checking. Most Resource Estimates Were Not Well-documented: Only the JPALS, Increment 2 resource estimate substantially met our criteria for a well-documented resource estimate, while four of the seven CDD resource estimates partially met our criteria, and two of the resource estimates did not meet or minimally met the criteria. The JPALS, Increment 2 and CVLSP resource estimates sufficiently described the calculations performed and estimating methodologies used to derive each program element's cost. Additionally, the JPALS, Increment 2, Ship to Shore Connector, and AIAMD SOS, Increment 2 documentation clearly discusses the technical baseline description, and the data in the technical baseline are consistent with the resource estimate. However, none of the documents effectively described how the resource estimates were developed in a manner that a cost analyst unfamiliar with the program could understand what was done and replicate it. We generally found that the resource estimates were not consistent with our best practices for a well-documented resource estimate. Documentation is essential for validating and defending a resource estimate. Without a well-documented resource estimate, a convincing argument of an estimate's validity cannot be presented, and decision makers' questions cannot be effectively answered. Poorly documented resource estimates cannot explain the rationale of the methodology or the calculations underlying the cost elements. Further, a well- documented resource estimate is essential for an effective independent review to ensure that the resource estimate is valid and credible. Unless the estimate is fully documented, it will not support reconciliation with an independent cost estimate, hindering understanding of cost elements and their differences. Military Services Generally Presented Resource Estimates That Were Fully Funded to JROC: The JROC required the military services to show that the proposed programs were fully funded to the resource estimates presented by the military services before it validated requirements for five of the seven proposed MDAPs we reviewed from fiscal year 2010; the two other proposed MDAPs were funded at more than 97 and 99 percent respectively.[Footnote 23] However, we found that these resource estimates were generally unreliable, which undermined the JROC's efforts. In 2007, the JROC issued guidance instructing the military services to commit to funding the requirements that the JROC validates. The guidance emphasized the need for full funding in an effort to facilitate sound fiscal and risk decisions.[Footnote 24] However, the JROC does not explicitly consider a requirement's affordability in a broader context during JCIDS reviews. DOD funding plans are captured in the future-years defense program, which presents resource information for the current year and the following 4 years. The future-years defense program is updated twice per year to reflect the military services' input and the budget the President submits to Congress. Statute and DOD acquisition policy also require programs to be fully funded through the period covered by the future-years defense program.[Footnote 25] One of the seven proposed MDAPs we reviewed from fiscal year 2010 included a funding shortfall when its requirements were being reviewed through JCIDS, but its CDD was not approved until the shortfall had been addressed. Specifically, when the JCB reviewed the CVLSP CDD, the funding plan included a $1.3 billion shortfall through fiscal year 2015. The JCB chairman directed the Air Force to modify the program's funding plan before proceeding to the JROC review. When the Air Force briefed the JROC on the CVLSP CDD approximately 8 months later, it presented a funding plan that fully funded the program through the future-years defense program time frame. The revised funding plan also included more money for the program beyond the future-years defense program time frame, and the total program cost increased from $14.2 billion to $15.2 billion. Despite JROC efforts to ensure programs are fully funded, the military services retain primary control over their budgets, and ultimately, JROC decisions are influential but not binding. When the JCB reviewed the JPALS, Increment 2 CDD, it requested clarification on the Air Force's funding plan, and emphasized the need for full funding prior to program start. The funding plan presented to the JCB included a $77.7 million shortfall through fiscal year 2015, and the Air Force had cut JPALS funding in the past. Following the JCB review, the JROC issued a decision memorandum that documented the Air Force's commitment to fully funding JPALS, Increment 2. However, in fiscal years 2011 and 2012, the Air Force only funded approximately 30 percent of the resource estimate presented to the JCB. JROC Did Not Consistently Prioritize Requirements and Capability Gaps: The JROC does not currently prioritize requirements, consider redundancies across proposed programs, or prioritize and analyze capability gaps in a consistent manner. As a result, the Joint Staff is missing an opportunity to improve military service and departmentwide portfolio management efforts. A portfolio management approach to weapon system investments would involve taking a disciplined, integrated approach to prioritizing needs and allocating resources in order to eliminate redundancies, gain efficiencies, and achieve a balanced mix of executable programs. According to Army, Air Force, and Navy officials, having a better understanding of warfighter priorities from the JROC would be useful to inform both portfolio management efforts and service budgets. A DOD review team examining the JCIDS process is considering changes that would address the prioritization of requirements. During its review of the capability gaps presented in 12 ICDs in fiscal year 2010, the JROC did receive some information on priorities and potential redundancies; however, the sponsors presented this information in an inconsistent manner, making it difficult for the JROC to assess the relative priority of capability gaps across different ICDs. JROC Does Not Prioritize Requirements or Consider Redundancies Across Proposed Programs: Under the current JCIDS process, the JROC does not prioritize requirements or consider redundancies across proposed programs during CDD reviews. In the National Defense Authorization Act for Fiscal Year 2008, Congress amended the U.S. Code to direct the JROC to help assign priority levels for joint military requirements and ensure that resource levels associated with those requirements are consistent with the level of priority.[Footnote 26] The House Armed Services Committee report accompanying the authorization act stated that clear JROC priorities and budget guidance would allow for joint decision making, as opposed to service-centric budget considerations.[Footnote 27] In addition, we have previously recommended that DOD develop an analytic approach within JCIDS to better prioritize and balance the capability needs of the military services, combatant commands, and other defense components.[Footnote 28] According to the Joint Staff and military service officials, prioritization across programs still primarily occurs through the Planning, Programming, Budgeting and Execution process, which is the responsibility of the military services and the Office of the Under Secretary of Defense (Comptroller). The JCIDS manual does not currently require an analysis of potential redundancies during CDD reviews. In our recently issued report on government duplication, we noted that service-driven requirements and funding processes continue to hinder integration and efficiency and contribute to unnecessary duplication in addressing warfighter needs. [Footnote 29] We have also previously reported that ineffective collaboration precluded opportunities for commonality in unmanned aircraft systems.[Footnote 30] In fiscal year 2010, the JROC met to consider joint efficiencies between two such systems: the Navy's Broad Area Maritime Surveillance system and the Air Force's Global Hawk system. The JROC requested that the Navy and Air Force ensure that a common component was interoperable between the two systems, and that the Air Force consider an all-weather capability developed by the Navy. The JROC has also supported joint development efforts for these programs and requested annual status updates. According to Broad Area Maritime Surveillance program officials, the Air Force and Navy programs are investigating commonality opportunities, including sense- and-avoid capabilities, a consolidated maintenance hub, and basing options for both systems. The JROC did not meet to consider any other joint efficiencies across military services in fiscal year 2010. The Joint Staff has acknowledged that the JROC should play a larger role in prioritizing needs and addressing redundancies. In July 2010, the Vice Chairman of the Joint Chiefs of Staff initiated a review of the JCIDS process. One of the goals of the review team was to develop metrics and criteria to ensure the JCIDS process has the ability to rank or prioritize needs. The review team's charter states that these metrics must enable more structured reviews of portfolio gaps and redundancies. According to the Joint Staff, the review team is considering a number of recommendations including asking the JROC to prioritize requirements based on the urgency and significance of the need. This list of priorities could be used to inform military service budgets. Joint Staff officials have also stated that redundancies may be addressed more directly in the future as part of an enhanced portfolio management effort. Lack of JROC Prioritization Results in Missed Opportunities to Manage Portfolios Better: We have previously reported that DOD has not taken a portfolio management approach to weapon system investments, which would involve taking a disciplined, integrated approach to prioritizing needs and allocating resources in order to eliminate redundancies, gain efficiencies, and achieve a balanced mix of executable programs. [Footnote 31] In September 2010, USD AT&L issued guidance intended to increase efficiencies and eliminate redundancies, and it presented the Army's portfolio management activities as an example to emulate. [Footnote 32] The Army uses capability portfolio reviews of capability gaps and proposed and existing programs to revalidate, modify, or terminate requirements and ensure the proper allocation of funds between them. The Army has established 17 portfolios, including aviation, air and missile defense, and combat vehicle modernization. An Army official involved in the portfolio reviews said that he has requested on several occasions for the Joint Staff to prioritize warfighter needs; however, the JROC has not done so. Instead, the Army relies on its own prioritization information during the portfolio reviews to help determine the capability areas where it is willing to assume risk. Air Force and Navy officials have also stated that they could benefit from JROC prioritization of requirements, and that this information would be useful in order to better allocate resources during their budget formulation activities. Sponsors Do Not Prioritize Capability Gaps or Analyze Potential Redundancies in a Consistent Manner: The JROC has required that capability sponsors prioritize capability gaps and identify redundancies when developing ICDs,[Footnote 33] and capability sponsors generally complied with these requirements in the 12 validated ICDs we reviewed from fiscal year 2010.[Footnote 34] However, the sponsors presented this information in an inconsistent manner, making it difficult for the JROC and the military services to assess priorities and redundancies across ICDs or use this information to inform resource allocation decisions. For example, the Electronic Health Record ICD prioritized its gaps in numerical order from 1 to 10,[Footnote 35] but the Command and Control On-The-Move ICD labeled half its gaps medium priority and the other half high priority. [Footnote 36] The JCIDS operation manual provides limited guidance on how capability sponsors should prioritize the gaps, stating only that the prioritization should be based on the potential for operational risk associated with the shortfalls. The JCIDS manual also directs capability sponsors to identify redundancies and assess whether the overlap is operationally acceptable or whether it should be evaluated as part of the trade-offs to satisfy capability gaps. Three of the 12 validated ICDs we reviewed from fiscal year 2010 did not address redundancies. Furthermore, only one of these ICDs presented to the JROC in fiscal year 2010 included an evaluation of the overlaps. The JROC did not address these omissions when it validated the documents. Conclusions: In the last several years, Congress has passed legislation to give the JROC a greater role in prioritizing military requirements and shaping sound acquisition programs by encouraging cost, schedule, and performance trade-offs. Taken together, these steps have the potential to improve the affordability and execution of DOD's portfolio of major defense acquisition programs. However, the JROC has largely left prioritization and trade-off decisions to the military services, despite having a unique, joint perspective, which would allow it to look across the entire department to identify efficiencies and potential redundancies. To more effectively leverage its unique perspective, the JROC would have to change the way it views its role, more regularly engage the acquisition community in trade-off discussions at early acquisition milestones, and more effectively scrutinize the quality of the resource estimates presented by the military services. Until it does so, the JROC will only be a marginal player in DOD's efforts to align the department's available resources with its warfighting requirements. Recommendations for Executive Action: To enhance the JROC's role in DOD-wide efforts to deliver better value to the taxpayer and warfighter, we recommend that the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, take the following five actions: * Establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority. * Require that capability sponsors present resource estimates that have been reviewed by a military service's cost analysis organization to ensure best practices are being followed. * Require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program's current level of knowledge. * Assign priority levels to the CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes. * Modify the JCIDS operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements. Agency Comments and Our Evaluation: The Joint Staff provided us written comments on a draft of this report. The comments are reprinted in appendix V. The Joint Staff also provided technical comments, which we addressed in the report, as appropriate. In its comments, the Joint Staff partially concurred with all five of our recommendations, generally agreeing that there is a need to take action to address the issues we raised, but differing in terms of the specific actions that should be taken. The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff, as chairman of the JROC, establish a mechanism to review the final AOA report prior to Milestone A to ensure that trade-offs have been considered and to provide military advice on these trade-offs and the proposed materiel solution to the Milestone Decision Authority. The Joint Staff noted that its ongoing review of JCIDS will include a recommendation that AOA results be briefed to FCBs. However, the FCB will only elevate these briefings to the JCB or JROC on an exception basis. The Joint Staff explained that this approach would allow the JROC to provide more informed advice to a Milestone Decision Authority without adding another round of staffing, an additional JCIDS document, or an official validation of AOA results. We agree that the Joint Staff should seek to implement this recommendation in the most efficient and effective way possible; however, given our finding that the most significant trade-off decisions are made as a result of an AOA, we continue to believe that the results should be reviewed by the JROC. The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff require that capability sponsors present resource estimates that have been reviewed by a military service's cost analysis organization to ensure best practices are being followed. The Joint Staff stated that program office cost estimates are compared to independent cost estimates during CDD reviews. However, none of the seven CDD cost estimates we reviewed effectively met our criteria for an independent cost estimate. As a result, we believe that the Joint Staff needs to take additional action to ensure that resource estimates presented by capability sponsors have been reviewed by a military service's cost analysis organization. The Joint Staff also stated that its ongoing review of JCIDS will examine how to highlight this area during CDD reviews. The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program's current level of knowledge. The Joint Staff stated that our recommendation needs further study to understand the expected outcomes and the required authorities for the JROC, and its ongoing review of JCIDS will examine how to highlight this area. We believe that the JROC cannot fully consider trade-offs or the affordability of a proposed program unless it receives information on the risk and uncertainty associated with resource estimates; it does not need additional authority to require capability sponsors to present the results of this type of analysis before it approves proposed requirements. The Joint Staff also noted that the Director, CAPE, has cost analysis responsibilities for resource estimates. CAPE cost assessment officials reported that they rarely participated in JCIDS reviews. As a result, the JROC may have to be more proactive in reaching out to CAPE to help it understand the risk and uncertainty associated with the resource estimates it receives. The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff assign priority levels to CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller) and the military services to be used for resource allocation purposes. The Joint Staff agreed that the identification of joint priorities could enhance a number of processes, including program and budget reviews. It noted that its ongoing review of JCIDS will recommend a prioritization framework through which CDDs will inherit priority levels based on the requirements and capability gaps identified in ICDs or Joint Urgent Operational Needs. However, the Joint Staff argued against prioritizing based on CDDs directly because it would provide less flexibility. We believe that the proposed approach could be effective if the Joint Staff addresses the inconsistencies we found in the way ICDs prioritize gaps. In addition, we continue to believe that the prioritization framework should facilitate an examination of priorities across CDDs. The Joint Staff partially concurred with our recommendation that the Vice Chairman of the Joint Chiefs of Staff modify the JCIDS operations manual to require that CDDs discuss potential redundancies across proposed and existing programs, and address these redundancies when validating requirements. The Joint Staff stated that its ongoing review of JCIDS will address this issue by establishing unique requirements as a higher priority than unnecessarily redundant requirements, and by establishing a post-AOA review, which could also be used to identify unnecessary redundancies. The Joint Staff did not address whether it would update the JCIDS operations manual as recommended and stated that reviewing assessments of redundancies in CDDs would be late in the JCIDS process. We believe that potential redundancies should be discussed at multiple points, including during CDD reviews, because we found that several years can pass between the conclusion of an AOA and this review. During that time, new redundancy issues could emerge. We are sending copies of this report to the Secretary of Defense; the Chairman and Vice Chairman of the Joint Chiefs of Staff; the Secretaries of the Army, Navy, and Air Force; and the Director of the Office of Management and Budget. In addition, the report will be made available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in Appendix VI. Signed by: Michael J. Sullivan: Director, Acquisition and Sourcing Management: [End of section] Appendix I: Scope and Methodology: To conduct our work, we reviewed relevant sections of Title 10 of the U.S. Code, the Weapon Systems Acquisition Reform Act of 2009 (WSARA), and the National Defense Authorization Act for Fiscal Year 2008 to establish the role of the Joint Requirements Oversight Council (JROC) in considering trade-offs among cost, schedule, and performance objectives; reviewing the estimated level of resources needed to fulfill these requirements; and prioritizing requirements. We also reviewed Department of Defense (DOD), Joint Staff, and military service guidance documents, as well as those for the Joint Capabilities Integration and Development System (JCIDS) for developing and validating military requirements, to determine how these roles have been implemented in policy. To determine how these policies have been implemented in practice, we analyzed information and capability documents contained in the Joint Staff's Knowledge Management/Decision Support tool.[Footnote 37] To do so, we first established how many requirements documents--Initial Capabilities Documents (ICD), Capability Development Documents (CDD), and Capability Production Documents (CPD)--were reviewed by the JROC and Joint Capabilities Board (JCB) during fiscal year 2010. We selected fiscal year 2010 as our time frame because WSARA was enacted in May 2009, and this would allow for any changes the JROC would implement as a result of this legislation. We then focused our analysis on the unclassified requirement documents reviewed by the JROC and JCB in fiscal year 2010 which identified capability gaps or defined performance requirements for new major defense acquisition programs: 13 ICDs and 7 CDDs. We assessed these documents, as well as briefings presented to the JROC or the JCB, associated meeting minutes, and JROC decision memos. We also examined 15 JROC reviews of programs that incurred substantial cost growth after program start in fiscal year 2010 to determine if cost, schedule, and performance trade-offs were made. We chose this time period to allow for any changes the JROC would implement as result of the enactment of WSARA in May 2009. To determine the extent to which the JROC has considered trade-offs among cost, schedule, and performance objectives within programs, we reviewed the seven CDDs submitted to the JROC and analyzed the information presented on trade-offs. We focused on CDDs because they are the first requirements documents that contain cost, schedule, and performance objectives. We also examined JROC decision memos to identify whether the JROC provided input on cost, schedule, and performance objectives for the seven proposed programs and analyses of alternatives (AOA) conducted by the military services prior to JROC reviews. We also met with officials from the Joint Staff; Department of the Air Force; Department of the Army; Department of the Navy; Office of the Director of Cost Assessment and Program Evaluation (CAPE); Office of the Under Secretary of Defense (Comptroller); Office of the Assistant Secretary of Defense for Research and Engineering; and respective program offices about these issues. To obtain combatant command views on their participation in the joint requirements process since the implementation of WSARA, we developed a survey administered to DOD's 10 combatant commands.[Footnote 38] The survey addressed a range of topics related to the joint requirements process, including the means for combatant commands to provide information on their capability needs. To understand the Joint Staff's ongoing internal JCIDS review, we assessed the review charter and met with the Joint Staff officials managing the review to discuss the recommendations from the review and how they might affect the JROC's consideration of trade-offs. We also observed joint requirements meetings and reviewed prior GAO reports. To determine the quality and effectiveness of efforts to estimate the level of resources needed to fulfill joint military requirements, we assessed the resource estimates used to support the seven unclassified proposed major defense acquisition programs reviewed by the JROC in fiscal year 2010 against the best practices in our cost estimating guide.[Footnote 39] We used these criteria to determine the extent to which these resource estimates were credible, well documented, accurate, and comprehensive. We scored each best practice as either being Not Met--DOD provided no evidence that satisfies any of the criterion, Minimally Met--DOD provided evidence that satisfies a small portion of the criterion, Partially Met--DOD provided evidence that satisfies about half of the criterion, Substantially Met--DOD provided evidence that satisfies a large portion of the criterion, and Met--DOD provided complete evidence that satisfies the entire criterion. We determined the overall assessment rating by assigning each individual rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the average of the individual assessment ratings to determine the overall rating for each of the four characteristics. To perform this analysis, we obtained and analyzed program resource estimate supporting documentation, including service cost positions, technical descriptions, work breakdown structures, technology readiness assessments, program schedules, and AOA reports. We also interviewed program and cost estimating officials, when necessary, to gather additional information on these resource estimates and the cost models used to produce them. Each program was also provided with a copy of our assessment of their resource estimates for review and comment. To determine the extent to which the JROC prioritized requirements and capability gaps, we reviewed the 13 ICDs and 7 CDDs submitted to the JROC and any discussions of priorities and redundancies contained in each document. We also met with officials from the Joint Staff; Department of the Air Force; Department of the Army; Department of the Navy; and Office of the Under Secretary of Defense (Comptroller) to discuss the extent to which the JROC and its supporting bodies have addressed prioritization issues. To understand the Joint Staff's ongoing internal JCIDS review, we assessed the review charter and met with the Joint Staff officials managing the review to discuss the recommendations from the review and how they might affect the JROC's prioritization of requirements. We also observed joint requirements meetings and reviewed prior GAO reports. We conducted this performance audit from June 2010 to June 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010: In fiscal year 2010, the Joint Capabilities Boards (JCB) and Joint Requirements Oversight Council (JROC) combined to review a total of 45 new requirements documents, including 11 that were classified, 2 that were information technology programs, and 8 documents that were not associated with major defense acquisition programs. The remaining 24 requirements documents are identified in figure 4.[Footnote 40] Figure 4: Requirements Documents Reviewed by the JCB or JROC in Fiscal Year 2010: [Refer to PDF for image: table] ICD: Sponsor: Air Force; Capability Gap: Advanced Pilot Trainer. Sponsor: Air Force; Capability Gap: Joint Future Theater Lift. Sponsor: Army; Capability Gap: Ground Combat Vehicle. Sponsor: Army; Capability Gap: Joint Direct-Support Airborne Intelligence, Surveillance, and Reconnaissance. Sponsor: Army; Capability Gap: Positioning, Navigation, and Timing Assurance. Sponsor: Defense Intelligence Agency; Capability Gap: Biometric Enabled Intelligence. Sponsor: Defense Intelligence Agency; Capability Gap: Joint Intelligence Operations Center Information Technology Enterprise. Sponsor: Deputy Assistant Secretary of Defense (Force Health Protection and Readiness) and Joint Staff Surgeon/J4 Health Service Support Division; Capability Gap: Joint Force Health Protection. Sponsor: TRICARE Management Agency; Capability Gap: Electronic Health Record. Sponsor: US Joint Forces Command; Capability Gap: Command and Control On-The-Move. Sponsor: US Joint Forces Command; Capability Gap: Global Force Management. Sponsor: US Northern Command; Capability Gap: Maritime Domain Awareness Joint Integrating Concept. Sponsor: US Strategic Command and US Joint Forces Command; Capability Gap: Joint Aerial Layer Network. CDD: Sponsor: Air Force; Capability Gap: Common Vertical Lift Support Platform. Sponsor: Air Force; Capability Gap: HH-60 Recapitalization. Sponsor: Army; Capability Gap: Integrated Air and Missile Defense System of Systems, Increment 2. Sponsor: Army; Capability Gap: Ground Soldier System, Increment 1. Sponsor: Navy; Capability Gap: Joint Precision Approach and Landing System, Increment 2. Sponsor: Navy; Capability Gap: P-8A Poseidon Multi-mission Maritime Aircraft, Increment 2 and 3. Sponsor: Navy; Capability Gap: Ship to Shore Connector. CPD: Sponsor: Air Force; Capability Gap: Joint Air-to-Surface Standoff Missile - Extended Range System, Increment 1. Sponsor: Army; Capability Gap: Apache Block III. Sponsor: Marine Corps; Capability Gap: Global Combat Support System - Marine Corps/Logistics Chain Management. Sponsor: Marine Corps; Capability Gap: V-22 Block C/20. Source: DOD's Knowledge Management and Decision Support Tool. [End of figure] [End of section] Appendix III: JROC Cost Breach Reviews in Fiscal Year 2010: The Joint Requirements Oversight Council (JROC) conducted 15 reviews following cost breaches in fiscal year 2010--6 Nunn-McCurdy reviews and 9 tripwire reviews. Table 2 identifies these reviews. Table 2: JROC Cost Breach Reviews in Fiscal Year 2010: Nunn-McCurdy reviews: Apache Block III. Advanced Threat Infrared Countermeasure/Common Missile Warning System. DDG 1000. Joint Strike Fighter. Remote Minehunting System. Wideband Global Satellite Communication. Tripwire reviews: Apache Block III. Apache Longbow Block II. C-130 Avionics Modernization Program. CH-47F. Family of Advanced Beyond Line-of-Sight Terminals, Increment 1. Global Hawk. Patriot Advance Capability 3. Tactical Tomahawk. Vertical Take-off and Landing Tactical Unmanned Air Vehicle. Source: GAO analysis of DOD information. [End of table] [End of section] Appendix IV: Best Practice Criteria for Resource Estimates: Table 3 below presents the best practice criteria against which we assessed the resource estimates presented to the Joint Requirements Oversight Council during fiscal year 2010 Capability Development Document (CDD) reviews. Table 3: Best Practice Criteria for Resource Estimates: Characteristic: Comprehensive; Best practices: * The cost estimate includes all life cycle costs. * The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable. * The cost estimate work breakdown structure is product-oriented, traceable to the statement of work/objective, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted. * The estimate documents all cost-influencing ground rules and assumptions. Characteristic: Well-documented; Best practices: * The documentation should capture the source data used, the reliability of the data, and how the data were normalized. * The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each element's cost. * The documentation describes step by step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. * The documentation discusses the technical baseline description and the data in the baseline is consistent with the estimate. The documentation provides evidence that the cost estimate was reviewed and accepted by management. Characteristic: Accurate; Best practices: * The cost estimate results are unbiased, not overly conservative or optimistic and based on an assessment of most likely costs. * The estimate has been adjusted properly for inflation. * The estimate contains few, if any, minor mistakes. * The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Characteristic: Credible; Best practices: * The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs. * A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors. * Major cost elements were crossed-checked to see whether results were similar. * An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. Source: GAO-09-3SP. [End of table] [End of section] Appendix V: Comments from the Department of Defense: The Joint Staff: Washington, DC: June 7, 2011: Reply ZIP Code: 20318-0300. Mr. Michael J. Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, N.W. Washington, D.C. 20548: Dear Mr. Sullivan, Thank you for the opportunity to review GAO Draft Report GA0-11-502, "DOD Requirements Reviews: Missed Trade-off Opportunities," dated 4 May 2011 (GAO Code 120916). Please find enclosed the DOD response to the report's recommendations. The Joint Staff point of contact is Mr. Randolph Wood; J-8; 703-614- 9628, randolph.wood@js.pentagon.mil. Sincerely, Signed by: Craig A. Franklin: Major General, USAF: Vice Director, Joint Staff: [End of letter] Enclosure: GAO Draft Report Dated 4 May 2011: GAO-11-502 (GAO Code 120916): "DOD Requirements Reviews: Missed Trade-Off Opportunities" Department Of Defense Comments On The GAO Recommendations: Recommendation 1: The GAO recommends that the Vice Chairman of the Joint Chiefs of Staff, as Chairman of the Joint Requirements Oversight Council (JROC), establish a mechanism to review the final analysis of alternatives (AoA) report prior to Milestone A to ensure that trades- offs have been considered and to provide military advice on these trades-offs and the proposed material solution to the milestone decision authority. (See page 27/GAO Draft Report.) DOD Response: Partially concur. One of the Joint Capability Development Process Review (JCDPR) recommendations is to have AoA results and recommended path forward briefed to the appropriate Functional Capabilities Board (FCB). At the discretion of the FCB Chair, by exception, the briefing could be elevated to the Joint Capabilities Board (JCB)/JROC for further review. This recommendation does not add another round of staffing, an additional Joint Capabilities Integration and Development System document, or an official validation of the AoA results. It does allow the JROC to provide more informed advice to the milestone decision authority. Concurrently, continued participation by Joint Staff representatives in the AoA Senior Advisory Group, Executive Steering Committee, and Integrated Product Team will ensure that timely military advice on trade-offs will be considered during the execution of the AoA. Recommendation 2: The GAO recommends that the Vice Chairman of the Joint Chiefs of Staff, as Chairman of the JROC, require that capability sponsors present resource estimates that have been reviewed by a military Service's cost analysis organization to ensure best practices are being followed. (See page 27/GAO Draft Report.) DOD Response: Partially concur. When capability development documents (CDDs) are entered into staffing for Joint Staff review and approval, a comparison of an independent cost estimate to the program office's cost estimate is included. A method to highlight and examine this area will be incorporated in the staffing and briefing process leading to a JROC validation, as part of the ongoing JCDPR. Recommendation 3: The GAO recommends that the Vice Chairman of the Joint Chiefs of Staff, as Chairman of the JROC, require that capability sponsors present key results from sensitivity and uncertainty analyses, including the confidence levels associated with resource estimates, based on the program's current level of knowledge. (See page 27/GAO Draft Report. DOD Response: Partially concur. The Director, Cost Assessment and Program Evaluation has oversight responsibility for analyses of alternatives and cost analyst responsibilities for resource estimates. This recommendation needs further study to understand the expected outcomes and the required authorities for the JROC. The JROC approves joint military requirements to inform the decision points of the acquisition process. However, a method to highlight and examine this area will be incorporated in the staffing and briefing process leading to a JROC validation as part of the ongoing JCDPR. Recommendation 4: The GAO recommends that the Vice Chairman of the Joint Chiefs of Staff, as Chairman of the JROC, assign priority levels to the CDDs based on joint force capability gaps and redundancies against current and anticipated threats, and provide these prioritization levels to the Under Secretary of Defense (Comptroller/Chief Financial Officer) and the military Services to be used for resource allocation purposes. (See page 27 through 28/GAO Draft Report.) DOD Response: Partially concur on the prioritization means. One of the JCDPR recommendations is a prioritization framework. Details are still being finalized, but the current recommendation includes prioritizing capability requirements to provide traceability from the capability requirement to capability gap to the solution being pursued (CDD). Therefore, the CDDs will "inherit" priority from the requirements and gaps being satisfied, and these requirements and gaps may come from one or more Initial Capability Documents or Joint Urgent Operational Needs. Priorities based upon CDDs directly would be a less flexible construct for prioritization. DOD Response: Partially concur on the prioritization usage. The prioritization framework developed under the JCDPR has been designed to provide more informed decision making to a wide variety of "consumers" across the Department. Processes that could be enhanced or streamlined by the availability of joint prioritization include the Program and Budget Review, combatant command Integrated Priority Lists and Capability Gap Assessments, and Service Program Objective Memorandum generation. In addition, prioritization could inform Joint Experimentation and Science and Technology efforts. Recommendation 5: The GAO recommends that the Vice Chairman of the Joint Chiefs of Staff, as Chairman of the JROC, modify the Joint Capabilities Integration and Development System operations manual to require that a CDD discuss potential redundancies when validating requirements. (See page 28/GAO Draft Report.) DOD Response: Partially concur. Reviewing assessment of redundancies in the CDD is late in the process for this activity; however, a method to examine this area during CDD staffing and briefing will be addressed as part of the ongoing JCDPR. Two of the recommendations from the JCDPR effort address the issue of unnecessary redundancy prior to the generation of a CDD. One of the Joint assessment criteria in the prioritization framework looks specifically at the notion of requirement uniqueness or redundancy. Unique requirements without suitable alternatives gain a higher priority than requirements that would be unnecessarily redundant with other existing requirements or capability solutions. This helps encourage the drive to commonality and collaboration where multiple organizations have similar or identical requirements. The recommendation for a post-AoA review provides a second opportunity for the FCB (or JCB/JROC) to review the alternatives studied and the recommended path forward. If the recommended approach introduces unnecessary redundancy, the JROC will provide appropriate recommendations to the MDA for adjustments to the path forward. [End of section] Appendix VI: GAO Contact and Acknowledgments: GAO Contact: Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov: Acknowledgments: In addition to the contact named above, Ronald E. Schwenn, Assistant Director; Noah B. Bleicher; Stephen V. Marchesani; Kenneth E. Patton; Karen A. Richey; Anna K. Russell; and Nathan A. Tranquilli made key contributions to this report. [End of section] Footnotes: [1] Pub. L. No. 111-23, § 201(b), (codified at 10 U.S.C. § 181 (b)(1)(C)). [2] Pub. L. No. 111-23, § 201(a)(2). [3] Pub. L. No. 111-23, § 201(a). [4] Pub. L. No. 111-23, § 105(b), (codified at 10 U.S.C. § 181(d)(2)). [5] Pub. L. No. 111-23, § 105(c). GAO, Defense Management: Perspectives on the Involvement of the Combatant Commands in the Development of Joint Requirements, [hyperlink, http://www.gao.gov/products/GAO-11-527R] (Washington, D.C.: May 2011) reports on the extent to which the JROC solicits and considers input from combatant commanders. [6] Pub L. No. 111-383, § 862. [7] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [8] Pub. L. No. 110-181, § 942(d). [9] Pub. L. No. 111-23, § 201 (b) and (c). [10] Pub. L. No. 111-383 § 841. [11] Major defense acquisition programs are those programs identified by DOD that require eventual total research, development, test, and evaluation expenditures, including all planned increments, of more than $365 million or procurement expenditures, including all planned increments, of more than $2.19 billion in fiscal year 2000 constant dollars. [12] The Milestone Decision Authority is an acquisition official with the authority to approve a program's entry into the next phase of the acquisition process. For MDAPs, the Milestone Decision Authority is the USD AT&L, DOD component head, or a component acquisition executive. [13] 10 U.S.C. § 2433a. [14] Pub. L. No. 111-23 § 201(a)(2)(B). [15] GAO, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, [hyperlink, http://www.gao.gov/products/GAO-08-619] (Washington, D.C.: July 2008). [16] Prior to WSARA, the Director of the Office of Program Analysis and Evaluation, which is named as a JROC advisor in the National Defense Authorization Act of Fiscal Year 2008, performed some of the cost estimating and program evaluation roles now performed by CAPE. The Deputy Director of Cost Assessment performs independent cost estimates for MDAPs, among other roles. The Deputy Director of Program Evaluation performs evaluations of the net costs and benefits of a proposed system compared with alternatives, among other roles. [17] [hyperlink, http://www.gao.gov/products/GAO-11-527R]. [18] The Joint Capabilities Development Process Review Integrated Process Team is providing recommendations to improve JCIDS' responsiveness and decision support to the JROC, combatant commands, military services, and defense agencies. The target completion date for implementing the review's recommendations is June 30, 2011. [19] Pub. L. No. 111-23 § 201(d)(1). [20] Under Secretary of Defense for Acquisition, Technology and Logistics, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (Washington, D.C.: Sept. 14, 2010). [21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [22] A rough-order-of-magnitude estimate is a quick, high-level estimate that generally involves less time and effort than a budget- quality estimate. [23] Full funding refers to a budgetary allocation in the future-years defense program. [24] JROC, Funding Guidance for Joint Requirements Oversight Council Directed Action, JROC Memorandum 067-07 (Washington, D.C.: Mar. 23, 2007). [25] An MDAP may also not receive Milestone B approval until the Milestone Decision Authority certifies that funding is available to execute the program through the period covered by the future-years defense program. 10 U.S.C. § 2366b(a)(1)(D). DOD Instruction 5000.02, Operation of the Defense Acquisition System (Dec. 8, 2008) also requires programs to be fully funded through the future-years defense program. [26] Pub. L. No. 110-181, § 942, (codified at 10 U.S.C. § 181(b)). [27] H.R. Rep. No. 110-146 at 381 (2007). [28] GAO, Defense Acquisitions: DOD's Requirements Determination Process Has Not Been Effective in Prioritizing Joint Capabilities, [hyperlink, http://www.gao.gov/products/GAO-08-1060] (Washington, D.C.: September 2008). [29] GAO, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue, [hyperlink, http://www.gao.gov/products/GAO-11-318SP] (Washington, D.C.: March 2011). [30] GAO, Defense Acquisitions: Opportunities Exist to Achieve Greater Commonality and Efficiencies among Unmanned Aircraft Systems, [hyperlink, http://www.gao.gov/products/GAO-09-520] (Washington, D.C.: July 2009). [31] [hyperlink, http://www.gao.gov/products/GAO-07-388]. [32] Under Secretary of Defense for Acquisition, Technology and Logistics, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (Washington, D.C.: Sept. 14, 2010). [33] Manual for the Operation of the Joint Capabilities Integration and Development System (July 31, 2009). [34] While the JROC validated 12 ICDs during fiscal year 2010, according to Joint Staff officials, an additional ICD received during this time period was pending final certification as of January 2011. [35] Electronic Health Record ICD is designed to address 10 gaps in the Joint Force Health Protection concept of operations, including comprehensive medical and dental documentation, inpatient and outpatient order entry and management, and consult and referral management. [36] Command and Control On-The-Move ICD is the capability to maintain situational awareness and make timely and informed decisions while nonstationary. It is designed to address six gaps, including the limited capability to share information with mission partners and an inability to plan collaboratively while on the move. [37] This system is designed to track the status of requirements documents--ICDs, CDDs, CPDs---submitted to the JCIDS process for review. It is the key system that the Joint Staff and other DOD components use to review and comment on proposals. To assess the reliability of the data system, we interviewed Joint Staff officials knowledgeable about the system, and cross-checked information in the system. We determined that the data were sufficiently reliable for the purposes of this report. [38] Combatant commands are DOD's operational commanders. There are six combatant commands with geographic responsibilities and four with functional responsibilities. However, in August 2010, the Secretary of Defense recommended the closure of the U.S. Joint Forces Command. [39] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [40] Joint Staff officials reported that in fiscal year 2010, two key requirements documents were reviewed by a Functional Capabilities Board, but not a JCB or JROC. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: