This is the accessible text file for GAO report number GAO-05-273 entitled 'Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed before Development Starts' which was released on March 15, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Chairman, Subcommittee on Tactical Air and Land Forces, Committee on Armed Services, House of Representatives: United States Government Accountability Office: GAO: March 2005: Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed before Development Starts: GAO-05-273: GAO Highlights: Highlights of GAO-05-273, a report to the Chairman, Subcommittee on Tactical Air and Land Forces, Committee on Armed Services, House of Representatives Why GAO Did This Study: The Air Force is on the verge of making a major commitment to the multi- billion dollar E-10A Multi-sensor Command and Control Aircraft program. Due to the substantial investment needed and technological challenges in developing the aircraft, the Subcommittee on Tactical Air and Land Forces asked GAO to examine the soundness of the E-10A business case as well as the risks associated with the current acquisition strategy. What GAO Found: As the E-10A Multi-sensor Command and Control Aircraft program nears its official starting point, questions remain regarding critical elements of its business case, including the need for the aircraft, the maturity level of its technology, and its funding. Plans call for the E- 10A to couple a new radar system with a sophisticated and software intensive battle management command and control system aboard a Boeing 767. E-10A is planned to fill a current gap in U.S. capabilities and provide a defense against weapons such as cruise missiles. The Office of the Secretary of Defense is still working on a study to determine whether the E-10A program is the most cost-effective way to fill that gap. E-10A program funding plans changed dramatically in December 2004 when the DOD proposed reducing the total program budget by about 45 percent for the next 2 fiscal years. The business case for starting a development program requires demonstrated evidence that (1) the warfighter need exists and that it can best be met with the chosen concept and (2) the concept can be developed and produced within existing resources--including design knowledge, demonstrated technologies, adequate funding, and adequate time to deliver the product. E-10A requirements and resources are still in flux. GAO found risks associated with the current E-10A acquisition strategy that could lead to costly changes later in the program. The program is set to move into production before critical knowledge is acquired. For example, the first fully assembled E-10A, outfitted with its radar and battle management command and control systems, would not be delivered in time to complete testing before the decision is made to begin production. Testing and production are scheduled to start at the same time in 2010. Furthermore, four of six E-10As are scheduled to begin production before the results of testing are available. By not demonstrating that the system can perform as expected before entering production, the program increases the risk of changes and delays later in the program. This strategy requires significant concurrency among the technology development, product development, and production phases. E-10A Acquisition Strategy: [See PDF for image] [End of figure] What GAO Recommends: GAO recommends that the Secretary of Defense ensure that open questions about the E-10A business case are answered before the program advances into the development and demonstration phase and officially begins. GAO also recommends that if the E-10A program goes forward adequate time be allotted to test a prototype before moving into production. DOD concurred with the first recommendation, but noted that the questions may not be resolved until the Milestone B decision, which could now be delayed until 2010. DOD partially concurred with the second recommendation, stating that DOD policy did not require demonstration of a design at that point in the acquisition process. GAO disagrees with this interpretation. www.gao.gov/cgi-bin/getrpt?GAO-05-273. To view the full product, including the scope and methodology, click on the link above. For more information, contact Michael Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Uncertainties Exist in E-10A Business Case as It Approaches Its Decision Point: E-10A Acquisition Strategy Does Not Capture Critical Knowledge before Program Decision Points: Conclusions: Recommendations: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Comments from the Department of Defense: Figures: Figure 1: Comparison of E-10A Strategy and Best Practices Model: Figure 2: E-10A Test and Production Schedule: Abbreviations: DOD: Department of Defense: OSD: Office of the Secretary of Defense: United States Government Accountability Office: Washington, DC 20548: March 15, 2005: The Honorable Curt Weldon: Chairman: Subcommittee on Tactical Air and Land Forces: Committee on Armed Services: House of Representatives: Dear Mr. Chairman: In April 2005, the Air Force plans to make a major commitment to the E- 10A Multi-sensor Command and Control Aircraft (E-10A) program and officially begin its development and demonstration phase. The program, estimated to cost over $7.3 billion through fiscal year 2013, is being designed to fill a gap in U. S. capabilities and provide a defense against weapons such as cruise missiles. It is also being designed to be an airborne battle management platform capable of directing forces to respond to moving targets in the air and on the ground. Given the technical challenges and significant investment associated with this aircraft, you requested that we review the E-10A program to determine whether its business case is clearly defined and its acquisition process is well executed. This report examines (1) the soundness of the Air Force's business case[Footnote 1] including E-10A requirements, technologies, and cost; and (2) the risks associated with the planned acquisition strategy. Because the E-10A is not officially a weapons system acquisition program until the Milestone B[Footnote 2] decision has been approved, the Air Force considered key elements of the business case as pre- decisional and subject to change. Therefore, we did not have the opportunity to review all elements related to the critical E-10A technology assessments, the cost estimate, or the funding profile. The Air Force did provide, however, a technology readiness assessment of the radar subsystem, the planned overall acquisition strategy, and the analysis of alternatives supporting the selection of the specific E-10A aircraft platform. Despite these limitations, we were able to assess the relative state of the E-10A business case as compared to the expectations of best practices. We did this by drawing on other information available that indicated the current instability of estimated cost and funding and that questioned the maturity of the technology associated with a key subsystem. We conducted this review from January 2004 to January 2005 in accordance with generally accepted government auditing standards. More details about our methodology are in appendix I. Results in Brief: Questions remain in the E-10A business case over it being the most cost- effective solution as well as its technology maturity, cost, and funding. Officials in the Office of the Secretary of Defense (OSD) are still studying E-10A alternatives to determine if it is the most cost- effective way to satisfy the planned mission to identify, track, and target time-sensitive threats like cruise missiles.[Footnote 3] The study is also assessing on board command and control needs. Current plans are for 25 crew to carry out mission requirements and 2 crew to fly the aircraft. The OSD expects to present the results of the study by March 2005. We found that radar technologies are ready for system development, but because the readiness assessment of the battle management technologies was not finalized, there was no evidence that these technologies were sufficiently mature. Review organizations within the Air Force and OSD are examining Air Force assessments of technology maturity and costs as well as the annual allocation of funds needed to develop and produce the E-10A. We were not able to review these assessments because they were not yet complete. Program officials also stated that the December 2004 reduction of $600 million in E-10A funding proposed by OSD will require significant changes to the business case because it would reduce planned funding for the total program by about 45 percent in fiscal years 2006 and 2007. The Air Force is determining how this will impact the program schedule and costs as it will likely cause them to truncate key activities in these years. An OSD official said the reduction will likely slow development of the E-10A airframe activities because the available funds will need to be applied to the radar improvement program that is also supporting the Global Hawk program. The Air Force acquisition strategy for acquiring the E-10A calls for moving through development and into production before critical knowledge is captured about design, manufacturing, and reliability. For example, the strategy does not allow for adequate product integration and prototype demonstration to ensure the design is stable at the critical design review stage. Additionally, a fully integrated prototype--a working model of the E-10A with the radar system and command and control computer system working together--will not be delivered in time to allow testing prior to the production decision. Both testing and production are scheduled to concurrently start in 2010. By not demonstrating the system can perform as expected before entering production, the program risks costly design changes and delays later in the program. Furthermore, four of six E-10As are scheduled to begin production before testing is completed. We have found this to be consistently a high-risk acquisition approach in our past reviews of Department of Defense (DOD) acquisitions. We are making recommendations to the Secretary of Defense to ensure that the open business case questions are answered before starting the E-10A program. We also recommend that the Secretary direct the Air Force to revise the acquisition strategy to ensure sufficient time to integrate and demonstrate the design in flight testing of an E-10A prototype before moving the program into production. DOD concurred with our first recommendation, but noted that the open questions will not be fully resolved until the Milestone B decision, which could be delayed until 2010 due to recent budget reductions. DOD partially concurred with our second recommendation. It stated that DOD policy did not require integration and demonstration of a design prior to critical design review. We disagree with this interpretation because DOD policy requires a demonstration of the integrated prototype prior to entering demonstration. Nevertheless, DOD acknowledged that it is restructuring the program to demonstrate the key technologies in a prototype prior to starting system development and demonstration. Background: The E-10A program comprises three primary elements: the aircraft, radar, and battle management command and control subsystem. The aircraft is a Boeing 767-400ER, the largest 767 variant Boeing makes. The Air Force has only contracted for one aircraft to date, because a final decision on the operational platform has not been made. This aircraft is a commercial product that will be modified for military use and used as a testbed. At this time there is only 1 unfilled order for the 400 model in the Boeing assembly line and 25 other unfilled orders for other smaller 767 models.[Footnote 4] If the Boeing production line were to close down before the Air Force is positioned to make a production decision on the E-10A it would have to find an alternative. Alternatives could include a different aircraft type or model or the purchase of 767-400ER aircraft from commercial airline companies. The radar planned for the E-10A began development in 1997 as a response to the growing concern about cruise missile proliferation. Initially, it was intended to upgrade the radar on the Joint Surveillance Target Attack Radar System (Joint STARS). The upgraded radar was to have advanced sensor technology, providing air-to-air capability for cruise missile defense and significant increases in ground surveillance capability. Shortly after the program began development, the Air Force restructured the program to develop a modular, scalable radar suitable for use on a variety of airborne platforms. OSD approved the development of the multiple platform radar in 2003. It is being designed for inclusion on the Global Hawk and E-10A programs. The Air Force began evaluating the need to improve its airborne battle management command and control capabilities in 2002. The planned E-10A battle management command and control subsystem is software intensive and intended to enable the E-10A to process and display sensor data from the radar and eventually from off board sensors so that the onboard crew can take actions against time sensitive targets. The Air Force issued a contract in September 2004 to begin preliminary design efforts for this subsystem. We have a body of work focused on best practices in product development and weapon systems acquisition.[Footnote 5] This work has found that key to success is formulation of a business case that matches product requirements to available resources--proven technologies, sufficient engineering capabilities, time, and funding. Several basic factors are critical to establishing a sound business case for undertaking a new product development. First, the needs of the party seeking the new product, the user, must be accurately defined, alternative approaches to satisfying these needs properly analyzed, and quantities needed for the chosen system must be well understood. The developed product must be producible at a cost that matches the users' expectations and budgetary resources. Finally, the developer must have the resources to design and deliver the product with the features that the customer wants when it is needed. If the financial, material and intellectual resources to develop the product properly are not available, development does not go forward. Additionally, an evolutionary and knowledge-based acquisition strategy that captures critical knowledge before key decision points in the program is needed to execute the business plan. This calls for a realistic assessment of risks and costs; doing otherwise undermines the intent of the business case and invites failure. Ultimately, preserving the business case and attaining critical knowledge in time for decisions strengthens the ability of managers to say "no" to pressures to accept high risks or unknowns. If best practices are not followed, we have found a cascade of negative effects becomes magnified in the product development and production phases of an acquisition program. These have led to acquisition outcomes that included significant cost increases and schedule delays, poor product quality and reliability, and delays in getting the new capability to the warfigher. These outcomes have been demonstrated in other programs such as the F/A-22 fighter, C-17 airlifter, V-22 tiltrotor aircraft, PAC-3 missile, and others. Uncertainties Exist in E-10A Business Case as It Approaches Its Decision Point: Questions remain as the Air Force develops the E-10A program's business case to support the decision to begin development in April 2005. The DOD has identified a need for a cruise missile defense capability and the Air Force has selected the E-10A to meet this need. There are, however, unanswered questions in both the requirement and resource elements of the E-10A business case. OSD is still studying whether the E-10A is the most cost-effective alternative for the cruise missile requirement and the extent of battle management command and control needed on board to satisfy the intended need. Finally, assessments of the technology maturity, estimated costs, and funding availability are still in process. OSD Is Still Studying the Need for the E-10A: OSD officials from the Program Analysis and Evaluation Directorate are not satisfied that the studies done by the Air Force to select the E- 10A sufficiently analyzed alternative systems.[Footnote 6] As a result, they are reviewing alternative systems and attempting to determine the most cost-effective solution to satisfy the warfighter's needs. OSD officials agree that the E-10A could provide an increased capability in identifying and tracking ground moving and time-sensitive targets. However, they believe that if there are less costly systems that can provide similar capabilities, it could be more cost-effective to buy those systems. The Air Force began efforts in 1997 to develop a radar sensor that would detect cruise missiles as part of the Joint STARS program. The Air Force examined different size and power combinations for the radar and which platforms had the capacity to carry the radar and still perform multiple missions. These analyses assumed that only manned airborne platforms could meet these requirements. The Air Force completed a formal analysis of alternatives in February 2002 of different possible host platforms for the radar. The study indicated that other aircraft could meet many of the requirements but were based on older commercial technology that was less efficient to operate. The Air Force analysis concluded that the Boeing 767-400ER was the optimal choice given the future multi-mission purpose of the system, and the size, weight, and performance requirements of the radar. OSD officials are also uncertain about the degree of battle management command and control capability needed onboard the E-10A versus transmitting the information gathered by the E-10A to other command and control centers. According to the Air Force, the need for an onboard capability is driven by the large amounts of data that would be collected and analyzed, the limited bandwidth to transmit the data, and the need to have line-of-sight communications for time-sensitive targeting, particularly against cruise missiles. OSD officials said they are looking at whether the battle management subsystem has to be part of the E-10A platform to meet the timelines identified by the Air Force. They expect to present their results by March 2005. Air Force officials told us that some of the battle management functions are currently performed by ground units, but these ground units cannot adequately respond to real-time events involving moving targets like cruise missiles. The E-10A's primary function will be battle management command and control of cruise missile detection and time-sensitive targeting activities. As a result, its battle management capabilities will be tailored to support those functions. These capabilities were validated in October 2004 by the Joint Requirements Oversight Council in preparation for the program's upcoming Milestone B decision. To provide these capabilities, an onboard crew will be required. The current E-10A crew size is estimated at 27 staff--2 flight crew, 21 mission operators, and 4 technicians. According to the Air Force, the crew size could change depending on the mission and the degree of automation on the system. However, the Air Force has not performed any incremental analysis to show crew size for individual specific missions, such as doing cruise missile defense only. Assessments of Needed Resources to Develop and Produce the E-10A Are Incomplete: To date, the Air Force has not identified sufficient or available resources to meet the warfighter's requirements and to start the development program. The Air Force program office has completed its assessments of E-10A critical technologies, cost estimates, and funding needs but these assessments are being reviewed by OSD. While some resources will meet the requirements, others are either unproven or in a state of flux. Radar development started under a separate program, the Radar Technology Insertion Program, and most radar technologies were reported as mature. Because the Air Force did not provide GAO its technical assessment of the battle management command and control system critical technologies, we consider the maturity levels unproven, even though program officials told us these technologies meet minimum maturity standards. In addition to technologies, the financial resources for the program are in a state flux. The E-10A cost estimate for development and production is still a work in process and funding was recently reduced by $600 million for fiscal years 2006 and 2007, which according to DOD officials will substantially impact the program. Status of Technologies: Most radar technologies are at a high level of maturity, but evidence was not provided to support stated maturity levels of the battle management command and control subsystem. The Air Force assessed radar technologies prior to the October 2003 start of the Radar Technology Insertion Program. The critical technologies identified in the radar improvement program included the radar architecture, modes, receiver/ exciter, and signal processor among others. Of the nine technologies identified, six were assessed as mature to our best practice standard; the remaining three were one level below the best practice requirement for mature technologies, a level DOD policy states is sufficient to begin development. These three technologies are the pulse compression unit, the structure, and the modes. Since the 2003 radar technology assessment, the radar improvement program completed its final design review in June 2004. Numerous tests have been conducted on small-scale radar prototypes to mitigate program risks. These tests electronically drove a signal through the radar, demonstrating the basic functionality of the design. However, the radar subsystem being designed for the E- 10A has demonstrated neither form nor fit, nor has it been integrated on the aircraft platform. Although the integration process is an inherently high-risk endeavor, Air Force officials stated they have a process in place to manage these risks. The actual size of the E-10A's radar will be significantly larger than the tested prototype and will require the E-10A testbed aircraft in order to complete the demonstration currently scheduled to occur in 2010. The process of scaling the radar to the appropriate size and ensuring that all the individual modules work together has yet to be accomplished. Recognizing this, program officials have identified the integration of the radar as a critical technology for the E-10A weapon system. The level of this technology's maturity has not yet been finalized. OSD officials accepted the Air Force's assessment of the radar technologies but expect more detailed information on the technologies when the E-10A weapon system undergoes its Milestone B review in April 2005. An assessment of the battle management command and control subsystem technologies was not provided for our review. This subsystem is complex and software intensive. E-10A program officials told us these technologies would meet the minimum DOD standard for starting a program. However, the Air Force only recently directed the contractor to begin systems engineering efforts to determine a preliminary design for this subsystem. Development of critical software needed to demonstrate the technologies has not started. The first increment of software is not scheduled to be delivered until January 2008. On other major weapon system development programs, we have found software development to be a substantial cause for delays in technology development, system deliveries, and increased costs. Therefore, even though program officials have stated technologies are sufficiently mature, we think stronger evidence will be needed to demonstrate their claim. Status of Cost Estimate and Funding Needs: The Air Force has completed its cost estimate for the total E-10A program and released it to OSD for review. The cost estimates for each of the three major program elements contain risk. The biggest area of cost uncertainty is the battle management command and control subsystem. It is a highly complex software-intensive system. A contract was issued in September 2004 for about $71 million to begin early design and engineering efforts to support a preliminary design review in late 2005. Until this initial design and engineering effort is completed, the program will not be able to establish high confidence in its estimated costs. In addition, the aircraft contract only calls for the delivery of one commercial 767-400ER for testing. To convert this aircraft to military use, there will be additional costs for installing communication antennas, a refueling receptacle, hull hardening, and FAA airworthiness certification. According to the Air Force, these costs have been factored into its latest program estimate. The initial cost estimate for the radar program, managed separately from the E-10A program, has grown. Prior to entering system development, OSD determined that projected costs were understated and directed the Air Force to increase its funding by $154 million. The Air Force acknowledges that funding for the E-10A program is also a major concern. Funding cuts have delayed its start. It has undergone two congressional budget reductions; the first cut in fiscal year 2003 ($343 million) required a significant program replanning effort. The second cut in fiscal year 2005 ($115 million) resulted in schedule delays for the planned test program, system integration lab, testbed aircraft delivery, and the E-10A's first flight. The Air Force states these cuts have caused the planned initial operating capability date to slip 3 years to 2015. A third cut, recently proposed by OSD in December 2004, reduces the program's budget request by $300 million in both fiscal year 2006 and 2007--a total reduction of $600 million. The program office is in the process of evaluating the impact of these reductions and officials indicated that because these represent a reduction of about 45 percent in each year, they will have a significant impact on the program if they are sustained. OSD officials indicated that efforts related to aircraft development and the delivery of the test aircraft will likely bear the bulk of the reductions. This will have an impact on planned program milestones. They said it was important to keep the radar program funded because it is developing the radar planned for the new Global Hawk unmanned aerial vehicle in addition to the E-10A. E-10A Acquisition Strategy Does Not Capture Critical Knowledge before Program Decision Points: The E-10A acquisition strategy raises concern as key decisions are planned before critical product knowledge is available. For example, the strategy for developing the first E-10A increment does not allow for adequate integration or prototype demonstration to ensure the design is stable at the system critical design review. System integration allows program officials to measure the stability of a product's design and its ability to meet established requirements. Both commercial companies and DOD recognize the attainment of this knowledge as being demonstrated by the completion of most engineering drawings and some demonstration of the system level capabilities in a prototype. A stable design that meets requirements should be achieved by critical design review, before system demonstration and initial manufacturing of production representative products begins. However, the Air Force does not expect to deliver the battle management command and control and radar subsystems to the integration laboratory until 2008 and 2009, after critical design review, scheduled in 2007. The transition of the battle management command and control and radar subsystems from the integration lab to the 767-400ER test airframe is not scheduled to begin until late-2009, nearly 2 years after the critical design review and only a few months prior to the program's production commitment decision. As a result, critical knowledge about the basic performance of key subsystems integrated into an actual E-10A prototype will not occur until 2010 (see fig. 1). Figure 1: Comparison of E-10A Strategy and Best Practices Model: [See PDF for image] Note: In the above figure, technology development is extended into production because there are several technologies that cannot be assessed as mature until they are actually integrated and flight tested. [End of figure] Additionally, the fully integrated E-10A prototype will not be available for testing prior to the scheduled decision to begin production. This strategy requires significant concurrency among the technology development, product development, and production phases and places decision makers at a disadvantage by not knowing if the E-10A can demonstrate it meets system performance and reliability requirements before transitioning into production. In fact, the results of operational testing are not scheduled to be available until four of the six planned E-10As are already in production in 2011, greatly increasing the risks of costly design changes and schedule delays later in the program (see fig. 2). Our past reviews have found this to be a high-risk acquisition approach.[Footnote 7] Figure 2: E-10A Test and Production Schedule: [See PDF for image] Note: The schedule depicted above is based on the program plan in effect at the time of our review. Since that time, the program's budget has been reduced, necessitating revisions to this schedule. We have not had the opportunity to review the new program plan. [End of figure] The Air Force is planning to use an incremental approach to achieve the E-10A's full capability with each subsequent increment adding capability. Although an incremental approach can reduce risks, the failure to capture critical knowledge while developing the first increment will likely reduce the benefits of such an approach. As currently planned there will be four distinct E-10A increments. Program officials are planning to conduct major program decision reviews prior to beginning development and demonstration of each increment. This approach, if implemented as planned, will provide decision makers with an opportunity to review the program's progress and risk before making further investment decisions thus reducing risk in the program. The first increment is expected to provide the users with many of the system's basic required capabilities.[Footnote 8] Those capabilities include cruise missile defense and on-board command and control capability for processing, displaying, and communicating the data needed to address time-sensitive targets. Subsequent increments will enhance the system's capabilities, moving them closer to objective levels by increasing the amount of data processing and analysis done by computers and decreasing the amount done by human analysts with computer assistance, thus shortening the time it takes to make decisions. However, if the first increment falters, the Air Force will likely spend increasing amounts of time and money to achieve this initial capability, thereby delaying subsequent increments. Conclusions: The current conditions surrounding the development of the E-10A business case portend the potential for poor outcomes if requirement, resource, and acquisition strategy deficiencies are not resolved before system development and demonstration begins. The decision to start a major weapon systems acquisition program for the E-10A requires an executable business case that demonstrates the E-10A is the best way to satisfy the gap in warfighter's capability and that the concept can be developed and produced within existing resources. An evolutionary and knowledge-based acquisition strategy is needed to ensure this business case can be executed within planned goals. The Air Force and OSD are still determining if a sound business case exists. Questions still surrounding the business case include: * Is the E-10A the most cost-effective alternative? * How extensive of a battle management command and control capability is needed? * Are technologies at a high level of maturity? * Is there sufficient funding to develop and deliver the capability in time? The acquisition strategy also fails to capture critical design, manufacturing, and reliability data in time to make investment decisions for moving the program through the development program into production. The gaps in knowledge increase the likelihood that the Air Force will not be able to deliver on the cost, schedule, and performance goals in its business case. Recommendations for Executive Action: Because gaps exist in the information needed to make a sound business case to start a major acquisition program, we recommend that the Secretary of Defense ensure that the open business case questions are answered before a decision is made to start the E-10A program. Additionally, to ensure a greater likelihood of success, if the E-10A program is approved to begin, we recommend the Secretary direct the Air Force to revise the acquisition strategy to ensure sufficient time is included in the schedule to (1) integrate and demonstrate the design before moving past the critical design review and (2) test a production representative E-10A prototype before starting production. Agency Comments and Our Evaluation: DOD provided us with written comments on a draft of this report. The comments appear in appendix II. DOD concurred with our recommendation that the Secretary ensure that the open business case questions are answered before a decision is made to start the E-10A program. DOD provided some information on the current status of these questions and implies that some of the business case questions had been answered. We believe that until the OSD/Program Analysis and Evaluation study is completed and final results are provided to OSD acquisition decision makers, the business case questions remain open. DOD partially concurred with our recommendation that the Secretary direct the Secretary of the Air Force to revise the E-10A acquisition strategy to ensure sufficient time is available to (1) integrate and demonstrate the design before moving past the critical design review and (2) test a production representative E-10A before starting production. Regarding (1), DOD stated that OSD policy does not require the integration and demonstration of a design before critical design review. We disagree. Section E1.1.14 of Department of Defense Directive 5000.1, The Defense Acquisition System, states that "PMs…shall reduce integration risk and demonstrate product design prior to the design readiness review." DOD's design readiness review is required to end the system integration phase of system development and demonstration. Additionally, DOD's entrance criterion for the demonstration phase requires a demonstration of the integrated product in a prototype. Nonetheless, DOD stated that it is restructuring the program with the goal of demonstrating the radar and battle management technologies in a prototype before starting systems development and demonstration. This approach incorporates the knowledge-based approach inherent in commercial best practices and endorsed by DOD policy. In its comments, DOD acknowledges that this approach will increase confidence in the program's cost estimate and allow time to evaluate the aircraft platform. Regarding (2), DOD stated that the Milestone C production decision for low rate initial production decision will be based on the initial test results from a representation E-10A aircraft system. While the program schedule in effect at the time of our review did not indicate this, we believe this approach is more consistent with a knowledge based acquisition strategy. By testing a production representative aircraft prior to committing to production, DOD will be able to reduce program risks and make informed decisions based on actual system capabilities and performance information. DOD also provided technical comments to our report. We made changes where appropriate but many of these comments were based on a new acquisition strategy that plans to delay the E-10A program Milestone B decision until 2010. We did not make DOD's recommended changes to the report that reflected this new schedule because it has not been approved and we have not had the opportunity to review it. We are sending copies of this report to the Secretary of Defense and the Secretaries of the Air Force, the Army, and the Navy. We will also provide copies to others on request. In addition, the report will be available at no charge on the GAO Website at http://www.gao.gov. Please contact me at (202) 512-4841 if you have any questions concerning this report. Other key contributors to this report were Martin Campbell, Michael Hazard, Travis Masters, Rae Ann Sapp, David Schilling and John Krump. Sincerely yours, Signed by: Michael J. Sullivan: Director: Acquisition and Sourcing Management: [End of section] Appendix I: Scope and Methodology: During our review we discussed the E-10A program with officials from the following organizations in the Office of the Secretary of Defense, Undersecretary of Defense for Acquisition Technology and Logistics; the Director, Defense Systems/Developmental Test and Evaluation; the Director, Operational Test and Evaluation; the Director, Defense Research and Engineering; and the Director, Program Analysis and Evaluation. We also discussed the E-10A with the technical director of the Joint Theater Air Missile Defense Organization. In addition, we discussed the program with officials from several organizations in the Air Force. These officials included representatives from the Information Dominance Directorate with the Office of the Assistant Secretary for Acquisition; the Directorate of Operational Requirements; the Command Control Communications Intelligence and Reconnaissance Center at Langley Air Force Base; the Electronic Systems Center at Hanscom Air Force Base; and the Aeronautical Systems Center at Wright Patterson Air Force Base. To determine the progress the Air Force had made in developing the business case for the E-10A, we obtained available information on the system's requirements and resources. However, the information we received on resources such as technology maturity, cost, funding, quantities, and schedule was limited. We discussed this information with knowledgeable program office and oversight officials. We also contacted officials studying force structure issues that could impact the requirements for the E-10A program. To assess the validity of the proposed business case, we compared the E-10A information with best commercial practices and DOD policy guidance for new development programs. Because the E-10A program has not yet been approved to enter system development and demonstration, specific information on the system's technology readiness assessment and total program cost and funding were not available. As a result, we could not conduct a detailed assessment of these elements of the business case. However, because of other related information, such as the status of the software intensive battle management command and control subsystem, the significant reduction in funding for fiscal years 2006 and 2007, and the ongoing studies to answer OSD concerns, we were able to conclude that at the time of our review key business case elements were still not mature enough to begin product development. For example, complex and software intensive subsystems in other programs have caused major problems that have delayed achieving technology maturity and the Air Force has only recently directed the contractor to begin early systems engineering effort to determine a preliminary design for the E-10A battle management subsystem. Additionally, the $600 million reduction in funding planned for the first 2 years will almost certainly require the program to extend its planned schedule resulting in additional costs and funding requirements not yet estimated. These are business case elements that need to be firmly established before entering the upcoming Milestone B decision point. To determine the soundness of the E-10A's acquisition strategy, we obtained available information on the program's original and revised acquisition plans from the program office and discussed it with functional oversight and program officials. In addition, we compared the E-10A's planned strategy to best commercial practices and DOD's knowledge-based acquisition policy. However, since our analysis, the program's budget request was reduced by a total of $600 million in fiscal years 2006 and 2007. We conducted our review from January 2004 to January 2005 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Comments from the Department of Defense: OFFICE OF THE UNDER SECRETARY OF DEFENSE: ACQUISITION, TECHNOLOGY AND LOGISTICS: 3000 DEFENSE PENTAGON: WASHINGTON, DC 20301-3000: MAR 09 2005: Mr. Michael J. Sullivan: Director, Acquisition and Sourcing Management: Government Accountability Office: Washington, D.C. 20548: Dear Mr. Sullivan: This is the Department of Defense (DoD) response to the Government Accountability Office (GAO) Draft Report, dated February 4, 2005, "Defense Acquisitions: Changes in E-10A Acquisition Strategy Needed Before Development Starts" (GAO code 120314/GAO-05-273). The DoD concurs with the draft report's first recommendation and partially concurs with the second. The rationale for the DoD's position is provided at enclosure 1. Recommended changes to enhance clarity and correctness are at enclosure 2. Sincerely, Signed by: Glenn F. Lamartin: Director: Defense Systems: Enclosures: 1. DoD Comments to the GAO Recommendations. 2. Recommended Changes: Enclosure 1: GAO DRAFT REPORT - DATED FEBRUARY 4, 2005: GAO CODE 120314/GAO-05-273: "DEFENSE ACQUISITIONS: CHANGES IN E-10A ACQUISITION STRATEGY NEEDED BEFORE DEVELOPMENT STARTS" DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Secretary of Defense ensure that the open business case questions are answered before a decision is made to start the E-10A program. (p. 13 & 14/GAO Draft Report): DOD RESPONSE: Concur. The GAO conducted the Business Case Analysis from January 2004 to January 2005. During this period, the E-10A Systems Group was drafting the required statutory and regulatory documents to support a Milestone B decision, which is considered the initiation of a major defense systems acquisition program. The draft documents were at the level of maturity expected leading up to a milestone, and the program was on schedule to meet the Milestone B Defense Acquisition Board in March 2005. During the Business Case Analysis, the E-10A Systems Group discussed the content of the draft documents with the GAO and agreed to provide the final documents once they were approved. However, since the draft documents were not fully coordinated through Air Force and OSD channels when the GAO study concluded in January 2005, final documents were never released to the GAO. Current status of answers to the open questions is as follows: 1) Is the E-10A the most cost effective alternative? Yes; based on an Analysis of Alternatives (AoA) completed in March 2002. The conclusions were accepted by the OSD Overarching Integrated Product Team. That said, additional information is expected from the results of an OSD/ Program Analysis and Evaluation study that was directed in the Multi- Platform Radar Technology Insertion Program (MP-RTIP) Acquisition Decision Memorandum (ADM). 2) How extensive of a battle management command and control capability (BMC2) is needed? The BMC2 requirement is defined in the E-10A Capabilities Development Document (CDD) approved by the Joint Requirements Oversight Council (JROC) in November 2004. Supporting analysis was presented at the BMC2 Analysis Meeting in September 2004. This justification is being considered along with on-going Office of the Secretary of Defense (OSD) analysis to determine the amount of required E-10A onboard BMC2 capability. 3) Are technologies at a high level of maturity? Technology maturity is addressed in the completed E-10A Technology Readiness Assessment (TRA). Technology maturity varies with the subsystem. Of 25 critical technologies, 9 are at technology readiness level (TRL) 6 and 16 are at TRL level 7 or higher. This meets requirements, or exceeds preferred levels, for entering the System Design and Development phase. A recent Department systems engineering evaluation of the combined E-10A and radar program revealed sound system-level planning and effective processes. The Department agrees, however, that system integration of the radar subsystem is inherently high risk and is working to mitigate that risk. 4) Is there sufficient funding to develop and deliver the capability in time? The current OSD policy is for programs to be fully funded at Milestone B. The Air Force has consistently budgeted the E-10A program to meet its agreed initial operational capability (IOC) date; however, the Department accepts that the $600 million proposed in PB06 for reduction in FY06-07 will slip the IOC from the original schedule. The program is being restructured in response to the PB06 funding reductions. Milestone B might not occur until 2010. Since the answers to the GAO's open business case questions can be found in Milestone B documents that are required by statute and regulation, that information will be available at the restructured program's Milestone B. The Air Force will seek formal approval of its restructure plan in June 2005. RECOMMENDATION 2: The GAO recommended that the Secretary of Defense direct the Secretary of the Air Force to revise the acquisition strategy to ensure sufficient time is included in the schedule to: (1) integrate and demonstrate the design before moving past the critical design review, and (2) test a production representative E-10A prototype before starting production. (p. 14/GAO Draft Report): DOD RESPONSE: Partially Concur. A restructured plan is under review. For (1), while OSD policy does not require the integration and demonstration of a design before moving past the critical design review as recommended by the GAO, the Air Force is restructuring the program with the goal of demonstrating the MP-RTIP radar and key BMC2 technologies in a prototype prior to starting System Development and Demonstration. This approach will also increase confidence in program cost estimates and allow time to evaluate the aircraft platform aspects. With respect to (2), the Milestone C production decision for low rate initial production will be based on the initial test results of a representation aircraft E-10A system. [End of section] FOOTNOTES [1] The business case is defined as demonstrated evidence that (1) the warfighter need exists and that it can best be met with the chosen concept, and (2) the concept can be developed and produced within existing resources--including design knowledge, demonstrated technologies, adequate funding, and adequate time to deliver the product. [2] Milestone B is the entrance point for the system development and demonstration phase, which is considered the initiation of a major defense systems acquisition program. It requires authorization by the Milestone Decision Authority based on advice from senior-level advisors on the Defense Acquisition Board. The Defense Acquisition Board meeting to discuss the E-10A Milestone B is scheduled for April 12, 2005. Department of Defense Instruction 5000.2, Operation of the Defense Acquisition System, Paragraph 3.7, System Development and Demonstration, Subparagraphs 3.7.1.2 and 3.7.2.3. [3] A time-sensitive target is a target of such high priority that it requires an immediate response, either because it poses a danger to friendly forces or it is a highly lucrative, fleeting target of opportunity. [4] Source: Boeing Website, http://www.boeing.com. [5] GAO, Defense Acquisitions: DOD's Assessments of Major Weapon Programs, GAO-04-248 (Washington, D.C.: Mar. 31, 2004). Other recent reports discussing best practices include GAO, Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 15, 2002) and Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices, GAO- 02-469T (Washington, D.C.: Feb. 27, 2002). [6] Both the Air Force and the Joint Theater Air and Missile Defense Organization conducted analyses to assess future mixtures of systems, including the E-10A. Both studies determined that if the E-10A performs as expected it could significantly increase the capability of the warfighters. However, the Air Force study focused only on ground-moving target indication and the Joint Theater Air and Missile Defense Organization focused primarily on cruise missile defense. [7] GAO, Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C; July 15, 2002). [8] The E-10A Capabilities Development Document lists 157 separate capabilities/ requirements to be achieved by increment 1.3. Of those requirements, 50 will be upgraded or changed as subsequent increments are developed. Thus the remaining 107 (68 percent) will be met by increment 1.0. The individual capabilities/requirements were not weighted in this analysis. GAO's Mission: The Government Accountability Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: