This is the accessible text file for GAO report number GAO-04-643R entitled 'Uncertainties Remain Concerning the Airborne Laser's Cost and Military Utility' which was released on May 19, 2004. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. May 17, 2004: The Honorable Bill Nelson: Ranking Minority Member: Subcommittee on Strategic Forces: Committee on Armed Services: United States Senate: The Honorable Daniel K. Akaka: United States Senate: Subject: Uncertainties Remain Concerning the Airborne Laser's Cost and Military Utility: In 1996, the Air Force launched an acquisition program to develop and produce a revolutionary laser weapon system, the Airborne Laser (ABL), capable of defeating an enemy ballistic missile during the boost phase of its flight. Over the last 8 years, the program's efforts to develop this technology have resulted in significant cost growth and schedule delays. These events led you to request that we answer the following questions: (1) How much and why has the ABL's cost increased since the program's inception? (2) What is the expected military utility of the initial ABL aircraft? (3) What support systems will be required when the ABL is fielded and what is the likely cost of those systems? (4) Have recent program changes resulted in a more cost effective strategy for developing the weapon? After we began our review, the Missile Defense Agency (MDA) refocused the ABL program to pursue a more knowledge-based approach, where program knowledge is demonstrated at critical points in the development cycle.[Footnote 1] We examined the planned changes to determine whether they will result in a more cost-effective strategy for developing the ABL element. This report summarizes that information and transmits the briefing charts that have been revised to address program changes made by MDA after our presentation to your staff on March 4, 2004. In conducting our review, we analyzed Department of Defense (DOD), MDA, Air Force, and ABL program documents and interviewed key program officials. To determine how much ABL's cost increased since the program was initiated and why this growth occurred, we reviewed both the ABL prime contract awarded in 1996 and significant modifications made to the contract through July 2003. We also assessed the contractor's cost and schedule performance by analyzing contractor cost performance reports using Earned Value Management principles. To determine the expected military utility of the initial ABL aircraft, we interviewed key DOD, MDA, and intelligence officials and analyzed relevant documents, including studies and reports. To determine the necessary support systems and their costs, we examined relevant documents and held discussions with the Air Force Air Combat Command. We also reviewed the Air Force Total Ownership Cost database to identify support systems and costs for other high value air assets. To determine whether recent program changes resulted in a more cost-effective strategy for developing the ABL, we compared the program office's previous plans, goals and activities with its recent efforts to refocus the program. We conducted our work from October 2003 through May 2004 in accordance with generally accepted government auditing standards. Results in Brief: The prime contractor's costs for developing ABL have nearly doubled from the Air Force's original estimate and additional cost growth is occurring. The cost growth occurred primarily because the program did not adequately plan for and could not fully anticipate the complexities involved in developing the system. MDA continues to face significant challenges in developing the ABL's revolutionary technologies and in achieving cost and schedule stability. From 1996 through 2003, the value of the prime contract, which accounts for the bulk of the program's cost, increased from about $1 billion to $2 billion. According to our analysis, costs could increase between $431 million to $943 million more through first full demonstration of the ABL system. Cost growth has been spurred by rework that was necessary because rapid prototyping forced the program to integrate components before all subcomponents were fully tested. In addition, fabricating ABL's unique components and developing its complex software proved more costly and time-consuming than anticipated. Although ABL's prime contractor has added additional personnel to the contract, the program is faced with a bow wave of uncompleted work from prior years. Recognizing that the technology development activities directed by the contract could not be completed within the contract's cost ceiling, the ABL program office began development of a new cost estimate for completing these activities. Predictions of the military utility of the initial ABL aircraft are still highly uncertain because these forecasts are not based on any demonstrated capability of the system, but rather on modeling, simulations, and analysis. These assessment tools predict that the initial Airborne Laser will be militarily useful against most theater and intercontinental ballistic missiles; but flight-test data are not yet available to anchor these tools. According to test officials, the models and simulations are adequate for establishing system parameters, but may not be sufficient for estimating the effectiveness of a fielded system. Additionally, other factors will influence ABL's military utility, including the availability of support infrastructure and the number of aircraft available. When it is fielded, ABL is expected to require unique support for its laser and beam control and fire control components in addition to the support burdens attached to all high-value air assets.[Footnote 2] For example, to carry out a wider range of missions, ABL will need laser fuel production facilities close to the theater of operations. ABL will also require unique maintenance, such as re-calibration and re-coating of beam control and fire control subcomponents. In addition, it will require the typical support systems needed by other high-value air assets, such as escort aircraft for protection. MDA has not yet determined the cost of ABL's unique support systems, but operating costs[Footnote 3] for other high-value air assets range from about $24,000 per hour to $92,000 per hour. MDA refocused the ABL program in February 2004 to pursue a more cost- effective development strategy. The program now plans to follow a knowledge-based rather than a schedule-driven approach to the element's development. For example, the program is no longer working to deliver a contingency sensor capability in Block 2004. Instead it will concentrate on maturing and demonstrating ABL's critical technologies. In addition, the program has delayed indefinitely the purchase of a second aircraft and a test facility. To provide better information to decision makers as they consider whether to continue investing in the ABL program, we are recommending that DOD provide an analysis that quantifies the confidence that decision makers should have in the new cost estimate. DOD concurred. Background: From 1996 to 2001, the ABL program was an Air Force major defense acquisition program. However, in October 2001, the Department of Defense (DOD) transferred responsibility for the program to the Ballistic Missile Defense Organization (now MDA) where the ABL became one element of the Ballistic Missile Defense System (BMDS). The ABL's mission is to destroy enemy ballistic missiles in the boost phase as part of the layered defense strategy. Similar to other BMDS elements, the ABL is being developed incrementally in a series of 2-year blocks. During the first block, known as Block 2004, MDA's goal is to mature the ABL's critical technologies. The ABL includes four major components: a modified 747 aircraft, which serves as the platform for other components; a high-energy chemical laser; a beam control and fire control system; and a battle management and command and control system. The high-energy laser is a chemical oxygen-iodine laser that generates energy through chemical reactions. The prototype, or demonstrator aircraft, that is under development will include six laser modules that will be linked together to produce a laser beam with megawatts of power. The beam control and fire control component is designed to track and stabilize the beam so that its energy remains focused on a small area of an enemy missile. ABL uses its high-energy laser to defeat enemy missiles by rupturing a missile's motor casing, causing the missile to lose power. MDA expects that the battle management and command and control component will plan and execute each ABL engagement. Program Complexity Drives Cost Growth: Because it did not adequately plan for and could not fully anticipate the complexity of developing and demonstrating ABL's critical technologies, the program has experienced continual cost growth and schedule delays. The prime contract, which consumes the bulk of the program's funding, will reach its cost ceiling in May 2004. The program office is in the process of developing a new cost estimate as the basis for raising the contract's ceiling price. Additionally, in recognition of the contract's rising cost, MDA has increased its cost goal for completing ABL's technology development phase during Block 2004. Contract's Current Value Has Doubled Since 1996 and Continues to Increase: From the contract's award in 1996 through 2003, the cost of ABL's primary research and development contract increased from about $1 billion to about $2 billion. In fiscal year 2003 alone, work completed by the contractor cost about $242 million more than expected. The contractor also experienced schedule delays and was unable to complete $28 million of work planned for the fiscal year. Based on the contractor's 2003 cost and schedule performance, we estimate that the prime contract will exceed the contractor's July 2003 cost estimate of about $2.1 billion by between $431 million and $943 million through first full demonstration of the ABL system. Since 1996, the ABL program has experienced several major restructurings and contract re-baselines, due primarily to the unforeseen complexity in manufacturing and integrating critical technology. According to program officials, rapid prototyping (integrating components into a prototype system prior to demonstrating the maturity of all critical technologies) limited subcomponent testing, causing rework and changing requirements. Today the program faces a bow wave of incomplete work from previous years even though the prime contractor has increased the number of people devoted to the program and has added additional shifts to bring the work back on schedule. In addition, unanticipated difficulties in software coding and integration issues, as well as difficulty in manufacturing advanced optics and laser components, have caused cost growth. Increasing Cost of Prime Contract Causes Revisions in Technology Development Cost Goal: According to program officials, the prime contract will reach its cost ceiling in May 2004. The ABL program office is in the process of developing a new cost estimate for the remaining technology development activities, including component ground tests, beam control and fire control flight-tests, integration of components into a working ABL prototype, and a demonstration of the prototype's lethality against a boosting ballistic missile. The program will use this estimate as the basis for raising the contract's ceiling price. MDA also recognized the impact that the prime contract's rising cost is having on the agency's cost goal for completing ABL's technology development. In February 2003, MDA reported in its fiscal year 2004 Budget Estimate Submission[Footnote 4] that its cost goal for maturing and demonstrating ABL's critical technologies during Block 2004 was about $494 million. However, in its fiscal year 2005 submission, MDA increased its Block 2004 cost goal by $583 million to $1.077 billion by moving ABL funds from Blocks 2006 and 2008 to Block 2004. The program manager told us that MDA plans to continue updating its cost goal for completing technology development as part of each year's budgeting process. To better assess the program's annual funding needs, the contractor will plan work in about 12-month increments. Successful developers have found that it is difficult to make highly reliable cost and schedule estimates until the maturity of a product's critical technologies has been demonstrated. The ABL program has not yet reached this point. One well-established tool for providing decision makers with an increased understanding of the reliability of developed cost estimates is an "uncertainty analysis." An uncertainty analysis simulates a model by randomly and repeatedly generating values for certain variables. After hundreds or thousands of trials, one can view the statistical results and determine the confidence level in any outcome. For example, the model may show that there is a 10-percent chance that the project can be completed for $50 million, a 50-percent chance that it can be completed for $70 million, or a 90-percent chance that the project will cost $100 million or less. Although there remains a gap in the knowledge MDA needs to make highly reliable estimates of the cost to complete ABL's technology development, other programs have found that uncertainty analyses help to understand the size of such a gap. Military Utility of Initial ABL Aircraft Is Highly Uncertain: Predictions of the military utility of the initial ABL aircraft are highly uncertain because these forecasts are based on modeling, simulations, and analysis, rather than the demonstrated capability of the system. The Airborne Laser program office predicts that the ABL prototype being developed during Block 2004 will have a capability to defeat most classes of theater and intercontinental ballistic missiles. However, program officials base these predictions on analyses, models, and simulations that are not yet anchored by flight test data. These assessment tools take into consideration various factors, such as the projected power output of the 6-module laser weapon, the stand-off range that ABL maintains to protect itself from threat weapons, and the characteristics of the threat missiles. Should any of the assumptions made in modeling these factors prove incorrect, the output of the assessment tools might prove similarly incorrect. According to test officials, models and simulations are adequate for establishing system parameters, but may not be sufficient for understanding the effectiveness of a fielded system.[Footnote 5] In addition, the Department of Defense has not yet assessed the Airborne Laser's military utility and has no fixed time frame for doing so. Even if the program office's assessment proves to be correct, other factors, such as reduced time on station and limited forward operating capability, will limit ABL's initial military utility. For example, with only one aircraft, 24-hour operations are not possible. In addition, in order to operate from a forward location the aircraft will require a forward-based chemical facility for replenishment of the chemicals that fuel the high-energy laser. Without these facilities, the Airborne Laser is capable of performing three constrained missions. Support Costs Will Likely Be Higher Than Those of Other High Value Air Assets: The ABL will require unique support in addition to the already substantial support required for all high-value air assets. For example, to remain on station for extended periods of time, ABL will need a production facility close to the theater of operations that can store and mix chemicals for the high-energy laser. Likewise, ABL will require unique maintenance, such as re-calibration and re-coating of beam control and fire control subcomponents; chemical facilities must be secured; and a ground support squadron and chemicals must be transported to a forward location. The ABL will also require support systems, such as a protective escort, that are typical of all high-value air assets. Our review of the U.S. Air Force Total Ownership Costs Database[Footnote 6] shows that operating costs for other high-value air assets, such as the Airborne Warning and Control System or Joint Surveillance Target Attack Radar System, range from about $24,000 per hour to $92,000 per hour. While the ABL program office has identified some of the support systems ABL will require once it is fielded, the program does not plan to fully address this issue, including support costs, before Block 2006. Program Changes Likely to Result in a More Cost-Effective Strategy: MDA made program changes in February 2004 that should make its strategy for developing ABL more cost-effective. The primary change is that the program will no longer try to develop ABL according to a set schedule, but will follow a more knowledge-based approach. For example, MDA no longer plans to deliver a contingency sensor capability in the Block 2004 time frame. Instead it plans to focus on ground-and flight-testing components and demonstrating technologies before proceeding to the next development phase. However, it does not intend to lose sight of its overall objective of testing the prototype ABL aircraft against a short-range ballistic missile in what has become known as a lethal demonstration. Program officials also told us that they plan to defer indefinitely the purchase of a second aircraft and other hardware, which MDA initially planned to initiate during Block 2004. Conclusion: We believe the ABL program made a sound and cost-effective decision to defer the purchase of the second aircraft and additional hardware and to pursue a more knowledge-based approach to the element's development. It has also taken an important first step by setting a cost goal for ABL's technology development phase. However, good investment decisions depend upon an understanding of the total funds that will be needed to obtain an expected benefit, and, at this time, MDA has not been able to provide decision makers assurances that the agency's cost projections to complete technology development can be relied upon. Decision makers could make more informed decisions about further investments in the ABL program if they understood the likelihood and confidence associated with MDA's cost projections. Recommendation: To provide a better framework for making investment decisions during the program's research and development phase, we recommend that the Secretary of Defense direct the Director, MDA, to complete an uncertainty analysis on the contractor's new cost estimate that quantifies the confidence that may be placed in the estimate. Agency Comments and Our Evaluation: In written comments to a draft of this report (see encl. II), the DOD concurred with our recommendation. DOD also provided separate technical comments, which we have incorporated as appropriate. As requested by your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. At that point, copies of this report will be sent to the Secretary of Defense; the Director, Missile Defense Agency; and other interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Should you or your staff have any questions on matters discussed in this report, please contact me on (202) 512-4841 or Barbara Haynes at (256) 922-7535. Principal contributors to this report were Beverly Breen, Alan Frazier, LaTonya Miller, and Karen Richey. Signed by: Robert E. Levin Director, Acquisition and Sourcing Management: Enclosure: Comments From the Department of Defense: ACQUISITION, TECHNOLOGY AND LOGISTICS: OFFICE OF THE UNDER SECRETARY OF DEFENSE: 3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000: Mr. Robert E. Levin: Director, Acquisition and Sourcing Management U. S. General Accounting Office: 441 G. Street, N.W. Washington, DC 20548: Dear Mr. Levin: This is the Department of Defense's (DoD's) response to the GAO Draft Report, "MISSILE DEFENSE: Uncertainties Remain Concerning the Airborne Laser's Cost and Military Utility" dated April 9, 2004 (GAO Report 04- 643R/Audit 120290). The DoD has reviewed the draft report. We concur with the recommendation, with the specific comments enclosed. We also recommend some factual corrections. My action officer for this effort is Lt Col Mark Arbogast, (703) 695-7328, mark.arbogast@osd.mil. We appreciate the opportunity to comment on the draft report. Sincerely, Signed for: Glenn F. Lamartin Director: Defense Systems: Enclosure: As stated: MAY 3 2004: GAO DRAFT REPORT - DATED April 9, 2004 GAO Report 04-643R/Audit 120290: "MISSILE DEFENSE: Uncertainties Remain Concerning the Airborne Laser's Cost and Military Utility" (GAO Report 04-643R/Audit 120290)": DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATION: RECOMMENDATION 1: The GAO recommended that the Secretary of Defense direct the Director, Missile Defense Agency (MDA) to complete an uncertainty analysis on the contractor's new cost estimate that quantifies the confidence that may be placed in the estimate (page 10, GAO Draft Report). DOD RESPONSE: Concur. However, it should be noted that the program office is producing the new cost estimate, as mentioned on page 3 of the report's discussion, not the contractor. The program office has already been using uncertainty analysis in its early formulation of the new estimate. (120290): FOOTNOTES [1] A knowledge-based approach contains three phases that are each distinguished by the knowledge attained. During technology development, scientists apply scientific knowledge to a practical engineering problem and demonstrate that components with the desired form, fit, and function can be developed. During product development, the second phase, engineers integrate components into a stable design and demonstrate that the design will result in a product that meets the customer's needs and can be produced with the time and money available. The final phase, production, is the manufacturing of the product. [2] Examples of other high-value air assets are the Airborne Warning and Control System and the Joint Surveillance Target Attack Radar System. [3] According to the U.S. Air Force Total Ownership Costs Database, operating costs include costs for mission personnel, unit-level consumption, intermediate maintenance, depot maintenance, contractor support, sustaining support, and indirect support. [4] For the last 2 fiscal years, MDA has issued a document known as a Budget Estimate Submission that lays out planned expenditures for each near-term block by element and the activities planned during those blocks. [5] We did not examine the models and simulations that MDA uses to forecast ABL's capability or attempt to replicate MDA's predictions of ABL's military utility. [6] The Air Force Total Ownership Costs database provides detailed cost information on all Air Force major weapon systems.