This is the accessible text file for GAO report number GAO-03-645T entitled 'Best Practices: Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from F/A-22 Program' which was released on April 11, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Testimony: Before the Subcommittee on National Security, Emerging Threats, and International Relations, Government Reform Committee, House of Representatives: United States General Accounting Office: GAO: For Release on Delivery Expected at 10:00 a.m. EDT: Friday, April 11, 2003: Best Practices: Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from F/A-22 Program: Statement of David M. Walker, Comptroller General of the United States: GAO-03-645T: GAO Highlights: Highlights of GAO-03-645T, a testimony before the Subcommittee on National Security, Emerging Threats, and International Relations, Government Reform Committee, House of Representatives Why GAO Did This Study: Over the next 5 years, DOD’s overall investments are expected to average $150 billion a year to modernize and transition our forces. In addition, DOD must modernize its forces amid competing demands for federal funds, such as health care and homeland security. Therefore, it is critical that DOD manage its acquisitions in the most cost efficient and effective manner possible. DOD’s newest acquisition policy emphasizes the use of evolutionary, knowledge-based concepts that have proven to produce more effective and efficient weapon systems outcomes. However, most DOD programs currently do not employ these practices and, as a result, experience cost increases, schedule delays, and poor product quality and reliability. This testimony compares the best practices for developing new products with the experiences of the F/A-22 program. What GAO Found: GAO’s reviews of commercial best practices have identified key enablers to the success of product development programs and focused on how DOD can better leverage its investments by shortening the time it takes to field new capabilities at a more predictable cost and schedule. First, commercial firms use an approach that evolves a product to its ultimate capabilities on the basis of mature technologies and available resources. This approach allows only the product features and capabilities achievable with available resources in the initial development. Further product enhancements are planned for subsequent development efforts when technologies are proven to be mature and other resources are available. Second, commercial firms ensure that a high level of knowledge exists at key junctures during a product’s development. The knowledge-based process includes three points: * Before a program is launched, successful programs match customer needs and available resources—technology, engineering knowledge, time, and funding. * About midway through development, the ability of the product’s design is demonstrated to be stable and meet performance requirements. * Before production begins, programs must show that a product can be manufactured within cost, schedule, and quality targets. In contrast, the F/A-22 program illustrates what can happen when a major acquisition program is not guided by the principles of evolutionary, knowledge-based acquisition. When the program was started, several key technologies were not mature. Program managers proceeded through development without the requisite knowledge to effectively manage program risk and, at the start of production, key manufacturing processes were not under control. The F/A-22 program has undergone significant cost increases. Instead of fielding early capabilities to the war fighter, the development cycle has extended to 19 years, so far, and original quantities have been significantly reduced, raising concerns about the capability the program will eventually deliver. DOD recognizes the need to get better weapon system outcomes, and its newest acquisition policy emphasizes the use of evolutionary, knowledge-based acquisition concepts proven to produce better outcomes in developing new products. However, policy changes alone are not enough. Leadership commitment and attention to putting the policy into practice for individual programs is needed to avoid the problems of the past. DOD will have many opportunities to do so over the next several years with its force modernization investments. What GAO Recommends: GAO is not making recommendations in this testimony. However, in a number of prior reports, GAO has recommended that DOD adopt policies with metrics for technology, design, and manufacturing maturity to support knowledge-based decision making. These policies should apply when making decisions on individual weapons programs. www.gao.gov/cgi-bin/getrpt?GAO-03-645T. To view the full report, including the scope and methodology, click on the link above. For more information, contact Katherine Schinasi at (202) 512-4841 or schinasik@gao.gov. [End of section] Mr. Chairman and Members of the Subcommittee: I am pleased to be here today to participate in the Subcommittee’s hearing on how the Department of Defense (DOD) can--and must--get better outcomes from its weapon system investments. DOD is on the threshold of several major investments in acquisition programs that are likely to dominate budget and doctrinal debates well into the next decade. These programs include such systems as the Missile Defense Agency’s suite of land, sea, air, and space defense systems; the Army’s Future Combat Systems; and the Air Force’s and Navy’s Joint Strike Fighter. Over the next 5 years, DOD’s overall investments are expected to average $150 billion a year as DOD works to keep legacy systems as well as modernize and transform our national defense capabilities for the future. Therefore, to meet these challenges, it is essential that sound foundations for these and other weapon system investments be laid now so that the resulting programs can be executed within estimates of available resources. Any discussion of improvements to DOD’s modernization efforts must be set in the context of overall expected budget availability. There are important competing priorities. Health care costs are growing at double-digit rates, and spending on homeland security will likely grow as we seek to defeat terrorism worldwide. We face an oncoming demographic tidal wave, and by 2035 the number of people who are 65 or older will have doubled, creating much larger demands on the federal budget. The demand of funding for entitlement programs continues to grow, creating increasing pressures on discretionary funding for other federal priorities like education and defense. Therefore, it is critical that DOD manage its acquisitions in the most cost efficient and effective manner possible. My testimony today is about improving the outcomes of major weapon system acquisitions by using best practices to capture and use the right product knowledge at the right time for better decision making during product development. As per your request, I will compare acquisition practices and decisions made for the F/A-22 with these best practices for developing new products. The divergence between F/A-22 experiences and best product development practices, we believe, largely explains why the F/A-22 has been in development for over 16 years and its cost has grown substantially. It is also a primary contributor to other performance issues that are currently faced by the program. My testimony will also include observations on what can be done at this time to limit further negative outcomes in the F/A-22 program. Lastly, I will discuss the need for enforcing DOD’s newest acquisition policy, which on paper embraces best practices but in operation does not always do so, if DOD really expects to get improved outcomes in its major weapon system acquisitions. Improving Major Weapon System Acquisition Outcomes: Clearly, the acquisition process has produced superior weapons, but it does so at a high price. Weapon systems routinely take much longer time to field, cost more to buy, and require more support than investment plans provide for. These consequences reduce the buying power of the defense dollar, delay capabilities for the war fighter, and force unplanned--and possibly unnecessary--trade-offs in desired acquisition quantities and an adverse ripple effect among other weapons programs or defense needs. Because of the lengthy time to develop new weapons, many enter the field with outdated technologies and a diminished supply base needed for system support. Frequently, this requires upgrades to the capability as soon as the new system is fielded. As previously noted, these inefficiencies have often led to reduced quantities of new systems. In turn, legacy systems remain in the inventory for longer periods, requiring greater operations and support cost that pull funds from other accounts, including modernization. DOD is facing these problems with its tactical air force assets now. We believe DOD can learn lessons from the experiences with the F/A-22 program as it frames the acquisition environment for its many transformational investments. DOD recognizes the need to get better weapon system outcomes, and its newest acquisition policy emphasizes the use of evolutionary, knowledge-based acquisition concepts proven to produce more effective and efficient outcomes in developing new products. It incorporates the elements of a knowledge-based acquisition model for developing new products, which we have recommended in our reviews of commercial best practices. Our body of work focuses on how DOD can better leverage its investments by shortening the time it takes to field new capabilities at a more predictable cost and schedule. However, policy changes alone will not guarantee success. Unless written policies are consistently implemented in practice through timely and informed decisions on individual programs, outcomes will not change. This requires sustained leadership and commitment and attention to the capture and use of key product knowledge at critical decision points to avoid the problems of the past. The Case for an Evolutionary Product Development Environment: A key enabler to the success of commercial firms is using an approach that evolves a product to its ultimate capabilities on the basis of mature technologies and available resources. This approach allows commercial companies to develop and produce more sophisticated products faster and less expensively than their predecessors. Commercial companies have found that trying to capture the knowledge required to stabilize the design of a product that requires significant amounts of new technical content is an unmanageable task, especially if the goal is to reduce development cycle times and get the product to the marketplace as quickly as possible. Therefore, product features and capabilities not achievable in the initial development are planned for subsequent development efforts in future generations of the product, but only when technologies are proven to be mature and other resources are available. DOD’s new policy embraces the idea of evolutionary acquisition. Figure 1 compares evolutionary and single step (“big bang”) acquisitions. Figure 1: Comparison of Evolutionary and Big Bang Acquisition Approaches: [See PDF for image] [End of figure] An evolutionary environment for developing and delivering new products reduces risks and makes cost more predictable. While the customer may not receive an ultimate capability initially, the product is available sooner, with higher quality and reliability, and at lower, more predictable cost. Improvements are planned for future generations of the product. The Case for Knowledge-Based Product Development Process: Leading commercial firms expect that their program managers will deliver high-quality products on time and within budgets. Doing otherwise could result in losing a customer in the short term and losing the company in the longer term. Thus, in addition to creating an evolutionary product development environment that brings risk in control, these firms have adopted practices that put their individual program managers in a good position to succeed in meeting these expectations on individual products. Collectively, these practices ensure that a high level of knowledge exists about critical facets of the product at key junctures during its development. Such a knowledge- based process enables decision makers to be reasonably certain about critical facets of the product under development when they need to be. The knowledge-based process followed by leading firms is shown in detail in table 1, but in general can be broken down into three knowledge points. First, a match must be made between the customer’s needs and the available resources--technology, engineering knowledge, time, and funding--before a program is launched. Second, a product’s design must demonstrate its ability to meet performance requirements and be stable about midway through development. Third, the developer must show that the product can be manufactured within cost, schedule, and quality targets and is demonstrated to be reliable before production begins. The following table illustrates more specifically what we have learned about how successful programs gather knowledge as they move through product development. Table 1: Highlights of Specific Best Practices for Acquisitions: Knowledge Point 1 (Should occur before program launch): Separate technology from product development. Knowledge Point 1 (Should occur before program launch): Have clear measures and high standards for assessing technology maturity-- technology readiness levels. Knowledge Point 1 (Should occur before program launch): Use a disciplined systems engineering process for translating and balancing customer’s desires with product developer’s technology, design, and production limitations; in other words, bring the right knowledge to the table when laying down a program’s foundation. Knowledge Point 1 (Should occur before program launch): Identify the mismatches between desired product features and the product developer’s knowledge and either (1) delay the start of the new product development until knowledge deficit can be made up or (2) reduce product features to lessen their dependence on areas where knowledge is insufficient (evolutionary acquisition). The main opportunities for trading off design features to save time and money occur here, before a program is started. Knowledge Point 1 (Should occur before program launch): When do you know you have achieved this knowledge point? When technologies needed to meet essential product requirements have been demonstrated to work in their intended environment and the producer has completed a preliminary design of the product. Knowledge Point 2 (Should occur midway between system integration and demonstration).: Hold a major decision review between system integration and system demonstration that determines that the product design is stable and includes specific criteria to move into the system demonstration phase. Knowledge Point 2 (Should occur midway between system integration and demonstration): Use integrated engineering prototypes to demonstrate design stability and prove with testing that the design meets the customer’s requirements. It is important that this happen before initial manufacturing begins--a point when investments are increased to produce an item. Knowledge Point 2 (Should occur midway between system integration and demonstration): Identify critical manufacturing processes and establish a plan to bring these under statistical control by the start of production; also establish reliability goals and a growth plan to achieve these by production. This facilitates the achievement of process control and reliability goals at the completion of knowledge point 3. Knowledge Point 2 (Should occur midway between system integration and demonstration): When do you know you have achieved this knowledge point? When 90 percent of engineering drawings are releasable to manufacturing organizations. Drawings are the language used by engineers to communicate to the manufacturers the details of the new product--what it looks like, how its components interface, how to build it and the critical materials and processes needed to fabricate it. This makes drawings a key measure of whether the design is stable or not. Knowledge Point 3 (Should occur before production): Demonstrate that all critical manufacturing processes are under statistical control and consistently producing items within the quality standards and tolerances for the overall product before production begins. This is important, since variation in one process can reverberate to others and result in defective parts that need to be repaired or reworked. Knowledge Point 3 (Should occur before production): Demonstrate product reliability before the start of production. This requires testing to identify the problems, design corrections, and retest the new design. Commercial firms consider reliability important and its achievement a measure of design maturity. Knowledge Point 3 (Should occur before production): When do you know you have achieved this knowledge point? When all key manufacturing processes have come under statistical control and product reliability has been demonstrated. [End of table] DOD programs often do not employ these practices. We found that if the evolutionary, knowledge-based acquisition concepts were not applied, a cascade of negative effects became magnified in the product development and production phases of an acquisition program. These led to acquisition outcomes that included significant cost increases and schedule delays, poor product quality and reliability, and delays in getting new capability to the war fighter. This is often the case in DOD programs as shown in our past work on systems like F/A-22 fighter, C-17 airlifter, V-22 tiltrotor aircraft, PAC-3 missile, BAT antitank munition, and others. We did find some DOD programs that employed best practice concepts and have had more successful program outcomes to date. These included the Global Hawk unmanned vehicle, AIM-9X missile, and Joint Direct Attack Munitions guided bomb. Figure 3 shows a notional illustration of the different paths and effects of a product development. Figure 2: Different Paths That A Product’s Development Can Take: [See PDF for image] [End of figure] It is clear that knowledge about the product’s technology, design, and processes captured at the right time can reduce development cycle times and deliver a more cost effective, reliable product to the customer sooner than programs that do not capture this knowledge. In applying the knowledge-based approach, the most leveraged decision point of the three, is matching the customer’s needs with the developer’s resources--technology, design, timing, and funding. This initial decision sets the stage for the eventual outcome--desirable or problematic. The match is ultimately achieved in every development program, but in successful development programs, it occurs prior to program launch. In successful programs, negotiations and trade-offs occur before a product development is launched to ensure that a match exists between customer expectations and developer resources. The results achieved from this match are balanced and achievable requirements, sufficient investment to complete the development, and a firm commitment to deliver the product. Commercial companies we have visited usually limit product development cycle-time to less than 5 years. In DOD, this match is seldom achieved. It is not unusual for DOD to bypass early trade-offs and negotiations, instead planning to develop a product based on a rigid set of requirements that are unachievable within a reasonable development time frame. This results in cost and schedule commitments that are unrealistic. Although a program can take as long as 15 years in DOD, the program manager is expected to develop and be accountable for precise cost and schedule estimates made at the start of the program. Because of their short tenures, it normally takes several program managers to complete product development. Consequently, the program manager that commits to the cost and schedule estimate at the beginning of the program is not the same person responsible for achieving it. Therefore, program accountability is problematic. Ironically, this outcome is rational in the traditional acquisition environment. The pressures put on program managers to get programs approved encourage promising more than can be delivered for the time and money allotted. They are not put in a position to succeed. The differences in the practices employed by successful commercial firms and DOD reflect the different demands imposed on programs by the environments in which they are managed. Specific practices take root and are sustained because they help a program succeed in its environment. The way success and failure are defined for commercial and defense product developments differs considerably, which creates a different set of incentives and evokes different behaviors from managers. Attempts at reforming weapon system acquisitions have not succeeded because they did not change these incentives. All of the participants in the acquisition process play a part in creating incentives. The F/A-22 program, advertised as a flagship of acquisition reform in its early days, failed to establish this match before program launch and today we are discussing the resulting outcomes to-date. F/A-22 Did Not Employ Evolutionary or Knowledge-Based Process: The F/A-22 provides an excellent example of what can happen when a major acquisition program is not guided by the principles of evolutionary, knowledge-based acquisition. The program failed to match requirements with resources and make early trade-offs and took on a number of new and unproven technologies. Instead of fielding early capability and then evolving the product to get new capabilities to the war fighter sooner, the Air Force chose a “big bang” product development approach that is now planned to take about 19 years. This created a challenging and risky acquisition environment that has delayed the war fighter the capabilities expected from this new aircraft. Program leaders did not capture the specific knowledge identified as key for each of the three critical knowledge points in product development. Instead, program managers proceeded through the F/ A-22’s development without the requisite knowledge necessary for reducing program risk and achieving more successful program outcomes. Now the optimism underlying these decisions has resulted in significant cost increases, schedule delays, trade-offs--making do with less than half the number of originally desired aircraft--and concerns about the capability to be delivered. F/A-22 Program Outcomes: Since the F/A-22 acquisition program was started in October 1986, the F/A-22 cost and schedule estimates have grown significantly to where, today, the Air Force estimates the total acquisition unit cost of a single aircraft is $257.5 million.[Footnote 1] This represents a 74 percent increase from the estimate at the start of development and a commensurate loss in the buying power of the defense dollar. Intended to replace the aging F-15 fighter, the F/A-22 program is now scheduled to reach its initial operational capability in December 2005--making its development cycle about 19 years. During this cycle, the planned buy quantity has been reduced 63 percent from 750 to 276 aircraft[Footnote 2]. In addition, since fiscal year 2001, funding for F/A-22 upgrades has dramatically increased from $166 million to $3.0 billion, most of which is to provide increased ground attack capability, a requirement that was added late in the development program. F/A-22 Did Not Use Evolutionary Acquisition or Capture Knowledge Required at Key Decision Junctures: The F/A-22 acquisition strategy from the outset was to achieve full capability in a “big bang” approach. By not using an evolutionary approach, the F/A-22 took on significant risk and onerous technological challenges. While the big bang approach may have allowed the Air Force to more successfully compete for early funding, it hamstrung the program with many new undemonstrated technologies, preventing the program from knowing cost and schedule ramifications throughout development. Cost, schedule, and performance problems resulted. The following table summarizes the F/A-22 program’s attainment of critical knowledge and key decision junctures during the development program and the changes in development cost and cycle time at each point. Table 2: Knowledge Attainment in the F/A-22 Program: Best practice; Program start--1986: Attain knowledge point 1. Separate technology and product development, deliver mature technology, and have preliminary design.; Design review--1995: Attain knowledge point 2. 90 percent of systems and structures engineering drawings releasable and subsystem design reviews completed.; Production start--2001: Attain knowledge point 3. 100% of critical manufacturing processes in statistical control and reliability goals demonstrated. F/A-22 practice; Program start--1986: Knowledge point 1 not attained. Failed to separate technology and product development. Three critical technologies immature: Low-observable materials, propulsion, and integrated avionics. Knowledge point 1 not attained until September 2000.; Design review--1995: Knowledge points 1 and 2 not attained. Only 26 percent of drawings released at the critical design review in February 1995. Knowledge point 2 not attained until September 1998, after delivery of second test aircraft.; Production start--2001: Knowledge point 3 not attained. Less than 50 percent of critical manufacturing processes in control. Only 22 percent of reliability goal demonstrated with many outstanding deficiencies. F/A-22’s estimated development cost[A]; Program start--1986 $12.6 billion; Design review--1995: $21.2 billion; (68 percent increase); Production start--2001: $28.7 billion; (128 percent increase). Estimated development cycle time; Program start--1986: 9.4 years; Design review--1995: 18.1 years; (93 percent increase); Production start--2001: 19.2 years; (104 percent increase). [A] The development estimate includes all F/A-22 RDT&E costs. [End of table]: Technology--The F/A-22 did not have mature technology at the start of the acquisition program. The program included new low-observable (stealth) materials, integrated avionics, and propulsion technology that were not mature at this time. The Air Force did not complete an evaluation of stealth technology on a full-scale model of the aircraft until several years into development. It was not until September 2000, or 9 years into development, that the integrated avionics reached a maturity level acceptable to begin product development. During development, the integrated avionics was a source of schedule delays and cost growth. Since 1997, avionics software development and flight- testing have been delayed, and the cost of avionics development has increased by over $980 million dollars. Today, the avionics still has problems affecting the ability to complete developmental testing and begin operational testing, and the Air Force cannot predict when a solution will be found. Design--The effects of immature technologies cascaded into the F/A-22 development program, making it more difficult to achieve a stable design at the right time. The standard measure of design stability is 90 percent of design drawings releasable by the critical design review. The F/A-22 achieved only 26 percent by this review, taking an additional 43 months to achieve the standard. Moving ahead in development, the program experienced several design and manufacturing problems described by the F/A-22 program office as a “rolling wave” effect throughout system integration and final assembly. These effects included numerous design changes, labor inefficiencies, parts shortages, out of sequence work, cost increases, and schedule delays. Production--At the start of production, the F/A-22 did not have manufacturing processes under control and was only beginning testing and demonstration efforts for system reliability. Initially, the F/A-22 had taken steps to use statistical process control data to gain control of critical manufacturing processes by full rate production. However, the program abandoned this best practice approach in 2000 with less than 50 percent of its critical manufacturing processes in control. In March 2002,[Footnote 3] we recommended that the F/A-22 program office monitor the status of critical manufacturing process as the program proceeds toward high rate production. The reliability goal for the F/A-22 is 3 hours of flying time between maintenance actions. The Air Force estimated that in late 2001, when it entered production, it should have been able to demonstrate almost 2 flying hours between maintenance actions. Instead, it could fly an average of only 0.44 hours between maintenance actions. Since then there has been a decrease in reliability. As of November 2002, development test aircraft have been completing only 0.29 hours between maintenance actions. Additionally, the program was slow to fix and correct problems that had affected reliability. At the time of our review in July 2002, program officials had identified about 260 different types of failures and had identified fixes for less than 50 percent of the failures. To achieve reliability goals will require additional design changes, testing, and modifications. Therefore, additional problems and costs can be expected if the system is fielded with the level of reliability achieved to date. It Is Too Late for the F/A-22 Program to Gain Full Benefit of a Knowledge-Based Process: The F/A-22 did not take advantage of evolutionary, knowledge-based concepts up front and now, the best it can hope for is to limit cost increases and performance problems by not significantly increasing its production until development is complete--signified by developmental and operational testing and reliability demonstrations. To that end, we have recommended that the Air Force reconsider its decision to increase the aircraft production rate beyond 16 aircraft per year.[Footnote 4] The program is nearing the end of developmental testing and plans to start initial operational testing in October 2003. If developmental testing goes as planned, which is not guaranteed, operational testing is expected to be completed around September 2004. By the end of this fiscal year, 51 F/A-22s will be on contract as low rate production began in 2001. Our March 2003 report identifies various problems still outstanding that could have further impacts on cost, schedule, and delivered performance that are in addition to undemonstrated reliability goals. The problems identified are of particular concern, given Air Force plans to increase production rates and make a full rate production decision in 2004. The problems include: * unexpected shutdowns (instability) of the avionics, * excessive movement of the vertical tails, * overheating in rear portions of the aircraft, * separations of the horizontal tail material, * inability to meet airlift support requirements, and: * excessive ground maintenance actions. : These problems are still being addressed, and not all of them have been solved as yet. For example, Air Force officials stated they do not yet understand the problems associated with the avionics instability well enough to predict when they would be able to resolve them, and certain tests to better understand the vertical tail problem have not yet begun. Despite remaining testing and outstanding problems, the Air Force plans to continue acquiring production aircraft at increasing annual rates and make the full rate production decision in 2004. This is a very risky strategy, given outstanding issues in the test program and the system’s less than expected reliability. The Air Force may encounter higher production costs as a result of acquiring significant quantities of aircraft before adequate testing and demonstrations are complete. In addition, remaining testing could identify problems that require costly modifications in order to achieve satisfactory performance. In a February 28, 2003 report to Representative John Tierney,[Footnote 5] we found that F/A-22 production costs are likely to increase more than the latest $5.4 billion cost growth recently estimated by the Air Force and the Office of Secretary of Defense (OSD). First, the current OSD production estimate does not include $1.3 billion included in the latest Air Force acquisition plan. Second, schedule delays in developmental testing could further postpone the start of the first F/ A-22 multiyear contract, which has already been delayed until fiscal year 2006. This could result in lower cost savings from multiyear procurement. Last, we found several risk factors that may increase future production costs, including the dependency of certain cost reduction plans on Air Force investments that are not being made to improve production processes, the availability of funding, and a reduction in funding for support costs. In addition, DOD has not informed Congress about the quantity of aircraft that can be procured within existing production cost limits, which we believe could be fewer than the 276 currently planned. Further details on F/A-22 cost growth and the Air Force’s attempt to offset it are provided in appendix I. Real Change in Acquisition Outcomes Requires Disciplined Enforcement of Acquisition Policy: While DOD’s new acquisition policy is too late to influence the F/A-22 program, it is not too late for other major acquisition programs like the Missile Defense Agency’s suite of land, sea, air, and space defense systems; the Army’s Future Combat Systems; and the Air Force and Navy’s Joint Strike Fighter. DOD’s revised acquisition policy represents tangible leadership action to getting better weapon system acquisition outcomes, but unless the policies are implemented through decisions on individual programs, outcomes are not likely to change. Further, unless pressures are alleviated in DOD to get new acquisition programs approved and funded on the basis of requirements that must stand out, programs will continue to be compromised from the outset with little to no chance of successful outcomes. If new policies were implemented properly, through decisions on individual programs, managers would face less pressure to promise delivery of all the ultimate capabilities of a weapon system in one “big bang.”: Both form and substance are essential to getting desired outcomes. At a tactical level, we believe that the policies could be made more explicit in several areas to facilitate such decisions. First, the regulations provide little or no controls at key decision points of an acquisition program that force a program manager to report progress against knowledge-based metrics. Second, the new regulations, once approved, may be too general and may no longer provide mandatory procedures. Third, the new regulations may not provide adequate accountability because they may not require knowledge-based deliverables containing evidence of knowledge at key decision points. At a strategic level, some cultural changes will be necessary to translate policy into action. At the very top level, this means DOD leadership will have to take control of the investment dollars and to say “no” in some circumstances if programs are inappropriately deviating from sound acquisition policy. In my opinion, programs should follow a knowledge-based acquisition policy--one that embraces best practices--unless there is a clear and compelling national security reason not to. Other cultural changes instrumental to implementing change include: * Keeping key people in place long enough so that they can affect decisions and be held accountable. * Providing program offices with the skilled people needed to craft acquisition approaches that implement policy and to effectively oversee the execution of programs by contractors. * Realigning responsibilities and funding between science and technology organizations and acquisition organizations to enable the separation of technology development from product development. * Bringing discipline to the requirements-setting process by demanding a match between requirements and resources. * Requiring readiness and operating cost as key performance parameters prior to beginning an acquisition. * Designing and implementing test programs that deliver knowledge when needed, including reliability testing early in design. : Ultimately, the success of the new acquisition policy will be seen in individual program and resource decisions. Programs that are implementing knowledge-based policies in their acquisition approaches should be supported and resourced, presuming they remain critical to national needs and affordable within current and projected resource levels. Conversely, if programs that repeat the approaches of the past are approved and funded, past policies--and their outcomes--will be reinforced with a number of adverse implications. Conclusions: DOD will continue to face challenges in modernizing its forces with new demands on the federal dollar created by changing world conditions. Consequently, it is incumbent upon DOD to find and adopt best product development practices that can allow it to manage its weapon system programs in the most efficient and effective way. Success over the long term will depend not only on policies that embrace evolutionary, knowledge-based acquisition practices but also on DOD leadership’s sustaining its commitment to improving business practices and ensuring that those adopted are followed and enforced. DOD’s new acquisition policy embraces the best practice concepts of knowledge-based, evolutionary acquisition and represents a good first step toward achieving better outcomes from major acquisition programs. The F/A-22 program followed a different path at its beginning, a big bang, high-risk approach whose outcomes so far have been increased cost, quality and reliability problems, growing procurement reductions, and delays in getting the aircraft to the war fighter. Since this program is nearing the end of development and already into production, it is too late to adopt a knowledge approach, but it can limit further cost increases and adverse actions by not ramping up production beyond current levels until developmental and operational testing are completed and reliability goals have been demonstrated. Regardless of the F/A-22’s current predicament, the new policy can and should be used to manage all new acquisition programs and should be adapted to those existing programs that have not progressed too far in development to benefit. At a minimum, the F/A-22 should serve as a lesson learned from which to effect a change in the future DOD acquisition environment. The costs of doing otherwise are simply too high for us to tolerate. Mr. Chairman, this concludes my prepared statement. I would be happy to respond to any questions that you or other members of the Subcommittee may have. [End of section] Appendix I: F/A-22 Production Cost Growth: Over the last 6 years, DOD has identified about $18 billion in estimated production cost growth during the course of two DOD program reviews. As a result, the estimated cost of the production program currently exceeds the congressional cost limit. The Air Force has implemented cost reduction plans designed to offset a significant amount of this estimated cost growth. But the effectiveness of these cost reduction plans has varied. During a 1997 review, the Air Force estimated cost growth of $13.1 billion.[Footnote 6] The major contributing factors to this cost growth were inflation, increased estimates of labor costs and materials associated with the airframe and engine, and engineering changes to the airframe and engine. These factors made up about 75 percent of the cost growth identified in 1997. In August 2001, DOD estimated an additional $5.4 billion in cost growth for the production of the F/A-22, bringing total estimated production cost to $43 billion. The major contributing factors to this cost growth were again due to increased labor costs and airframe and engine costs. These factors totaled almost 70 percent of the cost growth. According to program officials, major contractors’ and suppliers’ inability to achieve the expected reductions in labor costs throughout the building of the development and early production aircraft has been the primary reason for estimating this additional cost growth. Mixed Success With Cost Reduction Plans: The Air Force was able to implement cost reduction plans and offset cost growth by nearly $2 billion in the first four production contracts awarded. As shown in table 3, the total offsets for these contracts slightly exceeded earlier projections by about $.5 million. Table 3: Comparison of Planned Versus Implemented Cost Reduction Offsets for Awarded Production Contracts: Dollars in millions. Fiscal year 1999 (2 aircraft); Planned offsets: $199.0; Implemented offsets: $200.5; Difference: $1.5. Fiscal year 2000 (6 aircraft); Planned offsets: 329.3; Implemented offsets: 336.4; Difference: 7.1. Fiscal year 2001 (10 aircraft); Planned offsets: 580.2; Implemented offsets: 611.1; Difference: 30.9. Fiscal year 2002 (13 aircraft); Planned offsets: 827.2; Implemented offsets: 788.2; Difference: (39.0). Total; Planned offsets: $1,935.7; Implemented offsets: $1,936.2; Difference: $0.5. Source: Air Force. [End of table] Cost reduction plans exist but have not yet been implemented for subsequent production lots planned for fiscal years 2003 through 2010 because contracts for these production lots have not yet been awarded. If implemented successfully, the Air Force expects these cost reduction plans to achieve billions of dollars in offsets to estimated cost growth and to allow the production program to be completed within the current production cost estimate of $43 billion.[Footnote 7] However, this amount exceeds the production cost limit of $36.8 billion. In addition, while the Air Force has been attempting to offset costs through production improvement programs (PIPs), recent funding cutbacks for PIPs may reduce their effectiveness. PIPs focus specifically on improving production processes to realize savings by using an initial government investment. The earlier the Air Force implements PIPs, the greater the impact on the cost of production. Examples of PIPs previously implemented by the Air Force include manufacturing process improvements for avionics, improvements in fabrication and assembly processes for the airframe, and redesign of several components to enable lower production costs. As shown in figure 3, the Air Force reduced the funding available for investment in PIPs by $61 million for lot 1 and $26 million for lot 2 to cover cost growth in production lots 1 and 2[Footnote 8]. As a result, it is unlikely that PIPs covering these two lots will be able to offset cost growth as planned. Figure 3: Planned Versus Actual F/A-22 Production Improvement Program Investment for Production Lots 1 (Fiscal Year 2001) and 2 (Fiscal Year 2002): [See PDF for image] [End of figure] Figure 4 shows the remaining planned investment in PIPs through fiscal year 2006 and the $3.7 billion in estimated cost growth that can potentially be offset through fiscal year 2010 if the Air Force invests as planned in these PIPs. Figure 4: Planned Offsets to Cost Growth From Investing in and Implementing PIPs: [See PDF for image] [End of figure] In the past, Congress has been concerned about the Air Force’s practice of requesting fiscal year funding for these PIPs but then using part of that funding for F/A-22 airframe cost increases. [Footnote 9] Recently, Congress directed the Air Force to submit a request if it plans to use PIP funds for an alternate purpose. [End of section] Related GAO Products: Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. Defense Acquisitions: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996. Tactical Aircraft: Status of the F/A-22 Program. GAO-03-603T. Washington, D.C.: April 2, 2003. Tactical Aircraft: DOD Should Reconsider Decision to Increase F/A-22 Production Rates While Development Risks Continue. GAO-03-431. Washington, D.C.: March 14, 2003. Tactical Aircraft: DOD Needs to Better Inform Congress about Implications of Continuing F/A-22 Cost Growth. GAO-03-280. Washington, D.C.: February 28, 2003. Tactical Aircraft: F-22 Delays Indicate Initial Production Rates Should Be Lower to Reduce Risks. GAO-02-298. Washington, D.C.: March 5, 2002. Tactical Aircraft: Continuing Difficulty Keeping F-22 Production Costs within the Congressional Limitation. GAO-01-782. Washington, D.C.: July 16, 2001. Tactical Aircraft: F-22 Development and Testing Delays Indicate Need for Low-Rate Production. GAO-01-310. Washington, D.C.: March 15, 2001. Defense Acquisitions: Recent F-22 Production Cost Estimates Exceeded Congressional Limitation. GAO/NSIAD-00-178. Washington, D.C.: August 15, 2000. Defense Acquisitions: Use of Cost Reduction Plans in Estimating F-22 Total Production Costs. GAO/T-NSIAD-00-200. Washington, D.C.: June 15, 2000. Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal Year 2001. GAO/OCG-00-8. Washington, D.C.: March 31, 2000. F-22 Aircraft: Development Cost Goal Achievable If Major Problems Are Avoided. GAO/NSIAD-00-68. Washington, D.C.: March 14, 2000. Defense Acquisitions: Progress in Meeting F-22 Cost and Schedule Goals. GAO/T-NSIAD-00-58. Washington, D.C.: December 7, 1999. Fiscal Year 2000 Budget: DOD’s Procurement and RDT&E Programs. GAO/ NSIAD-99-233R. Washington D.C.: September 23, 1999. Budget Issues: Budgetary Implications of Selected GAO Work for Fiscal Year 2000. GAO/OCG-99-26. Washington, D.C.: April 16, 1999. Defense Acquisitions: Progress of the F-22 and F/A-18E/F Engineering and Manufacturing Development Programs. GAO/T-NSIAD-99-113. Washington, D.C.: March 17, 1999. F-22 Aircraft: Issues in Achieving Engineering and Manufacturing Development Goals. GAO/NSIAD-99-55. Washington, D.C.: March 15, 1999. F-22 Aircraft: Progress of the Engineering and Manufacturing Development Program. GAO/T-NSIAD-98-137. Washington D.C.: March 25, 1998. F-22 Aircraft: Progress in Achieving Engineering and Manufacturing Development Goals. GAO/NSIAD-98-67. Washington, D.C.: March 10, 1998. Tactical Aircraft: Restructuring of the Air Force F-22 Fighter Program. GAO/NSIAD-97-156. Washington, D.C.: June 4, 1997. Defense Aircraft Investments: Major Program Commitments Based on Optimistic Budget Projections. GAO/T-NSIAD-97-103. Washington, D.C.: March 5, 1997. F-22 Restructuring. GAO/NSIAD-97-100BR. Washington, D.C.: February 28, 1997. Tactical Aircraft: Concurrency in Development and Production of F-22 Aircraft Should Be Reduced. GAO/NSIAD-95-59. Washington, D.C.: April 19, 1995. Tactical Aircraft: F-15 Replacement Issues. GAO/T-NSIAD-94-176. Washington, D.C.: May 5, 1994. Tactical Aircraft: F-15 Replacement Is Premature as Currently Planned. GAO/NSIAD-94-118. Washington, D.C.: March 25, 1994. FOOTNOTES [1] All references to F/A-22 costs in this testimony are in then-year dollars in order to maintain consistent reporting with our prior reports on the F/A-22. [2] Between 1986 and the start of engineering and manufacturing development in 1991, the quantity was reduced from 750 to 648 aircraft. [3] U.S. General Accounting Office, Tactical Aircraft: F-22 Delays Indicate Initial Production Rates Should Be Lower to Reduce Risks, GAO-02-298 (Washington, D.C.: Mar. 5, 2002). [4] U.S. General Accounting Office, Tactical Aircraft: DOD Should Reconsider Decision to Increase F/A-22 Production Rates While Development Risks Continue, GAO-03-431 (Washington, D.C.: Mar. 14, 2003). [5] U.S. General Accounting Office, Tactical Aircraft: DOD Needs to Better Inform Congress about Implications of Continuing F/A-22 Cost Growth, GAO-03-280 (Washington, D.C.: Feb. 28, 2003). [6] Based on a plan to procure 438 aircraft. [7] The F/A-22 President’s budget for fiscal year 2004 would transfer $876 million in production funding to help fund estimated cost increases in development. As a result, the current production cost estimate is $42.2 billion. [8] Production lot 1 was awarded in fiscal year 2001 and production lot 2 was awarded in fiscal year 2002. [9] Report 107-298, Nov. 19, 2001.