This is the accessible text file for GAO report number GAO-02-469T entitled 'Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices' which was released on February 27, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office: GAO: Testimony: Before the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate. For Release on Delivery: Expected at 10:00 a.m., EDT: Wednesday, February 27, 2002: Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices: Statement for the Record by Jack L. Brock, Jr. Managing Director, Acquisition and Sourcing Management: and: Randolph C. Hite: Director, Information Technology Architecture and Systems: GAO-02-469T: Mr. Chairman and Members of the Subcommittee: Thank you for the opportunity to submit this statement for the record. At your request, we are discussing our work on best practices related to the acquisition of services, information technology, and weapon systems. Additionally, we are providing you with information on our other work for the subcommittee related to acquisitions. The Department of Defense (DOD) relies heavily on acquisitions. It spends close to $100 billion annually to research, develop, and acquire weapon systems and tens of billions more for services and information technology. Moreover, this investment is expected to grow substantially. From 1995 to 2007, investments in weapon systems are planned to escalate from $90 billion to $157 billion-—about a 74- percent increase. And over the next 5 years, starting in fiscal year 2003, DOD's request for weapon system development and acquisition funds is estimated to be about $700 billion. Similarly, DOD's spending on services is expected to continue to grow, largely attributable to increased purchases of information technology services and professional administrative and management support services. The goals for this investment are ambitious. While continuing to keep legacy systems, DOD plans to fund newer programs such as Global Hawk and Predator, as well as future capabilities such as unmanned airplanes, satellite networks, and information and communication systems. Additionally, information technology is expected to play a critical role in DOD's business transformation. Despite these heavy investments, our work continues to show that DOD is not carrying out acquisitions cost-effectively and that the acquisitions themselves are not always achieving DOD's objectives. Although the department has many acquisition reform initiatives in process, pervasive problems persist regarding the use of high risk acquisition strategies; questionable requirements and solutions that are not the most cost-effective available; and unrealistic cost, schedule, and performance estimates. For these reasons, we have reported DOD contract management, information technology management, and weapon systems acquisition as high risk areas for more than a decade.[Footnote 1] To help DOD meet these challenges, you have asked us over the past several years to identify how leading organizations are addressing similar problems. Our work has identified numerous practices and principles that have consistently resulted in better outcomes— including dramatic cost savings, improved services and products, and ultimately, a better return on investment. The changes leading organizations make often reflected common sense approaches, but they are nevertheless dramatically different from the traditional ways of doing business and they each enhance performance. For example: * In analyzing just how much was being spent on acquiring services and where the dollars were going, leading organizations were able to substantially reduce the number of suppliers and negotiate lower rates. Sometimes, thousands of suppliers were reduced to just a few. * Leading organizations were able to make sure that their business systems could interoperate and truly help to achieve corporate-—rather than business unit—-objectives by using enterprise architectures to guide and constrain their investments. These architectures are essentially blueprints that define where the organization is going in terms of mission, business operations, and technology. * In developing and manufacturing complex products, leading organizations have learned to treat technology development and product development differently and manage them separately. Doing so helped them to reduce design and production difficulties and to deliver more sophisticated products quicker and cheaper. DOD is committed to adopting many best practices and has already taken steps to change its policies and procedures. Implementing these practices, however, will be extremely challenging. For instance, the sheer size of the department, the number of acquisitions, and the hundreds of organizations involved will make it difficult to gain much- needed visibility over spending on services as well as to implement enterprisewide management and oversight mechanisms. Moreover, the changes DOD makes must extend well beyond policies and procedures. Incentives driving traditional ways of doing business, for example, must be changed, and cultural resistance to new approaches must be overcome. Undoubtedly, DOD will need strong and sustained commitment from its leadership to tackle these more elusive challenges-—not just to initiate changes but to continually support them. Best Practices in the Acquisition of Services: DOD is, by far, the government's largest purchaser of services. In fiscal year 2000 alone, it bought more than $53 billion in services ranging from clerical support and consulting, to information technology services, to the management and operation of facilities. However, this spending is not being managed efficiently. Responsibility for acquiring services is spread among individual military commands, weapon system program offices, or functional units on military bases, with little visibility or control at the DOD- or military department level. And when it comes to making procurements, our work, as well as that of other oversight agencies, continues to show that requirements are not always clearly defined, alternatives are not fully considered, and contractors are not adequately overseen. DOD leadership has recognized the need to change current practices for acquiring services and is seeking to adapt the same revolutionary business and management practices that helped the commercial sector gain a competitive edge. GAO Findings: In view of private sector successes with service acquisitions, this subcommittee asked us to examine how leading companies reengineered their approach and the extent to which DOD is pursuing a similar approach. Our first report, which describes the general framework adopted by leading companies, was issued in January 2002.[Footnote 2] The leading companies we studied made a number of dramatic changes to the way they bought services and found that these changes, in turn, resulted in significant cost savings and service improvements. These changes generally began with a corporate decision to pursue a more strategic approach to acquiring services. Taking a strategic approach involves a range of activities—from developing a better picture of what the company is spending on services, to taking an enterprisewide approach to procuring services, to developing new ways of doing business. For example: * The companies we visited analyzed their spending on services to answer the basic questions of how much was being spent and where the dollars were going. In doing so, they realized that they were buying similar services from numerous providers, often at greatly varying prices. The companies used this data to rationalize their supplier base, or in other words, to determine the right number of suppliers that met their needs. * The companies we studied changed how they acquired services in significant ways. They elevated or expanded the role of the company's procurement organization; designated commodity managers to oversee key services; and made extensive use of cross-functional teams to help identify their service needs, conduct market research, evaluate and select providers, and manage performance. Bringing about new ways of doing business was challenging. For example, some companies spent months piecing together data from various financial management information systems and examining individual purchase orders just to get a rough idea of what they were spending on services. Other companies found that in establishing new procurement processes, they needed to overcome resistance from individual business units reluctant to share decision-making responsibility and to involve staff that traditionally did not communicate with each other. To do so, the companies found they needed to have sustained commitment from their senior leadership; to clearly communicate the rationale, goals, and expected results from the reengineering efforts; and to measure whether the changes were having their intended effects. The figure below highlights specific principles and practices the companies we studied followed. Figure 1: Principles and Practices of Leading Companies: [Refer to PDF for image: list] Commitment: Secure up front commitment from top leaders; * Recognize and communicate the urgency to change service spending practices; * Provide clear and strong executive leadership, including goals and targets. Knowledge: Obtain improved knowledge on service spending; * Develop information system to identify how much is being spent with which service provider for what services; * Analyze the data to identify opportunities to reduce costs, improve service levels, and provide better management of service providers. Change: Create supporting structure, processes, and roles; * Create or identify organizations responsible for coordinating or managing service purchases; * Establish proactive business relationships between end users, purchasing units, and other stakeholders; * Implement more integrated team-based sourcing processes; * Create commodity/service experts. Support: Enable success through sustained leadership, communication, and metrics; * Obtain sustaining support from senior leadership to facilitate change; * Establish clear lines of communication between all affected parties; * Demonstrate value and credibility of new processes through use of metrics. Source: GAO analysis. [End of figure] Taking a strategic approach clearly paid off, as companies found that they could save millions of dollars and improve the quality of services received by instituting these changes. In some cases, thousands of suppliers were reduced to a few, enabling the companies to negotiate lower rates. In other cases, new information systems enabled companies to better match their business managers' needs with potential providers. The strategic approach taken by the leading firms we visited could serve as a general framework to guide DOD's service contracting initiatives. DOD has certain elements critical to taking a strategic approach already in place, such as the commitment by senior leadership to improve its practices for acquiring services and to adopting best commercial practices. However, DOD has not conducted a comprehensive analysis of its spending on services or thoroughly assessed its current structure, processes, and roles—two elements that companies found to be crucial to reengineering their approaches to purchasing services. It also lacks a strategic plan that integrates or coordinates the various initiatives underway within the department or that provides a road map for identifying or prioritizing future efforts. To achieve the significant improvements possible by the use of best practices in the acquisition of services, we recommended that the secretary of defense evaluate how a strategic reengineering approach, such as that employed by the leading companies we visited, could be used as a framework to guide DOD's reengineering efforts. Specifically, we recommended that DOD assess (1) whether current or planned financial or management information systems can provide the type of spending data that DOD needs to identify opportunities to leverage its buying power and improve oversight and (2) whether its current organizational structure, processes, and roles are adequate to support a more strategic approach to acquiring services. DOD concurred with our recommendations, and it is implementing improvements on several fronts. For example, it will be upgrading the Federal Procurement Data System to provide more detail on service acquisitions. DOD is also required by the National Defense Authorization Act for Fiscal Year 2002 to establish and implement a management structure for the procurement of services comparable to the management structure that applies to the procurement of products by DOD, and to establish a data collection system to provide management information on each purchase of services in excess of the simplified acquisition threshold. Challenges Still Ahead: Undoubtedly, DOD will find it challenging adopting best practices for buying services. First, DOD's size and the range and complexity of the services it acquires may mean that it cannot adopt a "one-size-fits- all" approach to services acquisitions. According to DOD officials, there are individual commands that are comparable to a Fortune 500 company, each spending billions of dollars annually on services. Further, while some services can be acquired departmentwide, others (such as ship support and maintenance) may be unique to specific commands, units, or geographic locations. Other challenges that could affect DOD's service contracting initiatives include existing problems in its information technology and financial management systems and the unique requirements of the federal environment. As noted earlier, our January report provided an overall framework of practices. We plan to assess each practice area in more depth, looking further at such questions as what are the best practices for conducting a spending analysis and how to ensure an organization is getting the right information. Best Practices in Information Technology Acquisitions: DOD is the federal government's largest consumer of information technology (IT) resources, spending almost $22 billion on IT in fiscal year 2001. For this reason, it is critical that DOD adopt effective IT acquisition practices. Our past reviews have shown that this is not always done. Particularly, because of inefficient and ineffective processes, DOD is at risk of pursing systems and services that do not deliver value commensurate with costs, and that are duplicative, are not well integrated, and do not help to optimize mission performance. GAO Findings: Our work in recent years has looked at best practices for acquiring IT systems and acquiring IT services (e.g., network support or help desk support). Like the framework we described for services acquisitions, these practices do not represent cookie cutter approaches; rather, they need to be tailored to the type of system and service being acquired, how it will be used, and its importance to an organization. IT System Acquisitions: The goals of any IT system acquisition, whether commercial-off-the-shelf (COTS)-based or customized, can be viewed as threefold: (1) to deliver needed functional and performance capabilities by a certain time for a certain cost, (2) to reasonably ensure that, over the system's useful life, these capabilities will provide mission or business value in excess of costs, and (3) to ensure that the system is defined, designed, and implemented in a manner that properly fits within the context of the organizationwide systems environment. In pursuit of these goals, we have categorized IT system acquisition best practices into three corresponding groups, and we apply these practices, as appropriate, in our evaluations of system acquisition across the federal government, including recent and ongoing work at DOD for the Senate Committee on Armed Services. A brief description of the three categories follows: * Is the system being acquired in accordance with mature software acquisition processes? The Software Engineering Institute (SEI), [Footnote 3] recognized for its expertise in software processes, publishes best practice models and methods governing software engineering, acquisition, and development. Collectively, these best practice tools provide logical frameworks for understanding the strengths and weaknesses of an organization's existing software practices, including acquisition practices, and a structured approach for incrementally implementing them. For example, SEI has defined a five stage software acquisition capability maturity model with specific best practices associated with each stage of maturity, including practices governing acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, evaluation, and risk management.[Footnote 4] Other examples of SEI best practice models include its emerging Integrating Technology by a Structured Evolutionary Process model, which addresses the unique challenges associated with COTS-based systems, and its IDEALSM,[Footnote 5] model, which provides a systematic, five phase best practices-based approach to continuously improving software practices. * Is the system being acquired in a series of economically justified incremental builds? Both federal law and guidance[Footnote 6] advocate the use of incremental investment management when acquiring or developing large systems. Incremental investment management can be broken into three major practices: (1) acquiring/developing the system in a series of smaller system increments, (2) individually justifying investment in each separate increment on the basis of costs, benefits, and risks, and (3) monitoring actual benefits achieved and costs incurred on ongoing increments and applying these lessons learned to future increments. Using these system investment practices helps to prevent discovering too late that a given acquisition/development effort is not cost beneficial. * Is the system's proposed architecture compliant with the organization's relevant enterprise architecture(s)? Enterprise architectures[Footnote 7] are essential tools for effectively and efficiently reengineering business processes and for acquiring and evolving supporting systems. As such, using them is a recognized best practice that is embodied in federal guidance.[Footnote 8] An enterprise architecture can be viewed as a master blueprint that defines operational and technological change across a given entity, which can be an organization (e.g., a military service or Defense agency) or a functional or mission area spanning more than one organization (e.g., financial management or combat system identification). In some cases, both organizational and functional/mission area architectures are appropriate because organizations interrelate closely, sharing functional and mission area responsibilities. This is the case for DOD and its component organizations. IT Service Acquisitions: At the request of this committee, we studied IT services acquisition best practices and captured these practices in a framework, which includes seven phases as described below.[Footnote 9] Embedded within each of the phases are specific practices. For example, during the first phase, the business and technical reasons for undertaking an outsourcing effort are explicitly described. This practice ensures, among other things, that the organization has evaluated the appropriateness of outsourcing in its environment. During the second phase, the boundary of responsibilities between provider and acquirer is defined. This takes place prior to even developing the proposal so that the acquirer understands what resources will be required of it, and the prospective provider understands its responsibilities prior to bidding. The remaining phases of the framework focus on managing and monitoring provider performance. Table 1: Description of Phases for IT Outsourcing: Phase: 1. Determine sourcing strategy; Definition: Determine whether internal capability or external expertise can more effectively meet IT needs. Phase: 2. Define operational model; Definition: Formalize executive leadership, team composition, client responsibilities, and operating relationships between client and provider organizations. Phase: 3. Develop the contract; Definition: Establish the legal terms for the IT outsourcing relationship. Phase: 4. Select provider(s); Definition: Find one or more providers who can help reach IT outsourcing goals. Phase: 5. Transition to provider(s); Definition: Transfer responsibility of IT functions to one or more providers. Phase: 6. Manage provider(s) performance; Definition: Make sure each provider is meeting performance requirements. Phase: 7. Ensure services are provided; Definition: Periodically benchmark whether end-users needs are being met to assess whether the organization is still getting good value. [End of table] In addition, the framework recognizes three critical success factors that transcend the seven phases: executive leadership, partner alignment, and relationship management. For example, relationship management describes a process of managing the vendor's performance that goes beyond the specifics of the contract. In relationship management, the acquirer and the provider work together to identify issues and concerns before they evolve into situations requiring official action. DOD Has Inconsistently Applied Best Practices: Our recent work has shown that DOD has not consistently applied best practices when acquiring IT. For example, we recently reported to this committee that two Defense Logistics Agency (DLA) system acquisitions—- the Business Systems Modernization (BSM) and the Fuels Automated System (FAS)[Footnote 10]-—represented a "tale of two cities" with regard to application of the best practices embodied in SEI's software acquisition maturity model.[Footnote 11] Specifically, while DLA was implementing the vast majority of these best practices on BSM, it was not on FAS because of resource constraints. By not following these practices, we concluded that FAS was at risk of not delivering promised system capabilities on time and within budget. To address the weaknesses we identified, we made a number of specific recommendations that DLA intends to implement, including launching a software process improvement program. We also recently reported to this committee that DOD components varied in the degree to which they were implementing software process improvement.[Footnote 12] In particular, we reported that the Air Force, the Army, and the Defense Finance and Accounting Service generally satisfied the best practice tenets of SEI's IDEALSM model, as did certain Navy units. However, DLA, the Marine Corps, and other Navy units did not. This particular model defines a systematic, five- phased approach for software process improvement. Accordingly, we made recommendations to correct these weaknesses, which DOD is implementing. In July 2000, we reported on DOD system acquisitions that were not employing incremental investment management best practices. 13 As an example, we reported that the department had divided its multi-year, billion dollar Standard Procurement System (SPS)[Footnote 14] into a series of incremental system releases. However, it had not treated each of these system increments as a separate investment decision. Instead, it had treated investment in all SPS increments as a single decision that it made when the acquisition was begun. Moreover, it was not attempting to validate whether expected system benefits were actually accruing from deployed system releases. This type of approach to making investment decisions has historically resulted in agencies' investing huge sums of money in systems that do not provide commensurate benefits, and thus has been abandoned by successful organizations. Accordingly, we made a series of recommendations to correct the situation. DOD is in the process of addressing our recommendations. Lastly, in June 2001, we reported that DOD was in the process of investing billions of dollars in acquiring various financial and logistics management systems without having enterprise architectures for either functional area to guide and constrain these investments. [Footnote 15] As part of these reports, we made a series of recommendations to systematically correct this IT acquisition weakness. DOD has initiated steps to implement the recommendations. Challenges Ahead: While DOD has taken steps to implement our recommendations, the challenges ahead are still substantial. To make the most out of its investment in IT, DOD needs to fully incorporate best practices into its policies and procedures and implement our recommendations. Until this is done, DOD risks not meeting its objectives—leading to costly scheduling delays and rework. Our work for the committee will continue to look at systems acquisitions important to DOD operations and determine whether best practices are being effectively applied to them. Currently, we are evaluating how effectively best practices have been applied to DOD's Composite Health Care System. We are also working with this committee to identify DOD IT acquisitions to evaluate against our IT outsourcing framework. Best Practices for the Acquisition of Weapon Systems: As noted earlier, DOD spends close to $100 billion annually to research, develop, and acquire weapons systems. Moreover, it is seeking to considerably ramp up spending to replace a force it believes is becoming outdated and too costly to operate. (See figure 2 for DOD's planned investments in weapon systems.) Our reviews over the past 20 years have consistently found the same problems with these investments—-cost increases, schedule delays, and performance shortfalls Clearly, it is critical to find better ways of doing business—to make sure that weapon systems are delivered on time, at cost, and effectively. Failure to do so can jeopardize other programs in the department and limit DOD's ability to effectively execute warfighting operations. Figure 2: Research, Development, Test and Evaluation and Procurement Funding for Fiscal Years 1995 to 2007: [Refer to PDF for image: multiple line graph] The graph depicts years 1995 through 2007 in billions of then year dollars: Research: Procurement: Total: Total from 2003 through 2007: $700 billion. Source: DOD. [End of figure] GAO Findings: At the request of the committee, we have undertaken an extensive body of work that examines weapon acquisition issues from a different, more cross-cutting perspective-—one that draws lessons learned from the best commercial product development efforts to see if they apply to weapon system improvement. This work has consistently shown that leading commercial firms expect that their program managers will deliver high quality products on time and within budget. Doing otherwise could result in the customer walking away. Thus, the firms have created an environment and adopted practices that put their program managers in a good position to succeed in meeting these expectations. Collectively, these practices ensure that a high level of knowledge exists about critical facets of the product at key junctures during development. Such a knowledge- based process enables decision makers to be reasonably certain about critical facets of the product under development when they need it. The process followed by leading firms can be broken down into three knowledge points: * At program launch, when a match must be made between the customer's needs and the available resources—technology, design, time, and funding; * Midway through development, when the product's design must demonstrate its ability to meet performance requirements; and; * At production start, when it must be shown that the product can be manufactured within cost, schedule, and quality targets. Figure 3 further illustrates how this process works, while figure 4 highlights some specific best practices within this process as well as criteria used to move forward. Figure 3: Knowledge-based Process for Applying Best Practices to the Development of New Products: [Refer to PDF for image: illustration] Technology Development: Knowledge Point 1: Technologies match requirements. Product Development: Program Launch; Integration; Knowledge Point 2: Design performs as expected; Demonstration; Knowledge Point 3: Production can meet cost, schedule, and quality targets. Production. Source: GAO analysis. [End of figure] Figure 4: Highlights of Specific Best Practices: [Refer to PDF for image: list] Knowledge Point 1: * Separate technology from product development. * Have clear measures and high standards for assessing technology maturity, such as technology readiness levels. * Use a disciplined systems engineering process for translating and balancing customer desires with the product developer's technology, design, and production limitations; in other words, bring the right knowledge to the table when laying down a program's foundation. * Identify the mismatches between desired product features and the product developer's knowledge and either (1) delay the start of the new product development until the knowledge deficit can be made up or (2) reduce product features to lessen their dependence on areas where knowledge is insufficient (evolutionary acquisition). The main opportunities for trading off design features to save time and money occur here, before a program is started. When do you know you have achieved this knowledge point? When technologies needed to meet essential product requirements have been demonstrated to work in their intended environment and the producer has completed a preliminary design of the product. Knowledge Point 2: * Hold a major decision review between system integration and system demonstration that determines the product design is stable and includes specific criteria to move into the system demonstration phase. * Use integrated engineering prototypes to demonstrate design stability and prove with testing that the design meets the customer requirements. It is important that this happen before initial manufacturing begins--a point when investments are increased to produce an item. * Identify critical manufacturing processes and establish a plan to bring these under statistical control by the start of production; also establish reliability goals and a growth plan to achieve these by production. This facilitates the achievement of process control and reliability goals at the completion of knowledge point 3. When do you know you have achieved this knowledge point? When 90 percent of engineering drawings are released to manufacturing organizations. Drawings are the language used by engineers to communicate to the manufacturers the details of a new product--what it looks like, how its components interface, how to build it and the critical materials and processes needed to fabricated it. This makes drawings a key measure of whether the design is stable or not. Knowledge Point 3: * Demonstrate that all critical manufacturing processes are under statistical control and consistently producing items within the quality standards and tolerances for the overall product before production begins. This is important since variation in one process can reverberate to others and result in defective parts that need to be repaired or reworked. * Demonstrate product reliability before the start of production. This requires testing to identify the problems, design corrections, and retest the new design. Commercial firms consider reliability important and its achievement a measure of design maturity. When do you know you have achieved this knowledge point? When all key manufacturing processes have come under statistical control and product reliability has been demonstrated. Source: GAO analysis. [End of figure] We have found that when DOD programs employed similar practices, they have experienced outcomes similar to leading firms. The AIM-9X air-to- air missile program is a good example. By adopting practices that mature technology before going into product development and stabilized the design by releasing over 90 percent of the drawings, the program has experienced very minimal cost increases and scheduling delays. Conversely, problems occur in programs when best practices are not adopted. For example, the PAC-3 missile program began nearly 5 years before most of the technical discovery was complete, and only 20 percent of design drawings were released at the point when knowledge point 2 should have been achieved. As a result, the early part of the program was plagued with technical difficulties that impaired efforts to stabilize design, and manufacturing the missile has been difficult. The result was costs that doubled and over a 3-year schedule delay. Knowledge-based acquisitions embraced in DOD policy: In 2000 and 2001, DOD made constructive changes to its acquisition policy that embrace best practices. These focused primarily on (1) making sure technologies are demonstrated to a high level of maturity before beginning a weapon system program and (2) taking an evolutionary, or phased, approach to developing the system. DOD's policy changes are a positive step. First, they would separate technology development from a weapon system development program. This would help to curb incentives to overpromise the capabilities of a new weapon system and to rely on immature technologies. By having a baseline requirement, decisionmakers would also have a means for deciding not to launch a program if a match between requirements and resources was not made. Second, the changes recommend an evolutionary approach to developing requirements and making improvements to a system's capabilities. This is substantially different than the historical approach, which sought to deliver all desired capabilities in one "big bang." For example, the F-22 fighter program was justified on the basis of achieving stealth, supercruise propulsion and fuzed avionics in one-leap with the first product off the production line. But the technologies, and even the funds, were not available to make good on such a promise. While DOD's policy changes are a good step, implementation has been mixed. There have been some successes with evolutionary acquisitions, but they are exceptional cases in that they required significant and unusual intervention from top leadership in the services and DOD. For example, the Global Hawk and Tactical Unmanned Aerial Vehicle programs-—both born from Advanced Concept Technology Demonstrations—- have so far been successful in reducing the time it takes to develop and field a new weapon. In the Tactical Unmanned Aerial Vehicle program, the top military acquisition executive met with the head of the user representative's organization, struck an agreement that the product was to be fielded in stages, with the first stage being a very basic system, and then enforced the agreement. The personal involvement of the under secretary of defense for acquisition, technology, and logistics helped set the stage for Global Hawk's evolutionary approach to meeting requirements. In both cases, this top- level intervention allowed requirements to be flexible and gave the product developers parity with the requirements setters in influencing requirements. Equally important, we believe the intervention signaled support for the programs, which eased some of the pressures that normally accompany efforts to get programs approved. In some of DOD's larger, more complex programs, best practices recommended by the new policy have not been effectively implemented. For example, we recently reported that although the Joint Strike Fighter program has made good progress in some technology areas, the program is at risk of not meeting its affordability objective because critical technologies are not projected to be matured to levels that we believe would indicate a low risk program at the planned start of product development.[Footnote 16] Earlier this week, we also reported that while the Crusader program has made considerable progress in developing key technologies and reducing its size and weight, it was also likely to enter product development with the majority of its critical technologies less mature than best practices recommend. [Footnote 17] As stressed in both reports, failure to make sure technologies are sufficiently mature before product development could result in increases in both product and long-term ownership costs, schedule delays, and compromised performance. Challenges Still Ahead: New policies will not produce better outcomes unless they influence decisions made on weapon systems. A major challenge ahead for DOD is taking steps necessary to make this happen. Specifically: * Programs must be structured so that requirements will not outstrip resources. This means getting requirements-setting organizations to be open to redefining their needs to better match resources available. * DOD's funding process must provide assurance to evolutionary programs that the end-state capability will eventually be achieved. This means getting decisionmakers to commit to providing funding for later, more improved versions of a system. * The role of the science and technology community must change to accept more responsibility for maturing relevant technologies-—without harming DOD's long-term basic research needs. This may require DOD to increase funding and support for science and technology. * Measures for success need to be defined for each stage of the development process so that decisionmakers can be assured that sufficient knowledge exists about critical facets of the product before investing more time and money. * Responsibility for making decisions must be squarely positioned in those with authority to adhere to best practices and to make informed tradeoff decisions. Our work in this area continues to take aspects of the best practice framework and look deeper into how specific practices can enhance how weapon systems are developed and managed. We are currently looking at the management of product design and manufacturing and the question of how leading firms reduce total ownership costs of capital equipment. Additional Ongoing Work of Interest to the Subcommittee: We were also asked to provide updates on our ongoing work related to (1) competition under task- or delivery-order contracts, (2) spare parts price increases, and (3) DOD's use of waivers for certified cost data in negotiating contracts. The following sections highlight our findings and describe ongoing work. Competition Under Task- or Delivery-Order Contracts: The government acquires billions of dollars worth of products and services each year using task- or delivery-order contracts—also known as multiple award contracts—that are available for use by all federal agencies. A task- or delivery-order contract provides for an indefinite quantity of supplies or services (within specific limits) to be furnished during a fixed period, with deliveries scheduled through orders with the contractor. There have been persistent concerns that agencies avoid competition when ordering under such contracts. To prevent this from occurring, the Congress, through the Federal Acquisition Streamlining Act,[Footnote 18] imposed statutory requirements on the use of these contracts. Agencies must now consider awarding multiple contracts rather than a single contract when planning a task- or delivery-order contract. Even with this change, concerns about a lack of competition when ordering under task- and delivery-order contracts have persisted. In 1998, we examined how multiple award contracts were being administered by six organizations, including several within DOD such as the Defense Information Systems Agency, the Standard Systems Group, and the Electronic Systems Center's Hanscom Air Force Base operations. We found that efforts to promote competition for orders placed under multiple award contracts varied at the six organizations. Two organizations achieved consistent competition for orders under their contracts while four others experienced more difficulty obtaining competition. One organization, for example, issued 64 percent of orders (accounting for 20 percent of dollars awarded) on a sole-source basis through the end of fiscal year 1997. Another organization named preferred vendors in announcements of opportunities, resulting in only one proposal being received on most orders.[Footnote 19] In 2000, at the request of this subcommittee, we expanded our review to examine DOD's use of large orders placed under multiple award contracts to acquire IT products and services. We reported that most of the 22 large orders we reviewed had been awarded without competing proposals having been received. Agencies frequently issued orders on a sole-source basis using one of the statutory exceptions to the fair opportunity requirement, and contractors frequently did not submit proposals when provided an opportunity to do so. In most cases, the proposals received involved incumbent contractors.[Footnote 20] The DOD Office of the Inspector General (IG) reported in September 2001 that competition was limited for orders under multiple award contracts DOD organizations administered.[Footnote 21] According to the IG, contracting offices continued to direct awards to selected sources without providing contractors a fair opportunity to be considered--304 of the 423 orders reviewed had been awarded on a sole- source or directed source basis. As a result, the IG concluded, DOD was not obtaining the benefits of sustained competition and the reduced costs that the Congress envisioned multiple award contracts providing. Also, since our reviews began, steps have been taken by the executive branch and the Congress to promote broader competition. First, the executive branch revised procurement regulations in June 1999 to prohibit agencies from designating preferred vendors for orders-—the practice we had reported on the previous year. Second, the Congress directed, through the National Defense Authorization Act for Fiscal Year 2000,[Footnote 22] that the procurement regulations be revised to provide guidance on steps agencies should take to ensure contractors are provided a fair opportunity to be considered. An initial revision to the regulations was issued in April 2000 that directed contracting officers to avoid situations where contractors will specialize in one or a few areas within the contract's scope, creating the likelihood that orders in those areas will be awarded on a sole-source basis. The revision directs contracting officers to consider such factors as the scope and complexity of the requirement, the expected duration and frequency of task orders, and the mix of resources a contractor must have to perform expected task-or delivery-order requirements. The revision also changed the requirements for placing individual orders under these contracts. An additional revision to the regulations has been proposed that identifies issues contracting officers should consider when developing ordering procedures for multiple award contracts. The National Defense Authorization Act for Fiscal Year 2000 directs us to evaluate conformance of the guidance it mandates with existing law. Although the act indicates this guidance should be in place by April 2, 2000, the regulations have not yet been issued in final form. We will initiate the evaluations the act requires once the regulations are finalized, and we look forward to continuing to work with you and your staff on issues related to multiple-award contracting. Increases in Spare Part Prices: In recent years, the military services have expressed concern to the Congress that spare part prices have been increasing at a higher rate than inflation and have taken an unanticipated bite out of the limited funds available to meet readiness requirements. Because the planned price changes for spare parts drive the ordering units' budget requests, unexpected price increases could affect their ability to purchase all the parts they need. In response to these concerns, this subcommittee asked us to determine whether spare part prices had been increasing and to identify some of the factors driving the escalation. In 2000, we issued reports on the prices DOD activities paid for Navy- managed aviation reparable parts, consumable spare parts[Footnote 23] purchased from DLA, and Marine Corps ground system reparable parts. [Footnote 24] At the request of the subcommittee, we are following-up on our prior work at the Navy and are examining the status of DLA's efforts to address spare part price increases. Overall, we found that price increases at the Defense organizations we reviewed were high for certain categories of parts. At the Navy, for example, the prices increased an average of 12 percent annually, but parts with high sales volume increased substantially more than parts overall. Results from our ongoing review indicate that these price increases are continuing. At DLA, the annual price change was less than 5 percent for most parts; but for about 14 percent of the parts, price changes were considerably higher, with a very small percentage experiencing price changes of 1,000 percent or more. Our specific findings at the Navy, DLA, and Marine Corps are described below. Navy: We found that prices for all Navy-managed aviation parts increased at an average annual rate of 12 percent from 1994 to 1999. However, prices for parts with high sales volume increased substantially more, at an average annual rate of 27 percent. From year to year, there were strong fluctuations in prices, making it difficult for the Navy to project price changes. These fluctuations were largely driven by dramatic swings in the surcharge rate. The lack of stability in prices affects ordering units, which may not have sufficient funds budgeted if unexpected price increases occur. In addition, we reported that the Navy had sought to alleviate concerns about high surcharge rates by moving certain overhead costs from the surcharge rate to the repair cost. This approach merely reallocated the overhead costs rather than reducing them. Our follow-on work indicates that prices continue to increase. We will report on reasons for the increases later this year. Our work thus far suggests that the primary reason for the increase is higher material cost associated with repair of the items. Factors we are looking at as drivers for the higher material costs include (1) increased cost of parts ordered from DLA, (2) new, more expensive material being used in the repair process, (3) increased material usage in the repair process, and (4) a change in the mix of parts used in repair. DLA: We found that from 1989 through 1998, an average of 70 percent of the consumable parts requisitioned by DLA's customers experienced an annual price change of less than 5 percent. However, a relatively small number of parts experienced significant annual price increases; that is, increases of 50 percent or more. The proportion of parts with increases of 50 percent or more had been increasing since 1994, reaching nearly 14 percent in 1998. In addition, a very small percentage of the parts experienced extreme increases in price-1,000 percent or more from one year to the next. These extreme price increases are due to outdated or estimated prices in the catalog that DOD units consult when ordering parts. When the catalog price reflects an outdated or estimated price, ordering agencies experience "sticker shock" when confronted with the actual price of the item, which in some cases is significantly higher than the listed—-and anticipated—- price. Since our review, DLA has undertaken a range of efforts to respond to concerns about significant spare part price increases. For example, the agency has recently completed two procurement management reviews concerning price reasonableness determinations and is developing two computer software programs to assist buyers in evaluating contractor- offered prices. These efforts, however, are in various stages of completion and it is too early to assess the results. In March 2001, DLA reported to the secretary of defense on price increases for commercially available spare parts. DLA reported that, from fiscal year 1993 to 2000, materiel costs grew 10.8 percent for competitively purchased commercial items, but increased more than twice as much for noncompetitive purchases. DLA is examining the causes of the price increases and plans to provide the Secretary of Defense with more detailed explanations of cost growth disparities and any remedies. This analysis will be part of DOD's third report to Congress as required in the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999.[Footnote 25] Marine Corps: We focused our work on the ground system spare parts that end-users actually procured during fiscal years 1997-99. We found that the prices for these parts had increased at an average annual rate of about 14 percent from 1995-99. Increases in the surcharge rates charged by the Marine Corps were a major cause of the price escalation. We also found that the Marine Corps did not follow DOD pricing regulations in setting prices and that mathematical and computer errors had occurred in price-setting. As a result, the prices of most parts sold to Marine Corps customers were not correct. The Marine Corps has since corrected its prices and implemented a number of corrective actions pertaining to its pricing methodology. Waiving the Requirement for Certified Cost Data in Negotiations of Contracts: To maximize the value of taxpayer dollars, the federal government generally seeks to compete its contracts. However, DOD buys many unique products and services for which it cannot always rely on competitive forces of the marketplace to get fair prices and values. Instead, it must turn to just a few sources or even a sole source for its procurements. Each year, DOD purchases billions of dollars in weapons systems without competition. In these cases, contractors and subcontractors normally provide the government with cost or pricing data supporting their proposed prices and they certify that the data submitted are accurate, complete, and current. This requirement, established by the Truth-in-Negotiation Act, is meant to put the government on an information parity with sole- source contractors and protect against inflated prices. The act, as amended, specifically provided that the requirements did not apply in "exceptional cases," but it did not include an explanation of what constituted an exceptional case and it has never been amended to define that term. In September 1995, the Federal Acquisition Regulation was amended to allow the head of contracting activities to authorize exceptional case waivers if contracting officers have sufficient information available to negotiate fair and reasonable contract prices without requiring certified data. Concerned about the waiver process, this subcommittee requested that we look at cases where waivers have been made to identify the circumstances in which the waiver was used and techniques used to negotiate prices in place of requiring certified data. We identified 20 cases in which the requirement for certified data was waived, covering fiscal year 2000 contracting actions of $5 million or more at six buying activities. The total contract value for which certified data was waived was $4.4 billion. Contracting officers cited the authority contained in the Federal Acquisition Regulation as the basis for all 20 waivers we reviewed. We are evaluating the agencies' bases for the decisions to waive cost and pricing data. We will be reporting on our findings later this year. This concludes our statement. We appreciate the opportunity to have it placed in the record. If you have questions about our work on service acquisitions and other contracting issues, please call David Cooper at (202) 512-4125, on information technology issues, please call Randolph Hite at (202) 512-3439, and on weapon system issues, please call Katherine Schinasi at (202) 512-4841. [End of section] Related GAO Products: Contract Management Acquisition Reform: Multiple-award Contracting at Six Federal Organizations. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-215]. Washington, D.C.: September 30, 1998. Acquisition Reform: Review of Selected Best-Value Contracts. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-99-93R]. Washington, D.C.: April 14, 1999. Best Practices: Taking a Strategic Approach Could Improve DOD's Acquisition of Services. [hyperlink, http://www.gao.gov/products/GAO-02-230]. January 18, 2002. Contract Management: Few Competing Proposals for Large DOD Information Technology Orders. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-56]. March 20, 2000. Contract Management: Not Following Procedures Undermines Best Pricing Under GSA's Schedule. [hyperlink, http://www.gao.gov/products/GAO-01-125]. November 28, 2000. Contract Management: Service Contracting Trends and Challenges. [hyperlink, http://www.gao.gov/products/GAO-01-1074R]. August 22, 2001. Contract Management: Trends and Challenges in Acquiring Services. [hyperlink, http://www.gao.gov/products/GAO-01-753T]. May 22, 2001. Information Technology: DOD Information Technology: Software and Systems Process Improvement Programs Vary in Use of Best Practices. [hyperlink, http://www.gao.gov/products/GAO-01-116]. March 30, 2001. DOD Systems Modernization: Continued Investment in the Standard Procurement System Has Not Been Justified. [hyperlink, http://www.gao.gov/products/GA0-01-682]. July 31, 2001. DOD's Standard Procurement System: Continued Investment Has Yet to Be Justified. [hyperlink, http://www.gao.gov/products/GAO-02-392T]. February 7, 2002. Information Technology: Architecture Needed to Guide Modernization of DOD's Financial Operations. [hyperlink, http://www.gao.gov/products/GAO-01-525]. May 17, 2001. Information Technology: DLA Should Strengthen Business Systems Modernization Architecture and Investment Activities. [hyperlink, http://www.gao.gov/products/GAO-01-631]. June 29, 2001. Information Technology: Inconsistent Software Acquisition Processes at the Defense Logistics Agency Increase Project Risks. [hyperlink, http://www.gao.gov/products/GAO-02-9]. January 10, 2002. Information Technology: Leading Commercial Practices for Outsourcing of Services. [hyperlink, http://www.gao.gov/products/GAO-02-214]. November 30, 2001. Defense Acquisitions: Collection and Reporting of Information Technology Purchases. [hyperlink, http://www.gao.gov/products/GAO-02-331]. January 28, 2002. Weapon System Acquisition: Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-199]. July 31, 2000. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-99-162]. July 30, 1999. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. [hyperlink, http://www.gao.gov/products/GA0-01-288]. March 8, 2001. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-96-162]. August 26, 1996. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-87]. March 17, 1998. Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-99-206]. August 16, 1999. Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD's Environment. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-56]. February 24, 1998. Defense Acquisition: Best Commercial Practices Can Improve Program Outcomes. [hyperlink, http://www.gao.gov/products/GAO/T-NSIAD-99-116]. March 17, 1999. Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions. [hyperlink, http://www.gao.gov/products/GAO/T-NSIAD-00-137]. April 26, 2000. Defense Acquisition: Improved Program Outcomes Are Possible [hyperlink, http://www.gao.gov/products/T-NSIAD-98-123]. March 18, 1998. Defense Acquisitions: Steps to Improve the Crusader Program's Investment Decisions. [hyperlink, http://www.gao.gov/products/GAO-02-201]. February 25, 2002. Joint Strike Fighter Acquisition: Mature Critical Technologies Needed to Reduce Risks. [hyperlink, http://www.gao.gov/products/GAO-02-39]. October 19, 2001. Pricing Issues: Defense Acquisitions: Prices of Navy Aviation Spare Parts Have Increased. [hyperlink, http://www.gao.gov/products/GAO-01-23]. November 6, 2000. Defense Acquisitions: Price Trends for Defense Logistics Agency's Weapon System Parts. [hyperlink, http://www.gao.gov/products/GAO-01-22]. November 3, 2000. Defense Acquisitions: Prices of Marine Corps Spare Parts Have Increased. [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-123]. July 31, 2000. Additional GAO Reports: High Risk Series: An Update. [hyperlink, http://www.gao.gov/products/GAO-01-263]. January 2001. Major Management Challenges and Program Risks: Department of Defense. [hyperlink, http://www.gao.gov/products/GAO-01-244]. January 2001. [End of section] Footnotes: [1] U.S. General Accounting Office, High Risk Series: An Update, [hyperlink, http://www.gao.gov/products/GAO-01-263] (Washington, D.C.: Jan. 2001). [2] U.S. General Accounting Office, Best Practices: Taking A Strategic Approach Could Improve DOD's Acquisition of Services, [hyperlink, http://www.gao.gov/products/GAO-02-230] (Washington, D.C.: Jan. 18, 2002). [3] SEI is a nationally recognized, federally funded research and development center established at Carnegie Mellon University to advance the state of software engineering, development, and acquisition practices. [4] Software Acquisition Capability Maturity Model (SA-CMM), Version 1.02. Capability Maturity ModeISM is a service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and Trademark Office. [5] IDEALSM is a service mark of Carnegie Mellon University and stands for initiating, diagnosing, establishing, acting, and leveraging. [6] Clinger-Cohen Act of 1996, P.L. 104-106, and Office of Management and Budget Circular A-130 (Nov. 30, 2000). [7] These architectures systematically capture-—in useful models, diagrams, and narrative-—the relevant breadth and depth of the mission- based mode of operation for a given enterprise. Moreover, they describe these operations in both (1) logical terms, such as interrelated processes, information needs and flows, work locations, and system applications, and (2) technical terms, such as hardware, software, data, communications, and security attributes, and standards. They also provide these perspectives both for the enterprise's current or "as is" environment for its target or "to be" environment, as well as a plan or road map for moving between the two environments. [8] Chief Information Officer's Council, A Practical Guide to Federal Enterprise Architecture, Version 1.0 (February 2001). [9] U.S. General Accounting Office, Information Technology: Leading Commercial Practices for Outsourcing of Services, [hyperlink, http://www.gao.gov/products/GAO-02-214] (Washington, D.C.: Nov. 30, 2001). [10] BSM is intended to modernize DLA's materiel management business functions, thereby enabling the agency to manage supply chains. BSM is based on commercially available software products and is expected to cost $658 million from fiscal years 2000 through 2005. FAS is intended to help the Defense Energy Support Center annually manage about $5 billion in contracts with petroleum suppliers. FAS also relies on a commercially available software package and is expected to cost $293 million from fiscal year 1995 through 2002. [11] U.S. General Accounting Office, Information Technology: Inconsistent Software Acquisition Processes at the Defense Logistics Agency Increase Project Risks, [hyperlink, http://www.gao.gov/products/GAO-02-9](Washington, D.C.: Jan. 10, 2002). [12] U.S. General Accounting Office, DOD Information Technology: Software and Systems Process Improvement Programs Vary in Use of Best Practices, [hyperlink, http://www.gao.gov/products/GAO-01-116] (Washington, D.C.: Mar. 30, 2001). [13] U.S. General Accounting Office, DOD Systems Modernization: Continued Investment in the Standard Procurement System Has Not Been Justified, [hyperlink, http://www.gao.gov/products/GAO-01-682] (Washington, D.C.: July 31, 2001) and U.S. General Accounting Office, Information Technology: DLA Should Strengthen Business Systems Modernization Architecture and Investment Activities, [hyperlink, http://www.gao.gov/products/GAO-01-631] (Washington, D.C.: June 29, 2001). [14] SPS is intended to be DOD's single, standard procurement system and is expected to cost $3.7 billion over a 10-year period. [15] U.S. General Accounting Office, Information Technology: Architecture Needed to Guide Modernization of DOD's Financial Operations, [hyperlink, http://www.gao.gov/products/GAO-01-525] (Washington, D.C.: May 17, 2001) and U.S. General Accounting Office, Information Technology: DLA Should Strengthen Business Systems Modernization Architecture and Investment Activities, [hyperlink, http://www.gao.gov/products/GAO-01-631] (Washington, D.C.:, June 29, 2001). [16] U.S. General Accounting Office, Joint Strike Fighter Acquisition: Mature Critical Technologies Needed to Reduce Risks, [hyperlink, http://www.gao.gov/products/GAO-02-39] (Washington, D.C.: Oct. 19, 2001). [17] U.S. General Accounting Office, Defense Acquisitions: Steps to Improve the Crusader Program's Investment Decisions, [hyperlink, http://www.gao.gov/products/GAO-02-201] (Washington D.C.: Feb. 25, 2002). [18] P.L. 103-355 (Oct. 13, 1994). [19] U.S. General Accounting Office, Acquisition Reform: Multiple- award Contracting at Six Federal Organizations, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-215] (Washington, D.C.: Sept. 30, 1998). [20] U.S. General Accounting Office, Contract Management: Few Competing Proposals for Large DOD Information Technology Orders, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-56] (Washington, D.C.: Mar. 20, 2000). [21] Office of the Inspector General, Department of Defense, Multiple Award Contracts for Services (Sept. 30, 2001). [22] P.L. 106-65 (Oct. 5, 1999). [23] Consumable items are those that are consumed in use or discarded when worn out or broken because they cannot be cost-effectively repaired. [24] U.S. General Accounting Office, Defense Acquisitions: Prices of Marine Corps Spare Parts Have Increased, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-00-123] (Washington, D.C.: July 31, 2000); U.S. General Accounting Office, Defense Acquisitions: Price Trends for Defense Logistics Agency's Weapon System Parts, [hyperlink, http://www.gao.gov/products/GAO-01-22] (Washington, D.C.: Nov. 3, 2000); U.S. General Accounting Office, Defense Acquisitions: Prices of Navy Aviation Spare Parts Have Increased, [hyperlink, http://www.gao.gov/products/GAO-01-23] (Washington, D.C.: Nov. 6, 2000). [25] P.L. 105-261. [End of section]