This is the accessible text file for GAO report number GAO-03-371 entitled 'Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects' which was released on April 25, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate: April 2003: Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects: GAO-03-371: GAO Highlights: Highlights of GAO-03-371, a report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate Why GAO Did This Study: Given the magnitude of its reported spending on information technology (IT) services—more than $6.2 billion in fiscal year 2001—it is critical that the Department of Defense (DOD) adopt effective practices for acquiring IT services. GAO researched leading commercial practices for the outsourcing of IT services, and, in November 2001, published a framework consisting of seven phases that span the full range of activities that are performed during the outsourcing of those services (this is an acquisition in which a client organization transfers responsibility for performing services to an external provider). GAO was asked to determine (1) the extent to which selected DOD projects for outsourcing IT services use leading commercial practices as specified in GAO’s framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department. What GAO Found: The projects in GAO’s review substantially used leading commercial practices as specified in GAO’s framework for outsourcing IT services. Specifically, the agencies fully implemented 88 percent of the practices (not including practices not applicable to a particular project). This framework consists of practices organized into seven phases: (I) determine sourcing strategy, (II) define operational model, (III) develop the contract, (IV) select the provider(s), (V) transition to provider(s), (VI) manage the performance of the provider(s), and (VII) ensure services are provided. The figure below shows the percentage of practices that were implemented in each phase. Although DOD has acted on gathering and disseminating lessons learned and commercial leading practices related to general acquisition issues, its actions have generally not been focused on outsourcing or on sharing the lessons learned from IT services outsourcing across the department. By not systematically capturing and disseminating such information across the department, DOD is losing the opportunity to leverage the knowledge gained on IT services projects like those in GAO’s review. Lessons learned that are pragmatic and easily accessible would give DOD managers a more informed understanding of important issues to be addressed when making outsourcing decisions, as well as the factors to be considered to help ensure the success of these endeavors. What GAO Recommends: GAO is making recommendations to the Secretary of Defense aimed at leveraging lessons learned across the department from its components’ IT outsourcing experiences. DOD agreed that capturing lessons learned related to IT outsourcing initiatives is important and stated that it intends to explore a variety of mechanisms to do so. DOD’s plans are consistent with our recommendations. www.gao.gov/cgi-bin/getrpt?GAO-03-371. To view the full report, including the scope and methodology, click on the link above. For more information, contact Randolph C. Hite at (202) 512-3439 or hiter@gao.gov. [End of section]: Letter: Results in Brief: Background: Projects Substantially Used Leading Commercial Practices: Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects: Conclusions: Recommendations: Agency Comments and Our Evaluation: Appendixes: Appendix I: Objectives, Scope, and Methodology: Appendix II: Projects’ Implementation of Phase I: Determine Sourcing Strategy: Appendix III: Project’s Implementation of Phase II: Define Operational Model: Appendix IV: Project’s Implementation of Phase III: Develop the Contract: Appendix V: Projects’ Implementation of Phase IV: Select the Provider(s): Appendix VI: Projects’ Implementation of Phase V: Transition to Provider(s): Appendix VII: Projects’ Implementation of Phase VI: Manage Provider(s) Performance: Appendix VIII: Projects’ Implementation of Phase VII: Ensure Services are Provided: Appendix IX: Comments from the Department of Defense: GAO Comments: Appendix X: GAO Contact and Staff Acknowledgments: GAO Contact: Staff Acknowledgments: Tables: Table 1: Definition of Phases for IT Outsourcing: Table 2: Profile of Outsourcing Projects Reviewed: Table 3: Percentage of Practices Implemented, by Project: Table 4: Summary of Projects' Use of Phase I Practices: Table 5: Summary of Projects' Use of Phase II Practices: Table 6: Summary of Projects' Use of Phase III Practices: Table 7: Summary of Projects' Use of Phase IV Practices: Table 8: Summary of Projects' Use of Phase V Practices: Table 9: Summary of Projects' Use of Phase VI Practices: Table 10: Summary of Projects' Use of Phase VII Practices: Figures: Figure 1: Percentage of Practices Implemented in Each Phase: Figure 2: Roles of the Client and Provider in an Outsourcing Relationship: Figure 3: GAO's Framework for Outsourcing IT Services: Figure 4: Use of Leading Commercial Practices for Outsourcing of IT Services, by Project: Figure 5: Percentage of Practices Implemented in Each Phase: Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy: Figure 7: Project Implementation of Phase II: Define Operational Model: Figure 8: Project Implementation of Phase III: Develop the Contract: Figure 9: Project Implementation of Phase IV: Select the Provider(s): Figure 10: Project Implementation of Phase V: Transition to Provider(s): Figure 11: Project Implementation of Phase VI: Managing Provider Performance: Figure 12: Project Implementation of Phase VII: Ensure Services Are Provided: Abbreviations: C3I: command, control, communications, and intelligence: C4: command, control, communications, and computer : CIO: chief information officer: DOD: Department of Defense: IT: information technology: IT/IS: Information Technology/Information Services: MHS/ITO: Military Health System/Information Technology Organization: NETCOM: Network Enterprise Technology Command: NIMA: National Imagery and Mapping Agency: NMCI: Navy and Marine Corps Intranet: : OMB: Office of Management and Budget: RFP: request for proposals: SLA: service-level agreement: TAC-SWA: Total Army Communications--Southwest Asia: This is a work of the U.S. Government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. It may contain copyrighted graphics, images or other materials. Permission from the copyright holder may be necessary should you wish to reproduce copyrighted materials separately from GAO’s product. Letter April 25, 2003: The Honorable John Ensign Chairman The Honorable Daniel K. Akaka Ranking Minority Member Subcommittee on Readiness and Management Support Committee on Armed Services United States Senate: The Department of Defense (DOD) is the government's largest purchaser of information technology (IT) services, such as desktop support, network operations, and software development services. In fiscal year 2001, DOD reportedly obligated more than $6.2 billion on IT services,[Footnote 1] and this amount is expected to grow substantially. Given the magnitude of DOD's spending on such services, it is critical that the department adopt effective practices for acquiring IT services. Since 1996, we have conducted a series of studies for the Senate Committee on Armed Services concerning how DOD can improve its acquisition processes by adopting proven practices of leading commercial organizations. In this vein, in November 2001, we issued a guide that organized leading commercial practices for the outsourcing[Footnote 2] of IT services into a framework of seven phases that span the full range of activities that are performed during IT services outsourcing.[Footnote 3] This report responds to your request that we determine (1) the extent to which selected DOD IT services outsourcing projects use leading commercial practices as specified in our framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department. To address the first objective, we selected five projects from a group of projects to outsource IT services that were identified by the military services and other DOD components.[Footnote 4] We then asked the component responsible for each project to perform a self-assessment against selected practices in our framework for outsourcing IT services.[Footnote 5] Next, we obtained and reviewed agencies' supporting documentation and interviewed the appropriate agency and provider project officials to independently determine whether a practice was met. To address the second objective, we reviewed applicable DOD approaches for capturing and disseminating lessons learned from IT services outsourcing projects and interviewed the applicable acquisition and IT officials. Details of our objectives, scope, and methodology are discussed in appendix I. Results in Brief: The projects in our review substantially used leading commercial practices as specified in our framework on outsourcing IT services. Specifically, the agencies fully implemented 88 percent of the practices.[Footnote 6] This framework consists of practices organized into seven phases that span the full range of activities that are performed during IT outsourcing: (I) determine sourcing strategy, (II) define operational model, (III) develop the contract, (IV) select the provider(s), (V) transition to provider(s), (VI) manage provider(s) performance, and (VII) ensure services are provided. Figure 1 illustrates the percentage of practices that were followed in each phase. Collectively, the projects fully implemented from 70 to 97 percent of the practices in each phase. Figure 1: Percentage of Practices Implemented in Each Phase: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Although implementing the leading commercial practices in our framework does not guarantee the success of an outsourcing project, the consensus view of the leading commercial activities that we studied is that these practices are the most critical to success when acquiring IT services.[Footnote 7] In addition, not implementing or only partially implementing particular practices can produce negative consequences or add risk to a project. For example, the Department of the Navy project's baseline of its existing environment (a phase I practice) was limited because it did not include an assessment of its legacy applications. Instead, project officials decided to rely on a preexisting inventory developed to address the Year 2000 challenge. The Navy subsequently found that it had substantially underestimated the number of legacy applications, which, according to program officials, later contributed to the transition period slipping from 2-½ years to 3-½ years. As DOD gathers more experience in implementing projects for outsourcing IT services, it can benefit from leveraging the lessons derived from these initiatives. For example, the projects in our review have identified lessons learned in such areas as transitioning to the provider and partnering with the provider. Although DOD has taken action to gather and disseminate lessons learned and best practices on general acquisition issues, these efforts generally do not focus on outsourcing or include sharing the lessons learned from IT outsourcing projects across the department. By not capturing and disseminating such information in a systematic manner across the department, DOD is losing the opportunity to leverage the knowledge gained on IT services projects like those in our review. We are making recommendations to the Secretary of Defense aimed at leveraging lessons learned across the department from its components' IT services outsourcing experiences. In written comments on a draft of this report signed by the DOD Principal Director, Deputy Assistant Secretary of Defense (Deputy Chief Information Officer), DOD agreed that capturing lessons learned in the development and implementation of its IT outsourcing initiatives is important to continually improving its outsourcing methods and results achieved. The department also stated that it intends to explore a variety of mechanisms for best exploiting lessons learned from its IT outsourcing initiatives. We agree that it is prudent to consider alternative means to leveraging these lessons learned, and we believe that this is consistent with our recommendations. Background: To protect the security of the United States, DOD relies on a complex array of computer-dependent and mutually supportive organizational components, including the military services, Commanders in Chief, and Defense agencies. As such, it invests tens of billions of dollars each year in a broad array of computer systems, which include weapon systems, command and control systems, satellite systems, inventory management systems, transportation management systems, health systems, financial systems, personnel systems, and payment systems. In addition, DOD spends billions of dollars annually on IT services, which include database management, help-desk operations, software maintenance, and network services. In fiscal year 2001, DOD reportedly obligated more than $6.2 billion on IT services alone.[Footnote 8] Decisions regarding the purchasing of services are critical to ensuring the effectiveness of DOD's operations as well as those of the government as a whole. Our November 2001 report recognizes the importance of such sourcing decisions and provides a framework that spans the full range of activities that are performed during IT services outsourcing.[Footnote 9] At the same time, governmentwide policies, initiatives, and challenges exist that significantly influence the government's sourcing decisions. GAO's Framework for Outsourcing IT Services: Outsourcing of IT services has become increasingly popular in both the public and private sector. For example, according to the Giga Information Group, Inc., a leading research firm, such outsourcing is expected to grow an average of 5 to 6 percent in 2003.[Footnote 10] The federal sector's outsourcing is predicted to rise at an even greater rate. For example, INPUT, an IT market research firm, forecasts that defense IT outsourcing will increase about 143 percent between fiscal years 2002 and 2007.[Footnote 11] IT outsourcing involves the activities associated with acquiring services from one or more external providers. During outsourcing, a client organization transfers responsibilities for performing one or more IT services to one or more external providers. This responsibility is executed through control and management of the processes, people, and technology associated with these services. Figure 2 depicts the roles of the client and provider organizations in an outsourcing relationship. Figure 2: Roles of the Client and Provider in an Outsourcing Relationship: [See PDF for image] [End of figure] Our November 2001 guide on leading commercial practices for outsourcing IT services provides a generic framework of practices from leading commercial organizations that can improve purchasing decisions and manage the resulting government/provider relationship.[Footnote 12] The framework is represented in figure 3 as a hierarchy of phases, practices, and critical success factors. Figure 3: GAO's Framework for Outsourcing IT Services: [See PDF for image] Note: The arrow from phase VII to phase I represents the need to reflect on lessons learned from previous phases. The arrows between phase III and IV represent the iterative nature of developing the contract and selecting the provider. Although there is a logical order to the sequence of the common phases, the order of the practices within each phase does not imply any priority or sequence. [End of figure] Table 1 provides a definition of each phase of the framework. Each of the phases has specific practices associated with it. Implementing these practices does not guarantee the success of an outsourcing project. However, our November 2001 study reflected a consensus view that these practices were the most critical to success when IT services are being acquired.[Footnote 13] Restated, application of these practices increases the probability of a successful outsourcing project. Table 1: Definition of Phases for IT Outsourcing: Phase number: I; Title: Determine sourcing strategy; Definition: Client organizations determine whether internal capability or external expertise can more effectively meet IT needs. Phase number: II; Title: Define operational model; Definition: Client organizations formalize executive leadership, team composition, client responsibilities, and operating relationships between client and provider organizations. Phase number: III; Title: Develop the contract; Definition: Client organizations establish the legal terms for the IT outsourcing relationship. Phase number: IV; Title: Select the provider(s); Definition: Client organizations find one or more providers who can help them reach their IT outsourcing goals. Phase number: V; Title: Transition to provider(s); Definition: Client organizations transfer responsibility of IT functions to one or more providers. Phase number: VI; Title: Manage provider(s) performance; Definition: Client organizations make sure each provider is meeting performance requirements. Phase number: VII; Title: Ensure services are provided; Definition: Client organizations make sure that services are provided and end-user needs are met. Source: GAO. [End of table] The organizations that we studied also identified certain capabilities (identified as critical success factors) that were essential for implementing the practices identified in our framework. First, executive leadership strengthens the interaction between executive management and the employees of the client organization. Second, partner alignment strengthens the interaction between the client and provider organization at the executive level, which ensures that the goals and objectives of these organizations support each other. Third, relationship management strengthens the interaction between the client and provider organization at the operational level. Influences on Government Sourcing Decisions: The federal government is one of the world's largest users of services. Because of the large dollar value and the number of private-and public- sector jobs involved, deciding whether the public or private sector would be the most appropriate provider of the services the government needs (IT or otherwise) is an important, and often highly charged, question. Among the factors that agencies must consider as they determine how best to meet their missions is whether the public or private sector would be the most appropriate provider of the services the government needs. Phase I of our framework, determine the sourcing strategy, addresses the client's assessment of whether expertise from within or outside of the organization can more effectively meet the client's needs. The National Defense Authorization Act for Fiscal Year 2001 required the Comptroller General of the United States to convene a panel of experts to study the current process used by the government to make sourcing decisions. The resulting Commercial Activities Panel conducted a year-long study and heard repeatedly about the importance of competition and its central role in fostering economy, efficiency, and continuous performance improvement. In particular, the panel reviewed the government's implementation of the Office of Management and Budget's (OMB) Circular A-76, which sets forth federal policy for determining whether federal employees or private contractors will perform commercial activities for the government.[Footnote 14] Circular A-76 (1) outlines conditions under which agencies are permitted to perform a commercial activity with government employees or by contract and (2) provides guidance for whether, and if so, how, agencies should conduct a cost comparison when they are considering transferring the performance of commercial activities from the public to the private sector (or vice versa). The panel reported that there were positive elements to Circular A-76 but that both federal employees and private firms complained that it does not meet the standard of a clear, transparent, and consistently applied process. For example, both federal employees and private firms criticized the Circular A-76 process as unequal and therefore unfair. The Commercial Activities Panel strongly supported continued emphasis on competition and concluded that whenever the government is considering converting work from one sector to another, public/private competitions should be the norm. In addition, the panel made four recommendations, including that all sourcing decisions be consistent with the principles adopted unanimously by the panel, such as the principle that federal policy provide for accountability in connection with all sourcing decisions.[Footnote 15] As part of the administration's efforts to implement the recommendations of the Commercial Activities Panel, OMB has published proposed changes to Circular A-76. Key highlights of the proposed changes include presuming that all functions are commercial in nature unless they are justified as inherently governmental;[Footnote 16] limiting the length of time for competitions; and emphasizing awarding contracts on the basis of best value, not just lowest cost. Best value allows the contracting official to consider technical superiority, quality, innovation, and past performance as well as price. However, we reported that there are several areas in which the proposed revisions to the circular are not consistent with the principles or recommendations of the Commercial Activities Panel.[Footnote 17] Specifically, the proposed revision does not include a link between sourcing policy and agency missions, has unnecessarily complicated source selection procedures, contains certain unrealistic time frames, and includes insufficient guidance on calculating savings. Beyond the Commercial Activities Panel, other bodies have identified challenges that the federal government faces in reaching and executing effective sourcing decisions. For example, members of the Coalition for Government Procurement, the Professional Services Council, and the Information Technology Association of America told us that organizational culture is one of the biggest differences between the commercial sector and the federal government and one of the greatest barriers to the government's use of commercial practices. Also, as we have previously reported, moving to outsourcing solutions can involve a cultural change for government organizations because it may require a change to an agency's operating model, such as using a contractor to provide IT services previously performed by government staff or using a performance-based contract.[Footnote 18] This view was echoed by a 2001 study of DOD competitive sourcing that found cultural, process, execution, and training barriers.[Footnote 19] The study stated that these barriers need to be understood and mitigated before the benefits of outsourcing can be fully realized. Barriers such as these can be overcome by strong executive leadership, which is a critical success factor in our framework. Another challenge is creating a productive agency/provider relationship--another critical success factor in our framework. According to a report sponsored by the PricewaterhouseCoopers Endowment for the Business of Government, such public/private partnerships are based on trust, commitment to problem or conflict resolution, and the recognition that flexibility is necessary and that the relationship will evolve and change over time.[Footnote 20] If deadlines are not met, or public agency goals change with differing political climates, the partners need to discuss the basis of the partnership and construct a different relationship. Our prior report on desktop outsourcing found that developing a productive agency/contractor relationship is not always easy.[Footnote 21] Both sides must recognize and understand each other's underlying motives and strive to achieve established expectations. Finally, human capital issues are another challenge facing federal agencies that affect their ability to implement outsourcing. Our framework recognizes the importance of having the right skills in place to support the outsourcing relationship. However, as we have previously reported, procurement reforms and technological changes have placed unprecedented demands on the acquisition workforce.[Footnote 22] Contracting personnel are now expected to have a much greater knowledge of market conditions, industry trends, and the technical details of the commodities and services they procure. The Commercial Activities Panel report stated that developing and maintaining a skilled acquisition workforce is the critical first step in managing this more complex procurement environment. The panel also reported that DOD bore the brunt of a 22 percent downsizing of the federal acquisition workforce in the last decade, going from 96,000 staff in 1991 to about 68,000 in fiscal year 2001. Addressing human capital issues is not just a matter of the size of the workforce; it is also a knowledge and skills issue. According to the Commercial Activities Panel, it is critically important that federal agencies adequately address human capital needs in meeting the current and emerging needs of government and its citizens in the most effective, efficient, and economical manner possible. This will require increased emphasis on training and development, particularly in the area of technology. Description of Five Projects Reviewed: The five projects in our study varied in how they approached outsourcing IT services, such as in using various solicitation methods, including holding a public/private competition under the policies outlined in OMB Circular A-76 or carrying out a negotiated competitive procurement. In addition, the types of services being outsourced differed: services ranged from the narrowly focused (e.g., help-desk services) to the very broad (e.g., enterprisewide end-to-end information services); contract terms ranged from 5 to 15 years (assuming all option years are exercised); and estimated contract values ranged from $23 million to $8.8 billion. Table 2 provides information on the variety of IT services and outsourcing approaches taken by the projects. : Table 2: Profile of Outsourcing Projects Reviewed: Agency/Project: Air Force/Kirtland Air Force Base's Command, Control, Communications, and Computer (C4) Services; Solicitation method: Competitive, under OMB Circular A-76 and small business set-aside rules; Date of contract award: April 2000; Contract term: 1 year, with 4 option years; Contract type: Firm, fixed-price; Estimated total contract value[A]: $23 million; Project description: Management, operations, and maintenance of command, control, communications, and computer systems, multimedia services, and information management for the Kirtland Air Force Base. Agency/Project: Army/Network Enterprise Technology Command's (NETCOM) Total Army Communications--Southwest Asia (TAC-SWA); Solicitation method: Negotiated competitive solicitation; Date of contract award: March 2001; Contract term: 1 year, with 4 option years; Contract type: Firm, fixed-price; Estimated total contract value[A]: $204 million; Project description: Operation and maintenance services, including repair, installation, and supply, for communications equipment in Southwest Asia. Agency/Project: Military Health System/Information Technology Organization (MHS/ITO) Help Desk; Solicitation method: Negotiated competitive solicitation; Date of contract award: June 2001; Contract term: 1 year, with 7 option years; Contract type: Firm, fixed-price with incentive awards; Estimated total contract value[A]: $71 million; Project description: Call and help-desk services for all MHS software applications. Agency/Project: Department of the Navy/Navy and Marine Corps Intranet (NMCI); Solicitation method: Negotiated competitive solicitation; Date of contract award: October 2000; Contract term: 7 years, with an option for 3 additional years; Contract type: Firm, fixed-price with incentive awards; Estimated total contract value[A]: $8.8 billion; Project description: Department of the Navy-wide end-to-end information services through a common computing and communication environment. Agency/Project: National Imagery and Mapping Agency's (NIMA) Information Technology/ Information Services (IT/IS); Solicitation method: Sole-source, using a statutory preferential provider (Alaska native corporation)[B]; Date of contract award: December 2001; Contract term: 1 year, with 14 option years; Contract type: Cost plus award fees; Estimated total contract value[A]: $2.1 billion; Project description: NIMA-wide IT/IS support services for printing, digital replication, networks, distributed and centralized systems and services, video and voice communications, information research, and help-desk operations.[C]. Source: DOD. [A] Estimated value if all option years are exercised. [B] NIMA performed an OMB Circular A-76 analysis and, on the basis of this analysis, implemented a direct conversion to a preferential provider rather than holding a public/private competition or obtaining an agency cost-comparison waiver. [C] As of February 24, 2003, NIMA had transitioned four of these functions to the provider: printing, digital replication, video and voice communications, and help-desk operations. [End of table] Projects Substantially Used Leading Commercial Practices: As illustrated in figure 4, the five IT services projects substantially used leading commercial practices. Specifically, each project used at least 76 percent of the practices.[Footnote 23] Reasons for projects implementing different percentages of the practices include differences in their individual circumstances and objectives. For example, the Army's Total Army Communications--Southwest Asia (TAC-SWA) project, which used the fewest number of practices and had the largest number of practices that were not applicable, was largely a continuation of an existing approach that already relied on the private sector but with fewer providers. In contrast, the National Imagery and Mapping Agency's (NIMA) Information Technology/ Information Services (IT/IS) project, which fully or partially implemented all of the applicable practices, involved a significant operational shift (e.g., functions previously performed by NIMA staff are now performed by a contractor) and was intended to result in substantial process improvements. In addition, the three projects that implemented the largest percentage of practices also used third-party assistance--including employing a contractor with sourcing expertise--to help formulate their sourcing strategy, which could account for the extent of their compliance. Figure 4: Use of Leading Commercial Practices for Outsourcing of IT Services, by Project: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] In addition, figure 5 illustrates that project compliance extended to each of the phases of our framework. Collectively, the projects fully implemented from 70 to 97 percent of the practices in each phase. Phase I, determine sourcing strategy, had the lowest percentage of practices implemented by the projects (70 percent). This result is not inconsistent with a recent Giga Information Group, Inc., survey, which found that only half of the respondent organizations had documented an IT sourcing strategy.[Footnote 24] This approach carries risk since phase I sets the tone for the outsourcing initiative within the client organization. Figure 5: Percentage of Practices Implemented in Each Phase: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] In addition, collectively the projects fully implemented 88 percent of the practices (see table 3).[Footnote 25] Table 3: Percentage of Practices Implemented, by Project: Phase: Phase I: Determine sourcing strategy (6 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 50; Percentage of practices implemented[A]: Army TAC-SWA project: 33; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 83; Percentage of practices implemented[A]: Navy NMCI project: 83; Percentage of practices implemented[A]: NIMA IT/IS project: 100; Percentage of practices implemented[A]: Overall: 70. Phase: Phase II: Define operational model (13 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 92; Percentage of practices implemented[A]: Army TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 77; Percentage of practices implemented[A]: NIMA IT/IS project: 100; Percentage of practices implemented[A]: Overall: 90. Phase: Phase III: Develop the contract (16 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 64; Percentage of practices implemented[A]: Army TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 94; Percentage of practices implemented[A]: NIMA IT/IS project: 100; Percentage of practices implemented[A]: Overall: 89. Phase: Phase IV: Select provider(s) (7 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 86; Percentage of practices implemented[A]: Army TAC-SWA project: 86; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS project: 71; Percentage of practices implemented[A]: Overall: 89. Phase: Phase V: Transition to provider(s) (11 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 82; Percentage of practices implemented[A]: Army TAC-SWA project: 88; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 82; Percentage of practices implemented[A]: NIMA IT/IS project: 100; Percentage of practices implemented[A]: Overall: 90. Phase: Phase VI: Manage provider(s) performance (11 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 73; Percentage of practices implemented[A]: Army TAC-SWA project: 73; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 91; Percentage of practices implemented[A]: NIMA IT/IS project: 82; Percentage of practices implemented[A]: Overall: 84. Phase: Phase VII: Ensure services are provided (6 practices).; Percentage of practices implemented[A]: Air Force C4 Services project: 83; Percentage of practices implemented[A]: Army TAC-SWA project: 100; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 100; Percentage of practices implemented[A]: Navy NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS project: 100; Percentage of practices implemented[A]: Overall: 97. Phase: Overall; Percentage of practices implemented[A]: Air Force C4 Services project: 76; Percentage of practices implemented[A]: Army TAC-SWA project: 79; Percentage of practices implemented[A]: MHS/ITO Help Desk project: 99; Percentage of practices implemented[A]: Navy NMCI project: 89; Percentage of practices implemented[A]: NIMA IT/IS project: 94; Percentage of practices implemented[A]: Overall: 88. Source: GAO. [A] These calculations do not include practices that were not applicable to a particular project. [End of table] The following provides additional information on the projects' implementation of each phase of our framework. * Phase I: Determine sourcing strategy. In the first phase of our outsourcing framework, the client organization determines whether internal capability or external expertise can more effectively meet its IT needs. The purpose of a sourcing strategy is to achieve the optimal balance between internal and external capabilities, activities, processes, and services to ensure the achievement of strategic business objectives at the lowest risk.[Footnote 26] The five projects' implementation of this phase was uneven. In particular, two of the six practices in this phase were fully implemented by all five projects, but the other four practices were not. Among the practices that were implemented by all of the projects was determining the business reasons for outsourcing. In addition, the three projects that implemented the largest percentage of practices in our framework--the Military Health System/Information Technology Organization (MHS/ITO) Help Desk, the Department of the Navy's Navy and Marine Corps Intranet (NMCI), and NIMA IT/IS projects--used the third-party assistance practice in this phase to help formulate their sourcing strategy, which could account for the extent of their compliance. For example, the MHS/ITO Help Desk project, which implemented the largest percentage of practices, worked with the Department of the Interior's GovWorks Program,[Footnote 27] the Defense Acquisition University staff, and a private-sector contractor to obtain expertise on sourcing strategies. The practice in this phase that was the most unevenly implemented was the benchmarking[Footnote 28] and baselining of existing internal services. Of the five projects in our review, (1) one fully benchmarked and baselined the productivity of the activity being outsourced before making the final sourcing decision, (2) two partially baselined their existing activities, and (3) two did not perform benchmark and baseline analyses at all. The agencies' reasons for not fully implementing this practice included that an executive decision had been made to conduct a public/private competition following the OMB Circular A-76 policy, so such an analysis would not have affected the sourcing decision, or that available documentation to perform such an analysis was limited. Leading research firms suggest benchmarking and baselining the entity's current processes before outsourcing because only then would it be able to determine whether the arrangement has been successful.[Footnote 29] In addition, the risk of not fully baselining the existing environment is illustrated by the NMCI project. Specifically, the NMCI project's baseline of its existing environment was limited because it did not include an assessment of its legacy applications since project officials decided to rely on a preexisting inventory developed to address the Year 2000 challenge. The Navy subsequently found that it had substantially underestimated its number of legacy applications, that, according to program officials, later contributed to the transition period slipping from 2-½ years to 3-½ years. Appendix II provides additional information on projects' implementation of the practices comprising this phase. * Phase II: Define operational model. The operational model is an important mechanism for an organization to compare its plans with the expectations that were set when the decision to outsource was made and to ascertain whether these plans will enable the organization to meet expectations. The five projects had largely implemented the 13 practices contained in this phase. Specifically, about 90 percent[Footnote 30] of the practices were implemented. For example, all projects implemented the practice that executive leadership be established to facilitate the outsourcing effort. NIMA, for instance, formed a strategic sourcing office to oversee the IT/IS project. Another practice--training the provider on the organization's business environment and goals--was fully implemented by one project (in two cases, the practice was not applicable). One project that did not fully implement this practice was the Air Force's Kirtland Air Force Base's Command, Control, Communications, and Computer (C4) Services project. Although some training was provided (e.g., Kirtland held an orientation session for potential bidders), provider officials stated that they did not receive adequate training, which made the transition period more difficult. Appendix III provides additional information on projects' implementation of the practices comprising this phase. * Phase III: Develop the contract. A well-written contract is necessary for the outsourcing organization to meet its requirements while allowing the service provider to make a fair profit. It sets the expectations for service levels, delivery of essential services, and continuous improvement and should protect the interests of all parties. The five projects largely implemented the practices in this phase. Specifically, about 89 percent[Footnote 31] of the practices were implemented in this phase, and two projects (the MHS/ITO Help Desk and NIMA IT/IS projects) implemented all of the practices. Several practices in this phase address performance requirements. For example, all five projects implemented the practices that called for basing performance requirements on business requirements and reviewing and updating them periodically. One practice that was not fully implemented by two projects was including performance measures that address both technical and end-user satisfaction aspects of performance. For example, the Army TAC-SWA project included technical performance measures in its contract but not measures related to end-user satisfaction, even though the contract included help-desk services. According to the project official that developed the performance work statement in the contract, the command did not include customer satisfaction measures because it did not think that it was necessary to have a performance standard for the help-desk service. However, without such measures, the agency does not have a contractual standard with which to judge the provider's performance. Appendix IV provides additional information on projects' implementation of the practices comprising this phase. * Phase IV: Select the provider(s). Critical to the success of any outsourcing project for IT services is identifying potential providers and ultimately selecting a provider(s) that will best meet the needs of the agency. The five projects had largely implemented the seven practices contained in this phase. About 89 percent of the practices were implemented in this phase and two projects (the MHS/ITO Help Desk and Navy NMCI projects) used each applicable practice. For example, the five projects implemented the practice related to conducting due diligence activities to verify provider capabilities before signing the contract. In the case of the Army TAC-SWA project, the Network Enterprise Technology Command (NETCOM) evaluated the provider's financial and past performance information. The Department of the Navy's NMCI project also evaluated bidders' past performance and performed reference checks. Moreover, as part of its due diligence activities, the Navy required all bidders to demonstrate that they had experience in implementing large seat management contracts.[Footnote 32] A practice that was not implemented by two projects was using third-party assistance when selecting the provider. Projects that did not implement this practice believed that they had adequate in-house expertise with outsourcing, making third-party assistance unnecessary. However, because third-party assistance provides an independent resource that can suggest options or processes that the client organization may not be aware of, these projects may have missed an opportunity to implement their outsourcing projects more effectively. Appendix V provides additional information on projects' implementation of the practices comprising this phase. * Phase V: Transition to provider(s). This phase focuses on the client organization's transfer of the IT function to one or more providers. As part of this transition, the clear definition of responsibilities and the careful consideration of employees' needs matched against the client organization's needs enable both the client and provider to focus on execution and give staff confidence in their future employment. The five projects largely implemented the 11 practices associated with this phase. Specifically, about 90 percent[Footnote 33] of the practices were implemented in this phase, and two projects implemented all of the practices (the MHS/ITO Help Desk and NIMA IT/IS projects). Several practices in this phase address dealing with employees affected by the outsourcing projects. For example, in the four projects in which federal employees were affected, the projects provided assistance to those who did not want to transfer to the provider, including helping to place them in other positions and helping with résumé writing. A related practice is to clearly communicate to all employees what is going to happen and when it is going to happen. Two projects did not fully implement this practice. For example, the Navy used its normal chain of command to communicate transition information, but found that implementation of this practice was uneven. As a result, some staff did not know current information about how NMCI would affect them until the provider was ready to contact them regarding their possible transition to the contractor. However, according to the NMCI's Director's office, this problem was somewhat mitigated by the provider's Web site that provides transition information to all NMCI customers/users. Appendix VI provides additional information on projects' implementation of the practices comprising this phase. * Phase VI: Manage provider(s) performance. The effectiveness with which the provider(s) performance is managed is critical to the successful implementation of an outsourcing project. Indeed, according to Gartner, Inc., a leading research firm, an outsourcing project can be thwarted by poorly designed, funded, and delivered processes for managing the delivery of services.[Footnote 34] The five projects generally implemented the 11 practices contained in this phase, with about 84 percent of the practices being implemented in this phase. For example, the practices related to obtaining feedback on provider performance were largely implemented. This is important because different levels of an agency can have different perceptions about the value of the outsourcing project. For example, an outsourcing project may be considered successful by the agency's executive management if it is focused on controlling costs, but be considered inadequate by business managers and users who may be expecting higher levels of service. Each of these viewpoints is valid and should be taken into account when the provider's performance is evaluated. Two other practices--including incentives and penalties in contracts--were fully implemented by two projects. Incentives and penalties are important because they can help motivate the provider to exceed or meet performance requirements. Nonetheless, two projects did not include monetary incentives and two projects did not include monetary penalties in their contracts. For example, the Army TAC-SWA project did not include monetary incentives, although the contracting officer stated that incentives might have been useful to motivate the provider to exceed performance requirements. Incentives can also help control risks. According to a guide on performance-based services acquisition, if the incentives in the contract are right and if the provider and agency share the same goals, risk is largely controlled and effective performance is "almost the inevitable outcome."[Footnote 35] Appendix VII provides additional information on projects' implementation of the practices comprising this phase. * Phase VII: Ensure services are provided. Although outsourcing transfers responsibility for performing the service to the provider(s), the client organization is ultimately responsible for ensuring that services are provided and that end-user needs are met. Accordingly, it is critical that the agency ensure that services are provided. The projects had implemented 97 percent of these practices, and four projects implemented all of them. For example, every project monitored the providers' work. In the case of the Air Force's C4 Services project, quality assurance evaluators monitored the provider's work to identify problems or trends in accordance with the project's quality assurance surveillance plan. The results were reported to the contracting officer and to the functional area chief for resolution. Another practice, using customer satisfaction surveys, was fully implemented by four of the five projects. However, the Air Force project did not conduct, or require its contractor to conduct, customer satisfaction surveys. Although the provider surveys staff annually, the Air Force is nevertheless relying on the provider to voluntarily implement an important practice for determining how customers view the services being delivered and whether changes need to be made. Appendix VIII provides additional information on projects' implementation of the practices comprising this phase. Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects: We have previously reported on the importance of collecting and disseminating lessons learned.[Footnote 36] For example, a critical activity in IT investment management is establishing a process for developing and capturing lessons learned in a written product or knowledge base and disseminating them to decision-makers.[Footnote 37] In addition, one of the practices in our framework for outsourcing IT services addresses incorporating lessons learned from peers who have engaged in similar sourcing decisions. Use of lessons learned is a principal component of an organizational culture committed to continuous improvement. Sharing such information serves to communicate acquired knowledge more effectively and to ensure that beneficial information is factored into planning, work processes, and activities. Lessons learned can be based on positive experiences or on negative experiences that result in undesirable outcomes. Although DOD has taken action to gather and disseminate lessons learned and best practices on general acquisition issues, these efforts generally do not focus on outsourcing or include sharing the lessons learned from IT outsourcing projects across the department. Specifically, a number of DOD Web sites provide guidance, lessons learned, and best practices related to general acquisition issues. However, using these sites to locate specific information on IT outsourcing best practices and lessons learned can be time-consuming and difficult because so many topics and information sources are provided. Specifically, MHS/ITO Help Desk project officials said that searching numerous Web sites to get relevant information to address questions and concerns about outsourcing IT services can consume hours. For example, when we entered the keywords "IT outsourcing" and "best practices" into the search feature on the Office of the Undersecretary for Defense of Acquisition, Technology, and Logistics site, ACQWeb (www.acq.osd.mil) in early March, it provided us with links to 1,251 documents. Likewise, lessons learned covers so many topics that it is difficult to search for an applicable IT lesson. For example, when we used the phrase "lessons learned in IT outsourcing" no documents were identified on the ACQWeb, but when we inserted "lessons learned" and "IT outsourcing" links to more than 1,700 documents were produced. One DOD Web site, Share A-76!, was established to address one of our previous recommendations,[Footnote 38] that is, to establish a framework for identifying and analyzing best practices and lessons learned from competitive sourcing studies and disseminating them DOD- wide. Share A-76! promotes the sharing of best practices and lessons learned related to one form of outsourcing that was conducted under the OMB Circular A-76 competitive sourcing process. Among other things, the site contains guidance, links to other relevant sites, sample documents, and a best practices library that communicates field staff experiences and advice about the Circular A-76 process. A NIMA project official said that NIMA's staff routinely accesses the Web site because it contains a wealth of information on policies, procedures, lessons learned, and links to other outsourcing sites. The DOD analyst responsible for Share A-76! estimated that the site receives about 12,000 visits per month and said that on the basis of E-mails and anecdotes, user satisfaction is favorable. However, this Web site is specific to the OMB Circular A-76 process, which may not apply to other types of outsourcing. For example, Circular A-76's policy pertains to public/private competitions and requires that the final evaluation between the government and the private sector be based exclusively on cost. DOD acquisition and IT officials acknowledged that there is no mechanism in DOD to easily share and leverage lessons learned relating to outsourcing IT services. However, these officials agreed that a departmentwide effort to identify, capture, and disseminate lessons learned and leading practices of projects with experience in carrying out IT outsourcing could offer valuable insights and new ideas that would benefit others. Moreover, officials from three of the projects in our review told us that there is value in collecting and disseminating the knowledge acquired from IT outsourcing projects in a systematic manner across the department. Each of the projects in our review identified knowledge and experience gained from their approaches to outsourcing IT services that could offer insights and practices for other ongoing and future projects to consider. For example: * MHS developed specific guidance and lessons learned for implementing a performance-based incentive contract for help-desk operations. * The Department of the Navy's NMCI project has developed a series of lessons learned related to transitioning to the provider that is being shared within the NMCI community; one example was that all personnel should be available during scheduled testing and deployment. * NIMA has experience in contracting techniques emphasizing a partnering approach with providers to refine requirements and establish a common understanding of costs. In addition, a departmental IT outsourcing knowledge-sharing approach could include links to information about other government agencies' IT outsourcing projects. For example, our 2002 report on desktop outsourcing includes an extensive discussion of lessons learned by agencies that have implemented this type of IT services outsourcing.[Footnote 39] Developing an effective lessons learned activity is not easy. For example, NMCI officials said that for a lessons learned initiative to be effective, a process must exist that is clearly understood by everyone and allows capturing and sharing of knowledge to occur with minimum effort. Other challenges in developing an effective lessons learned process were outlined by the Share A-76! analyst. The analyst stated that only a small number of site users have contributed lessons learned to the Share A-76! Web site, which she attributed, in part, to the amount of time and effort needed to document and obtain agreement by all levels of the organization on the lessons learned. In addition, the analyst stated that there is reluctance to share negative lessons, and often the review and approval process sanitizes best practice information so that it becomes too general to be most helpful to users. Such challenges can be overcome by executive-level support. Indeed, DOD acquisition and IT officials stated that for lessons learning activities to be effective, senior management must devote support and resources to the effort. This is consistent with our prior work, which showed that knowledge can be effectively shared only when employees are given adequate time as well as established places where they can actually transfer knowledge.[Footnote 40] Last year, we outlined a generic lessons learned process that could be used to guide the development of such a process for outsourcing IT services.[Footnote 41] Although the mechanism or processes used to collect, share, and disseminate lessons learned may vary, in general such a process comprises four main elements: collection, verification, storage, and dissemination. The collection process involves the capture of information through structured and unstructured processes. Verification serves to verify the correctness and applicability of lessons submitted. The storage aspect of lessons learned usually involves incorporating the lessons into an electronic database for the dissemination and sharing of information, including the ability to conduct information searches. The final element, and the most important, is the dissemination of lessons learned, since lessons are of little benefit unless they are distributed and used by people who will benefit from them. Lessons can be "pushed," or automatically delivered to a user, or "pulled" in situations where a user must manually search for them. Lessons can also be disseminated with an assigned priority descriptor, which denotes the risk, immediacy, and urgency of the lessons learned content. Conclusions: The projects in our review substantially implemented leading commercial practices for outsourcing IT services, which has increased each project's probability of success. Capturing how these projects operationalized leading commercial practices could help other IT services outsourcing projects succeed. Although currently there is no such DOD-wide mechanism, such as an electronic tool, to easily share and leverage lessons learned, DOD IT and acquisition officials agreed that a departmentwide effort to identify, capture, and disseminate lessons learned could offer valuable insights and new ideas that would benefit others. Lessons learned that are pragmatic and easily accessible could give DOD managers a more informed understanding of the important issues to be addressed when making outsourcing decisions, as well as the factors to be considered to help ensure the success of these endeavors. DOD managers can also benefit from lessons learned on the basis of negative experiences. The projects in our review were well into implementation, and therefore, at this late stage, we see little advantage for them to revisit practices that were not implemented. Nevertheless, an electronic tool for capturing and disseminating lessons learned would allow the rest of DOD to benefit from the negative consequences and increased risks associated with those practices that the projects did not implement. Developing a lessons learned mechanism is not easy; thus, senior management support and resources are keys to success. Without such support driving the capture and dissemination of lessons learned, DOD is losing an opportunity for wider application of leading practices and thus better ensuring that its IT outsourcing efforts are successful. Recommendations: To assist DOD organizations in planning and implementing outsourcing projects for IT services, we recommend that the Secretary of Defense direct the Undersecretary of Defense for Acquisition, Technology, and Logistics, working in conjunction with the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I), to provide senior management support and adequate resources to develop and implement an electronic tool to capture and disseminate examples and lessons learned from actual IT outsourcing projects. These examples and lessons learned, at a minimum, should include the results of our review of the five projects discussed in this report. We also recommend that the Secretary of Defense direct the Undersecretary of Defense for Acquisition, Technology, and Logistics, working in conjunction with the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I), to ensure that the method used to gather information for this electronic tool incorporate the main elements of a lessons learned process--namely, collection, verification, storage, and dissemination. Agency Comments and Our Evaluation: In written comments on a draft of our report, signed by DOD's Principal Director, Deputy Assistant Secretary of Defense (Deputy Chief Information Officer), the department partially concurred with our recommendations. Specifically, DOD agreed that capturing lessons learned in the development and implementation of its IT outsourcing initiatives is important to continually improve its outsourcing methods and results achieved. The department also stated that before deciding on a specific method to achieve this aim, it intends to explore a variety of mechanisms that could be used. In particular, the department stated that it currently has several processes and communities of interest that collect and disseminate lessons learned in other areas, which are logical starting points for determining the best path forward. DOD's written comments are reproduced in appendix IX. We agree that it is prudent to explore various alternatives to leveraging lessons learned from DOD's IT services outsourcing experiences. Our recommendations are not prescriptive as to the electronic method to be used to capture and disseminate lessons learned. Therefore, the department's plan to explore various alternatives is consistent with our recommendations. : We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Director of the Office of Management and Budget. We will also provide copies to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions on matters discussed in this report, please contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, at (202) 512-9556. We can also be reached by E-mail at hiter@gao.gov and lambertl@gao.gov, respectively. Other contacts and key contributors to this report are listed in appendix X. Randolph C. Hite Director, Information Technology Architecture and Systems Issues: Signed by Randolph C. Hite: [End of section] Appendixes : Appendix I: Objectives, Scope, and Methodology: Our objectives were to determine (1) the extent to which selected Department of Defense (DOD) information technology (IT) services outsourcing projects use leading commercial practices as specified in our framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department. To determine the extent to which selected DOD outsourcing projects for IT services use leading commercial practices, we identified the practices in our November 2001 report on leading commercial practices[Footnote 42] that (1) are typically applied at the project level and (2) were verifiable through documentation and interviews. Because DOD did not centrally maintain a list of outsourcing projects for IT services, we asked the department to identify candidate projects for our evaluation. From this list, we selected the following five projects for our review: (1) Air Force Kirtland Air Force Base's Command, Control, Communications, and Computer (C4) Services project; (2) Army Network Enterprise Technology Command's (NETCOM) Total Army Communications - Southwest Asia (TAC-SWA) project; (3) Military Health System/Information Technology Organization (MHS/ITO) Help Desk project; (4) Department of the Navy's Navy and Marine Corps Intranet (NMCI) project; and (5) National Imagery and Mapping Agency (NIMA) Information Technology/Information Services (IT/IS) project. We chose each project on the basis of the following criteria: (1) no more than one project from each military service and two agencies, (2) illustrative example of DOD IT outsourcing, (3) dollar value greater than $10 million, and (4) enough time elapsed for services to have been delivered and performance measured. At our request, each project completed a self-assessment on whether and how it implemented leading commercial practices. We reviewed the agency self-assessments and accompanying documentation and interviewed the appropriate agency project officials to verify whether the practices were followed. In addition, we interviewed representatives from each of the providers associated with these projects. We also researched additional information on commercial practices in our November 2001 guide[Footnote 43] and the challenges the federal government faces in implementing them. Specifically, we performed a literature search, which included reviewing reports issued by leading research firms, such as Gartner, Inc., and Giga Information Group, Inc. In addition, we interviewed representatives from industry organizations that have an interest in outsourcing IT services, including the Coalition for Government Procurement; the Information Technology Association of America; the Professional Services Council; and Acquisition Solutions, Inc. To determine whether DOD is sharing lessons learned from its IT outsourcing projects across the department, we identified and reviewed various approaches that DOD currently uses to capture and disseminate such information. This included identifying and reviewing various Web sites and performing key word searches on these sites to identify lessons learned for outsourcing IT services. We also interviewed applicable DOD acquisition and IT officials. Finally, we identified and reviewed a generic lessons learned process contained in our January 2002 report on the National Aeronautics and Space Administration's lessons learned mechanisms.[Footnote 44] We performed our work at the Army's NETCOM in Ft. Huachuca, Arizona; the MHS/ITO Help Desk project office in Falls Church, Virginia; the NMCI Director's office in Crystal City, Virginia; and NIMA's headquarters in Bethesda, Maryland. We conducted our review between May 2002 and early March 2003 in accordance with generally accepted government auditing standards. [End of section] Appendix II: Projects' Implementation of Phase I: Determine Sourcing Strategy: [See PDF for image] [End of figure] In the first phase of our outsourcing framework, the client organization determines whether internal capability or external expertise can more effectively meet its IT needs. The purpose of this sourcing strategy is to achieve the optimal balance between internal and external capabilities, activities, processes, and services to ensure that strategic business objectives are achieved at the lowest risk. Among the factors that an organization should evaluate in crafting this strategy are technology, business, financial, and personnel requirements and whether it has skilled business and IT managers. In addition, according to Gartner, Inc., sound sourcing decisions depend on whether IT organizations (1) know and understand their business priorities, (2) are prepared to invest in some skills and divest others, and (3) identify and assess trade-offs.[Footnote 45] The six practices in this phase that we used to evaluate the five projects are as follows: * Use third-party assistance with experience in a variety of sourcing arrangements when formulating a sourcing strategy. * Incorporate lessons learned from peers who have engaged in similar sourcing decisions. * Estimate impact of sourcing decision on internal organization. * Benchmark and baseline productivity of internal services before making the final sourcing decision. * Determine the business reasons for outsourcing IT. * Determine reasons for outsourcing IT that can improve the organization's ability to use and manage technology. Figure 6 shows that the implementation of the practices by the five projects in our review was uneven. Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 4 provides detailed information on whether and how each project implemented each of the six practices in this phase. Table 4: Summary of Projects' Use of Phase I Practices: Did The Project Use the Practice?: Practice: Use third-party assistance with experience in a variety of sourcing arrangements when formulating a sourcing strategy.; Did the project use the practice?: Air Force C4 Services: No--According to Air Force project officials, they decided that they had sufficient in-house expertise, and the project had no funding available to employ a contractor. However, the former functional area chief told us that using third-party assistance would have been beneficial because the requirements would have been better written. According to this official, the requirements had to be substantially rewritten 2 years after contract award.; Did the project use the practice?: Army TAC-SWA: No--According to a TAC-SWA project official, the Army decided that it had sufficient expertise in-house since the TAC-SWA contract was a consolidation of three existing contracts.; Did the project use the practice?: MHS/ITO Help Desk: Yes--For example, MHS worked with the Department of the Interior's GovWorks Program and the Defense Acquisition University staff to obtain expertise on sourcing strategies. Also, MHS employed a contractor with sourcing expertise.; Did the project use the practice?: Navy NMCI: Yes--The Navy employed a contractor with sourcing expertise and contacted other government entities about their experiences.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA employed a contractor with sourcing expertise. Practice: Incorporate lessons learned from peers who have engaged in similar sourcing decisions.; Did the project use the practice?: Air Force C4 Services: Yes--Primarily from other Department of Defense (DOD) projects that implemented Office of Management and Budget (OMB) Circular A-76 policies.; Did the project use the practice?: Army TAC- SWA: Yes--According to a TAC-SWA project official, it used lessons learned from its prior contracts for these services and another similar Army contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Primarily from industry peers and MHS's prior help-desk function.; Did the project use the practice?: Navy NMCI: Yes--Navy officials stated that there were no peers that had engaged in similar sourcing decisions because no other outsourcing project was of as large a scale as NMCI. However, the Navy did talk to members of private industry and the National Aeronautics and Space Administration on their more limited efforts.; Did the project use the practice?: NIMA IT/IS: Yes--A NIMA contractor provided the agency with a report on industry best practices. In addition, NIMA held discussions with the National Security Agency on its outsourcing effort. Practice: Estimate impact of sourcing decision on internal organization.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force estimated that there would be substantial internal impact, such as to its staff, due to its decision to outsource.; Did the project use the practice?: Army TAC-SWA: No--The Army did not analyze the impact on its internal organization because, according to the TAC-SWA program manager, NETCOM was outsourcing a function that was already contracted out.[A]; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS assessed the staff and financial impact of its sourcing decision.; Did the project use the practice?: Navy NMCI: Yes--The Navy performed an assessment of staffing and other impacts, such as cost, related to its sourcing decision.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA performed an assessment of staffing and other impacts, such as cost, related to its sourcing decision. Practice: Benchmark and baseline productivity of internal services before making the final sourcing decision.; Did the project use the practice?: Air Force C4 Services: No--The Air Force did not implement this practice because, according to project officials, an executive decision was made to outsource the C4 services following the policies contained in OMB Circular A-76. Accordingly, Air Force officials stated that such analyses would not have affected the final sourcing decision.; Did the project use the practice?: Army TAC-SWA: No--The Army did not perform a benchmark or baseline analysis because, according to the TAC-SWA program manager, NETCOM was outsourcing a function that was already contracted out.; Did the project use the practice?: MHS/ITO Help Desk: Limited--MHS attempted to establish a baseline of its prior environment but, according to project officials, available documentation was limited to historical trouble ticket workload data; therefore, this baseline was a best estimate. In addition, MHS project officials stated that a contractor performed a benchmark analysis, but they did not provide supporting documentation.; Did the project use the practice?: Navy NMCI: Limited--The Navy had a private-sector firm benchmark its environment against seven large public and private organizations. In addition, the Navy and the Marine Corps performed an analysis at a sample of representative locations to obtain a baseline. However, this baseline did not include an assessment of the Department of the Navy's legacy applications since project officials decided to rely upon an inventory developed in addressing the Year 2000 challenge. The Navy subsequently found that it had substantially underestimated its number of legacy applications. According to NMCI program officials, this underestimation contributed to the transition period slipping from 2-½ years to 3-½ years.; Did the project use the practice?: NIMA IT/IS: Yes--A NIMA contractor baselined the existing NIMA environment and benchmarked it to peers. Practice: Determine the business reasons for outsourcing IT.; Did the project use the practice?: Air Force C4 Services: Yes--To achieve cost savings and to shift military personnel to other work.; Did the project use the practice?: Army TAC-SWA: Yes--To address shortages in military personnel to perform its mission.; Did the project use the practice?: MHS/ITO Help Desk: Yes--To achieve increased productivity and customer satisfaction and to decrease costs.; Did the project use the practice?: Navy NMCI: Yes--To have private industry capitalize infrastructure improvements that were needed to quickly and securely share knowledge around the globe.; Did the project use the practice?: NIMA IT/IS: Yes- -To obtain improved customer services and decreased costs. Also, this function was determined to be a commercial function under the Federal Activities Inventory Reform Act.[B]. Practice: Determine reasons for outsourcing IT that can improve the organization's ability to use and manage technology.; Did the project use the practice?: Air Force C4 Services: No--According to Kirtland Air Force Base officials, the Air Force decided to hold a public/private competition following OMB Circular A-76 policies; therefore, improving its ability to use and manage technology was not a factor in determining its outsourcing strategy.; Did the project use the practice?: Army TAC-SWA: No--According to TAC-SWA project officials, improving its ability to use and manage technology was not a factor in determining its outsourcing strategy.; Did the project use the practice?: MHS/ITO Help Desk: Yes--To improve IT expertise and knowledge.; Did the project use the practice?: Navy NMCI: Yes--To improve, for example, system security, interoperability, reliability, and network response.; Did the project use the practice?: NIMA IT/IS: Yes--To achieve better IT management performance. Source: GAO. [A] Having had the activity previously performed by a contractor does not obviate the need to estimate the impact of a sourcing decision on the internal organization--there are still risks involved, such as the potential disruption of services during the transfer to the new contractor. In fact, the staff from the incumbent contractor did not transition to the TAC-SWA provider, and the provider had difficulty filling these slots within the schedule outlined in the contract. [B] The Federal Activities Inventory Reform Act of 1998 requires federal agencies to prepare and submit to OMB, by June 30 of each year, inventories of the commercial activities performed by federal employees. [End of table]: [End of section] Appendix III: Project's Implementation of Phase II: Define Operational Model: [See PDF for image] [End of figure] Critical to the successful outsourcing relationship is an operational model for guiding the structure of the contract and the plans for transition. In defining the operational model, client organizations formalize executive leadership, team composition, client responsibilities, and operating relationships between the client and provider. The operational model helps the organization to compare its plans with the expectations that were set as the initial decision to outsource was made and to ascertain whether these plans will enable the organization to meet those objectives. An important aspect of the operational model is an explicit understanding of how the client organization plans to communicate its needs and provide feedback to the provider. In addition, communication between the business and IT offices within the client organization is always critical. This is particularly true in the case of outsourcing because the IT service provider is outside the client organization and disconnects are more likely to occur. Therefore, organizational processes to facilitate good communication are critical. The 13 practices in this phase that we used to evaluate the five projects are as follows: * Establish executive leadership for IT to facilitate the outsourcing initiative. * Continually communicate/clarify outsourcing objectives, while correcting misinformation that affects the organization. * Establish a core group of people who will be involved in all phases of outsourcing. * Select a person involved in the negotiation of the contract to manage the outsourcing relationship. * Create and define a contract management structure with operational points of contact and managers. * Define the role of internal IT managers and business leaders. * Ensure that the right skills are in place to support the outsourcing relationship. * Establish a point of contact high in the provider management structure for elevating provider performance concerns. * Have provider establish an on-site support team to serve as liaison between client and provider. * Train provider on client business environment and goals. * Select or develop standard tools for managing the relationship. * Use third-party assistance to take advantage of expertise from a variety of outsourcing arrangements in defining the operational model. * Ensure that the provider management team has prior experience in the client's field of business. Figure 7 illustrates that the five projects in our review largely implemented the practices. Figure 7: Project Implementation of Phase II: Define Operational Model: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 5 provides detailed information on whether and how each project implemented each of the 13 practices in this phase. Table 5: Summary of Projects' Use of Phase II Practices: Practice: Establish executive leadership for IT to facilitate the outsourcing initiative.; Did the project use the practice?: Air Force C4 Services: Yes--The initiative was planned by a steering group made up of representatives from various major offices at Kirtland Air Force Base. Subsequent to contract award, the Air Force established a functional area chief to manage the initiative, and a Lieutenant Colonel was appointed to be responsible for this project.; Did the project use the practice?: Army TAC-SWA: Yes--NETCOM used the existing leadership in its offices of operations and logistics to provide executive leadership.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS used its existing leadership structure and processes. This structure includes a program executive office, steering committee, and program review board.; Did the project use the practice?: Navy NMCI: Yes--In 1999, the Navy established a program executive office for IT primarily to support the NMCI outsourcing effort. The Congress later directed the Navy to identify a single individual whose sole responsibility would be to oversee and direct the NMCI program. As a result, in February 2002, the Navy established the NMCI Director's Office to take over responsibility for NMCI.; Did the project use the practice?: NIMA IT/IS: Yes--The NIMA enterprise transformation directorate formed a strategic sourcing office to provide executive oversight of the effort. Practice: Continually communicate/clarify outsourcing objectives, while correcting misinformation that affects organization.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force provided communication through, for example, briefings and meetings with employees.; Did the project use the practice?: Army TAC-SWA: Yes- -According to TAC-SWA project officials, the applicable NETCOM officials were informed about the initiative and provided comments on the draft performance work statement. TAC-SWA project officials also brief new commanders on the contract before they are transferred to the military theater.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS provided updates to the Chief Information Officer (CIO) and program review boards, provided briefings to the deputy surgeon general, and published questions and answers.; Did the project use the practice?: Navy NMCI: Yes--The Navy established an action collaboration team structure to involve the Navy community in the NMCI communications process. Also, the NMCI Information Bureau initiated press conferences, briefings, site visits, informational pamphlets, and promotional material. In addition, the NMCI Director and other staff provide numerous briefings and presentations to commands, industry gatherings, and government officials. Finally, Web sites and Web-based collaboration sites were established to facilitate communications.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA created an internal Web site to post information on the outsourcing project, distributed periodic global electronic mails, and held town hall meetings. Practice: Establish a core group of people who will be involved in all phases of outsourcing.; Did the project use the practice?: Air Force C4 Services: Yes--The core group included the contracting officer, functional area chief, and manpower specialist.; Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-SWA project officials, the core group included representatives from logistics, operations, and resource management as well as the contracting officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- According to program officials, the core group included representatives from the MHS program executive office, MHS IT program offices, military department chief information officers' offices, and Interior's GovWorks organization, which provided contracting services.; Did the project use the practice?: Navy NMCI: Yes--The program executive office for IT and the NMCI Director's organization comprise the core group responsible for managing the outsourcing initiative.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA established the Strategic Sourcing Office, which was dedicated to managing and facilitating the outsourcing initiative. Practice: Select a person involved in the negotiation of the contract to manage the outsourcing relationship.; Did the project use the practice?: Air Force C4 Services: Yes--The contracting officer.[A]; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- According to MHS program officials, the chair of the contract evaluation committee and the individual in charge of transition are responsible for managing the outsourcing relationship.; Did the project use the practice?: Navy NMCI: Yes--The primary contracting officer.; Did the project use the practice?: NIMA IT/IS: Yes--A lead contracting officer. Practice: Create and define contract management structure with operational points of contact and managers.; Did the project use the practice?: Air Force C4 Services: Yes--The structure was defined by the roles and responsibilities of the contracting officer and her staff.; Did the project use the practice?: Army TAC-SWA: Yes--The overall responsibility for the contract rests with the contracting officer. The contracting officer's representative acts as a liaison between the government and the contractor.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--Responsibility for contract management is jointly held by the contracting officer, who is part of Interior's GovWorks organization, and the MHS program office, which provides day-to-day oversight of the provider.; Did the project use the practice?: Navy NMCI: Yes--The primary contracting officer establishes procedures and controls necessary for effective contractual oversight of the NMCI initiative and has a matrix relationship with the NMCI Director. The primary contracting officer certifies contracting officer representatives for the NMCI contract to provide technical coordination efforts.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has appointed an operational point of contact for each of the seven functional areas being outsourced, which are documented in a "rules of engagement" agreement between the government and the contractor. Practice: Define the role of internal IT managers and business leaders.; Did the project use the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base did not initially define the roles of its internal IT and business managers. However, subsequent to contract award, Kirtland Air Force Base established a CIO committee and configuration control board comprising the functional area chief (Kirtland's CIO) and representatives from various business areas. Although not exclusively devoted to the C4 Services project, these groups help provide direction to the project.; Did the project use the practice?: Army TAC-SWA: Limited--TAC-SWA project officials explained the roles of the principal staff involved with this project but generally did not have supporting documentation defining these roles.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS's help- desk performance assessment plan defines the roles of various groups and individuals associated with the program.; Did the project use the practice?: Navy NMCI: Yes--The NMCI execution plan describes the roles of internal and external organizations that directly or indirectly affect the management of NMCI and explains in detail the duties and responsibilities of the program executive office for IT and the NMCI program management offices.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA transformation teams for the activities being outsourced define the roles of their managers and leaders. Practice: Ensure that the right skills are in place to support the outsourcing relationship, including those dealing with; ; * contract management,; * financial management,; * IT management,; * negotiation strategies,; * teaming and interpersonal relationships,; * project management, and; * relationship management.; Did the project use the practice?: Air Force C4 Services: Yes--According to Air Force officials, the right skills are in place.; Did the project use the practice?: Army TAC-SWA: Yes--According to the TAC-SWA project officials, the Army has created several positions to ensure that the right skills are in place to manage the outsourcing relationship, including a contracting officer, legal advisor, operations and logistics personnel as well as a NETCOM unit commander. According to Army TAC-SWA officials, each position employs an individual with the necessary skills to support the outsourcing relationship.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, the Tri-Service Management Program Office, GovWorks (the Interior organization that provides contracting assistance to MHS), and a contractor collectively employ the skill sets needed to support the outsourcing relationship.; Did the project use the practice?: Navy NMCI: Limited--The Director, NMCI, has staff that report directly or are matrixed with him that are responsible for performing all but two of these functions. Specifically, at this time, NMCI does not have staff assigned to support teaming and interpersonal relationships and relationship management. The Navy recognizes the need for these skills and is taking, or plans to take, various actions to obtain these skills.; Did the project use the practice?: NIMA IT/IS: Yes--According to NIMA, its transformation teams provide the skills necessary to support the outsourcing relationship. Practice: Establish a point of contact high in the provider management structure for elevating provider performance concerns.; Did the project use the practice?: Air Force C4 Services: Yes--According to the contracting officer, concerns can be addressed to the senior vice president at the provider headquarters.; Did the project use the practice?: Army TAC-SWA: Yes--The provider established a program manager as the point of contact for elevating concerns.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The provider established an executive program manager as the point of contact for elevating concerns.; Did the project use the practice?: Navy NMCI: Yes- -The NMCI Director interacts directly with the provider's program executive on a regularly scheduled basis; such meetings would include any performance issues.; Did the project use the practice?: NIMA IT/IS: Yes--The transformation team charters define procedures for addressing issues, including possibly elevating concerns to the provider's general manager. Practice: Have provider establish an on-site support team to serve as a liaison between client and provider.; Did the project use the practice?: Air Force C4 Services: Yes--The provider has established an on-site support team, which is led by the on-site manager.; Did the project use the practice?: Army TAC-SWA: Yes--Site managers have been established in accordance with the contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The provider's transition plan identifies the on-site support team.; Did the project use the practice?: Navy NMCI: Yes--The provider establishes site leads that remain on-site as the location goes through the cutover to NMCI.; Did the project use the practice?: NIMA IT/IS: Yes--The rules of engagement define the on-site support team for each major NIMA location. Practice: Train provider on client business environment and goals.; Did the project use the practice?: Air Force C4 Services: Limited--Prior to contract award, Kirtland Air Force Base provided an orientation session in which they discussed their operations, business environment, and goals. Subsequent to contract award, specific Kirtland Air Force Base government personnel were responsible for aiding the provider during the transition period. However, provider officials stated that they did not receive adequate training, which made the transition period more difficult.; Did the project use the practice?: Army TAC-SWA: Not applicable--The Army contract was a consolidation of prior contracts for this activity, and the winning bidder was one of the incumbent contractors.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS officials stated that they provided some training to the provider on their business environment. In particular, MHS trained and certified the provider's staff on the agency's application systems. In addition, the provider had previous MHS experience.; Did the project use the practice?: Navy NMCI: Limited--The Navy did not provide training on its business environment and goals; instead it relied on the NMCI design reference mission document, which was included with the request for proposals (RFP). This document defined the NMCI operational environment. However, a provider official stated that although the document was useful, it did not identify Navy enterprisewide operations. According to NMCI program officials, the provider somewhat mitigated this problem by hiring several highly knowledgeable staff from the Department of the Navy shortly after contract award.; Did the project use the practice?: NIMA IT/IS: Not applicable--According to the project's contracting officer, such training was not necessary because the senior officers of the provider are all recent NIMA management employees and were already knowledgeable of these areas. For example, among those who transitioned to the provider was a former director of the hydrographic topographic center of the Defense Mapping Agency, a predecessor agency to NIMA. Practice: Select or develop standard tools for managing the relationship (e.g., performance scorecards, enterprise resource management system).; Did the project use the practice?: Air Force C4 Services: Yes--For example, the revised contract outlines the use of specific software to help manage the provider's performance.; Did the project use the practice?: Army TAC-SWA: Yes--NETCOM uses a monthly report from the contracting officer's representative to evaluate the provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- According to MHS officials, the program office and the provider have selected, for example, an enterprise resource management system.; Did the project use the practice?: Navy NMCI: Yes--The Navy is using a balanced scorecard process to provide Navy and Marine Corps leadership with information to judge how well NMCI is supporting the missions and strategies of the department.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA used various tools, including twice-a-day performance reports for the operational help-desk function and monthly performance reports. Practice: Use third-party assistance to take advantage of expertise from a variety of outsourcing arrangements in defining the operational model (i.e., defining roles and responsibilities).; Did the project use the practice?: Air Force C4 Services: Yes--Although third-party assistance was not used at the onset of the C4 Services project, a person with prior experience in managing IT services contracts for the Air Force was brought in to provide advice. He was subsequently put in charge of the project.; Did the project use the practice?: Army TAC- SWA: No--The Army believed that it had sufficient expertise in-house.; Did the project use the practice?: MHS/ITO Help Desk: Yes--For example, MHS contacted the Interior's GovWorks Program, Defense Acquisition University, and others. Also, MHS employed a contractor with sourcing expertise.; Did the project use the practice?: Navy NMCI: No--The Navy believed it had sufficient internal experience and expertise.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA employed contractors to develop contractual roles and assist in evaluating the contract. Practice: Ensure that the provider management team has prior experience in the client's field of business.; Did the project use the practice?: Air Force C4 Services: Yes--Provider management capabilities and experience in providing IT services were evaluation factors in the technical evaluation process.; Did the project use the practice?: Army TAC-SWA: Yes--Vendor past performance was part of the evaluation criteria during source selection and the provider was an incumbent contractor with NETCOM.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The request for quote laid out MHS's evaluation criteria for selecting a provider, which included prior experience in health care.; Did the project use the practice?: Navy NMCI: Yes-- According to NMCI program officials, the Navy required bidders to provide information on their prior experience in related business fields. For example, the solicitation required bidders to demonstrate experience managing a similar effort of 100,000 or more seats at one time.[B]; Did the project use the practice?: NIMA IT/IS: Yes--It was expected that most of the provider's workforce would be composed of former NIMA employees. Among those who transitioned to the provider was a former director of the hydrographic topographic center of the Defense Mapping Agency. Source: GAO. [A] The Air Force's initial contract was not negotiated because Kirtland Air Force Base used an OMB Circular A-76 2-step, sealed bid process. However, the contracting officer was involved in the bid process as well as in managing the winning contractor. [B] This threshold could be met through multiple efforts, but at least one had to include at least 20,000 seats. [End of table]: [End of section] Appendix IV: Project's Implementation of Phase III: Develop the Contract: [See PDF for image] [End of figure] Phase III focuses on the development of the contract, which defines the legal terms of the relationship between client and provider. While other phases in the outsourcing process describe the need for mutual trust and a close, flexible working relationship, this phase focuses on the development of the contract, which is the foundation on which a working relationship will be built. A well-written contract helps the outsourcing organization meet its requirements while allowing the service provider to make a fair profit. It sets the expectations for service levels, delivery of essential services, and continuous improvement and is intended to protect the interests of all parties. The 16 practices in this phase that we used to evaluate the five projects are as follows: * Base performance requirements on business outcomes. * Include measures that reflect end-user satisfaction as well as technical IT performance. * Review and update performance requirements periodically. * Require the provider to meet the minimum performance in each category of service. * Require the provider to achieve escalating performance standards at agreed-upon intervals. * Incorporate sufficient flexibility so that minimum acceptable performance can be adjusted as conditions change, as the provider becomes more adept at satisfying customer demands, and as improvement goals are achieved. * Use service-level agreements (SLA)[Footnote 46] to clearly articulate all aspects of performance, including management, processes, and requirements. * Client and provider work together to define the appropriate number of SLAs and appropriate structure for each. * Specify circumstances under which the provider is excused from performance levels mandated by master service agreements. * Client and provider work together to identify SLAs for which compensation is based, while additional SLAs may be defined to manage performance. * The contract should include clauses for (1) determining pricing structures; (2) performing customer satisfaction surveys and using the results to redefine performance levels; (3) terminating the contract; (4) resolving disputes in a timely manner; (5) taking work away, without penalty, from provider for nonperformance; (6) declaring a significant event that can lead to a change in the contract; (7) defining performance requirements; and (8) conducting regularly scheduled meetings. * Consider setting up a master services agreement under which all arrangements between client and provider operate. * Include the appropriate representation from each major organizational unit on the contract negotiation team. * Specify the use of volume purchases to obtain optimal discounts. * Use third-party assistance in negotiating and developing the contract. * Sign the contract after contract negotiations and final vendor selections. As figure 8 shows, the five projects largely implemented the practices. Figure 8: Project Implementation of Phase III: Develop the Contract: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 6 provides detailed information on whether and how each project implemented each of the 16 practices in this phase. Table 6: Summary of Projects' Use of Phase III Practices: Practice: Base performance requirements on business outcomes.; Did the project use the practice?: Air Force C4 Services: Yes--The performance requirements are contained in the contract and are based on business outcomes. For example, the requirements define the percentage of system availability needed to support users.; Did the project use the practice?: Army TAC-SWA: Yes--The performance requirements are outcome based and stated in terms of the level of operation and maintenance services required. According to TAC-SWA officials, multiple trips to overseas locations were made to identify the performance requirements.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Industry standards and metrics were used to base performance requirements on business outcomes.; Did the project use the practice?: Navy NMCI: Yes--The performance requirements are included in the contract and are based on business outcomes. SLAs establish the performance standards and service quality for all types of NMCI seats.; Did the project use the practice?: NIMA IT/IS: Yes--Performance-based requirements are in the contract, including those related to customer satisfaction, process improvement, quality control, and timely and accurate completion of requirements. Practice: Include measures that reflect end-user satisfaction as well as technical IT performance.; Did the project use the practice?: Air Force C4 Services: Limited--The Air Force measures include technical IT performance. However, end-user satisfaction measures are limited to the percentage of complaints received and do not measure overall customer satisfaction. Project officials did not know why overall end-user satisfaction measures were not established.; Did the project use the practice?: Army TAC-SWA: Limited--The Army's contract includes technical IT performance measures but not end-user satisfaction performance measures. For example, the contract includes a requirement for help-desk services, but does not include measures related to customer satisfaction associated with such services. According to the project official that developed the performance work statement in the contract, NETCOM did not include such customer satisfaction measures because it did not think that it was necessary to have a performance standard for that service.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS measures the service provider's technical IT performance and uses an electronic customer satisfaction survey to assess the quality of help-desk services. Satisfaction rates and number of survey responses are tracked and used for both evaluation and incentive payments (positive and negative).; Did the project use the practice?: Navy NMCI: Yes--The SLAs include measures for technical IT performance and customer satisfaction. In addition, the contract requires the service provider to measure and report on overall customer satisfaction with NMCI services.; Did the project use the practice?: NIMA IT/IS: Yes--The SLAs contain technical IT performance measures. In addition, after a function is transitioned, the contractor must provide a plan that includes measures for customer satisfaction, which the provider is contractually obligated to meet. Practice: Review and update performance requirements periodically.; Did the project use the practice?: Air Force C4 Services: Yes--The performance requirements were revised about 2 years after contract award.; Did the project use the practice?: Army TAC-SWA: Yes--The contract is reviewed and modified as needed on the basis of additions or changes to such requirements. According to TAC-SWA and provider officials, changes are coordinated between the government and the contractor before they are finalized.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, performance requirements are updated as required and reviewed on an annual basis before the decision is made to execute the option year on the contract.; Did the project use the practice?: Navy NMCI: Yes-- Performance reviews and adjustments are ongoing. For example, the Navy is in the process of refining its SLAs to ensure that they adequately reflect technical performance and customer satisfaction needs.; Did the project use the practice?: NIMA IT/IS: Yes--According to NIMA, requirements are updated as needed or once a year when the government is assessing whether to exercise its annual option. Practice: Require the provider to meet the minimum performance in each category of service.; Did the project use the practice?: Air Force C4 Services: Yes--The minimum performance is defined in the contract.; Did the project use the practice?: Army TAC-SWA: Yes--The contract defines minimum performance requirements.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The minimal acceptable performance criteria the contractor must meet are based upon commercial industry standards and are defined in the contract.; Did the project use the practice?: Navy NMCI: Yes--The SLAs provide the basic level of service the contractor must deliver for every NMCI seat.; Did the project use the practice?: NIMA IT/IS: Yes--The provider must meet the performance standards as specified in the contract, SLAs, and NIMA's performance assessment plan. Practice: Require the provider to achieve escalating performance standards at agreed-upon intervals.; Did the project use the practice?: Air Force C4 Services: No--According to Kirtland Air Force Base officials, they did not include escalating performance standards because the focus of this contract was on meeting requirements at the least cost.[A]; Did the project use the practice?: Army TAC-SWA: Not applicable--According to a TAC-SWA project official, this practice was not applicable because NETCOM believed that the performance standards in the contract were already high. Provider officials also stated that the performance requirements in the contract were high. The Army expected, and the contractor agreed, to meet these standards immediately.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract defines negative, acceptable, and positive incentive ranges associated with escalating performance standards.; Did the project use the practice?: Navy NMCI: No--According to NMCI program officials, the Navy expected vendors to develop their pricing proposal assuming 8 years[B] of providing service in accordance with the SLAs included in the RFP.; Did the project use the practice?: NIMA IT/IS: Yes--The contract allows for the redefinition of service levels. In addition, according to NIMA program officials, the need for escalating performance standards will be evaluated on an as-needed basis as well as during the semiannual award fee analysis period and during the annual review on whether to exercise the contract option. However, such escalating standards have not yet been identified. Practice: Incorporate sufficient flexibility so that minimum acceptable performance can be adjusted as conditions change, as the provider becomes more adept at satisfying customer demands, and as improvement goals are achieved.; Did the project use the practice?: Air Force C4 Services: No--The contracting officer said that the requirements should be built into the contract, which can be modified if necessary.; Did the project use the practice?: Army TAC-SWA: Not applicable--According to a TAC-SWA project official, NETCOM believed that the performance standards in the contract were already high, and provider officials agreed. According to the contracting officer, the military theater where performance is delivered is not an area where anything less than meeting the defined standards is acceptable.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, the contract incorporates sufficient flexibility to adjust minimal acceptable performance as conditions change. For example, the contract includes incentives for the service provider to introduce new capabilities and new services within the scope of the performance work statement without further competition.; Did the project use the practice?: Navy NMCI: Yes--The contract includes a provision requiring the provider to submit an annual plan for technology refreshment and deployment. This plan is to include proposed revisions to the contract and an estimate of changes in performance that would result.; Did the project use the practice?: NIMA IT/IS: Yes- -According to NIMA, the performance measures are designed to first stabilize NIMA's IT environment and then to be adjusted to bring minimum performance into alignment with industry best practices. Also, the contract indicates that the SLAs may be further defined throughout the contract life. Practice: Use SLAs to clearly articulate all aspects of performance, including management, processes, and requirements.; Did the project use the practice?: Air Force C4 Services: Yes--The SLA defines the requirements, processes, and who is responsible for meeting the requirements.; Did the project use the practice?: Army TAC-SWA: Yes-- The TAC-SWA project did not use SLAs, but the contract addresses the management, processes, and requirements associated with the project.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract defines the requirements, service performance standards, thresholds, objectives, and metrics as well as the requirements for quarterly management reviews and quality assurance plans.; Did the project use the practice?: Navy NMCI: Yes--According to NMCI officials, SLAs cover all aspects of provider performance. For example, the E-mail services SLA includes location and frequency of the service, performance categories, the performance measures, and methods of measurement.; Did the project use the practice?: NIMA IT/IS: Yes--The contract incorporates SLAs, which articulate performance requirements and take effect when the function is transitioned to the provider. Practice: Client and provider work together to define the appropriate number of SLAs and appropriate structure for each.; Did the project use the practice?: Air Force C4 Services: Yes--Although the original SLA was prepared by the Air Force, the revised SLA included contractor input and concurrence.; Did the project use the practice?: Army TAC- SWA: Yes--SLAs are not used but, according to TAC-SWA and provider officials, they work together on any contract modifications dealing with requirements and performance levels. In addition, the partnering clause in the contract emphasizes a mutual commitment between government and industry to work as a team.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS did not work with the service provider to determine the number and structure of these requirements. However, according to MHS and provider officials, they have worked together on revisions to these requirements.; Did the project use the practice?: Navy NMCI: Yes--The SLAs were developed by the Navy with the help of a third-party contractor. According to the NMCI Deputy Director for Enterprise Operations, it would have been inappropriate to work with individual competing contractors before the contract was awarded. However, NMCI staff and the service provider are now working together on SLA revisions.; Did the project use the practice?: NIMA IT/IS: Yes- -According to NIMA and service provider officials, they used a partnering contracting approach to jointly define, develop, and structure the SLAs. Practice: Specify circumstances under which the provider is excused from performance levels mandated by master service agreements.; Did the project use the practice?: Air Force C4 Services: Yes--The contract includes a clause that the contractor is excused in the event of government delay of work.; Did the project use the practice?: Army TAC- SWA: Yes--The contract includes clauses that the provider is excused from performance levels if there are government delays or factors beyond its control.; Did the project use the practice?: MHS/ITO Help Desk: Yes--For example, the provider's proposal, which is incorporated as part of the contract, states that during transition periods, metrics will not be reported for incentive and penalty purposes. As a result, according to program officials, the provider would be excused from required performance levels when baseline requirements are being established for new applications.; Did the project use the practice?: Navy NMCI: Yes--Contract terms and conditions specify the situations where the service provider does not have to meet the SLAs. For example, during transition, the service provider does not have to meet the performance levels set by the SLAs.; Did the project use the practice?: NIMA IT/IS: Yes--The contractor can be excused from meeting performance levels with the permission of the contracting officer if circumstances occur beyond the service provider's control. Practice: Client and provider work together to identify SLAs for which compensation is based, while additional SLAs may be defined to manage performance.; Did the project use the practice?: Air Force C4 Services: Yes--According to Kirtland Air Force Base officials, the service provider had input on the revised SLAs, including those for compensation.; Did the project use the practice?: Army TAC-SWA: Not applicable--SLAs are not used.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--Before the contract was awarded, MHS did not work with the contractor to identify SLAs for which compensation is based. However, according to MHS and provider officials, after award, there have been instances in which they have worked together on refining the SLAs, including those affecting compensation.; Did the project use the practice?: Navy NMCI: Yes--Meetings were held with potential bidders to obtain their input in defining the SLAs, including discussions of compensation. In addition, Navy and the provider are now refining the SLAs, including those affecting compensation.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA and the service provider work together in transformation teams to develop SLAs, including those affecting compensation. Practice: The contract should include clauses for; ; * determining pricing structures;; * performing customer satisfaction surveys and using the results to redefine performance levels;; * terminating the contract, including early terminations;; * resolving contract disputes in a timely manner;; * taking work away, without penalty, from the provider for nonperformance;; * declaring a significant event that can lead to a change in the contract;; * defining performance requirements; and; * conducting regularly scheduled meetings.; Did the project use the practice?: Air Force C4 Services: Limited--All clauses are included in the contract, except a clause pertaining to customer satisfaction surveys.; Did the project use the practice?: Army TAC-SWA: Yes--All contract clauses are included.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--All contract clauses are included.; Did the project use the practice?: Navy NMCI: Yes--All contract clauses are included.; Did the project use the practice?: NIMA IT/IS: Yes--All contract clauses are included. Practice: Consider setting up master services agreement under which all arrangements between client and provider operate.; Did the project use the practice?: Air Force C4 Services: Yes--According to Kirtland Air Force Base officials, the contract, which incorporates the contractor proposal, governs all arrangements between the government and the contractor and is considered the master services agreement.; Did the project use the practice?: Army TAC-SWA: Yes--According to TAC- SWA project officials, the contract is considered the master services agreement.; Did the project use the practice?: MHS/ITO Help Desk: Yes- -According to MHS program officials, the contract is considered the master services agreement.; Did the project use the practice?: Navy NMCI: Yes--According to NMCI program officials, the contract is considered to be a master services agreement.; Did the project use the practice?: NIMA IT/IS: Yes--According to NIMA, the contract is considered the master services agreement. Practice: Include appropriate representation from each major organizational unit on contract negotiation team.; Did the project use the practice?: Air Force C4 Services: Not applicable--The C4 Services contract was awarded under a sealed bid process and was not negotiated.; Did the project use the practice?: Army TAC-SWA: Yes-- Representatives from the field and headquarters commands and the contracting office participated in contract negotiation.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Staff from affected program management offices and the chief information offices were on the negotiation team.; Did the project use the practice?: Navy NMCI: Yes--The contracting team was built with experts from each of the major systems commands. Also, the source selection evaluation board consisted of more than 50 people from various commands.; Did the project use the practice?: NIMA IT/IS: Yes--The NIMA contract negotiating team included representatives from each major organizational unit. Practice: Specify the use of volume purchases to obtain optimal discounts.; Did the project use the practice?: Air Force C4 Services: Not applicable--According to Kirtland Air Force Base officials, such volume purchases were not relevant to this contract.; Did the project use the practice?: Army TAC-SWA: Not applicable--According to TAC-SWA project officials, such volume purchases were not relevant to this contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- Call bands[C] are used to obtain optimal discounts on the number of calls being fielded to the help desk.; Did the project use the practice?: Navy NMCI: Yes--The contract includes volume discounts. For example, seat moves, adds, and changes are purchased in bulk only, because prices for these actions are lower when purchased in groups of 250.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has authorized the provider to use government sources, such as the Federal Supply Schedule, in procuring products, services, and supplies related to this contract. According to the contracting officer, this authorization was made so that the provider could take advantage of discounts available to the government. Practice: Use third-party assistance in negotiating and developing the contract.; Did the project use the practice?: Air Force C4 Services: No--This was not done because the Air Force staff believed that it had adequate expertise available in-house.; Did the project use the practice?: Army TAC-SWA: No--NETCOM officials believed that the Army's designated contracting agency had adequate expertise and experience.; Did the project use the practice?: MHS/ITO Help Desk: Yes- -Private-sector and other government organizations provided assistance.; Did the project use the practice?: Navy NMCI: Yes--The Navy used private-sector firms to assist in developing the overall NMCI concept and negotiation strategies as well as in drafting the contract documents.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA obtained help from (1) private contractors to help develop the performance work statement, SLAs, and award fee plan; (2) another agency on a particular contract technique; and (3) a private-sector firm to compare the vendor's proposal with industry best practices and trends and to attend some negotiation meetings to answer questions. Practice: Sign contract after contract negotiations and final vendor selections.; Did the project use the practice?: Air Force C4 Services: Yes--The contract was signed after negotiations and final vendor selection.; Did the project use the practice?: Army TAC-SWA: Yes--The contract was signed after negotiations and final vendor selection.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract was signed after negotiations and final vendor selection.; Did the project use the practice?: Navy NMCI: Yes--The contract was signed after negotiations and final vendor selection.; Did the project use the practice?: NIMA IT/IS: Yes--The contract was signed after negotiations with the selected vendor. Source: GAO. [A] Requiring a provider to meet escalating requirements and focusing on achieving results at the least cost are not mutually exclusive goals. As the provider becomes more familiar with the client organization, it may be able to exceed the original performance requirements at the same, or possibly lower, cost. [B] The NMCI contract was subsequently extended to 7 years, with an option for an additional 3 years. [C] Call bands are call volume ranges used to determine contractor pricing. [End of table]: [End of section] Appendix V: Projects' Implementation of Phase IV: Select the Provider(s): [See PDF for image] [End of figure] Critical to the success of any project to outsource IT services is the identification of potential providers and the ultimate selection of a provider(s) that will best meet the needs of the organization. Developing a strategy that will lead to the selection of the "right contractor" is especially important in a performance-based acquisition.[Footnote 47] The overall success of the outsourcing project requires the contractor to understand the performance-based approach, know or develop an understanding of the organization's requirement, have a history of performing exceptionally in the field, and have the processes and resources in place to support the mission. The seven practices in this phase that we used to evaluate the five projects are as follows: * Conduct research on the state of the market, vendors, and technology before defining vendor selection criteria. * Identify and evaluate various sourcing solutions (e.g., single vendor, multivendor, and alliance). * Define a process for selecting vendors to be providers. * Define vendor selection and evaluation (acceptance) criteria at the outset. * When issuing an RFP, identify services with expected performance levels and define client and provider roles and responsibilities. * Use third-party assistance with expertise in a variety of outsourcing arrangements when selecting provider(s), including developing the RFP. * Conduct due diligence activities to verify vendor capabilities before signing the contract. As figure 9 illustrates, all five projects largely implemented the practices. Figure 9: Project Implementation of Phase IV: Select the Provider(s): [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 7 provides detailed information on whether and how the projects in our review implemented each of the seven practices in this phase. Table 7: Summary of Projects' Use of Phase IV Practices: Practice: Conduct research on state of the market, vendors, and technology before defining vendor selection criteria.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force advertised in the Commerce Business Daily and interested parties submitted statements of capabilities. This was done before the Air Force identified the vendor selection criteria.; Did the project use the practice?: Air Force C4 Services: Yes--A Commerce Business Daily notice was published identifying TAC-SWA requirements to interested parties. In addition, a draft RFP was issued before the final solicitation to solicit comments from industry that might affect the requirements. According to the contracting officer, comments received from industry were incorporated into the final solicitation, where applicable. Additionally, a preproposal conference was conducted in the overseas operations location to inform industry representatives about issues and the procurement strategy.; Did the project use the practice?: Army TAC- SWA: Yes--MHS had a third-party contractor perform market research.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The Navy and a private-sector firm performed market research to, for example, help define the market conditions and vendor selection criteria. The Navy also held meetings with leading companies that had outsourced IT services on an enterprisewide level. Finally, more than 200 companies participated in the July 1999 NMCI Industry Day Conference, which informed companies about the NMCI vision, requirements, and procurement strategy.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA staff and a contractor researched preferential providers before defining vendor selection criteria. NIMA program officials noted that they limited their analysis to such providers because they had previously decided on a strategy to directly convert their IT/IS activities to a preferential provider. Practice: Identify and evaluate various sourcing solutions (e.g., single-vendor, multivendor, and alliance).; Did the project use the practice?: Air Force C4 Services: Yes--As outlined in OMB Circular A- 76, the Air Force evaluated private-sector and internal government staff proposals to satisfy the C4 services requirements.; Did the project use the practice?: Air Force C4 Services: Yes--A multivendor approach has been used in the past, which the Army opted not to continue. Instead, NETCOM chose to consolidate its requirements and contractor oversight by choosing a single-vendor solution.; Did the project use the practice?: Army TAC-SWA: Yes--According to an MHS program official, MHS evaluated continuing with the status quo, using another federal agency's help-desk services as part of a cross-services agreement, and contracting with a commercial firm. According to the MHS program office, MHS chose a single-vendor solution to consolidate requirements and performance accountability.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The Navy evaluated single-vendor and multivendor approaches. A joint Navy and Marine Corps team determined that single-vendor point of contact for accountability and responsibility was critical to mission success.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA considered the sourcing solutions allowed under OMB Circular A-76, including the direct conversion to a preferential provider, holding a public/private competition, or obtaining an agency cost comparison waiver. NIMA decided that directly converting the selected IT functions to a preferential procurement source[A] was the lowest risk to its mission and NIMA employees. Practice: Define a process for selecting vendors to be providers (e.g., issuing an RFP and prequalifying vendors).; Did the project use the practice?: Air Force C4 Services: Yes--As explained in the invitation to bid, the Air Force followed the two-step process outlined in OMB Circular A-76 for public/private competitions.; Did the project use the practice?: Air Force C4 Services: Yes--The contract was awarded on "best value" considerations of technical and management capabilities, past performance, and price, as defined in the solicitation.; Did the project use the practice?: Army TAC-SWA: Yes--MHS used a selection process that included the prequalification of 13 industry leaders in both medical and commercial help-desk operations. MHS also issued a request for comment and a request for quote, which defined vendor evaluation and selection criteria on the basis of technical approach, past performance, key personnel qualifications, organizational experience, and price.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The NMCI RFP used a negotiated commercial items evaluation process. The vendor evaluation criteria included technical approach, management plan, small business utilization, past performance, and price.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA's decision to use a phased direct conversion of its IT/IS functions to a statutory, preferential procurement vendor limited its vendor selection process to the identification of potential Alaska Native Corporation companies capable of performing the requirements. NIMA decided to take this approach because it believed that it would avoid schedule delays and mission risk that it thought would result from taking other approaches. In addition, NIMA program officials believed that the preferential provider approach would maximize the retention of institutional knowledge whether employees stayed at NIMA or transitioned to the provider. Practice: Define vendor selection and evaluation (acceptance) criteria at the outset.; Did the project use the practice?: Air Force C4 Services: Yes--The criteria were included in the invitation to bid.; Did the project use the practice?: Air Force C4 Services: Yes--The solicitation included vendor evaluation and selection criteria. The source selection evaluation plan describes the evaluation process.; Did the project use the practice?: Army TAC-SWA: Yes--Vendor evaluation and selection were based upon industry-defined help-desk criteria. MHS used a multistep process that included an assessment of minimum qualifications, evaluation of written technical and cost proposals, and oral presentations for qualified vendors.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The final RFP required bidders to have implemented and provided service to at least 100,000 seats, of which 20,000 were to be on the same contract. In addition, bidders were evaluated on their technical approach, management plan, small business utilization, past performance, and price. Finally, vendors in the competitive range had to provide demonstrations of technical network operations centers to verify their ability to achieve required service levels set forth in technical proposals.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA evaluated the past performance and financial capabilities of Alaska Native Corporation companies. According to NIMA, only two companies had demonstrated successful performance in service environments similar to NIMA's requirements. However, to be able to fully meet NIMA's requirements, these vendors formed a joint venture, which was awarded the contract. Practice: When issuing an RFP, identify services with expected performance levels and define client and provider roles and responsibilities.; Did the project use the practice?: Air Force C4 Services: Yes--The services and performance levels were included in the original performance work statement included in the invitation to bid.; Did the project use the practice?: Air Force C4 Services: Yes--The RFP identified the government and contractor roles and responsibilities, and system operational and availability requirements.; Did the project use the practice?: Army TAC-SWA: Yes--The request for quote identified services and expected performance levels and specified a performance- based, incentivized, shared-risk relationship with the service provider. It also defined client and provider roles and responsibilities.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The NMCI solicitation documents identified the required services and expected performance levels. These documents, along with the contract, define the client and provider roles and responsibilities.; Did the project use the practice?: NIMA IT/IS: Yes- -Because this was a sole source contract, an RFP was not issued. However, according to NIMA and provider officials, they worked jointly to refine the requirements and expected performance levels and to define client and provider roles and responsibilities in the contract. Practice: Use third-party assistance with expertise in a variety of outsourcing arrangements when selecting provider(s), including developing the RFP.; Did the project use the practice?: Air Force C4 Services: No--This was not done because the Air Force believed that adequate in-house expertise was available.; Did the project use the practice?: Air Force C4 Services: No--The Army believed that it had adequate government experience and expertise covering technical, resource management, and contracting areas.; Did the project use the practice?: Army TAC-SWA: Yes--A private-sector contractor, the Defense Acquisition University, and GovWorks (an Interior organization offering procurement services to government agencies) provided assistance in this area.; Did the project use the practice?: MHS/ITO Help Desk: Yes- -A private-sector firm assisted with market research and developing vendor pass/fail criteria. Another contractor assisted the Navy with developing SLAs and the technical evaluation of vendors.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA used a private-sector firm and the NIMA Acquisition Center to support the evaluation of the vendor proposal. Practice: Conduct due diligence activities to verify vendor capabilities before signing the contract.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force performed a technical evaluation of the vendor's capabilities. Also, as part of its due diligence activities, the Air Force reviewed the debarred list and DOD's Central Contractor Registry.; Did the project use the practice?: Air Force C4 Services: Yes--NETCOM evaluated the contractor's financial and past performance information before signing the contract.; Did the project use the practice?: Army TAC-SWA: Yes--Minimum vendor qualifications were established and evaluated early in the process for selecting the provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes--As part of due diligence, the Navy ensured that all bidders had relevant experience implementing large seat management contracts. In addition, past performance, including reference checks, was a source-selection evaluation factor.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA researched both parent companies of the new joint venture corporation to ensure that they had financial and technical backing. Source: GAO. [A] Preferential procurement programs are special commercial source programs, such as Federal Prison Industries and the workshops administered by the Committee for the Purchase from the Blind and Other Severely Handicapped under the Javis-Wagner O'Day Act. [End of table]: [End of section] Appendix VI: Projects' Implementation of Phase V: Transition to Provider(s): [See PDF for image] [End of figure] In this phase, client organizations transfer responsibility of IT functions to one or more providers. A clear definition of responsibilities among the various parties and the careful consideration of employees' needs matched against the organization's needs enable both the client and provider to focus on execution and give staff confidence in their future employment. If the contractor is assuming responsibility for functions previously performed by federal employees, it is especially important that the organization communicate a clear transition process. Without such communication, an outsourcing project can be negatively affected if misinformation and mistrust ensues. The 11 practices in this phase that we used to evaluate the five projects are as follows: * Communicate a clear transition process to all key players from both client and provider organizations. * Clearly communicate to employees what is going to happen and when it is going to happen. * Establish a client transition team with representatives from across the organization to facilitate the transition. * Place the transition under a single program manager. * Create client/provider transition teams to address short-term transition tasks as required. * Recognize that it takes time to effect transition and plan accordingly. * Encourage the transition of staff to the provider, where appropriate, using bonuses, stock options, and other appropriate methods. * Develop employee-retention programs and offer bonuses to keep key people, where appropriate. * When consistent with organizational objectives, assist employees who do not want to transfer in finding other jobs, either within an organization or at another organization. * Document key information to preserve organizational knowledge in the event that one or more providers change. * Use change management strategies to help client employees deal with the transition. Figure 10 demonstrates that the five projects in our review were largely using the practices. Figure 10: Project Implementation of Phase V: Transition to Provider(s): [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 8 provides information on whether and how the projects implemented each of the 11 practices in this phase. Table 8: Summary of Projects' Use of Phase V Practices: Practice: Communicate a clear transition process to all key players from both client and provider organizations.; Did the project use the practice?: Air Force C4 Services: No--There were two transition plans prepared, one by the government and one by the provider. However, the government's plan contained conflicting direction. In addition, provider officials said that the two separate transition plans were not well coordinated and that the Air Force did not have a strong advocate to ensure that the transition process was well planned and executed. Both provider and Kirtland Air Force Base officials acknowledged that there were problems with the transition process, including incorrect information about upgrades that would be made before the transition and staff morale problems that hampered knowledge transfer to the provider.; Did the project use the practice?: Army TAC-SWA: Yes--The provider developed a phase-in plan that addressed personnel issues, the integration of added locations, updating plans, and finalizing new subcontract/ sponsorship arrangements. This phase-in plan was discussed at a postaward conference, attended by representatives from both the client and provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract delivery order and the provider's transition plan laid out a transition process. In addition, a kickoff meeting was held between the client and provider to communicate the transition process to all key players.; Did the project use the practice?: Navy NMCI: Yes- -The Navy developed numerous guides and Web sites to communicate with the NMCI community about site, technical, and personnel transition processes, procedures, and tasks.; Did the project use the practice?: NIMA IT/IS: Yes--Several NIMA transformation teams were formed, consisting of both client and provider personnel, to develop a single transition plan. The plan was developed to formalize communications with all key players. Practice: Clearly communicate to employees what is going to happen and when it is going to happen.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force kept its employees informed of the outsourcing project through town hall meetings and internal briefings. As part of this process, employees were informed about the reduction-in-force process and the procedure for those interested in being transitioned or reassigned.; Did the project use the practice?: Army TAC-SWA: Limited--According to TAC-SWA officials, the Army communicated with its employees, but they did not provide documentation supporting any communication efforts. However, the provider was required to submit a phase-in plan, which included a section on communicating with the incumbent contractors' staff.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The government task manager for the Tri-Service Infrastructure Management Office held regular meetings with MHS program officials to discuss transition issues. The MHS program office also periodically sent global E-mails to MHS staff on transition activities and MHS leadership made on-site visits to San Antonio, the location of the help-desk function.; Did the project use the practice?: Navy NMCI: Limited--The Navy has used its normal chain of command to communicate transition information, but found that the implementation was uneven. As a result, some staff did not know current information about how NMCI would affect them until the provider was ready to contact them regarding their possible transition to the contractor. However, according to the NMCI Director's office, this problem was somewhat mitigated by the provider's Web site that provides transition information to all NMCI customers/users.; Did the project use the practice?: NIMA IT/IS: Yes--With help from a contractor, NIMA developed a communications plan. The plan included town hall meetings, global E-mail messages, and a Web site posting that provided employees with access to transition information. Practice: Establish client transition team with representatives from across the organization to facilitate the transition.; Did the project use the practice?: Air Force C4 Services: Yes--The government's transition team included personnel from the functional areas being outsourced (i.e., communications and IT services) as well as personnel from other offices.; Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-SWA officials, the contracting officer's representative assisted the provider with the transition. The officials also stated that Army technical points of contact at each site also helped facilitate the transition to the new contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS program officials for the Tri-Service Infrastructure Management Office established transition teams with representatives from the former Tri-Service Medical System Support Center contract.; Did the project use the practice?: Navy NMCI: Yes--The headquarters-level transition team consists of representatives from the NMCI Director's Office and the program management office. Also, every site has a transition team made up of customer and provider personnel.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA's transition team included representatives from various offices within NIMA to help facilitate the transition. Practice: Place transition under single program manager.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force established a single program manager, who is referred to as the functional area chief.; Did the project use the practice?: Army TAC- SWA: Yes--The Army designated the contracting officer's representative to oversee the transition.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The transition was assigned to the government task manager in the Tri-Service Infrastructure Management Office.; Did the project use the practice?: Navy NMCI: Yes--NMCI's Director has overall responsibility for the transition.; Did the project use the practice?: NIMA IT/IS: Yes--The transition process is under a single program manager. Practice: Create client/provider transition teams to address short-term transition tasks as required.; Did the project use the practice?: Air Force C4 Services: Yes--During the transition phase, management from both the government and the provider met weekly to track the progress of the transition.; Did the project use the practice?: Army TAC-SWA: Yes--Personnel from both the Army and provider formed three transition teams, one for each site location. These teams worked with the contracting officer's representative and the provider's project manager to deal with transition issues.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Client/provider transition teams were established to address short-term transition tasks. The teams met weekly to address any transition issues.; Did the project use the practice?: Navy NMCI: Yes--NMCI and the provider designated staff to work together on short-term transition issues.; Did the project use the practice?: NIMA IT/IS: Yes--An integrated NIMA transformation team comprising NIMA and provider staff, was responsible for addressing short-term tasks, including (1) transition planning, (2) resources and recruitment, (3) program budget, and (4) contract development and costing. Practice: Recognize that it takes time to effect transition and plan accordingly.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force allowed 60 days for the transition.; Did the project use the practice?: Army TAC-SWA: Yes--The contract allowed a 60-day phase-in period.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract provided for a transition period of 30 to 90 days. According to MHS program officials, the transition period lasted 90 days.; Did the project use the practice?: Navy NMCI: Limited--The Navy said that it had initially underestimated the scope of the project and the magnitude of the problems brought on by legacy applications and associated information assurance and cyber-security issues. According to NMCI program officials, addressing these problems and completing the operational testing that was mandated by the Congress subsequent to contract award led to the transition period being lengthened from 2-1/ 2 to 3-1/2 years. NMCI officials also said the extension will allow the Department of the Navy to have time to operate NMCI as a fully transitioned enterprise before having to decide whether to exercise the contract option.; Did the project use the practice?: NIMA IT/IS: Yes-- According to NIMA, it has scheduled its seven functional areas to be transitioned to the provider over a 3-to 5-year period. The provider will not assume responsibility for a given functional area until the NIMA transition review board has given its approval to the provider's turnover plan. Practice: Encourage transition of staff to provider, where appropriate, using bonuses, stock options, and other appropriate methods.; Did the project use the practice?: Air Force C4 Services: Yes--The Air Force staff were given the right to transfer to the contractor. It was left up to the contractor to decide whether to offer incentives or not, which it chose not to do.; Did the project use the practice?: Army TAC- SWA: Yes--The Army had no personnel expected to transition to the provider since the activity being outsourced had previously been contracted out. However, the provider was interested in retaining existing contractor staff and had a program to recruit them.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to provider officials, they extended offers to employees from the prior contractor and some MHS staff.; Did the project use the practice?: Navy NMCI: Yes--The NMCI contract provides that displaced federal employees that transition to the provider under employment openings as a result of NMCI will be given guaranteed 3 years of employment with the provider, a 15 percent salary increase, and a sign-on bonus.; Did the project use the practice?: NIMA IT/IS: Yes--The contract requires the provider to actively recruit, hire, and make reasonable efforts to retain NIMA staff. If the provider meets certain target thresholds for recruiting NIMA staff, it receives a monetary incentive. According to the provider's general manager, the company offered signing bonuses to NIMA employees that wanted to transition. Practice: Develop employee-retention programs and offer bonuses to keep key people, where appropriate.; Did the project use the practice?: Air Force C4 Services: No--According to Kirtland Air Force Base officials, they did not develop an employee-retention program because they did not believe that they could offer incentives.; Did the project use the practice?: Army TAC-SWA: Not applicable--Since this activity was previously contracted out, employee retention of Army civilian personnel was not relevant.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The government task manager persuaded key government employees from the Tri-Service Medical System Support Center to temporarily work at the Tri-Service Infrastructure Management Office until positions became available with a contractor that provides support to this office. In addition, according to a program official, to keep expertise in particular applications, key employees were transferred to the MHS program executive office and Tri-Service Infrastructure Management Office.; Did the project use the practice?: Navy NMCI: Yes--NMCI program officials noted that the Navy has human resources programs that would allow its organizations a range of options, including bonuses, to help keep key staff. Additionally, the Navy directed all Navy and Marine Corps commands affected by NMCI to develop civilian personnel transition plans to mitigate the impact of NMCI implementation on employees.; Did the project use the practice?: NIMA IT/IS: Yes--According to NIMA program officials, NIMA categorized affected employees into three tiers, on the basis of whether they would be allowed to transition to the provider or whether there would be restrictions on their activities if they did transition. This distinction was made to comply with government conflict of interest rules. According to the program manager, NIMA also sought personnel who wanted to remain at the agency to help monitor the provider's performance and considered the qualifications of those who wished to serve in this role prior to deciding who to retain. Practice: When consistent with organizational objectives, assist employees who do not want to transfer in finding other jobs, either within an organization or at another organization.; Did the project use the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base officials stated that they have reduction-in-force procedures to help place civilian employees in other Air Force jobs. Employees also receive priority status for other DOD job openings. Further, the base education office provided assistance with outplacement; résumé writing; and, for those nearing retirement, planning advice.; Did the project use the practice?: Army TAC-SWA: Not applicable--According to the TAC- SWA contracting officer, no Army personnel were displaced by the contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- According to MHS program officials, the government task manager assisted government employees with the Tri-Service Medical System Support Center staff in finding new positions with other MHS organizations. For example, he located open positions and contacted the organizations.; Did the project use the practice?: Navy NMCI: Yes--NMCI program officials said that existing Navy civilian personnel programs are in place to assist employees. They noted that two Navy organizations found other jobs within their command for their small number of employees affected by NMCI. Other Navy organizations are also reassigning affected personnel to other government jobs, where applicable.; Did the project use the practice?: NIMA IT/IS: Yes--For staff that did not want to transition, NIMA provided various types of assistance, such as retirement planning, résumé writing, and interviewing skills. In addition, NIMA reported that if an employee did not want to transition to the provider, that it would consider re- adjusting the employee's work assignment or provide training to support the individual's placement within other areas. Practice: Document key information to preserve organizational knowledge in the event that one or more providers change.; Did the project use the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base's work functions and workload size are documented in the contract.; Did the project use the practice?: Army TAC-SWA: Yes--Documentation produced by the provider, including maintenance logs, become the property of the government.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The Tri-Service Medical System Support Center processes and procedures were captured during transition. The provider's processes and procedures are currently being captured. The contract also includes requirements on documenting key information.; Did the project use the practice?: Navy NMCI: Yes--NMCI maintains a Web portal with organizational knowledge information. In addition, a new Intranet site is being developed with up-to-date interactive organizational knowledge relevant to the many varied NMCI communities.; Did the project use the practice?: NIMA IT/IS: Yes--The executing NIMA transformation team was charged with documenting the activities for each functional area that is being outsourced. This includes developing and capturing operational processes and procedures for each of the functional areas. Practice: Use change management strategies to help client employees deal with the transition.; Did the project use the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base held town hall meetings and internal briefings to communicate the objectives of the outsourcing project and the changes that would take place internally. It also had one-on-one meetings with each employee that would be displaced.; Did the project use the practice?: Army TAC-SWA: Not applicable--The activity had been previously contracted out.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, the government task manager for the Tri-Service Infrastructure Management Office assisted government employees with finding new positions and kept them informed of transition issues. Weekly E-mails were also sent out to all affected employees.; Did the project use the practice?: Navy NMCI: Yes--The Navy has provided employees affected by NMCI with a range of employment options and incentives.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA's change management strategies included training seminars offered by its human resources office on résumé writing, interviewing skills, career transition workshops, and employment trends. Source: GAO. [End of table]: [End of section] Appendix VII: Projects' Implementation of Phase VI: Manage Provider(s) Performance: [See PDF for image] [End of figure] The effectiveness with which the performance of the provider(s) is managed--the focus of this phase--is critical to the successful implementation of an outsourcing project. Indeed, according to Gartner, Inc., an outsourcing project can be thwarted by poorly designed, funded, and delivered processes for managing the delivery of services.[Footnote 48] This firm also points out that an enterprise needs to retain the resources to oversee the planning and implementation of the IT services being delivered by the provider to ensure that the contractor meets the client's business needs throughout the life of the agreement.[Footnote 49] Moreover, frequent and clear communication between the client and provider ensures that potential problems are resolved before they cause disruptions. In addition, performance reviews should take place regularly to keep the project on course, measure performance levels, and make adjustments as necessary.[Footnote 50] The 11 practices in this phase that we used to evaluate the five projects are as follows: * Consider incentives to motivate provider(s) to exceed performance requirements. * Use penalties to motivate provider(s) to meet performance requirements. * Periodically undertake studies to assess (1) how the provider's performance compares with the value being delivered to similar clients and (2) the extent to which the provider's performance is improving over time. * Schedule periodic working-level meetings with both the end-user groups and the provider to review the provider's performance. * Conduct executive-level oversight meetings with the provider's senior management to review provider's performance. * Distribute performance data to stakeholders. * Reserve audit rights on performance data supplied by the provider. * Ensure that the provider measures and reports on performance. * Work with the provider to redefine service levels, as appropriate. * Sample performance data frequently enough to perform trend analysis and to permit extrapolation based on historical data. * Allow employees and possibly stakeholders to rate the provider on a regular basis. As shown by figure 11, the five projects in our review generally implemented the practices. Figure 11: Project Implementation of Phase VI: Managing Provider Performance: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 9 depicts whether and how the five projects in our review implemented each of the 11 practices in this phase. Table 9: Summary of Projects' Use of Phase VI Practices: Practice: Consider incentives to motivate provider(s) to exceed performance requirements.; Did the project use the practice?: Air Force C4 Services: No--The Air Force did not include incentives in the contract because the agency expects the contractor to meet the contract's requirements without incentives. Specifically, according to the former functional area chief, because the contract was issued under the rules established by OMB Circular A-76, the focus of the project was on achieving cost savings, and incentives were not included in the contract. As a result, he noted that the provider did not have any incentive to be innovative or creative.; Did the project use the practice?: Army TAC-SWA: No--There are no monetary incentives, but, according to TAC-SWA officials, the fact that the agency will prepare a performance evaluation report can help motivate a provider to meet requirements. However, the contracting officer said that incentives might have been useful to motivate the provider to exceed performance requirements.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract includes incentives linked to each performance requirement. Each requirement has a positive, negative, and acceptable performance range that provides a basis for monetary incentives (as well as penalties).; Did the project use the practice?: Navy NMCI: Yes- -The NMCI contract includes a one-time payment of $10 million if the provider successfully completes full operational capability. The contract also has incentives for customer satisfaction, information assurance, and small business and small disadvantaged business participation.; Did the project use the practice?: NIMA IT/IS: Yes--The contract includes monetary incentives in accordance with an award fee plan. It also includes share-in-savings provisions to encourage process improvements. Practice: Use penalties to motivate provider(s) to meet performance requirements, such as; ; * assess penalties for failure to perform at required individual service as well as aggregate service levels;; * apply penalties in the form of credit to the client;; * increase penalty for recurring deficient performance;; * hold back a percentage of provider's pay for a particular service until performance requirements are met;; * refund a penalty if the provider returns to agreed-upon performance levels within a designated period of time; and; * ensure that the provider will cover costs, but not profit, when a particular performance requirement is not met.; Did the project use the practice?: Air Force C4 Services: Yes--The contract contains monetary penalties that are linked to each of the performance requirements. If imposed, they would reduce the amount of the payment owed to the provider.; Did the project use the practice?: Army TAC-SWA: Limited-- The contract does not have specific monetary penalties linked to performance requirements. However, the contract states that unacceptable work must be redone at the provider's expense, and, if the defects and services cannot be corrected, the government may reduce the contract's price to reflect the reduced value of the services performed. TAC-SWA project officials stated that monetary penalties were not included in the contract because the Army was concerned that contractors might not bid on the contract, and administrating this type of contract is more difficult.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--The contract includes monetary penalties linked to each performance requirement. Each requirement has positive, negative, and acceptable performance ranges that provide a basis for monetary penalties as well as incentives.; Did the project use the practice?: Navy NMCI: Yes--The contract includes monetary penalties in the form of credits to the agency if the provider fails to perform to the levels specified in the SLAs.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA did not include monetary penalties in its contract. However, if the provider does not meet certain minimum performance standards, only its costs would be covered. Practice: Periodically undertake studies to assess: (1) how the provider's performance compares with the value being delivered to similar clients and (2) the extent to which the provider's performance is improving over time.; Did the project use the practice?: Air Force C4 Services: Limited--(1) This has not been done because, according to a project official, the Air Force was not aware of similar clients.; (2) This is done through periodic monitoring of performance by quality assurance evaluators and through monthly review meetings held by the functional area chief to identify and address any problems that are starting to occur.; Did the project use the practice?: Army TAC-SWA: No--TAC-SWA officials stated that they address performance issues at the time that they occur and that they have not performed any studies.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The provider's performance is reviewed monthly by the government task manager and in quarterly management reviews of the provider's performance. According to MHS program officials, the results are compared with peers and reviewed for how the provider has improved over time.; Did the project use the practice?: Navy NMCI: Limited--(1) NMCI only has performance data since October 2002. The Navy intends to do such a study but has not established a schedule for it.; (2) The provider's improvements over time are being monitored monthly.; Did the project use the practice?: NIMA IT/IS: Limited--(1) At this time, NIMA has not undertaken such studies because it only recently transitioned functions to the provider. However, NIMA program officials stated that they intend to use a private-sector firm to periodically compare the provider's performance with those of similar organizations.; (2) The provider's improvements over time are being monitored monthly. Practice: Schedule periodic working-level meetings with both the end- user groups and the provider to review the provider's performance.; Did the project use the practice?: Air Force C4 Services: Yes--Monthly meetings are held at which the contractor briefs the functional area chief and other Kirtland Air Force Base representatives.; Did the project use the practice?: Army TAC-SWA: Yes--According to the provider and TAC-SWA officials, they hold daily meetings to discuss any issues.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Periodic working-level meetings are held with the government task manager and the program management office representatives to obtain help-desk feedback.; Did the project use the practice?: Navy NMCI: Yes--At each implemented site, the provider and government managers meet on a frequent, as-needed basis to review performance and resolve any issues.; Did the project use the practice?: NIMA IT/IS: Yes--Quarterly performance management review meetings are held and are attended by NIMA and the provider. In addition, the provider hired an ombudsman to obtain anonymous comments from NIMA staff and management on its performance. Practice: Conduct executive-level oversight meetings with the provider's senior management to review provider's performance.; Did the project use the practice?: Air Force C4 Services: Yes--Air Force schedules executive-level oversight meetings whenever they are necessary. A recent meeting included the president of the provider.; Did the project use the practice?: Army TAC-SWA: Yes--According to provider and TAC-SWA officials, quarterly in-progress reviews are held with the provider's corporate management and the battalion commander.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Executive oversight meetings are held quarterly with the provider's senior management to review performance.; Did the project use the practice?: Navy NMCI: Yes--The NMCI Director and the provider executive meet weekly to review performance and discuss other NMCI implementation issues. In addition, the Department of the Navy recently established an operations advisory board consisting of Department of the Navy leadership and the provider. The goal of this board is to focus senior leadership on issues affecting NMCI in order to establish priorities and make decisions.; Did the project use the practice?: NIMA IT/IS: Yes--This is done through the quarterly review meetings. Practice: Distribute performance data to stakeholders.; Did the project use the practice?: Air Force C4 Services: Yes--Performance data were not distributed to stakeholders at the beginning of the project, but began after complaints from stakeholders about the need for such information. Metrics, such as system reliability or "uptime," are now provided to stakeholders on line.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative distributes monthly performance reports to Army stakeholders for review. Performance issues or problems identified are discussed with the contracting officer's representative and the contracting officer, and corrective actions are planned to prevent reoccurrence.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The help-desk monthly reports, including results against performance metrics, are e- mailed to stakeholders monthly.; Did the project use the practice?: Navy NMCI: Yes--The NMCI Navy and Marine Corps program managers are responsible for providing SLA performance data to their commands.; Did the project use the practice?: NIMA IT/IS: Yes--Performance data are distributed at quarterly review meetings, which according to the contracting officer, are attended by stakeholders. Practice: Reserve audit rights on performance data supplied by the provider.; Did the project use the practice?: Air Force C4 Services: Yes--Addressed in the contract.; Did the project use the practice?: Army TAC-SWA: Yes--Addressed in the contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Addressed in the contract.; Did the project use the practice?: Navy NMCI: Yes--Addressed in the contract.; Did the project use the practice?: NIMA IT/IS: Yes--Addressed in the contract. Practice: Ensure that provider measures and reports on performance.; Did the project use the practice?: Air Force C4 Services: Yes--As required by the quality control plan in the contract, the provider makes information available on product or service quality and any actions needed to correct decreases in quality. In addition, according to the former functional area chief, the provider provides performance information during monthly meetings.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative receives performance information in daily meetings with the provider's project manager.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract requires the provider to submit monthly and quarterly performance reports that are then reviewed and verified.; Did the project use the practice?: Navy NMCI: Yes--The contract specifies that the provider must measure and report on whether its SLA performance goals are being met.; Did the project use the practice?: NIMA IT/IS: Yes--The contract requires the provider to submit data, including measurements of service, quarterly. Practice: Work with provider to redefine service levels, as appropriate.; Did the project use the practice?: Air Force C4 Services: Yes--This is done on an as-needed basis. For example, the service levels were redefined in April 2002.; Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-SWA and provider officials, they have worked together to redefine performance requirements in the contract.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--MHS and the provider have worked together to redefine SLAs. For example, the first call closure performance metric was redefined to be more realistic.; Did the project use the practice?: Navy NMCI: Yes--The Navy and service provider have and continue to refine NMCI SLAs to ensure more precise performance measures and to more accurately capture user satisfaction with the system. For example, a contract modification standardized many of the performance categories that dealt with service availability.; Did the project use the practice?: NIMA IT/IS: Yes--The redefinition of service levels is provided for in the contract. According to NIMA, the redefinition of service levels is expected to occur in the future, generally at the time that the annual contract option is exercised. Practice: Sample performance data frequently enough to perform trend analysis and to permit extrapolation based on historical data.; Did the project use the practice?: Air Force C4 Services: Yes--Air Force quality assurance evaluators monitor performance in accordance with the quality assurance surveillance plan and the performance requirements summary. Any trends identified are addressed in monthly status meetings.; Did the project use the practice?: Army TAC-SWA: Yes--The provider's proposal and quality control plan state that it will provide various trend analyses to the government. According to TAC-SWA project officials, these data are submitted to the contracting officer's representative for analysis.; Did the project use the practice?: MHS/ ITO Help Desk: Yes--Performance data are sampled and reported monthly and quarterly. They are analyzed and verified, including any supporting data. In addition, the provider's knowledge management system provides analysis and trend data to MHS.; Did the project use the practice?: Navy NMCI: Yes--The NMCI Director monitors provider performance to identify trends by assessing provider-supplied information as well as information from Navy independent verification and validation testing and customer satisfaction survey results. In addition, NMCI program officials said that they have requested funding to develop a performance measurement data repository to support trend analyses.; Did the project use the practice?: NIMA IT/IS: Yes--The contract requires data to be provided periodically for NIMA to use for trend and other types of analyses. Practice: Allow employees and possibly stakeholders to rate the provider on a regular basis (e.g., scorecards and quarterly report cards).; Did the project use the practice?: Air Force C4 Services: Limited--The provider conducts annual customer satisfaction surveys, but it is not required to submit the results to the Air Force. Kirtland Air Force Base also obtains ad hoc feedback from employees who, after they report a problem, are asked to provide information about how well the problem was addressed and their overall satisfaction level.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative and on-site points of contact obtain feedback from employees on a regular basis.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The customer satisfaction survey is one method used to rate the provider. In addition, according to MHS program officials, stakeholder input is also obtained from program management reviews and meetings held by the government task manager.; Did the project use the practice?: Navy NMCI: Yes--User surveys are used to measure satisfaction with specific services such as E-mail access, help desk, etc., and overall user satisfaction with the service provider's performance.; Did the project use the practice?: NIMA IT/IS: Yes--The provider surveys NIMA employees on their satisfaction with the help- desk function. The results of these surveys are reviewed by NIMA during quarterly meetings. In addition, stakeholders are also responsible for providing monthly assessments of contractor performance. Moreover, customer satisfaction is a factor in determining the amount of the semiannual award fee earned by the provider. Source: GAO. [End of table]: [End of section] Appendix VIII: Projects' Implementation of Phase VII: Ensure Services are Provided: [See PDF for image] [End of figure] Although outsourcing focuses on the provider's ability to perform, the client organization is ultimately responsible for ensuring that services are provided and that end-user needs are met. The previous phases addressed the extensive preparation that must precede the provider's assuming responsibility for the client organization's services. This phase encompasses practices designed to ensure that an effective oversight approach is established. In addition, over the course of the outsourcing project, the client learns more about the capabilities of the provider, and market conditions may change. As a result, it is important to monitor service levels internally as well as maintain an external view of the performance of other providers in order to identify opportunities to improve and ensure that the outsourcing arrangement maintains its value to the client. The six practices in this phase that we used to evaluate the five projects are as follows: * Monitor the provider's work to anticipate issues for resolution. * Make sure that the provider uses the standard tools and processes defined as part of the operational model. * Use provider performance data to continuously improve processes. * Pursue improvement based on customer satisfaction surveys. * Ensure that an appropriately empowered individual from the client organization oversees the work. * Set realistic time frames that are agreed to by the provider. As illustrated by figure 12, the five projects in our review largely implemented the practices. Figure 12: Project Implementation of Phase VII: Ensure Services Are Provided: [See PDF for image] Note: Not applicable--The practice was not relevant to the project's particular circumstances. No--The agency did not implement the practice. Limited--The agency fully implemented some but not all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes--The agency fully implemented the practice or took an alternative action that fully satisfied its intent. [End of figure] Table 10 provides details on whether and how each project implemented each of the six practices associated with this phase. Table 10: Summary of Projects' Use of Phase VII Practices: Practice: Monitor the provider's work to anticipate issues for resolution.; Did the project use the practice?: Air Force C4 Services: Yes--Quality assurance evaluators monitor the quality of the provider's work and identify problems or trends. The results and any problems are reported to the contracting officer and the functional area chief.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative reports monthly on contractor performance. In addition, he meets daily with provider officials to discuss performance results.; Did the project use the practice?: MHS/ITO Help Desk: Yes-- The provider's work is monitored through monthly reports and meetings as well as through on-site meetings and readiness assessments.; Did the project use the practice?: Navy NMCI: Yes--Navy representatives at each implemented site (including headquarters) monitor the provider's work and identify issues. According to NMCI program officials, the NMCI Enterprise Management System enables the government to monitor the performance of the network and associated service delivery.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA reviews the provider's work in quarterly program management reviews and as needed in response to issues raised by NIMA's performance monitoring officials. Practice: Make sure the provider uses the standard tools and processes defined as part of the operational model.; Did the project use the practice?: Air Force C4 Services: Yes--The provider uses software tools to help monitor system performance against performance standards to monitor and manage the help-desk function.; Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-SWA project officials, the provider uses various standard tools outlined in its proposal.; Did the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, the provider uses the standard tools outlined in the contract.; Did the project use the practice?: Navy NMCI: Yes--The Navy has monitored the service provider's use of standard tools and processes. Tools are being used to support legacy migration, client installation, and help-desk procedures.; Did the project use the practice?: NIMA IT/IS: Yes--The tools are documented in the quality management plan. For example, the provider is using a specific quality and process improvement methodology. Practice: Use provider performance data to continuously improve processes.; Did the project use the practice?: Air Force C4 Services: Yes--Although Kirtland Air Force Base does not rely on data provided by its provider, its quality assurance evaluators continually monitor provider performance.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative monitors provider performance data, and any potential areas for improvement are incorporated in the monthly report.; Did the project use the practice?: MHS/ITO Help Desk: Yes--For example, MHS used provider performance data to improve the customer satisfaction survey process.; Did the project use the practice?: Navy NMCI: Yes--The Navy uses the provider's data to help determine ways to improve processes, such as to improve NMCI implementation procedures and the timeliness of help-desk problem resolution.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA's performance monitors use provider data to assess progress and ways to improve performance. Practice: Pursue improvement based on customer satisfaction surveys.; Did the project use the practice?: Air Force C4 Services: Limited--The Air Force does not perform, or require the provider to perform, customer surveys. However, the provider has initiated such surveys but is not required to distribute the results to the Air Force.; Did the project use the practice?: Army TAC-SWA: Yes--Surveys are received by the contracting officer's technical representatives and provided as input for the contracting officer's representative report.; Did the project use the practice?: MHS/ITO Help Desk: Yes--Negative feedback that is received based on the results of the customer satisfaction surveys is reviewed by the MHS help-desk manager.; Did the project use the practice?: Navy NMCI: Yes--Surveys of user satisfaction with specific services and overall satisfaction with provider performance are used to identify areas needing improvement.; Did the project use the practice?: NIMA IT/IS: Yes--To pursue improvement, NIMA uses both customer satisfaction surveys and interviews with senior-level customers. Practice: Ensure that an appropriately empowered individual from the client organization oversees the work.; Did the project use the practice?: Air Force C4 Services: Yes--The functional area chief oversees the work, and the quality assurance evaluators support this oversight effort.; Did the project use the practice?: Army TAC-SWA: Yes--The contracting officer's representative is the empowered individual from NETCOM.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The government task manager is the individual empowered to oversee the work of the provider.; Did the project use the practice?: Navy NMCI: Yes--The NMCI Director oversees the work and discusses performance with the NMCI provider executive during weekly meetings.; Did the project use the practice?: NIMA IT/IS: Yes--The contracting officer's representative and a staff of technical monitors oversee the work. Practice: Set realistic time frames that are agreed to by the provider.; Did the project use the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base and the provider agree that reasonable time frames for performance have now been set. Some revisions were made to them in April 2002.; Did the project use the practice?: Army TAC- SWA: Yes--Time frames for new tasks are established by contract modifications that are agreed to by the provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes--The monthly performance reports and quarterly program reviews set time frames that have been agreed to by the provider.; Did the project use the practice?: Navy NMCI: Yes-- According to NMCI program officials, Navy's practice is to work with the provider in establishing schedules.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA and the provider have agreed on dates associated with specific actions. Source: GAO: [End of table] [End of section] Appendix IX: Comments from the Department of Defense: DEPARTMENT OF DEFENSE: 6000 DEFENSE PENTAGON WASHINGTON, DC 20301-6000: April 8, 2003: CHIEF INFORMATION OFFICER: Mr. Randolph C. Hite Director: Information Technology Architecture and Systems Issues United States General Accounting Office: Washington, DC 20548: Dear Mr. Hite: This is the Department of Defense (DoD) response to GAO draft Report No. GAO-03-371, "INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons Learned from its Outsourcing Projects," dated March 11, 2002 (GAO Code 310239). The Department partially concurs with the report. The DoD agrees that capturing lessons learned in the development and implementation of its IT outsourcing initiatives is important to continually improving the methods used and the results achieved. However, specifying a particular method of accession is premature at this time. The DoD currently has several processes and communities of interest that collect and disseminate lessons learned in other areas; one noteworthy example is "Share A-76." The processes used by these groups would be a logical starting point to determine the best path forward. Before the Department commits to a specific means of provision, we intend to explore a variety of mechanisms by which we can exploit lessons learned in IT outsourcing initiatives. One minor correction to the matrix on page 57 is requested. The National Imagery and Mapping Agency (NIMA) has authorized its contractor to procure supplies, but not products or services under the Federal Supply Schedule. Therefore, under the NIMA IT/IS column, in the third box (at the bottom), please change the sentence from: "Yes-NIMA has authorized the provider to use government sources, such as the Federal Supply Schedule, in procuring products, services, and supplies related to this contract.": to: "Yes-NIMA has authorized the provider to use government sources, such as the Federal Supply Schedule, in procuring supplies related to this contract.": If you require further information, please contact the action officer, Leo Milanowski, at (703) 602-2720 x142 or leo.milanowskiAosd.mil. Sincerely, Margaret E. Myers, Principal Director Deputy Assistant Secretary of Defense (Deputy CIO): Signed by Margaret E. Myers: Attachment: Response to Recommendations: GAO DRAFT REPORT DATED MARCH 11, 2003 GAO-03-371 (GAO CODE 310239): "INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons Learned from Its Outsourcing Projects: DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATIONS: RECOMMENDATION 1: The GAO recommended that the Undersecretary of Defense for Acquisition, Technology, and Logistics, working in conjunction with the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, provide senior management support and adequate resources to develop and implement an electronic tool to capture and disseminate examples and lessons learned from actual IT outsourcing projects. DOD RESPONSE: Partially concur; DoD agrees that capturing lessons learned in the development and implementation of its IT outsourcing initiatives is important to continually improving the methods used and the results achieved. However, specifying a particular method of accession is premature at this time. DoD currently has several processes and communities of interest that collect and disseminate lessons learned. Before the Department commits to a specific means of provision, we intend to explore a variety of mechanisms by which we can exploit lessons learned in IT outsourcing initiatives. RECOMMENDATION 2: The GAO recommended that the DoD ensure that the method used to gather information for this electronic tool incorporate the main element of a lessons learned process, namely, collection, verification, storage, and dissemination. DOD RESPONSE: Partially concur; see above: The following are GAO's comments on the Department of Defense's letter dated April 8, 2003. GAO Comments: 1. Addressed in the Agency Comments and Our Evaluation section of this report. 2. On January 29, 2003, NIMA granted the provider authorization to use government sources in performing the IT/IS contract that was limited to "products, services, and supplies that support the performance of the specific functional areas and miscellaneous items required under this contract." Accordingly, we did not modify this report. 3. Addressed in the Agency Comments and Our Evaluation section of this report. [End of section] Appendix X: GAO Contact and Staff Acknowledgments: GAO Contact: Linda J. Lambert, (202) 512-9556: : Staff Acknowledgments: Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and Thomas Wright made key contributions to this report. (310239): FOOTNOTES [1] This figure is from the Federal Procurement Data System, which contains detailed information on contract actions over $25,000. [2] IT services outsourcing is a type of acquisition in which a client organization transfers responsibility for performance of one or more IT services to one or more external providers. [3] U.S. General Accounting Office, Information Technology: Leading Commercial Practices for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001). [4] We asked the military services and other DOD components to identify candidate projects because DOD does not maintain a central list of IT services outsourcing projects. We chose each project on the basis of the following criteria: (1) no more than one project from each military service and two agencies, (2) illustrative example of DOD IT outsourcing, (3) dollar value greater than $10 million, and (4) enough time elapsed for services to have been delivered and performance measured. [5] We identified 70 practices in our November 2001 report on leading commercial practices that (1) are typically applied at the project level and (2) were verifiable through documentation and interviews. [6] This calculation does not include practices that were not applicable to a particular project. [7] This consensus view was based on interviews with managers in leading commercial organizations, discussions with academic and professional authorities, and extensive research on IT acquisition practices. [8] This figure is from the Federal Procurement Data System, which contains detailed information on contract actions over $25,000. [9] GAO-02-214. [10] Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 19, 2002). We did not independently verify these data. [11] INPUT, The Federal IT Outsourcing Market View (December 2002). We did not independently verify these data. [12] GAO-02-214. [13] This consensus view was based on interviews with managers in leading commercial organizations, discussions with academic and professional authorities, and extensive research on IT acquisition practices. [14] Office of Management and Budget, Circular A-76, Performance of Commercial Activities (Washington, D.C.: June 14, 1999). In November 2002, OMB issued proposed revisions that would substantially change this circular. As of April 8, 2003, these revisions have not yet been made final. [15] Final report of the Commercial Activities Panel, Improving the Sourcing Decisions of the Government (Washington, D.C.: Apr. 30, 2002). [16] Section 5 of P.L. 105-270, codified at 31 U.S.C. 501 note (1998), defines an inherently governmental function as a "function that is so intimately related to the public interest as to require performance by Federal Government employees." [17] U.S. General Accounting Office, Proposed Revisions to OMB Circular A-76, GAO-03-391R (Washington, D.C.: Jan. 16, 2003). [18] U.S. General Accounting Office, Desktop Outsourcing: Positive Results Reported, but Analyses Could Be Strengthened, GAO-02-329 (Washington, D.C.: Mar. 29, 2002). [19] The Industrial College of the Armed Forces, Case Study: Complex Business Management for Competitive Sourcing (2001). [20] PricewaterhouseCoopers Endowment for the Business of Government, Contracting for the 21ST Century: A Partnership Model (January 2002). [21] GAO-02-329. [22] U.S. General Accounting Office, Contract Management: Taking a Strategic Approach to Improving Service Acquisitions, GAO-02-499T (Washington, D.C.: Mar. 7, 2002). [23] This calculation does not include practices that were not applicable to a particular project. [24] Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key Stages and Phases of the IT Sourcing Process (Jan. 31, 2003). [25] This calculation does not include practices that were not applicable to a particular project. [26] Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing Strategy, Research Note R-18-1099 (Sept. 23, 2002). [27] The Department of the Interior's GovWorks Program is a federal fee-for-service acquisition center that helps other federal agencies acquire supplies and services for their programs on a project-by- project basis. [28] Gartner, Inc., defines benchmarking as a method to compare the cost or price of an IT environment to peer groups with the same workload characteristics. [29] Giga Information Group, Inc., Payment and Incentives for Outsourcing Management (July 27, 2000) and Gartner, Inc., Benchmarking Helps Outsourcing Deals Stay Competitive, Research Note COM-16-8055 (June 14, 2002). [30] This calculation does not include practices that were not applicable to a particular project. [31] This calculation does not include practices that were not applicable to a particular project. [32] Seat management generally refers to service provision arrangements in which contractor-owned desktop and other computing hardware, software, and related services are bundled and provided to a client organization at a fixed price per unit (or seat). [33] This calculation does not include practices that were not applicable to a particular project. [34] Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note COM-16-8425 (June 17, 2002). [35] An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based Services Acquisition, Benchmark Version (January 2002). [36] U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). [37] U.S. General Accounting Office, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO/AIMD-10-1.23, Exposure Draft (Washington, D.C.: May 2000). [38] U.S. General Accounting Office, DOD Competitive Sourcing: Lessons Learned System Could Enhance A-76 Study Process, GAO/NSIAD-99-152 (Washington, D.C.: July 21, 1999). [39] GAO-02-329. [40] GAO-02-195. [41] GAO-02-195. [42] U.S. General Accounting Office, Information Technology: Leading Commercial Practices for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001). [43] GAO-02-214. [44] U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). [45] Gartner, Inc., Five Tough Questions About Skill Sourcing, Research Note SPA-13-2537 (Mar. 28, 2001). [46] SLAs define the agency's expectations and are used to track and measure a contractor's performance. [47] An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based Services Acquisition, Benchmark Version (January 2002). [48] Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note COM-16-8425 (June 17, 2002). [49] Gartner, Inc., Successful Outsourcing Means Retaining Some Staff, Research Note COM-18-9692 (Dec. 18, 2002). [50] An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based Services Acquisition, Benchmark Version (January 2002). GAO's Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to daily E-mail alert for newly released products" under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: