This is the accessible text file for GAO report number GAO-02-1020 entitled 'Space Transportation: Challenges Facing NASA's Space Launch Initiative' which was released on October 02, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to the Subcommittee on Space and Aeronautics, Committee on Science, House of Representatives: United States General Accounting Office: GAO: September 2002: Space Transportation: Challenges Facing NASA’s Space Launch Initiative: GAO-02-1020: Contents: Letter: Results in Brief: Background: Important Decisions to Be Made Before Requirements Can Be Defined: Key Management Controls Are Not Yet Implemented: Conclusions: Recommendations: Agency Comments: Scope and Methodology: Appendix I: Comments from the National Aeronautics and Space Administration: Appendix II: GAO Contact and Staff Acknowledgments: Table: Table 1: Activities Related to the Space Launch Initiative: Figures: Figure 1: NASA Illustration of the 2nd Generation Reusable Launch Vehicle: Figure 2: Illustration of NASA’s Overall Space Transportation Plans: Abbreviations: DOD: Department of Defense EVM: Earned Value Management NASA: National Aeronautics and Space Administration RFP: Request for Proposal SLI: Space Launch Initiative TRL: technology readiness level: United States General Accounting Office: Washington, DC 20548: September 17, 2002: The Honorable Dana Rohrabacher Chairman The Honorable Bart Gordon Ranking Member Subcommittee on Space and Aeronautics Committee on Science House of Representatives: In 2001, the National Aeronautics and Space Administration (NASA) began undertaking a new effort--the Space Launch Initiative (SLI)--to develop a new generation of space transportation vehicles. SLI is expected to result in the development of the second generation of reusable launch vehicles, the space shuttle being the first generation. NASA plans to spend $4.8 billion on the program through fiscal year 2006. SLI is part of a broader program--known as NASA’s Integrated Space Transportation Plan- -to address future space transportation needs. Under the plan, NASA envisions making upgrades to extend the life of the space shuttle and undertaking longer-term research and development of future transportation technologies and transportation systems, including space vehicles that can reach orbit in just one-stage. SLI is a highly ambitious program. It will require NASA to develop and advance new technologies, such as propulsion and airframe systems, which in turn can potentially be used by U.S. industry to create new business opportunities in space. The undertaking will also require a high level of communication and coordination between a range of partners, including private-sector contractors, academia, and the Department of Defense (DOD). Moreover, it will require effective controls and oversight to reduce cost, scheduling, and technical risks. NASA’s previous attempts to develop a new generation of space vehicles were unsuccessful largely because NASA did not successfully implement and adhere to critical project management controls and activities. You requested that we assess NASA’s progress with the Space Launch Initiative, particularly with respect to defining requirements and implementing management controls. Results in Brief: NASA plans to define basic requirements for its second-generation reusable launch vehicle--that is, what the crew size will be, what the payload capacity will be, and what designs or architectures are worth pursuing--by November 2002. But considerable challenges must be addressed before NASA can accomplish this. First, NASA has to complete a reassessment of its overall space transportation plans. In doing so, it must decide whether it should continue pursuing the development of second-generation vehicles as planned, pursue alternative ways to develop the second generation in order to more quickly replace the space shuttle, or postpone these efforts altogether indefinitely until there is a major breakthrough in technology that could vastly improve performance and reduce costs. This decision will be difficult, given the uncertainties about the availability of technologies needed to reduce costs and enhance performance for future space flight. Second, NASA is currently reassessing the future of the International Space Station. The decisions it will make as part of this evaluation, such as how many crew will operate the station, will have a dramatic impact on NASA’s requirements for a second-generation vehicle. But they will be difficult to reach, since they require NASA to come to agreement with international partners who are concerned about planned cutbacks to the station’s capabilities. Third, NASA needs to decide whether the SLI program will be developed jointly with DOD and, if so, how it can accommodate DOD’s requirements for a reusable launch vehicle. So far, indications are that NASA and DOD will share many of the same objectives for the vehicle, but there are significant differences in priorities and requirements. Until NASA finalizes its basic requirements for SLI, it cannot implement management controls that are essential to predicting what the total costs of the program will be and to minimizing risks with NASA’s planned initial investment of $4.8 billion. These include cost estimates, controls designed to provide early warnings of cost and schedule overruns, and risk mitigation plans. Moreover, there are potential impediments to NASA’s development and effective use of a detailed cost estimate, including the lack of a modern integrated financial management system. Lastly, NASA does not plan to develop several measures that are important to assessing how the program as a whole is making progress toward achieving its key objectives, reducing risks, and maturing technology. It is important for NASA to implement management controls for SLI as soon as possible, so that it can provide its managers and the Congress with the information needed to ensure that the program is on track and able to meet expectations. We are making recommendations to NASA that focus on the need to make decisions with regard to the future of NASA’s overall space transportation plan, the future of the space station, and DOD’s participation in the SLI program before setting requirements for SLI. We are also making recommendations aimed at implementing management controls for the SLI program. In its comments to a draft of this report, NASA stated that it concurs with the recommendations. NASA believes that much of the SLI program’s success is directly related to the implementation of project management controls and appropriate levels of insight. NASA’s response is included as appendix 1. Background: NASA’s Space Launch Initiative is an effort to develop and build a second generation of reusable space transportation vehicles. (See fig. 1 and table 1.) NASA’s current transportation vehicle, the space shuttle, has been in use for 20 years and requires a significant portion of NASA’s resources to operate and maintain. The primary goals for SLI are to reduce the risk of crew loss as well as substantially lower the cost of space transportation so that more funds can be made available for scientific research, technology development, and exploration activities. Currently, NASA spends nearly one-third of its budget on space transportation. Figure 1: Illustration of the 2nd Generation Reusable Launch Vehicle: [See PDF for image] Source: NASA. SLI is part of a broader program--known as NASA’s Integrated Space Transportation Plan--to address space transportation needs. Under the plan, NASA could operate the space shuttle through 2020 and make software and hardware upgrades to the shuttle in order to extend its use to this point. It envisions the deployment of second-generation cargo vehicles to begin around 2011 and crew vehicles around 2014. As with the shuttle, NASA envisions that the second-generation vehicle will reach orbit in two stages. NASA also anticipates building a third generation of vehicles in 2025 and even a fourth generation in 2040. (See fig. 2.) NASA anticipates that these vehicles would reach orbit in one stage; travel beyond low-earth orbit to far-reaching interstellar missions; and employ revolutionary technologies, such as (1) rocket engines that breath oxygen from the air during the climb to orbit rather than carrying heavy oxidizers onboard, (2) propellantless electromagnetic propulsion, and (3) solar-powered space sails. Figure 2: Illustration of NASA’s Overall Space Transportation Plans: [See PDF for image] Source: NASA. Building the second-generation vehicle will be a considerably complex and challenging endeavor for NASA--from both a technical and business standpoint. For example, NASA plans to develop and advance new technologies for the new vehicle. These include (1) new airframe technologies that will include robust, low-cost, low-maintenance structure, tanks, and thermal protection systems, using advanced ceramic and metallic composite materials, and (2) new propulsion technologies, including main propulsion systems, orbital maneuvering systems, main engines, and propellant management. If successfully developed, NASA believes that these new technologies could substantially reduce the costs to maintain and operate the vehicle and increase its reliability. NASA also plans to develop the new vehicle through partnerships with private industry, academia, and DOD. It is partnering with the private sector so that it can help create business opportunities in space, and it is partnering with DOD because of military needs for a reusable launch vehicle. Such partnerships will require a high level of coordination and communication, since agreements need to be reached on what the basic capabilities of the new vehicle will be, what designs or architectures[Footnote 1] should be pursued, how development costs will be shared, and what individual partner responsibilities will be. NASA recognizes that there are substantial technical and business risks with SLI, and it is undertaking activities aimed at reducing them. To reduce technical risks, for example, NASA is exploring, developing, and testing technologies to make sure that they can be used on the new vehicle, rather than proceeding with the program with uncertainty about whether technologies will be mature enough when they are needed. To reduce business and costs risks, NASA’s goal is to ensure that there is adequate competition in the development of the architecture and that the architectures that NASA pursues will enable convergences with NASA’s requirements and commercial and military missions, and will not pursue vehicles that cannot meet program goals. To date, NASA has already explored hundreds of concepts and technologies that could be used for future government and commercial launch systems and space transportation operations. The “leap-ahead” technologies studied include crew survival systems, advanced tanks and airframe structures, long-life rocket engines, and thermal protection systems. In March 2002, NASA selected 15 of the most promising candidates to go forward into more detailed development. In November 2002, NASA plans to narrow the field down to 3 candidates and in September 2003 to at least 2 candidates. From fiscal year 2003 through fiscal 2006, two competing system architectures will be developed, ready for a full-scale development decision in fiscal 2006. NASA plans to spend $4.8 billion for the SLI program through fiscal year 2006. This investment is to be used for (1) systems engineering and activities concerning the definition of requirements; (2) efforts to compete designs for the reusable launch vehicle as well as efforts to reduce business and technical risks; (3) activities to develop and demonstrate designs, technologies, and system-level integration issues associated with such NASA-unique transportation elements as a crew transport vehicle and cargo carriers; and (4) activities to develop alternative access to the space station as well as to provide contingency backup or relief for the space shuttle. NASA’s efforts to compete designs for the vehicle and reduce technical and business risks consist of approximately $3 billion of the planned initial investment. At the present time, no prototype vehicle integrating all new SLI technologies is planned before NASA’s fiscal year 2006 decision on whether to proceed to full-scale development. Table 1: Activities Related to the Space Launch Initiative: Year: 2001; Month: Feb.; Event: * SLI program approved; Narrative: NASA plans to budget $4.8 billion for the program through fiscal year 2006.. Year: 2001; Month: May; Event: * Initial contracts awarded to 22 contractors; Narrative: The contracts, valued at almost $800 million, were not intended to provide a specific vehicle design, but rather to explore concepts and technologies that could be used for future government and commercial launch systems and space transportation operations. The leap-ahead technologies explored included crew survival systems, advanced tanks and airframe structures, long-life rocket engines, and thermal protection systems.. Year: 2002; Month: Mar.; Event: * Interim Architecture and Technology Review; Narrative: Design concepts for the second-generation vehicle narrowed down from hundreds to 15.. Year: 2002; Month: Nov.; Event: * Systems Requirements Review; Narrative: This review is to focus attention on fewer space transportation architectures and technology areas, to select three architectures that can be pursued, and reach agreement on the development of system requirements.. Year: 2003; Month: Feb.; Event: * Request for Proposals (RFP) to select SLI designs; Narrative: RFPs are to focus on selecting the most promising architectures to proceed toward a detailed preliminary design.. Year: 2003; Month: Sept.; Event: * Contract awards for selection of two designs; Narrative: Second phase of formulation program to further develop concepts and risk-reduction activities.. Year: 2006; Month: [Empty]; Event: * SLI full-scale development decision; Narrative: Decision for selecting architecture to enter full- scale development.. Year: 2009; Month: [Empty]; Event: * Potential prototype vehicle available; Narrative: Reusable unmanned vehicle with limited capabilities.. Year: 2012; Month: [Empty]; Event: * Cargo vehicle available; Narrative: Reusable unmanned vehicle with advanced engines and tanks.. Year: 2014; Month: [Empty]; Event: * Crew vehicle available; Narrative: Reusable launch vehicle with crew capabilities. [End of table] Source: Discussions with and documentation from NASA. Important Decisions to Be Made Before Requirements Can Be Defined: According to a NASA official, NASA plans to define the basic requirements for its second-generation space transportation--that is, what the crew size will be, what the payload capacity will be, and what designs or architectures are worth pursuing--by November 2002. These decisions will have a significant impact on the cost, size, and design of the new vehicle. For example, as the payload capacity increases, so does the thrust requirement for the propulsion system, and in turn, the cost to develop and build the system, as well as the cost to operate the system. However, NASA is facing a considerable challenge in reaching the point to where it can finalize SLI requirements. This is primarily because NASA must first make some difficult broader decisions regarding the future of space transportation and other NASA projects, including (1) whether NASA should skip development of a second-generation vehicle in favor of concentrating on the third generation, (2) what the future of the International Space Station will be, and (3) what DOD’s role in developing the new vehicle will be. Moreover, in making these decisions, NASA will need to reach consensus with a wide range of partners who have priorities and concerns different from NASA’s, and it will need to make trade-offs amid uncertainties as to the availability of advanced technologies. First, NASA must complete its ongoing reassessment of its overall space transportation plan. This evaluation is being done as part of NASA’s development of a budget proposal for fiscal year 2004. The options NASA is considering could have a far-reaching impact on the SLI program. One option, in fact, involves postponing efforts to develop a new generation of vehicles indefinitely until there is a major breakthrough in technology that could vastly improve performance and reduce costs. Other options being examined could drastically change NASA’s timetable and requirements for SLI. For example, NASA is looking at developing a crewed vehicle more quickly than currently planned and launching it atop an expendable rocket until it can field a reusable launch vehicle. NASA is also considering deferring some development efforts so that it can aggressively pursue a goal of building a relaunchable vehicle prototype by 2009. Finalizing its decisions on where to go with the space transportation plan will be difficult for NASA, given the uncertainties about the availability of technologies needed to reduce costs and achieve NASA’s performance and safety objectives. Second, NASA is currently reassessing the future of the International Space Station. One decision it needs to make that could significantly affect the SLI program is whether the station should support a crew of seven astronauts, as originally planned, or three. Because of cost growth, NASA plans to cut back to a crew of three, but its international partners have not agreed yet to this decision and are concerned that the cutback will severely undermine planned scientific research. This decision could significantly affect the design and cost of the second-generation vehicle. For example, with a bigger crew size, the overall vehicle will have to be larger; a larger crew cabin and additional backup systems will be required; and, as a result, the cost to develop the vehicle will increase. NASA is also planning to cut back the number of flights to the station from seven to four per year, which could also have an impact on SLI. But again, NASA’s international partners have concerns about this decision, since it would limit the deliveries of resources needed to carry out research activities. At the same time that NASA is looking at cutting back on space station capabilities, it is planning to extend the life of its space shuttle to 2020 with software and hardware upgrades. If NASA can successfully extend the life of the shuttle, it may well find that a second-generation vehicle, which is not expected to begin transporting crews to the space station until at least 2014, would generally be duplicative in capability and therefore unnecessary. NASA plans to make final decisions on the space station by November/ December 2002. Third, while DOD and NASA have explored potential common areas of interest for a new space transportation vehicle, DOD has not yet formally defined its requirements. In fact, it is still uncertain whether SLI will be a joint DOD/NASA program. A study conducted by the Air Force and NASA earlier this year revealed that both NASA and DOD shared similar objectives when it came to technologies needed for the new vehicle as well as cargo lift requirements and launch architecture elements. But there were differences with priorities and certain requirements. The Air Force would like the vehicle to operate in stronger winds, more precipitation, and a wider range of temperatures. It would also like the vehicle to operate from an inland U.S. Air Force base. NASA would like the vehicle to stay in orbit for a longer duration and have more maneuverability because of its mission to service the International Space Station. Additionally, NASA will have a higher weight-delivered-to-orbit requirement for its crew vehicle. It is apparent that some of DOD’s objectives and priorities may not match up with NASA’s. For example, DOD envisions developing an unpiloted vehicle, while NASA is focusing on a crewed design. DOD also envisions developing a vehicle that could be capable of relaunching within 12 to 48 hours and making as many as 20 flights in a 2-week time frame. At this time, NASA does not share these objectives. According to a DOD official, DOD expects to finalize its requirements before the end of 2002. Since DOD’s decisions will affect the size, design, and capabilities of the vehicle, it is important that these requirements be known before NASA finalizes its own requirements in November 2002. Otherwise, NASA will need to reexamine its requirements after narrowing down potential system architectures to three possibilities. While significant cost and operational benefits may accrue with a vehicle capable of satisfying both DOD’s and NASA’s requirements, NASA will need to guard against making compromises that might negatively affect its goal of substantially lowering launch costs. Key Management Controls Are Not Yet Implemented: NASA cannot implement key management controls for the SLI program until it defines its basic requirements. These include cost estimates, controls designed to provide early warnings of cost and schedule problems, as well as risk mitigation plans. Moreover, NASA does not have plans to implement some performance measures, including ones that would assess overall progress toward achieving key objectives, reducing risks, and maturing technology. It is essential that these controls be implemented quickly so that NASA can predict what the total costs of the program will be and provide assurance that its investment in SLI is being spent wisely. Importance of Management Controls for SLI: Undertaking ambitious, technically challenging efforts like SLI--which involve multiple contractors and technologies that have to be developed and proven--requires careful oversight and management. Importantly, accurate and reliable cost estimates need to be developed, technical and program risks need to be anticipated and mitigated, and performance and readiness need to be closely monitored. Such undertakings also require a high level of communication and coordination. Not carefully implementing such project management tools and activities is a recipe for failure. Without realistically estimating costs and risks, and providing budgetary reserves needed to mitigate those risks, management may not be in a position to effectively deal with the technical problems that cutting-edge projects invariably face. In fact, we found that NASA did not successfully implement and adhere to a number of critical project management tools and activities in its previous efforts to build new space transportation vehicles. As we testified in June 2001, neither of NASA’s X-33 or X-34 programs--which attempted to build a new transportation vehicle that would reach orbit in one stage---fully assessed the costs associated with developing new, unproven technologies; provided for the financial reserves needed to deal with technical risks and accommodate normal development delays; developed plans to quantify and mitigate risks to NASA; or established performance targets showing a clear path to an operational launch vehicle.[Footnote 2] Underlying these difficulties were problems with the agreements and contracts that established the relationship between NASA and its industry partners and the eventual erosion of commercial prospects for the development of new reusable launch vehicles. We testified that lax management controls led to numerous problems with both the X-33 and X-34 programs. Technical problems, for example, resulted in significant schedule and cost overruns, which NASA was ill prepared to deal with. NASA has taken steps to avoid the problems it encountered with the X-33 and X-34 programs. In our testimony last year, for example, we pointed out that NASA planned to increase the level of insight into SLI projects by providing more formal reviews and varying levels of project documentation from contractors depending on the risk involved and the contract value. NASA also required that all proposals submitted in response to its research announcement be accompanied by certifiable cost and pricing data. Finally, NASA discouraged the use of cooperative agreements, since these agreements did not prove to be effective contracting mechanisms for research and development efforts where large investments are required. Cost Estimates Cannot Be Developed Until Requirements Are Defined: NASA cannot develop cost estimates for the SLI program until it defines the basic requirements for the new transportation vehicle and narrows the field of possible architectures. Moreover, until requirements are defined, NASA cannot determine whether the $4.8 billion already planned for SLI through fiscal year 2006 will achieve the desired results for the program’s formulation. NASA guidance requires that life-cycle costs be estimated, assessed, and controlled throughout a program’s life cycle.[Footnote 3] The estimates are to be prepared to support major program reviews and the development of budget submissions. In our recent review of the International Space Station, we reported that NASA is facing additional challenges to developing reliable cost estimates and effectively using them to manage programs.[Footnote 4] For example, a recent study performed by the Rand Corporation for the Office of Science and Technology Policy found that NASA has “very good” cost and risk modeling capabilities. However, the study also found that NASA’s in-house capabilities were not well integrated into the program’s planning and management and that NASA programs had been reluctant to integrate cost estimate and control practices that were sufficiently robust to yield confidence in budget estimates. In addition, a task force appointed last year to conduct an independent external review and assessment of the space station’s cost, budget, and management found that NASA tends to take a short-term focus on executing programs--that is, rather than managing programs to come within overall cost and scheduling goals, it manages them around annual budgets. The task force cited NASA’s culture of managing the space station program in adherence to its annual budgets as perhaps the single greatest factor in the space station program’s cost growth. We also reported that NASA’s ability to develop good cost estimates for programs is hampered by NASA’s lack of a modern integrated financial system to track and maintain data needed for estimating and controlling costs. NASA has made it a top priority to develop and implement a new system. However, the first major component of the system--the core financial system--is not expected to be implemented until June 2003. It will be important for NASA to overcome barriers that relate to estimating costs for SLI and to use estimates to take a long-term perspective in managing the SLI program. Without good cost information, decision makers at NASA and in the Congress will not know whether the $4.8 billion investment is sufficient for the early phases of the program or how much more it will cost to actually develop and deploy the new vehicle. Moreover, NASA managers will lack the information they need to monitor costs, schedule, and performance. Other Management Controls Cannot Be Implemented Without a Cost Estimate: Other management controls that are integral to successfully managing the SLI program cannot be implemented until NASA has a cost estimate. First, NASA has a system--known as Earned Value Management--intended to help provide program managers and others with early warnings of cost and schedule problems. But this system cannot be effectively implemented without having baseline requirements defined or detailed cost estimates, schedules, and timelines developed for the SLI program. Earned Value Management goes beyond the two-dimensional approach of comparing budgeted costs with actual costs. It also attempts to compare the value of work accomplished during a given period with the value of work scheduled for that period. By using the value of work done as a basis for estimating the cost and time to complete it, the earned value concept should alert program managers to potential problems sooner than expenditures alone can. The communities that have a vested interest in earned value are the (1) program managers, who are charged with overall management responsibility for acquisition programs; (2) contractors, who are responsible for the contract’s successful execution; and (3) overseers, such as acquisition executives, financial managers, contract surveillance officials, and cost estimators, who are tasked with tracking and estimating program costs. Second, individual SLI activities have prepared risk mitigation plans. However, a program official told us that these plans cannot be quantified at the overall program cost and schedule level without complete cost estimates. Moreover, without a cost estimate, risks can be measured only on a year-by-year basis and not on a multiyear basis- -for example, to 2006. Risk mitigation plans identify, assess, and document the risks associated with the cost, resource, schedule, and technical aspects of a project and determine the procedures that will be used to manage those risks. In doing so, they help ensure that a system will meet performance requirements and be delivered on schedule and within budget. NASA’s guidance requires that these plans be developed during the formulation phase of a project. Measures to Assess Performance Not Implemented: Agencies are required to prepare annual performance plans that establish performance goals with measurable target levels of performance for each program activity in the agency’s budget and to provide a basis for comparing actual performance with performance goals.[Footnote 5] Doing so enables agencies to gauge the progress of programs like SLI and, in turn, to take quick action when performance goals are not being met. Individual SLI activities are using computer simulations to help gauge whether the technologies under development can meet specific performance targets. However, NASA has not yet established broader measures that can show NASA decision makers and the Congress whether the program as a whole is meeting objectives such as reducing the payload cost to approximately $1,000 per pound and reducing the risk of crew loss to approximately 1 in 10,000 missions. Like cost and risk controls, the development of such metrics depends on NASA’s definition of requirements and cost estimation for SLI. It is important that NASA develop such measures as soon as possible. We testified last year that one problem that hampered the previous X-33 and X-34 efforts was the fact that NASA had not developed performance targets that establish a clear path leading to a reusable launch vehicle. In addition, while NASA plans to assess the readiness of technology to gauge the maturity of individual technologies, it does not plan to verify and validate the maturity of technology for the SLI program as a whole. To ensure that individual technologies are sufficiently mature by NASA’s planned full-scale development decision for 2006, NASA intends to use technology readiness levels (TRLs). Our prior reports have shown that TRLs, which were pioneered by NASA, are a good way to gauge the maturity of technologies. Readiness levels are measured along a scale of 1 to 9, starting with paper studies of the basic concept, proceeding with laboratory demonstrations, and ending with a technology that has proven itself on the intended product. NASA would like to achieve a TRL of 6 by 2006 for key technologies, such as the propulsion system. At this level, a prototype is tested in a relevant environment, such as a high- fidelity laboratory environment or in a simulated operational environment. Currently, most of the technology areas are at levels 3 or 4. At level 3, analytical studies and laboratory studies are performed to physically validate analytical predictions of separate elements of the technology. At level 4, basic technological components are integrated to establish that the pieces will work together. While assessing technologies separately should help NASA decide when and where to insert new technologies into the SLI program, it is still important for NASA to look at the readiness of the product as a whole because how well various components being developed will work together is unknown. Another measure that NASA officials told us they do not plan to implement is one that would assess the extent or percentage of total risks that have been reduced with NASA’s initial $4.8 billion investment as well as the amount of risk remaining. Program officials told us that it would be too resource-intensive to develop this measure. However, in the absence of such information, decision makers at NASA and in the Congress have little assurance that the $4.8 billion investment in SLI can fulfill the goals expected of the program, and they have reduced confidence that what has already been spent has placed NASA on track to meeting its primary goals. Conclusions: NASA aims to be able to define system requirements for SLI by November 2002. But meeting this goal may not be realistic. NASA must first decide whether developing a second-generation vehicle to be deployed in 2014 to 2015 is still a worthwhile endeavor, given plans to extend the life of the space shuttle and cut back on the space station’s capabilities, and if so, what specific direction the program should take and how it will fit in with DOD efforts. Making such decisions within the short time remaining before the SLI systems requirements review will be difficult because it will require NASA to (1) resolve differences with its space station partners, who have concerns about planned cutbacks to the station; (2) reach consensus with DOD on requirements and priorities, which are now considerably different; and (3) make trade-offs as to what capabilities and technologies it should pursue without really knowing when advancements can be achieved. When NASA is able to resolve these challenges and finalize requirements for SLI, it will be critical for NASA to swiftly implement effective management to oversee the effort. Until it does so, NASA will not be able to assure its managers and the Congress that the initial investment is being spent wisely and that risks are being reduced, and it will not be able to predict what the total costs of the program will be. Recommendations: We recommend that the NASA Administrator do the following: 1. Reassess the schedule for defining the requirements for the Space Launch Initiative in order to ensure that the agency takes the following actions before making final decisions on basic requirements and selecting three architectures to pursue: (1) complete the reassessment of NASA’s integrated space transportation plan, (2) reach consensus with its international partners on the future of the space station, and (3) reach consensus with the Department of Defense on its role in the SLI effort. 2. If DOD is to jointly develop the second-generation vehicle, reach consensus with DOD on priorities and objectives for SLI and factor DOD’s requirements into NASA’s own. 3. After NASA completes its system requirements review, ensure that a cost estimate is promptly developed for the SLI investment and that this estimate is detailed and reliable enough to be used to complete risk mitigation plans and carry out earned value management activities. 4. Ensure that NASA can demonstrate how the $4.8 billion initial investment supports the requirements that NASA decides to pursue in November 2002. 5. After system requirements are defined, ensure that performance measures are developed to assess the progress of the program, as a whole, toward (1) meeting key performance objectives, including lowering the cost of delivering payloads to low-Earth orbit to less than $1,000 per pound and reducing the risk of crew loss to approximately 1 in 10,000 missions, and (2) achieving an overall technology readiness level of 6 by 2006. 6. Ensure that measures are developed to assess the amount of risk reduced following each year of expenditure and the amount of risk remaining. Agency Comments: In written comments on a draft of this report, NASA’s Associate Deputy Administrator said the agency concurs with the recommendations, adding that the program completed its first year as planned and within budget. According to the Associate Deputy Administrator, the agency is reassessing the Integrated Space Transportation Plan and working with DOD to develop their requirements. Furthermore, NASA concurs with the recommendation concerning the development of a cost estimate. However, the agency notes that the RLV industry suffers uncertainties in predicting costs, primarily because of a limited set of models and data to validate those models. NASA also concurred with the need to validate SLI requirements and the need for performance measures to assess the progress of the program towards meeting objectives and achieving technology readiness levels. Our draft report contained a recommendation that NASA develop measures to assess the percentage of risk reduced following each year of expenditures and the percentage of risk remaining. NASA believes that its process of narrowing potential SLI technology alternatives to the most promising concepts meriting further funding enables the successful candidates to be built and become operational. In that way, the SLI program is reducing the risks inherent in an advanced research and technology program of this magnitude. According to NASA, it will develop a reporting mechanism that communicates the amount of risk reduction achieved and remaining on an annual basis. Our intent is for NASA to provide a quantifiable means of measuring progress. Thus, we modified that recommendation accordingly. Scope and Methodology: To assess the coordination of requirements with DOD and other NASA programs, we interviewed officials within the Department of Defense and NASA. We evaluated studies conducted by DOD and NASA on efforts to address requirements and coordination. To address NASA’s management controls for the SLI program, we interviewed NASA officials regarding cost estimates and the process by which cost information is studied and communicated throughout NASA. We reviewed NASA’s and the Office of Management and Budget’s guidance regarding earned value management and discussed the program’s methods for coordinating activities. We also reviewed risk mitigation plans to determine how cost affects these plans and discussed these plans with program officials. To assess program performance measures, we reviewed NASA’s policies and procedures governing program management and the program’s plans for assessing performance measures. We also interviewed SLI program officials to understand the program’s plans to meet goals and objectives. To accomplish our work, we interviewed officials from NASA’s headquarters, Washington, D.C.; Marshall Space Flight Center, Alabama; Johnson Space Center, Texas; Kennedy Space Center, Florida; and Independent Program Assessment Office, Virginia; and DOD’s Space Command, Colorado. We performed our review from October 2001 through August 2002 in accordance with generally accepted government auditing standards. Unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies to the NASA Administrator; Director, Office of Management and Budget; and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http:// www.gao.gov. Please contact me at (202) 512-4841 if you or your staffs have any questions about this report. Major contributors to this report are listed in appendix II. Allen Li Director Acquisition and Sourcing Management: Signed by Allen Li [End of section] Appendix I: Comments from the National Aeronautics and Space Administration: National Aeronautics and Space Administration: Office of the Administrator Washington, DC 20546-0001: SEP 5 2002: Mr. Allen Li: Director, Acquisition and Sourcing Management Team: United States General Accounting Office Washington, DC 20548: Dear Mr. Li: NASA has reviewed the draft audit report (GAO-02-1020) and concurs with the recommendations as submitted for the Administrator. The program has completed its first year as planned and within budget. Much of this success is directly related to the successful implementation of stringent project management controls coupled with appropriate levels of insight. In parallel with the ongoing requirements definition efforts, the program has continued to implement its strategic plan for SLI, which will enable the program to provide the Agency with the necessary scientific and technological data required to support the design and development of NASA’s future Space Transportation Systems. The next several paragraphs provide the planned approach to implement each of the recommendations outlined in the report. 1. Reassess the schedule for defining the requirements for SLI in order to ensure that the Agency takes the following actions before making final decisions on basic requirements and selecting three architectures to pursue: (1) complete reassessment of NASA’s integrated space transportation plan, (2) reach consensus with its international partners on the future of space station, and (3) reach consensus with the Department of Defense on its role in the SLI effort. NASA concurs with this recommendation. The Agency is reassessing the Integrated Space Transportation Plan (ISTP) and will provide program guidance in mid-September 2002. It is acknowledged that the importance of the major Agency decisions cited in Recommendation 1 warrant careful consideration, as their resolution will ultimately affect the architecture design and associated required technologies. The Agency continues to actively work with the DoD in the development of their requirements and potential partnership for the development of a next generation RLV. The DoD is expecting to complete the development of their requirements early in FY 2003. 2. If DoD is to jointly develop the 2nd Generation vehicle, reach consensus with DoD on priorities and objectives for SLI and factor DoD’s requirements into NASA’s own. NASA concurs with the recommendation that the Agency reach consensus with the DoD on the requirements and missions for the next generation of reusable launch vehicles. The analysis and activities to achieve this consensus are ongoing and will continue through FY 2003. We are working closely with our counterparts within the Air Force to develop a converged set of requirements. 3. After NASA completes its system requirements review, ensure that a cost estimate is promptly developed for the SLI investment and that this estimate is detailed and reliable enough to be used to complete risk mitigation plans and to carry out earned value management. NASA concurs with this recommendation with the qualifications that, the program’s purpose is to achieve the necessary resolution in cost estimation. SLI is building the analysis capability to perform reliable development and operational RLV cost estimation for the Agency and the industry. As you are aware, the RLV industry suffers with tremendous uncertainties in cost predictions, primarily due to a limited set of models and data to validate those models. The recommendation (understandably) reiterates a general lack of confidence by our stakeholders in the Agency’s ability to predict costs of complex systems, including RLV’s. It underlines the need for the development of this capability, and SLI continues to work this task. 4. Ensure that NASA can demonstrate how the $ 4.8 billion initial investment supports the requirements that NASA decides to pursue in November 2002. NASA concurs with this recommendation. Existing NASA guidelines specify that validation is required prior to approval of Level I requirements. The validation process includes analyses to support the reasonableness and achievability of the requirements. S. After system requirements are defined, ensure that performance measures are developed to assess the progress of the program as a whole toward (1) meeting key performance objectives, including lowering the cost of delivering payloads to low-Earth orbit to less than $1000 per pound and reducing the risk of crew loss to approximately 1 in 10,000 missions and (2) achieving an overall technology readiness level 6 by 2006: NASA concurs with the recommendation. The SLI program maintains detailed performance measures at the project and program level, including key milestones, progress reviews, system and subsystem tests, product deliveries, and Government Performance and Reporting Act (GPRA) indicators. These metrics are continually assessed and frequently reviewed for applicability toward the program goals and objectives. The program has successfully implemented Earned Value Management (EVM) on the SLI program content. 6. Ensure that measures are developed to assess the percentage of risk reduced following each year of expenditure and the percentage of risk remaining. NASA concurs with this recommendation with the following amplification. The Agency will develop a reporting mechanism that will properly communicate the amount of risk reduction achieved and remaining on an annual basis. The process of narrowing architectures completed a major milestone - the Interim Architecture and Technology Review - in the second quarter of FY 2002. The next phase will continue to focus space transportation system designs from many concepts to several of the most promising candidates to go forward into more detailed development. As technology trade studies are focused and validated through a rigorous systems engineering process, the two (technology trade studies and transportation system designs) will narrow the field to a select few designs and define which technologies require further investment to enable the eventual winner to be built and become operational. In this way, the SLI is reducing the risks inherent in an advanced research and development program of this magnitude, while fostering a fair business environment for industry and ensuring the wise use of valuable resources. We are developing the detailed information that will help to quantify the risk reduced versus the investment and provide the necessary knowledge to meet the SLI program objectives. Cordially, Daniel R. Mulville: Associate Deputy Administrator: [Signed by Daniel R. Mulville] [End of section] Appendix II: GAO Contact and Staff Acknowledgments: GAO Contact: Jerry Herley (202) 512-7609: Acknowledgments: In addition to the person named above, Cristina Chaplain, Ivy Hubler, Danny Owens, and Dana Solomon made key contributions to this report. [End of section] GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061 To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470 Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: FOOTNOTES [1] “Architecture” broadly includes an Earth-to-orbit reusable launch vehicle; on-orbit transfer vehicles and upper stages; mission planning; ground and flight operations; and support infrastructure, both on the ground and in orbit. [2] See U.S. General Accounting Office, Space Transportation: Critical Areas NASA Needs to Address in Managing Its Reusable Launch Vehicle Program, GAO-02-826T (Washington, D.C.: June 20, 2001). [3] See NASA Policy Directive 7120.4A, Program/Project Management and NASA Procedures and Guidelines 7120.5A, Program and Project Management Processes and Requirements. [4] See U.S. General Accounting Office, Space Station: Actions Under Way to Manage Cost, but Significant Challenges Remain, GAO-02-735 (Washington, D.C.: July 17, 2002). [5] 31 U.S.C. §1115.