This is the accessible text file for GAO report number GAO-11-67 entitled 'Information Security: National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations' which was released on December 9, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: December 2010: Information Security: National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations: GAO-11-67: GAO Highlights: Highlights of GAO-11-67, a report to congressional requesters. Why GAO Did This Study: In the absence of underground nuclear weapons testing, the National Nuclear Security Administration (NNSA) relies on its supercomputing operations at its three weapons laboratories to simulate the effects of changes to current weapons systems, calculate the confidence of future untested systems, and ensure military requirements are met. GAO was requested to assess the extent to which (1) NNSA has implemented contingency and disaster recovery planning and testing for its classified supercomputing systems, (2) the laboratories are able to share supercomputing capacity for recovery operations, and (3) NNSA tracks the costs for contingency and disaster recovery planning for supercomputing assets. To do this work, GAO examined contingency and disaster recovery planning policies and activities, and analyzed classified supercomputing capabilities at the weapons laboratories, and NNSA budgetary data. What GAO Found: All three NNSA weapons laboratories—-Los Alamos, Sandia, and Lawrence Livermore—-have implemented some components of a contingency planning and disaster recovery program. NNSA, however, has not provided effective oversight to ensure that the laboratories have comprehensive and effective contingency and disaster recovery planning and testing. Further, due to lack of planning and analysis by NNSA and the laboratories, the impact of a system outage is unclear. Only one of the three laboratories-—Los Alamos-—had conducted a business impact analysis to assess the criticality of resources and acceptable outage time frames; yet, NNSA and all three laboratories consider the consequence associated with the loss of system availability to be low impact and do not consider the classified supercomputers to be mission critical. Nonetheless, NNSA classified supercomputing capabilities serve as a computational surrogate to nuclear weapons testing and are used to address other areas of national security. Despite the absence of business impact analyses, all laboratories had key components of a contingency planning program in place. However, shortcomings existed. For example, all laboratories had backup processes in place and had developed contingency plans, but the plans were not comprehensive. Specifically, one plan did not address the supercomputing operations, and none of the plans had been tested at the time of GAO’s review. In addition, the laboratories addressed disaster recovery to a limited extent, but not specifically for the supercomputers. These shortcomings existed, at least in part, because NNSA’s component organizations, including the Office of the Chief Information Officer, were unclear about their roles and responsibilities for providing oversight in the laboratories’ implementation of contingency and disaster recovery planning. Until the agency fully implements a contingency and disaster recovery planning program for its weapons laboratories, it has limited assurance that vital information can be recovered and made available to meet national security priorities and requirements. Although the laboratories have the technological capability to share supercomputing capacity across all three weapons laboratories, barriers exist that could impede recovery operations. For example, the laboratories do not know the minimum supercomputing capacity needed to meet program requirements, such as simulating the effects of changes to weapons systems, should a disruption occur. In addition, the laboratories have not tested the technological capability to share the capacity on an on-demand basis for recovery operations. Without having an understanding of capacity needs and subsequent testing, the laboratories have little assurance that they could effectively share capacity if needed. Although NNSA obligated approximately $1.7 billion to help implement its classified supercomputing program from fiscal years 2007 through 2009, the agency has not tracked costs for contingency and disaster recovery planning and is uncertain of actual funds that were spent toward these efforts. What GAO Recommends: GAO recommends, among other things, that NNSA clearly define roles and responsibilities for its component organizations in providing oversight for contingency and disaster recovery planning for the classified supercomputing environment. NNSA agreed with most of GAO’s recommendations, but did not concur with recommendations relating to capacity planning and cost tracking. View [hyperlink, http://www.gao.gov/products/GAO-11-67] or key components. For more information, contact Gregory C. Wilshusen at (202) 512-6244 or wilshuseng@gao.gov, Gene Aloise at (202) 512-3841 or aloisee@gao.gov, or Naba Barkakati at (202) 512-6415 or barkakatin@gao.gov. [End of section] Contents: Letter: Background: NNSA Has Not Fully Implemented Contingency and Disaster Recovery Planning and Testing for Its Classified Supercomputing Assets: The Laboratories Have the Ability to Share Supercomputing Capacity, but Barriers Exist: NNSA Does Not Track the Costs for Ensuring Contingency and Disaster Recovery Planning for Its Supercomputing Assets: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: NNSA Annual Obligations for Its Advanced Simulation and Computing Program, Fiscal Years 2007 through 2009: Appendix III: Comments from the National Nuclear Security Administration: Appendix IV: GAO Contacts and Staff Acknowledgments: Table: Table 1: Inventory of NNSA-Deployed Classified Supercomputing Systems (as of October 2010): Figures: Figure 1: Common Hardware Components of a Supercomputing System: Figure 2: NNSA's Classified Supercomputing Network Infrastructure: Figure 3: Total Usable Supercomputing Capacity at Each Weapons Laboratory, 2010 and 2011: Figure 4: Annual Obligations for NNSA's Advanced Simulation and Computing Program, Fiscal Years 2007 through 2009: Abbreviations: ASC: Advanced Simulation and Computing: BIA: business impact analysis: CNSS: Committee on National Security Systems: DISCOM: Distance Computing: DOD: Department of Defense: DOE: Department of Energy: FISMA: Federal Information Security Management Act of 2002: FLOPS: floating-point operations per second: Livermore: Lawrence Livermore National Laboratory: Los Alamos: Los Alamos National Laboratory: NIST: National Institute of Standards and Technology: NNSA: National Nuclear Security Administration: Sandia: Sandia National Laboratories: [End of section] United States Government Accountability Office: Washington, DC 20548: December 9, 2010: The Honorable Henry Waxman: Chairman: Committee on Energy and Commerce: House of Representatives: The Honorable Edward J. Markey: Chairman: Subcommittee on Energy and the Environment: Committee on Energy and Commerce: House of Representatives: The Honorable Bart Stupak: Chairman: Subcommittee on Oversight and Investigations: Committee on Energy and Commerce: House of Representatives: The National Nuclear Security Administration[Footnote 1] (NNSA) provides classified supercomputing capabilities for assessing the performance of nuclear weapons. In the absence of nuclear weapons testing--which ceased in 1992--the simulation capabilities of NNSA's supercomputers are a necessary means to determine the effects of changes to current weapons systems and to determine a level of confidence in the performance of future untested systems.[Footnote 2] These simulation capabilities also contribute to the enhancement of NNSA's ability to predict the performance of weapons systems to ensure the systems meet all military requirements established by the Department of Defense (DOD). NNSA's three nuclear weapons laboratories--Los Alamos National Laboratory (Los Alamos) in New Mexico, Lawrence Livermore National Laboratory (Livermore) in California, and the Sandia National Laboratories (Sandia) with locations in New Mexico and California--use these supercomputing simulation capabilities to obtain a comprehensive understanding of the entire nuclear weapons life cycle, from design to safe processes for dismantlement. These classified supercomputing capabilities are a considerable investment and serve as a cornerstone for NNSA's Stockpile Stewardship Program.[Footnote 3] In addition, classified supercomputing capabilities are essential for informing critical decisions related to the nuclear stockpile, including all stockpile modernization and warhead studies. NNSA classified supercomputing capabilities are also used to address other areas of national security, including intelligence analyses, nuclear forensics, and emergency response. Because of the importance of these classified supercomputing capabilities to issues central to national security, contingency and disaster recovery planning[Footnote 4] are key to ensuring that, when unexpected events occur, NNSA can recover and reconstitute its classified supercomputing systems, data, and operations. Our objectives were to assess the extent to which (1) NNSA has implemented contingency and disaster recovery planning and testing for its classified supercomputing assets, (2) the three laboratories are able to share classified supercomputing capacity for recovery operations, should service disruptions occur, and (3) NNSA tracks the costs for ensuring contingency and disaster recovery planning for its classified supercomputing assets. To accomplish these objectives, we examined contingency and disaster recovery planning controls for the systems within the classified supercomputing environment that are a necessary means for NNSA's achievement of its nuclear weapons mission. In addition, we performed technical assessments of classified supercomputing capabilities at each weapons laboratory, including each laboratory's ability to share supercomputing capacity. Further, we obtained information from NNSA and laboratory officials to determine how expenditures were tracked for contingency and disaster recovery planning of the classified supercomputing systems at each of the laboratories, as well as projected future cost estimates for ensuring the recovery of these assets. We conducted this performance audit from December 2009 through December 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. A more detailed description of our objectives, scope, and methodology is contained in appendix I. Background: NNSA relies on its Stockpile Stewardship Program to ensure the safety, security, and effectiveness of the nuclear weapons stockpile. The Stockpile Stewardship Program is comprised of various elements, including, but not limited to: (1) the Advanced Simulation and Computing (ASC) Campaign, which provides the computational science and simulation tools to understand the behaviors and effects of nuclear weapons; (2) Directed Stockpile Work, which provides evidence of the health of the nuclear weapons stockpile and involves day-to-day maintenance of these weapons, including life extension efforts; (3) the Science Campaign, which provides tools and capabilities geared toward advancing the general understanding of all nuclear weapons systems; and (4) the Engineering Campaign, which provides a sustained basis for stockpile certification and assessments throughout the life cycle of each weapon. The coordination among the Stockpile Stewardship elements is instrumental to increasing NNSA's confidence in the performance of nuclear weapons. To help accomplish its Stockpile Stewardship mission, NNSA relies on the three weapons laboratories--Los Alamos, Livermore, and Sandia. Los Alamos and Livermore are the two design laboratories that are responsible for designing the nuclear weapons' explosive package and conducting research to better understand nuclear weapons phenomena. Sandia is an engineering laboratory and has principal responsibility for the research, design, and development of nonnuclear warhead components; integration of these components with Los Alamos and Livermore; and overall warhead systems integration with DOD. In accordance with NNSA, management and operations contractors, who are responsible for day-to-day operations of the laboratories, are required to adhere to agency policies.[Footnote 5] At the time of our review, NNSA's classified supercomputing resources consisted of 12 classified supercomputing systems. Figure 1 shows the hardware configuration of a supercomputing system. Figure 1: Common Hardware Components of a Supercomputing System: [Refer to PDF for image: illustration] Compute chip: Compute card: Node card: Cabinet: System: Source: GAO, data provided by Los Alamos, Livermore, and Sandia. [End of figure] NNSA classified supercomputing systems employ a large number of interdependent processors, which are the core unit of a computer that gathers instructions and data. These processors are mounted onto a compute chip, which is the portion of the system that carries out the instructions of a computer program. These compute chips are inserted onto a compute card, which also holds memory for the compute chips to use. A number of compute cards are attached to a node card, which have one or more processors with a common memory and are connected by high- speed interconnection networks. Each node card is inserted into a single cabinet, and that configuration is repeated many times to build a single supercomputing system. Each supercomputing system has a peak performance, which is the maximum rate of floating-point operations per second (FLOPS) that the system can sustain.[Footnote 6] Currently, almost all NNSA classified supercomputer systems operate at the teraFLOP level, which represents a trillion FLOPS. According to NNSA, the laboratories have three types of classified supercomputing systems: Capacity: Small systems that execute parallel problems with more modest computational requirements. These systems serve as the workhorse for the ASC program and are responsible for processing the day-to-day supercomputing workload. Capability: This type of supercomputer is used to solve the largest and most demanding problems that other computing systems cannot manage. Advanced architecture: Research and development systems that assist the ASC program in preparing to rapidly deploy and exploit the next generation of supercomputing technology. These systems have a targeted workload and serve as the foundation for the next generation of NNSA supercomputers. Table 1 shows the classified supercomputing systems currently in use at the three weapons laboratories. Table 1: Inventory of NNSA-Deployed Classified Supercomputing Systems (as of October 2010): Site: Los Alamos: System name: Roadrunner Base; System type: Capacity; Delivery date: 10/2006; Total processors: 18,432; Peak performance (TeraFLOPS): 76.0. System name: Roadrunner Phase-3; System type: Advanced architecture; Delivery date: 9/2008; Total processors: 24,480; Peak performance (TeraFLOPS): 1,280.0. System name: Hurricane; System type: Capacity; Delivery date: 9/2008; Total processors: 5,760; Peak performance (TeraFLOPS): 51.2. Site: Livermore: System name: BlueGene/L; System type: Advanced architecture; Delivery date: 11/2004; Total processors: 131,072; Peak performance (TeraFLOPS): 367.0. System name: Purple[A]; System type: Capability; Delivery date: 6/2005; Total processors: 12,288; Peak performance (TeraFLOPS): 93.4. System name: Rhea; System type: Capacity; Delivery date: 9/2006; Total processors: 4,608; Peak performance (TeraFLOPS): 22.1. System name: Minos; System type: Capacity; Delivery date: 6/2007; Total processors: 6,912; Peak performance (TeraFLOPS): 33.2. System name: Juno; System type: Capacity; Delivery date: 5/2008; Total processors: 18,432; Peak performance (TeraFLOPS): 162.2. System name: Dawn; System type: Advanced architecture; Delivery date: 1/2009; Total processors: 147,456; Peak performance (TeraFLOPS): 501.4. Site: Sandia-NM: System name: Red Storm; System type: Advanced architecture; Delivery date: 3/2005; Total processors: 31,680; Peak performance (TeraFLOPS): 284.0. System name: Unity; System type: Capacity; Delivery date: 3/2009; Total processors: 4,352; Peak performance (TeraFLOPS): 38.0. Site: Sandia-CA; System name: Whitney; System type: Capacity; Delivery date: 3/2009; Total processors: 4,352; Peak performance (TeraFLOPS): 38.0. Source: GAO summary of data from Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Sandia National Laboratories. [A] Although Purple was the capability system in use at the time of our site visits, Livermore retired the system in November 2010. [End of table] NNSA's classified supercomputing capabilities consist of supporting resources, including (1) parallel files systems, which store transitory data; (2) network file systems, which store user and project data for a calculation; (3) archival storage systems, which serve as storage for data; and (4) visualization systems, which enable users to better comprehend the results of their computations. NNSA's classified supercomputing systems are connected via its Enterprise Secure Network and the Distance Computing (DISCOM) network, which function as supporting resources for the classified supercomputing environment. The Enterprise Secure Network provides classified communications across the nuclear weapons complex, including security services and other activities that ensure the flow of NNSA's data sharing and business missions. DISCOM provides secure, high-speed remote access for intra-and inter-site file transfers and enables users, across the three weapons laboratories, to operate on remote computing resources as if they were local. DISCOM and the Enterprise Secure Network serve as the backup networks to each other. Figure 2 shows the composition of NNSA's classified supercomputing network infrastructure. Figure 2: NNSA's Classified Supercomputing Network Infrastructure: [Refer to PDF for image: illustration] The illustration depicts the following connections: Connected to DISCOM Network (10 Gigabits per second): and to NNSA Enterprise Secure Network (1 Gigabit per second): Lawrence Livermore National Laboratory: * BlueGene/L; * Purple; * Rhea; * Minos; * Juno; * Dawn. Los Alamos National Laboratory: * Road Runner; * Road Runner Phase 3; * Hurricane. Sandia National Laboratories, California: * Whitney. Sandia National Laboratories, New Mexico: * Red Storm; * Unity. DISCOM Network and NNSA Enterprise Secure Network are interconnected. Source: GAO, data provided by Los Alamos, Livermore, and Sandia. [End of figure] NNSA reported obligating approximately $1.7 billion from fiscal years 2007 through 2009 to support ASC program activities at the three weapons laboratories.[Footnote 7] The $1.7 billion was predominantly associated with three efforts: Weapons codes and models. This effort is intended to develop and improve weapons simulation models and codes for predicting the behavior of weapons systems and devices in the nuclear stockpile. Computational systems and software environment. This effort is intended to provide ASC users a stable, seamless computing environment for ASC-deployed platforms. It is responsible for procuring, delivering, and deploying ASC computational systems and user environments via technology development and integration across the three weapons laboratories. Facility operations and user support. This effort is intended to provide both the necessary physical facility and operational support for reliable supercomputing and storage environments, as well as a suite of user services for effective use of the three weapons laboratories' computing resources. Facility operations cover physical space, power and other utility infrastructure, and local-and wide-area networking, as well as system administration, cyber security, and operations services for ongoing support. The user support function includes planning, development, integration and deployment, continuing product support, and quality and reliability activity collaborations. To strengthen the security of information and information systems across the federal government, including those at NNSA's weapons laboratories, the Federal Information Security Management Act of 2002 (FISMA) requires each agency to develop, document, and implement an agencywide information security program that supports the operations and assets of the agency, including those provided or managed by another agency or contractor on its behalf.[Footnote 8] This security program is to include plans and procedures to ensure the continuity of operations for information systems that support the agency's operations.[Footnote 9] Pursuant to its FISMA responsibilities, the National Institute of Standards and Technology (NIST) has issued federal standards and guidelines on information security, such as a contingency planning guide for federal information systems, and recommended security controls, which address contingency and disaster recovery planning and testing.[Footnote 10] To further ensure the security of national security systems, the Committee on National Security Systems (CNSS)[Footnote 11] requires federal agencies with national security systems to implement a comprehensive set of security controls and enhancements for these systems.[Footnote 12] CNSS requires that each agency implement a contingency and disaster recovery planning capability that ensures the integrity and availability of its national security information and information systems.[Footnote 13] FISMA, NIST guidelines,[Footnote 14] and CNSS policies all call for contingency and disaster recovery planning--also referred to as continuity of operations for information systems--for critical components of information protection. DOE and NNSA policies also regard contingency and disaster recovery plans as being necessary for information protection. If normal operations are interrupted, contingency and disaster recovery plans allow senior agency officials to detect, mitigate, and recover operations. Examples of the key components that make up contingency and disaster recovery planning programs include (1) assessing the criticality and sensitivity of computerized operations and identification of supporting resources such as developing business impact analyses (BIA), (2) taking steps to prevent and minimize potential damage and interruption such as establishing data backup processes, (3) developing comprehensive contingency and disaster recovery plans,[Footnote 15] and (4) conducting periodic testing of contingency and disaster plans. The extent to which controls--such as contingency and disaster recovery planning--are implemented depends on a level of risk assigned to the system or information maintained on the system. NIST standards and guidelines, CNSS instructions, and NNSA policy allow consideration of risk in determining the level of protection of systems and data. These standards and policies require that organizations consider the impact or consequences of loss as it relates to the confidentiality, integrity, and availability of the information, and assign a value of low, moderate, or high impact levels. For contingency and disaster recovery planning, consideration of "availability" is the key element. NNSA policy defines the values for the consequences of loss associated with availability as follows: High: Loss of life might result from loss of availability; information must always be available on request, with no tolerance for delay; loss of availability will have an adverse effect on national-level interests; federal requirement (i.e., requirement for material control and accountability inventory); or loss of availability will have an adverse effect on confidentiality. Moderate: Information must be readily available with minimum tolerance for delay; bodily injury might result from loss of availability; or loss of availability will have an adverse effect on organizational- level interests. Low: Information must be available with flexible tolerance for delay. NNSA Has Not Fully Implemented Contingency and Disaster Recovery Planning and Testing for Its Classified Supercomputing Assets: Contingency and disaster recovery planning and testing for NNSA's classified supercomputing systems have not been fully implemented at each of the three weapons laboratories--Los Alamos, Sandia, and Livermore. Specifically, NNSA did not ensure that the laboratories (1) developed BIAs to determine the impact of potential service disruptions, (2) fully tested data backup processes, and (3) developed and tested contingency and disaster recovery plans. These shortcomings existed, at least in part, because NNSA's component organizations were unclear of their roles and responsibilities for providing oversight in the laboratories' implementation of contingency and disaster recovery planning. Until the agency fully implements a contingency and disaster recovery planning program for its classified supercomputing assets at the weapons laboratories, it has limited assurance that vital information can be recovered and made available to meet national security priorities and requirements. Not All of the Laboratories Assessed the Criticality and Sensitivity of Supercomputer Operations and Resources, or Potential Outage Impact: To assess the criticality and sensitivity of computerized operations and identification of supporting resources, NIST guidelines state that agencies should determine their recovery strategies by performing business impact analyses of their systems. A BIA is an analysis of information technology system requirements, processes, and interdependencies used to characterize system contingency requirements and priorities in the event of a significant disruption. NIST guidelines state that agencies conduct a BIA to identify critical information systems to fully characterize the system's requirements, processes, and interdependencies to determine contingency requirements and priorities. In addition, according to NIST guidelines, the BIA process should follow three main steps: (1) identify critical data and information technology resources, (2) identify outage impacts and allowable outage times,[Footnote 16] and (3) develop recovery priorities and strategies. NNSA policy also requires a BIA to identify systems that provide critical services to site operations and prioritize these systems and their components. One of the laboratories--Los Alamos--had conducted a BIA that addressed its classified supercomputing systems, generally following the three steps of a BIA. However, the BIA was not always specific. For example, the laboratory identified critical information technology resources for each of its classified supercomputing systems, but did not specifically identify the critical data. Instead, Los Alamos noted that the systems are not considered mission critical nor mission essential to the business needs of the laboratory,[Footnote 17] and that the consequence of loss for system availability is low. Additionally, it defined a specific number of days for the allowable time frames for fully and partially disabled systems, but did not provide specifics on allowable outage impacts. Further, the analyses indicated high-level recovery priorities, but did not provide specifics regarding the recovery process or strategies that would be used for recovery efforts. The other two laboratories did not conduct BIAs specifically for classified supercomputing systems, but plan to do so. Livermore has a BIA in place for its logical assets--the applications and services that provide basic operational support to the Livermore computing environment, but the BIA did not address any of the classified supercomputing systems. However, at the time of our site visit, Livermore officials stated they were beginning the process of developing a BIA that would address their information technology needs for their classified supercomputing systems, but the process was still in the planning stage. Similarly, according to Sandia officials, the laboratory has BIAs that address its unclassified information technology systems, but does not currently have one specifically for its classified supercomputing systems. However, Sandia officials indicated that they plan to conduct a BIA for classified supercomputing systems in 2011. Although the two laboratories have not conducted any BIAs--in line with the BIA conducted by Los Alamos--they have considered the risk of consequence of loss from availability as low impact. NNSA also considers the consequence of loss as low impact. In addition, NNSA and the three laboratories do not consider the classified supercomputers to be "mission critical." One laboratory categorized the systems as "mission essential," while another referred to them as "mission support elements, not mission essential elements." However, NNSA's mission includes maintaining the safety, security, and effectiveness of the nuclear deterrent without nuclear testing. The supercomputers provide a necessary means to determine the effects of changes to current weapons systems and to determine a level of confidence in the performance of future untested systems. The classified supercomputing capabilities serve as the computational surrogate to nuclear weapons testing and are central to national security. Regarding recovery priorities and strategies, each of the laboratories indicated that it would likely rely on a process that is currently being used for the capability system shared among the laboratories. The laboratories generally rely on the Capability Computing Campaign to prioritize the workload and develop priorities for jobs that need to be run on the capability system.[Footnote 18] In the event of a service disruption or emergency, laboratory officials told us that they would likely rely on the same process for all of their systems. However, this process has not been documented as a means for establishing overall recovery priorities across the laboratories. Until all of the laboratories have a BIA in place for their classified supercomputing systems that (1) identifies and categorizes critical data, (2) identifies acceptable allowable outage impacts and time frames, and (3) establishes emergency processing priorities and strategies, the potential impact of a system outage will remain unclear. The Laboratories Have Backup Processes in Place, but One Storage Site May Be Susceptible to Damage: Data backup processes offer a means of taking steps to prevent and minimize potential damage and interruption to computerized services. NIST guidelines, as well as CNSS instructions and NNSA policies, call for agencies to conduct backups of user-level information, system- level information, and information system documentation. In addition, NIST, CNSS, and NNSA all provide that agencies establish an alternate storage site that is separated from the primary storage site so that both are not susceptible to the same hazards. To ensure the availability of data stored in the alternate storage site, NIST and CNSS require that agencies test the backup information to verify the integrity of the data. All of the laboratories had backup processes in place. Each of the laboratories follows similar data backup processing--both manual and automated procedures--to back up user-level information, system-level information, and information system documentation. For example, this information can include global directories, user home directories, project directories, desktop systems, and critical systems documentation. Backups occur in increments: daily incremental backups to disk, weekly full backups to tape, and monthly full-system backups to tape (with a 6-month on-site storage retention policy). The laboratories also have vendor-provided software that takes periodic snapshots of user directories for storage retention purposes. The snapshot process can be performed manually or can be set up for automatic processing. Users are encouraged to maintain their data in a shared environment on the network and are allowed to make their own determinations regarding what data should be backed up from the classified supercomputing systems. Not all of the laboratories have an alternate storage site sufficiently separated from the primary site to not be susceptible to the same hazards. Two of the three laboratories have alternate storage sites a considerable distance from their primary storage site. Livermore sends its system backups electronically to Los Alamos every 6 months. Sandia sends its backup data to its alternate site locations (e.g, the California site sends its data to the New Mexico site and the New Mexico site sends its data to the California site). However, Los Alamos maintains its alternate storage facility on-site in a building located less than 1 mile away from the primary local backup storage facility. Consequently, both sites could be susceptible to the same hazards, such as a wildfire. The laboratories had processes in place to verify the integrity of the backed up data. However, tests of their backup procedures rely predominantly on ad hoc recovery, rather than periodically planned tests. Los Alamos officials indicated that thousands of file recoveries have been performed over the years by end users as part of their testing. Livermore officials stated that the laboratory tests its local backup procedures through actual system usage on almost a daily basis, and tests their remote backup procedures at least once annually. Further, Sandia officials told us they had successfully tested a sample of data at their offsite facility. Not All Laboratories Had Developed and Tested Contingency and Disaster Recovery Plans: NIST guidelines and CNSS policies call for the development and testing of contingency plans and the development of disaster recovery plans for each information system to ensure that, in the event of a service disruption, the work and supporting functions of the agency can continue to be performed. According to NIST guidelines, at a minimum, the contingency plan should address the identification and notification of key personnel, plan activation, system recovery, and system reconstitution to meet the needs of the agency's critical supporting operations. The guidelines also state that the plan should be tested periodically; CNSS specifies that the frequency of testing should be annually. NIST also notes that the disaster recovery plan should be designed to restore operability of the targeted system, application, or computer facility at an alternate site after a major service disruption. DOE and NNSA policies also require the development of contingency and disaster recovery plans and the testing of these plans in line with NIST and CNSS. Each of the laboratories had developed contingency plans for their classified supercomputing systems; however, the plans were not always comprehensive, and at the time of our site visits, these plans had not been tested. The laboratories addressed disaster recovery planning to a limited extent; none specifically addressed the supercomputing environment. For example, * Two laboratories--Los Alamos and Sandia--had contingency plans that addressed the classified supercomputing systems. Although Livermore had an information technology contingency plan and a master security plan, neither specifically addressed the supercomputers. In addition, the plans for both Los Alamos and Sandia included key components such as the identification and notification of key personnel, plan activation, system recovery, and system reconstitution procedures; however, the sufficiency of the level of detail varied. For instance, one plan provided specific details regarding system recovery processes and the notification and identification of key personnel, but provided limited details regarding plan activation and system reconstitution procedures. * At the time of our site visits, none of the laboratories had tested their contingency plans, which were less than a year old. One of the laboratories--Los Alamos--had created testing guides but had not yet conducted formal testing. Subsequent to our site visit, Los Alamos officials indicated that the first test of their plan took place in September 2010 and noted that the results would be finalized in December. Additionally, although Sandia had a contingency plan in place, the plan states that testing is not required because, in the event of a service disruption, the laboratory would either wait until the equipment was fully operational or simply acquire new equipment. This is contrary to NIST guidelines, CNSS instructions, and DOE and NNSA policies. * Each of the laboratories had addressed disaster recovery planning to a limited extent. For example, - Los Alamos included disaster recovery planning as a section within their classified supercomputing system's contingency plans. Although it provided high-level instructions such as directing individuals to call 911 for all emergencies, it did not include information regarding the specifics for restoring operability of the classified supercomputing system at an alternate site after a major service disruption. - None of the plans submitted by Livermore specifically addressed the supercomputing environment, although in the disaster recovery section of the master security plan, the laboratory noted that it had no mission-essential systems in the computing environment, and systems may be offline for an extended period for system upgrades. - Sandia also had disaster recovery plans in place, designed for emergency preparedness and disease response planning needs for the laboratory. These plans focused on emergencies involving the facilities, operations, and activities for the laboratory, and provided individuals with emergency information should pandemics plague the laboratory. However, these plans did not include any information regarding the classified supercomputing systems. Unless each of the laboratories develops and sufficiently tests comprehensive contingency and disaster recovery plans in accordance with applicable policies and guidance for their classified supercomputing systems, they face a risk of not being able to successfully recover their supercomputing assets and operations after a service disruption. NNSA Component Organizations Were Unclear of Their Roles and Responsibilities for Providing Oversight: The aforementioned shortcomings existed, at least in part, because NNSA's component organizations were unclear of their roles and responsibilities for providing oversight in the laboratories' implementation of contingency and disaster recovery planning. FISMA requires that the chief information officer, in coordination with other senior agency officials, manage the development and implementation of an agencywide information security program that includes plans and procedures to ensure continuity of operations for information systems that support the operations and assets of the agency. NIST guidelines and DOE policies call for individuals with information system or security management and oversight responsibilities to take responsibility for the development, implementation, assessment, monitoring, reviewing, and updating of security planning policies and procedures, which includes contingency and disaster recovery plans. Further, the NNSA Safety Management Function and Responsibilities and Authorities Manual states that the chief information officer is responsible for information technology programs and initiatives and for ensuring the security of the agency's information and systems. Although roles and responsibilities are defined at a high level in FISMA, NIST guidelines, as well as DOE and NNSA policies, NNSA component organizations were confused about their roles in providing oversight of the laboratories' implementation of contingency and disaster recovery planning for the supercomputing systems. For example, at the beginning of our review, ASC officials told us that, although they were responsible for administering and managing the program that uses the classified supercomputing systems, they were not responsible for contingency and disaster recovery planning. Instead, they directed us to the Office of the Chief Information Officer (OCIO), where officials told us that they were not responsible for contingency and disaster recovery planning for these systems, and noted that they would only provide guidance if requested by ASC. Further, OCIO officials told us that ASC has not requested any assistance. ASC officials subsequently acknowledged that they had responsibility for contingency and disaster recovery planning; however, this organizational responsibility is contrary to NIST guidelines and DOE policies, as well as NNSA's own manual, which gives this responsibility to the OCIO. In the absence of effective oversight, the laboratories did not consistently comply with, or fully implement, federal requirements and guidance related to contingency planning and disaster recovery. Until NNSA clearly establishes and carries out defined roles and responsibilities for OCIO and ASC pertaining to contingency and disaster recovery planning for the classified supercomputing environment, it will not be able to effectively manage and oversee the recovery of its supercomputing operations should service disruptions occur. The Laboratories Have the Ability to Share Supercomputing Capacity, but Barriers Exist: Technologically, the weapons laboratories have demonstrated the ability to share classified supercomputing capacity using their capacity and capability systems under normal operating conditions. Although these supercomputers process unique workloads and operate independently, they are designed with a similar operating system, resource manager, and job scheduler, which is built on a LINUX foundation. These supercomputers also include application codes that are portable across supercomputing systems and a data network, which allows authorized users local and remote access to the systems. Although the weapons laboratories have the ability to share supercomputing capacity, barriers exist. One barrier to sharing supercomputing capacity is that the weapons laboratories do not know the minimum supercomputing capacity needed to achieve processing priorities in the event of a service disruption. NIST guidelines recommend, and NNSA policy requires, that capacity planning be conducted so that there is adequate capacity for information processing and supporting resources during contingency operations. Although the weapons laboratories have identified the supercomputing processing needed for normal business operations, they have not identified the minimum supercomputing capacity needed to achieve processing priorities in the event of a service disruption. Another barrier to sharing supercomputing processing is the disparity in usable supercomputing processing across the laboratories. Figure 3 depicts this disparity by identifying the amount of total usable supercomputing capacity, in teraFLOPS, for each of the three weapons laboratories for 2010 and 2011.[Footnote 19] Figure 3: Total Usable Supercomputing Capacity at Each Weapons Laboratory, 2010 and 2011: [Refer to PDF for image: vertical bar graph] TeraFLOPS: Los Alamos; Total usable capacity: 2010: 127; Total usable capacity: 2011: 1,427. TeraFLOPS: Livermore; Total usable capacity: 2010: 311; Total usable capacity: 2011: 218. TeraFLOPS: Sandia; Total usable capacity: 2010: 76; Total usable capacity: 2011: 76. Source: GAO analysis of supercomputing capacity data provided by Los Alamos, Livermore, and Sandia. [End of figure] For example, in 2010, total usable capacity at Livermore has been 311 teraFLOPS, whereas Los Alamos and Sandia have had 127 teraFLOPS and 76 teraFLOPS, respectively. Should Livermore experience service disruptions for a sustained amount of time, neither Los Alamos nor Sandia possesses the necessary usable supercomputing capacity to accommodate the additional workload and NNSA will have to reprioritize the computational workloads across the other two laboratories. As previously noted, officials at the laboratories told us that, should disruptions occur, they would use the Capability Computing Campaign model for re-prioritizing the workload. However, this process has not been documented for recovery activities. Further limiting the ability of the weapons laboratories to recover from a service disruption, in 2011, there will be a significant disparity in projected usable supercomputing capacity. For example, in 2011, Los Alamos' usable capacity is projected to be 1,427 teraFLOPS, whereas usable capacity at Livermore and Sandia is to be 218 teraFLOPS and 76 teraFLOPS, respectively. Should Los Alamos' supercomputing systems become unavailable for an extended period of time, neither Livermore nor Sandia possesses sufficient usable supercomputing capacity to achieve its workload and accommodate the additional potential computational workload from Los Alamos. According to laboratory officials, an additional supercomputer will be deployed at Los Alamos in 2011. This supercomputer is a replacement for the single capability supercomputer currently at Livermore that was retired in 2010. Therefore, a significant amount of usable supercomputing capacity will be centralized at Los Alamos. Because the weapons laboratories have not determined the minimum supercomputing capacity requirements for their emergency processing priorities, they may not be able to meet the minimum computational workload required to meet Stockpile Stewardship milestones. Another barrier to sharing supercomputing capacity across the weapons laboratories is that the capability to share usable capacity on an "on- demand"[Footnote 20] basis has not been fully tested in a recovery scenario. According to officials at the laboratories, during normal operating conditions, simulation programs have run on other supercomputing systems. However, consideration has not been given to include and test these abilities in a disaster recovery scenario should a service disruption occur. As a result, NNSA has limited assurance that its disaster recovery approach would work effectively should a service disruption occur. NNSA Does Not Track the Costs for Ensuring Contingency and Disaster Recovery Planning for Its Supercomputing Assets: Although NNSA reported obligating approximately $1.7 billion from fiscal 2007 through 2009 to implement its ASC program activities at the three weapons laboratories, the costs for ensuring the recovery of its classified supercomputing operations are unknown. Under GAO's Standards for Internal Control in the Federal Government, financial information should be recorded and communicated to program managers who need this information to make operational decisions and to effectively allocate resources for program activities. NNSA officials reported obligating approximately $390 million for facility operations and user support activities, which include the funds associated with contingency and disaster recovery planning activities, but they were unable to provide detailed financial information for contingency and disaster recovery planning activities. According to NNSA officials, costs for contingency and disaster recovery planning for classified supercomputing systems are unknown because ASC program expenditures are part of the NNSA ASC operational budget, whose costs are tracked at an aggregate level. As a result, neither NNSA nor the three weapons laboratories can track what has been spent since fiscal year 2007 for ensuring the recovery of classified supercomputing operations and, consequently, they do not know whether funding levels for these activities have been adequate. Although certain components of contingency and disaster recovery planning are in place at the three weapons laboratories, NNSA is uncertain as to what funds were spent on these information protection activities. Furthermore, NNSA has not developed contingency planning and disaster recovery cost estimates for its classified supercomputing assets. For fiscal years 2011 through 2014, NNSA projects that it needs about $2.2 billion to implement its ASC activities, which support its Stockpile Stewardship program--$984 million for weapons codes and models, $604 million for computational systems and software environment, and $588 million for facility operations and user support. Although NNSA has developed its out-year funding needs over the next 4 years, it has not developed estimates regarding the future costs for ensuring the recovery of its classified supercomputing assets in the event of service disruptions. Until NNSA develops a means for tracking current contingency and disaster recovery costs and for developing estimates of future costs, the agency will not have the information needed to determine whether it is meeting its goals for effective stewardship of public resources. Conclusions: All three NNSA weapons laboratories have implemented some components of a contingency planning and disaster recovery program. NNSA, however, has not provided effective oversight to ensure that the laboratories have comprehensive and effective contingency and disaster recovery planning and testing. For example, BIAs that identify critical resources and outage impacts have not been developed for all classified supercomputing systems and existing contingency plans at the laboratories have not been thoroughly tested. Although one laboratory's analysis is not comprehensive and the other two laboratories have not completed a BIA, NNSA and the laboratories consider the consequence of loss of availability of the classified supercomputers as a low-risk impact, and do not consider them to be mission critical. However, it is unclear how NNSA made this determination given that (1) the analyses have not been completed; (2) NNSA's mission includes maintaining the safety, security, and effectiveness of the nuclear deterrent without nuclear testing; (3) the classified supercomputing capabilities serve as the computational surrogate to underground nuclear weapons testing and are central to our national security; and (4) NNSA has obligated about $1.7 billion over 3 fiscal years to support the Advanced Simulation and Computing program, which includes classified supercomputing activities. Beyond the activities undertaken by the laboratories, NNSA has not developed a means for identifying, tracking, or re-prioritizing the classified supercomputing workload across the operating environment. In addition, the laboratories have not tested offsite recovery capabilities and the agency has not tested the laboratories' ability to share "on-demand" capacity if needed or determined the minimum capacity needed to meet Stockpile Stewardship Program requirements, particularly in the event that it may need to establish emergency processing priorities. Further, although over a billion dollars have been obligated to support the classified supercomputing capabilities within the last 3 years, NNSA has not tracked the costs for ensuring the recovery of the classified supercomputing systems, data, and supporting resources should a service disruption occur. The classified supercomputing program represents a significant investment, and accountability for these systems is essential. Until NNSA clearly defines its component organizations' roles and responsibilities and fully implements an effective contingency and disaster planning program, it has limited assurance that, in the event of a service disruption, vital information could be recovered and made available to meet national security priorities. Recommendations for Executive Action: To improve the effectiveness of contingency and disaster recovery planning for NNSA's classified supercomputing capabilities, we recommend that the Administrator of NNSA direct the weapons laboratories to take the following four actions, where not already implemented: * Develop business impact analyses that, among other things, (1) identify and prioritize critical systems, data, and supporting resources; (2) identify allowable outage times and impacts for classified supercomputing capabilities; and (3) identify recovery priorities and strategies. * Develop and implement comprehensive contingency and disaster recovery plans for all classified supercomputing systems that identify how each weapons laboratory's classified supercomputing capabilities will be recovered following service disruptions. * Conduct contingency and disaster recovery plan testing. * Test the three weapons laboratories' ability to share "on-demand" classified supercomputing capacity to ensure this capability will work in the event of unexpected service disruptions. In addition, we recommend that the Administrator of NNSA take the following five actions: * Document an agencywide means for reprioritizing the workload across NNSA's classified supercomputing systems should a disruption occur. * Clearly define the oversight responsibilities of the NNSA ASC program office and the NNSA Office of the Chief Information Officer, as they relate to contingency and disaster recovery planning for NNSA's classified supercomputing operations. * Identify, assess, and communicate the minimum classified supercomputing capacity needed to meet Stockpile Stewardship requirements in the event of a service disruption. * Develop, document, and implement a process that identifies and tracks expenditures for contingency and disaster recovery planning for NNSA's classified supercomputing assets. * Develop and document the total anticipated costs for contingency and disaster recovery planning of NNSA's classified supercomputing assets, which includes the replacement costs for these assets. Agency Comments and Our Evaluation: In providing written comments (reprinted in appendix III) on a draft of this report, NNSA's Associate Administrator for Management and Administration agreed that improvements can be made in contingency and disaster recovery planning for supercomputing operations. He indicated that NNSA agreed with six of our nine recommendations and outlined the agency's intent to conduct business impact analyses, develop and test appropriate contingency and disaster recovery plans, document workload prioritization, and clearly define roles and responsibilities. However, NNSA did not agree with our recommendation related to identifying the minimum capacity needed to meet Stockpile Stewardship requirements in the event of a service disruption. The Associate Administrator stated that this recommendation did not take into account that the different types of supercomputers--capacity and capability--serve different functions and are procured and managed differently. In our report, we recognize that different types of supercomputers exist and that they are used for different purposes, they process unique workloads and operate independently. However, as we point out in the report, although the weapons laboratories have identified supercomputing processing needed for normal business operations, they have not identified the minimum capacity needed to achieve processing priorities in the event of a service disruption. We believe that the recommendation appropriately focuses on meeting NNSA's Stockpile Stewardship mission and that capacity planning is essential to ensure that information processing and supporting resources exist during contingency operations, regardless of the type of system used. Although NNSA did not agree with the recommendation, the Associate Administrator stated that the agency will conduct a BIA and build appropriate contingency strategies for both types of supercomputers, as well as enhance capacity sizing actions to account for contingency and disaster recovery operations. Further, NNSA did not agree with two recommendations related to identifying and tracking expenditures for contingency and disaster recovery planning and documenting anticipated recovery planning costs, including replacement costs of the assets. The Associate Administrator asserted that this information would not add significant value to managing contingency and disaster recovery planning. However, we believe such actions reflect good government practices and would add value by providing NNSA program managers with useful expenditure and cost information to aid decision making with regards to contingency and disaster recovery planning. As our report points out, GAO's Standards for Internal Control in the Federal Government states that financial information should be recorded and communicated to program managers to help them make operational decisions and effectively allocate resources for program activities. Strong financial and internal controls are a major part of managing any organization because they help government program managers achieve desired results through effective stewardship of public resources. Accordingly, we believe our recommendations have merit. We are sending copies of this report to the Secretary of Energy; the Administrator of NNSA; and the Directors of Los Alamos, Livermore, and Sandia laboratories. Copies of the report will also be available to others at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions about this report, please contact Gene Aloise at (202) 512-6870, or aloisee@gao.gov; Nabajyoti Barkakati at (202) 512-6415 or barkakatin@gao.gov; or Gregory C. Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are included in appendix IV. Signed by: Gene Aloise: Director, Natural Resources and Environment: Signed by: Nabajyoti Barkakoti: Director, Center for Technology and Engineering: Signed by: Gregory C. Wilshusen: Director, Information Security Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: The objectives of our review were to assess the extent to which (1) the National Nuclear Security Administration (NNSA) has implemented contingency and disaster recovery planning and testing for its classified supercomputing assets, (2) the three laboratories are able to share classified supercomputing capacity for recovery operations, should service disruptions occur, and (3) NNSA tracks the costs for ensuring contingency and disaster recovery planning for its classified supercomputing assets. To address these objectives, we focused on contingency and disaster recovery planning activities at NNSA headquarters, as well as the operating environment for the 12 classified supercomputing systems at the three weapons laboratories-- Los Alamos National Laboratory, Livermore National Laboratory, and Sandia National Laboratories. To assess the extent to which NNSA has implemented contingency and disaster recovery planning and testing for its classified supercomputing assets, we examined contingency and disaster recovery planning controls for the systems within the classified supercomputing environment that are critical to NNSA's achievement of its nuclear weapons mission. We collected and reviewed policies, procedures, and guidelines from the National Institute of Standards and Technology, the Committee on National Security Systems, the Department of Energy, and NNSA. We also reviewed contingency plans and business impact analyses provided by the weapons laboratories and compared them to federal guidelines. We interviewed NNSA and laboratory officials to determine whether they had documented critical system, data, and supporting resources and whether contingency plans had been tested. Further, we interviewed NNSA officials to determine to what extent they have provided specific guidance and oversight for the laboratories to ensure that contingency and disaster recovery planning requirements are being met. To determine the extent to which the three weapons laboratories have the ability to share supercomputing capacity for backup and recovery operations, we visited each weapons laboratory and gained an understanding of the overall classified supercomputing infrastructure and identified interconnectivity and control points. We performed technical assessments of supercomputing capabilities at each weapons laboratory, including each laboratory's ability to share supercomputing capacity under normal operating conditions. We reviewed the weapons laboratories' efforts to determine the minimal supercomputing capacity needed to meet NNSA Stockpile Stewardship Program requirements along with the ability of the weapons laboratories to share supercomputing capacity on an "on-demand" basis, including the use of advanced architecture systems. In addition, we obtained documents describing the supercomputing system environment as well as capacity information, along with the views of officials from NNSA and the three weapons laboratories. To assess the extent to which NNSA tracks costs for ensuring contingency and disaster recovery planning for classified supercomputing assets, we interviewed NNSA and weapons laboratory officials to determine how expenditures were tracked for contingency and disaster recovery planning of the classified supercomputing systems at each of the laboratories. We also requested the amount of funds NNSA obligated to the three weapons laboratories, and the amount of funds the laboratories spent, in implementing NNSA's classified supercomputing capabilities from fiscal years 2007 through 2009. Further, we interviewed NNSA and laboratory officials to determine how they projected future cost estimates for ensuring the recovery of these assets for fiscal years 2011 through 2014. To assess the reliability of data provided, we reviewed (1) the fiscal year 2009 financial statement audit for the system and (2) responses NNSA provided to questions about processes and procedures for ensuring the accuracy and completeness of data. Based on this information, we determined the data are sufficiently reliable for the purposes of this report. We conducted this performance audit from December 2009 through December 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: NNSA Annual Obligations for Its Advanced Simulation and Computing Program, Fiscal Years 2007 through 2009: NNSA reported obligating approximately $1.7 billion from fiscal years 2007 through 2009 to support Advanced Simulation and Computing (ASC) program activities at the three weapons laboratories. The $1.7 billion was used mainly for three efforts: Weapons codes and models. This effort is intended to develop and improve weapons simulation codes and models for predicting the behavior of weapons systems and devices in the nuclear stockpile. Computational systems and software environment. This effort is intended to provide ASC users a stable, seamless computing environment for ASC-deployed platforms. It is responsible for procuring, delivering, and deploying ASC computational systems and user environments via technology development and integration across the three weapons laboratories. Facility operations and user support. This effort is intended to provide both the necessary physical facility and operational support for reliable supercomputing and storage environments, as well as a suite of user services for effective use of the three weapons laboratories' computing resources. Facility operations cover physical space, power and other utility infrastructure, and local-and wide-area networking, as well as system administration, cyber security, and operations services for ongoing support. The user support function includes planning, development, integration and deployment, continuing product support, and quality and reliability activity collaborations. Figure 4 depicts NNSA's annual obligations for each of the three efforts from fiscal years 2007 through 2009. Figure 4: Annual Obligations for NNSA's Advanced Simulation and Computing Program, Fiscal Years 2007 through 2009: [Refer to PDF for image: stacked vertical bar graph] Fiscal year: 2007; Weapons codes and models: $280 million; Computational systems and software environment: $183 million; Facility operations and user support: $128 million; Total: $591 million. Fiscal year: 2008; Weapons codes and models: $249 million; Computational systems and software environment: $183 million; Facility operations and user support: $114 million; Total: $546 million. Fiscal year: 2009; Weapons codes and models: $221 million; Computational systems and software environment: $161 million; Facility operations and user support: $147 million; Total: $529 million. Source: GAO analysis of data provided by NNSA. [End of figure] As shown in figure 4, NNSA annual obligations for its classified supercomputing operations decreased from about $591 million to $529 million between fiscal years 2007 and 2009. The largest obligation for the classified supercomputing program was for weapons codes and models, which accounted for approximately $750 million (or 45 percent) of total obligations. Obligations for computational systems and software environment accounted for approximately $527 million (or 32 percent) of total obligations. For the period, obligations for this effort decreased from $183 million to $161 million. The facility operations and user support activities, which includes, among other things, expenditures for contingency and disaster recovery planning, accounted for $390 million (or 23 percent) of total obligations over the period. These obligations ranged from $114 million to $147 million for the 3 fiscal years. [End of section] Appendix III: Comments from the National Nuclear Security Administration: Department of Energy: National Nuclear Security Administration: Washington, DC 20585: November 23, 2010: Mr. Gregory C. Wilshusen: Director: Information Security Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Wilshusen: The National Nuclear Security Administration (NNSA) appreciates the opportunity to review the Government Accountability Office's (GAO) draft report, GAO-11-67, Information Security: National Nuclear Security Administration Needs to Improve Contingency Planning for Its Classified Supercomputing Operations. I understand the House Committee on Energy and Commerce requested GAO to assess various aspects of NNSA's Continuity of Operations Program to ensure that, in case of service disruptions, the three weapons laboratories can maintain the computer simulation capabilities needed to meet nuclear weapons assessment and certification requirements. Specifically, GAO identified, (1) whether NNSA has Continuity of Operations planning and testing procedures in place across the classified supercomputing environment of the three weapons laboratories; (2) whether the weapons laboratories are able to share capacity for backup and recovery operations; and (3) the past, present and future resources needed to maintain supercomputing capabilities. We are pleased that GAO recognizes the importance of the simulation capabilities of NNSA's supercomputers to address stockpile stewardship and other national security matters. While the draft report implies, without explicitly stating, that the timeframe for reconstitution of supercomputing assets should be similar to that required for Continuity of Operations for national command and control, major financial systems, and health and emergency services, that time urgency is not consistent with existing policies for the recovery of research and development capabilities, nor should it be. Nevertheless, we agree that improvements can be made in contingency and disaster recovery planning for supercomputing operations. After careful review, additional time is required to establish further plans and schedule solutions for addressing the GAO's recommendations. NNSA will provide a more detailed response to the recommendations when the final report is issued. However, we are providing a summary of responses to the recommendations presented in the draft report. Recommendation 1: Develop business impact analyses that, among other things, (I) identify and prioritize critical systems, data, and supporting resources, (2) identiI5, allowable outage times and impacts for classified supercomputing capabilities, and (3) identify recovery priorities and strategies. Concur: NNSA will leverage current Business Impact Analysis (BIA) activities underway at Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories and perform a national level BIA to provide a consistent assessment across the laboratories for classified supercomputing. Recommendation 2: Develop and implement comprehensive contingency and disaster recovery plans for all classified supercomputing systems that identify how each weapons laboratory's classified supercomputing capabilities will be recovered following service disruptions. Concur: NNSA will develop appropriate plans based on the assessment results of the BIA performed per the GAO's first recommendation. Recommendation 3: Conduct contingency plan testing. Concur: NNSA will conduct contingency plan testing according to contingency and disaster recovery plans to be implemented based on the assessment results of the BIA performed per the GAO's first recommendation. Recommendation 4: Classified supercomputing capacity to ensure this capability will work in the event of unexpected service disruptions. Concur: NNSA will test the three weapons laboratories' ability to share classified capacity supercomputers according to contingency and disaster recovery plans to be implemented based on the assessment results of the BIA. Recommendation 5: Document an agency-wide means for reprioritizing the workload across NNSA's classified supercomputing systems should a disruption occur. Concur: The NNSA will adapt, apply and exercise procedures that are routinely being used for prioritizing workload in capability computing campaigns for use in contingencies and disasters. Recommendation 6: Clearly define the oversight responsibilities of the NNSA ASC program office and the NNSA Office of the Chief Information Officer, as they relate to contingency and disaster recovery planning for NNSA's classified supercomputing operations. Concur: In general, the NNSA Office of the Chief Information Officer provides policy and guidance and the Office of Advanced Simulation and Computing (ASC) has responsibilities for execution. Oversight responsibilities will be clearly defined through the BIA and development and implementation of the contingency and disaster recovery plans. Recommendation 7: Identify, assess, and communicate the minimum classified supercomputing capacity needed to meet Stockpile Stewardship requirements in the event of a service disruption. Nonconcur: This recommendation does not take into account that capacity and capability systems serve different functions under different cost of ownership models, and consequently are procured and managed differently. The ASC program has deployed its supercomputing assets to mitigate single site failures. For the future, the program will enhance capacity sizing actions to account for contingency and disaster recovery operations when planning host sites for capacity computing capabilities. We will conduct a BIA assessment recognizing the differences and build appropriate contingency strategies for both classes of computing. Recommendation 8: Develop, document, and implement a process that identifies and tracks expenditures for contingency and disaster recovery planning for NNSA 's classified supercomputing assets. Nonconcur: Almost all classified supercomputing contingency and disaster recovery planning leverages computing resources and activities funded as part of a production simulation environment for weapons designers and engineers. These expenses are integral to ASC's Facility Operations and User Support Program element and tracking them separately would not add significant value to managing contingency and disaster recovery. Recommendation 9: Develop and document the total anticipated costs for contingency and disaster recovery planning of NNSA 's classified supercomputing assets, which includes the replacement costs for these assets. Nonconcur: As stated in the response to Recommendation 8, these expenses are integral to ASC's Facility Operations and User Support program element and tracking them separately would not add significant value to managing contingency and disaster recovery. If you have any questions related to this response, please contact JoAnne Parker, Director, Office of Internal Controls, at 202-586-1913. Sincerely, Signed by: Gerald L. Talbot, Jr. Associate Administrator for Management and Administration: cc: Acting Chief Information Officer: Deputy Administrator for Defense Programs: [End of section] Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Gene Aloise (202) 512-3841 or aloisee@gao.gov Nabajyoti Barkakati (202) 512-6415 or barkakatin@gao.gov Gregory C. Wilshusen (202) 512- 6244 or wilshuseng@gao.gov: Staff Acknowledgments: In addition to the individuals named above, Glen Levis, Edward M. Glagola, Jr., and Jeffrey Knott (Assistant Directors); and Preston S. Heard, Jennifer R. Franks, Kevin Metcalfe, and Zsaroq Powe were key contributors to this report. Neil Doherty, Nancy Glover, Franklin Jackson, and Jonathan Kucskar also made key contributions to this report. [End of section] Footnotes: [1] NNSA was established in 2000 as a separately organized agency within the Department of Energy (DOE) and is responsible for the nation's nuclear weapons, nonproliferation, and naval reactors programs. [2] For nearly half a century, the United States' nuclear program was spearheaded by underground nuclear testing and never had to rely on weapon systems that had exceeded their design life times. The United States last produced a nuclear weapon in 1991 and performed its last underground nuclear test in 1992. [3] The National Defense Authorization Act for Fiscal Year 1994, Pub. L. No. 103-160, § 3138 (1993), directed DOE to establish the Stockpile Stewardship Program. In the absence of underground nuclear testing, the program encompasses a broad range of activities to increase understanding of the basic phenomena associated with nuclear weapons, provide better predictive understanding of the safety and reliability of weapons, and ensure a strong scientific and technical basis for future nuclear weapons policy objectives. The Stockpile Stewardship Program is carried out through the nuclear weapons complex, which includes three nuclear weapons laboratories. [4] Continuity of operations focuses on restoring an organization's mission-essential functions at an alternate site and performing those functions for a short period of time before returning to normal operations. Contingency and disaster recovery planning include a broad scope of activities designed to sustain and recover critical information and information system services for a range of potential service disruptions. Contingency and disaster recovery planning components may include the relocation of information systems and operations to an alternate site, recovery of information system functions using alternate equipment, or the performance of information system functions using alternative methods. For the purposes of this report, the term contingency and disaster recovery planning refer to the interim measures NNSA should use to recover information system services after an unexpected service disruption. [5] Los Alamos is managed and operated by Los Alamos National Security, LLC, which is a consortium of contractors that includes Bechtel National, the University of California, the Babcock and Wilcox Company, and the Washington Division of URS. Livermore is managed and operated by Lawrence Livermore National Security, LLC, which is comprised of a corporate management team that includes Bechtel National, the University of California, the Babcock and Wilcox Company, and the Washington Division of URS. Sandia is managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation. [6] FLOPS are a measure of a supercomputing system's performance. Floating-point performance is the rate at which a computer executes floating-point operations. [7] For additional information regarding budgetary information for the classified supercomputing program from fiscal years 2007 through 2009, see appendix II. [8] 44 U.S.C. § 3544(b); FISMA was enacted as title III, E-Government Act of 2002, Pub. L. No. 107-347, 116 Stat. 2899, 2946 (Dec. 17, 2002). [9] For the purposes of this report, we will refer to "continuity of operations procedures for information systems" as contingency and disaster recovery planning. [10] NIST Special Publication 800-34, Contingency Planning Guide for Information Technology Systems (Washington, D.C.: June 2002) and NIST Special Publication 800-53 Revision 3, Recommended Security Controls for Federal Information Systems and Organizations (Gaithersburg, Md.: August 2009). [11] Formerly known as the National Security Telecommunications and Information Systems Security Committee, CNSS provides a forum for the discussion of policy issues, sets national policy, and provides direction, operational procedures, and guidance for the security of national security systems. DOD chairs the committee under the authorities established by National Security Directive 42, National Policy for the Security of National Security Telecommunications and Information Systems, issued in July 1990. This directive designates the Secretary of Defense and the Director of the National Security Agency as the Executive Agent and National Manager, respectively. The committee has 21 voting representatives from various departments and agencies, including the Department of Energy. [12] National security systems include any information system used or operated by an agency, or by a contractor of an agency, that processes, stores, or transmits national security information. They do not include those systems used for routine administrative and business applications. [13] CNSS Instruction 1253 provides federal government departments, agencies, bureaus, and offices with a process for security categorization of national security systems that collect, generate, process, store, display, transmit, or receive national security information. In addition, this instruction serves as a companion document to NIST Special Publication 800-53, Revision 3. [14] Although NIST guidelines note they shall not apply to national security systems without the express approval of appropriate federal officials exercising policy authority over such systems, CNSS instructions, as well as DOE and NNSA policies for national security systems, refer to the NIST guidelines as being applicable. [15] A contingency plan is designed to maintain or restore business operations, including computer operations, possibly at an alternate location in the event of emergencies, system failures, or disaster. A disaster recovery plan is a written plan for processing critical applications in the event of a major hardware or software failure or destruction of facilities. [16] Outage impacts and allowable outage times enable the organization to develop and prioritize recovery strategies that personnel will implement during contingency plan activation. The effects of the outage may be tracked over time, which will enable the organization to identify the maximum allowable time that a resource may be unavailable before it inhibits the performance of an essential function. The effects of the outage can also be tracked across related resources, identifying any cascading effects that may occur as an effect of a service disruption. [17] The Department of Energy defines "mission critical" as an information system that supports an organization's core missions and goals, and "mission-essential (or business essential)" as an information system whose failure would not preclude organizations from accomplishing core business functions in the long term. [18] The Capability Computing Campaign includes a committee made up of staff from the NNSA ASC program office, as well as ASC executives located at the laboratories at Los Alamos, Livermore, and Sandia. [19] Total usable supercomputing capacity includes the supercomputers that have the ability to run all weapons program codes and could be used in the event of a service disruption, and includes capacity and capability systems. [20] The term "on demand" is defined as the ability to move an application (simulation program/code) from one supercomputer to a different supercomputer at a different physical facility and use the existing computational resources without the need for major modifications. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: