This is the accessible text file for GAO report number GAO-13-4 
entitled 'James Webb Space Telescope: Actions Needed to Improve Cost 
Estimate and Oversight of Test and Integration' which was released on 
December 3, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

December 2012: 

James Webb Space Telescope: 

Actions Needed to Improve Cost Estimate and Oversight of Test and 
Integration: 

GAO-13-4: 

GAO Highlights: 

Highlights of GAO-13-4, a report to congressional committees. 

Why GAO Did This Study: 

JWST is one of NASA’s most expensive and technologically advanced 
science projects, intended to advance understanding of the origin of 
the universe. In 2011, JWST was rebaselined with a life cycle cost 
estimate of $8.8 billion and a launch readiness date in October 2018—
almost nine times the cost and more than a decade later than 
originally projected in 1999. Concern about the magnitude of JWST's 
cost increase and schedule delay and their effects on NASA’s progress 
on other high-priority missions led conferees for the Consolidated and 
Further Continuing Appropriations Act, 2012, to direct GAO to report 
on the project. Specifically, GAO assessed (1) the extent to which NASA’
s revised cost and schedule estimates are reliable based on best 
practices, (2) the major risks and technological challenges JWST 
faces, and (3) the extent to which NASA has improved oversight of 
JWST. To do this, GAO compared NASA’s revised cost and schedule 
estimates with best practice criteria, reviewed relevant contractor 
and NASA documents, and interviewed project and contractor officials. 

What GAO Found: 

The National Aeronautics and Space Administration (NASA) has provided 
significantly more time and money to the James Webb Space Telescope 
(JWST) than previously planned and expressed high confidence in the 
project's new baselines. Its current cost estimate reflects some 
features of best practices for developing reliable and credible 
estimates. For example, the estimate substantially meets one of four 
cost characteristics—comprehensive—that GAO looks for in a reliable 
cost estimate, in part because all life cycle costs were included. The 
estimate, however, only partially met the other three characteristics—
well documented, accurate, and credible—which detracts from its 
reliability. For example, the estimate's accuracy, and therefore the 
confidence level assigned to the estimate, was lessened by the summary 
schedule used for the joint cost and schedule risk analysis because it 
did not provide enough detail to determine how risks were applied to 
critical project activities. The estimate's credibility was also 
lessened because officials did not perform a sensitivity analysis that 
would have identified key drivers of costs, such as workforce size. 
Program officials believe that it would have been difficult to fully 
address all best practice characteristics. GAO believes there is time 
to improve the estimate and enhance the prospects for delivering the 
project according to plan. 

Project officials report that the JWST schedule has 14 months of 
reserve, which meets Goddard guidance for schedule reserve; however, 
only 7 of the 14 months are likely to be available for the last three 
of JWST’s five complex integration and test efforts. GAO's prior work 
shows that the integration and test phases are where problems are 
commonly found and schedules tend to slip. Given that JWST has a 
challenging integration and test schedule, this could particularly be 
likely. The project has made some significant progress in the past 
year, notably successfully completing development of the 18 primary 
mirror segments—considered JWST's top technical risk. Nevertheless, 
ongoing challenges are indicative of the kinds of issues that can 
require significant effort to address. For example, instrument 
challenges have delayed the first integration and test effort. In 
addition, key long-term risks on subsystems with a significant amount 
of work remaining will not be retired until 2016. Currently, NASA’s 
plan for project oversight calls for one independent mission-level 
system integration review about 13 months before launch. While this is 
consistent with what NASA requires for its projects, this approach may 
not be sufficient for a project as complex as JWST. 

JWST has taken several steps to improve communications and oversight 
of the project and its contractors—such as taking over responsibility 
for mission systems engineering from the prime contractor; instituting 
meetings that include various levels of NASA, contractor, and 
subcontractor management; and implementing a new risk management 
system to allow for better tracking of risks. The enhancements to the 
oversight of the project are steps in the right direction, but it will 
take time to assess their effectiveness and ensure that the efforts 
are sustained by the project in the future. Reductions in travel 
budgets, however, could require the project to adjust the oversight 
approach that was adopted as a result of the replan. Additional 
reductions in travel budgets are anticipated in future years, but 
officials do not have a plan to address such reductions and their 
potential impact on continuing the current oversight approach. 

What GAO Recommends: 

GAO recommends NASA take six actions including, among others, to take 
steps to improve its cost estimate; to conduct an additional, earlier 
independent review of test and integration activities; and to develop 
a long-term oversight plan that anticipates planned travel budget 
reductions. In commenting on a draft of this report, NASA fully or 
partially concurred with the recommendations citing ongoing efforts, 
but GAO believes some do not go far enough. 

View [hyperlink, http://www.gao.gov/products/GAO-13-4]. For more 
information, contact Cristina Chaplain at (202) 512-4841 or 
chaplainc@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Revised JWST Cost Estimate Is Not Fully Consistent with Best Practices 
for Reliable and Credible Estimates and the Integrated Master Schedule 
Is Not Finalized: 

Technically Challenging JWST Project Lacks the Schedule Reserve 
Flexibility and Commensurate Oversight of Integration and Test Efforts: 

JWST Project Has Taken Steps to Enhance Communications with and 
Oversight of Its Contractors: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Subsystems of the JWST Observatory: 

Appendix III: Our Evaluation of JWST's Cost Estimate Process: 

Appendix IV: Comments from the National Aeronautics and Space 
Administration: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Issues with JWST Science Instruments Included in ISIM: 

Table 2: ICRP Recommendations Concerning Oversight and Communications 
and the Actions Taken by NASA: 

Table 3: NASA Office of Evaluation Review Team Concerns: 

Table 4: Summary Assessment of JWST Cost Estimate Compared to Best 
Practices: 

Figures: 

Figure 1: Subsystems of the JWST: 

Figure 2: Organizational Chart for the JWST Program: 

Figure 3: Summary of Results of GAO Assessment of JWST Cost Estimate 
Based on Best Practices Criteria: 

Figure 4: Project Integration and Test Time Line: 

Figure 5: Percentage of Budgeted Development Work Remaining for Major 
Subsystems: 

Figure 6: Current Mass Estimate for Spacecraft Subsystem: 

Figure 7: Changes in ISIM Schedule Since the Project Replan: 

Figure 8: Subsystems of the JWST: Interactive Information: 

Abbreviations: 

CSA: Canadian Space Agency: 

ESA: European Space Agency: 

FGS: Fine Guidance Sensor: 

ICRP: Independent Comprehensive Review Panel: 

ISIM: Integrated Science Instrument Module: 

JCL: Joint Cost and Schedule Confidence Level: 

JWST: James Webb Space Telescope: 

KDP: key decision point: 

MIRI: Mid-Infrared Instrument: 

NASA: National Aeronautics and Space Administration: 

NIRCam: Near-Infrared Camera: 

NIRISS: Near-Infrared Imager and Slitless Spectrograph: 

NIRSpec: Near-Infrared Spectrograph: 

NPR: NASA Procedural Requirements: 

OTE: Optical Telescope Element: 

OTIS: Optical Telescope Element and Integrated Science Instrument 
Module: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

December 3, 2012: 

Congressional Committees: 

The James Webb Space Telescope (JWST) is one of the National 
Aeronautics and Space Administration's (NASA) most expensive and 
technologically advanced science projects. It is intended to 
revolutionize our understanding of star and planet formation and 
advance the search for the origins of our universe. The project has 
been hampered by ineffective management, unrealistic estimates of 
costs, and insufficient resources, and is projected to cost 
significantly more and to take much longer to complete than expected 
when it was authorized to proceed in 1999. In 2011, JWST was 
rebaselined with a lifecycle cost estimate of $8.8 billion and a 
launch date in October 2018, which is almost nine times the cost and 
more than a decade later than originally projected. The magnitude of 
JWST's cost increase and schedule delay has and will continue to have 
a major effect on NASA's ability to implement or begin high-priority 
science missions for years to come. 

NASA has expressed high confidence in being able to execute the 
project within its new cost and schedule baselines, and officials 
report that the project is being executed to its revised estimates in 
the year since the rebaseline. The project, however, is less than 
halfway through its development with known and unknown technical risks 
and challenges remaining to be addressed. The on-time and on-budget 
delivery of JWST is a high congressional priority. In November 2011, 
the Joint Explanatory Statement of the Committee of Conference for the 
Consolidated and Further Continuing Appropriations Act, 2012, 
recommended that JWST's formulation and development costs, not 
including the cost to operate the mission after launch, be no more 
than $8 billion. The conferees also directed GAO to assess the program 
annually and to report to the Committees on Appropriations on key 
issues relating to program and risk management, achievement of cost 
and schedule goals, and program technical status. This report responds 
to that direction. Specifically, we assessed (1) the extent to which 
NASA's revised cost and schedule estimates are reliable based on GAO 
best practices, (2) the major risks and technological challenges the 
JWST project faces, and (3) the extent to which NASA has improved the 
oversight of the JWST project. 

Our approach included an examination of the revised cost and schedule 
estimates NASA developed during the replanning process using GAO's 
best practice criteria.[Footnote 1] This included review of 
documentation of the inputs that supported the detailed cost and 
schedule estimates used by decision-makers to rebaseline the program 
and interviews with project officials involved in developing the 
estimates. We also reviewed the proposal from the JWST prime 
contractor used as input to the replan and interviewed contractor 
officials. We did not assess the most current JWST integrated master 
schedule because of ongoing negotiations for contract modifications 
associated with the replan. To identify the major short and long term 
risks and technological challenges facing the project, we interviewed 
project officials for the major subsystems as well as contractor 
officials. We also reviewed the project's risk database, monthly 
status reviews, and other documentation provided by the project 
officials. To understand the steps NASA implemented to improve the 
oversight being performed on the project, we interviewed project and 
contractor officials and reviewed documentation from independent 
reviews prior to the replan and actions taken by NASA in response. We 
also interviewed the Director of NASA's Office of Evaluation 
concerning a recent independent review of the JWST project performed 
by his office. We took appropriate steps to address data reliability, 
such as clarifying data discrepancies and corroborating NASA-generated 
data with other sources where applicable. NASA provided comments and 
technical clarifications on a draft of this report, which were 
incorporated as appropriate. Appendix I contains detailed information 
on our scope and methodology. 

We conducted this performance audit from February 2012 to December 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

Initially referred to as the "Next Generation Space Telescope," JWST 
is a large deployable, infrared-optimized space telescope intended to 
be the successor to the aging Hubble Space Telescope. JWST is designed 
to be a 5-year mission to find the first stars and trace the evolution 
of galaxies from their beginning to their current formation, and is 
intended to operate in an orbit approximately 1.5 million kilometers--
or 1 million miles--from the Earth. In a 2001 decadal survey, the 
National Research Council rated the JWST as the top-priority new 
initiative for astronomy and physics. With its 6.5-meter primary 
mirror, JWST will be able to operate at 100 times the sensitivity of 
the Hubble Space Telescope. A tennis-court-sized sunshield will 
protect the mirrors and instruments from the sun's heat to allow the 
JWST to look at very faint infrared sources. The Hubble Space 
Telescope operates primarily in the visible and ultraviolet regions. 
[Footnote 2] 

History of Cost Growth and Schedule Delays: 

JWST has experienced significant increases to project costs and 
schedule delays. Prior to being approved for development, cost 
estimates of the project ranged from $1 billion to $3.5 billion with 
expected launch dates ranging from 2007 to 2011. In March 2005, NASA 
increased the JWST's life-cycle cost estimate to $4.5 billion and 
slipped the launch date to 2013. We reported in 2006 that about half 
of the cost growth was due to schedule slippage--a 1-year schedule 
slip because of a delay in the decision to use a European Space Agency-
supplied Ariane 5 launch vehicle and an additional 10-month slip 
caused by budget profile limitations in fiscal years 2006 and 2007. 
More than a third of the cost increase was caused by requirements and 
other changes. An increase in the program's contingency funding 
accounted for the remainder--about 12 percent--of the growth. NASA 
Headquarters chartered an Independent Review Team to evaluate the 
project that same year. In April 2006, the review team's assessment 
confirmed that the program's technical content was complete and sound, 
but expressed concern over the project's contingency reserve funding--
funding used to mitigate issues that arise but which were previously 
unknown--reporting that it was too low and phased in too late in the 
development life cycle. The team reported that for a project as 
complex as the JWST, a 25 to 30 percent total contingency was 
appropriate. At that time, JWST's total contingency was about 19 
percent. The team cautioned that this contingency compromised the 
project's ability to resolve issues, address risk areas, and 
accommodate unknown problems. The team also concluded that the 2013 
launch date was not viable for the project based on its anticipated 
budget. It recommended that before the project was formally approved 
for development and baselined, NASA should take steps to provide the 
JWST project with adequate time-phased reserve funding to secure a 
stable launch date. Additional reserves were added and the project was 
baselined in April 2009 with a life-cycle cost estimate of $4.964 
billion and a launch date in June 2014. 

Shortly after JWST was approved for development and its cost and 
schedule estimates were baselined, project costs continued to 
increase. In 2010, Senator Barbara Mikulski, chair of the Senate 
Committee on Appropriations, Subcommittee on Commerce, Justice, 
Science, and Related Agencies, asked NASA to initiate another 
independent review in response to the project's cost increases and 
reports that the June 2014 launch date was in jeopardy. The 
Independent Comprehensive Review Panel (ICRP) was commissioned by NASA 
and began its review in August 2010. In October 2010, the ICRP issued 
its report and cited several reasons for the project's problems 
including management, budgeting, oversight, governance and 
accountability, and communication issues.[Footnote 3] The panel 
concluded JWST was executing well from a technical standpoint, but 
that the baseline funding did not reflect the most probable cost with 
adequate reserves in each year of project execution, resulting in an 
unexecutable project. The review panel recommended that additional 
resources be considered along with organizational and management 
restructuring. Following this review, the JWST program underwent a 
replan in 2011. In November 2011, the JWST project was reauthorized, 
but not before it was recommended for termination by the House 
Appropriations Committee. On the basis of the replan, NASA announced 
that the project would be rebaselined at $8.835 billion--a 78 percent 
increase to the project's life-cycle cost from the confirmed baseline--
and would launch in October 2018--a delay of 52 months. The revised 
life-cycle cost estimate included 13 months of funded schedule 
reserve. In the President's Fiscal Year 2013 budget request, NASA 
reported a 66 percent joint cost and schedule confidence level 
associated with these estimates. A joint cost and schedule confidence 
level (JCL) is the process NASA uses to assign a percentage to the 
probable success of meeting cost and schedule targets and is part of 
the project's estimating process. 

Current JWST Organization: 

The JWST project is divided into three major segments: the launch 
segment, the ground segment, and the observatory segment. The launch 
segment is primarily provided by the European Space Agency (ESA), 
which is contributing the Ariane 5 launch vehicle and launch site 
operations in French Guiana. The ground segment will be responsible 
for collecting the data obtained by JWST in space and making it usable 
for scientists and researchers. This includes the development of 
software that will translate data into usable formats as well as 
operation of the software once the telescope is in space. The Space 
Telescope Science Institute, operated by the Association of 
Universities for Research in Astronomy (AURA) on a contract awarded by 
NASA, which currently performs science operations for the Hubble Space 
Telescope, is developing the science and operations and flight 
operations center for JWST and will conduct the first 6 months of 
flight and science operations. The NASA contract with the Space 
Telescope Science Institute extends through the first 6 months of JWST 
operations. A contract to manage the long term operations of JWST is 
planned to be awarded approximately 2 years prior to launch. The 
observatory segment will be launched into space and includes five 
major subsystems.[Footnote 4] These subsystems are being developed 
through a mixture of NASA, contractor, and international partner 
efforts. See figure 1. 

Figure 1: Subsystems of the JWST: 

[Refer to PDF for image: Interactive Graphic] 

Rollover the white dots to see description. See appendix II for the 
non-interactive, printer-friendly version. 

Source: GAO analysis; NASA (data and images. 

[End of figure] 

JWST is a single project program reporting directly to the NASA 
Associate Administrator for programmatic oversight and to the 
Associate Administrator for the Science Mission Directorate for 
technical and analysis support.[Footnote 5] Goddard Space Flight 
Center is the NASA center responsible for the management of JWST. See 
figure 2 for the current JWST organizational chart. 

Figure 2: Organizational Chart for the JWST Program: 

[Refer to PDF for image: organizational chart] 

NASA HQ: Associate Administrator: 
* NASA HQ: JWST Program Director: 
- NASA HQ: Science Mission Directorate (programmatic); 
* NASA HQ: JWST Program Manager; 
* Goddard: JWST Program Manager: 
- Goddard Center Director (Execution oversight); 
- Goddard: JWST Project Scientist (programmatic); 
* Goddard: In-house project contributions: 
- ISIM; 
- OTIS; 
- Johnson Space Center Chamber A; 
- Integration & Test. 

Major contractor contributions: 

University of Arkansas: 
* NIRCam; 
* Detectors. 

Northrup Grumman: 
* OT; 
* Spacecraft; 
* Sunshield; 
* Integration & Test. 

Space TelescopeScience Institute: 
* Ground Systems. 

International partner contributions: 

European Space Agency: 
* NIRSpec; 
* Launch Vehicle. 

Canadian Space Agency: 
* FGS/NIRISS. 

European Consortium: 
* MIRI. 

Source: GAO analysis of NASA information. 

[End of figure] 

Revised JWST Cost Estimate Is Not Fully Consistent with Best Practices 
for Reliable and Credible Estimates and the Integrated Master Schedule 
Is Not Finalized: 

Our analysis of JWST's revised cost estimate showed that it is not 
fully consistent with best practices for developing reliable and 
credible estimates, although project officials took some steps in line 
with best practices in the development of the estimate. For example, 
as part of its cost estimation process, the project conducted a joint 
cost and schedule risk analysis, or joint cost and schedule confidence 
level (JCL), which assigned a 66 percent confidence level to the 
estimate.[Footnote 6] In addition, we found that the cost estimate 
included all life cycle costs for the project. Although NASA's methods 
for developing the JWST cost estimate reflect some features of best 
practices, our review of the estimate showed that based on best 
practice criteria, it did not fully meet the four characteristics of a 
reliable estimate. See figure 3. 

Figure 3: Summary of Results of GAO Assessment of JWST Cost Estimate 
Based on Best Practices Criteria: 

[Refer to PDF for image: table] 

Characteristic: Comprehensive; 
Best Practice Criteria: Include costs of the program over its full 
life-cycle, provide a level of detail appropriate to ensure that cost 
elements are neither omitted nor double-counted, and document all cost-
influencing ground rules and assumptions; 
Overall Assessment: Substantially met. 

Characteristic: Well documented; 
Best Practice Criteria: Be supported by detailed documentation 
describing: 
* the purpose of the estimate,
* the program background and system description,
* the scope of the estimate,
* the ground rules and assumptions,
* all data sources,
* estimating methodology and rationale, and, 
* the results of the risk analysis. 
This documentation should show how the data used to derive the 
estimate can be traced back to, and verified against, their sources. 
Overall Assessment: Partially met. 

Characteristic: Accurate; 
Best Practice Criteria: Be based on an assessment of most likely costs 
(adjusted for inflation), documented assumptions, and historical
cost estimates and actual experiences on other comparable programs.
Be checked for accuracy, double counting, and omissions. Be updated to 
reflect any changes; 
Overall Assessment: Partially met. 

Characteristic: Credible; 
Best Practice Criteria: Discuss any limitations of the analysis 
because of uncertainty, or biases surrounding data or assumptions.
Risk and uncertainty analysis should be performed to determine the 
level of risk associated with the estimate.
Have results that are cross-checked against an independent estimate; 
Overall Assessment: Partially met. 

Source: GAO analysis of NASA data. 

Note: Met: NASA provided complete evidence that satisfies the entire 
criterion; Substantially Met: NASA provided evidence that satisfies a 
large portion of the criterion; Partially Met: NASA provided evidence 
that satisfies about half of the criterion; Minimally Met: NASA 
provided evidence that satisfies a small portion of the criterion; and 
Not Met: NASA provided no evidence that satisfies any of the 
criterion. See appendix III for our evaluation of JWST's cost estimate 
process. 

[End of figure] 

Specifically, the project's estimate was found to substantially meet 
the best practice criteria for being comprehensive, and the remaining 
three characteristics of being well documented, accurate, and credible 
were found to be only partially met. For example, the accuracy of the 
cost estimate, and therefore the confidence level assigned to the 
estimate, was lessened by the schedule used in the JCL analysis 
because it prevented us from, among other things, identifying the 
activities that were on the critical path--defined as time associated 
with activities that drive the overall schedule. The credibility of 
the estimate was lessened because project officials did not perform a 
sensitivity analysis that would have identified key drivers of costs, 
such as workforce size. Although NASA is not required to adhere to 
these best practices, our prior work has shown that not following best 
practices for cost estimating can make the cost estimate less 
reliable, putting projects at risk of experiencing cost overruns, 
missed deadlines, and performance shortfalls.[Footnote 7] The best 
practices stem from practices federal cost estimating organizations 
and industry use to develop and maintain reliable cost estimates, 
including the Department of Defense and NASA. According to program 
officials, it would have been difficult, if not impossible, for the 
project to have met all of the best practice criteria given the 
complexity of the project and that some elements of the project are 
quite mature in their development. Instead, the program manager stated 
that the project followed a tailored process to develop the cost 
estimate that was appropriate for the project. Furthermore, officials 
report the project is currently meeting a majority of its milestones 
and executing as planned to the revised estimates for the JWST. 

Comprehensive: The JWST cost estimate substantially met the criteria 
for being comprehensive because it included all life-cycle costs, 
documented cost-influencing ground rules and assumptions, and used a 
work breakdown structure, or the structure used to define in detail 
the work necessary to accomplish its objectives, that is detailed and 
traceable to the cost of each work element and the contract statement 
of work.[Footnote 8] The cost estimate is not considered fully 
comprehensive, however, because we were unable to align the schedule 
to the cost estimate and could not align all of the subcontractor work 
to the work breakdown structure of the cost estimate. Although NASA 
officials believe that the project has visibility into, and can map, 
subcontractor activities, we were unable to confirm that cost elements 
were neither omitted nor double counted. In addition, we were unable 
to map the project's earned value management data to the work 
breakdown structure because information between the two was not 
compatible.[Footnote 9] Finally, although the project outlined and 
documented the ground rules and assumptions, we were unable to 
determine whether risks associated with any assumptions were 
identified and traced to specific elements. 

Well documented: The JWST cost estimate only partially met the 
criteria for being well documented because it did not include a step-
by-step description of how the estimate was developed, the raw data 
used to develop the estimate, or the calculations and estimating 
methodology for specific cost elements of the work breakdown 
structure. Without good documentation, a cost analyst unfamiliar with 
the program will not be able to replicate the estimate, because he or 
she will not understand the logic behind it. Good documentation, for 
example, assists management and oversight in assessing the credibility 
of the estimate, helps to keep a history of reasons for cost changes 
and to record lessons learned, defines the scope of the analysis, and 
answers questions about the approach or data used to create the 
estimate. Project documentation, however, does provide evidence that 
NASA management reviewed and accepted the cost estimate because 
managers were briefed on the technical aspects of the estimate and 
were provided an overview of the joint cost and schedule risk analysis 
that was conducted. 

Accurate: The JWST cost estimate only partially met the criteria for 
being accurate because the projected costs of schedule reserve did not 
reflect actual data, the summary schedule used to derive the JCL 
prevented us from sufficiently understanding how risks were 
incorporated, and the project did not provide evidence that it 
regularly updates the estimate or plans to conduct another JCL. For 
example, using historical actual cost data from Northrop Grumman, we 
estimated that 13 months of schedule reserve is likely to be $204 
million instead of NASA's estimate of $121 million--a potential 
underestimation of 69 percent related to the schedule reserve. Project 
officials, however, believe they have adequate reserves available to 
offset any underestimation. In addition, the summary schedule the 
project used as an input to the JCL, although deemed acceptable by 
NASA, contained many long-duration activities, some with 1,000 days or 
more. Because of these long durations in the summary schedule used for 
the JCL, the lack of detail prevented us from identifying the 
activities that were on the critical path, as well as which risks were 
applied to remaining activities.[Footnote 10] As a result, there is no 
way to ensure that risks were appropriately assigned to activities in 
the schedule to account for the impact of the risks during the JCL 
analysis. 

Finally, it was unclear whether the cost estimate was regularly 
updated to reflect material changes in actual costs and in the project 
itself, such as when schedules or other assumptions change, due to a 
lack of detailed documentation for the cost estimate. Project 
officials stated that in keeping with NASA policy they do not plan, 
nor are they required, to conduct another JCL analysis. GAO's cost 
estimating best practices call for estimates to be continually updated 
through the life of the project, ideally every month as actual costs 
are reported in earned value management reports, and that a risk 
analysis and risk simulation exercise--like the JCL analysis--be 
conducted periodically through the life of the program, as risks can 
materialize or change throughout the life of a project. Unless 
properly updated on a regular basis, the cost estimate cannot provide 
decision makers with accurate information to assess the current status 
of the project. NASA officials state that the life-cycle cost estimate 
is updated annually for the budgeting process, and that historical 
records such as earned value data were used to develop the estimate. 
They also stated that this information is updated in several different 
documents being provided to management; however, we were unable to 
determine how this information was used in updating the cost estimate 
on a regular basis. 

Credible: The JWST cost estimate only partially met the criteria for 
being credible because project officials did not adequately test and 
verify the reasonableness of the cost estimate and the schedule used 
in conducting the JCL did not have a valid critical path and contained 
durations that were too long to properly account for risks. For 
example, project officials said they did not perform a sensitivity 
analysis for the cost estimate. A sensitivity analysis identifies key 
elements that drive cost and permits analysis of different outcomes 
and is often used to develop cost ranges and risk reserves. NASA 
officials stated that the largest cost driver for the JWST project is 
the size of the workforce, which could have been subjected to a 
sensitivity analysis; yet, the cost model did not include a 
sensitivity analysis that would show how staff increasing or 
decreasing over time affects cost. In addition, NASA officials believe 
that all risks were sufficiently accounted for when conducting the 
JCL, however, the software used to conduct the JCL analysis does not 
recognize certain risks that officials had placed on activities in the 
project schedule and, therefore, some risks were discarded during the 
simulation. 

The schedule used to conduct the JCL was also summarized at such a 
high level that the durations were too long to effectively model the 
risks. For example, one of the activities that drove the launch date 
was over 4 years in duration and should have been broken down further 
prior to conducting the simulation. Moreover, the critical path in the 
JCL schedule consisted of six level of effort activities all with the 
same duration of 2,238 days in length.[Footnote 11] Level of effort 
activities should never be on the critical path because support 
activities should never drive any milestone finish date. As a result 
of the schedule used in the JCL not fully meeting best practices, we 
question the results of the analysis. Furthermore, the risk of having 
to carry the JWST workforce to support the project if delayed was not 
included since a sensitivity analysis was not performed. Project 
officials report that, instead, risk associated with the workforce was 
factored in when establishing cost reserves. 

In addition, project officials did not commission an independent cost 
estimate, which is considered one of the best and most reliable 
estimate validation methods because it shows whether other estimating 
procedures produce similar results, and it provides an independent 
view of expected program costs that tests the program office's 
estimate for reasonableness.[Footnote 12] An estimate that has not 
been reconciled with an independent cost estimate has an increased 
risk of being underfunded because the independent cost estimate 
provides an objective and unbiased assessment of whether the project 
estimate can be achieved. Notably, however, project officials provided 
evidence that an independent cost assessment was done for the project 
at the request of the JWST Standing Review Board, the independent 
review team for the project, and the assessment was within 2 percent 
of the project's estimated cost for the rebaseline. Project officials 
contend that the approach they used in developing the life-cycle cost 
estimate for the project is more accurate than the types of approaches 
often used to develop and independent estimate. 

Complete Schedule Not Finalized; Full Assessment of Schedule Needed: 

We did not conduct a full schedule assessment to determine the 
reliability of the revised schedule based on best practices due to on-
going contract negotiations.[Footnote 13] The project has an 
integrated master schedule developed as part of the replan; however, 
it is not finalized because major contract modifications have yet to 
be negotiated and definitized.[Footnote 14] Specifically, the 
modification to the Northrop Grumman contract, which accounts for 
approximately 40 percent of the total project cost and spans much of 
the work on the spacecraft and OTE, remains undefinitized more than a 
year after the project was rebaselined. Once the project completes 
negotiations for the contract modification and all schedule dates are 
set, the project can then have a measurable integrated master 
schedule. Project officials stated that the negotiation process and 
updating of associated schedules are planned to be complete in January 
2013 for the Northrop Grumman contract modification--a year after 
submission of the latest update to its proposal for the replan. The 
project also reported that multiple audits of the proposals submitted 
by Northrop Grumman and its subcontractor by the Defense Contract 
Audit Agency have delayed definitization. Negotiations for the 
modification to NASA's contract with the Space Telescope Science 
Institute to incorporate the October 2018 launch readiness date are 
not scheduled to be complete until spring 2013. 

Once all the contracts have been definitized and the project's 
integrated master schedule is baselined, we plan to conduct a 
comprehensive best practices assessment of the reliability of the 
project's schedule estimates. 

Technically Challenging JWST Project Lacks the Schedule Reserve 
Flexibility and Commensurate Oversight of Integration and Test Efforts: 

Project officials report that the JWST schedule has 14 months of 
reserve, which meets Goddard guidance for schedule reserve; however, 
only 7 of the 14 months are likely to be available for the last three 
of JWST's five complex integration and test efforts. GAO's prior work 
shows that it is during integration and test where problems are 
commonly found and schedules tend to slip. Given that JWST has a 
challenging integration and test schedule, this could particularly be 
the case. The project has made some significant progress in the past 
year, notably successfully completing development of the 18 primary 
mirror segments--considered JWST's top technical risk. Nevertheless, 
ongoing challenges are indicative of the kinds of issues that can 
require a significant amount of effort to address. For example, 
instrument challenges have delayed the first integration and test 
effort. In addition, key long-term risks on subsystems with a 
significant amount of work remaining will not be retired until 2016. 
Currently, NASA's plan for project oversight calls for one independent 
system integration review about 13 months before launch. While this is 
consistent with what NASA requires for its projects, this approach may 
not be sufficient for a project as complex as JWST. As a result, the 
current plan may be inadequate to ensure key technical and management 
issues are identified early enough to be addressed within the current 
integration and test phase schedule. 

Test and Integration Schedule Lacks Schedule Reserve Late in the 
Process: 

JWST has a complex and lengthy integration and test phase, which 
includes five major integration and test efforts--ISIM, OTE, OTIS, 
spacecraft, and observatory. See figure 4 for the project reported 
dates for the major integration and test efforts and the schedule 
reserve allocated for each effort. 

Figure 4: Project Integration and Test Time Line: 

[Refer to PDF for image: time line] 

July 2011 - October 2015: 
ISIM I&T: integrates and test the 4 science instruments. 

March 2011 - October 2015: 
OTE I&T: integrates and tests the opt and support structure; 
Schedule reserve: 7 months. 

May 2016 - June 2017: 
OTIS I&T: integrates the ISIM and the OTE; 
Schedule reserve: 3 months. 

April 2016 - July 2017: 
Spacecraft I&T: integrates the spacecraft and the sunshield; 
Schedule reserve: 2 months. 

September 2017 - May 2018: 
Observatory I&T: integrates spacecraft and OTIS; 
Schedule reserve: 4 months. 

October 2018: 
Launch date. 

Source: GAO analysis of NASA data. 

[End of figure] 

Overall, project officials report that the critical path schedule has 
14 months of reserve with 7 months after the ISIM and OTE integration 
and test efforts. If these efforts are delayed beyond those 7 months, 
they will impinge on the schedule for the remaining three integration 
and test efforts. Project officials stated that the baseline plan is 
for the OTIS integration and test effort to not begin earlier than May 
2016. These officials reported it is likely that all of the 7 months 
of schedule reserve held by the OTE subsystem will be utilized during 
its integration and test prior to delivery to OTIS and that the OTE 
effort is on the critical path for the project. Therefore, the 
remaining integration and test efforts--OTIS, Spacecraft, and 
Observatory--will likely have at most 7 months divided among them to 
use if issues are found during integration and test. 

In addition to not likely being able to conserve any of the unused 
first 7 months of schedule reserve, the project has limited time 
allocated to the final three integration and test efforts, with 
between 2 to 4 months for each. This time could be used easily by the 
project if an issue were to arise during integration and test. An 
example of this is seen in the OTIS integration and test schedule, 
which currently has 3 months of schedule reserve. The final event in 
the OTIS integration and test effort is a lengthy cryo-vacuum test--
the first time that the optics integrated with the instruments will be 
tested at operational temperatures near absolute zero (less than -400 
degrees Fahrenheit)--that takes approximately 3 months, due to the 
requirements of the test. If an issue were to arise during this test 
that requires shutting the test down and working on the hardware, the 
chamber would have to be slowly warmed to a temperature safe for 
removal of the hardware from the chamber, work would be performed, and 
the 3-month test process would need to begin again. This could easily 
exhaust the available schedule reserve. Prior GAO work shows that it 
is during integration and test when problems are commonly found, and 
schedules tend to slip. A project official confirmed that this is the 
case because during integration and test the process is more 
sequential and there is less flexibility to move work around if 
problems are found. A NASA Inspector General report on the Mars 
Science Laboratory, another complex and high-cost mission, found that 
historically the probability that schedule-impacting problems will 
arise is commensurate with the complexity of the project.[Footnote 15] 
JWST is one of NASA's most technologically complex projects to date. 

Despite Overcoming Several Technical Challenges, JWST Has a 
Significant Amount of Work with Complex Technical Challenges Remaining: 

The project has made significant progress overcoming several technical 
challenges over the last year. In December 2011, for example, the 
project completed development of the 18 segments of the primary mirror-
-the project's primary technology risk--approximately 6 weeks ahead of 
schedule. In addition, project officials stated that during the last 
year they were also able to accelerate other optics-related work, 
which added one month of funded reserve to the schedule, bringing the 
total to 14 months. Finally, the project successfully addressed an 
increase in the estimated amount of heat on the instruments, which 
otherwise could have pushed observatory temperatures close to where 
the optics would not function correctly. 

Although technical challenges are being overcome, the project will 
likely continue to experience additional challenges over the remainder 
of the project, given the significant portion and complexity of the 
work remaining. Four of six major subsystems have nearly 50 percent or 
more of their development work remaining based on its current budget 
information, although the dollar amounts associated with the work 
vary. See figure 5. 

Figure 5: Percentage of Budgeted Development Work Remaining for Major 
Subsystems: 

[Refer to PDF for image: horizontal bar graph] 

Percentage of work to go (as of July 2012: 

Spacecraft: 49%; 
Sunshield: 48%; 
Ground system: 67%; 
OTIS (Integrated OTE and ISIM): 83%; 
Integrated Science Instrument Module (ISIM): 17%; 
Optical Telescope Element (OTE): 24%. 

Source: GAO analysis of NASA budget data. 

[End of figure] 

Currently, the project is experiencing several technical issues that 
have required a significant amount of time and effort to address. For 
example, the spacecraft subsystem, which experienced delays in 
development prior to the replan, is currently estimated to be heavier 
than its mass limit.[Footnote 16] Spacecraft development has lagged 
behind other subsystems because it was viewed as a lower risk part of 
the project and was therefore not allocated funding when budgets were 
limited prior to the replan. In March 2010, the project passed its 
mission critical design review, which evaluated the project design and 
its ability to meet mission requirements and indicated that the design 
was ready for fabrication phase; however, the spacecraft was not 
included in this review due to its delayed development. Under the 
initial replan, which had constrained funding in fiscal years 2011 and 
2012, the spacecraft critical design review was scheduled for June 
2014; however, due to additional funding in the final agency-approved 
replan, the project was able to accelerate work and this review is now 
planned for December 2013. 

Project officials have been concerned with the mass of JWST since its 
inception because of the telescope size and the limits of available 
launch vehicles. Accordingly, mass limits have been allocated for each 
subsystem, including the spacecraft. Project officials stated that 
they expected to encounter mass growth on the spacecraft, but that the 
magnitude of the mass growth on the spacecraft was unexpected. As 
shown in figure 6, the current spacecraft projected mass exceeds its 
mass allocation. 

Figure 6: Current Mass Estimate for Spacecraft Subsystem: 

[Refer to PDF for image: vertical bar graph] 

Spacecraft subsystem mass allocation w/o reserve: 1,754 kilograms; 
Estimated spacecraft mass as of Sept 2012: 1,960 kilograms; 
Difference between spacecraft allocation and mass estimate: 206 
kilograms (-21%). 

Source: GAO analysis of NASA data. 

[End of figure] 

Primary drivers of the mass growth on the spacecraft are increases in 
the estimated weight of the wiring harnesses, which distribute power 
and electric signals between different parts of the observatory, the 
solar array, and other structures that make up the spacecraft. The 
burden to find ways to reduce mass has been primarily placed with the 
spacecraft because it was assessed by the project to have the least 
technical risk and because it is the least mature subsystem and can 
more easily accommodate design changes. Over 100 kilograms, or 220 
pounds, of mass savings options are being evaluated by the project and 
Northrop Grumman, which is developing the spacecraft. Potential mass 
solutions have been identified by Northrop Grumman and the project; 
however, cost and risk vary with each solution and the project is 
still evaluating the trade-offs of the various solutions. Project 
officials stated that final decisions for all tradeoffs will need to 
occur before spacecraft critical design review in December 2013. 

The ISIM subsystem is experiencing technology and engineering 
challenges that resulted in the use of 18 of ISIM's 26 months of 
schedule reserve. The schedule for the instruments needed for ISIM 
continues to slip, which could result in use of more schedule reserve. 
Based on the replan, all four instruments were to be delivered by 
September 2012; however, only two instruments were delivered by that 
time and those still have issues that must be addressed. The remaining 
two instruments are currently scheduled to be delivered at least 11 
months late. See table 1 below for the instrument specific issues. 

Table 1: Issues with JWST Science Instruments Included in ISIM: 

Instrument: MIRI; 
Instrument specific issues: MIRI was delivered in May 2012, 1 month 
later than the replan date, which did not impact the ISIM integration 
and test schedule. However, the instrument team is still working to 
address an instrument sensitivity issue. The instrument team is 
conducting data analysis to determine a root cause for why the 
instrument is less sensitive than required. A formal report on the 
root cause and recommended corrective action is expected in November 
2012. NASA officials report that preliminary findings indicate that 
hardware changes will not be necessary. 

Instrument: NIRSpec; 
Instrument specific issues: NIRSpec is currently planned for delivery 
in August 2013, a delay of 11 months from the replan date as a result 
of several technical issues. First, in July 2011, three cracks were 
found in the part that holds the optics components for the instrument. 
A failure review board was held in January 2012. ESA, which is 
building the instrument, will need to reassemble the instrument using 
the flight spare part. This part cannot be rebuilt in time to include 
the instrument in the first of three cryo-vacuum tests of ISIM. The 
project team adjusted the integration and test schedule to be able to 
accept delivery of NIRSpec in February 2013. However, additional 
issues during testing, including failure of the microshutters to close 
and slower than planned progress on the reintegration by the ESA 
contractor, have further delayed NIRSpec until August 2013. The 
microshutters were provided to ESA by Goddard Space Flight Center and 
enable the instrument to study 100 targets simultaneously. To mitigate 
schedule risk, the project is studying an option to build new 
microshutters that will replace the current ones, if necessary. 
Delivery of NIRSpec after August 2013 may threaten ISIM's critical 
path schedule. 

Instrument: FGS/NIRISS; 
Instrument specific issues: FGS/NIRISS was delivered as planned in 
July 2012; 
however, it was delivered with a known gear motor issue that could 
result in the early loss of NIRISS in space. In July 2012, the 
Canadian Space Agency (CSA), which built the instrument, began to 
redesign the gear motors and they are being rebuilt. Around the time 
of the replan, CSA had to redesign its instrument because the original 
design was not meeting performance requirements. Part of instrument 
design was simplified, which resulted in some loss of expected science 
performance; 
however, NIRISS science is not required for JWST mission success. 

Instrument: NIRCam; 
Instrument specific issues: NIRCam is currently planned for delivery 
in August 2013, a delay of 11 months from the replan date as a result 
of technical and workmanship issues. The project determined the need 
to electrically ground a mirror on the instrument and the team is 
reviewing several potential options to address this issue. Workmanship 
issues, such as poor soldering quality of the wires in the instrument 
controls electronic box, resulted in the need for corrective actions. 
The project has decided to run the first ISIM cryo-vacuum test without 
NIRCam. 

Source: GAO analysis of NASA information. 

[End of table] 

In addition to the instrument delays, two other technical challenges 
associated with ISIM are: (1) the detectors used by three of the four 
instruments to capture infrared light in space are degrading and may 
need to be replaced, resulting in the addition of another round of 
cryo-vacuum testing--in which a test chamber is used to simulate the 
near absolute zero temperatures in space, and (2) issues with the 
development of the cryo-cooler system that removes heat and cools 
MIRI. In December 2010 the project became aware that the detectors in 
three of the instruments were degrading.[Footnote 17] As a result, 
approximately $42 million and 15 months of schedule reserve to replace 
the detectors were included in the replan. These additions covered the 
cost of manufacturing the detectors; fabrication, assembly, and test 
of new focal plane assemblies; changing the detectors on three 
instruments, and the addition of a third ISIM cryo-vacuum test. The 
manufacturing process for new detectors takes approximately 30 months, 
which means that they cannot be delivered until after the second round 
of ISIM cryo-vacuum testing in 2014. As a result, $2 million of the 
$42 million in the replan was used to add a third round of cryo-vacuum 
testing for ISIM. The third test will validate the performance 
requirement of the ISIM and is the only time the instruments are 
tested with the flight detectors. Changing the detectors requires 
disassembling the instruments from ISIM, a process that will risk 
damage to the structure and instruments. Project officials stated that 
they will continue to monitor the degradation rate of the current 
detectors because if the degradation rate is low, they may not replace 
the detectors. 

Development issues with a part of the cryo-cooler needed for MIRI have 
delayed its delivery to ISIM. In 2010, project officials realized that 
an essential valve in the cryo-cooler was leaking at rates that 
exceeded requirements. Following the results of a failure review 
board, the contractor manufactured a newly designed valve, but it also 
did not meet leak rate requirements. Project officials stated that a 
new valve design will not be manufactured in time for use in the first 
ISIM cryo-vacuum test. The project is concurrently developing three 
alternatives and authorized manufacturing for one of the alternatives 
in October 2012. Project officials stated that the MIRI cryo-cooler is 
particularly complex because it spans approximately 10 meters--or 
approximately 33 feet--through the entire JWST observatory. These 
issues combined required the use of 18 months of schedule reserve, 
which reduced ISIM's schedule reserve from the 26 months established 
in the replan to 8 months before it is needed for integration with the 
OTIS subsystem. 

These types of issues are not uncommon among NASA programs as 
technical issues tend to arise when disparate parts are integrated and 
tested together for the first time. Given the complexity and cutting 
edge technology developed and used on JWST, it is expected that these 
kinds of issues will continue to materialize as the program moves 
through its complex integration and test program. Figure 7 shows the 
delay of instrument deliveries as well as changes to the ISIM 
integration and test and final delivery dates over the last year. 

Figure 7: Changes in ISIM Schedule Since the Project Replan: 

[Refer to PDF for image: table] 

MIRI: 
2012: 
Replan delivery date (as of October 2011); 
Dates provided in March 2012; 
Dates provided in November 2012. 

FGS/NIRISS: 
2012: 
Replan delivery date (as of October 2011); 
Dates provided in March 2012; 
Dates provided in November 2012. 

NIRCam: 
2012: 
Replan delivery date (as of October 2011); 
Dates provided in March 2012; 
2013: 
Dates provided in November 2012. 

NIRSpec: 
2012: 
Replan delivery date (as of October 2011); 
2013: 
Dates provided in March 2012; 
Dates provided in November 2012. 
2014: 
2015: 

Cryo-vacuum 1: 
2013: 
Replan delivery date (as of October 2011); 
Dates provided in March 2012; 
Dates provided in November 2012. 

Cryo-vacuum 2: 
2013: 
Replan delivery date (as of October 2011); 
2014: 
Dates provided in March 2012; 
Dates provided in November 2012. 

Cryo-vacuum 3: 
2014: 
Dates provided in March 2012; 
2015: 
Dates provided in November 2012. 

Complete ISIM: 
2014: 
Replan delivery date (as of October 2011); 
2015: 
Dates provided in March 2012; 
Dates provided in November 2012. 

Delivery to OTIS: 
2014: 
Replan delivery date (as of October 2011); 
2015: 
Dates provided in March 2012; 
Dates provided in November 2012. 

Source: GAO analysis of NASA data. 

[End of figure] 

Until the project is able to overcome the major issues with the 
instruments and other parts of the ISIM, it is likely that the 
schedule would continue to slip and may begin to affect the overall 
project schedule. ISIM still has 8 months of schedule reserve before 
the slipping of its schedule would affect the schedule for the 
remainder of the project. The instrument, detector, and cryo-cooler 
issues have all contributed to the delay in the ISIM integration and 
test schedule and the reduction of objectives that can be achieved in 
the first two rounds of cryo-vacuum testing. The first round of 
testing will not include two instruments, a final design of the cryo-
cooler hardware, or new detectors. As a result project officials will 
only be able to gather risk reduction information on the FGS/NIRISS, 
MIRI, test procedures, and test support equipment from the first cryo-
vacuum test. 

The project also has several known long term risks and challenges 
remaining. For example, risks related to OTIS, the sunshield, and the 
ground system subsystems are not scheduled to be addressed until late 
in project development. As of October 2012, seven of the top 10 
project risks were related to the long-term risks associated with the 
OTIS and sunshield, most of which will not be resolved until 2016 or 
later. For example, several risks relating to OTE are not scheduled to 
be closed until the OTIS testing in the chamber at Johnson Space 
Center in February 2017. Project officials are adding risk mitigation 
through early and additional testing, where possible, to these 
subsystems. Prior to the replan, the ground system software was at 
high risk for not being completed before launch and many tasks were 
planned for completion after launch. Space Telescope Science Institute 
officials stated that the replan allows them to plan for completion of 
their work before launch on a more realistic time schedule, which 
decreases schedule and operational risk. A continuing challenge on the 
ground system is that some development and testing is dependent on the 
final design of subsystems such as the instruments, which continue to 
slip delivery dates. 

Independent and Management Oversight during Test and Integration May 
Not Be Commensurate with JWST Test and Integration Complexity: 

The project plans to hold independent and management reviews required 
for all projects during the integration and test phase, but this phase 
for JWST is particularly complex. JWST has five major integration and 
test efforts that span 7 years and only one independent mission-level 
technical review--the system integration review. The system 
integration review evaluates the readiness of the project and 
associated supporting infrastructure to begin system assembly, 
integration, and test, and evaluates whether the remaining project 
development can be completed within available resources. For JWST, 
this review is scheduled in September 2017, only 13 months prior to 
launch. Projects we reviewed that had recently launched, however, held 
their system integration review on average approximately 22 months 
prior to launch. The project has an internal review with participation 
from standing review board members planned before the beginning of 
OTIS integration and test activities begin, and it will be subject to 
independent lower level reviews conducted by the Goddard Systems 
Review Office of the integration and test process. In addition, key 
decision point D (KDP-D)--when the senior agency decision authority 
would approve the project to proceed into the system integration and 
test phase--is scheduled for December 2017, 3 months after the 
commencement of the final major integration and test activity. 
According to NASA policy, this review should be held prior to the 
start of the system integration and test phase of the project. 
[Footnote 18] Project budget information shows that over 90 percent of 
expected integration and test funding will be spent on four major 
integration and test activities prior to the scheduled mission-level 
system integration review and KDP-D approval by NASA senior 
management. As a result, the current plan may be inadequate to ensure 
that key technical and management issues are identified early enough 
to be addressed within the current integration and test phase schedule. 

JWST Project Has Taken Steps to Enhance Communications with and 
Oversight of Its Contractors: 

The JWST project has taken steps to improve communications and 
oversight of its contractors as part of the replanning activities. For 
example, based on recommendations from the ICRP, the project has 
instituted meetings at various levels throughout NASA and its 
contractors and subcontractors. In addition, the project has added 
personnel at contractor facilities, which has allowed for more direct 
interaction and quicker resolution of issues. The project also assumed 
responsibility of the mission-level systems engineering function from 
Northrop Grumman, a move that shifts the authority to make trades or 
decisions to NASA. An independent NASA review of the project conducted 
in May 2012 found, however, that agencywide reductions in travel 
budgets have put the effectiveness of the JWST project's oversight 
plans in jeopardy. While the project received partial relief from 
travel budget reductions in fiscal year 2012, project officials are 
concerned that the current level of oversight will not be sustained if 
similar cuts in travel funding occur in future years as anticipated. 
The project is also taking steps to enhance its oversight of project 
risks by implementing a new risk management system. The new project 
manager found that the previous system lacked rigor and was relatively 
ineffective for managing project risks, especially for a project as 
complex as JWST. The new system should allow for better tracking of 
risks than did the previous system. While these enhancements to the 
oversight of the project are steps in the right direction, it will 
take time to assess their effectiveness. 

NASA Responded to Recent Independent Review Panel Report with 
Increased Communication and Oversight: 

Based on recommendations in the ICRP report, NASA has taken action to 
enhance oversight and communications. See table 2 for the ICRP 
recommendations and actions taken by NASA in response. 

Table 2: ICRP Recommendations Concerning Oversight and Communications 
and the Actions Taken by NASA: 

ICRP recommendation: Move the JWST management and accountability from 
the Astrophysics Division to a new organizational entity at NASA 
headquarters that has responsibility only for the management and 
execution of JWST; 
Action taken: Completed: The management of JWST at Headquarters has 
been reorganized. JWST is now a stand-alone program reporting to the 
NASA Associate Administrator for programmatic oversight and to the 
Associate Administrator for the Science Mission Directorate for 
technical and analysis support. A similar approach was taken in the 
past with both the Hubble Space Telescope and the Mars Exploration 
Program. 

ICRP recommendation: Assign management and execution responsibility 
for the JWST Project to the Goddard Center Director, with 
accountability to the Science Mission Directorate Associate 
Administrator at NASA headquarters; 
Action taken: Completed: NASA has reorganized the JWST project at 
Goddard to report directly to the Center Director, who is responsible 
for the management and execution of the JWST Project and accountable 
to the NASA Administrator. 

ICRP recommendation: Assign at least one senior Goddard project person 
to be resident at Northrop Grumman throughout the Project; 
Action taken: In progress: In August 2012, the Deputy Observatory 
Manager began spending at least half time at the Northrop Grumman 
facility, with a permanent move scheduled in early 2013. 

ICRP recommendation: Improve communications between the JWST Project 
and both Goddard management and NASA headquarters Science Mission 
Directorate; 
Action taken: Completed: The JWST project office provides weekly 
updates of risks and new issues to the Goddard Center Director. In 
addition, the Project Manager at Goddard has daily interaction with 
the Program Manager at NASA HQ. 

ICRP recommendation: Conduct monthly or bi-monthly JWST Project 
Executive meetings, attended by the NASA Associate Administrator and 
the President of Northrop Grumman Aerospace Systems; 
Action taken: Completed: Meetings were first held in December 2010 
between the NASA Associate Administrator, the new JWST Program 
Director, and the Goddard Center Director and senior management of the 
prime contractor. Since that time, senior level management reviews 
have occurred on a quarterly basis and are open to subcontractors as 
well. Topics at these meeting have included issues such as the 
spacecraft mass problems, instrument delivery delays, and the process 
to find alternative solutions for the MIRI cryo-cooler valve problem. 

ICRP recommendation: Strengthen the role and the independent voice of 
the science team in the Project; 
Action taken: Completed: The project added a Deputy Senior Project 
Scientist/Technical position to the project science team. This 
individual: 
* is responsible for day-to-day interactions with senior project 
management on all aspects of the mission; scientific, technical, 
budgetary, and schedule; 
* meets regularly with other members of the project science team to 
ensure better communication with other functions in the project; 
* assists the Senior Project Scientist to facilitate better 
integration of the science activities with the hardware development 
activities and enable closer coordination and understanding of 
technical drivers to science performance so fully informed decisions 
can be made. 

Source: GAO analysis of NASA data. 

[End of table] 

NASA has taken steps to increase communication between the project and 
its contractors and subcontractors in an effort to enhance oversight. 
According to project officials, the increased communication has 
allowed them to better identify and manage project risks by having 
more visibility into contractors' activities. The project reports that 
a great deal of communication existed across the project prior to the 
ICPR and replan; however, improvements have been made. For example, 
monthly meetings between project officials at Goddard and all of the 
contractors have continued on a regular basis and include half-day 
sessions devoted to business discussions. The project reports that 
these meetings have benefits over other forms of communication. For 
example, it was through dialogue with several technical leads at 
Northrop Grumman during detailed reviews of analytical models that the 
project identified that the mass issue on the spacecraft was likely to 
occur. 

In addition, the project has increased its presence at contractor 
facilities as necessary to provide assistance with issues. For 
example, the project has had two engineers working on a recurring 
basis at Lockheed Martin to assist in solving problems with the NIRCam 
instrument. The ISIM manager said that these engineers have insight 
into Lockheed Martin's work and are having a positive effect as they 
offer technical help and are involved in devising the solutions to 
issues. He added that these engineers have authority to make decisions 
on routine issues to allow the work flow to continue, but decisions 
that are more complex or require a commitment of funds are 
communicated to project management for disposition. The project 
reports that the Jet Propulsion Laboratory, responsible for NASA 
contribution to the MIRI instrument and its associated cryo-cooler, 
has an in-house representative in the responsible Northrop Grumman 
division to monitor the work being performed on the cryo-cooler. 

The JWST project also assumed full responsibility for the mission 
system engineering functions from Northrop Grumman in March 2011. NASA 
and Northrop Grumman officials both said that NASA is better suited to 
perform these tasks. Project officials stated the systems engineering 
requires the ability to make trades and decisions across the entire 
observatory, and because Northrop Grumman is only responsible for 
portions of the observatory, it did not have the authority to make 
trades or decisions for areas outside of its control. Although 
responsibility for the overall mission systems engineering function 
was removed from Northrop Grumman, it retains system engineering 
responsibility for work still under its contract, such as development 
of the spacecraft and sunshield. The ICRP noted that a highly capable, 
experienced systems engineering group is fundamental to project 
success and appropriate to ensure accountability especially for a 
project of JWST's complexity and visibility. 

While these enhancements to the oversight of the project are steps in 
the right direction, it will take time to assess their effectiveness. 
In addition, sustainment of these efforts on the part of the project 
will be important. Project and contractor officials we spoke with 
believe that the increased communication has had a positive effect on 
the relationships between them. We will continue to monitor the 
interaction between the project and its contractors and its frequency 
in future reviews to identify whether the changes have had the desired 
results. 

Travel Budget Reductions May Hamper Planned Oversight Activities: 

The JWST project reported that its travel budget was reduced by 
approximately $200,000 from the $1.2 million planned in fiscal year 
2012 as a result of NASA's implementation of an Executive Order to 
promote more efficient spending.[Footnote 19] According to project 
officials, the changes in oversight necessitated by a reduction in 
travel funds represent a major shift away from the management paradigm 
adopted during the replan. Proposed reductions in future fiscal years 
could significantly reduce the project's travel budget. The project 
reports that the travel requirements for fiscal years 2013 through 
2015 are $1.6 million, $1.7 million, and $1.8 million, respectively. 
Officials reported that while travel is a small percentage of the 
project's annual budget, the majority of expected travel--about 87 
percent--is for oversight functions put in place as a result of the 
ICRP recommendations, such as having a permanent on-site presence at 
Northrop Grumman. These oversight functions include attending and 
participating in contractor monthly programmatic and technical 
reviews, technical interface meetings, recurring on-site presence at 
contractor facilities for quality assurance reviews and inspection of 
hardware. JWST project officials are concerned that decreased 
oversight could translate into the project increasing its use of cost 
and schedule reserves as they will not be conducting planned oversight 
to better ensure success. A recent NASA Office of Evaluation review 
concluded that by not having an adequate travel budget, the project is 
at risk of cost/schedule growth and/or technical risk due to the late 
identification of issues or timely resolution strategies. The project 
has made adjustments to absorb the reduction in fiscal year 2012 and 
plans to identify instances of increased cost or schedule risk due to 
late identification of issues. However, the project does not have a 
strategy to address anticipated future reductions. Ensuring adequate 
oversight is particularly important as the project begins its complex 
and lengthy test and integration phase, where issues will likely 
surface. 

NASA Headquarters Independent Review of the JWST Program Identified 
Concerns and Recommends Indicators to Measure Progress: 

As part of NASA's approach to increase oversight of the project at 
headquarters, NASA's Office of Evaluation recently conducted an 
independent review of the JWST project to assess the progress since 
the September 2011 rebaseline was approved. According to the Director 
of the Office of Evaluation, the goal of the review was not to 
reproduce the replan assessment, but rather to assess progress based 
on cost, schedule, and technical performance of the project and the 
status of oversight functions within NASA headquarters, the JWST 
Program Office, and Goddard Space Flight Center. The intended outcome 
of the review was 1) to obtain a snapshot of performance to determine 
if the program was progressing in accordance with its plan, and 2) to 
identify leading indicators for upper management to use when tracking 
future performance. The review team identified several areas of 
concern within the program, many of which we have highlighted, and 
recommended a list of leading indicators that project management 
should consider tracking. 

The Director of the Office of Evaluation said that the project is 
generally performing the activities and maintaining the schedule set 
forth in the replan; however, the team identified key areas that 
should be monitored as the project moves forward. The review team also 
recommended a set of leading indicators for project management to 
consider tracking to measure and monitor progress. The Director added 
that these indicators are for the project to use and would not be 
specific criteria for use by independent review boards such as the 
Standing Review Board. These indicators are a positive step to ensure 
that NASA management has the information necessary to monitor the 
progress of the JWST project. See table 3 below for the concerns 
raised by the review team. 

Table 3: NASA Office of Evaluation Review Team Concerns: 

Concern identified: Resolving the on-going mass issue[A]; 
NASA Office of Evaluation Review Team finding: The technical and 
programmatic risks involved in possible solutions need to be 
explicitly understood. 

Concern identified: Understanding root causes for the NIRSpec Optical 
Bench cracking issue; 
NASA Office of Evaluation Review Team finding: There needs to be a 
better understanding of the root causes and investigate what would 
constitute a flyable structure. 

Concern identified: Monitoring of schedule erosion by the project and 
recognition of the dynamics of four competing critical paths[A]; 
NASA Office of Evaluation Review Team finding: There are several 
competing critical paths for the JWST program to include development 
and integration and test of the OTE, ISIM, sunshield, and the 
spacecraft. 

Concern identified: Monitoring the stability of the schedule going 
forward; 
NASA Office of Evaluation Review Team finding: The master schedule 
took 8 months to baseline and stability of the schedule going forward 
is unknown. The project will need an accurate reporting system at the 
integrated master schedule level. 

Concern identified: Ensuring that Northrop Grumman will have an 
adequate workforce and that skills are available in fiscal year 
2013[A]; 
NASA Office of Evaluation Review Team finding: The project needs to 
work with Northrop Grumman to ensure that it has an adequate workforce 
and skill sets available to accomplish the planned ramp up for fiscal 
year 2013. 

Concern identified: Monitoring the increased risk of cost/schedule 
growth and technical risk; 
NASA Office of Evaluation Review Team finding: Without an adequate 
travel budget to execute the original plan, there is an increased risk 
of cost or schedule growth and technical risks due to late 
identification of issues or timely resolution strategies. 

Source: GAO analysis of NASA data. 

[A] These issues were also identified as leading indicators the review 
team recommended that project management should consider tracking. 

[End of table] 

Changes Made to Project Risk Management System: 

The new JWST project manager re-emphasized the importance of the 
project's risk management system and, in August 2012, a new risk 
management database was implemented to support the system. The project 
manager told us that he evaluated the risk management system being 
utilized by the project when he assumed his position and found it to 
be ineffective and not robust, especially for a project as complex as 
JWST. While the basic risk management methodology remains unchanged, 
the project manager wanted a more regimented system. For example, the 
project utilizes a hierarchy of risk boards that periodically reviews 
and provides disposition of all new and existing risks. These risk 
boards reviewed and assessed new risks and lower level risk board 
actions and met on an ad hoc basis. The project manager instituted a 
more regimented system that re-emphasized and revised the weekly 
project risk board meetings. Lower level risk boards meet a minimum of 
once a month depending on activity. 

The project manager also determined that a new risk management 
database needed to be put in place that would bring more rigor to the 
risk management process. The project manager told us that he directed 
an overhaul of the risk management database to provide more complete 
information to management on the purpose and history for each risk. 
The goal was to improve consistency in how the project determined the 
potential for a risk to occur and its impact, and provide greater 
detail on mitigation and better tracking of the status for each risk. 
For example, the new system puts more emphasis on understanding and 
capturing the key events in the mitigation plan that are intended to 
result in a change in likelihood or consequence of a risk. The new 
system has a provision where the mitigation plan will be entered and 
updated over time, and the capability to store data such as mitigation 
steps throughout the life of the risk. In addition, the new system now 
archives data automatically to provide a traceable history of the 
risk. The prior data system did not have as robust of an archiving 
function. Furthermore, the project manager wanted to improve the 
linkage between the risk database entries and financial records to 
ensure consistency of the data in the risk database with regard to 
cost and schedule for risk mitigations with project office financial 
records. As the changes to the risk management system and database, as 
well as other changes we identified that were put in place to enhance 
oversight were just recently implemented, we will continue to monitor 
their continued use and assess the impact they may be having on the 
project. 

Conclusions: 

The JWST project is among the most challenging and high-risk projects 
NASA has pursued in recent years. It is also one of the most 
expensive, with a recent major replan resulting in a total cost of 
$8.8 billion. The reasons for cost and schedule growth were largely 
recognized by an independent review team to be rooted in ineffective 
funding, management, communication, and oversight. NASA has invested 
considerable time and resources replanning the project and instituting 
management and oversight improvements in order to ensure that it (1) 
can be executed within its new estimates and (2) has addressed the 
majority of issues raised in the recent independent review. It appears 
that communications with contractors and within NASA have improved, 
that a more robust risk mitigation system is in place, that more is 
known about what it will take to complete the project and how much it 
will cost, and that the project is currently meeting the majority of 
its milestones. Nevertheless, over the course of the next several 
years, the project will be executing a large amount of work with 
several extremely complex and challenging integration and test 
efforts. Because three major test and integration efforts must be 
completed in the last 2 years of the JWST schedule, it is essential 
that issues are identified and addressed early enough to be handled 
within the project's current schedule. While the JWST oversight plan 
is consistent with NASA's requirements for all project's required 
reviews, a single independent review scheduled just over a year before 
launch may not be sufficient to identify and resolve problems early 
for a project of this magnitude. A key element of overseeing project 
progress is monitoring how the project is executing to its cost 
baseline. To that end, while NASA took some steps that were in line 
with best practices to develop its revised baseline, some of the 
deficiencies we found in its process could impact the reliability of 
the cost estimate and the joint cost and schedule confidence level 
that was provided to headquarters decision-makers. Without higher-
fidelity, regularly updated information related to costs, as well as 
an oversight regime during later phases of test and integration that 
is commensurate with the complexity of that effort, NASA risks late 
identification of technical and cost issues that could delay the 
launch of JWST and increase project costs beyond established 
baselines. Also important to oversight for the remainder of the 
project is the ability of officials to sustain improvements to 
communication with and oversight of contractors. Anticipated travel 
restrictions, however, could decrease the project team's ability to 
sustain these actions. Without a plan to address such reductions in 
future years, the project could once again become susceptible to 
communication and oversight problems identified in earlier reviews, 
which could also have a detrimental impact on continued project 
performance. 

Recommendations for Executive Action: 

To ensure that the JWST life-cycle cost estimate conforms to best 
practices, GAO recommends that the NASA Administrator direct JWST 
officials to take the following three actions to provide high-fidelity 
cost information for monitoring project progress: 

* improve cost estimate documentation and continually update it to 
reflect earned value management actual costs and record any reasons 
for variances, 

* conduct a sensitivity analysis on the number of staff working on the 
program to determine how staff variations affect the cost estimate, 
and: 

* perform an updated integrated cost/schedule risk analysis, or joint 
cost and schedule confidence level analysis, using a schedule that 
meets best practices and includes enough detail so that risks can be 
appropriately mapped to activities and costs; historical, analogous 
data should be used to support the risk analysis. 

To ensure that technical risks and challenges are being effectively 
managed and that sufficient oversight is in place and can be 
sustained, GAO recommends that the NASA Administrator direct JWST 
officials to take the following three actions: 

* conduct a separate independent review prior to the beginning of the 
OTIS and spacecraft integration and test efforts to allow the 
project's independent standing review board the opportunity to 
evaluate the readiness of the project to move forward, given the lack 
of schedule flexibility once these efforts are under way, 

* schedule the management review and approval to proceed to 
integration and test (key decision point D or KDP-D) prior to the 
start of observatory integration and test effort, and: 

* devise an effective, long-term plan for project office oversight of 
its contractors that takes into consideration the anticipated travel 
budget reductions. 

Agency Comments and Our Evaluation: 

NASA provided written comments on a draft of this report. These 
comments are reprinted in appendix IV. NASA also provided technical 
comments, which were incorporated as appropriate. 

In responding to a draft of this report, NASA concurred with three 
recommendations and partially concurred with three other 
recommendations and commented on actions in process or planned in 
response. In some cases, these actions meet the intent and are 
responsive to issues we raise; however, some of the responses do not 
fully address the issues we raised in the report. 

NASA partially concurred with our recommendation to improve the cost 
estimate documentation of the JWST project, and to continually update 
it to reflect earned value management actual costs and record any 
reasons for variances between planned and actual costs. In response to 
this recommendation, NASA officials stated that the project currently 
receives earned value data from some of its contractors and performs 
monthly analysis of that data to understand the contractors' estimates 
at completion, and then compares these numbers to similar figures 
independently assessed by the JWST project. NASA also highlighted its 
efforts to improve the agency's documentation of the earned value 
variances and to extend the earned value management analysis to areas 
where it is not yet implemented, such as ground systems development at 
the Space Telescope Science Institute. In addition, NASA responded 
that its annual budget process generates a requirements-driven budget 
plan consistent with the rebaseline. NASA stated that this information 
is updated in several different documents that are provided to 
management and it does not plan to revise its JCL documentation 
developed during the replan. Despite these steps, we could not 
independently confirm that they were leading to an updated cost 
estimate, which is the basis of our recommendation. If the estimate is 
not updated, it will be difficult to analyze changes in project costs 
and collecting cost and technical data to support future estimates 
will be hindered. Furthermore, if not properly updated on a regular 
basis, the cost estimate cannot provide decision makers with accurate 
information for assessing alternative decisions. Without a documented 
comparison between the current estimate (updated with actual costs) 
and the old estimate, the cost estimator cannot determine the level of 
variance between the two estimates and cannot see how the project is 
changing over time. Therefore, we continue to believe NASA will be 
well served by following best practices and updating its cost estimate 
with current information and documenting reasons for any variances. We 
encourage the project to improve the cost estimate documentation and 
record any reasons for variances between planned and actual costs and 
we intend to review the documentation as a part of our ongoing review 
of the project. 

NASA officials partially concurred with our recommendation that the 
project conduct a sensitivity analysis on the number of staff working 
on the project to determine how staff variations affect the cost 
estimate. In its response, the agency stated that it believes it met 
the intent of this recommendation when staffing levels were determined 
in the 2011 JWST rebaseline based on programmatic experience from the 
accomplishment of similar activities. To accommodate the possibility 
of increased costs based on increased staffing hours, NASA reports 
that funded schedule reserve was built into the JWST rebaseline, in 
addition to unallocated future expenses being held at various levels 
of the organization. NASA believes that these reserves will be 
sufficient to cover increases for the duration of specific activities 
that result in increased staffing cost, and that an additional 
workforce sensitivity analysis is not warranted. NASA added that the 
joint cost and schedule confidence level analysis performed provided a 
de facto workforce sensitivity analysis and does not plan any further 
action. A joint cost and schedule confidence level analysis, however, 
is not the same as a sensitivity analysis wherein the sources of the 
workforce variation should be well documented and traceable. While we 
appreciate the steps NASA took to account for workforce variation, the 
JWST cost model does not show how staff levels increasing or 
decreasing over time affects cost. Furthermore, best practices call 
for a risk analysis to be conducted in conjunction with a sensitivity 
analysis, not to be a substitute for it. As a best practice, a 
sensitivity analysis should be included in all cost estimates because 
it examines the effects of changing assumptions and ground rules. 
Since uncertainty cannot be avoided, it is necessary to identify the 
cost elements that represent the most risk and, if possible, cost 
estimators should quantify the risk. Without performing a sensitivity 
analysis that reveals how the cost estimate is affected by a change in 
a single assumption, such as workforce size, the cost estimator will 
not fully understand which variable most affects the cost estimate. 
Therefore, we continue to believe that NASA should conduct a 
sensitivity analysis for the JWST project, given the large number of 
staff working on the program, to determine how staff variations 
positively or negatively affect the cost estimate rather than relying 
on schedule reserve and unallocated future expenses to offset any 
shortfall. 

NASA concurred with our recommendation to perform an updated 
integrated cost and schedule risk analysis using a schedule that meets 
best practices and includes enough detail so that risks can be 
appropriately mapped to activities and costs. In response to this 
recommendation, NASA stated that the agency is already using tools and 
a method to conduct programmatic assessments of projects after the 
baseline was established using the JCL methodology. While these may be 
good tools, the key point is the need to address shortcomings of the 
schedule that supports the baseline itself. For example, the lack of 
detail in the summary schedule used for the joint cost and schedule 
risk analysis prevented us from sufficiently understanding how risks 
were incorporated; therefore, we question the results of that 
analysis. Since the JCL was a key input to the decision process of 
approving the project's new cost and schedule baseline estimates, we 
maintain that the JWST project should perform an updated JCL analysis 
using a schedule with sufficient detail to map risks to activities and 
costs. Doing so could help increase the reliability of the cost 
estimate and the confidence level of the JCL. Furthermore, risk 
management is a continuous process that constantly monitors a 
project's health. Given that JWST is many years from launch and the 
risks that the project faces are likely to change, a risk analysis 
should be conducted periodically throughout the life of the project. 

NASA concurred with our recommendation to conduct a separate 
independent review prior to the beginning of the OTIS and spacecraft 
integration and test efforts. In response to this recommendation, NASA 
stated that it will request members of the independent JWST Standing 
Review Board participate in OTIS Pre-Environmental Review scheduled 
prior to the beginning of OTIS environmental testing. A member of the 
Standing Review Board will co-chair this review and report its 
findings to the NASA Associate Administrator, which is the practice of 
all Standing Review Board reviews. In addition, NASA plans to direct 
Northrop Grumman, the spacecraft developer, to add members of the 
Standing Review Board, as well as members of the Goddard Independent 
Review Team, to the spacecraft element integration readiness review 
and report their findings to the NASA Associate Administrator. We 
believe these actions meet the intent of our recommendation and will 
afford an independent evaluation of the readiness of the project to 
move forward with its major integration and test efforts. 

NASA partially concurred with our recommendation to schedule the 
management review and approval to proceed to integration and test (KDP-
D) prior to the start of the observatory integration and test effort. 
In response to this recommendation, NASA stated that it will reduce 
the 3-month gap between the scheduled system integration review and 
the KDP-D review, which it believes will provide NASA management and 
the NASA Associate Administrator with the full independent assessment 
earlier than currently planned. While we agree that this change will 
move the review earlier than previously planned, based on its 
response, NASA still plans to hold the review after the observatory 
integration and test is already underway. Holding this review after 
the observatory integration and test effort is already underway does 
not meet agency policy and will lessen the impact of the review as it 
may be inadequate to ensure key technical and management issues are 
identified early enough to be addressed. KDP-D is the point in which 
management approval is given to transition to the test and integration 
phase. We reiterate our recommendation that NASA should hold this 
important key decision point prior to the beginning of this last major 
integration and test effort, as required by agency policy. 

NASA concurred with our recommendation to devise an effective, long-
term plan for project office oversight of its contractors that takes 
into consideration the anticipated travel budget reductions. In 
response to this recommendation, NASA stated that it will develop a 
plan based on fiscal year 2013 travel allocations and will take into 
consideration anticipated travel budget reductions. In addition, NASA 
stated that the plan will enable the project to maintain oversight of 
JWST contractors and their ability to meet performance and delivery 
deadlines and work closely with the international partners. We believe 
such a plan will be critical to ensuring adequate oversight, which is 
particularly important as the project enters into the complex 
integration and test efforts where issues will likely surface. In 
addition, we agree with the concerns of project officials that the 
current efforts to increase communication and oversight may not be 
sustained if reductions to future travel budgets occur as anticipated. 
We encourage the project to complete this plan in a timely manner and 
intend to review it as a part of our ongoing assessment of the 
project's oversight efforts. 

We will send copies of the report to NASA's Administrator and 
interested congressional committees. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on GAO's web-site at http://www.gao.gov. 

Should you or your staff have any questions on matters discussed in 
this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. GAO staff who 
made major contributions to this report are listed in appendix V. 

Signed by: 

Cristina T. Chaplain: 
Director: 
Acquisition and Sourcing Management: 

List of Committees: 

The Honorable Barbara A. Mikulski:
Chairwoman:
The Honorable Kay Bailey Hutchison:
Ranking Member:
Subcommittee on Commerce, Justice, Science, and Related Agencies:
Committee on Appropriations:
United States Senate: 

The Honorable Frank R. Wolf:
Chairman:
The Honorable Chaka Fattah:
Ranking Member:
Subcommittee on Commerce, Justice, Science, and Related Agencies:
Committee on Appropriations:
House of Representatives: 

The Honorable Ralph M. Hall:
Chairman:
The Honorable Eddie Bernice Johnson:
Ranking Member:
Committee on Science, Space, and Technology:
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to assess (1) the extent to which NASA's revised 
cost and schedule estimates are reliable based on GAO best practices, 
(2) the major risks and technological challenges the James Webb Space 
Telescope (JWST) project faces, and (3) the extent to which the 
National Aeronautics and Space Administration (NASA) has improved the 
oversight of the JWST project. In assessing the project's cost and 
schedule estimates, we performed various checks to determine that the 
provided data were reliable enough for our purposes. Where we 
discovered discrepancies, we clarified the data accordingly. Where 
applicable, we confirmed the accuracy of NASA-generated data with 
multiple sources within NASA. 

To assess the current life cycle cost estimate of the JWST project, we 
used the GAO cost guide to evaluate the estimating methodologies, 
assumptions, and results to determine whether the cost estimate was 
comprehensive, accurate, well-documented, and credible.[Footnote 20] 
We developed standardized data collection instruments to request 
relevant cost and schedule documentation and questionnaires for 
project officials to complete. Through the data collection instruments 
and questionnaires, we gathered basic information pertaining to the 
project's cost and schedule estimation process. We examined documents 
from the JWST replan efforts pertaining to the revised cost estimate 
and the joint cost and schedule confidence level (JCL) analysis, 
including detailed spreadsheets that contained cost, schedule and 
project risk information. Project documents we reviewed included basis 
of estimates for the replan from contractors and the project office, 
contractor engineering change proposals, the JWST program plan and 
mission requirements, as well as the NASA cost estimating handbook. 
[Footnote 21] To assess how management and independent review teams 
were involved in the cost estimate process, we also reviewed the 
project's monthly status presentations to Goddard management, Standing 
Review Board reports concerning the project's revised baseline 
estimates, cost estimates conducted by NASA's Independent Program 
Assessment Office, and the Agency Program Management Council's 
decision memo for the rebaselined estimate and JCL analysis. We 
supplemented our analysis by interviewing officials from Northrop 
Grumman, the Space Telescope Science Institute, and the JWST program 
and project offices. After reviewing cost estimate documentation 
submitted by NASA and conducting numerous interviews with relevant 
sources within the project office, we calculated the assessment rating 
of each criteria within the four characteristics by assigning each 
individual assessment rating: Not Met = 1, Minimally Met = 2, 
Partially Met = 3, Substantially Met = 4, and Met = 5. We then took 
the average of the individual assessment ratings for the criteria to 
determine the overall rating for each of the four characteristics. The 
resulting average becomes the "Overall Assessment" as follows: Not Met 
= 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, 
Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We discussed the 
results of our assessments with officials within the program office at 
NASA headquarters and the project office at Goddard Space Flight 
Center. 

We supplemented the assessment of the revised 2011 cost estimate with 
an assessment of the summary schedule used for the JCL, which was a 
part of the project's cost estimation process, and followed criteria 
laid out in the GAO schedule guide.[Footnote 22] These practices 
address whether the schedule (1) captured all activities; (2) 
sequenced all activities--that is, listed in the order in which they 
are to be carried out; (3) assigned resources to all activities; (4) 
established the duration of all activities; (5) integrated schedule 
activities horizontally and vertically, which identifies whether 
products and outcomes associated with other sequenced activities are 
arranged in the right order, and that varying levels of activities and 
supporting subactivities are also aligned properly; (6) established 
for all activities, the critical path, which is the longest continuous 
sequence of activities that is necessary to examine the effects of 
activities slipping in the schedule; (7) identified between activities 
float, which is the amount of time by which a predecessor activity can 
slip before the delay affects the program's estimated finish date; (8) 
identified a level of confidence using a schedule risk analysis; and 
(9) was updated using logic and durations to determine dates. We also 
reviewed the inputs to the JCL model, the document outlining the 
methodology of the analysis that accompanied the electronic files, and 
interviewed cognizant project officials to discuss their use of the 
summary schedule. 

Because the project's detailed integrated master schedule has not been 
finalized because of ongoing negotiations and contract modifications, 
we did not conduct a complete schedule analysis using the GAO schedule 
assessment guide. We plan to perform this assessment in a subsequent 
review of the JWST project. 

To assess the major short-and long-term risks and technological 
challenges facing the project, we reviewed the project's risk list, 
monthly status reviews, and other documentation provided by projects 
and contractor officials. This information covered the risks, 
mitigation plans, and timelines for addressing risk and technological 
challenges. We also interviewed project officials for each major 
observatory subsystems to clarify information and to obtain additional 
information on risks and technological challenges. Further, we 
interviewed officials from the Jet Propulsion Laboratory, Northrop 
Grumman Aerospace Systems, Lockheed Martin Advanced Technology 
Company, Teledyne Imaging Sensors, the University of Arizona, and the 
Space Telescope Science Institute concerning risks and challenges on 
the subsystems, instruments, or components they were developing. We 
reviewed GAO's prior work on NASA Large Scale Acquisitions, NASA 
Office of Inspector General reports, and NASA's Space Flight Program 
and Project Management Requirements and Systems Engineering Processes 
and Requirements policy documents.[Footnote 23] We compared NASA's 
controls as outlined in these agency policies with the project plan to 
assess the extent to which the JWST's plan followed the intent of the 
policies with regard to independent oversight and management approval 
processes. 

To assess the extent to which NASA is performing enhanced oversight of 
the JWST project, we reviewed documentation from the Independent 
Comprehensive Review Panel and the project to determine actions taken 
by NASA in response to the panel's recommendations. We interviewed 
project officials to understand the impact of these changes on the 
oversight processes for the project and communication between the 
project and its contractors. We also interviewed officials from the 
Jet Propulsion Laboratory, Northrop Grumman Aerospace Systems, 
Lockheed Martin Advanced Technology Company, Teledyne Imaging Sensors, 
the University of Arizona, and the Space Telescope Science Institute 
concerning project oversight of work they were performing and the 
effectiveness of oversight changes. In addition, we reviewed a 
presidential directive and Office of Management and Budget and project 
documentation and interviewed project officials concerning the 
reductions to travel budgets and their impact on project oversight 
activities. We interviewed the Director of NASA's Office of Evaluation 
about a recent internal review of the JWST project and reviewed 
documentation from that review. We also reviewed documentation and 
interviewed project officials concerning the changes made to the 
project's risk management system. 

Our work was performed primarily at NASA Headquarters in Washington, 
D.C., and Goddard Space Flight Center in Greenbelt, Maryland. We also 
visited Johnson Space Center in Houston, Texas, and the Jet Propulsion 
Laboratory in Pasadena, California. In addition, we met with 
representatives from Northrop Grumman Aerospace Systems, Lockheed 
Martin Advanced Technology Company, Teledyne Imaging Sensors, the 
University of Arizona, and the Space Telescope Science Institute. 

We conducted this performance audit from February 2012 to December 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Subsystems of the JWST Observatory: 

Figure 8: Subsystems of the JWST: Interactive Information: 

[Refer to PDF for image: Subsystem information with 6 illustrations] 

Integrated Science Instrument Module: 
Acronym: ISTM; 
Contractor/Center: Goddard Space Flight Center; 
Description: Combines the 4 instruments. 

Mid Infrared Instrument: 
Acronym: MIRI; 
Contractor/Center: Jet Propulsion Lab and European Consortium; 
Description: Science Instrument. 

Near Infrared Spectograph: 
Acronym: NIRSpec; 
Contractor/Center: European Space Agency; 
Description: Science Instrument. 

Fine Guidance Sensor/Near-Infrared Imager and Slitless Spectrograph: 
Acronym: FGS/NIRISS; 
Contractor/Center: Canadian Space Agency; 
Description: Telescope guider and Science Instrument. 

Near Infrared Camera: 
Acronym: NIRCam; 
Contractor/Center: University of Arizona; 
Description: Science Instrument and Wave Front Sensor. 

Spacecraft: 
Contractor/Center: Northrup Grumman Aerospace Systems; 
Description: Con tans the power, communications, and avionics needed 
to operate the observatory. Contains the cryocooler needed to achieve 
MIRI operational temperature approximating 7K. 

Sunshield: 
Contractor/Center: Northrup Grumman Aerospace Systems; 
Description: Tennis court sized series of 5 thin membranes, provides 
passive cooling to achieve operational temperatures approximating 45K 
for the OTE and ISIM. 

Ground systems: 
Contractor/Center: Space Telescope Science Institute; 
Description: The ground system (located on Earth) communicates with 
and operates the spacecraft. 

Optical Telescope & Integrated Science Instrument Module: 
Acronym: OTIS (OTE+ISIM); 
Contractor/Center: Goddard Space Flight Center; 
Description: Hardware configuration created when OTE and ISIM are 
integrated. 

Optical Telescope Element: 
Acronym: OTE; 
Contractor/Center: Northrup Grumman Aerospace Systems; 
Description: 18 primary mirror segments, secondary mirror, tertiary 
mirror, backplane support structure. 

Sources: GAO (analysis); NASA (data and images). 

Note: The hardware configuration created when the Optical Telescope 
Element and the Integrated Science Instrument Module are integrated, 
referred to as OTIS, is not considered a subsystem by NASA, but we 
categorize it as such for ease of discussion. 

[End of figure] 

[End of section] 

Appendix III: Our Evaluation of JWST's Cost Estimate Process: 

In determining that the National Aeronautics and Space 
Administration's (NASA) processes for developing the James Webb Space 
Telescope (JWST) cost estimate do not fully comply with best 
practices, we evaluated the project's cost estimation methods against 
our 2009 Cost Estimating and Assessment Guide. (See table 4.) We 
applied the following scale across the four categories of best 
practices: 

* Not met: NASA provided no evidence that satisfies any portion of the 
criterion. 

* Minimally met: NASA provided evidence that satisfies less than one-
half of the criterion. 

* Partially met: NASA provided evidence that satisfies about one-half 
of the criterion. 

* Substantially met: NASA provided evidence that satisfies more than 
one-half of the criterion. 

* Met: NASA provided complete evidence that satisfies the entire 
criterion. 

Table 4: Summary Assessment of JWST Cost Estimate Compared to Best 
Practices: 

Characteristic: Comprehensive; 
Overall assessment: Substantially met; 

Best practice: The cost estimate includes all life-cycle costs; 
Individual assessment: Met. 

Best practice: The cost estimate completely defines the program, 
reflects the current schedule, and is technically reasonable; 
Individual assessment: Substantially met. 

Best practice: The cost estimate work breakdown structure is product 
oriented, traceable to the statement of work/objective, and at an 
appropriate level of detail to ensure that cost elements are neither 
omitted nor double counted; 
Individual assessment: Substantially met. 

Best practice: The estimate documents all cost-influencing ground 
rules and assumptions; 
Individual assessment: Substantially met. 

Characteristic: Well documented; 
Overall assessment: Partially met; 

Best practice: The documentation should capture the source data used, 
the reliability of the data, and how the data were normalized; 
Individual assessment: Partially met. 

Best practice: The documentation describes in sufficient detail the 
calculations performed and the estimating methodology used to derive 
each element's cost; 
Individual assessment: Minimally met. 

Best practice: The documentation describes step-by-step how the 
estimate was developed so that a cost analyst unfamiliar with the 
program could understand what was done and replicate it; 
Individual assessment: Minimally met. 

Best practice: The documentation discusses the technical baseline 
description, and the data in the baseline are consistent with the 
estimate; 
Individual assessment: Substantially met. 

Best practice: The documentation provides evidence that the cost 
estimate was reviewed and accepted by management; 
Individual assessment: Substantially met. 

Characteristic: Accurate; 
Overall assessment: Partially met; 

Best practice: The cost estimate results are unbiased, not overly 
conservative or optimistic and based on an assessment of most likely 
costs; 
Individual assessment: Partially met. 

Best practice: The estimate has been adjusted properly for inflation; 
Individual assessment: Partially met. 

Best practice: The estimate contains few, if any, minor mistakes; 
Individual assessment: Partially met. 

Best practice: The cost estimate is regularly updated to reflect 
significant changes in the program so that it is always reflecting 
current status; 
Individual assessment: Partially met. 

Best practice: Variances between planned and actual costs are 
documented, explained, and reviewed; 
Individual assessment: Minimally met. 

Best practice: The estimate is based on a historical record of cost 
estimating and actual experiences from other comparable programs; 
Individual assessment: Partially met. 

Characteristic: Credible; 
Overall assessment: Partially met; 

Best practice: The cost estimate includes a sensitivity analysis that 
identifies a range of possible costs based on varying major 
assumptions, parameters, and data inputs; 
Individual assessment: Minimally met. 

Best practice: A risk and uncertainty analysis was conducted that 
quantified the imperfectly understood risks and identified the effects 
of changing key cost driver assumptions and factors; 
Individual assessment: Partially met. 

Best practice: Major cost elements were cross-checked to see whether 
results were similar; 
Individual assessment: Minimally met. 

Best practice: An independent cost estimate was conducted by a group 
outside the acquiring organization to determine whether other 
estimating methods produce similar results; 
Individual assessment: Partially met. 

Source: GAO analysis of NASA data. 

[End of table] 

[End of section] 

Appendix IV: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration: 
Office of the Administrator: 
Washington, DC 20546-0001: 

November 20, 2012: 

Ms. Christina Chaplain: 
Director: 
Acquisition and Sourcing Management: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Ms. Chaplain: 

The National Aeronautics and Space Administration (NASA) appreciates 
the opportunity to review the Government Accountability Office (GAO) 
draft correspondence entitled "James Webb Space Telescope: Actions 
Needed to Improve Cost Estimate and Oversight of Test and Integration" 
(GA0-13-4). NASA values the continued open communications between NASA 
and the GAO team on the James Webb Space Telescope (JWST) and 
appreciates the constructive comments arising as a result of this 
dialogue. The JWST Program is fully compliant with NASA's programmatic 
requirements and is committed to making improvements as the Agency 
deems appropriate. 

In the draft report, GAO makes two recommendations to the 
Administrator, specifically: 

Recommendation 1: To ensure that the JWST life-cycle cost estimate 
conforms to best practices, GAO recommends that the NASA Administrator 
direct JWST project officials to take the following three actions to 
provide high-fidelity cost information for monitoring project progress: 

a) improve cost estimate documentation and continually update it to 
reflect earned value management actual costs and record any reasons 
for variances, 

b) conduct a sensitivity analysis on the number of staff working on 
the program to determine how staff variations affect the cost 
estimate, and, 

c) perform an updated integrated cost/schedule risk analysis, or joint 
cost and schedule confidence level analysis, using a schedule that 
meets best practices and includes enough detail so that risks can be 
appropriately mapped to activities and costs; historical, analogous 
data should be used to support the risk analysis. 

Management's Response: Partially Concur. 

la) Partially Concur: NASA uses a number of performance metrics to 
assess the project's performance against plan, including Earned Value 
Management (EVM). NASA is currently receiving EVM data from multiple 
JWST contractors (i.e., Northrop Grumman Aerospace Systems INCAS], 
Lockheed Martin Advanced Technology Center [LMATC], ITT/Exelis), and 
performs monthly analysis of that data to understand contractor 
estimates at completion (EAC) as compared to the Project's Independent 
EAC (IEAC), documenting the reasons for any variances. NASA has also 
recently initiated EAC analysis for in-house Integrated Science 
Instrument Module (ISIM) work. NASA will continue to perform this 
analysis throughout the development phase of JWST. 

NASA is committed to improving its documentation of the variances and 
to extending the EVM analysis to include areas where it is not yet 
implemented. For example, NASA is working with the Space Telescope 
Science Institute (STSI) to implement BVM for its ground system 
development. In addition, the Agency's annual budget planning activity
(Planning, Programming, Budgeting, and Execution, or PPBE) generates a 
requirements-driven budget plan consistent with the rebaseline, which 
is well documented with Basis of Estimates (BOBs) and incorporates 
information on performance-to-date against plan and EAC estimates. 
JWST does not plan to revise its Joint cost and schedule Confidence
Level (JCL) documentation developed during the re-plan. 

1b) Partially Concur. NASA met the intent of this recommendation by 
taking a number of steps to determine how staffing variations affect 
its cost estimate and to ensure the project has sufficient margin to 
address changes in workforce requirements. Staffing levels, such as 
the number of full-time equivalents working on a specific activity at 
any given time, were determined in the 2011 JWST rebaseline based on 
programmatic experience in the accomplishment of similar activities. 
The primary driver of increased staffing hours required for a specific 
activity would be an increase in the duration of the activity (e.g., 
integration and test). To accommodate this possibility, funded schedule
reserve was built into the 2011 JWST rebaseline in many areas 
including the NGAS contract, IS1M, and also at the JWST Project level. 
This funded schedule reserve is in addition to Unallocated Future 
Expenses (UFE) held by NGAS, ISIM, the JWST Project, and the JWST 
Program. 

In addition, the integrated cost, schedule, and risk analysis (JCL) 
performed as part of the 2011 JWST rebaseline addressed the effect of 
changes in the duration of activities on staffing costs, the largest 
driver of Program cost. In the JCL analysis, the duration of 
activities are simulated as being both shortened and lengthened, 
enabling a calculation of the resultant cost associated with those 
variations in duration. Thus, the JCL analysis provided a defacto 
workforce sensitivity analysis. 

Consequently, because of the funded schedule reserve and UFE built-in 
to the rebaseline and the JCL analysis, NASA considers that there is 
sufficient funded schedule reserve and UFE to cover increases in the 
duration of specific activities that result in increased staffing cost 
and that additional workforce sensitivity analysis is not warranted. 
NASA believes it met the intent of this recommendation and requests it 
be closed without requiring further action. 

1c) Concur. To ensure the ongoing viability of the Project's plan, 
NASA performs monthly integrated programmatic and cost/schedule risk: 
analyses through its assessments and updates of the Project's risks, 
costs and schedule to reflect actual progress and changes to-date. For 
example, JWST conducts ongoing risk management by identifying risks, 
assigning probability to the risks, and determining a dollar value of 
each risk; assesses encumbrances/liens/threats against its UFE; tracks 
actual costs against planned costs to assess the viability of current 
estimates; uses earned value management, including independent 
estimates at complete that are factored into cost estimates; and
performs schedule analyses. NASA uses these tools and methods to 
conduct ongoing programmatic assessment of projects after the baseline 
is established using the JCL methodology at KDP-C. 

Recommendation 2: To ensure that technical risks and challenges are 
being effectively managed and that sufficient oversight is in place 
and can be sustained, GAO recommends that the NASA Administrator 
direct JWST project officials to take the following three actions: 

a) conduct a separate independent review prior to the beginning of the 
OTIS and spacecraft integration and test efforts to allow the 
project's independent standing review board the opportunity to 
evaluate the readiness of the project to move forward, given the lack 
of schedule flexibility once these efforts are underway, 

b) schedule the management review and approval to proceed to 
integration and test (key decision point D or KDP-D) prior to the 
start of observatory integration and test effort, and, 

c) devise an effective, long-term plan for project office oversight of 
its contractors that takes into consideration the anticipated travel 
budget reductions. 

Management's Response: Partially concur. 

2a) Concur. The current Optical Telescope element-Integrated Science 
instrument module (OTIS) integration and test plan includes II Pre-
Environmental Review (PER) prior to environmental testing of OTIS. 
Currently. that review is conducted by II NASA Goddard Space Flight 
Center (GSFC) Independent Review Team (IRT) composed of independent 
technical reviewers from the Center and chaired by the Center's Mission
Assurance organization, reporting their findings to the Center 
Director. In response to this GAO recommendation, JWST Standing Review 
Board (SRB) members will be requested to participate in the OTIS PER 
as well as co-chair the review and report their findings to the 
Decision Authority, the NASA Associate Administrator (AA), as is the
practice for all JWST SRB reviews. 

Because the spacecraft integration is a separate activity taking place 
at a different location than OTIS integration, to not unduly delay 
OTIS or spacecraft: element integration and test, the current plan 
calls for the spacecraft developer (NGAS) to conduct a spacecraft 
element integration readiness review. In response to this GAO 
recommendation, NASA will direct NGAS to add members of the JWST SRB 
to this review as well as include some members of the GSFC IRT and 
report the review findings to the NASA AA. 

The System Integration Review (SIR), a JWST SRB review scheduled for 
2017, will remain a part of the review plan. 

2b) Partially concur. Consistent with NASA policy, the SRB-chaired SIR 
is scheduled prior to. the integration of the JWST science payload 
(OTIS) and the spacecraft element (consisting of the spacecraft bus 
and sunshield) with the SRB providing an independent assessment report 
to the NASA Decision Authority (NASA AA) within one week of the SIR 
prior to the beginning of observatory I&T. To respond specifically to 
this recommendation, the currently scheduled 3-month period between 
the SIR and KDP-D will be reduced, thereby providing NASA management 
and the NASA AA with the full independent assessment earlier than 
currently planned. 

2c) Concur. NASA will develop a plan based on FY2013 travel 
allocations and will take into consideration anticipated travel budget 
reductions. The plan will enable the Project to maintain oversight of 
JWST contractors and their ability to meet performance and delivery 
deadlines and work closely with the Project's international partners. 

Thank you for the opportunity to comment on this draft report. If you 
have any questions or require additional information, please contact 
Ray Taylor at (202) 358-0766. 

Signed by: 

Eric P. Smith, for: 

Geoffrey Yoder: 
JWST Program Director: 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Shelby S. Oakley, Assistant 
Director; Karen Richey, Assistant Director; Richard A. Cederholm; 
Laura Greifner; Cheryl M. Harris; David Hulett; Jason Lee; Kenneth E. 
Patton; Sylvia Schatz; Stacey Steele; Roxanna T. Sun; Jay Tallon; and 
Jade A. Winfree made key contributions to this report. 

[End of section] 

Footnotes: 

[1] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2, 
2009); and GAO Schedule Assessment Guide: Best Practices for Project 
Schedules, [hyperlink, http://www.gao.gov/products/GAO-12-120G] 
(Washington, D. C.: May 30, 2012). 

[2] The electromagnetic spectrum is the wavelengths of all the visible 
and invisible light. The infrared part of the spectrum, also known as 
radiant heat, has wavelengths that go from about 0.75 microns to a few 
hundred microns. The Hubble is designed to operate primarily in the 
ultraviolet and visible wavelengths of the spectrum from 0.1 to 0.8 
microns. Humans cannot see in the ultraviolet region. 

[3] James Webb Space Telescope (JWST) Independent Comprehensive Review 
Panel (ICRP): Final Report (Oct. 29, 2010). 

[4] The hardware configuration created when the Optical Telescope 
Element and the Integrated Science Instrument Module are integrated, 
referred to as OTIS, is not considered a subsystem by NASA, but we 
categorize it as such for ease of discussion. 

[5] A project typically reports to a program within a mission 
directorate, such as the Science Mission Directorate. A similar 
reporting structure was also instituted in the past with both the 
Hubble Space Telescope and the Mars Exploration Program when they 
began experiencing significant cost and schedule issues. For JWST, the 
change was made in response to recommendations from the ICRP. 

[6] The JCL is a quantitative probability analysis that requires the 
project to combine its cost, schedule, and risks into a complete 
quantitative picture to help assess whether the project will be 
successfully completed within cost and on schedule. NASA introduced 
the analysis in 2009, and it is among the agency's initiatives to 
reduce acquisition management risk. The move to probabilistic 
estimating marks a major departure from NASA's prior practice of 
establishing a point estimate and adding a percentage on top of that 
point estimate to provide for contingencies.NASA's procedural 
requirements state that Mission Directorates should plan and budget 
programs and projects based on a 70 percent JCL, or at a different 
level as approved by the Decision Authority--which in the case of the 
JWST project would be the Agency Program Management Council--and any 
JCL approved at less than 70 percent must be justified and documented. 
NASA Procedural Requirements (NPR) 7120.5E, NASA's Space Flight 
Program and Project Management Requirements, paragraph 2.4.4 (Aug. 14, 
2012). 

[7] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[8] A work breakdown structure reflects the requirements and what must 
be accomplished to develop a program, and it provides a basis for 
identifying resources and tasks for developing a program cost 
estimate. The work breakdown structure should be used to define all 
program activities and tasks to ensure that the schedule encompasses 
the entire work. 

[9] Earned value management is a project management tool that 
integrates the technical scope of work with schedule and cost elements 
and compares the value of work accomplished in a given period with the 
value of the work expected in that period. When used properly, earned 
value management can provide objective assessments of project 
progress, produce early warning signs of impending schedule delays and 
cost overruns, and provide unbiased estimates of anticipated costs at 
completion. As a best practice the work breakdown structure should 
match the schedule, cost estimate, and earned value management system 
at a high level so that it clearly reflects the work to be done. 

[10] A summary schedule is derived from the integrated master schedule 
and is often used in a JCL analysis when, for example, time or 
resources needed to simulate the full detailed schedule are limited. 

[11] Level of effort activities are those activities that have no 
measurable output and cannot be associated with a physical product or 
defined deliverable. These activities are typically related to 
management and other oversight that continues until the detailed 
activities they support have been completed. 

[12] An independent cost estimate is conducted by a group outside the 
acquiring organization that uses the same detailed technical 
information as the program estimate to determine whether other 
estimating methods produce similar results. 

[13] [hyperlink, http://www.gao.gov/products/GAO-12-120G]. 

[14] An integrated master schedule constitutes a program schedule that 
includes the entire required scope of effort, including the effort 
necessary from all government, contractor, and other key parties for a 
program's successful execution from start to finish. As a key focal 
point for management, the integrated master schedule documents and 
integrates the planned work, the resources necessary to accomplish 
that work, and the associated budget. It also consists of logically 
related activities whose forecasted dates are automatically 
recalculated when activities change allowing for management to respond 
to the consequences of these changes. 

[15] NASA, Office of Inspector General, NASA's Management of the Mars 
Science Laboratory Project. IG-11-019. (Washington, D.C.: June 2011). 

[16] Mass is a measurement of how much matter is in an object. It is 
related to an object's weight and is mathematically equal to mass 
multiplied by acceleration due to gravity. The project uses mass for 
JWST because when it goes into space, its weight changes with gravity, 
but its mass stays the same. 

[17] NIRCam, NIRSpec, and FGS/NIRISS use detectors fabricated by the 
same vendor; whereas MIRI uses different detectors that do not need to 
be replaced. 

[18] NPR 7120.5E, table 2-5 and appendix A (Aug. 14, 2012). 

[19] Executive Order 13589, Promoting Efficient Spending, The White 
House, November 9, 2011. NASA provided allocations to the Centers 
based on this order. In turn, Goddard conducted an internal travel 
prioritization process and center management determined the allocation 
for JWST and other Goddard projects. 

[20] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. For the cost 
guide, GAO cost experts assessed measures consistently applied by cost-
estimating organizations throughout the federal government and 
industry and considered best-practices for the development of reliable 
cost-estimates. 

[21] NASA, 2008 NASA Cost Estimating Handbook, (2008). 

[22] [hyperlink, http://www.gao.gov/products/GAO-12-120G]. 

[23] NPR 7120.5E (Aug. 14, 2012) and NASA Procedural Requirements 
7123.1A, NASA Systems Engineering Processes and Requirements with 
Change 1. (Nov.4, 2009). 

[End of section] 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]