This is the accessible text file for GAO report number GAO-15-29 
entitled 'Project and Program Management: DOE Needs to Revise 
Requirements and Guidance for Cost Estimating and Related Reviews' 
which was released on November 25, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to the Committee on Armed Services, U.S. Senate: 

November 2014: 

Project and Program Management: 

DOE Needs to Revise Requirements and Guidance for Cost Estimating and 
Related Reviews: 

GAO-15-29: 

GAO Highlights: 

Highlights of GAO-15-29, a report to the Committee on Armed Services, 
U.S. Senate. 

Why GAO Did This Study: 

NNSA is responsible for the nation's nuclear security programs. These 
programs often include the design and construction of large projects 
to meet program needs. NNSA has a history of struggling to complete 
these and other projects and programs within cost estimates. 

Senate Report 112-73 mandated that GAO report on NNSA's cost 
estimating practices. This report examines: (1) the extent to which 
DOE and NNSA cost estimating requirements and guidance for projects 
and programs reflect best practices and (2) the extent to which recent 
NNSA project and program reviews identified cost estimating 
weaknesses, and the extent to which the weaknesses can be attributed 
to not following best practices. GAO reviewed DOE and NNSA cost 
estimating practices and compared them to best practices, NNSA project 
and program reviews, and two programs selected among the largest and 
most complex NNSA programs. GAO also interviewed DOE and NNSA 
officials about requirements and guidance for cost estimates. 

What GAO Found: 

The Department of Energy's (DOE) and its National Nuclear Security 
Administration's (NNSA) cost estimating requirements and guidance for 
projects and programs do not fully reflect best practices for 
developing cost estimates. In regard to cost estimating requirements 
for projects, DOE's 2010 project management order requires 1 of the 12 
cost estimating best practices-—conducting an independent cost 
estimate—-for larger projects at certain stages of development. In 
contrast, DOE's 2011 cost estimating guide describes recommended 
approaches for using 10 of 12 best practices and partially contains 
information about the other 2. Furthermore, because DOE's cost 
estimating guide was issued in 2011—after DOE's 2010 project order was 
issued—it is not referenced in the order. As a result, users of the 
order may not be aware of the guide's availability and may not benefit 
from its usefulness. In addition, although NNSA programs are required 
to follow DOE's budget formulation order and NNSA's budget process, 
both of which require the development of cost estimates, neither the 
order nor the process requires the use of best practices in developing 
the estimates. In February 2014, for example, GAO found that NNSA's 
lifecycle cost estimate for the Plutonium Disposition Program did not 
follow all key steps for developing high-quality cost estimates, in 
part because the agency did not have a requirement to develop a life-
cycle cost estimate. In the absence of a requirement for using best 
practices, it is unlikely that DOE, NNSA, and their contractors will 
consistently develop reliable cost estimates. 

NNSA's project and program reviews issued during fiscal year 2009 
through March 2014 identified cost estimating weaknesses that can be 
attributed to not following best practices. DOE and NNSA require 
independent project reviews, including reviews of cost estimates at 
certain stages of development and at the discretion of project 
managers. Of the 50 reviews GAO analyzed, 39 identified a total of 113 
cost estimating weaknesses. GAO determined that 71 of the 113 
weaknesses—-or about 63 percent-—can be attributed to not following 
four best practices: (1) determining the estimating structure, (2) 
identifying ground rules and assumptions, (3) conducting risk and 
uncertainty analysis, and (4) documenting the estimate. Neither DOE 
nor NNSA, however, requires reviews of program cost estimates. Of the 
three program reviews conducted during fiscal years 2009 to 2013, two 
were of the B61 Life Extension Program, which is to extend the 
operational life of this nuclear weapon. Both reviews identified 
weaknesses in the cost estimates that can be attributed to not 
following three best practices: (1) determining the estimating 
structure, (2) defining program characteristics, and (3) obtaining 
data. In addition, a February 2014 GAO report on NNSA's program to 
dispose of weapons-grade plutonium found that NNSA did not follow 
several cost estimating best practices, such as conducting an 
independent cost estimate and as a result, the program cost estimate 
was not reliable. While the program reviews and GAO's February 2014 
report indicate weaknesses in a few program cost estimates, the extent 
of program cost estimate weaknesses is largely unknown because neither 
DOE nor NNSA requires reviews of program cost estimates. Without a 
requirement for conducting independent program reviews, NNSA does not 
have the internal control necessary for assessing program performance 
over time. 

What GAO Recommends: 

GAO recommends that DOE, among other things, revise DOE requirements 
and guidance that apply to projects and programs to ensure: (1) the 
guidance is referenced in the project management order, (2) DOE and 
NNSA and its contractors develop cost estimates in accordance with the 
12 best practices, (3) its cost estimating guide fully reflects best 
practices, and (4) independent reviews of programs are conducted 
periodically. DOE agreed with the report and its recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-15-29]. For more 
information, contact David Trimble at (202) 512-3841 or 
trimbled@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DOE and NNSA Cost Estimating Requirements and Guidance Generally Do 
Not Reflect Best Practices for Developing Cost Estimates: 

Project Reviews Indicate Cost Estimating Weaknesses and Program 
Reviews Are Not Required and Therefore the Extent of Weaknesses Is 
Largely Unknown: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Comparison of DOE Cost Guide to GAO's Best Practices for 
Cost Estimating: 

Appendix II: Cost Estimating Weaknesses Identified in All Completed 
NNSA Project Reviews February 2009 - February 2014 and GAO's Best 
Practices for Cost Estimating: 

Appendix III: Comments from the Department of Energy: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: DOE and NNSA Critical Decision Review Requirements for 
Projects: 

Table 2: Four Characteristics of a High-Quality Cost Estimate with 
Corresponding Best Practices: 

Figure: 

Figure 1: NNSA Nuclear Security Enterprise: 

Abbreviations: 

CD: critical decision: 

DOE: Department of Energy: 

ICE: independent cost estimate: 

LEP: Life Extension Program: 

M&O: management and operating: 

MOX: Mixed Oxide: 

NNSA: National Nuclear Security Administration: 

OAPM: Office of Acquisition and Project Management: 

PPBE: Planning, Programming, Budgeting, and Evaluation: 

WBS: work breakdown structure: 

[End of section] 

United States Government Accountability Office: 
GAO:
441 G St. N.W. 
Washington, DC 20548: 

November 25, 2014: 

The Honorable Carl Levin: 
Chairman: 
The Honorable James Inhofe: 
Ranking Member: 
Committee on Armed Services: 
United States Senate: 

The National Nuclear Security Administration (NNSA)-—a semiautonomous 
agency within the Department of Energy (DOE)—-is responsible for 
managing the nation's nuclear security missions. These missions 
include maintaining the safety, security, and effectiveness of the 
U.S. nuclear weapons stockpile and reducing the threat posed by 
nuclear proliferation. To achieve its missions, NNSA manages numerous 
programs such as the Stockpile Life Extension Program and the 
Plutonium Disposition Program.[Footnote 1] These and other programs 
frequently rely on the acquisition of capital assets to meet program 
needs.[Footnote 2] For example, to meet the needs of the Plutonium 
Disposition Program, NNSA is currently managing the construction of 
two capital asset projects—-the Mixed Oxide (MOX) Fuel Fabrication 
Facility and the Waste Solidification Building-—at DOE's Savannah 
River Site in South Carolina.[Footnote 3] NNSA has a history of 
struggling to complete these and other projects within their initial 
cost and schedule estimates. Since 1990, DOE's management of major 
contracts and projects, including those executed by NNSA, has been on 
our list of areas at high risk for fraud, waste, abuse, and 
mismanagement.[Footnote 4] We have also recently reported that cost 
estimates for several NNSA projects were inaccurate, and significant 
cost increases were likely. For example, in 2014, we reported that the 
estimated cost of the MOX facility had increased from $4.8 billion in 
April 2007 to $7.7 billion in 2012.[Footnote 5] In that report, we 
recommended, among other things, that DOE and NNSA conduct an analysis 
of the root causes of the increase and identify and prioritize 
recommended solutions. NNSA concurred with this recommendation and 
stated that it was planning to conduct such an analysis. 

To support its missions, NNSA is organized into program offices, which 
include Defense Programs, Defense Nuclear Nonproliferation, and Naval 
Reactors.[Footnote 6] These offices manage a wide variety of programs 
that can be generally placed into one of two categories. The first 
category includes those programs that are ongoing in nature with 
activities that contribute to a broad range of objectives and goals 
across the nuclear enterprise. Programs in this category consist of 
programs that have defined start and end points and a total cost that 
includes all costs incurred to complete the program, like capital 
asset projects programs of this type include NNSA's Science Campaign 
and the Directed Stockpile Work program.[Footnote 7] The second 
category consists of programs that have defined start and end points 
and a total cost that includes all costs incurred to complete the 
program.[Footnote 8] Programs in this category include the B61 Life 
Extension Program (LEP) and the Plutonium Disposition Program. 
Programs in this second category are the programs discussed in this 
report.[Footnote 9] 

In April 2008, to assess the underlying causes of its contract and 
project management challenges, DOE conducted a root-cause analysis, 
which involved identifying the most significant issues that were 
impeding the department's ability to complete projects within their 
cost, scope, and schedule estimates. The analysis identified an 
inadequate cost estimating capability as one of the top five reasons 
that DOE was unable to successfully complete these projects. The 
analysis also identified DOE's lack of cost estimating policy or 
standards, personnel without the appropriate skills, and databases 
that lack current or historical information as root causes 
contributing to cost estimating problems. In July 2008, DOE published 
a corrective action plan to address the deficiencies identified in the 
root-cause analysis. The plan included actions to address cost 
estimating issues by establishing “a federal independent government 
cost estimating capability.” In line with these efforts, DOE and NNSA 
have undertaken a number of activities to improve their cost 
estimating practices, including revising DOE's project management 
order, issuing a supplemental cost estimating guide, and strengthening 
training courses and certification requirements. In February 2011, DOE 
issued a report documenting the actions taken and progress made to 
implement corrective actions. In the report, DOE stated that the 
actions taken effectively mitigated most of the root causes of their 
most significant contract and project management challenges and 
decreased the inherent risks involved with managing large, technically 
complex projects. 

The Senate Report accompanying a proposed bill for the National 
Defense Authorization Act for Fiscal Year 2013 mandated that GAO 
report on NNSA's cost estimating practices and provide 
recommendations, if appropriate, to help NNSA conform to cost 
estimating best practices for projects and programs.[Footnote 10] This 
report examines: (1) the extent to which DOE and NNSA cost estimating 
requirements and guidance for its projects and programs reflect best 
practices for developing and reviewing cost estimates, and (2) the 
extent to which NNSA project and program reviews identified weaknesses 
in cost estimates, and the extent to which any weaknesses can be 
attributed to not following cost estimating best practices. In May 
2013, we testified before the U.S. Senate Subcommittee on Strategic 
Forces, Committee on Armed Services, relying on prior work and on 
preliminary observations from our work in this area.[Footnote 11] 

To examine the extent to which DOE and NNSA cost estimating 
requirements and guidance for its projects and programs reflect best 
practices for developing and reviewing cost estimates, we reviewed DOE 
and NNSA requirements and guidance related to cost estimating for 
projects and programs and compared them with the best practices 
identified in our 2009 Cost Estimating and Assessment Guide. [Footnote 
12] This guide is a compilation of cost estimating best practices 
drawn from across industry and government. We also interviewed DOE and 
NNSA officials at headquarters and NNSA project and program managers 
and their contractors at NNSA's Los Alamos and Sandia National 
Laboratories in New Mexico to discuss the requirements and guidance 
they follow in preparing, documenting, and reviewing project and 
program cost estimates. These NNSA sites were chosen because both 
sites have recently completed or have ongoing projects and are 
involved in the execution of several NNSA programs. We focused on 
projects and programs managed by NNSA's Offices of Defense Programs 
and Defense Nuclear Nonproliferation because they manage a diverse 
array of programs and manage NNSA's largest ongoing capital asset 
projects—these two offices account for approximately 85 percent of NNSA'
s fiscal year 2015 budget request. We also analyzed NNSA program-
specific cost-estimating policies and guidance for the B61 LEP. We 
focused on the B61 LEP and the Plutonium Disposition Program because 
they represent two of the largest and most complex NNSA programs and 
have defined start and end points and a total cost that includes all 
costs incurred to complete the program. 

To determine the extent to which recent NNSA project and program 
reviews identified any weaknesses associated with the cost estimates 
for projects and programs and whether any of these weaknesses can be 
attributed to not following cost estimating best practices, we 
obtained project and program reviews completed from February 2009 
through February 28, 2014. We limited our assessment to those reviews 
conducted since February 2009 to coincide with NNSA's issuance of a 
requirement in February 2009 to conduct independent cost estimates 
(ICE) or independent reviews for certain projects.[Footnote 13] The 
reviews included independent cost assessments, independent project 
reviews, peer reviews, and external independent reviews conducted 
primarily by DOE's Office of Acquisition and Project Management 
(OAPM), NNSA's Office of Acquisition and Project Management, and U.S. 
Army Corps of Engineers, and other subject matter experts including 
former federal contractors.[Footnote 14] We obtained and assessed a 
total of 50 reviews of 21 projects and the results from two program 
reviews. We analyzed these reviews to determine whether they included 
any references to weaknesses in the cost estimates associated with 
these projects and programs, and we determined whether the weaknesses 
could be attributed to not following specific cost estimating best 
practices. When a weakness was noted, we determined whether the 
weakness fit into any of the 12 best practices as described in the 
Cost Estimating and Assessment Guide. As a measure to collect reliable 
data, two GAO analysts reviewed each report independently and compared 
their responses. 

We conducted this performance audit from November 2012 to November 
2014, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

This section describes: (1) NNSA organization and management, (2) DOE 
and NNSA cost estimating requirements and guidance for projects, (3) 
DOE and NNSA cost estimating requirements and guidance for programs, 
(4) our 2009 cost estimating best practices, and (5) federal standards 
for internal controls. 

NNSA Organization and Management: 

To fund NNSA's projects and programs, the President requested 
approximately $11.7 billion for NNSA in the fiscal year 2015 budget 
submission to Congress, the majority of which is intended to fund 
program operations. Work activities on both projects and programs are 
largely carried out by management and operating (M&O) contractors at 
NNSA's eight government-owned, contractor-operated sites--collectively 
referred to as the nuclear security enterprise. Among other things, 
these contractors operate and maintain the government-owned facilities 
and infrastructure deemed necessary to support the nuclear security 
enterprise and to support the capabilities to conduct scientific, 
technical, engineering, and production activities. 

As shown in figure 1, the nuclear security enterprise sites include 
national research and development laboratories, as well as production 
plants. Specifically, NNSA manages three national nuclear weapons 
design laboratories--Lawrence Livermore National Laboratory in 
California, Los Alamos National Laboratory in New Mexico, and Sandia 
National Laboratories in New Mexico and California. It also manages 
four nuclear weapons production plants--the Pantex Plant in Texas, the 
Y-12 National Security Complex in Tennessee, the Kansas City Plant in 
Missouri, and Tritium Operations at DOE's Savannah River Site in South 
Carolina. NNSA also manages the Nevada National Security Site, 
formerly known as the Nevada Test Site. Various headquarters 
organizations within NNSA develop policies and NNSA site offices, 
collocated with NNSA's sites, conduct day-to-day oversight of the M&O 
contractors, and evaluate the M&O contractors' performance in carrying 
out the sites' missions.[Footnote 15] 

Figure 1: NNSA Nuclear Security Enterprise: 

[Refer to PDF for image: illustrated U.S. map] 

Kansas City Plant (Kansas City, MO); 
Lawrence Livermore National Laboratory (Livermore, CA); 
Los Alamos National Laboratory (Los Alamos, NM); 
Nevada National Security Site (Mercury, NV); 
Pantex Plant (Amarillo, TX); 
Sandia National Laboratories (Albuquerque, NM; Livermore, CA); 
Savannah River Site-Tritium Operations (Aiken, SC); 
Y-12 National Security Complex (Oak Ridge, TN). 

Sources: National Nuclear Security Administration; Map Resources 
(map). GAO-15-29. 

[End of figure] 

The Secretary of Energy is authorized under the National Nuclear 
Security Administration Act to establish policy and provide direction 
to NNSA.[Footnote 16] The NNSA Administrator, however, is also vested 
with the authority to establish NNSA-specific policies, unless 
disapproved by the Secretary of Energy. NNSA does this through the 
issuance of Policy Letters.[Footnote 17] According to NNSA officials 
responsible for project management, DOE directives and orders and NNSA 
Policy Letters, including business operating procedures, collectively 
establish requirements for the development of NNSA project and program 
cost estimates, and we refer to them as such throughout this document. 

DOE and NNSA Cost Estimating Requirements and Guidance for Projects: 

NNSA is required to manage its projects, including the development of 
cost estimates, in accordance with DOE Order 413.3B.[Footnote 18] The 
purpose of the order is to provide program and project management 
direction for the acquisition of capital assets, and it includes 
requirements for developing project cost estimates. The order also 
provides management direction for NNSA and other DOE offices with the 
goal of delivering projects within the original performance baseline 
that are fully capable of meeting mission performance and other 
requirements such as environmental, safety, and health 
standards.[Footnote 19] The order defines five major milestones--or 
critical decision points--that span the life of a project. These 
critical decision (CD) points are: 

* CD-0: Approve mission need. 

* CD-1: Approve alternative selection and cost range. 

* CD-2: Approve project performance baseline. 

* CD-3: Approve start of construction. 

* CD-4: Approve start of operations or project completion. 

* The order specifies requirements that must be met, along with the 
documentation necessary, to move a project past each CD point. In 
addition, the order requires senior management to review the 
supporting documentation and decide whether to approve the project at 
each CD. DOE also provides suggested approaches for meeting the 
requirements contained in Order 413.3B through a series of guides, 
such as guides for project reviews and risk management. 

* The order includes several cost estimating requirements for 
projects. Specifically, the order requires: 

* a cost estimate be provided at each CD point; 

* the degree of rigor and detail for a cost estimate be carefully 
defined, depending on the degree of confidence in project scale and 
scope that is reasonable to expect at that stage; 

* a cost estimate range for CD-0 and 1 explicitly noting any relevant 
caveats concerning uncertainties; and: 

* a program sponsor never be the sole cost estimator at any critical 
decision given the inherent conflict of interest, and a second cost 
estimator should come from outside of the line manager's chain of 
command to avoid conflict of interest. 

* The order also establishes requirements for the review of cost 
estimates for certain projects at CD-0 through CD-3, depending on the 
estimated cost of the project. Table 1 shows these requirements and an 
NNSA requirement for review. 

Table 1: DOE and NNSA Critical Decision Review Requirements for 
Projects: 

Critical decision (CD): CD-0; 
Action required: Independent cost review; 
Source of requirement: DOE Order 413.3B; 
Office responsible[A]: Office of Acquisition and Project Management 
(OAPM); 
Required for projects with a total estimated cost of $750 million or 
greater, or for projects as designated by the DOE (or NNSA) 
acquisition executive responsible for the project. 

Critical decision (CD): CD-1; 
Action required: Independent cost estimate OR independent cost review; 
Source of requirement: DOE Order 413.3B; 
Office responsible[A]: OAPM; 
Required for projects with a total estimated costs of $100 million or 
greater or as OAPM deems appropriate. 

Critical decision (CD): CD-2; 
Action required: Independent cost estimate; 
Source of requirement: DOE Order 413.3B; 
Office responsible[A]: OAPM; 
Required for projects with a total estimated costs of $100 million or 
greater. 

Critical decision (CD): CD-2; 
Action required: Independent cost review OR independent cost estimate; 
Source of requirement: National Nuclear Security Administration (NNSA) 
Business Operating Procedure 06.03; 
Office responsible[A]: NNSA Office of Acquisition and Project 
Management; 
Independent cost estimate or independent cost review required for 
projects with a total estimated cost greater than $20 million and less 
than $100 million[B]. 

Critical decision (CD): CD-3; 
Action required: Independent cost estimate; 
Source of requirement: DOE Order 413.3B; 
Office responsible[A]: OAPM; 
Required for projects with a total estimated cost greater than or 
equal to $100 million if warranted by risk and performance indicators, 
or as designated by the DOE (or NNSA) acquisition executive 
responsible for the project[C]. 

Source: GAO. GAO-15-29. 

[A] Order 413.3B currently refers to the Office of Engineering and 
Construction Management as the office responsible for conducting these 
independent reviews. OECM was replaced as part of DOE's reorganization 
in June 2012 by OAPM. 

[B] NNSA Business Operating Procedure 06.03, Independent Cost 
Estimating Procedure, February 27, 2014. According to the procedure, 
the decision of whether to conduct an ICE or an independent cost 
review is determined by NNSA's Director of Enterprise Project 
Management, which is in NNSA's Office of Acquisition and Project 
Management, in consultation with the NNSA acquisition executive 
responsible for the project. 

[C] In addition to the ICE requirements in Order 413.3B for CD-3, the 
Consolidated Appropriations Act, 2012, prohibited expenditure of 
fiscal year 2012 funds to approve CD-2 or CD-3 for construction 
projects with a total estimated cost of $100 million or greater prior 
to the development of an ICE. DOE officials treat this as an ongoing 
requirement. 

[End of table] 

As a supplement to Order 413.3B, DOE provides guidance for the 
development of cost estimates in its Cost Estimating Guide, which was 
issued in May 2011.[Footnote 20] According to the guide, its purpose 
is to provide uniform guidance and best practices that describe 
recommended methods and procedures that projects and programs could 
use for preparing cost estimates. The guide states that it is based on 
accepted industry practices and processes, including practices 
outlined in our 2009 Cost Estimating and Assessment Guide, to meet 
federal and DOE requirements.[Footnote 21] 

DOE and NNSA Cost Estimating Requirements and Guidance for Programs: 

DOE and NNSA cost estimating requirements and guidance for programs 
are primarily defined in four documents.[Footnote 22] 

* DOE Order 130.1. This order requires all DOE and NNSA program budget 
requests be based on cost estimates that have been thoroughly reviewed 
and deemed reasonable prior to their inclusion in department budgets. 
[Footnote 23] 

* DOE Cost Estimating Guide 413.3-21. DOE's cost estimating guide is 
primarily intended for use in managing the phases involved in the 
acquisition of a capital asset project. However, the guide states that 
it also could be used by DOE and NNSA programs in preparing cost 
estimates. 

* NNSA Planning, Programming, Budgeting, and Evaluation (PPBE) 
Requirements. NNSA's PPBE process provides a framework for the agency 
to plan, prioritize, fund, and evaluate its program activities. This 
process includes a budget validation step to review the process 
contractors follow in developing program cost estimates for budgeting 
purposes.[Footnote 24] 

* NNSA Phase 6.X Process. NNSA manages life extension programs, 
including the B61 LEP, according to a process called Phase 6.X that 
was developed by DOE in collaboration with the Department of Defense. 
[Footnote 25] The Phase 6.X process provides NNSA with a framework to 
conduct and manage LEP activities, including the development and 
documentation of a program cost estimate. 

GAO Cost Estimating Best Practices: 

Drawing from federal cost estimating organizations and industry, our 
cost estimating guide provides best practices about the processes, 
procedures, and practices needed for ensuring development of high-
quality--that is, reliable cost estimates.[Footnote 26] A high-quality 
cost estimate helps ensure that management is given the information it 
needs to make informed decisions. The guide identifies the following 
four characteristics of a high-quality cost estimate. Specifically, 
such an estimate is: 

* Credible when it has been cross-checked with an ICE, the level of 
confidence associated with the point estimate has been identified 
through the use of risk and uncertainty analysis, and a sensitivity 
analysis has been conducted;[Footnote 27] 

* Well-documented when supporting documentation is accompanied by a 
narrative explaining the process, sources, and methods used to create 
the estimate and contains the underlying data used to develop the 
estimate; 

* Accurate when it is not overly conservative or too optimistic and 
based on an assessment of the costs most likely to be incurred; and: 

* Comprehensive when it accounts for all possible costs associated 
with a project, it contains a cost estimating structure in sufficient 
detail to ensure that costs are neither omitted nor double-counted, 
and the estimating teams' composition is commensurate with the 
assignment. 

To develop a cost estimate that embodies these four characteristics, 
our cost estimating guide lays out 12 best practice steps. For 
example, one step--determining the estimating structure--includes the 
need to develop a "product-oriented" work breakdown structure (WBS) 
that reflects the requirements and basis for identifying resources and 
tasks necessary to accomplish the project's objectives. A product-
oriented WBS is organized to reflect the cost, schedule, and technical 
performance of project components. Such a WBS allows a project to 
track cost by defined deliverables, promote accountability by 
identifying work products that are independent of one another, and 
provides a basis for identifying resources and tasks for developing a 
cost estimate. Table 2 shows the four characteristics of a high-
quality cost estimate with the corresponding steps. 

Table 2: Four Characteristics of a High-Quality Cost Estimate with 
Corresponding Best Practices: 

Characteristic: Credible; 
12 best practice steps: 
* Compare the point estimate to an independent cost estimate[A]; 
* Conduct sensitivity analysis; 
* Conduct risk and uncertainty analysis. 

Characteristic: Well-documented; 
12 best practice steps: 
* Define the estimate's purpose, scope, and schedule; 
* Define the program's characteristics; 
* Identify ground rules and assumptions; 
* Obtain the data; 
* Document the estimate; 
* Present the estimate to management for approval. 

Characteristic: Accurate; 
12 best practice steps: 
* Develop the point estimate[A]; 
* Update the estimate to reflect actual costs and changes. 

Characteristic: Comprehensive; 
12 best practice steps: 
* Develop the estimating plan; 
* Determine the estimating structure. 

Source: GAO. GAO-15-29. 

[A] As described in the GAO Cost Estimating and Assessment Guide, 
developing the point estimate and comparing it with an independent 
cost estimate are separate parts of a single step. For purposes of 
assessing the extent to which a cost estimate achieves the 
characteristics of a high-quality cost estimate, developing the point 
estimate contributes to accuracy, and comparing the point estimate 
with an independent cost estimate contributes to credibility. 

[End of table] 

Federal Standards for Internal Controls: 

In addition to the requirements for cost estimating established by DOE 
and NNSA, according to Standards for Internal Control in the Federal 
Government, federal agencies are to employ internal control 
activities, such as functional reviews by management of projects and 
programs to compare actual performance to planned or expected results 
throughout the organization and analyze significant differences. Such 
reviews, which may include reviews of program cost estimates, are to 
help ensure that management's directives are carried out and to 
determine if agencies are effectively and efficiently using resources. 
[Footnote 28] 

DOE and NNSA Cost Estimating Requirements and Guidance Generally Do 
Not Reflect Best Practices for Developing Cost Estimates: 

DOE and NNSA cost estimating requirements and guidance for projects 
and programs generally do not reflect best practices for developing 
cost estimates. For projects, DOE requires one best practice, and 
DOE's cost estimating guide does not fully reflect cost estimating 
best practices, and is not mandatory. For programs, DOE does not 
require the use of any of the best practices to develop cost 
estimates, and NNSA has taken steps to close gaps in the existing cost 
estimating framework for programs, such as creating a cost analysis 
office, but these steps have not included a requirement to use best 
practices. 

Projects Are Not Required to Meet Most Best Practice Steps, and 
Guidance Describes Most Best Practice Steps but Is Not Mandatory: 

DOE and NNSA cost estimating requirements and guidance for projects 
generally do not reflect best practices for developing cost estimates. 
DOE's 2010 project management order requires the use of only 1 of the 
12 cost estimating best practice steps. Specifically, the order 
requires an ICE be prepared at CD-2 and CD-3 for projects with an 
estimated cost of $100 million or greater. The order requires the 
development of an ICE at CD-3 if warranted by risk and performance 
indicators or as designated by DOE or NNSA management. In addition, 
NNSA's 2014 requirement for an ICE or independent cost review could 
subject additional projects with an estimate of a cost of less than 
$100 million to an ICE, but this would depend on whether NNSA chooses 
to conduct an ICE rather than the less rigorous independent cost 
review. None of the other cost estimating requirements in the order, 
such as the need for a cost estimate at each CD point, ensure that 
project cost estimates will be prepared in accordance with cost 
estimating best practices. For example, the order does not require any 
of the other 11 best practice steps such as conducting a risk and 
uncertainty analysis, identifying ground rules and assumptions, 
documenting the estimate, developing a point estimate, or determining 
the estimating structure. According to the DOE officials responsible 
for developing DOE's project management order, DOE has chosen to not 
require all cost estimating best practices in the order and instead 
includes suggested approaches for developing cost estimates in the DOE 
cost estimating guide that accompanies the order. However, because 
neither DOE nor NNSA requires the use of most cost estimating best 
practices for its projects, it is unlikely that NNSA and its 
contractors will consistently develop reliable cost estimates. 

DOE's 2011 cost estimating guide describes most best practices, but it 
is not mandatory and it is not referenced in the order. We found that 
the guide fully or substantially describes 10 of the 12 best 
practices. However, the guide only partially or minimally contains 
information about the other 2 best practice steps--determining the 
estimating structure and conducting a sensitivity analysis.[Footnote 
29] For the estimating structure, the guide only partially describes 
how to develop this structure, which primarily involves establishing 
an adequate WBS. For example, we found that the guide describes the 
need for a WBS, but it does not describe the need for a product-
oriented WBS and does not provide enough information on how to create 
a WBS.[Footnote 30] For the sensitivity analysis, the guide only 
minimally describes the information needed. For example, the guide 
suggests briefing management on the results of a sensitivity analysis, 
but it does not address the best practice steps on how to perform a 
sensitivity analysis. As a result, DOE and NNSA have not provided its 
contractors with all the detailed guidance needed to consistently 
develop reliable cost estimates. When we discussed our analysis of 
this guide and the shortcomings we found in regard to the guide with 
the DOE officials responsible for developing the guide, they said that 
they would review our analysis and consider revisions to the guide as 
appropriate. Appendix I provides a comparison of the extent to which 
DOE's cost guide contains information on the 12 best practice steps. 

In addition to the shortcomings in the description for two of the best 
practices, the use of the guide is not mandatory and, as a result, 
neither NNSA staff nor its contractors who develop cost estimates are 
required to use it. The guide states that it does not impose any new 
cost estimating requirements or establish departmental policy; it 
describes only suggested recommended approaches for preparing cost 
estimates. Further, DOE's cost estimating guide is not referenced in 
the order. To supplement DOE's project management order, DOE has 
issued a series of guides that provide suggested approaches to meeting 
the requirements, some of which are specified in the order. There are 
a total of 18 separate guides, including guides that provide suggested 
approaches on topics such as systems engineering, earned value 
management systems, and project closeout.[Footnote 31] The order 
references some of these guides as appropriate so that the user of the 
order is made aware of the existence of the suggested approaches that 
are included in the guides. However, because the cost guide was issued 
in 2011, a year after DOE's latest version of the order was issued, 
the order contains no reference to this guide that a preparer of a 
cost estimate could use in developing an estimate, or that a reviewer 
could use in reviewing one, and users of the order may not be aware of 
the guide's availability and may not benefit from its usefulness. 

Programs Are Not Required to Meet Any Cost Estimating Best Practices: 

DOE and NNSA programs are not required to meet any cost estimating 
best practices. NNSA officials explained that NNSA cost estimating 
practices for programs are limited, de-centralized, and inconsistent, 
and are not governed by a cost estimating policy or single set of NNSA 
requirements and guidance. According to these officials, each NNSA 
program office uses different practices and procedures for the 
development of cost estimates that are included in the NNSA annual 
budget. In addition, these officials also explained that, while there 
are no specific requirements on how program cost estimates should be 
developed or reviewed, all NNSA programs must follow DOE's budget 
formulation order and NNSA's PPBE process, but the order does not 
require, and the PPBE process does not direct the use of best 
practices in developing cost estimates. In 2012, we reported on NNSA's 
PPBE process, and we noted significant deficiencies in NNSA's 
implementation of its PPBE process.[Footnote 32] Specifically, we 
found that the process does not ensure the development of fully 
credible program cost estimates, and that the review of the cost 
estimates supporting DOE's budget was not sufficiently thorough to 
ensure the credibility and reliability of NNSA's budget estimates. We 
also found that NNSA was not following DOE's budget formulation order 
that requires that budget requests be based on cost estimates that 
have been thoroughly reviewed and deemed reasonable by the cognizant 
program organization. NNSA officials told us that they do not follow 
DOE's budget formulation order because it expired in 2003. 
Additionally, according to NNSA officials, NNSA's trust in its 
contractors minimizes the need for formal review of its budget 
estimates. 

In our 2012 report on the NNSA's PPBE process we recommended that, 
among other things, DOE update the departmental order for budget 
reviews, improve the formal process for reviewing budget estimates, 
and reinstitute an independent analytical capability to provide senior 
decision makers with independent program reviews. The agency agreed in 
principle with these recommendations, but it has not taken action to 
fully implement them. According to NNSA officials, as of August 2014, 
NNSA has suspended the budget review process and several options for 
replacing this process are under review. 

In the absence of a DOE or NNSA requirement for programs to follow 
cost estimating best practices, we found that managers for the 
programs we reviewed--the B61 LEP and the Plutonium Disposition 
Program--used different processes in developing cost estimates for 
these programs. Specifically: 

* B61 LEP. NNSA managers for the B61 LEP use the Phase 6.X Process 
which, among other things, involves the direction to establish a cost 
and schedule baseline for the program. However, according to the NNSA 
B61 LEP project manager, since the Phase 6.X Process does not define 
how cost estimates should be developed, the B61 management team 
developed an approach for developing cost estimates for the program 
using various sources, including direction under the Phase 6.X 
Process, as well as DOE's project management order and cost guide and 
our cost-estimating guide. This effort resulted in the management team 
producing a document that defines the strategy and provides guidance 
for completing the cost estimate for the B61 LEP. We reviewed this 
document and found that it does not stipulate that NNSA program 
managers or its contractors must follow any DOE or NNSA requirements 
or guidance for the development of a program cost estimate when 
developing the estimate for the B61 LEP. 

* Plutonium Disposition Program. In February 2014, we found that 
NNSA's life-cycle cost estimate for the Plutonium Disposition Program 
did not follow all key steps for developing high-quality cost 
estimates in part because it did not have a requirement to develop a 
life-cycle cost estimate[Footnote 33]. According to NNSA officials, 
DOE's project management order includes requirements for development 
of cost and schedule estimates for a capital asset project, such as 
the MOX facility or Waste Solidification Building, but does not 
specify equivalent requirements for a program like Plutonium 
Disposition, which includes multiple projects as well as supporting 
activities. As a result, when developing the life-cycle cost estimate 
for the Plutonium Disposition Program, NNSA officials used an ad hoc 
approach to adapt DOE requirements for managing projects in DOE's 
project management order. NNSA officials also said that its April 2013 
life-cycle cost estimate did not include all the steps of a high-
quality, reliable estimate in part because NNSA considered the 
estimate to be draft and therefore had not fully implemented plans for 
developing it. We recommended in this report that DOE conduct a root 
cause analysis of the Plutonium Disposition Program's cost increases 
and ensure that future estimates of the program's life-cycle cost and 
schedule for the program meet all best practices for reliable 
estimates. We also recommended that DOE revise its project management 
order to require life-cycle cost estimates covering the full cost of 
programs that include both construction projects and other efforts and 
activities not related to construction. DOE concurred with 
recommendations to analyze the programs cost increases and revise and 
update the program life-cycle cost estimate following best practices. 
DOE did not agree to update its project management order to require 
life-cycle cost estimates of programs. However, we continue to believe 
that this recommendation has merit and should be fully implemented. 

* NNSA took some actions to improve independent review and analysis of 
program estimates. For example, in April 2013, NNSA created the Office 
of Program Review and Analysis. According to NNSA, this office is 
intended to improve NNSA's ability to plan and budget by providing 
senior leadership independent advice on resource allocations to ensure 
the best use of the agency's resources, including evaluating cost 
estimates of NNSA projects and programs. 

* DOE and NNSA began taking some actions to improve cost estimating. 
For example, DOE OAPM has embarked on a Cost Estimating and Scheduling 
Initiative to systematically improve DOE's policies and guidance. Some 
of the near term efforts include development of a Life-Cycle Cost 
handbook, a Key Performance Parameters and Statement of Work handbook, 
and Analysis of Alternatives guidance. In addition, NNSA's Office of 
Defense Programs has taken steps to fill the existing gaps in the cost 
estimating framework for programs. For example, in 2011, Defense 
Programs established an Office of Cost Policy and Analysis to provide 
it with a cost analysis capability. In March 2014, NNSA's Office of 
Program Integration issued a cost estimating improvement plan, which 
includes proposed guidance for conducting cost estimate briefings to 
the Office of Defense Programs Assistant Deputy Administrator, 
establishing a defense programs database, and implementing various 
process improvements to improve cost accounting and performance. While 
these efforts are intended to improve, among other things, cost 
estimating practices, none of these efforts include establishing 
requirements to follow cost estimating best practices. 

Project Reviews Indicate Cost Estimating Weaknesses and Program 
Reviews Are Not Required and Therefore the Extent of Weaknesses Is 
Largely Unknown: 

NNSA's project and program reviews for fiscal years 2009 to 2014 
identified cost estimating weaknesses that can be attributed to not 
following best practices. DOE and NNSA require reviews of projects, 
including reviews of cost estimates at various CD points and at the 
discretion of project managers; however, because DOE and NNSA do not 
require reviews of program cost estimates, the extent of weaknesses in 
program cost estimates is largely unknown. 

Project Reviews Identified Cost Estimating Weaknesses That Can Be 
Attributed to Not Following Best Practices: 

Of the 50 NNSA project reviews conducted from February 2009 through 
February 2014, 39 identified a total of 113 cost estimating 
weaknesses. We determined that 71 of the 113 weaknesses--or about 63 
percent--can be attributed to not following 4 of the 12 best practice 
steps. These four steps are: 

* determining the estimating structure, 

* identifying ground rules and assumptions, 

* conducting risk and uncertainty analysis, and: 

* documenting the estimate. 

* For example, as part of this review, we examined NNSA's 2012 
independent review of the Nuclear Material Safeguards and Security 
Upgrades Project at Los Alamos National Laboratory and it identified 
three weaknesses, one of which can be attributed to not following the 
best practice of updating the estimate to reflect actual costs and 
changes. In particular, the review found deficiencies in the estimate 
that could lead to an inaccurate cost and schedule estimate for the 
project. In another example, a 2012 review of NNSA's Waste 
Solidification Building identified four weaknesses, including not 
following the best practice of conducting risk and uncertainty 
analysis. Several other reviews that we examined found similar 
weaknesses in project cost estimating. For example, the 2012 
Independent Project Review of the MOX Fuel Fabrication Facility at the 
Savannah River Site identified flaws in the project's cost estimate 
that include: (1) basic assumptions concerning project reserves and 
contingency cost; (2) data reliability of cost information considering 
the design maturity of the project; (3) the sensitivity analysis; and 
(4) the project risk analysis. The review also concluded that the most 
significant risk to the project's success was NNSA's under-estimation 
of the cost to complete the facility. 

* We also found that the remaining 42 weaknesses could be attributed 
to not following seven other best practice steps. These best practices 
are: 

* developing an estimating plan, 

* defining program characteristics, 

* obtaining data, 

* developing a point estimate and comparing it to an ICE, 

* conducting a sensitivity analysis, 

* presenting the estimate to management for approval, and: 

* updating the estimate to reflect actual costs and changes. 

Appendix II includes the cost estimating weaknesses we identified for 
each of the project reviews we assessed and the best practice step not 
followed to which the weakness can be attributed to not following. 

Program Reviews Are Limited to LEPs So the Extent of Cost Estimating 
Weaknesses in NNSA Programs Is Largely Unknown: 

DOE and NNSA do not require reviews of programs, including reviews of 
program cost estimates. As a result, reviews of cost estimates for 
programs are limited, and the extent to which program cost estimates 
have weaknesses is largely unknown. While program reviews are not 
required, we identified and analyzed three LEP program reviews from 
the beginning of fiscal year 2009 through February 2014, two of which 
were reviews of the B61 LEP. These identified several weaknesses in 
the cost estimates for this program that can be attributed to not 
following three best practice steps: (1) obtaining data, (2) defining 
program characteristics, and (3) determining the estimating structure. 
For example, both reviews identified weaknesses in obtaining the data 
needed to compile the B61 cost estimate. In addition, one of the 
reviews noted that the estimate was not based on a sound program 
definition, while the other review stated that a standard WBS was not 
used to develop the B61 LEP cost estimate. Both reviews concluded that 
the B61 cost estimate is inaccurate, with one review noting that the 
program will cost approximately $3.6 billion more than NNSA's 2011 
$6.5 billion Weapon Design and Cost Report estimate. 

While we did not identify any additional reviews of cost estimates for 
NNSA programs, we reported in February 2014 on the results of our 
review of the cost estimates for the Plutonium Disposition Program. 
[Footnote 34] We concluded that the life-cycle cost estimate for the 
overall program was not reliable and found that it did not fully 
reflect the characteristics of high-quality, reliable estimates as 
established by best practices. Specifically, in developing its April 
2013 draft life-cycle cost estimate of $24.2 billion for the Plutonium 
Disposition Program, we found that NNSA followed several cost 
estimating best practices; including obtaining the data, defining the 
estimate's purpose, and defining the program's characteristics. 
However, we also found that NNSA did not follow other key steps such 
as conducting an independent cost estimate and, as a result, the 
estimate was not reliable. We recommended, among other things, that 
NNSA revise and update the program's life-cycle estimate using cost 
estimating best practices, including conducting an independent cost 
estimate. NNSA agreed with this recommendation. 

Without a requirement for conducting program reviews, NNSA does not 
have the appropriate internal controls necessary for assessing program 
performance. According to the Standards for Internal Control in the 
Federal Government, federal agencies are to employ internal control 
activities, such as functional reviews by management of projects and 
programs to compare actual performance to planned or expected results 
throughout the organization and analyze significant differences. Such 
reviews are to help ensure that management's directives are carried 
out and to determine if agencies are effectively and efficiently using 
resources. 

Conclusions: 

DOE and NNSA have taken action in recent years to improve cost-
estimating practices, including the corrective actions that were 
implemented as part of the department's 2008 root-cause analysis, 
which DOE later reported as having effectively mitigated most of the 
root causes of the most significant contract and project management 
challenges. Nonetheless, NNSA continues to struggle with significant 
cost overruns on its major projects. Because DOE does not require the 
use of most of the 12 cost estimating best practices for its projects 
and programs, it is unlikely that NNSA and its contractors will 
consistently develop reliable cost estimates. In addition, while DOE 
has developed a cost estimating guide, it does not fully describe all 
12 cost estimating best practices. As a result, DOE and NNSA have not 
provided its contractors with all the detailed guidance needed to 
consistently develop reliable cost estimates. Also, because DOE Order 
413.3B has not been updated since 2010, it omits reference to the 
supplemental cost estimating guide; users of the order may not be 
aware of the guide's availability and may not benefit from its 
usefulness. Finally, without a requirement for conducting reviews of 
programs with project-like characteristics, including the life-cycle 
cost estimates of these programs, neither DOE nor NNSA have 
appropriate internal controls to assess the quality of program 
performance over time. 

Recommendations for Executive Action: 

To enhance NNSA's ability to develop reliable cost estimates for its 
projects and for its programs that have project-like characteristics, 
we recommend the Secretary of Energy take the following five actions: 

* Revise DOE's project management order to require that DOE, NNSA, and 
its contractors develop cost estimates in accordance with the 12 cost 
estimating best practices. 

* Revise DOE's cost estimating guide so that it fully reflects the 12 
cost estimating best practices. 

* Revise DOE's project management order to include references to the 
DOE cost estimating guide, where applicable. 

* Revise DOE directives that apply to programs to require that DOE and 
NNSA and its contractors develop cost estimates in accordance with the 
12 cost estimating best practices, including developing life-cycle 
cost estimates for programs. 

* Revise DOE requirements and guidance that apply to programs to 
ensure that program reviews are conducted periodically, including 
reviews of the life-cycle cost estimates for programs. 

Agency Comments: 

We provided DOE with a draft of this report for its review and 
comment. In its written comments, reproduced in appendix III, DOE 
agreed with the report's recommendations. In regard to the report's 
first recommendation--revise DOE's project management order to require 
that DOE, NNSA, and its contractors develop cost estimates in 
accordance with the 12 cost estimating best practices--DOE stated in 
its written comments that DOE's order for project management (DOE O 
413.3B) will be assessed for revision following the issuance of the 
revision to DOE-STD-1189, Integration of Safety into the Design 
Process, currently scheduled for November 2016. DOE stated that it has 
established a log of issues to be addressed when revising Order 
413.3B, which includes this recommendation. DOE also stated that this 
recommendation will be fully considered during this revision process. 
In the interim, DOE stated that its cost estimating guide incorporates 
the 12 cost estimating best practices, albeit not in the same format 
as our guidance. DOE also stated that it has begun internal efforts 
for the publication of a Departmental Cost Estimating and Schedule 
Policy and that the policy will complement existing cost estimating 
guidance and will incorporate our cost estimating guidance. DOE stated 
that the time frame for this policy to be issued is still to be 
determined. Further, DOE stated that DOE's Office of Acquisition and 
Project Management will continue to incorporate the 12 cost estimating 
best practices in its independent cost estimating activities. 
Additionally, DOE explained that the curriculum of the Project 
Management Career Development Program requires cost and schedule 
estimating courses and that this training incorporates the 12 cost 
estimating best practices. We are pleased that DOE agreed with our 
recommendation and that it has interim measures to improve cost 
estimating before it plans to implement the recommendation. However, 
while these may be useful interim measures, the unspecified, open-
ended date for updating the project management order that contains 
requirements (i.e., sometime after November 2016) and the statement 
that the cost estimating best practices will be considered, not 
incorporated, may indicate DOE's lack of urgency or concern about the 
need to implement this recommendation. 

In regard to the report's second recommendation--revise DOE's cost 
estimating guide so that it fully reflects the 12 cost estimating best 
practices--DOE stated in its written comments that it will begin 
updating the cost estimating guide in the first quarter of fiscal year 
2015. The update efforts are to elevate the significance of the 12 
cost estimating best practices within the content of the guide, 
although DOE did not specify when it might complete this update. We 
are pleased that DOE agreed with our recommendation and that it plans 
to take action during the current quarter of this fiscal year. 
However, we are concerned that in its written comments DOE did not 
specify whether it plans to revise the guide to better include the two 
areas in the guide we found were deficient and that the lack of a 
completion date may indicate DOE's lack of urgency or concern about 
the need to implement this recommendation. 

In regard to the report's third recommendation--revise DOE's project 
management order to include references to the DOE cost estimating 
guide, where applicable--DOE stated in its written comments that its 
project management order will be assessed for revision following the 
issuance of the revision to DOE-STD-1189, currently scheduled for 
November 2016, and that this recommendation will be fully considered 
during the order revision process. In the interim, DOE stated that its 
directives website points to the guides that accompany Order 413.3B as 
the best practices guidance for implementation of Order 413.3B 
requirements. We are pleased that DOE agreed with our recommendation 
and that its website includes links to the guides that accompany the 
project management order. However, our report noted the deficiencies 
associated with the existing cost estimating guide, and DOE did not 
specify plans for addressing those deficiencies. In addition, we are 
concerned that the unspecified, open-ended date for updating the 
project management order that contains requirements (i.e., sometime 
after November 2016) and the statement that referencing the cost 
estimating guide will be considered, not completed, may indicate DOE's 
lack of urgency or concern about the need to implement this 
recommendation. 

In regard to the report's fourth recommendation--revise DOE directives 
that apply to programs to require that DOE and NNSA and its 
contractors develop cost estimates in accordance with the 12 cost 
estimating best practices, including developing lifecycle cost 
estimates for programs-
-DOE stated in its written comments that it is in the process of 
substantially revising the existing 1995 DOE Order 130.1, Budget 
Formulation, and that as part of this effort, DOE will assess the 
requirement for program cost estimates and will revise the order to 
provide more specificity on the cost estimating requirements. Further, 
DOE stated that the revised order will (1) define which DOE and NNSA 
program budget requests require cost estimates and (2) clarify that 
cost estimates for program budget submissions shall be conducted in 
accordance with the DOE cost estimating guide (or its successor 
policy). DOE estimated that it would complete this process in 
September 2016. We are pleased that DOE agreed with our recommendation. 

In regard to the report's fifth recommendation--revise DOE 
requirements and guidance that apply to programs to ensure that 
program reviews are conducted periodically, including reviews of the 
lifecycle cost estimates for programs--DOE again stated that it is in 
the process of substantially revising the existing DOE Order 130.1, 
and that as part of this effort, the department will assess 
requirements for program reviews and the linkage between program 
reviews and the budget formulation process. Further, DOE stated that 
the revised order will clarify requirements for program reviews and 
specify how such reviews-
-to include lifecycle cost estimates--can best support the budget 
formulation process. NNSA is to review these requirements and adjust 
NNSA-specific policies and guidance as appropriate. DOE estimated that 
it would complete this process in September 2016. We are pleased that 
DOE agreed with our recommendation. 

We are sending copies of this report to the appropriate congressional 
committees; the Secretary of Energy; the Director, Office of 
Management and Budget; and other interested parties. In addition, the 
report is available at no charge on the GAO website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff members have any questions about this report, 
please contact me at (202) 512-3841 or trimbled@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff who made major 
contributions to this report are listed in appendix IV. 

Signed by: 

David C. Trimble: 
Director, Natural Resources and Environment: 

[End of section] 

Appendix I: Comparison of DOE Cost Guide to GAO's Best Practices for 
Cost Estimating: 

12 key steps/best practices: 

Define estimate's purpose, scope, and schedule: Fully. 

Develop the estimating plan: Fully. 

Define the program: Substantially. 

Determine the estimating approach--work breakdown structure (WBS): 
Partially. 

Identify the ground rules and assumptions: Substantially. 

Obtain the data: Substantially. 

Develop the point estimate and compare it to an independent cost 
estimate: Fully. 

Conduct sensitivity analysis: Minimally. 

Conduct risk and uncertainty analysis: Substantially. 

Document the estimate: Fully. 

Present the estimate to management: Substantially. 

Update the estimate to reflect actual costs and changes: Fully. 

Source: GAO analysis of DOE Guide 413.3-21 (2011). GAO-15-29: 

Note: The ratings we used in this analysis are as follows: "Fully" 
means that the guide included information that satisfied the best 
practice; "Substantially" means that the guide included the majority 
of the information to satisfy the best practice; "Partially" means 
that the guide included information satisfying part of the best 
practice; "Minimally" means that the guide included information 
satisfying a minor part of the best practice; and "Not" means that the 
guide did not include any information that satisfied the best practice. 

[End of table] 

[End of section] 

Appendix II: Cost Estimating Weaknesses Identified in All Completed 
NNSA Project Reviews February 2009 - February 2014 and GAO's Best 
Practices for Cost Estimating: 

Idaho National Laboratory: 
Material Security and Consolidation Project: 

#1; Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 1, 8, 9; 
Total weaknesses: 3. 

Kansas City Plant: 
Responsive Infrastructure Manufacturing & Sourcing & National Secure 
Manufacturing Center Occupancy Project: 

#2; Peer review; 
Review date: 2011; 
GAO 12 steps (see table legend): [Empty]; 
Total weaknesses: 0. 

Los Alamos National Laboratory: 
Chemistry and Metallurgy Research Replacement Facility: 

#3; Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): [Empty]; 
Total weaknesses: 0. 

#4; Independent project review; 
Review date: 2011; 
GAO 12 steps (see table legend): 5, 9; 
Total weaknesses: 2. 

Los Alamos Neutron Science Center Refurbishment Project: 

#5; Independent project review/independent cost review; 
Review date: 2009; 
GAO 12 steps (see table legend): 5, 9, 10; 
Total weaknesses: 3. 

Nuclear Materials Safeguards and Security Upgrades Project Phase I & 
II: 

#6; Independent project Review; 
Review date: 2009; 
GAO 12 steps (see table legend): [Empty]; 
Total weaknesses: 0. 

#7; Peer review; 
Review date: 2011; 
GAO 12 steps (see table legend): 7; 
Total weaknesses: 2. 

#8; 
Review type: Peer review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5, 6, 12; 
Total weaknesses: 3. 

#9; Peer review; 
Review date: 2013; 
GAO 12 steps (see table legend): 5, 9; 
Total weaknesses: 2. 

Radioactive Liquid Waste Treatment Facility Upgrades Project: 

#10; 
Review type: Independent cost estimate; 
Review date: 2010; 
GAO 12 steps (see table legend): 6, 10; 
Total weaknesses: 2. 

#11; 
Review type: Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): 3, 4, 5, 9; 
Total weaknesses: 4. 

#12; 
Review type: Independent cost review; 
Review date: 2012; 
GAO 12 steps (see table legend): 3, 5, 7, 8, 10, 12; 
Total weaknesses: 6. 

#13; Independent project review; 
Review date: 2012; 
GAO 12 steps (see table legend): 9; 
Total weaknesses: 1. 

TA-55 Reinvestment Project - Phase II: 

#14; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 1, 2, 3, 7, 10; 
Total weaknesses: 5. 

Transuranic (TRU) Waste Facility: 

#15; 
Review type: Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): 1, 3, 4, 9; 
Total weaknesses: 4. 

#16; 
Review type: Independent project review; 
Review date: 2011; 
GAO 12 steps (see table legend): 5, 12; 
Total weaknesses: 2. 

#17; 
Review type: Independent cost estimate; 
Review date: 2013; 
GAO 12 steps (see table legend): 7; 
Total weaknesses: 1. 

Sanitary Effluent Reclamation Facility Expansion: 

#18; 
Review type: Independent project review; 
Review date: 2009; 
Total weaknesses: 0. 

Science and Engineering Complex: 

#19; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 1, 2, 3, 7, 8; 
Total weaknesses: 5. 

Nevada National Security Site: 

Device Assembly Facility Argus Installation Project: 

#20; 
Review type: Independent cost review; 
Review date: 2013; 
GAO 12 steps (see table legend): 3, 9; 
Total weaknesses: 2. 

Pantex Plant: 

High Explosives Pressing Facility: 

#21; 
Review type: Peer review; 
Review date: 2012; 
GAO 12 steps (see table legend): 9; 
Total weaknesses: 1. 

#22; 
Review type: Peer review; 
Review date: 2013; 
Total weaknesses: 0. 

Operations Systems Development and Integration Project: 

#23; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 3, 4, 5, 6, 9, 10; 
Total weaknesses: 6. 

#24; 
Review type: Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): 5, 10; 
Total weaknesses: 2. 

Pantex Renewable Energy Project: 

#25; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 3, 4, 5, 8, 9,, 10; 
Total weaknesses: 6. 

#26; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 4, 5; 
Total weaknesses: 2. 

Sandia National Laboratory: 

Test Capabilities Revitalization Project: 

#27; 
Review type: Independent project review; 
Review date: 2011; 
GAO 12 steps (see table legend): 10, 11, 12; 
Total weaknesses: 3. 

Savannah River Site: 

Mixed Oxide (MOX) Oxide Fuel Fabrication Facility Project: 

#28; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 4, 9, 12; 
Total weaknesses: 2. 

#29; 
Review type: Peer review; 
Review date: 2011; 
GAO 12 steps (see table legend): 5, 8, 12; 
Total weaknesses: 3. 

#30; 
Review type: Independent project review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5, 6, 8, 9, 12; 
Total weaknesses: 5. 

#31; 
Review type: Peer review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5, 9; 
Total weaknesses: 2. 

Pit Disassembly and Conversion Project: 

#32; 
Review type: Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): 3, 4, 9; 
Total weaknesses: 3. 

Waste Solidification Building Project: 

#33; 
Review type: Independent project review; 
Review date: 2010; 
Total weaknesses: 0. 

#34; 
Review type: Peer review; 
Review date: 2011; 
GAO 12 steps (see table legend): 2, 4, 10; 
Total weaknesses: 3. 

#35; 
Review type: Peer review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5, 6, 9, 10; 
Total weaknesses: 4. 

#36; 
Review type: Peer review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5, 6, 10; 
Total weaknesses: 3. 

#37; 
Review type: Construction project review; 
Review date: 2012; 
GAO 12 steps (see table legend): 5; 
Total weaknesses: 1. 

#38; 
Review type: Construction project review; 
Review date: 2013; 
GAO 12 steps (see table legend): 4, 5, 9, 10; 
Total weaknesses: 4. 

Y-12 Security Complex: 

Nuclear Facility Risk Reduction Project: 

#39; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 4, 5; 
Total weaknesses: 2. 

#40; 
Review type: Independent project review; 
Review date: 2010; 
Total weaknesses: 0. 

#41; 
Review type: Independent project review; 
Review date: 2011; 
Total weaknesses: 0. 

#42; 
Review type: Construction project review; 
Review date: 2013; 
GAO 12 steps (see table legend): 9; 
Total weaknesses: 1. 

Security Improvements Project: 

#43; 
Review type: Independent project review; 
Review date: 2009; 
GAO 12 steps (see table legend): 3, 4; 
Total weaknesses: 2. 

Uranium Processing Facility: 

#44; 
Review type: Independent cost estimate; 
Review date: 2010; 
GAO 12 steps (see table legend): 4, 5, 9; 
Total weaknesses: 3. 

#45; 
Review type: Independent project review; 
Review date: 2010; 
GAO 12 steps (see table legend): 5, 8, 9; 
Total weaknesses: 3. 

#46; 
Review type: Independent cost estimate; 
Review date: 2011; 
GAO 12 steps (see table legend): 4, 5, 6, 10; 
Total weaknesses: 4. 

#47; 
Review type: Independent project review; 
Review date: 2011; 
GAO 12 steps (see table legend): 8; 
Total weaknesses: 1. 

#48; 
Review type: Independent project review; 
Review date: 2012; 
Total weaknesses: 0. 

#49; 
Review type: Other; 
Review date: 2012; 
Total weaknesses: 0. 

#50; 
Review type: Peer review; 
Review date: 2013; 
Total weaknesses: 0. 

Total Weaknesses; 
GAO 12 steps (see table legend): 
1: 0; 
2: 4; 
3: 10; 
4: 15; 
5: 23; 
6: 7; 
7: 5; 
8: 8; 
9: 20; 
10: 13; 
11: 1; 
12: 7; 
Total weaknesses: 113. 

Source: GAO analysis of National Nuclear Security Administration 
project reviews. GAO-15-29. 

[A] The 2013 Independent Cost Estimate (ICE) review of the Transuranic 
(TRU) Waste Facility did not identify the lack of a point estimate and 
comparison of the estimate to an independent cost estimate as a 
weakness. However, in our analysis of the ICE review, we found that 
this step was not completed and have therefore identified it as a 
weakness. 

[End of table] 

Legend: GAO 12 Best Practice Cost Estimating Steps: 

Step 1. 
Description: Define estimate's purpose; 
Summary of associated tasks: Determine purpose, scope, required level 
of detail of estimate, as well as who will receive estimate. 

Step 2. 
Description: Develop estimating plan; 
Summary of associated tasks: Determine cost estimating team, schedule, 
and outline tasks in writing. 

Step 3. 
Description: Define program characteristics; 
Summary of associated tasks: Identify technical characteristics of 
planned investment, quality of data needed, and plan for documenting 
and updating information. 

Step 4. 
Description: Determine estimating structure (WBS); 
Summary of associated tasks: Define the elements of the cost estimate, 
including best method for estimating costs and potential cross-checks, 
and standardized structure. 

Step 5. 
Description: Identify ground rules and assumptions; 
Summary 
of associated tasks: Define what the estimate will include and exclude, 
key assumptions (such as life-cycle of investment), schedule or budget 
constraints, and other elements that affect estimate. Assumptions 
should be measurable, specific, and consistent with historical data. 
Assumptions should be based on expert and technical judgment. 

Step 6. 
Description: Obtain data; 
Summary of associated tasks: 
Create data collection plan, identify sources, collect valid and useful 
data, analyze data for cost drivers and other factors, and assess data 
for reliability and accuracy. 

Step 7. 
Description: Develop a point estimate and compare it to an 
independent cost estimate; 
Summary of associated tasks: Develop cost estimation model and 
calculate estimate, in constant dollars for investments that occur 
over multiple years, and other cross checks and validation, and 
compare estimate to an independent estimate and previous estimates. 
Update as more data are available. 

Step 8. 
Description: Conduct sensitivity analysis; 
Summary of associated tasks: Test the sensitivity of cost elements to 
changes in input values, ground rules, and assumptions. 

Step 9. 
Description: Conduct risk and uncertainty analysis; 
Summary of associated tasks: Determine which cost elements pose 
technical, cost, or schedule risks; analyze those risks; and recommend 
a plan to track and mitigate risks. A range of potential costs, based 
on risks and uncertainties, should be identified around a point 
estimate. 

Step 10. 
Description: Document the estimate; 
Summary of associated tasks: Document all steps used to develop the 
estimate so it can be recreated, describing methodology, data, 
assumptions, and results of risk, uncertainty, and sensitivity 
analysis. 

Step 11. 
Description: Present estimate to management for approval; 
Summary of associated tasks: Develop briefing on results, including 
information on estimation methods and risks, making content clear and 
complete so those unfamiliar with analysis can comprehend estimate and 
have confidence in it. 

Step 12. 
Description: Update the estimate to reflect actual costs and changes; 
Summary of associated tasks: As technical aspects of project change, 
the complete cost estimate should be regularly updated and, as project 
moves forward, cost and schedule estimates should be tracked. 

Source: GAO-09-3SP. GAO-15-29. 

[End of table] 

[End of section] 

Appendix III: Comments from the Department of Energy: 

Department of Energy: 
Washington, DC 20585: 

October 29, 2014: 

Mr. David Trimble: 
Director, Natural Resources and Environment: 
U.S. Government Accountability Office: 
Washington, D.C. 20548: 

Dear Mr. Trimble: 

This letter provides the U.S. Department of Energy's (DOE's) response 
to the recommendations contained in the U.S. Government Accountability 
Office (GAO) report entitled, "Modernizing the Nuclear Security 
Enterprise — DOE Needs to Revise Requirements and Guidance for Cost 
Estimating and Program Reviews (GAO-15-29, October 2014). We 
appreciate GAO's perspective and recommendations for improvement. 

The recommendations are consistent with our commitment to continuous 
improvement in cost estimation. We will take the necessary steps to 
address the recommendations and strive to adhere to best practices in 
cost estimating. Enclosed are our response and technical comments on 
the report. 

Sincerely, 

Signed by: 

Ingrid Kolb: 
Director: 
Office of Management: 

Enclosure: 

Attachment 1: DOE Responses to Recommendations: 

The Department has the following responses regarding the draft 
report's five recommendations. 

1. Recommendation: Revise DOE's project management order to require 
that DOE, NNSA, and its contractors develop cost estimates in 
accordance with the 12 cost estimating best practices. 

Response: The Departmental order for project management (DOE Order 
413.3B) will be assessed for revision following the issuance of the 
revision to DOE-STD-1189, Integration of Safety into the Design 
Process, currently scheduled for November 2016. The Department has 
established a log of issues to be addressed when revising Order 
413.3B, which includes GAO's recommendation. The recommendation to 
require that cost estimates follow the GAO best practices will be 
fully considered during this revision process. 

The DOE cost estimating guide (DOE Guide 413.3-21) already 
incorporates the twelve cost estimating best practices, albeit not in 
the same format as the GAO Guidance. The crosswalk to the GAO twelve 
cost estimating best practices is found in Appendices J and L of the 
Guide, Additionally, DOE has begun internal efforts for the 
publication of a Departmental Cost Estimating and Schedule Policy. 
This policy will complement the existing cost estimating guidance and 
will incorporate GAO's cost estimating guidance. The timeframe for the 
Policy to be finalized through the Department's directives process is 
still to be determined. 

DOE's Office of Acquisition and Project Management (OAPM) will 
continue to incorporate the twelve cost estimating best practices in 
its independent cost estimating activities. Currently, the standard 
operating procedures for independent cost estimates, independent cost 
reviews and external independent reviews within OAPM incorporate the 
twelve cost estimating practices. Additionally, the curriculum of the 
Project Management Career Development Program requires cost and 
schedule estimating courses for individuals pursuing certification as 
Federal Project Directors; this training incorporates the twelve cost 
estimating best practices. 

2. Recommendation: Revise DOE's cost estimating guide so that it fully 
reflects the 12 cost estimating best practices. 

Response: The Department will begin updating the cost estimating guide 
in the first quarter of FY 2015. The DOE cost estimating guide already 
incorporates the twelve cost estimating best practices albeit not in 
the same format as the GAO Guidance. The cross-walk to the GAO twelve 
cost estimating best practices is found in Appendices .1 and L of DOE 
Guide 413.3-21. The update efforts will elevate the significance of 
the twelve cost estimating best practices within the content of the 
guidance. 

3. Recommendation: Revise DOE's project management order to include 
references to the DOE cost estimating guide, where applicable. 

Response: The Departmental order for project management (DOE Order 
413.38) was published before DOE Guide 413.3-21 was issued; therefore, 
it was not possible to reference the guide. The order will be assessed 
for revision following the issuance of the revision to DOE-STD-1189, 
Integration of Safety into the Design Process, currently scheduled for 
November 2016. This recommendation will be fully considered during the 
order revision process. 

In the interim, the DOE Directives website points to the DOE 413.3 
Series Guides as the best practices guidance for implementation of DOE 
Order 413.3B requirements. 

4. Recommendation: Revise DOE directives that apply to programs to 
require that DOE and NNSA and its contractors develop cost estimates 
in accordance with the 12 cost estimating best practices, including 
developing lifecycle cost estimates for programs. 

Response: The Department is in the process of substantially revising 
the existing DOE Order 130.1, Budget Formulation, dated September 29, 
1995. As part of this effort, the Department will assess the 
requirement for program cost estimates and will revise the order to 
provide more specificity on the cost estimating requirement. The 
revised order will (1) define which DOE and NNSA program budget 
requests require costs estimates and (2) clarify that cost estimates 
for program budget submissions shall be conducted in accordance with 
the DOE cost estimating guide (or its successor policy). The estimated 
closure date for this GAO recommendation is September 2016. 

5. Recommendation: Revise DOE requirements and guidance that apply to 
programs to ensure that program reviews are conducted periodically, 
including reviews of the lifecycle cost estimates for programs. 

Response: The Department is in the process of substantially revising 
the existing DOE Order 130.1, Budget Formulation, dated September 29, 
1995. As part of this effort, the Department will assess requirements 
for program reviews and the linkage between program reviews and the 
budget formulation process. The revised order will clarify 
requirements for program reviews and specify how such reviews-—to 
include lifecycle cost estimates-—can best support the budget 
formulation process. NNSA will review these requirements and adjust 
NNSA-specific policies and guidance as appropriate. The estimated 
closure date for this GAO recommendation is September 2016. 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

David C. Trimble, (202) 512-3841 or trimbled@gao.gov: 

Staff Acknowledgments: 

In addition to the individual named above, Daniel Feehan, Assistant 
Director; Antoinette Capaccio; Jennifer Echard; Mark Braza; Mike 
Meleady; Alison O'Neill; and Peter Ruedel made key contributions to 
this report. 

[End of section] 

Footnotes: 

[1] DOE defines a program as an organized set of activities directed 
toward a common purpose or goal in support of an assigned mission 
area, and typically includes labor and operations and maintenance 
costs. The Stockpile Life Extension Program is intended to extend, 
through refurbishment, the operational lives of the weapons in the 
nuclear stockpile. The Plutonium Disposition Program is intended to 
dispose of 34 tons of surplus, weapons-grade plutonium. 

[2] DOE defines a capital asset as land, structures, equipment, and 
intellectual property, which are used by the federal government and 
have an estimated useful life of 2 years or more. The acquisition of 
capital assets typically includes projects involving the design and 
construction of facilities with specialized equipment, such as 
facilities needed to maintain the nuclear weapons stockpile, conduct 
research and development, or process nuclear materials. DOE defines a 
capital asset project as having defined start and end points with an 
acquisition cost that includes all costs incurred to construct the 
project for its intended purpose, bringing it to a form and location 
suitable for its intended use, excluding operating expenses that are 
part of routine operations and maintenance functions. 

[3] The MOX capital asset project is designed to remove impurities 
from plutonium feedstock obtained from nuclear weapon pits, form the 
plutonium into MOX fuel pellets, and fabricate pellets into fuel 
assemblies for use in commercial nuclear power reactors. The Waste 
Solidification Building is a capital asset project that is designed to 
process radioactive waste from the MOX facility. 

[4] GAO, High-Risk Series: An Update, [hyperlink, 
http://www.gao.gov/products/GAO-13-283] (Washington, D.C.: February 
2013). Major contracts and projects are those with values of $750 
million or greater. GAO designated DOE contract management as a high-
risk area in 1990, and, in 2013, GAO narrowed the focus of its high-
risk designation for DOE to the major contracts and projects of DOE's 
Office of Environmental Management and NNSA. 

[5] GAO, Plutonium Disposition Program: DOE Needs to Analyze the Root 
Causes of Cost Increases and Develop Better Cost Estimates, 
[hyperlink, http://www.gao.gov/products/GAO-14-231] (Washington, D.C.: 
Feb.13, 2014). DOE's fiscal year 2015 budget request stated that the 
MOX facility will be significantly more expensive than anticipated, 
and the request proposes to place the MOX facility in cold standby 
while NNSA evaluates plutonium disposition options. 

[6] NNSA's other program offices include Emergency Operations, Defense 
Nuclear Security, Counterterrorism and Counterproliferation, 
Infrastructure and Environment, Management and Administration, and the 
Office of the Administrator. 

[7] The Science Campaign is a program intended to improve the 
understanding of the complex processes that occur during a nuclear 
explosion by supporting several NNSA activities, including annual 
nuclear stockpile assessments and stockpile life extension program 
efforts. The Directed Stockpile Work program is intended to ensure the 
reliability of the nation's nuclear weapons stockpile, which includes 
activities such as LEP efforts and weapons dismantlement and 
disposition. Both of these programs are managed by the Office of 
Defense Programs. 

[8] Ideally, according to DOE, execution of such projects should take 
no more than 4 years from when the start of construction or execution 
was approved. Programs, however, can continue for defined periods of a 
decade or more. 

[9] The B61 is one of the nuclear weapons in the stockpile and the 
goal of the B61 LEP is to extend the operational life of this weapon. 
The B61 LEP is managed by the Office of Defense Programs. The 
Plutonium Disposition Program is a part of the Fissile Materials 
Disposition Program, which is managed by the Office of Defense Nuclear 
Nonproliferation. 

[10] S. Rep. No. 112-173, 293-4 (2012). 

[11] GAO, Department of Energy: Observations on Project and Program 
Cost Estimating in NNSA and the Office of Environmental Management, 
[hyperlink, http://www.gao.gov/products/GAO-13-510T] (Washington, 
D.C.: May 8, 2013). 

[12] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[13] An ICE is generated by an entity that has no stake in approval of 
the project but uses the same detailed technical information as the 
project estimate. Having an independent entity conduct an ICE and 
compare it to a project team's estimate provides an unbiased test of 
whether a project team's cost estimate is reasonable. As our cost 
estimating guide points out, the most rigorous independent review is 
an ICE. In contrast, an independent cost review addresses a project 
cost estimate's high-value, high-risk, and high-interest aspects 
without evaluating the remainder of the estimate. According to our 
cost guide, conducting an ICE is a cost estimating best practice, 
while an independent cost review is not. 

[14] These reviews are independent in the sense that they are 
conducted by offices other than the office that manages a particular 
program. 

[15] See GAO, National Nuclear Security Administration: Observations 
on NNSA's Management and Oversight of the Nuclear Security Enterprise, 
[hyperlink, http://www.gao.gov/products/GAO-12-473T] (Washington, 
D.C.: Feb. 16, 2012). 

[16] Pub. L. No. 106-65, § 3203, 113 Stat. 512, 954 (1999); codified 
at 42 U.S.C. §7144 (2012). 

[17] NNSA, Policy Letters: NNSA Policies, Supplemental Directives, and 
Business Operating Procedures, NA SD 251.1 (Washington, D.C.: July 5, 
2011). 

[18] DOE Order 413.3B, Program and Project Management for the 
Acquisition of Capital Assets, (November 2010). DOE's project 
management order applies to programs only in conjunction with a 
program's acquisition of capital assets. 

[19] A project's performance baseline consists of three interrelated 
targets--scope, cost, and completion date. A project's scope defines 
its technical goals and includes targets for specific technical 
requirements that the project is to deliver at completion. 

[20] DOE Guide 413.3-21, Cost Estimating Guide, May 2011. NNSA also 
issued a cost estimating guide that has many similarities to DOE's 
cost estimating guide. See NNSA, Cost Estimating Guide 50.005, Jan. 
11, 2010. According to NNSA officials, the NNSA guide was superseded 
and replaced by DOE's guide. 

[21] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[22] According to DOE officials, DOE also defines cost estimating 
requirements and guidance for the procurement of goods and services 
associated with management of department programs. These cost 
estimating requirements for procurements require the development of 
"independent government cost estimates." This report does not assess 
the cost estimating requirements and guidance associated with 
procurement activities and instead focuses on cost estimating related 
to the planning and budgeting for projects and programs. While 
programs rely on the acquisition of capital assets, this report does 
not address the effectiveness or efficiency of DOE or NNSA's 
acquisition process. 

[23] DOE Order 130.1, Budget Formulation, Sept. 29, 1995. 

[24] For more information about NNSA's PPBE process, see GAO, 
Modernizing the Nuclear Security Enterprise: NNSA's Reviews of Budget 
Estimates and Decisions on Resource Trade-offs Need Strengthening, 
[hyperlink, http://www.gao.gov/products/GAO-12-806] (Washington, D.C.: 
July 31, 2012). 

[25] Department of Defense and Department of Energy, Procedural 
Guideline for the Phase 6.X Process (Washington, D.C.: Apr. 19, 2000). 

[26] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. 

[27] A risk and uncertainty analysis assesses the variability in the 
cost estimate from such effects as schedules slipping, missions 
changing, and proposed solutions not meeting users' needs. A 
sensitivity analysis examines the effect of changing one assumption 
related to each project activity while holding all other variables 
constant in order to identify which variable most affects the cost 
estimate. 

[28] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[29] The ratings we used in this analysis are as follows: "Fully" 
means that the guide included information that satisfied the best 
practice; "Substantially" means that the guide included the majority 
of the information to satisfy the best practice; "Partially" means 
that the guide included information satisfying part of the best 
practice; "Minimally" means that the guide included information 
satisfying a minor part of the best practice; and "Not" means that the 
guide did not include any information that satisfied the best practice. 

[30] OAPM issued a WBS handbook in August 2012 to provide federal 
project directors and contractors suggested guidance and best 
practices for developing a product-oriented WBS that should be used 
for all projects within DOE. However, the handbook is not referenced 
in DOE's project management order and, as noted in the handbook, it 
does establish any new project requirements. 

[31] An earned value management system compares the value of the work 
accomplished during a given period with the value of the work 
scheduled to be accomplished during that period. Differences from the 
scheduled work plan are measured in both cost and schedule variances. 
Such a system tracks whether completed activities are costing more or 
less than expected. 

[32] [hyperlink, http://www.gao.gov/products/GAO-12-806]. 

[33] [hyperlink, http://www.gao.gov/products/GAO-14-231]. 

[34] [hyperlink, http://www.gao.gov/products/GAO-14-231]. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO's commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
"E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO's actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO's 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]