This is the accessible text file for GAO report number GAO-13-22 
entitled 'NASA: Earned Value Management Implementation across Major 
Spaceflight Projects Is Uneven' which was released on November 19, 
2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

November 2012: 

NASA: 

Earned Value Management Implementation across Major Spaceflight 
Projects Is Uneven: 

GAO-13-22: 

GAO Highlights: 

Highlights of GAO-13-22, a report to congressional requesters. 

Why GAO Did This Study: 

NASA historically has experienced cost growth and schedule slippage in 
its portfolio of major projects and has taken actions to improve in 
this area, including adopting the use of EVM. EVM is a tool developed 
to help project managers monitor risks. GAO was asked to examine (1) 
the extent to which NASA is using EVM to manage its major space flight 
acquisitions, (2) the challenges that NASA has faced in implementing 
an effective EVM system, and (3) NASA’s efforts to improve its use of 
EVM. To address these questions, GAO obtained contractor and project 
EVM data and used established formulas and tools to analyze the data 
and assess NASA’s implementation of EVM on 10 major spaceflight 
projects; interviewed relevant NASA headquarters, center and mission 
directorate officials on their views on EVM; and reviewed prior 
reports on EVM and organizational transformations. GAO compared NASA 
policies and guidance on EVM to best practices contained in GAO’s cost 
estimating best practices guide. 

What GAO Found: 

The National Aeronautics and Space Administration's (NASA) 10 major 
spaceflight projects discussed in this report have not yet fully 
implemented earned value management (EVM). As a result, NASA is not 
taking full advantage of opportunities to use an important tool that 
could help reduce acquisition risk. GAO assessed the 10 projects 
against three fundamental EVM practices that, according to GAO’s best 
practices cost guide, are necessary for maintaining a reliable EVM 
system. GAO found shortfalls in two of three fundamental practices. 
Specifically, we found that: 

* More than half of the projects did not use an EVM system that was 
fully certified as compliant with the industry EVM standard. 

* Only 4 of the 10 projects established formal surveillance reviews, 
which ensure that key data produced by the system was reliable. The 
remaining 6 projects provided evidence of monthly EVM data reviews; 
however, the rigor of both the formal and informal surveillance 
reviews is questionable given the numerous data anomalies GAO found. 

GAO also found that 3 projects had reliable EVM data while 7 had only 
partially reliable data. For the EVM data to be considered reliable 
per best practices it must be complete and accurate with all data 
anomalies explained. 

NASA EVM focal points, headquarters officials, project 
representatives, and program executives cited cultural and other 
challenges as impediments to the effective use of EVM at the agency. 
Traditionally, NASA's culture has focused on managing science and 
engineering challenges and not on monitoring cost and schedule data, 
like an effective EVM system produces. As a result, several 
representatives said this information traditionally has not been 
valued across the agency. This sentiment was also echoed in a NASA 
study of EVM implementation. Also cited as a challenge to the 
effective use of EVM was NASA’s insufficient number of staff with the 
skills to analyze EVM data. Without a sufficient number of staff with 
such skills, NASA’s ability to conduct a sound analysis of the EVM 
data is limited. However, NASA has not conducted an EVM skills gap 
analysis to determine the extent of its workforce needs. 

NASA has undertaken several initiatives aimed at improving the 
agency's use of EVM. For example, NASA strengthened its spaceflight 
management policy to reflect the industry EVM standard and has 
developed the processes and tools for projects to meet these standards 
through its new EVM system. While these are positive steps, the 
revised policy contains only the minimum requirements for earned value 
management. For example, it lacks a requirement for rigorous 
surveillance of how projects are implementing EVM and also does not 
require use of the agency’s newly developed EVM system to help meet 
the new requirements. NASA has attempted to address EVM shortcomings 
through policy changes over the years, but these efforts have failed 
to adequately address the cultural resistance to implementing EVM. 

What GAO Recommends: 

GAO recommends that NASA establish a time frame for requiring new 
spaceflight projects to implement its new EVM system; conduct an EVM 
skills gap assessment; develop a change management plan for EVM; and 
strengthen its EVM requirements by requiring projects to implement 
formal EVM surveillance. NASA concurred with two recommendations and 
partially concurred with two others citing resource constraints. 
Despite NASA's plans to address some issues GAO identified, not 
addressing all key issues lessens the usefulness of EVM at NASA. 

View [hyperlink, http://www.gao.gov/products/GAO-13-22]. For more 
information, contact Cristina Chaplain at (202) 512-4841 or 
chaplainc@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

NASA Projects Have Not Consistently Implemented Key EVM Practices and 
Most Did Not Have Access to Reliable EVM Data: 

Cultural, Technical, and Other Challenges Seen as Impediments to EVM 
Implementation: 

NASA Policy Is in Line with Best Practices but Implementation Remains 
the Challenge: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Importance of Earned Value Management: 

Appendix III: Case Studies of Selected Projects' Implementation of 
Earned Value Management: 

Appendix IV: Comments from the National Aeronautics and Space 
Administration: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: EVM Standards, Regulations, and Requirements Applicable to 
NASA: 

Table 2: Summary of the 10 NASA Spaceflight Projects Use of Three 
Fundamental EVM Practices and Reliability of the Data: 

Table 3: Examples of NASA EVM System Compliance with ANSI/EIA 
Guidelines: 

Table 4: NASA 10 Spaceflights Project EVM Data Summary: 

Table 5: Assessment of GPM EVM Practices: 

Table 6: Assessment of JWST EVM Practices: 

Table 7: Assessment of Landsat OLI EVM Practices: 

Table 8: Assessment of LADEE EVM Practices: 

Table 9: Assessment of MMS EVM Practices: 

Table 10: Assessment of MAVEN EVM Practices: 

Table 11: Assessment of OCO-2 EVM Practices: 

Table 12: Assessment of RBSP EVM Practices: 

Table 13: Assessment of SOFIA EVM Practices: 

Table 14: Assessment of TDRS EVM Practices: 

Figures: 

Figure 1: Example of a Project Work Breakdown Structure: 

Figure 2: GAO Earned Value Analysis of GMI-1 Data: 

Figure 3: GAO Earned Value Analysis of Observatory Data: 

Figure 4: GAO Earned Value Analysis of NIRcam Data: 

Figure 5: GAO Earned Value Analysis of Operational Land Imager Data: 

Figure 6: GAO Earned Value Analysis of LADEE Project Data: 

Figure 7: GAO Earned Value Analysis of MMS Project Data: 

Figure 8: GAO Earned Value Analysis of Instrument Suite Data: 

Figure 9: GAO Earned Value Analysis of MAVEN Project Data: 

Figure 10: GAO Earned Value Analysis of Spacecraft Data: 

Figure 11: GAO Earned Value Analysis of Science Operations Center Data: 

Figure 12: GAO Earned Value Analysis of Remote Sensing Instrument Data: 

Figure 13: GAO Earned Value Analysis of Langmuir Probe and Waves Data: 

Figure 14: GAO Earned Value Analysis of OCO-2 Project Data: 

Figure 15: GAO Earned Value Analysis of the Spacecraft Data: 

Figure 16: GAO Earned Value Analysis of Project Data: 

Figure 17: GAO Earned Value Analysis of the SOFIA Project Data: 

Figure 18: GAO Earned Value Analysis of the SOFIA Project Data: 

Figure 19: GAO Earned Value Analysis of the TDRS Spacecraft Project 
Data: 

Figure 20: GAO Earned Value Analysis of the TDRS Sustainment Data: 

Abbreviations: 

ANSI: American National Standards Institute: 

BAC: budget at completion: 

CPR: contract performance report: 

DCMA: Defense Contract Management Agency: 

EIA: Electronic Industries Alliance: 

EVM: earned value management: 

FAR: Federal Acquisition Regulation: 

GPM: Global Precipitation Measurement: 

GSFC: Goddard Space Flight Center: 

IBR: integrated baseline review: 

ICESat-2: Ice, Cloud, and Land Elevation Satellite 2: 

IMS: integrated master schedule: 

JWST: James Webb Space Telescope: 

LADEE: Lunar Atmosphere and Dust Environment Explorer: 

LDCM: Landsat Data Continuity Mission: 

LOE: level of effort: 

MAVEN: Mars Atmosphere and Volatile EvolutioN Mission: 

MMS: Magnetospheric Multiscale: 

NASA: National Aeronautics and Space Administration: 

NID: NASA Interim Directive: 

NPR: NASA Procedural Requirements: 

OCO-2: Orbiting Carbon Observatory 2: 

OMB: Office of Management and Budget: 

RBSP: Radiation Belt Storm Probes: 

SOFIA: Stratospheric Observatory for Infrared Astronomy: 

TDRS: Tracking and Data Relay Satellite System: 

VAC: variance at completion: 

WBS: work breakdown structure: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

November 19, 2012: 

The Honorable Kay Bailey Hutchison: 
Ranking Member: 
Committee on Commerce, Science, and Transportation: 
United States Senate: 

The Honorable Ralph M. Hall: 
Chairman: 
The Honorable Eddie Bernice Johnson: 
Ranking Member: 
Committee on Science, Space, and Technology: 
House of Representatives: 

The Honorable Bill Nelson: 
Chairman: 
Subcommittee on Science and Space: 
Committee on Commerce, Science, and Transportation: 
United States Senate: 

The National Aeronautics and Space Administration (NASA) has a 
portfolio of 21 major spaceflight projects that are expected to cost 
$19.4 billion to develop.[Footnote 1] Historically, NASA has 
experienced problems with persistent cost growth and schedule slippage 
in the majority of its major projects. As a result, NASA acquisition 
management has remained on GAO's high risk list since it was first 
introduced in 1990.[Footnote 2] Our work has shown that several 
factors contribute to NASA's cost and schedule performance, including 
poor cost estimating and underestimating risks associated with the 
development of its major systems. 

Over the years, NASA has undertaken a number of actions to improve its 
acquisition management function and has shown some progress in 
improving its performance. One of these actions has been to adopt 
earned value management (EVM). EVM is a project management tool 
developed by the Department of Defense in the 1960s to help project 
managers monitor risks. EVM measures the value of work accomplished in 
a given period and compares it with the planned value of work 
scheduled for that period and the actual cost of work accomplished. 
EVM's intended purpose is to integrate a project's cost, schedule, and 
technical efforts for management and provide reliable data to decision 
makers. 

Starting in 1997, NASA began to require its projects to implement EVM 
on all significant contracts.[Footnote 3] In 2005, NASA broadened its 
application of EVM to encompass significant project efforts 
implemented by in-house civil service and associated support 
contractor personnel. NASA has had uneven success in effectively 
implementing EVM, according to GAO and NASA Inspector General reports 
conducted since 1999.[Footnote 4] NASA has recognized a need for 
improved EVM implementation, and has an ongoing effort focused on 
doing so. In light of these issues, the House Science, Space, and 
Technology Committee and the Senate Committee on Commerce, Science, 
and Transportation asked GAO to examine the use of EVM at NASA. 
Specifically, we assessed (1) the extent to which NASA is using EVM to 
manage its major space flight acquisitions, (2) the challenges that 
NASA faces in implementing an effective EVM system, and (3) NASA's 
efforts to improve its use of EVM. 

To determine the extent to which NASA's major spaceflight projects are 
using EVM to manage the acquisition, we reviewed 10 major spaceflight 
projects, each with a life cycle cost estimate of more than $250 
million that had been approved to proceed into the implementation 
phase of development. Collectively, the projects we reviewed will cost 
over $14 billion to develop (See table 2 for a list of the 
projects).[Footnote 5] We collected all available EVM data for the 10 
projects for the period of August 2010 to August 2011 and used 
established earned value formulas and tools to identify cost and 
schedule variances and trends. We also assessed the projects' 
implementation of three fundamental EVM practices that, according to 
GAO's Cost Estimating and Assessment Guide, are necessary for 
maintaining a reliable EVM system--using a certified system, 
conducting integrated baseline reviews, and performing 
surveillance.[Footnote 6] We analyzed monthly project management 
review briefings to support our analysis. To assess the reliability of 
the cost data, we electronically tested the data for anomalies, and 
reviewed relevant project documentation and interviewed agency and 
project officials about the data. To determine the challenges the 
agency faces in its efforts to implement an effective EVM system, we 
developed standard sets of questions and interviewed NASA headquarters 
officials, program executives, and EVM focal points at each center and 
the Human Exploration and Operations and Science Mission Directorates 
to discuss their roles as well as the extent to which EVM data is used 
to inform decision making. We also reviewed prior GAO and relevant 
NASA Inspector General reports on EVM. Also, we submitted written 
questions to the project offices to obtain their views on implementing 
EVM on their projects. To determine the steps that NASA is taking to 
improve its use of EVM, we compared NASA policies and guidance 
intended to improve the agency's implementation of EVM to best 
practices for earned value management as documented in GAO's cost 
guide. We also interviewed agency officials responsible for developing 
and implementing the NASA's new system to discuss ongoing initiatives. 
For additional details on our scope and methodology, see appendix I. 

We conducted this performance audit from July 2011 through November 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

EVM is a project management tool that, when properly used, can provide 
accurate assessments of project progress, produce early warning signs 
of impending schedule delays and cost overruns, and provide unbiased 
estimates of anticipated costs at completion. Pulling together 
essential cost, schedule, and technical information in a meaningful, 
coherent fashion is a challenge for most projects. Without such 
information, managers can have a distorted view of a project's status 
and risks. EVM also allows individuals outside the project to see a 
standardized metric describing the cost and schedule performance of 
that particular project and compare it consistently with other 
projects. 

EVM measures the value of work accomplished in a given period and 
compares it with the planned value of work scheduled for that period 
and with the actual cost of work accomplished. Differences in these 
values are measured in both cost and schedule variances. Positive 
variances indicate that activities are costing less or are completed 
ahead of schedule. Negative variances indicate activities are costing 
more or are falling behind schedule. For example, if a contractor 
completed $5 million worth of work and the work actually cost $6.7 
million, there would be a $1.7 million negative cost variance. 
Schedule variances are also measured in dollars, but they compare the 
earned value of the work completed with the value of work that was 
expected to be completed. For example, if a contractor completed $5 
million worth of work at the end of the month but was budgeted to 
complete $10 million worth of work, there would be a $5 million 
negative schedule variance. 

Earned value provides information that is necessary for understanding 
the health of a project and an objective view of project status. Cost 
and schedule variances can also be used in estimating the cost and 
time needed to complete the project. While some data that an EVM 
system produces are retrospective and indicate performance to date, 
EVM data can also be used to predict future performance. For example, 
estimates at completion for a project can be calculated by using 
efficiency indices which are based on a project's past cost and 
schedule performance. See appendix II for additional information on 
the importance of EVM. 

A project's EVM data comes from multiple sources. For example, each 
contractor that supports a project will produce and deliver EVM 
reports to the project for the work they and their subcontractors 
perform, if the contract so requires. In addition, the project will 
collect EVM information for the work that it performs in-house at a 
NASA center. All of this lower level EVM data can then be consolidated 
at the project level to produce a project level EVM report. Pulling 
together EVM data from multiple levels into a project level report 
gives the project a comprehensive outlook of its cost and schedule, 
and provides the project manager with early warning of potential cost 
and schedule overruns. 

EVM has evolved from an industrial engineering tool to a government 
and industry best practice, providing improved information to conduct 
oversight of acquisition programs. As such, it is guided by industry 
best practices and standard, and is required by regulations and 
requirements at the federal government and the agency level at NASA. 
These requirements and standards are summarized in table 1 below. 

Table 1: EVM Standards, Regulations, and Requirements Applicable to 
NASA: 

American National Standards Institute/Electronic Industries Alliance 
(ANSI/EIA)-748: 
* Regarded as the national standard and an industry best practice for 
EVM systems; 
* Describes 32 guidelines that a certified EVM system must meet in the 
areas of organization; planning, scheduling, and budgeting; accounting; 
analysis and management reports; and revisions and data maintenance. 

American National Standards Institute/Electronic Industries Alliance 
(ANSI/EIA)-748: Office of Management and Budget (OMB) Circular A-11 
and Capital Programming Guide; 
* EVM is to be applied for parts of a major acquisition with 
developmental effort and is to be used for both in-house government 
and contractor work; 
* EVM system must be in compliance with the guidelines in; 
* ANSI/EIA-748; 
* When there is both government and contractor work, the data from the 
two EVM systems must be consolidated at the reporting level for total 
program management and visibility. 

American National Standards Institute/Electronic Industries Alliance 
(ANSI/EIA)-748: Federal Acquisition Regulation (FAR), Subpart 34.2; 
* EVM system required for major acquisitions in development; 
* Contractors and subcontractors are required to report EVM on a 
monthly basis; 
* When an EVM system is required, an integrated baseline review is 
required to verify technical content and realism of related 
performance budgets, resources, and schedules. 

American National Standards Institute/Electronic Industries Alliance 
(ANSI/EIA)-748: NASA FAR Supplement, Subpart 1834.2; 
* EVM system is required on acquisitions for development or production 
work for flight and ground support systems and components, prototypes, 
and institutional investments, such as facilities and IT 
infrastructure; 
* Certified ANSI/EIA-compliant EVM system (as determined by cognizant 
Federal agency) is required for cost or fixed-price incentive 
contracts and subcontracts valued at $50 million or more; 
* ANSI/EIA-compliant EVM system (as determined by the contracting 
officer) required for cost or fixed-price incentive contracts and 
subcontracts valued at $20 million or more but less than $50 million. 

American National Standards Institute/Electronic Industries Alliance 
(ANSI/EIA)-748: NASA Procedural Requirements (NPR) 7120.5; 
* NPR 7120.5E strengthens the previous version by requiring projects 
to perform EVM with a system that complies with the guidelines in 
ANSI/EIA-748. This directive establishes the requirements by which 
NASA will formulate and implement spaceflight programs and projects 
including flowing down EVM system requirements to applicable 
suppliers[A]; 
* Requires an integrated review of project baselines as part of their 
preparations for the confirmation review to ensure that the project's 
work is properly linked with its cost, schedule, and risk and that the 
systems are in place to conduct EVM. 

Source: GAO analysis of standards, regulations, and requirements. 

[A] For some of its projects, international partners contribute 
elements of a mission. According to NASA, the agreements reached with 
its international partners do not require the international partners 
to share EVM data for their contribution to a project. When NASA 
collaborates with an international partner, it is done on a no 
exchange of funds basis. Each partner commits to providing certain 
elements of the overall mission, such as an instrument, spacecraft 
bus, or launch vehicle, and are responsible for delivering the 
element(s) at the appropriate time. Each partner determines how they 
want to manage their development efforts and assumes all costs for 
their work. Partners are not required to share any information on how 
much their efforts cost. 

[End of table] 

Almost two decades of NASA's past efforts to improve its use of earned 
value management have had uneven success. An EVM Focal Point Council 
was created in 1996 and focal points were designated at each NASA 
center and the Office of Procurement and the Office of the Chief 
Financial Officer to provide an open forum to share experiences and 
develop a network of support within the NASA EVM community. In 1997, 
the agency issued NASA Policy Directive 9501.3 "Earned Value 
Management Performance" which established the basis for applying EVM 
to NASA contracts. Prior to this policy, centers used their own 
individual policies on performance measurement systems. However, in 
1999, the NASA Inspector General reported that EVM policy was not 
consolidated as an overall program management responsibility and that 
the Agency Program Management Council did not receive comprehensive 
EVM information.[Footnote 7] As a result, in 2003, NASA shifted 
responsibility for the EVM policy from the Office of the Chief 
Financial Officer to the Office of the Chief Engineer to emphasize EVM 
as a project management tool, rather than a financial management tool. 

In 2004, GAO reported that only 2 of 10 NASA projects reviewed used 
EVM consistently and appropriately.[Footnote 8] Several barriers to 
EVM implementation were identified, such as lack of reliable financial 
data, trained EVM staff, data analysis tools, and incentives. Among 
other things, GAO recommended that NASA take action to ensure that a 
true EVM system is used as an organizational management tool to bring 
cost to the forefront in NASA's decision-making process and that 
acquisition and EVM management policies and procedures be enforced. In 
response to our recommendations, NASA stated that it was updating NASA 
Procedural Requirements 7120.5, its program and project management 
processes and requirements policy, to improve its cost estimating and 
ensure that its cost estimate and earned value analyses were 
effectively used, and the updated policy was issued in 2005. Similar 
to other agencies in our 2009 report on governmentwide use of EVM, we 
reported on weaknesses in NASA's EVM policies and practices and 
recommended that the agency modify its policies governing EVM to 
ensure that they are consistent with best practices.[Footnote 9] In 
particular, we found problems with the EVM training requirements for 
personnel responsible for investment oversight and management 
responsibilities. In addition, NASA's policy for revising project cost 
and schedule baselines did not have adequately defined criteria on the 
acceptable reasons for permitting a rebaselining.[Footnote 10] We also 
found weaknesses in how the NASA projects we reviewed implemented EVM 
and managed their negative performance trends. NASA acknowledged the 
identified weaknesses and stated that it was revising its NASA 
procedural requirements for programs and projects to include expanded 
and strengthened policies governing EVM application and processes, and 
revised policies for rebaselining projects. 

In January 2010, the Agency Program Management Council approved the 
funding for the EVM capability project to develop an EVM system that 
complies with the guidelines in ANSI/EIA-748. Another goal of the 
project was to determine whether it was feasible to implement a single 
EVM system that would integrate the scope, schedule, and budget of EVM 
data for NASA's in-house managed efforts and contractor data across 
the agency. According to agency officials, NASA invested about $2 
million into the capability project to pilot the EVM system through 
two projects: the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-
2) and the Constellation Extra-Vehicular Activity project.[Footnote 
11] Through the pilots, the capability project demonstrated that an 
agency-wide EVM system was feasible. As a result, a finalized set of 
processes, tools, guidance, and training products that compose NASA's 
new EVM system was developed. This new system was peer reviewed and 
approved by a panel of EVM experts. 

NASA Projects Have Not Consistently Implemented Key EVM Practices and 
Most Did Not Have Access to Reliable EVM Data: 

Our assessment of 10 major spaceflight projects showed that NASA has 
not yet fully implemented EVM and thus is not taking full advantage of 
an important tool that could help reduce acquisition risk. GAO found 
that the projects had shortfalls in two of the three fundamental 
practices that we assessed. Specifically, we found that half of the 
projects did not use an EVM system that was certified as compliant 
with the ANSI/EIA-748 standard. Most of the projects conducted an 
integrated baseline review (IBR), a practice that ensures the 
performance measurement baseline reflects all requirements and that 
resources are adequate to complete the work. Specifically, 9 projects 
conducted an IBR for their overall efforts; however, the Stratospheric 
Observatory for Infrared Astronomy (SOFIA) project office only 
conducted an IBR of one of its major contractors. In addition, we 
found that only 4 of the 10 projects had established formal 
independent surveillance reviews to ensure that key elements of the 
EVM process were maintained over time so that the data produced by the 
system provided timely indications of actual or potential problems. 
For the 6 projects that did not have formal independent surveillance 
in place, each provided evidence that they instituted monthly EVM data 
reviews, which according to project officials, helps them to 
continually monitor cost and schedule performance. However, the rigor 
of both the formal and informal surveillance reviews is questionable 
given the numerous EVM data anomalies we found in the monthly EVM 
reports. Specifically, we found many unexplained anomalies, such as 
the presence of negative numbers or missing data, which caused us to 
question the reliability of the data. Out of the 10 projects we 
reviewed, we found that just 3 projects had reliable EVM data while 
the remaining 7 had only partially reliable data. 

Overall, the projects are using EVM, but NASA has not consistently 
implemented EVM across these projects. For example, we found that 
several projects were not implementing EVM at the project level, which 
is considered a best practice. Table 2 summarizes the performance of 
each of the 10 projects in meeting the three fundamental EVM practices 
and the reliability of the data. 

Table 2: Summary of the 10 NASA Spaceflight Projects Use of Three 
Fundamental EVM Practices and Reliability of the Data: 

Projects: Global Precipitation Measurement; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: James Webb Space Telescope; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: Landsat Data Continuity Mission; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: Lunar Atmosphere and Dust Environment Explorer; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: Magnetospheric Multiscale; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: Mars Atmosphere and Volatile Evolution Mission; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
partially met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion. 

Projects: Orbiting Carbon Observatory 2[A]; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion. 

Projects: Radiation Belt Storm Probes; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion. 

Projects: Stratospheric Observatory for Infrared Astronomy Project; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
partially met this criterion; 
Conducted an integrated baseline review: Agency partially met this 
criterion; 
EVM System surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Projects: Tracking and Data Relay Satellite Replenishment; 
Used A certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM System surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion. 

Source: GAO analysis of NASA and contractor data. 

Notes: For detailed information on the EVM performance of the projects 
selected for our case studies, see appendix III. 

[A] At the time of our review, the Orbital Sciences Corporation did 
not have a certified EVM system that complied with the ANSI/EIA-748 
standard; however, the Defense Contract Management Agency certified 
their EVM system in January 2012. 

[End of table] 

Few Projects Had EVM Systems Compliant with the Industry Standard: 

[Side bar: Criteria:
* Project obtained EVM data from certified system(s).
* Project shows a consistent work breakdown structure between the EVM 
report and the schedule.
* Project schedule identifies significant task interdependencies.
* Project uses a time-phased budget baseline for tracking cost and 
schedule variances. End of side bar] 

Of the 10 projects we reviewed, 4 projects had a certified EVM system, 
3 did not, and 3 had a mixture in which some contractors and 
subcontractors had certified systems and some did not. When an EVM 
system is certified, the agency has assurance that the implemented 
system was validated for compliance with the ANSI/EIA-748 standard by 
independent and qualified staff and therefore can be considered to 
provide reliable and valid data from which to manage a project. The 
Global Precipitation Measurement (GPM), Tracking and Data Relay 
Satellite System (TDRS), Landsat Data Continuity Mission (LDCM), and 
James Webb Space Telescope (JWST) were the only projects that provided 
evidence that the contract performance reports provided came from EVM 
systems that were certified as compliant with the ANSI/EIA-748 
standard. The Lunar Atmosphere and Dust Environment Explorer (LADEE), 
Magnetospheric Multiscale (MMS) and Radiation Belt Storm Probes (RBSP) 
projects did not have EVM systems that were certified to be compliant 
with the ASNI/EIA-748 standard. Finally, the Jet Propulsion 
Laboratory, a federally funded research and development center that 
the California Institute of Technology manages under a contract with 
NASA, was the only NASA Center with a certified EVM system. The Jet 
Propulsion Laboratory is responsible for managing the Orbiting Carbon 
Observatory 2 (OCO-2) project. The Mars Atmosphere and Volatile 
Evolution Mission (MAVEN) and SOFIA prime contractors also had 
certified systems; however, their project offices did not. NASA does 
not require a certified EVM system for their in-house work. 

Using the project's integrated master schedule and contract 
performance reports, we assessed the EVM data provided by the projects 
against selected fundamental ANSI/EIA guidelines to determine the 
extent to which each project's EVM system, whether certified or not, 
was meeting them. The guidelines we reviewed included whether the work 
breakdown structure (WBS)--which provides the basis of the project 
schedule--was consistent between the EVM report and the schedule, 
whether the schedule identified significant task interdependencies, 
and whether the project had identified a time-phased budget baseline 
for tracking cost and schedule variances. As shown in figure 1, a work 
breakdown structure breaks down product-oriented elements into a 
hierarchical structure that shows how elements relate to one another 
as well as to the overall end product. By subdividing a project into 
smaller elements, management can more easily plan and schedule the 
program's activities and assign responsibility for the work. 

Figure 1: Example of a Project Work Breakdown Structure: 

[Refer to PDF for image: illustration] 

Project WBS: 

1.0. Spaceflight project  
1.1. Project management  
1.2. Systems engineering  
1.3. Payloads  
1.3.1. Payload #1 development  
1.3.2. Payload #2 development  
Contract WBS: 
1.4 Spacecraft 
1.4.1 Spacecraft management 
1.4.2 Propulsion 
1.4.3 Architecture 
1.4.4 Avionics 
1.4.5 Thermal control 
1.4.5.1 Passive thermal control 
1.4.5.2 Active thermal control 
1.4.5.2.1 Active thermal control pumps 
1.4.5.2.2 Active thermal control piping and valves 
1.4.5.2.3 Active thermal control radiator 
1.4.6 Integration and testing 
1.5. Mission operations  
1.5.1. Mission operations requirements  
1.5.1. Operator training and certification  
1.6. Launch vehicle systems  
1.7. Ground systems  
1.7.1. Ground systems management  
1.8. System integration and testing  
1.8.1. Integration and testing management  
1.8.2. Systems integration  
1.8.3. Static loads testing  
1.8.4. Avionics testing  
1.9. Education and public outreach  

Source: GAO illustration based on NASA data. 

[End of figure] 

We found that even for the projects that had certified systems, there 
were problems with consistency between the WBS and the EVM report and 
the schedule. For example, we found discrepancies in the hierarchical 
structure and numbering of WBS elements for JWST, an $8.8 billion 
project. Specifically, the project's WBS dictionary showed mission 
assurance efforts numbered differently than contractor reports for two 
contractors, each of which had mission assurance labeled with 
different WBS numbers.[Footnote 12] NASA officials explained that 
neither the spacecraft nor the near infrared camera contractor was 
required to follow the project-level WBS structure or numbering 
scheme. NASA officials explained that while it is not a requirement 
for the project and contractor WBSs to be the same, it is recommended 
that the prime contractor lower-level WBS numbering scheme be 
consistent with the overall project WBS numbering format. Doing so 
allows easier total project integration of cost and EVM data for 
project reporting. Consistency of the WBS element between the cost 
estimate and the schedule facilitates updating the cost estimate with 
actual costs and ensures there is correlation between the cost 
estimate and schedule. 

Our review of the project schedules also revealed that about half of 
the schedules were missing predecessor and/or successor dependencies 
and had constraints that prevented the schedule from responding 
properly to updates. Since the schedule is the foundation for the EVM 
baseline, it must be properly sequenced. This means knowing how one 
activity (the predecessor) affects another (the successor) and how 
each affects the critical path.[Footnote 13] When the schedule is not 
sequenced correctly, the reliability of the EVM data is called into 
question. Our review found that the MMS project was missing 
dependencies for 31 percent of its remaining activities for its 
instrument suite contract and 36 percent of the remaining activities 
for the instrument suite were constrained.[Footnote 14] Due to the 
major sequencing issues in the MMS instrument schedule, we questioned 
the reliability of the overall network and the schedule's ability to 
correctly calculate float values and the critical path. MMS project 
officials said they believed many of the constrained activities we 
found were not valid because they reside in another schedule. In 
addition, officials said some of the constraints found are in the 
Harness area, and if removed, these constraints would have no effect 
on the overall schedule. Furthermore, MMS officials said some of the 
sequencing issues may be caused by manual integration because some 
instrument provider schedules are in Microsoft Project and others are 
in Primavera, and therefore it is not possible to ensure all tasks 
have been linked properly. As part of their schedule health check 
process, the MMS project scheduler tests the schedule for missing 
dependencies, logic errors, and reasonable durations and the results 
are shared with the project office and the contractor so that 
appropriate action can be taken. However, when we removed the 15 level 
of effort type activities from the missing dependencies count, the 
schedule still showed 28 percent of the remaining activities missing 
dependencies. We also removed the 14 level of effort type activities 
and 4 Harness from the constraint count, the schedule still showed 33 
percent of the remaining activities were constrained. MMS program 
officials said that they have a process in place to manage float and 
the critical path, however, the schedule we reviewed still showed 
significant sequencing issues. 

Finally, we found that 4 of the 14 the schedules we analyzed were not 
resource loaded. This means that the schedule did not have the 
required labor, materials, equipment, and other resources assigned to 
the appropriate activities. When the schedule is not resource loaded, 
then costs need to be spread over time using some other method that 
may not be as straightforward as having the costs integrated directly 
within the schedule. Having a resource-loaded schedule is a best 
practice for developing the time phased budget baseline. The time-
phased budget baseline represents the plan that performance is 
measured against for the life of a project. It takes into account that 
program activities occur in a sequenced order, based on finite 
resources, with budgets representing those resources spread over time. 
Deviations from the baseline identify areas where management should 
focus their attention. 

Majority of Projects Conducted an Integrated Baseline Review: 

[Side bar: Criteria: 
* Project performed an IBR to verify that the performance measurement 
baseline is realistic and to ensure that the contractor and government 
mutually understand the potential project risks. End of side bar] 

In keeping with best practices, 9 of the 10 projects conducted 
integrated baseline reviews. An IBR is an evaluation of the 
performance measurement baseline--the foundation for an EVM system--to 
determine whether all project requirements have been addressed, risks 
have been identified, mitigation plans are in place, and available and 
planned resources are sufficient to complete the work. Conducting an 
IBR increases confidence that the performance measurement baseline 
provides reliable cost and schedule data for managing the project and 
that it projects accurate estimated costs at completion. OMB has 
endorsed the IBR as a critical process for risk management on major 
investments and requires agencies to conduct IBRs for all contracts 
that require EVM. Since an IBR's goal is to verify that the technical 
baseline's budget and schedule are adequate for performing the work, 
it offers many key benefits such as laying a solid foundation for 
successfully executing the project and enabling better understanding 
of the risks. 

Officials for the SOFIA project did not conduct an integrated baseline 
review at the project level; however, its prime contractor for the 
engineering and modification of the airborne observatory platform did 
conduct an integrated baseline review. According to project officials, 
the lack of a project-level IBR is related to the EVM system being 
implemented "on the fly" late in the development phase for SOFIA, as a 
result of an audit recommendation in 2010.[Footnote 15] However, 
project officials noted that the EVM baseline was established 
concurrently with an agency approved re-plan and Joint Cost and 
Schedule Confidence Level analysis in 2009 and 2010 and was reviewed 
by a Standing Review Board as part of that process.[Footnote 16] 

Majority of Projects Do Not Have a Comprehensive Surveillance System 
in Place: 

[Side bar: Criteria: 
* Project verifies that EVM processes and procedures continue to 
satisfy the ANSI/EIA EVM guidelines.
* Independent surveillance organization conducts reviews to ensure the 
integrity of the organization's EVM system. End of side bar] 

Four of the 10 projects we assessed had a comprehensive EVM 
surveillance system in place. Of the remaining 6 projects, 1 had 
formal surveillance at the project level but its contractor did not, 
and 2 projects did not have a formal surveillance system at the 
project level, only their prime contractors did. The remaining 3 
projects did not have any formal surveillance, but provided evidence 
that EVM data, such as cost and schedule variances, were being 
reviewed during their monthly status reviews. Beyond reviewing cost 
and schedule variances and variances at completion, formal 
surveillance reviews ensure that the processes and procedures continue 
to satisfy the ANSI/EIA EVM guidelines. A formal surveillance plan 
involves establishing an independent surveillance organization with 
members who have practical experience using EVM. This organization 
then conducts periodic surveillance reviews to ensure the integrity of 
the contractor's EVM system and where necessary discusses corrective 
actions to mitigate risks and manage cost and schedule performance. 
Effective surveillance ensures that the key elements of the EVM 
process are maintained over time and on subsequent applications. 

NASA delegates surveillance of contractor EVM systems to the Defense 
Contract Management Agency (DCMA); however, NASA has no entity to 
perform independent surveillance reviews to ensure that the ANSI/EIA-
748 standard is being met for EVM efforts performed in-house or by 
nonprofit organizations. Without an independent surveillance function, 
an organization's ability to use EVM as intended may be hampered since 
surveillance monitors problems with the performance measurement 
baseline and EVM data. If the kinds of problems that formal 
surveillance can identify go undetected, EVM data may be distorted and 
may not be meaningful for decision making. 

Unreliable EVM Data Limit NASA's Ability to Measure Project 
Performance: 

[Side bar: Criteria: 
* Project EVM system(s) produces reports that are complete and accurate.
* All reported data anomalies are explained.
* EVM data can be mapped through various levels to precisely identify 
root cause(s) of issues. End of side bar] 

Only 3 of the 10 projects we reviewed, MAVEN, RBSP, and OCO-2, 
produced fully reliable data for managing the project and reporting 
status. The other projects only partially met the criterion to have an 
EVM system in place that produces reliable data. If done correctly, 
EVM data can provide an objective means for measuring project status 
and forecasting potential project cost overruns and schedule slippages 
so that timely action can be taken to minimize their impact. To do so, 
however, the underlying EVM data must be reliable, meaning that they 
are complete and accurate and all data anomalies are explained. In our 
analysis, we found multiple cases of data anomalies that caused us to 
question the reliability of the EVM data. For example, we found 
several EVM reports where a contractor reported that no work was 
planned or accomplished, but actual costs were incurred without an 
explanation in the variance analysis report to say why this happened. 
Additionally, we found cases where a contractor reported that work was 
planned and actual costs were incurred, but a negative amount of work 
was performed--work that was previously reported as completed was now 
reported as not completed. Further, we also found several instances 
where a project reported an estimate at completion but no budget at 
completion.[Footnote 17] Finally, we found instances of negative 
values in the EVM reports. When explanations were provided in the 
variance analysis reports, the reasons for these anomalies included 
use of estimated, rather than actual costs, or adjustments from prior 
periods due to mistakes or errors with the underlying EVM systems. For 
example, the SOFIA project said that many of the negative values in 
its EVM reports were due to over-reporting of earlier progress, 
mischarges by employees, delayed cost postings, inappropriate use of 
charge codes. When there are data anomalies such as those we 
identified for SOFIA, the EVM data can become skewed and can distort 
true performance. 

In some cases, the projects provided no explanations for the anomalies 
we found. In other cases, the projects said that the size of the 
anomalies did not breach the variance reporting threshold and 
therefore no explanation was required.[Footnote 18] However, whether 
the variance breaches the threshold or not, cost and schedule 
deviations from the baseline plan give management information about 
where corrective action plans are needed to bring the project back on 
track or to update completion dates. If programs only address those 
issues that breach the thresholds, management may be losing valuable 
insight into project risks and causes. EVM data should be valid and 
free from unexplained anomalies (e.g., missing or negative values) 
because without reliable performance data, project management's 
ability to identify potential problems and intervene early to mitigate 
the problems can be limited. Therefore, anomalies should be minimized 
and the reason for each should be fully explained in the monthly EVM 
variance analysis reports. To do less limits the completeness and 
accuracy of the EVM data and makes the resulting variance 
determinations unreliable. While an industry standard for what 
constitutes an acceptable volume of anomalies does not exist, EVM 
experts in the public and private sectors stated that the occurrence 
of EVM data anomalies should be rare. 

For four projects that provided subcontractor EVM data, we tried to 
map EVM data at the project level to lower level EVM data at the 
subcontractor level. However, we were only able to successfully map 
the data for one of the projects. This mapping allows project managers 
to track cost and schedule by defined deliverables to more precisely 
identify which components are causing cost or schedule overruns and to 
more effectively mitigate the root cause of the overruns. When the 
reports do not allow for traceability, project managers are not able 
to effectively measure progress, use the reports to monitor and 
control costs based on the original baseline, and/or to track where 
and why there were differences. For example, when we attempted to map 
the EVM data in the lower level reports for MAVEN's spacecraft, 
science operations center, remote sensing instrument, and Langmuir 
Probes and Waves instrument efforts to the overall MAVEN project EVM 
report, we were not able to see how the costs tracked from one report 
to another and therefore could not reconcile the costs between the 
reports. Such issues raise the question of which reports contain the 
true costs for these efforts. However, MAVEN officials walked us 
through the process they use to ensure that lower-level reports map to 
the project level reports. Furthermore, MAVEN officials said they do 
not mandate that their contractors follow a certain reporting format, 
instead any adjustments necessary to ensure that the lower-level 
reports map to the project-level reports are made by the project 
office. Though the MAVEN project does not prescribe to a standard 
reporting format, attempting to manually resolve incompatible pieces 
of data can become time-consuming, expensive and can lead to data 
reliability issues. We also had similar difficulty mapping various 
levels of EVM data for the MMS project. The MMS project was able to 
demonstrate how the Southwest Research Institute (SwRI) budget at 
completion in the lower-level report mapped to the SwRI budget at 
completion in the MMS project report, but because of the way the 
contractor submits their data, project officials said that the two 
reports will never match. Although the project was able to explain how 
the data tracked, again, attempting to manually resolve incompatible 
pieces of data can become time-consuming and can lead to data 
reliability issues. Since our review, MMS officials said the project 
is working to capture data at lower WBS levels which will allow for a 
closer tie between the cost and schedule data. 

Projects Have Not Consistently Applied EVM: 

[Side bar: Criteria: 
* Project uses an integrated EVM system across the entire project to 
measure cost, schedule, and performance goals.
* Project uses the 32 guidelines in ANSI/EIA-748. End of side bar] 

All the projects we reviewed were using some EVM to manage their work. 
However, the extent to which EVM is implemented across NASA's in-house 
projects and their contractors varies by project and center. For 
example, 3 of the 10 projects we reviewed did not report project level 
EVM data.[Footnote 19] Implementing EVM at the project level rather 
than just for the contract is considered a best practice. In addition, 
OMB policy requires the use of an EVM system for both in-house and 
contractor work and when there is both government and contractor work, 
the data from the two EVM systems must be consolidated at the 
reporting level for total program management and visibility. 
Integrating government and contractor cost, schedule, and performance 
status at the project level should result in better project execution 
through more effective management. In addition, some of the in-house 
projects we analyzed were only required to meet EVM principles and 
gather "EVM-like" data. Further, contractors such as the nonprofit 
organization managing the MMS instrument suite, do not report its EVM 
in the standard contract performance report format because they are 
only required to meet the intent of the standard. Other contractors 
for the JWST and MAVEN projects are required to provide EVM reports 
that show cost and schedule data by WBS elements. 

Cultural, Technical, and Other Challenges Seen as Impediments to EVM 
Implementation: 

NASA EVM focal points, headquarters officials, project 
representatives, and program executives cited cultural and technical 
challenges, as well as other challenges, as impediments to the 
effective use of EVM at the agency. NASA's culture traditionally has 
focused on solving science and engineering challenges and not on 
monitoring cost and schedule data such as data produced by an 
effective EVM system. Technical challenges were also cited as an 
impediment to effective EVM use, but opinions differed within NASA on 
the extent of their impact. The technical challenges cited involved 
difficulty in gathering sufficiently detailed data for timely 
inclusion and analysis in an EVM system. In addition, though NASA has 
not conducted an EVM skills gap analysis, NASA representatives said it 
is a challenge for the agency to implement EVM effectively due to a 
lack of sufficient staff with the skills and experience to analyze EVM 
data. 

NASA Culture Seen as Not Valuing EVM: 

Almost half of the more than 30 NASA representatives we interviewed, 
including a large number of those charged with implementing EVM 
agencywide--known as EVM focal points--and officials from the Chief 
Engineer's office, cited NASA's culture as a challenge to the 
effective use of EVM at the agency. Specifically, several NASA 
representatives said that historically, NASA's culture has not focused 
on or valued the kind of information that EVM can highlight. For 
example, a NASA EVM analyst said the culture has been focused on 
science and engineering and that accomplishment of that work has been 
the first priority for managers. Discussion of the cost of the work 
has been a secondary concern. A NASA EVM focal point described 
overcoming a culture that is based on applied and basic research 
rather than discrete tasks with discrete deliverables, which are among 
the requirements for effective EVM implementation. Further, a senior 
official at NASA headquarters told us that in project reviews, 
discussion of cost and schedule information, like EVM data and 
analysis, tends to be pushed to the very end of the review meeting and 
generally is not discussed in detail. Because EVM data is not 
universally valued within NASA, in some cases, the data generated to 
satisfy a project's EVM requirement may be of limited use. For 
example, one NASA EVM focal point said some managers are just 
"checking the box" with respect to using EVM. The task is performed 
but the requirement to collect EVM data was viewed as a nuisance that 
ultimately did not provide worthwhile information. Nonetheless, 
several of those we interviewed said persistent inquiries about EVM 
data from senior management at headquarters, especially over the last 
couple of years, are having a positive impact on the culture and 
forcing projects to pay more attention to the data. 

Technical and Other Challenges Cited as Making EVM Use More Difficult: 

The NASA representatives we interviewed also cited technical 
challenges as having an impact on the effective use of EVM, although 
their views varied on the extent of these challenges. For example, 
about half of the focal points we interviewed reported that a 
challenge to using an EVM system at NASA was aligning it with the 
agency's accounting system, the SAP Core Financial System. One of the 
problems cited was that EVM data collection may require more detailed 
data than a project has collected for the agency's accounting system 
and this may require the use of estimated costs instead of actual 
costs. Estimating these costs can create additional work for the 
project, delay the production of EVM data, and limit the reliability 
of the EVM data that is produced. Nonetheless, a NASA manager said, 
projects could more effectively plan their work to better accommodate 
the accounting system. For example, NASA's accounting system is set up 
to measure and report on labor in terms of full-time equivalents. A 
project, however, may have set up its earned value management system 
with a different measure for labor, such as productive hours. As a 
result, the accounting system cannot fill in the proper numbers for an 
earned value analysis, potentially causing more work for the project. 
Further, the EVM data could be less accurate due to the use of 
estimates rather the actual figures. However, if the project had 
planned from the outset to have the same measure of labor as the 
accounting system, there would not be a problem having this data fit 
the EVM system. 

An Office of the Chief Financial Officer representative did not 
believe that the projects have consistently demonstrated that the 
accounting system is a problem, but nonetheless agreed that potential 
work-arounds and slight changes to processes are potential solutions 
for these issues. The Office of the Chief Financial Officer has 
started an initiative to address both the level of detail of the data 
and improve the monitoring of contractor cost performance at levels 
that may be lower than levels at which obligations are made and costs 
are reported in the financial system.[Footnote 20] A report on a NASA 
EVM pilot project noted that the greatest impediment for implementing 
EVM is cultural resistance, not technical challenges. Specifically, it 
noted that "it's not the EVM Process. It's not the EVM Tools. It's not 
the SAP Accounting System. It is the NASA culture." 

NASA's use of contracts with one entity to provide goods or services 
to several different NASA projects was also cited as a challenge to 
use of EVM. For example, NASA centers may have a contract with one 
firm to provide engineering support services. Multiple projects may 
seek services using a single task order on this contract. Because of 
the way the NASA accounting system is configured, this approach can 
create artificial variances when looking at EVM data on a month-to-
month basis. According to a NASA representative, contractual 
requirements can correct these issues and allow for a closer 
accounting of the funds for EVM purposes and in fact, one NASA center 
has already instituted such requirements in a new contract for 
services. NASA plans to address this issue as current service 
contracts expire, but it will take time for the new data requirements 
that would provide the desired data to be implemented. 

Other challenges cited include the difficulty of gathering sound EVM 
data from nonprofit subcontractors, such as universities. One project, 
for example, reported that EVM data from nonprofit subcontractors were 
immature or non-existent. The nonprofits may be doing a significant 
amount of work for a center, but are not equipped to collect earned 
value data at the level of detail needed, Office of the Chief 
Financial Officer and center representatives reported. EVM focal 
points said the problem of collecting EVM data from nonprofits to feed 
into a larger project-level EVM system could be mitigated through 
contract language that clearly specifies what data are required from 
contractors. According to these officials, this kind of language has 
been included in Jet Propulsion Laboratory contracts and has been 
successfully demonstrated as a result. Furthermore, NASA is concerned 
that if nonprofits and small businesses are required to have a fully 
compliant or certified EVM system, they may not be able to bid on the 
work. However, the Federal Acquisition Regulation is clear that no 
offeror can be eliminated from consideration for a contract award 
because the entity does not have a compliant or certified EVM system. 
[Footnote 21] 

Understanding of EVM Varies Widely Across NASA: 

NASA representatives we interviewed said there was a need for improved 
abilities across the agency to analyze EVM data and implement EVM 
systems. Specifically, several focal points said the challenge for 
NASA is not as much in obtaining EVM data because most of the 
information comes from private contractors responsible for much of 
NASA's work, but in having a staff that can analyze the data and 
integrate it at the project level. A senior NASA official also noted 
that the career civil servants, who typically are the first level of 
review for EVM data, do not have background or training in EVM and 
cannot conduct a sound analysis of the data. A project representative 
echoed this comment and noted a general awareness of EVM within 
projects but a shortage of in-depth knowledge to understand EVM 
fundamentals and how to interpret the data it produces. For example, 
some projects seek to reset EVM baselines to match funding 
allocations, which thwarts efforts to examine cost and schedule 
trends. One EVM focal point told us it has been difficult to convince 
project managers that EVM can predict what will happen in their 
projects given the highly technical nature of their work. For example, 
a senior manager of a program that experienced significant schedule 
delays and cost overruns stated that he is an "EVM skeptic" and that 
he does not see EVM data as helpful in helping him track the 
performance of a project. 

Additionally, the employee skill sets available to analyze and 
implement EVM vary widely from center to center, headquarters 
officials said. In recent years, NASA has provided EVM training to a 
large number of employees; however, the agency has not conducted a 
skills gaps analysis, which could help to determine the number and 
extent of the staff's EVM expertise. NASA centers may have staff skill 
levels reflective of the level of EVM use at the center. Some centers 
have many projects producing EVM data while others may only rarely 
work on a project that uses EVM. Without a sufficient number of 
trained staff to analyze contractor data and implement in-house EVM 
efforts, NASA will likely continue to struggle to effectively use EVM 
as a valuable project management tool. 

NASA Policy Is in Line with Best Practices but Implementation Remains 
the Challenge: 

NASA has undertaken several initiatives aimed at improving the 
agency's use of EVM. For example, NASA strengthened its spaceflight 
management policy to require projects to comply with the 32 ANSI/EIA-
748 guidelines and has developed the processes and tools for projects 
to meet this requirement through its new EVM system. While these are 
positive steps, the policy continues to lack a requirement for 
rigorous oversight or surveillance of how projects are implementing 
EVM and NASA does not require projects to use the new EVM system to 
implement the EVM requirement of the revised policy. In addition, the 
issues that have impeded NASA's ability to effectively implement EVM, 
such as its culture, are longstanding and, as a result, NASA has not 
had much success implementing EVM in the past. 

The agency's recent revision of NASA Procedural Requirements 7120.5--
the policy that governs NASA's spaceflight projects and contains 
project EVM requirements--strengthened the EVM requirements over prior 
versions of the policy. For example, the revised policy requires all 
spaceflight projects to demonstrate compliance with each of the 32 EVM 
guidelines as set forth in ANSI/EIA-748, whereas the prior policy only 
required projects to comply with seven high-level EVM 
principles.[Footnote 22] The new requirements took effect through the 
release of an interim directive on September 28, 2011 and have since 
been made final in NASA's most recent update to 7120.5. As a result, 
projects meeting EVM reporting thresholds that enter implementation 
after that date are required to comply with the new requirement. 
According to an agency official, the Office of the Chief Engineer and 
the mission directorates will determine which projects that began 
development under the prior policy must comply with the new EVM 
requirements. At major milestones, Office of the Chief Engineer 
representatives will review whether the projects have implemented the 
32 EVM guidelines. 

However, the new policy still only contains the minimum requirements 
for earned value management, such as the thresholds for implementing 
EVM and the requirement to comply with ANSI/EIA-748 guidelines. 
[Footnote 23] The policy does not require projects to implement formal 
independent surveillance of their EVM systems. Without effective 
surveillance, agencies cannot ensure they are meeting the ANSI/EIA-748 
guidelines because internal management systems are not being reviewed 
to determine if they are providing reliable cost, schedule and 
technical performance data. In addition, effective surveillance helps 
pinpoint problems, and is useful for verifying the effectiveness of 
corrective action plans used to mitigate EVM system deficiencies. 
While projects are not required to implement formal independent 
surveillance, NASA does plan to conduct periodic surveillance of 
project EVM systems. For example, NASA plans to conduct EVM 
assessments at two key decision point life cycle reviews and through 
the Office of the Chief Engineer compliance surveys.[Footnote 24] 
While these methods will increase the agency's surveillance efforts, 
best practices call for project level surveillance to be an ongoing, 
continuous process conducted by an independent surveillance function. 

The policy also does not require projects to use NASA's new EVM 
system, although the system was designed to help projects meet the 
ANSI/EIA-748 guidelines. We found that the system meets the intent of 
the ANSI/EIA-748 guidelines. Examples of how NASA's EVM system will 
satisfy three key ANSI/EIA-748 guidelines are summarized in table 3 
below. 

Table 3: Examples of NASA EVM System Compliance with ANSI/EIA 
Guidelines: 

ANSI/EIA guideline: 1. Define the authorized work elements for the 
program. A work breakdown structure, tailored for effective internal 
management control, is commonly used in this process; 
NASA EVM system description: 
* Requires a work breakdown structure and outlines how the work will 
be structured and controlled. 

ANSI/EIA guideline: 6. Schedule the authorized work in a manner that 
describes the sequence of work and identifies significant task 
interdependencies required to meet the requirements of the program; 
NASA EVM system description: 
* Outlines how work will be scheduled including the methodology used 
to create the schedule; 
* Explains how the project will document activity and milestone 
sequences through the assignment of predecessor and successor 
interdependencies within detailed Control Account schedules; 
* Establishes a process for updating and making changes. 

ANSI/EIA guideline: 8. Establish and maintain a time-phased budget 
baseline, at the control account level, against which program 
performance can be measured. Initial budgets established for 
performance measurement will be based on either internal management 
goals or the external customer negotiated target cost, including 
estimates for authorized but undefinitized work. Budget for far-term 
efforts may be held in higher level accounts until an appropriate time 
for allocation at the control account level. If an over-target 
baseline is used for performance measurement reporting, prior 
notification must be provided to the customer; 
NASA EVM system description: 
* Outlines the Performance Measurement Baseline, which is the time-
phased budget expressed in dollars with separate identification of 
cost elements. It applies to the entire scope of authorized work and 
spans the project's period of performance; 
* Outlines the process and steps for creating a Performance 
Measurement Baseline that is time-phased via integration with the 
network schedule at the Work Package or Planning Package level. The 
Performance Measurement Baseline consists of the aggregation of 
individual time-phased Control Account budgets, including all burdens, 
for current project plans; 
* Includes how the control accounts should reflect the budget for the 
work planned and outlines who is responsible for completing the tasks. 

Source: GAO analysis of draft NASA EVM system description. 

[End of table] 

For the projects required to comply with the new policy, use of the 
agency-developed EVM system would meet the ANSI/EIA guidelines; 
however, some projects will be permitted to continue using their 
individual EVM systems as long as the 32 guidelines are met. According 
to agency officials, while future revisions to the policy may require 
use of the standardized agency-developed EVM system by all projects, 
at this time, the agency does not plan to require projects to use the 
agency-developed system in order to meet the guidelines. Instead, 
senior managers will determine on a case by case basis whether a 
project will use the agency's new EVM system. Currently, only the 
Space Launch System and ICESat-2 projects have been selected to 
implement the new EVM system. According to an agency senior official, 
the Agency Program Management Council approved a phased rollout of the 
new system because NASA does not have the resources to implement it 
agency-wide. For example, there are not enough NASA subject matter 
experts to provide the support needed by the projects when applying 
the new EVM system and to build the institutional capability at the 
centers. Their approach aims to incrementally build the capacity to do 
EVM, and seek increased acceptance of EVM as the requirement for its 
use is expanded. 

Strong Leadership Needed to Fully Implement EVM: 

Over the years, NASA has attempted to address its EVM shortcomings 
through a series of policy changes, but these efforts have failed to 
adequately address the cultural resistance to implementing EVM 
highlighted by many of the NASA officials we interviewed. NASA has 
made uneven progress since we reported in 2004 that the agency needed 
to improve its use of EVM as a project management tool. Furthermore, a 
2008 NASA internal study noted that projects needed to be educated on 
the value and approaches for using EVM and to provide support for 
setting up EVM within the projects early. Also, an internal agency 
briefing on EVM stated that a change management initiative would be 
necessary in order to successfully implement EVM at NASA centers. Our 
work has also shown that implementing a large-scale initiative, such 
as EVM, requires more than just policy changes.[Footnote 25] To see 
real change and, in effect, a cultural shift at NASA, top leadership 
must provide to employees a succinct and compelling reason that 
effective implementation of EVM is important. Articulating a 
compelling reason for implementing EVM enables employees and other 
stakeholders to understand the expected outcome of the management 
initiative and engenders not only their cooperation, but also their 
ownership of the outcome, which our work has shown can take at least 5 
to 7 years to fully implement. NASA, by having a policy that is not 
comprehensive, allowing projects to opt out of using the new EVM 
system, and not committing resources to adequately train staff, 
continues to limit progress in the cultural change needed to implement 
EVM. Without breaking through the cultural resistance to EVM and 
committing to efforts intended to strengthen the use of EVM, NASA is 
missing an opportunity to make full use of a key tool that could help 
it to manage its projects more effectively. 

Conclusions: 

Implementing an effective earned value management system and using it 
across a large federal agency with well-established processes is 
without doubt a challenging task. However, NASA has had uneven 
progress to date. NASA acknowledges that EVM can be a valuable tool 
for monitoring project development and has initiated an effort to 
implement an agencywide system. Currently, only a few of the 10 major 
spaceflight projects we reviewed were able to produce reliable EVM 
data, raising concern that they cannot produce reliable estimates of 
cost at completion. Moreover, until the data are sufficiently 
reliable, NASA, as well as external stakeholders, lose valuable 
insights into project performance that EVM provides. A sound EVM 
system is not merely an accounting tool; it can alert managers to 
developing problems so that they can be proactive in reducing the 
project's cost and schedule overruns. However, NASA is not making full 
use of a key tool that could help it address the cost and schedule 
issues that have kept NASA acquisition management on GAO's high risk 
list for more than 20 years. 

Though NASA's recent efforts to improve its EVM capability and 
strengthen its policy are steps in the right direction, implementation-
-fully integrating EVM into management processes--has been the biggest 
challenge and remains so today. NASA faces cultural and technical 
challenges that it must overcome to successfully implement an earned 
value system and to use this data on a regular basis to inform 
decision-making. Managing change will be key if NASA's latest effort 
to overcome these challenges and implement an agencywide EVM plan is 
to succeed. To accomplish effective earned value management, strong 
leadership is required to set an expectation that reliable and 
credible data are necessary to manage a successful project. This 
should be buttressed with a sound EVM policy and system that are 
required, and a commitment of resources to enable staff. Without 
sustained momentum and commitment, its current efforts could suffer 
the same consequence as those in the past. 

Recommendations for Executive Action: 

To improve NASA management and oversight of its spaceflight projects, 
we recommend that the NASA Administrator direct the appropriate 
offices to take the following four actions: 

* Establish a time frame by which all new spaceflight projects will be 
required to implement NASA's newly developed EVM system, unless the 
project is proposing to use a certified system, to ensure that in-
house efforts are compliant with ANSI/EIA-748. The time frame selected 
should take in to account the need to increase NASA's institutional 
capability for conducting EVM and analyzing and reporting the data. 

* Conduct an EVM skills gap analysis to identify areas requiring 
augmented capability across the agency. Based on the results of the 
assessment, develop a workforce training plan to address any 
deficiencies. 

* Develop an EVM change management plan to assist managers and 
employees throughout the agency with accepting and embracing earned 
value techniques while reducing the operational impact on the agency. 
The plan should include a strategy for having the agency's senior 
leadership communicate their commitment to implementation of EVM. 

* To improve the reliability of project EVM data, NASA Procedural 
Requirements (NPR) 7120.5 should be modified to require projects to 
implement a formal surveillance program that: 

- Ensures anomalies in contractor-delivered and in-house monthly 
earned value management reports are identified and explained, and 
report periodically to the center and mission directorate's leadership 
on relevant trends in the number of unexplained anomalies. 

- Ensures consistent use of WBSs for both the EVM report and the 
schedule. 

- Ensures that lower level EVM data reconcile to project level EVM 
data using the same WBS structure. 

- Improves underlying schedules so that they are properly sequenced 
using predecessor and successor dependencies and are free of 
constraints to the extent practicable so that the EVM baseline is 
reliable. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to NASA for comment. In its written 
comments, reproduced in appendix IV, NASA's Chief Engineer stated that 
the agency concurred with two recommendations and partially concurred 
with two other recommendations. In particular, the agency agreed with 
our recommendation to perform an EVM skills gap analysis and develop a 
workforce training plan to address any deficiencies identified. To 
that end, NASA plans to conduct a skills gap assessment and to augment 
its EVM training program to address the gaps identified. In addition, 
the agency also concurred with our recommendation to develop an EVM 
change management plan and is planning to develop a strategy targeted 
at all levels of the workforce from project team members to the 
agency's leadership. 

The agency partially concurred with our recommendation that NASA 
establish a time frame by which all new spaceflight projects will be 
required to implement NASA's newly developed EVM system, stating that 
they already require projects to perform EVM with an ANSI/EIA 748 
compliant system. NASA stated that its phased rollout approach for 
implementing the agency's EVM system is based on available resources, 
budgetary constraints, and institutional and project needs. However, 
NASA's approach does not include a timeframe for when projects will be 
required to use the new system. We recommended that NASA establish a 
timeframe for rolling out the system to all projects because a large 
number of projects are not in compliance with NASA's requirement, and 
very few are implementing the new EVM system. Using the newly 
developed EVM system could help projects better ensure NASA's projects 
are using a system that is compliant with the ANSI/EIA standard. The 
agency also noted its disagreement with the notion that all projects, 
in particular those that have a skilled EVM workforce and a compliant 
system in place, should be forced to use the agency's new system. 
Accordingly, we acknowledged in our report that there may be 
situations where a project should not be required to use the agency's 
EVM system, such as when a project already uses a certified system or 
for current, ongoing projects. Furthermore, we reported that NASA 
lacks the appropriate level of surveillance of its projects' EVM 
systems to monitor project adherence to the EVM standard; in addition, 
the extent to which EVM has been effectively implemented across NASA's 
projects varies. If NASA chooses not to require projects to use its 
new system it should take steps to ensure that it monitors their 
compliance with the standard. Finally, while we appreciate that NASA 
must balance its resources with its needs, the benefits that an 
effective EVM system can provide, such as allowing project managers to 
identify cost growth and take actions to stem further growth, warrants 
prioritization of resources to ensure earlier widespread 
implementation of NASA's EVM system. 

The agency also partially concurred with our recommendation that NPR 
7120.5 be modified to require projects to implement a formal EVM 
surveillance program. Citing resource constraints, NASA commented that 
it does not plan to implement a formal surveillance program, but 
agreed that the reliability and utility of the EVM data needed to be 
improved. As a result, the agency plans to establish a surveillance 
process, expand the workforce's EVM skills, and provide analytical 
tools including developing an EVM System Acceptance and Surveillance 
Guide. Furthermore, NASA said that it was not appropriate to 
incorporate the surveillance requirement in NPR 7120.5 because of the 
level of detail associated with requirements in that policy. The most 
important part of our recommendation is that EVM surveillance should 
be required to ensure better quality data. We reported that only 4 of 
the 10 projects we assessed had a comprehensive EVM surveillance 
system in place and the others had limited or no surveillance being 
performed and only 3 of the 10 projects had fully reliable data. 
Without an effective surveillance program, NASA cannot ensure its 
projects are meeting the ANSI/EIA-748 standard because internal 
management systems are not being reviewed to determine if they are 
providing reliable cost, schedule and technical performance data. 

In its response, NASA also noted that the project data we used in our 
report is over a year old and does not take in to account progress 
made over the past year. We disagree and note that in the report we 
discuss progress the agency has made over the past year, such as 
strengthening the EVM requirements in its policy and developing its 
new EVM system. Furthermore, we did not solely rely on project EVM 
data to develop our findings. For example, interviews with project 
officials and additional documentation they provided further validated 
our findings. Finally, it is important to note that NASA Acquisition 
Management has been on GAO's High Risk list for many years due to the 
agency's cost and schedule performance on its major projects. EVM is 
an important project management tool that can assist project managers 
in managing and assessing performance. Not addressing key issues that 
impact the availability of accurate and reliable data could lessen the 
usefulness of this key project management tool. 

NASA also provided technical comments, which have been addressed in 
the report, as appropriate. 

We are sending copies of this report to interested congressional 
committees, NASA's Administrator, and other interested parties. In 
addition, this report will also be available at no charge on the GAO 
website at [hyperlink, http://www.gao.gov]. 

Should you or your staff have any questions concerning this report, 
please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff who made key 
contributions to this report are listed in appendix V. 

Signed by: 

Cristina T. Chaplain: 
Director: 
Acquisition and Sourcing Management: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine to what extent NASA's major spaceflight projects are 
using earned value management (EVM) to manage acquisitions, we 
reviewed all NASA major spaceflight projects with a life cycle cost of 
over $250 million that were in the implementation phase and thus 
required to report EVM. There were 13 projects that met these 
criteria. Of these, 2 projects had recently launched and the launch of 
a 3rd was imminent. These 3 were excluded from our assessment because 
the work on these projects was nearly complete. Collectively, the 10 
projects we reviewed will cost over $14 billion to develop. 

Our review looked at EVM data for the period of August 2010 to August 
2011. While the majority of the 10 projects we reviewed had at least 6 
months of EVM data, a few did not because the project had only 
recently entered the implementation phase. Additionally, some projects 
were undergoing a re-plan and, therefore, were not required to provide 
EVM data for certain periods of time. In particular, the Tracking and 
Data Relay Satellite Sustainment contract had 5 months of data, the 
Mars Atmosphere Volatile Evolution Mission project had only 3 months 
of data, and the Orbiting Carbon Observatory 2 Orbital contract had 
just 4 months of data. The James Webb Space Telescope contract had EVM 
data for the whole period; however, the contractor underwent a major 
replan in which all EVM data except for the reporting of actual costs 
were suspended from January 2011 to April 2011. Although the Global 
Participation Measurement (GPM) project provided EVM reports for the 
entire project, we did not conduct an analysis of the project EVM 
data, because the performance reports did not contain the detailed 
data we needed for our analysis. However, we were able to assess the 
performance data for GPM's Microwave Imager Instrument. 

To determine cost and schedule performance for the selected major 
projects based on an evaluation of the earned value data, we analyzed 
project and contractor data and documentation including contract 
performance reports; project work breakdown structures; project 
schedules; integrated baseline review briefings; the extent to which 
surveillance of the EVM system was occurring; and monthly management 
briefings for the 10 major spaceflight projects. Specifically, we 
compared project documentation with EVM and scheduling best practices 
as identified in GAO's Cost Estimating and Assessment Guide and 
Schedule Assessment Guide.[Footnote 26] To the extent practicable, we 
assessed how each of the 10 projects' EVM data adhered to 3 of the 
American National Standard Institute's (ANSI) and Electronic 
Industries Alliance (EIA) 32 guidelines. In addition, we assessed the 
projects against 3 fundamental EVM practices that we believe are 
necessary for maintaining a reliable EVM system, as identified in our 
cost guide.[Footnote 27] We also analyzed the contract performance 
reports for each project to determine the level of data reliability. 
Specifically, we identified instances of the following: (1) negative 
planned value, earned value, or actual cost; (2) planned value and 
earned value without actual cost; (3) earned value and actual cost 
without planned value; (4) actual cost without planned value or earned 
value; (5) earned value without planned value and actual cost; (6) 
inconsistencies between the estimated cost at completion and the 
planned cost at completion; (7) actual cost exceeding estimated cost 
at completion; and (8) planned or earned values exceeding planned cost 
at completion. 

For the contracts that had more than 6 months of data, we used 
contract performance report data in order to generate our estimated 
overrun or underrun of the contract cost at completion by using 
formulas accepted by the EVM community and printed in the GAO Cost 
Estimating and Assessment Guide. To perform this analysis, we examined 
contractor performance reports over the period for which we had data 
to show trends in cost and schedule performances. We generated 
multiple formulas for the projected contract cost at completion that 
were based on how much of the contract had been completed up to August 
2011 or earlier for some projects. The ranges in the estimates at 
completion are driven by using different efficiency indices based on 
the project's past cost and schedule performance to forecast the cost 
of the remaining work and adding that cost to the actual costs to 
date. The efficiency indices capture how the project has performed in 
the past and can be useful in predicting how it will perform in the 
future. 

We also analyzed monthly project management review briefings to 
support our analysis. Finally, we analyzed the earned value data 
contained in EVM performance reports obtained from the projects. To 
perform this analysis, we compared the cost of work completed with 
budgeted costs for scheduled work to show trends in cost and schedule 
performances. 

To assess the reliability of the cost data, in addition to 
electronically testing the data for anomalies, we also reviewed 
relevant project documentation and interviewed agency and project 
officials about the data. We then followed up on these anomalies with 
the project offices that manage each of the spaceflight projects by 
sharing our preliminary analysis for each of the 10 projects. When 
warranted, we updated our analyses based on the agency's response and 
additional documentation provided to us. The data that we used were 
sufficiently reliable for how we portrayed them in our report and we 
are making recommendations to the agency to improve NASA's data 
reliability based on the findings discussed in our report. We did not 
test the adequacy of the agency or contractor accounting systems. 

To support and clarify information in our documentation reviews, we 
interviewed agency officials at NASA headquarters and EVM Focal Point 
Working Group members--the agency officials that are responsible for 
developing an integrated, consistent approach for implementing EVM 
throughout NASA, as well as addressing EVM review and surveillance 
issues and activities--at each center and the Human Exploration and 
Operations and Science mission directorates to discuss their roles as 
well as the extent to which EVM data is used to inform decision 
making. We interviewed officials at NASA headquarters in Washington, 
D.C.; and officials from Ames Research Center in Moffett Field, 
California; Dryden Flight Research Center in Edwards, California; 
Glenn Research Center in Cleveland, Ohio; Goddard Space Flight Center 
in Greenbelt, Maryland; Johnson Space Center in Houston, Texas; the 
Jet Propulsion Laboratory in Pasadena, California; Kennedy Space 
Center in Florida; Langley Research Center in Hampton, Virginia; 
Marshall Space Flight Center in Huntsville, Alabama; and Stennis Space 
Center in Mississippi. Additionally, we received responses to 
questions regarding the EVM data from each of the 10 projects we 
selected for review. These questions addressed how EVM practices are 
implemented at the project level and how the project utilizes EVM data. 

To determine the challenges that NASA has faced in implementing an 
effective EVM system, we interviewed NASA headquarters personnel to 
discuss the status and plans for implementing the agency-wide EVM 
system. We developed a standard set of questions and interviewed EVM 
Focal Point Working Group members at each center and the Human 
Exploration and Operations and Science mission directorates to assess 
the challenges of implementing EVM at individual centers and across 
the agency. We also interviewed a selection of senior officials and 
program executives at NASA headquarters that represent projects from 
each mission directorate and NASA center included in our engagement to 
obtain their perspective on the challenges of implementing and using 
EVM on their projects. We also reviewed prior GAO and NASA Inspector 
General reports that discuss the agency's prior efforts to implement 
EVM. We examined GAO and NASA Inspector General reports that discuss 
the importance of effective organizational change. Additionally, we 
received written responses to a standard set of questions regarding 
the challenges associated with implementing EVM from each of the 10 
projects we selected for review. 

To determine the steps that NASA is taking to improve its use of 
earned value management, we examined the results of NASA's EVM 
capability pilot projects and draft policies and guidance and compared 
these with best practices in EVM as discussed in GAO's Cost Estimating 
and Assessment Guide,[Footnote 28] the ANSI/EIA-748 standard, and OMB 
Circular A-11, Preparation, Submission, and Execution of the Budget 
and the Capital Programming Guide. In addition, we interviewed NASA 
headquarters personnel and EVM Focal Point Working Group members at 
each center and the Human Exploration and Operations and Science 
mission directorates to discuss and obtain information on ongoing 
initiatives the agency has undertaken. 

We conducted this performance audit from June 2011 to November 2012 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Importance of Earned Value Management: 

Pulling together essential cost, schedule, and technical information 
in a meaningful, coherent fashion is always a challenge for any 
project. Without this information, management of the project will be 
fragmented, presenting a distorted view of project status. For several 
decades, the Department of Defense (DOD) has utilized a tool called 
earned value management (EVM) to compare the value of work performed 
to the work's actual cost. Earned value management goes beyond the two-
dimensional approach of comparing budgeted costs to actual costs. It 
attempts to compare the value of work accomplished during a given 
period with the work scheduled for that period. By using the value of 
completed work as a basis for estimating the cost and time needed to 
complete the project, the earned value concept should alert project 
and senior managers to the potential problems early in the project. 

In 1996, DOD adopted 32 criteria for evaluating the quality of earned 
value management systems. These 32 criteria are organized into 5 basic 
categories: organization, planning and budgeting, accounting 
considerations, analysis and management reports, and revisions and 
data maintenance. In general terms, the criteria require contractors 
to: 

* define the contractual scope of work using a work breakdown 
structure; 

* identify organizational responsibility for the work; 

* integrate internal management subsystems; 

* schedule and budget authorized work; 

* measure the progress of work based on objective indicators; 

* collect the cost of labor and materials associated with the work 
performed; 

* analyze any variances from planned cost and schedules; 

* forecast costs at contract completion; and: 

* control changes. 

The criteria have evolved to become an American National Standards 
Institute (ANSI) and Electronic Industries Alliance (EIA) standard for 
EVM, which has been adopted by major U.S. government agencies, 
industry, and the governments of Canada and Australia. The full 
application of EVM system criteria is appropriate for large-cost 
reimbursable contracts where the government bears the cost risk. For 
such contracts, the management discipline described by the criteria is 
essential. In addition, data from an EVM system have been proven to 
provide objective reports of contract status, allowing numerous 
indices and performance measures to be calculated. These can then be 
used to develop accurate estimates of anticipated costs at completion, 
providing early warning of impending schedule delays and cost overruns. 

The standard format for tracking earned value is through a Contract 
Performance Report (CPR). The CPR is a monthly compilation of cost, 
schedule and technical data which displays the performance measurement 
baseline, any cost and schedule variances from that baseline, the 
amount of management reserve used to date, the portion of the contract 
that is authorized unpriced work, and the contractor's latest revised 
estimate to complete the project. 

As a result, the CPR can be used as an effective management tool 
because it provides the project manager with early warning of 
potential cost and schedule overruns. Using data from the CPR, a 
project manager can assess trends in cost and schedule performance. 
This information is useful because trends can be difficult to reverse. 
Studies have shown that once projects are 15 percent complete, the 
performance indicators are indicative of the final outcome. For 
example, a CPR showing a negative trend for schedule status would 
indicate that the project is behind schedule. By analyzing the CPR, 
one could determine the cause of the schedule problem such as delayed 
flight tests, changes in requirements, or test problems because the 
CPR contains a section that describes the reasons for the negative 
status. A negative schedule condition is a cause for concern, because 
it can be a predictor of later cost problems since additional spending 
is often necessary to resolve problems. For instance, if a project 
finishes 6 months later than planned, additional costs will be 
expended to cover the salaries of personnel and their overhead beyond 
what was originally expected. CPR data provides the basis for 
independent assessments of a project's cost and schedule status and 
can be used to project final costs at completion in addition to 
determining when a project should be completed. 

Examining a project's management reserve is another way that a project 
can use a CPR to determine potential issues early on. Management 
reserves, which are funds that may be used as needed, provide 
flexibility to cope with problems or unexpected events. EVM experts 
agree that transfers of management reserve should be tracked and 
reported because they are often problem indicators. An alarming 
situation arises if the CPR shows that the management reserve is being 
used at a faster pace than the project is progressing toward 
completion. For example, a problem would be indicated if a project has 
used 80 percent of its management reserve but only completed 40 
percent of its work. A project's management reserve should contain at 
least 10 percent of the cost to complete a project so that funds will 
always be available to cover future unexpected problems that are more 
likely to surface as the project moves into the testing and evaluation 
phase. 

An Integrated Baseline Review (IBR) is conducted to ensure the 
reliability of the EVM data and that the performance measurement 
baseline accurately captures all the work to be accomplished. Data 
from the CPR can then be used to assess project status--typically, 
monthly. Cost and schedule variances are examined and various 
estimates at completion are developed and compared to available 
funding. The results are shared with management for evaluating 
contractor performance. Finally, because EVM requires detailed 
planning for near-term work, as time progresses, planning packages are 
converted into detailed work packages. This cycle continues until all 
work has been planned and the project is complete. 

An IBR is an evaluation of the performance measurement baseline to 
determine whether all project requirements have been addressed, risks 
identified, and mitigation plans put in place and all available and 
planned resources are sufficient to complete the work. Too often, 
projects overrun because estimates fail to account for the full 
technical definition, unexpected changes, and risks. Using poor 
estimates to develop the performance measurement baseline will result 
in an unrealistic baseline for performance measurement. 

After the CPRs start being delivered to the government, it is 
important to oversee the project by conducting surveillance of the EVM 
system. Surveillance is reviewing a contractor's EVM system as it is 
applied to one or more projects. Its purpose is to focus on how well a 
contractor is using its EVM system to manage cost, schedule, and 
technical performance. For instance, surveillance checks whether the 
contractor's EVM system summarizes timely and reliable cost, schedule, 
and technical performance information directly from its internal 
management system; complies with the contractor's implementation of 
ANSI/EIA-748 guidelines; provides timely indications of actual or 
potential problems by performing spot checks, sample data traces, and 
random interviews; maintains baseline integrity; gives information 
that depicts actual conditions and trends; and provides comprehensive 
variance analyses at the appropriate levels, including corrections for 
cost, schedule, technical, and other problem areas. 

Effective surveillance ensures that the key elements of the EVM 
process are maintained over time and on subsequent applications. EVM 
system surveillance ensures that the contractor is following its own 
corporate processes and procedures and confirms that the contractor's 
processes and procedures continue to satisfy the ANSI guidelines. 

The surveillance team designated to perform project reviews should 
consist of a few experienced staff who fully understand the 
contractor's EVM system and the processes being reviewed. The 
surveillance organization should appoint the team leader and ensure 
that all surveillance team members are independent. This means that 
they should not be responsible for any part of the projects they 
assess. Key activities on the surveillance team's agenda include 
reviewing documents, addressing government project office concerns, 
and discussing prior surveillance findings and any open issues. 
Sufficient time should be allocated to all these activities to 
complete them. The documents for review should give the team an 
overview of the project's implementation of the EVM process. 

Successful surveillance is predicated on access to objective 
information that verifies that the project team is using EVM 
effectively to manage the contract and complies with company EVM 
procedures. Objective information includes project documentation 
created in the normal conduct of business. Besides collecting 
documentation, the surveillance team should interview control account 
managers and other project staff to see if they can describe how they 
comply with EVM policies, procedures, or processes. During interviews, 
the surveillance team should ask them to verify their responses with 
objective project documentation such as work authorizations, cost and 
schedule status data, variance analysis reports, and back-up data for 
any estimates at completion. 

[End of section] 

Appendix III: Case Studies of Selected Projects' Implementation of 
Earned Value Management: 

We conducted case studies of 10 major spaceflight system acquisition 
projects. This appendix provides a brief description of each project, 
including an analysis of the project's earned value data and trends. 
As part of our analysis, we assessed the projects' implementation of 
three fundamental earned value management (EVM) practices that we 
believe are necessary for maintaining a reliable EVM system--using a 
certified American National Standards Institute (ANSI) and Electronic 
Industries Alliance (EIA) compliant system, performing surveillance, 
and conducting integrated baseline reviews. 

We also assessed the projects' EVM data against three ANSI and EIA 
guidelines. These guidelines state that the authorized work elements 
for the project should be defined typically using a work breakdown 
structure (WBS) that has been tailored to the project and that the WBS 
is the same for the cost estimate, schedule, and EVM. The ANSI/EIA 
guidelines also state that projects should have a schedule that 
describes the sequence of work by listing activities in the order in 
which they are to be carried out and identifying significant task 
interdependencies required to meet project requirements. Finally, the 
ANSI/EIA guidelines state that a project should establish and maintain 
a time-phased budget baseline to track cost and schedule variances in 
an EVM system. 

As mentioned above, this appendix includes an analysis of each 
project's earned value trends from August 2010 to August 2011. These 
data and trends are often described in terms of cost and schedule 
variances. Cost variances compare the earned value of the completed 
work with the actual cost of the work performed. Schedule variances 
are also measured in dollars, but they compare the earned value of the 
completed work with the value of the work that was expected to be 
completed. Positive variances are good--they indicate that activities 
are costing less than expected or are completed ahead of schedule. 
Negative variances are bad--they indicate activities are costing more 
than expected or are falling behind schedule. Variances are merely 
measures that indicate that work is not being performed according to 
plan and that it must be assessed further to understand why. Although 
our EVM cost projections may show that a project is experiencing 
negative cost variances and schedule slippages, this does not mean 
that a project has exceeded its agency baseline commitment and will 
require additional funds to complete the project.[Footnote 29] These 
estimates use a project's EVM baseline, which represents only a 
portion of the agency baseline commitment for a project. The EVM 
baseline contains the cost and schedule contained in a project's 
management agreement minus unallocated future expenses and schedule 
margin held by the project and others above the project.[Footnote 30] 

As of August 2011, the budget at completion for the 10 projects was 
estimated to be $6.4 billion. To estimate the project variance at 
completion, we examined the trends for the earned value data for the 
entire project, if data was collected at that level, or elements of 
the project.[Footnote 31] Table 4 provides a summary of the projects' 
implementation of each EVM best practice we assessed and projected 
costs. 

Table 4: NASA 10 Spaceflights Project EVM Data Summary: 

Dollars in millions: 

Global Precipitation Measurement[C]; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: Agency partially met this criterion; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Global Precipitation Measurement[C]; 
Subcomponent[A]: GPM Microwave Imager; 
Data review period: August 2010-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $159; 
Cumulative cost variance: -$13; 
GAO VAC[B]: -$14 to -$22. 

James Webb Space Telescope; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

James Webb Space Telescope; 
Subcomponent[A]: Observatory; 
Data review period: May 2011-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $2,967; 
Cumulative cost variance: $2; 
GAO VAC[B]: NA. 

James Webb Space Telescope; 
Subcomponent[A]: Near infrared camera; 
Data review period: August 2010-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $205; 
Cumulative cost variance: -$33; 
GAO VAC[B]: -$34 to -$48. 

Landsat Data Continuity Mission; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Landsat Data Continuity Mission; 
Subcomponent[A]: Operational Land Imager; 
Data review period: August 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $129; 
Cumulative cost variance: -$45; 
GAO VAC[B]: -$49 to -$52. 

Lunar Atmosphere and Dust Environment Explorer; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Lunar Atmosphere and Dust Environment Explorer; 
Subcomponent[A]: Project; 
Data review period: September 2010-June 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $140; 
Cumulative cost variance: $3; 
GAO VAC[B]: -$0.1 to -$13. 

Magnetospheric Multiscale; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Magnetospheric Multiscale; 
Subcomponent[A]: Project; 
Data review period: August 2010-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $1,037; 
Cumulative cost variance: -$18; 
GAO VAC[B]: -$47 to -$80. 

Magnetospheric Multiscale; 
Subcomponent[A]: Instrument Suite; 
Data review period: August 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $233[D]; 
Cumulative cost variance: -$4; 
GAO VAC[B]: -$10 to -$24. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
partially met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Project; 
Data review period: June 2011-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $259; 
Cumulative cost variance: $5; 
GAO VAC[B]: NA. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Spacecraft; 
Data review period: January 2011-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $169; 
Cumulative cost variance: $6; 
GAO VAC[B]: $1 to $14. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Science Operations Center[E]; 
Data review period: November 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: [Empty]; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $3; 
Cumulative cost variance: $5; 
GAO VAC[B]: $0 to $0.4. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Remote sensing Package[E]; 
Data review period: November 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: [Empty]; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $20; 
Cumulative cost variance: $1; 
GAO VAC[B]: $1 to $7. 

Mars Atmosphere and Volatile Evolution Mission; 
Subcomponent[A]: Langmuir Probe and Waves Instrument[E]; 
Data review period: November 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: [Empty]; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $5; 
Cumulative cost variance: -$1; 
GAO VAC[B]: -$2 to -$3. 

Orbiting Carbon Observatory 2[F]; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Orbiting Carbon Observatory 2[F]; 
Subcomponent[A]: Project; 
Data review period: December 2010-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $129; 
Cumulative cost variance: $1; 
GAO VAC[B]: -$1 to -$26. 

Orbiting Carbon Observatory 2[F]; 
Subcomponent[A]: Spacecraft; 
Data review period: April 2011-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $30; 
Cumulative cost variance: 0; 
GAO VAC[B]: NA. 

Radiation Belt Storm Probes; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: [Empty] 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Radiation Belt Storm Probes; 
Subcomponent[A]: Project; 
Data review period: August 2010-August 2011[G]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $291; 
Cumulative cost variance: -$32; 
GAO VAC[B]: -$40 to -$41. 

Stratospheric Observatory for Infrared Astronomy Project; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
partially met this criterion; 
Conducted an integrated baseline review: Agency partially met this 
criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Stratospheric Observatory for Infrared Astronomy Project; 
Subcomponent[A]: Project; 
Data review period: August 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Agency did not meet this 
criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $384; 
Cumulative cost variance: $4; 
GAO VAC[B]: -$1.4 to -$76. 

Stratospheric Observatory for Infrared Astronomy Project; 
Subcomponent[A]: Airborne System; 
Data review period: August 2010-July 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $38; 
Cumulative cost variance: $3; 
GAO VAC[B]: $3 to $4. 

Tracking and Data Relay Satellite; 
Subcomponent[A]: Overall; 
Data review period: [Empty]; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: [Empty]; 
Cumulative cost variance: [Empty]; 
GAO VAC[B]: [Empty]. 

Tracking and Data Relay Satellite; 
Subcomponent[A]: Spacecraft; 
Data review period: August 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency met this 
criterion; 
BAC: $698; 
Cumulative cost variance: -$131; 
GAO VAC[B]: -$152 to -$185. 

Tracking and Data Relay Satellite; 
Subcomponent[A]: Sustainment; 
Data review period: August 2010-August 2011; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
met this criterion; 
Conducted an integrated baseline review: Agency met this criterion; 
EVM system surveillance is being performed: Agency met this criterion; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
BAC: $5; 
Cumulative cost variance: $0; 
GAO VAC[B]: NA. 

Source: GAO analysis of NASA and contractor data. 

[A] Items in italics represent subcomponents that are accounted for in 
higher level EVM reports. 

[B] Forecast range based on Actual cost + Budgeted Cost of Work 
Remaining/Performance factor where Performance factor was either the 
cumulative cost performance index (CPI) * schedule performance index 
(SPI), the current CPI, or current CPI * SPI. 

[C] In October 2011, the Global Precipitation Measurement mission was 
approved for a replan. 

[D] Only reflects Phases B-D. 

[E] The contract for the Science Operations Center, Remote sensing 
Package, Langmuir Probe and Waves Instrument does not exceed $50 
million. Therefore, the supplier is not required to have a certified 
system for this contract. 

[F] Orbiting Carbon Observatory 2 is in the process of being 
rebaselined due to a change in the launch vehicle. 

[G] Missing data for October and November 2010. 

NA - Not applicable means there was not enough data to forecast an 
Estimate at Completion. 

[End of table] 

With timely and effective action taken by project and executive 
management, it is possible to reverse negative performance trends so 
that the projected negative cost variances at completion may be 
reduced. To get such results, management needs to obtain reliable EVM 
data from EVM systems that adhere to the ANSI/EIA-748 standard for 
informed decision making. Until project offices undertake a rigorous 
validation of their EVM data, NASA faces an increased risk that 
managers may not be receiving the information they need to effectively 
manage their projects. 

The following information describes the key that we used in tables 5 
through 14 to convey the results of our assessment of the 10 case 
study projects' implementation of EVM practices. 

Key description: 
Fully met this criterion; 
Partially met this criterion; 
Did not meet this criterion. 

Global Precipitation Measurement: 

[Figure: artist depiction] 
* Project formulation start: July 2002; 
* Project confirmation date: December 2009; 
* Initial total project cost: $974.9 million; 
* Current total project cost: $932.8 million; 
* Launch readiness date: June 2014. 

Source: NASA GSFC. 

[End of figure] 

The Global Precipitation Measurement (GPM) mission, a joint NASA and 
Japan Aerospace Exploration Agency (JAXA) project, seeks to improve 
the scientific understanding of the global water cycle and the 
accuracy of precipitation forecasts. The GPM is composed of a core 
spacecraft carrying two main instruments: a Dual-frequency 
Precipitation Radar and a GPM Microwave Imager (GMI). GPM builds on 
the work of the Tropical Rainfall Measuring Mission, and will provide 
an opportunity to calibrate measurements of global precipitation when 
it launches in 2014. 

This analysis focuses only on the GMI-1 effort. Ball Aerospace and 
Technology Company is the prime contractor for GMI. GMI's current 
contract value is $217 million, which represents approximately 23 
percent of the total GPM project budget of $932.8 million. The GMI 
instrument was delivered to Goddard Space Flight Center in February 
2012 for integration into NASA's upcoming Earth science spacecraft. 
All remaining effort for GMI-1 is post delivery support, which is all 
level of effort.[Footnote 32] The GPM project provided EVM reports for 
the entire project, but we did not conduct an analysis of the project 
EVM data because the performance reports did not contain the detailed 
data we needed for our analysis. 

Table 5: Assessment of GPM EVM Practices: 

Dollars in millions. 

GPM Microwave Imager; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $159; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: -$14 to -$22. 

Source: GAO analysis of contractor EVM data as of July 2011. 

[End of table] 

GPM Microwave Imager Contractor Uses a Certified EVM System Compliant 
with the ANSI/EIA Standard: 

The GMI-1 contractor met the three fundamental ANSI/EIA-748 practices 
necessary for a reliable EVM system. The Defense Contract Management 
Agency (DCMA) certified that the GMI-1 contractor's EVM system 
complied with the ANSI/EIA standard in April 2008. Though the 
contractor has a certified system, the implementation of that system 
is questionable based on our findings below. We assessed GMI-1's EVM 
data against three ANSI/EIA guidelines. These guidelines state that 
the authorized work elements for the project should be defined 
typically using a WBS that has been tailored to the project and the 
WBS is the same for the cost estimate, schedule, and EVM. Our review 
found that the WBS in the GMI-1 schedule did not match the WBS used 
for the EVM data. Project officials said that the WBS for the project 
schedule did not match the WBS used in GMI contractor's EVM reports 
because GMI-1 is only one element of the total project and the project 
level schedule has a simplified summary of the GMI-1 schedule that was 
used for completeness. The ANSI/EIA guidelines also state that 
projects should have a schedule that describes the sequence of work by 
listing activities in the order in which they are to be carried out 
and identifying significant task interdependencies required to meet 
project requirements. Our review of the GMI-1 schedule found some 
sequencing issues. For example, about 5 percent of the remaining 
activities were missing predecessor and successor links, which are 
necessary for properly sequencing work so that the schedule will 
update in response to changes. We also found that 19 percent of the 
remaining activities had date constraints, which also hinder the 
schedule's ability to respond dynamically to status updates resulting 
in an artificial or unrealistic view of the project plan. These 
sequencing issues and constraint dates within the schedule affect the 
reliability of the overall network and the schedule's ability to 
correctly calculate float values and the critical path. Project 
officials agreed with our findings but said that these issues are 
corrected as they are discovered, so there is no impact to the 
project. Finally, though resource loading the schedule is not required 
to meet the ANSI/EIA guidelines, it is a best practice and therefore 
resources should be accounted for in the schedule in order to develop 
this baseline, according to the GAO cost guide. We found that the GPM 
schedule was not resource loaded, it is GAO's assessment that the 
project did not show evidence that it had established and maintained a 
time-phased budget baseline. 

Project Conducted an Integrated Baseline Review: 

The project conducted an integrated baseline review (IBR) in January 
2006. From that review, officials believed the results were less than 
satisfactory due to the contractor's inability to demonstrate the 
integration of contract schedule and cost in accordance with their EVM 
system description. In particular, 64 areas of concern were identified 
during the IBR, including major concerns with data continuity, cost/ 
schedule risk, and EVM processes. As a result of these issues and 
changes to the contract another IBR was performed in January 2011. 

EVM Surveillance Is Being Performed: 

Joint surveillance reviews of the EVM data are being performed by DCMA 
and the contractor. According to DCMA, although they have found some 
deficiencies in the contractor's EVM data, the contractor has 
responded with acceptable corrective action plans. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to July 
2011. Our review of the GMI-1 EVM data found various data anomalies 
that call into question the reliability of the data. For example, we 
found negative values for EVM data without any explanation in three 
monthly reports. Project officials responded that the negative values 
all fell within the contract threshold and therefore the GMI-1 
contractor did not need to provide an explanation in the variance 
analysis report. In addition, there were many instances of costs and 
performances being recorded when no work had been scheduled. In 
response to this finding, project officials explained that this work 
had been accounted for in previous months, which explain the missing 
values. Anomalous EVM data prevents the project from being able to 
gain meaningful and proactive insight into potential cost and schedule 
performance shortfalls, and take corrective action to avoid shortfalls 
in the future. Figure 2 below illustrates that as of July 2011, the 
project was reporting a negative cumulative cost variance of $13 
million and a negative cumulative schedule variance of $3 million. 

Figure 2: GAO Earned Value Analysis of GMI-1 Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: -$2.1 million; 
Cumulative schedule variance: -$7.6 million. 

September; 
Cumulative cost variance: -$1.8 million; 
Cumulative schedule variance: -$6.6 million. 

October; 
Cumulative cost variance: -$2.4 million; 
Cumulative schedule variance: -$5.6 million. 

November; 
Cumulative cost variance: -$1.1 million; 
Cumulative schedule variance: -$5.1 million. 

December; 
Cumulative cost variance: -$3.8 million; 
Cumulative schedule variance: -$4.6 million. 

2011:  

January; 
Cumulative cost variance: -$4.7 million; 
Cumulative schedule variance: -$3.4 million. 

February; 
Cumulative cost variance: -$5.9 million; 
Cumulative schedule variance: -$3.1 million. 

March; 
Cumulative cost variance: -$6.8 million; 
Cumulative schedule variance: -$3.2 million. 

April; 
Cumulative cost variance: -$8.1 million; 
Cumulative schedule variance: -$2.8 million. 

May; 
Cumulative cost variance: -$10.0 million; 
Cumulative schedule variance: -$2.8 million. 

June; 
Cumulative cost variance: -$11.6 million; 
Cumulative schedule variance: -$2.9 million. 

July; 
Cumulative cost variance: -$13.0 million; 
Cumulative schedule variance: -$2.9 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

GPM project officials said the cumulative negative cost and schedule 
variances were due to slips caused by suppliers not delivering flight 
hardware as planned, which pushed uncompleted work into the future. 
The negative variances were also caused by tasks being worked that 
were not included in the baseline. Officials also said the project 
experienced unfavorable variances in labor costs across all integrated 
project teams, which were further affected by the unfavorable 2010 
year end indirect rate adjustment.[Footnote 33] To address the 
negative schedule variance, officials said the contractor increased 
staffing and added extra shifts, which increased labor costs thereby 
increasing the contract value. Officials noted, however, that the 
increase in contract value did not translate into an increase in the 
baseline, just an increase in the project funding. Due to both the 
negative cost and schedule variances, we are forecasting a negative 
variance ranging from $14 million to $22 million at contract 
completion.[Footnote 34] According to NASA, the project is not 
overrunning its approved baseline commitment, because the EVM baseline 
does not include unallocated future expenses held at the project and 
headquarters level. 

James Webb Space Telescope: 

[Figure: artist depiction] 
* Project formulation start: March 1999; 
* Project confirmation date: July 2008; 
* Initial total project cost: $4,963.6 million; 
* Current total project cost: $8,835 million; 
* Launch readiness date: October 2018. 

Source: Northrop Grumman Aerospace Systems. 

[End of figure] 

The James Webb Space Telescope (JWST) is a large, infrared-optimized 
space telescope that is designed to find the first galaxies that 
formed in the early universe. Its focus will include searching for 
first light, assembly of galaxies, origins of stars and planetary 
systems, and origins of the elements necessary for life. Scheduled to 
launch in October 2018, JWST's instruments will be designed to work 
primarily in the infrared range of the electromagnetic spectrum, with 
some capability in the visible range. JWST will have a large primary 
mirror composed of 18 smaller mirrors, measuring 6.5 meters (21.3 
feet) in diameter, and a sunshield that is the size of a tennis court. 
A successor to the Hubble Space Telescope and the Spitzer Space 
Telescope, JWST will reside in an orbit about 1 million miles from the 
Earth. 

NASA rebaselined JWST in September 2011 and made changes in the 
project's management in response to cost and schedule performance 
issues and the recommendations of the Independent Comprehensive Review 
Panel report. As part of the rebaseline, NASA took the lead role for 
systems engineering from the prime contractor. The telescope, along 
with a segmented primary mirror, will deliver infrared light to the 
Fine Guidance Sensor and fine pointing updates to the Observatory and 
four scientific instruments including the Near-Infrared Camera 
(NIRCam), the Near-Infrared Spectrograph, the Mid-Infrared Instrument 
and the Fine Guidance Sensor/Near InfraRed Imager and Slitless 
Spectrograph. For work being performed by its international partners, 
such as the Near-Infrared Spectrograph, EVM data is not collected. 

At the time of our review, there was no EVM data for the overall JWST 
project or for the work done in-house on the Integrated Science 
Instrument Module. According to project officials, the Integrated 
Science Instrument Module effort began over a decade ago, and 
significant parts of the project, particularly those undertaken in-
house at GSFC were not structured to enable EVM to be implemented 
easily. However, the JWST project office is collecting EVM data from 
Northrop Grumman Aerospace Systems and obtains copies of the Lockheed 
Martin Space Systems-Advanced Technology EVM data, which is the 
University of Arizona's prime contractor. Northrop Grumman Aerospace 
Systems is responsible for developing and launching the JWST 
Observatory, which comprises the spacecraft, sunshield, and the 
optical telescope element, systems integration and test observatory 
verification, observatory commissioning, and ground and launch support 
equipment. The University of Arizona is responsible for developing the 
Near-Infrared Camera science instrument. 

Table 6: Assessment of JWST EVM Practices[A]: 

Dollars in millions. 

Observatory; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $2,967; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: NA[B]. 

Near infrared camera; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Agency partially met this 
criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $205; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: -$34 to -$48. 

Source: GAO analysis of contractor EVM data as of July 2011. 

[A] All budget and variance at completion data is before the JWST re-
plan. 

[B] NA - Not applicable means there was not enough data to forecast an 
Estimate at Completion. 

[End of table] 

Observatory Contractor Has a Certified EVM System Compliant with ANSI/ 
EIA Standard: 

The observatory contractor met all three fundamental ANSI/EIA-748 
practices necessary for a reliable EVM system. For the observatory 
portion of the JWST project, Northrop Grumman Aerospace Systems has a 
certified EVM System, which it uses to fulfill the earned value 
reporting requirement. Though the contractor has a certified system, 
the implementation of that system is questionable based on our 
findings below. We assessed the observatory contractor's EVM data 
against three fundamental ANSI/EIA guidelines. These guidelines state 
that the authorized work elements for the project should be defined 
typically using a WBS that has been tailored to the project and the 
WBS is the same for the cost estimate, schedule, and EVM. We found 
that while consistent naming conventions existed between the WBS and 
contract performance reports, there were discrepancies in the 
hierarchical structure and numbering of the WBS elements. For example, 
the WBS dictionary shows the Mission Assurance listed as 3.0, while 
the contract performance report provided by the observatory contractor 
has Mission Assurance listed as 2.0. NASA officials responded that the 
observatory contractor is not required to follow the project level WBS 
hierarchical structure nor the WBS numbering scheme. They further 
stated that the project and the observatory WBS structures are not 
identical because procurement of the observatory is only one element 
of the overall JWST project. NASA did not provide us with a schedule 
in a format that would allow us to determine if the schedule had the 
proper sequencing in place. As a result, we were unable to determine 
if significant task interdependencies such as predecessor and 
successor links were in place to ensure that the schedule would update 
in response to changes. In addition, because we could not view the 
schedule in its native software, we were unable to determine if the 
schedule was resource loaded, which is a best practice for 
establishing and maintaining a time-phased budget baseline. 

Project Conducted an Integrated Baseline Review: 

An Integrated Baseline Review was conducted in February 2010; however, 
because of the rebaseline in September 2011, according to the project 
an additional IBR was held for the observatory in October 2012. 

EVM Surveillance Is Being Performed: 

Surveillance is being performed by DCMA. In addition, EVM data is 
reviewed monthly by multiple individuals on the project office as well 
as at higher levels of NASA headquarters and the Goddard Space Flight 
Center. 

Data Resulting from the EVM System Was Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to July 
2011. Our review of the Observatory EVM data found various data 
anomalies that call into question the reliability of the data. For 
example, we found actual costs recorded without any work being 
performed, inconsistencies between the reported estimate at completion 
and budget at completion, large month to month performance swings, and 
unexplained variances. NASA officials explained that during this time 
period the observatory contractor was engaged in re-planning efforts 
so NASA did not want them to expend resources reporting performance 
management data to an outdated performance measurement baseline that 
did not reflect the new rebaseline assumptions. Further, while 
variances analyses were provided in the variance analysis reports for 
WBS elements that exceeded contractual thresholds, there was no 
explanation for the anomalies we found. A variance analysis report 
provides a detailed, narrative report explaining significant cost and 
schedule variances and other contract problems and topics. Without 
this information, management cannot understand the reasons for the 
variances and the contractor's plan for fixing them. When information 
is missing in a variance analysis report, the EVM data will not be 
meaningful or useful as a management tool. 

As of July 2011, the observatory portion of the JWST Project was 51 
percent complete with a positive cumulative cost variance of $2.4 
million. For the same period the project was also experiencing a 
positive cumulative schedule variance of $0.9 million as seen in 
figure 3 below. In January 2011, the observatory contractor began 
replanning the remaining effort in order to meet the October 2018 
launch readiness date. As a result, NASA suspended performance 
measurement reporting during the period of January 2011 through April 
2011. However, the observatory contractor was still required to submit 
contract performance reports depicting actual cost and estimate at 
completion data. 

Figure 3: GAO Earned Value Analysis of Observatory Data: 

[Refer to PDF for image: line graph] 

2010:  

August;  
Cumulative cost variance: -$27.1 million; 
Cumulative schedule variance: -$10.0 million. 

September; 
Cumulative cost variance: -$31.4 million; 
Cumulative schedule variance: -$10.2 million. 

October; 
Cumulative cost variance: -$35.2 million; 
Cumulative schedule variance: -$10.7 million. 

November; 
Cumulative cost variance: -$39.6 million; 
Cumulative schedule variance: -$8.1 million. 

December; 
Cumulative cost variance: -$40 million; 
Cumulative schedule variance: -$9.0 million. 

2011:  

January; 
Cumulative cost variance: -$1,434.6 million; 
Cumulative schedule variance: 0. 

February; 
Cumulative cost variance: -$1,448.5 million; 
Cumulative schedule variance: 0. 

March;
Cumulative cost variance: -$1.467.0 million; 
Cumulative schedule variance: 0. 

April; 
Cumulative cost variance: -$1,481.8 million; 
Cumulative schedule variance: 0. 

May; 
Cumulative cost variance: $0.014 million; 
Cumulative schedule variance: $0.3 million. 

June; 
Cumulative cost variance: $0.684 million; 
Cumulative schedule variance: $0.3 million. 

July; 
Cumulative cost variance: $2.4 million; 
Cumulative schedule variance: $0.9 million. 

Source: GAO analysis of NASA and contractor data. 

[End of figure] 

Since there were only 3 months of EVM data after the rebaseline, we 
were not able to forecast a variance at completion. Since our 
assessment, JWST project officials said they have made some 
significant improvements in implementation and use of EVM that 
includes an EVM approach for the in-house work that will provide EVM 
metrics to measure progress. In addition, the officials said the 
project is also doing managerial analysis on its contracts and project 
components and producing independent estimate of completion each month 
based on the EVM data. 

Near-Infrared Camera Contractor Has a Certified EVM System Compliant 
with ANSI/EIA Standard: 

The Near-Infrared Camera (NIRCam) contractor, Lockheed Martin, met one 
of the three fundamental ANSI/EIA-748 practices necessary for a 
reliable EVM system. The contractor has a certified EVM System that it 
is using to report EVM data for NIRCam. Our review found similar 
problems with the WBS hierarchical structure and numbering of the 
elements. For example, the WBS dictionary shows Mission Assurance 
listed as 3.0 while the NIRCam contract performance report shows this 
effort under element 5.5. NASA officials explained that the NIRCam 
contractor is not required to follow the project level WBS 
hierarchical structure or numbering of elements. As stated above, we 
did not receive a schedule in its native software, so we were unable 
to confirm whether the schedule was sequenced using predecessor and 
successor links or if it was resource loaded, necessary for 
establishing the time-phased budget baseline. 

Project Conducted an Integrated Baseline Review: 

As stated above, while the JWST project conducted an IBR in February 
2010, the rebaseline necessitated a new IBR, which occurred in March 
2012. 

EVM Surveillance Is not Being performed: 

While formal surveillance is not occurring for the Lockheed Martin EVM 
data, monthly reviews of the EVM data are performed by both the 
project and program offices and by independent groups. As part of 
these reviews, trending metrics are prepared and presented to 
management as part of the internal project reviews and monthly status 
reviews. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 through July 
2011 and found various data anomalies that call into question the 
reliability of the data. For example, we found EVM data with negative 
values, no work scheduled but work performed, and actual costs being 
incurred without any work being performed. NASA officials stated that 
variances that exceeded contractual thresholds should be reflected in 
the variance analysis reports, however many of these anomalies did not 
breach a variance threshold so the report provided no explanation. In 
addition, NASA officials explained that many of these anomalies 
occurred during the project replan, which was formally approved in 
September 2011, when the project was rebaselined. As a result of the 
replan, NASA suspended EVM data reporting, which resulted in many of 
the anomalies we found. 

As of July 2011, the NIRCam portion of the JWST project was 98 percent 
complete. In July 2011, there was a negative cumulative cost variance 
of $33 million and a negative cumulative schedule variance of $4.4 
million as seen in figure 4 below. The reasons for the downward trend 
reflected in the graph below were not explained because variance 
analysis reporting was suspended during the replan period. 

Figure 4: GAO Earned Value Analysis of NIRcam Data: 

[Refer to PDF for image: line graph] 

2010:  

August; 
Cumulative cost variance: -$2.7 million; 
Cumulative schedule variance: -$1.0 million. 

September;  
Cumulative cost variance: -$5.1 million; 
Cumulative schedule variance: -$0.5 million. 

October; 
Cumulative cost variance: -$7.7 million; 
Cumulative schedule variance: -$0.9 million. 

November; 
Cumulative cost variance: -$9.3 million; 
Cumulative schedule variance: -$1.3 million. 

December; 
Cumulative cost variance: -$11.5 million; 
Cumulative schedule variance: -$1.5 million. 

2011:  

January; 
Cumulative cost variance: -$14.7 million; 
Cumulative schedule variance: -$1.6 million. 

February; 
Cumulative cost variance: -$20.0 million; 
Cumulative schedule variance: -$3.7 million. 

March; 
Cumulative cost variance: -$23.0 million; 
Cumulative schedule variance: -$3.7 million. 

April; 
Cumulative cost variance: -$25.5 million; 
Cumulative schedule variance: -$3.7 million. 

May; 
Cumulative cost variance: -$28.1 million; 
Cumulative schedule variance: -$4.3 million. 

June; 
Cumulative cost variance: -$30.4 million; 
Cumulative schedule variance: -$4.4 million. 

July; 
Cumulative cost variance: -$33.0 million; 
Cumulative schedule variance: -$4.4 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

Based on the downward trend, we are forecasting a negative variance at 
completion ranging anywhere from $34 million to $48 million. This 
analysis is based on information prior to the project's 2011 replan. 
The variances projected do not take in to account the establishment of 
a new EVM baseline as a result of the replan. 

Landsat Data Continuity Mission: 

[Figure: photograph] 
* Project formulation start: August 2000; 
* Project confirmation date: December 2009; 
* Initial total project cost: $941.7 million; 
* Current total project cost: $931.2 million; 
* Launch readiness date: June 2013. 

Source: Orbital Science Corporation. 

[End of figure] 

The Landsat Data Continuity Mission (LDCM) is a joint mission between 
NASA and the United States Geological Survey (USGS) that seeks to 
extend the ability to detect and quantitatively characterize changes 
on the global land surface at a scale where natural and man-made 
causes of change can be detected and differentiated. It is the 
successor mission to Landsat 7. The Landsat data series, begun in 
1972, has provided the longest continuous record of changes in the 
Earth's surface as seen from space. 

Landsat data is a resource for people who work in agriculture, 
geology, forestry, regional planning, education, mapping, and global 
change research. The LDCM provides remotely sensed, highly calibrated, 
moderate resolution, multispectral imagery affording systematic global 
coverage of the Earth's land surfaces on a seasonal basis and makes 
the data readily available for large-scale and long-term Earth System 
Science and Land use/land cover change research and management. The 
project plans to launch early in February 2013. 

LDCM consists of an Operational Land Imager (OLI) and a Thermal 
Infrared Sensor (TIRS) science instrument, a spacecraft, and a mission 
operations element. LDCM does not collect EVM data at the project 
level. The decision not to perform EVM at the project level was 
reviewed extensively prior to proceeding into the design and 
development phase, according to project officials. Also, there is no 
EVM data for the spacecraft effort because this work is being done 
under a firm fixed price contract and NASA regulations do not require 
EVM for firm fixed price contracts. The TIRS instrument, built in-
house at NASA's Goddard Space Flight Center, was added late in the 
formulation phase with an aggressive delivery schedule and delivered 
in February 2012. The Ground System is being built and delivered by 
the USGS. The developmental part of the OLI contract with Ball 
Aerospace and Technology Corporation was completed with delivery of 
the instrument in early October 2011. Following on-orbit checkout, the 
contract will transfer to USGS for management. The OLI instrument is 
the only part of the project that performed EVM. 

Table 7: Assessment of Landsat OLI EVM Practices: 

Dollars in millions. 

Operational Land Imager; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $129; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: -$49 to -$52. 

Source: GAO analysis of contractor EVM data as of August 2011. 

[End of table] 

Operational Land Imager Contractor Has a Certified EVM System 
Compliant with ANSI/EIA Standard: 

The OLI contractor met two of the three fundamental ANSI/EIA-748 
practices necessary for a reliable EVM system. In 2007, the Defense 
Contract Management Agency (DCMA) certified Ball Aerospace and 
Technology Corporation's EVM system. Though the contractor has a 
certified system, the implementation of that system is questionable 
based on our findings below. We assessed the contractor EVM data 
against three ANSI/EIA guidelines. These guidelines state that the 
authorized work elements for the project should be defined typically 
using a WBS that has been tailored to the project and the WBS is the 
same for the cost estimate, schedule, and EVM. Our review found that 
the WBS in the OLI schedule was consistent with the WBS used for the 
EVM data. The ANSI/EIA guidelines also state that projects should have 
a schedule that describes the sequence of work by listing activities 
in the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. Our analysis found 11 percent of the remaining 
activities were missing dependencies, 13 percent had lags, and 24 
percent had constraints, among other things. When schedules are not 
sequenced properly, float values and the calculated critical path will 
not be valid. Project officials said the schedule sequencing is driven 
by external forces such as facilities availability, spacecraft and 
ground system interfaces, DCMA inspections, and so forth. The effects 
of these forces on schedule sequencing and critical path are reviewed 
extensively and the validity of the critical path is not typically an 
issue for project management. Finally, the ANSI/EIA guidelines state 
that a project should establish and maintain a time-phased budget 
baseline to track cost and schedule variances in an EVM system. Though 
resource loading the schedule is not required to meet the ANSI/EIA 
guideline, it is a best practice and therefore resources should be 
accounted for in the schedule in order to develop this baseline, 
according to the GAO cost guide. Our analysis found that the OLI 
schedule was resource loaded. 

Project Conducted an Integrated Baseline Review: 

The project conducted an IBR in 2007. The IBR identified 80 areas of 
concern and as a result the LDCM Project did not accept the 
Performance Measurement Baseline at the IBR. However since 2007, Ball 
Aerospace and Technology Corporation has addressed the areas of 
concern. 

EVM Surveillance Is Being performed: 

Joint surveillance reviews are being conducted on Ball Aerospace and 
Technology Corporation's EVM system by DCMA and the Defense Contract 
Audit Agency. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to August 
2011. While the EVM data reflected several data anomalies, Ball 
Aerospace provided detailed explanations for each of them. For 
example, negative values were attributed to year end rate savings, 
labor corrections, or material transfers that were greater than the 
current month actual costs which resulted in a negative number. 

The EVM data assessed below reflects all work associated with the LDCM 
OLI instrument. Figure 5 below illustrates that as of August 2011, the 
project was a reporting a negative cumulative cost variance of 
approximately $46 million and a negative cumulative schedule variance 
of $1.3 million. 

Figure 5: GAO Earned Value Analysis of Operational Land Imager Data: 

[Refer to PDF for image: line graph] 

2010:  

August; 
Cumulative cost variance: -$36.2 million; 
Cumulative schedule variance: -$2.9 million. 

September; 
Cumulative cost variance: -$37.1 million; 
Cumulative schedule variance: -$2.4 million. 

October; 
Cumulative cost variance: -$38.8 million; 
Cumulative schedule variance: -$2.7 million. 

November; 
Cumulative cost variance: -$39.6 million; 
Cumulative schedule variance: -$2.3 million. 

December; 
Cumulative cost variance: -$39.6 million; 
Cumulative schedule variance: -$2.3 million. 

2011:  

January; 
Cumulative cost variance: -$40.9 million; 
Cumulative schedule variance: -$2.1 million. 

February; 
Cumulative cost variance: -$42.2 million; 
Cumulative schedule variance: -$1.8 million. 

March; 
Cumulative cost variance: -$43.1 million; 
Cumulative schedule variance: -$2.0 million. 

April; 
Cumulative cost variance: -$43.9 million; 
Cumulative schedule variance: -$1.9 million. 

May; 
Cumulative cost variance: -$44.7 million; 
Cumulative schedule variance: -$2.0 million. 

June; 
Cumulative cost variance: -$45.6 million; 
Cumulative schedule variance: -$1.9 million. 

July; 
Cumulative cost variance: -$45.4 million; 
Cumulative schedule variance: -$1.2 million. 

August; 
Cumulative cost variance: -$45.6 million; 
Cumulative schedule variance: -$1.3 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The negative cumulative cost and schedule variances were due to 
various technical challenges experienced during instrument 
development, including detector fabrication issues, coatings issues 
that necessitated the build of a second calibration subassembly, and 
instrument integration onto the baseplate taking two weeks longer than 
expected to complete fabrication. Due to the negative variances, we 
are forecasting a negative variance at completion ranging from $49 
million to $52 million. According to a project official, the project 
is not overrunning its agency baseline commitment, because the EVM 
baseline does not include unallocated future expenses held by the 
project or NASA headquarters. The developmental part of the OLI 
instrument contract with Ball Aerospace and Technology Corporation was 
completed with delivery of the instrument in early October 2011. 

Lunar Atmosphere and Dust Environment Explorer: 

[Figure: artist depiction] 
* Project formulation start: February 2009; 
* Project confirmation date: August 2010; 
* Initial total project cost: $262.9 million; 
* Current total project cost: $262.9 million; 
* Launch readiness date: November 2013. 

Source: LADEE Project Office. 

[End of figure] 

The Lunar Atmosphere and Dust Environment Explorer (LADEE) will 
determine the global density, composition, and time variability of the 
lunar atmosphere. LADEE's measurements will determine the size, 
charge, and spatial distribution of electrostatically transported dust 
grains. Additionally, it will carry an optical laser communications 
demonstrator that will test high-bandwidth communication from lunar 
orbit.[Footnote 35] Finally, it will broaden the scientific 
understanding of other planetary bodies regarding exospheres or very 
thin atmospheres, like the moon. 

Table 8: Assessment of LADEE EVM Practices: 

Dollars in millions. 

Project; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $140; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: -$0.1 to -$13. 

Source: GAO analysis of NASA data as of June 2011. 

[End of table] 

Project Office Does not Have a Certified EVM System Compliant with the 
ANSI/EIA Standard: 

LADEE met only one of three key fundamental ANSI/EIA-748 practices for 
reliable EVM system. NASA's EVM guidance says that projects must start 
reporting EVM data once the project enters the project implementation 
phase, if the project's life-cycle cost is at or greater than $20 
million. While LADEE may not have a contract that exceeds $20 million, 
the overall LADEE project cost is about $262.9 million. Nevertheless, 
project officials said LADEE is responsible only for gathering "EVM-
like" data at the project level. The "EVM-like" data is collected 
using the "EVM Lite" process, which attempts to meet the ANSI/EIA-748 
standard where applicable. Project officials said that when LADEE was 
initiated in February 2008, NPR 7120.5D was still in effect which 
required application of EVM principles. This is, in effect, "EVM-Lite" 
or "EVM-Like." Officials further stated that prior to August 2011, the 
LADEE project evaluated candidate EVM techniques using sample data 
from January to March 2011. Based on that evaluation period, LADEE 
decided to use the "EVM Lite" technique to collect the necessary data 
to manage the project. From April to June 2011, additional evaluations 
of this technique continued. Therefore, standard reporting of the 
LADEE "EVM-like" project level data did not begin until August 1, 
2011. We reviewed all available EVM data from September 2010 to June 
2011, which was before the standard EVM reporting period began. 

Though LADEE does not have a certified system, we assessed how well 
LADEE project was meeting three ANSI/EIA guidelines. These guidelines 
state that the authorized work elements for the project should be 
defined typically using a WBS tailored to the project and the WBS 
should be the same for the schedule, cost estimate, and EVM. Our 
analysis found that the WBS in the LADEE schedule matched the WBS used 
for EVM data. The ANSI/EIA guidelines also states that projects should 
have a schedule that describes the sequence of work by listing 
activities in the order in which they are to be carried out and 
identifies significant task interdependencies required to meet project 
requirements. Our review of the LADEE schedule found 3 percent of the 
remaining activities were missing predecessor or successor links, 
which cause the schedule to not properly update in response to 
changes. We also found that about 6 percent of the remaining 
activities had date constraints and/or lags, which also hinder the 
schedule from responding dynamically to changes and can portray an 
artificial or unrealistic view of the project plan. While these issues 
may be relatively small, any missing dependencies, constraints, and 
lags may disrupt the reliability of the overall network. Finally, the 
ANSI/EIA guidelines state that a project should establish and maintain 
a time-phased budget baseline to track cost and schedule variances in 
an EVM system. Though resource loading the schedule is not required to 
meet the ANSI guidelines, it is a best practice and therefore 
resources should be accounted for in the schedule in order to develop 
this baseline, according to the GAO cost guide. We found that the 
LADEE schedule was not resource-loaded, therefore, it is GAO's 
assessment that the project did not show evidence that it had 
established and maintained a time-phased budget baseline. 

Project Conducted an Integrated Baseline Review: 

LADEE conducted an IBR in December 2010. From that review, a list of 
concerns about the EVM data and schedule were identified. 
Specifically, the project was not using a consistent approach to 
collect EVM data, calling into question the credibility of the data. 
In addition, it was unclear if an objective assessment of cost and 
schedule performance could be made using data generated by the "EVM 
Lite" approach. Since that review, officials said all issues and 
actions have been addressed and were formally closed by the IBR panel 
in July 2011. 

EVM Surveillance Is Not Being Performed: 

While formal surveillance is not occurring, EVM data assurance reviews 
are being performed during monthly management reviews with the Lunar 
Quest Program office.[Footnote 36] 

Data Resulting From the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from September 2010 to June 
2011. Our review of LADEE's EVM data found various data anomalies that 
call into question the reliability of the data. For example, from 
October 2010 to December 2010 there were some instances of negative 
values in the EVM reports that were unexplained. Since a variance 
analysis report provides a detailed narrative explaining significant 
cost and schedule variances, when this information is missing, 
management cannot understand the reasons for variances and the plan 
for fixing them. Also, the EVM data provided by the project office was 
not presented in a standard EVM format. This could be attributed to 
the fact that LADEE is required to provide only "EVM-like" data. 

Figure 6 below illustrates that as of June 2011, the LADEE project was 
reporting a positive cumulative cost variance of $3 million and a 
negative cumulative schedule variance of $10 million. 

Figure 6: GAO Earned Value Analysis of LADEE Project Data: 

[Refer to PDF for image: line graph] 

2010: 

Sept. 
Cumulative cost variance: $0.3 million; 
Cumulative schedule variance: -$0.5 million. 

Oct. 
Cumulative cost variance: $4.4 million; 
Cumulative schedule variance: -$5.2 million. 

Nov. 
Cumulative cost variance: $6.2 million; 
Cumulative schedule variance: -$6.3 million. 

Dec. 
Cumulative cost variance: $9.0 million; 
Cumulative schedule variance: -$4.6 million. 

2011:  

Jan. 
Cumulative cost variance: $8.2 million; 
Cumulative schedule variance: -$7.8 million. 

Feb. 
Cumulative cost variance: $8.0 million; 
Cumulative schedule variance: -$8.0 million. 

Mar. 
Cumulative cost variance: $7.2 million; 
Cumulative schedule variance: -$9.0 million. 

Apr. 
Cumulative cost variance: $5.8 million; 
Cumulative schedule variance: -$8.8 million. 

May 
Cumulative cost variance: $3.0 million; 
Cumulative schedule variance: -$9.4 million. 

June 
Cumulative cost variance: $2.9 million; 
Cumulative schedule variance: -$9.8 million. 

Source: GAO analysis of NASA and contractor data. 

[End of figure] 

LADEE project officials provided no information regarding positive 
cost and negative schedule variance drivers. As such, we have no 
insight into what could be causing deviations from the plan. Based on 
the positive cost variance and negative schedule variance thus far, we 
are forecasting a negative variance at completion ranging from $0.1 
million to $13 million dollars. According to NASA, the project is not 
overrunning its commitment because the EVM baseline does not include 
unallocated future expenses held at the project and NASA headquarters 
level. 

Magnetospheric Multiscale: 

[Figure: computer model] 
* Project formulation start: May 2002; 
* Project confirmation date: June 2009; 
* Initial total project cost: $1,082.7 million; 
* Current total project cost: $1,082.6 million; 
* Launch readiness date: March 2015. 

Source: MMS Project Office. 

[End of figure] 

The Magnetospheric Multiscale (MMS) is made up of four identically 
instrumented spacecraft. The mission will use the Earth's 
magnetosphere as a laboratory to study the microphysics of magnetic 
reconnection, energetic particle acceleration, and turbulence. 
Magnetic reconnection is the primary process by which energy is 
transferred from solar wind to Earth's magnetosphere and is the 
physical process determining the size of a space weather storm. The 
four spacecraft will fly in a pyramid formation, adjustable over a 
range of 10 to 400 kilometers. The data from MMS will be used as a 
basis for predictive models of space weather in support of exploration. 

The MMS spacecraft is being designed, developed, and tested in-house 
at Goddard Spaceflight Center (GSFC) while instrument development 
activities are under contract with Southwest Research Institute 
(SwRI). The Mission Operations Center and the Flight Dynamics 
Operations Area will be developed and operated at GSFC. The Science 
Operations Center for the instruments will be developed and operated 
at the Laboratory for Atmospheric and Space Physics at the University 
of Colorado and is under contract to SwRI. The MMS project office is 
collecting EVM data both at the project level as well as from SwRI, 
which is responsible for the entire instrument suite. Therefore, the 
SwRI Instrument Suite effort is a subset of the overall MMS project 
level EVM report. 

Table 9: Assessment of MMS EVM Practices: 

Dollars in millions. 

Project Level; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $1,037; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO estimate at completion: -$47 to -$80. 

Instrument Suite; 
Used a certified EVM system compliant with ANSI/EIA standard: Agency 
did not meet this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $233[A]; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO estimate at completion: -$10 to -$24. 

Source: GAO analysis of NASA EVM data as of July 2011 and contractor 
EVM data as of August 2011. 

[A] The SwRI budget at completion is only for Phases B-D. 

[End of table] 

Project Office Does Not Have a Certified EVM System Compliant with the 
ANSI/EIA Standard: 

At the project level, MMS met one of three fundamental ANSI/EIA-748 
practices for a reliable EVM system. MMS does not have a certified EVM 
system that complies with the ANSI/EIA-748 standard. NASA project 
officials said in-house projects are required only to be ANSI/EIA 
compliant, and are not required to have a certified system. Although 
the MMS project does not have a certified system, we assessed how well 
the MMS project was meeting three ANSI/EIA guidelines. These 
guidelines state that the authorized work elements for the project 
should be defined typically using a WBS that has been tailored to the 
project and the WBS is the same for the cost estimate, schedule, and 
EVM. Our review found that the WBS in the MMS schedule did not match 
the WBS used for the EVM data, which is not in line with best 
practices. According to project officials, the MMS project was started 
before the requirements for earned value management were developed. As 
a result, the schedule and WBS were created without significant 
consideration of a one-to-one correlation between the two. A project 
official stated that MMS has retrofitted its EVM system to provide as 
close a correlation as possible without having to rebuild the WBS. 

The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. Our review of the MMS schedule found some sequencing 
issues within the schedule. For example, 9 percent of predecessor and 
successor tasks were not linked to one another, which is necessary for 
properly sequencing work so that the schedule will update properly 
once changes are made. This number accounts for removing all external 
tasks and level of effort (LOE) type activities. We also found that 21 
percent of the remaining activities were constrained. This number also 
does not include LOE type activities. In fact, the majority of these 
constraints were hard constraints. Hard constraints can sometimes be 
impossible to meet, given the network characteristics, and can thereby 
result in schedules that are logically impossible to carry out. The 
presence of constraints also impacts the schedule's ability to respond 
dynamically to changes and may portray an unrealistic view of the 
project plan. As a result, these sequencing issues and date 
constraints within the schedule affect the reliability of the overall 
network and the schedule's ability to correctly calculate float values 
and the critical path. Finally, the ANSI/EIA guidelines state that a 
project should establish and maintain a time-phased budget baseline to 
track cost and schedule variances in an EVM system. Though resource 
loading the schedule is not required to meet the ANSI/EIA guideline, 
it is a best practice and therefore resources should be accounted for 
in the schedule in order to develop this baseline, according to the 
GAO cost guide. We found that the MMS schedule was resource loaded. 

Project Conducted an Integrated Baseline Review: 

The MMS project office conducted an IBR in June 2010. This resulted in 
positive comments, general observations, and constructive 
recommendations by the review team. 

EVM Surveillance Is Not being Performed: 

While formal surveillance is not occurring at the project level, EVM 
data assurance reviews are being performed by the Explorers and 
Heliophysics Program Office, GSFC Flight Projects Directorate 
managers, GSFC Chief Financial Officer's Office, Standing Review 
Board, and NASA headquarters. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to July 
2011. Our review of the MMS project level EVM data found some minor 
data reliability issues. The 12 months of project level EVM data 
reflect all of the performed in-house effort at GSFC for the project 
and the instrument suite contractor's summary level effort. Also, the 
data provided was not reported in the standard contract performance 
report format. Project officials said a defined contract performance 
report was not dictated to the MMS project, but that all reporting has 
the same information as a "standard" contract performance report even 
though the formatting may be different. However, beginning in October 
2011, MMS began reporting with the standard format 1 contract 
performance report, which provides cost and schedule data for each 
element in the project's product-oriented WBS. 

We tried to map the EVM data in the lower level report for the 
instrument suite to the MMS project level report and we were not able 
to see how the costs tracked from one report to another. For example, 
the project level EVM data showed that the instrument suite 
contractor's July 2011 budget at completion was $296 million, whereas 
the lower level EVM data in the instrument suite contractor's report 
showed the budget at completion to be $217 million. The MMS project 
was able to demonstrate how the SwRI budget at completion in the lower-
level report mapped to the SwRI budget at completion in the MMS 
project report. However, officials said because of the way the 
contractor submits their data the two reports will never match. Though 
we acknowledge that the project was able to explain how the data 
tracked, attempting to manually resolve incompatible pieces of data 
can become time-consuming, expensive and can lead to data reliability 
issues. Project officials said that MMS is working to capture data at 
lower WBS levels, which will allow for a closer tie between the cost 
and schedule data. In addition, MMS said that in addition to receiving 
the SwRI EVM reports, the project now internally calculates earned 
value metrics on the contractor provided instrument suite EVM data, 
which gives MMS completely internally derived earned value performance 
reports, based on the project team's assessment without bias from 
contractor data. 

Figure 7 below illustrates that as of July 2011, the project was 
reporting a negative cumulative cost variance of $18 million and a 
negative cumulative schedule variance of $25 million. 

Figure 7: GAO Earned Value Analysis of MMS Project Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: -$7.1 million; 
Cumulative schedule variance: -$24.6 million. 

September; 
Cumulative cost variance: -$1.0 million; 
Cumulative schedule variance: -$17.8 million. 

October; 
Cumulative cost variance: -$2.4 million; 
Cumulative schedule variance: $1.3 million. 

November; 
Cumulative cost variance: -$5.5 million; 
Cumulative schedule variance: -$2.2 million. 

December; 
Cumulative cost variance: -$8.5 million; 
Cumulative schedule variance: -$8.2 million. 

2011: 

January; 
Cumulative cost variance: -$8.1 million; 
Cumulative schedule variance: -$12.8 million. 

February; 
Cumulative cost variance: -$4.5 million; 
Cumulative schedule variance: -$12.6 million. 

March; 
Cumulative cost variance: -$5.0 million; 
Cumulative schedule variance: -$15.3 million. 

April; 
Cumulative cost variance: -$5.8 million; 
Cumulative schedule variance: -$15.1 million. 

May; 
Cumulative cost variance: -$11.1 million; 
Cumulative schedule variance: -$19.1 million. 

June; 
Cumulative cost variance: -$14.6 million; 
Cumulative schedule variance: -$24.8 million. 

July; 
Cumulative cost variance: -$18.2 million; 
Cumulative schedule variance: -$24.9 million. 

Source: GAO analysis of NASA and contractor data. 

[End of figure] 

In October 2010, the project was in the midst of a replan and not all 
data was available at the time of report submission to generate 
detailed variance explanations. The replan was conducted so that the 
earned value baseline was the same as the cost plan required by the 
agency for monthly plan versus actual reporting. The goal of the 
replan was to prevent the project from having to report variances 
against two different plans. Since the replan, however, the project 
has experienced a downward trend in both cost and schedule 
performance. As a result, we are forecasting a negative variance at 
completion ranging from $47 million to $80 million dollars. NASA 
stated that the project is not overrunning its approved baseline 
commitment, because the EVM baseline does not include unallocated 
future expenses held at the project and headquarters level. 

Instrument Suite Contractor Does not Have a Certified EVM System 
Compliant with ANSI/EIA Standard: 

SwRI, the contractor responsible for the entire instrument suite for 
MMS, met one of the three fundamental ANSI/EIA-748 practices for a 
reliable EVM system. SwRI does not have a certified EVM system that 
complies with the ANSI/EIA-748 standard. According to a project 
official, the SwRI contract does not require SwRI to have a certified 
system but only to be compliant with the ANSI/EIA-748. NASA convened 
an independent team to review the contractor's readiness for EVM 
system certification and concluded that while the contractor has 
qualified people to support implementation of EVM, a single point of 
failure exists without a documented process. Not documenting the 
process is a problem because if the people who know the process leave, 
new staff will not know what to do. In addition, the team found that 
even though the right software tools are in place to support EVM, more 
integration is needed to reduce manual inputs. Finally, the team 
reported that compliance with the ANSI/EIA-748 standard would not be 
achievable without management support and resources. Despite these 
findings, the project office believed that contractor's EVM data is 
useful in examining trends and overall performance of the instrument 
suite effort. 

Though SwRI does not have a certified system, we assessed how well the 
contractor was meeting three ANSI/EIA guidelines. These guidelines 
state that the authorized work elements for the project should be 
defined typically using a WBS that has been tailored to the project 
and the WBS is the same for the cost estimate, schedule, and EVM. Our 
review found that the WBS used in the schedule was not consistent with 
the WBS used for the EVM data. The ANSI/EIA guidelines also state that 
projects should have a schedule that describes the sequence of work by 
listing activities in the order in which they are to be carried out 
and identifying significant task interdependencies required to meet 
project requirements. Our review of the schedule found some sequencing 
issues. For example, the Primavera schedule provided showed 31 percent 
of the remaining activities were missing dependencies and 36 percent 
were constrained. Officials said majority of the constrained tasks 
were due to external dependencies. Due to the major sequencing issues 
in the MMS SwRI schedule, we question the reliability of the overall 
network and the schedule's ability to correctly calculate float values 
and the critical path. MMS project officials said they believed many 
of the constrained activities are not valid because they reside in 
another schedule. Also, some constraints, in the Harness area for 
example, if removed, have no effect on the overall schedule. In 
addition, officials also said some of the sequencing issues may be 
caused by the manual integration because since some instrument 
provider schedules are in Microsoft Project and others are in 
Primavera, and therefore it is not possible to ensure all tasks have 
been linked properly. However, the MMS project scheduler tests the 
schedule for missing dependencies, logic errors, and reasonable 
durations and the results are shared with the project office and the 
contractor so that appropriate action can be taken. Lastly, officials 
said several of the activities identified in our analysis are not 
really schedule items but level of effort type activities. When we 
removed the 15 LOE type activities from the missing dependencies 
count, the schedule still showed 28 percent of the remaining 
activities were missing dependencies. When we removed the 14 LOE type 
activities and the 4 Harness activities from the constraint count, the 
schedule still showed 33 percent of remaining activities were 
constrained. Because the schedule is the foundation for the EVM 
baseline, we question the reliability of the Instrument Suite EVM 
data. Finally, the ANSI/EIA guidelines state that a project should 
establish and maintain a time-phased budget baseline to track cost and 
schedule variances in an EVM system. Though resource loading the 
schedule is not required to meet the ANSI/EIA guidelines, it is a best 
practice and therefore resources should be accounted for in the 
schedule in order to develop this baseline, according to the GAO cost 
guide. We found that the MMS schedule was resource-loaded. 

Instrument Suite Contractor Conducted an Integrated Baseline Review: 

The MMS project conducted an IBR of the MMS instrument suite effort in 
January 2010 and all action items have been closed out. 

EVM Surveillance Is Not Being Performed: 

While the instrument suite contractor does not have a formal 
surveillance program, the MMS project office has an EVM analyst, 
schedule team, resources team, instrument management team, and project 
management team who all review the instrument suite EVM data on a 
monthly basis. In addition, the Solar Terrestrial Probes project 
office, Science Mission Directorate management, and The Aerospace 
Corporation review the instrument suite EVM data monthly. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to August 
2011. Our review of the instrument suite EVM data found various data 
anomalies that call into question the reliability of the data. For 
example, there were negative numbers reported for EVM data in four of 
the months that we reviewed. For some of the negative numbers, there 
was no explanation for the cause. For others, the negative values were 
due to correcting several months of translation errors brought on by a 
known issue with importing data from the schedule into the EVM system 
software. There were also errors such as inflated EVM data that once 
corrected, resulted in negative values. Figure 8 below illustrates 
that as August 2011, the project was reporting a negative cumulative 
cost variance of $4 million and negative cumulative schedule variance 
of $6 million. 

Figure 8: GAO Earned Value Analysis of Instrument Suite Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: -$0.5 million; 
Cumulative schedule variance: -$3.4 million. 

September; 
Cumulative cost variance: -$6.2 million; 
Cumulative schedule variance: -$5.2 million. 

October; 
Cumulative cost variance: -$6.2 million; 
Cumulative schedule variance: -$4.5 million. 

November; 
Cumulative cost variance: -$8.9 million; 
Cumulative schedule variance: -$5.3 million. 

December; 
Cumulative cost variance: -$10.0 million; 
Cumulative schedule variance: -$5.6 million. 

2011: 

January; 
Cumulative cost variance: -$11.1 million; 
Cumulative schedule variance: -$6.9 million. 

February; 
Cumulative cost variance: -$9.6 million; 
Cumulative schedule variance: -$6.7 million. 

March; 
Cumulative cost variance: -$10.0 million; 
Cumulative schedule variance: -$5.9 million. 

April; 
Cumulative cost variance: -$11.6 million; 
Cumulative schedule variance: -$6.4 million. 

May; 
Cumulative cost variance: -$13.5 million; 
Cumulative schedule variance: -$6.4 million. 

June; 
Cumulative cost variance: -$14.4 million; 
Cumulative schedule variance: -$7.0 million. 

July; 
Cumulative cost variance: -$16.8 million; 
Cumulative schedule variance: -$7.2 million. 

August; 
Cumulative cost variance: -$4.3 million; 
Cumulative schedule variance: -$6.3 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The cost variance in August 2011 dramatically improved from the 
downward trend during the previous months due to the project applying 
almost $13 million from its management reserve to the instrument suite 
contract. However, due to the negative cumulative cost and schedule 
variances, we are forecasting a negative variance at completion 
ranging from $10 million to $24 million. 

Mars Atmosphere and Volatile Environment: 

[Figure: artist depiction] 
* Project formulation start: September 2008; 
* Project confirmation date: October 2010; 
* Initial total project cost: $671.2 million; 
* Current total project cost: $671.2 million; 
* Launch readiness date: November 2013. 

Source: NASA GSFC MAVEN Project Office. 

[End of figure] 

The Mars Atmosphere and Volatile EvolutioN (MAVEN) mission is part of 
NASA's Mars Scout program, a robotic orbiter mission that will provide 
a comprehensive picture of the Mars upper atmosphere, ionosphere, 
solar energetic drivers, and atmospheric losses. Set to launch in 
2013, MAVEN will deliver comprehensive answers to long-standing 
questions regarding the loss of Mars' atmosphere, climate history, 
liquid water, and habitability. MAVEN will provide the first direct 
measurements ever taken to address key scientific questions about 
Mars' evolution. 

Lockheed Martin is building the MAVEN spacecraft and will carry out 
mission operations for MAVEN. NASA's Jet Propulsion Laboratory will 
navigate the spacecraft. The Laboratory for Atmospheric and Space 
Physics (LASP) at the University of Colorado at Boulder will 
coordinate the science team and science operations and lead the 
education and public outreach activities. NASA's Goddard Spaceflight 
Center will provide management and technical oversight for the mission 
and will also provide mission systems engineering, mission design, and 
safety and mission assurance. 

The MAVEN project office is using EVM at the project level as well as 
collecting EVM data from Lockheed Martin, the spacecraft contractor, 
and LASP, which is responsible for the Science Operations Center, 
Remote Sensing, and Langmuir Probe and Waves instrument efforts. Both 
the Lockheed Martin and LASP contracts are subsets of the overall 
MAVEN Project EVM report. 

Table 10: Assessment of MAVEN EVM Practices: 

Dollars in millions. 

Project level; 
Used a certified EVM system compliant with ANSI standard: Agency did 
not meet this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $259; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: NA[A]. 

Spacecraft; 
Used a certified EVM system compliant with ANSI standard: Fully met 
this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $169; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: $1 to $14. 

Science Operations Center[B]; 
Used a certified EVM system compliant with ANSI standard: [Empty]; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $3; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: $0 to $0.4. 

Remote sensing Package[B]; 
Used a certified EVM system compliant with ANSI standard: [Empty]; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $20; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: $1 to $7. 

Langmuir Probe and Waves Instrument[B]; 
Used a certified EVM system compliant with ANSI standard: [Empty]; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $5; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: -$2 to -$3. 

Source: GAO Analysis of NASA EVM data as of July 2011 and contractor 
EVM data as of August 2011. 

[A] NA - Not applicable means there was not enough data to complete a 
forecast. 

[B] A certified EVM system is not required due to the contract value. 

[End of table] 

Project Does Not Have a Certified EVM System Compliant with ANSI/EIA 
Standard: 

MAVEN fully met one of the three key practices for implementing EVM at 
the project level. Specifically, the project did not have a certified 
EVM system and is not required to have a certified system. 
Nevertheless, we assessed how well the MAVEN Project was meeting three 
ANSI/EIA guidelines. As part of our analysis, we assessed MAVEN's EVM 
data against three ANSI/EIA guidelines. These guidelines state that 
the authorized work elements for the project should be defined 
typically using a WBS that has been tailored to the project and that 
the WBS should be the same for the cost estimate, schedule, and EVM. 
We found that the project's WBS was consistent between the schedule 
and EVM data. 

The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. Our review found some sequencing issues in the schedule. 
For example, 5 percent of the activities were missing dependencies, 6 
percent had open ended logic links, and 20 percent had constraints, 
among other things. When schedules are not sequenced properly, float 
values and the calculated critical path will not be valid. In 
addition, the project conducted an integrated baseline review in July 
2011. The MAVEN project office provided a May 2012 Schedule Health 
Check Report that showed the number of missing dependencies and 
constraints had been reduced considerably. Though we cannot validate 
the improvement in the schedule without performing our own assessment, 
we believe that MAVEN is working towards producing a more reliable 
schedule. This review found that the project schedule and technical 
design were in good shape, but noted concerns that more resources were 
needed to implement and maintain EVM, there were cost and schedule 
integration issues that caused the budgets for some work packages to 
not be in sync with the schedule, and reliable critical path analysis 
was at risk because of missing schedule links and constraints. 
Finally, a project should establish and maintain a time-phased budget 
baseline at the control account level, against which performance can 
be measured. Though resource loading the schedule is not required to 
meet the ANSI/EIA guidelines, it is a best practice and therefore 
resources should be accounted for in the schedule in order to develop 
this baseline, according to the GAO cost guide. We found that the 
schedule was resource-loaded. 

Project Conducted an Integrated Baseline Review: 

Project conducted an IBR in July 2011 and all nine areas of concern 
were addressed and closed. 

EVM Surveillance Is Not Being Performed: 

While formal surveillance is not occurring at the project level, EVM 
data assurance reviews are being performed by the Mars Program Office 
representatives, MAVEN standing Review Board representatives, and 
Aerospace Corporation representatives at both the project level and 
for LASP efforts. 

Data Resulting from the EVM System Is Reliable: 

We reviewed contract performance reports from June 2011 to August 
2011. EVM data prior to spring 2011 was not available because MAVEN 
had not been confirmed into the implementation phase. Our review of 
the MAVEN project EVM data found that there was a mistake causing the 
costs to be overstated by twice their actual amount. Other than this 
mistake, which affected most of the data in the report, no other 
errors were found. The project EVM data reflects all work on this 
project including effort related to the project office, two in-house 
instruments, and the Space Sciences Laboratory, as well as major 
efforts from Lockheed Martin and LASP. Consolidated reporting of all 
components at a summary level began with the May 2011 data, with the 
first full summary report delivered on July 15, 2011. Therefore, we 
had only 3 months of data to review. Figure 9 below illustrates that 
as of August 2011, the project was reporting a positive cumulative 
cost variance of $5 million while also experiencing a negative 
cumulative schedule variance of $5 million. 

Figure 9: GAO Earned Value Analysis of MAVEN Project Data: 

[Refer to PDF for image: line graph] 

2011: 

June; 
Cumulative cost variance: $3.9 million; 
Cumulative schedule variance: -$4.3 million. 

July; 
Cumulative cost variance: $4.6 million; 
Cumulative schedule variance: -$4.3 million. 

August; 
Cumulative cost variance: $5.4 million; 
Cumulative schedule variance: -$4.8 million. 

Source: GAO analysis of NASA data. 

[End of figure] 

The positive cumulative cost variance was due to a decrease in labor 
charges, delayed material costs, a reduction in re-work, and the 
leveraging of common engineering products from other projects. The 
negative schedule variance was being driven by the Neutral Gas and Ion 
Mass Spectrometer instrument, which experienced technical issues such 
as vendor machines not being manufactured to specifications. Because 
we had only 3 months of data, we did not forecast an estimate at 
completion. 

In addition, we tried to map the EVM data in the lower level reports 
for spacecraft, Science Operations Center, Remote Sensing, and 
Langmuir Probe and Waves efforts to the overall MAVEN project EVM 
report and in some cases we were not able to see how the costs tracked 
from one report to another. For example, while we could easily trace 
the costs for the Science Operations Center effort from the lower 
level EVM report to the overall MAVEN project report, we could not 
clearly map the costs for the spacecraft, Remote Sensing or Langmuir 
Probe and Waves efforts. In particular, for Remote Sensing and 
Langmuir Probe and Waves efforts, the lower level EVM report cost 
elements did not have their costs burdened at the WBS level, which 
could account for some of the differences between the lower level 
report costs and the overall MAVEN project costs for those elements. 
MAVEN project officials walked us through their process of how they 
ensure lower-level reports map to the project level reports. In 
addition, MAVEN project provided supporting documentation that 
validated this assertion. Though MAVEN project officials helped 
explain the mapping, officials said they do not mandate that their 
contractors follow a certain reporting format, instead any adjustments 
necessary to ensure that the lower-level reports map to the project-
level reports are made manually by the project office. Though MAVEN 
project does not prescribe to a standard reporting format, attempting 
to manually resolve incompatible pieces of data can become time-
consuming, expensive and can lead to data reliability issues. Although 
the agency provided explanations for the mapping issues, the ability 
to reconcile the costs between the reports should be evident, without 
additional explanations. 

Lockheed Martin Has a Certified EVM System Compliant with the ANSI/EIA 
Standard: 

Lockheed Martin, the spacecraft contractor, met the three fundamental 
ANSI/EIA practices necessary for a reliable EVM system. In August 
2008, the Defense Contract Management Agency (DCMA) certified that 
Lockheed Martin's EVM system is compliant with the ANSI/EIA standard. 
However, the implementation of that EVM system is questionable based 
on our findings. We assessed the contractor's data against the three 
ANSI/EIA guidelines. These guidelines state that the authorized work 
elements for the project should be defined typically using a WBS that 
has been tailored to the project and the WBS is the same for the cost 
estimate, schedule, and EVM. We found that the project's WBS in the 
schedule was consistent with the WBS used in the EVM data. However, we 
found issues with the schedule. The ANSI/EIA guidelines also state 
that projects should have a schedule that describes the sequence of 
work by listing activities in the order in which they are to be 
carried out and identifying significant task interdependencies 
required to meet project requirements. Our review found some 
sequencing issues in the schedule. For example, 2 percent of the 
activities remaining were missing predecessor and successor links, 
which are necessary for properly sequencing work so that the schedule 
will update in response to changes. We also found that 37 percent of 
the remaining activities had constraints, which also hinder the 
schedule's ability to respond dynamically to status updates resulting 
in an artificial or unrealistic view of the project plan. These issues 
with the schedule affected the reliability of the overall network and 
the schedule's ability to correctly calculate float values and the 
critical path. Project officials further explained that some of the 
"start no earlier than" constraints were due to resource availability 
and "finish no later than" constraints were used intentionally to plan 
task activities to occur as late as possible. Finally, the ANSI/EIA 
guidelines state that a project should establish and maintain a time-
phased budget baseline to track cost and schedule variances in an EVM 
system. Though resource loading the schedule is not required to meet 
the ANSI/EIA guidelines, it is a best practice and therefore resources 
should be accounted for in the schedule in order to develop this 
baseline, according to the GAO cost guide. We found that the 
contractor schedule was resource loaded. 

Project Conducted an Integrated Baseline Review: 

The project conducted the spacecraft's IBR in April 2011. During the 
review, several areas of concern regarding the schedule were 
identified. The project office said that during the integrated 
baseline review, the review team identified many of the same 
observations with the schedule as our findings. As a result, the 
project office directed the contractor to eliminate the constraints, 
lags, and missing logic links in their integrated master schedule. 
Since the November 2011 schedule submittal, the contractor has 
decreased the number of sequencing issues in the schedule, according 
to project officials. The project office also said that the contractor 
continues to conduct schedule health checks to uncover any sequencing 
issues and provide the project office with schedule variance reports 
and critical path analysis, which are discussed during monthly 
management meetings. 

EVM Surveillance Is Being Performed: 

Joint surveillance reviews of the EVM data are being performed by 
DCMA, MAVEN project officials, and the contractor. 

Data Resulting from the EVM System Is Reliable: 

We reviewed contract management reports from January 2011 to August 
2011. Our review of the EVM data found no major issues with data 
reliability. However, from January to May 2011, there were no variance 
analysis reports produced to explain significant cost and schedule 
variances and other contract problems and topics because they did not 
meet reporting thresholds. Without this information, however, 
management cannot understand the reasons for variances and the 
contractor's plan for fixing them. 

Figure 10 below illustrates that as of August 2011, the project was 
reporting a positive cumulative cost variance of $6 million and a 
negative cumulative schedule variance of $3 million. 

Figure 10: GAO Earned Value Analysis of Spacecraft Data: 

[Refer to PDF for image: line graph] 
 
2011: 

January; 
Cumulative cost variance: -$0.2 million; 
Cumulative schedule variance: -$0.5 million. 

February; 
Cumulative cost variance: $0.8 million; 
Cumulative schedule variance: -$0.5 million. 

March; 
Cumulative cost variance: $2.1 million; 
Cumulative schedule variance: -$0.7 million. 

April; 
Cumulative cost variance: $2.7 million;
Cumulative schedule variance: -$1.6 million. 

May; 
Cumulative cost variance: $4.5 million;
Cumulative schedule variance: -$2.4 million. 

June; 
Cumulative cost variance: $5.7 million; 
Cumulative schedule variance: -$2.1 million. 

July; 
Cumulative cost variance: $5.7 million; 
Cumulative schedule variance: -$2.1 million. 

August; 
Cumulative cost variance: $6.1 million; 
Cumulative schedule variance: -$2.8 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

One reason for the positive cost variance was due to the ability to 
leverage a lower subcontractor rate than planned, which resulted in a 
cost savings. The negative schedule variance was also being driven by 
the mechanism subsystem falling behind schedule due to the shop being 
overloaded with work and the mechanism designers supporting other NASA 
efforts, among other things. Due to the positive cost variance, we are 
forecasting a positive variance at completion ranging from $1 million 
to $14 million. 

Laboratory for Atmospheric and Space Physics Does Not Have a Certified 
EVM System Compliant with ANSI/EIA Standard: 

LASP, the contractor responsible for the Science Operations Center, 
Remote Sensing package, and Langmuir Probe and Waves instrument 
efforts met one of three fundamental ANSI/EIA-748 practices necessary 
for a reliable EVM system. To date, LASP does not have a certified EVM 
system. Though the contractor is not required to have a certified 
system, we assessed how well the contractor was meeting three ANSI/EIA 
guidelines. These guidelines state that the authorized work elements 
for the project should be defined typically using a WBS that has been 
tailored for effective internal management control of the project and 
the WBS is the same for the cost estimate, schedule, and EVM. We found 
some slight inconsistencies in the WBS numbering between the EVM 
report and the schedule for the Remote Sensing package, which calls 
into question the reliability of their overall schedule network. 
Moreover, the Langmuir Probe and Waves Instrument had issues with 
consistency between the WBS and the schedule. In particular, there was 
varying levels of information between the two WBSs, making it hard to 
use the WBS as a common thread between the EVM data and the schedule. 
Since the WBS is a critical component of EVM, it should be the same 
for developing the EVM performance measurement baseline and the 
schedule. Without a common link between these two features, project 
managers cannot fully understand project cost and schedule variances. 

The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. We found some sequencing issues with the Langmuir Probe 
and Waves Instrument schedule. For example, 47 percent of the 
remaining activities had constraints, which defeated the purpose of a 
using a dynamic schedule. The quality of the schedule was also 
hampered by the presence of schedule lags on 18 percent of the 
remaining activities. Schedule lags must be justified because they 
cannot be easily monitored or included in risk assessments. Finally, 
the ANSI/EIA guidelines state that a project should establish and 
maintain a time-phased budget baseline to track cost and schedule 
variances in an EVM system. Though resource loading the schedule is 
not required to meet the ANSI/EIA guidelines, it is a best practice 
and therefore resources should be accounted for in the schedule in 
order to develop this baseline, according to the GAO cost guide. We 
found that all three components showed evidence of a time-phased 
budget baseline. 

Project Conducted an Integrated Baseline Review: 

The project conducted an IBR in March 2011 and several areas of 
concern were noted. MAVEN project officials stated that many of the 
schedule issues we found were also discovered during the IBR and have 
now been corrected. 

EVM Surveillance Is Not Being Performed: 

While formal surveillance is not occurring, EVM data assurance reviews 
are being performed by the, project office, Mars Program Office, and 
MAVEN Standing Review Board representatives. Also, The Aerospace 
Corporation is working as an advisor to Science Mission Directorate's 
Planetary Systems Division. 

Data Resulting from the EVM System Is Reliable: 

Our review of MAVEN's Science Operations Center, Remote Sensing 
package, and Langmuir Probe and Waves instrument EVM data found no 
major issues with data reliability. However, there was a lack of 
variance analysis reports for these efforts. For example, Science 
Operations Center had no variance analysis reports with the 
explanations for any of the months reported. In addition, the Remote 
Sensing package, and Langmuir Probe and Waves instrument variance 
analysis reports did not provide any explanation for major performance 
swings from one month to another. 

Figure 11 below illustrates that as of August 2011, the Science 
Operations Center portion of the MAVEN project was reporting a 
positive cumulative cost variance of $0.01 million and a slightly 
negative cumulative schedule variance of $0.01 million. 

Figure 11: GAO Earned Value Analysis of Science Operations Center Data: 

[Refer to PDF for image: line graph] 

2010:  

November; 
Cumulative cost variance: -$0.02 million; 
Cumulative schedule variance: -$0.017 million. 

December; 
Cumulative cost variance: -$0.005 million; 
Cumulative schedule variance: $0.002 million. 

2011: 

January; 
Cumulative cost variance: -$0.002 million; 
Cumulative schedule variance: $0.004 million. 

February; 
Cumulative cost variance: -$0.006 million; 
Cumulative schedule variance: $0.002 million. 

March; 
Cumulative cost variance: -$0.02 million; 
Cumulative schedule variance: $0. 

April; 
Cumulative cost variance: -$0.02 million; 
Cumulative schedule variance: -$0.002 million. 

May; 
Cumulative cost variance: -$0.004 million; 
Cumulative schedule variance: -$0.008 million. 

June; 
Cumulative cost variance: -$0.002 million; 
Cumulative schedule variance: -$0.014 million. 

July; 
Cumulative cost variance: -$0.004 million; 
Cumulative schedule variance: -$0.006 million. 

August; 
Cumulative cost variance: -$0.005 million; 
Cumulative schedule variance: -$0.006 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

However, since the variance analysis reports provided no information 
regarding what is driving the positive cost and slightly negative 
schedule variances, we have no insight into the causes for deviations 
from the plan. Based on the positive cost variance thus far, we are 
forecasting a positive variance at completion of less than $0.5 
million at contract completion. 

Figure 12 below illustrates that as of August 2011 the Remote Sensing 
package portion of the MAVEN project was reporting a positive 
cumulative cost variance of $0.8 million and a slightly negative 
cumulative schedule variance of $0.6 million. 

Figure 12: GAO Earned Value Analysis of Remote Sensing Instrument Data: 

[Refer to PDF for image: line graph] 

2010: 

November; 
Cumulative cost variance: $0.171129; 
Cumulative schedule variance: $0.135625 

December; 
Cumulative cost variance: $0.132754; 
Cumulative schedule variance: $0.175011 

2011: 

January; 
Cumulative cost variance: $0.48 million; 
Cumulative schedule variance: $0.32 million. 

February; 
Cumulative cost variance: $0.44 million; 
Cumulative schedule variance: $0.41 million. 

March; 
Cumulative cost variance: $0.59 million; 
Cumulative schedule variance: $0.25 million. 

April; 
Cumulative cost variance: $1.04 million; 
Cumulative schedule variance: $0.47 million. 

May; 
Cumulative cost variance: $0.64 million; 
Cumulative schedule variance: $0.20 million. 

June; 
Cumulative cost variance: -$0.044 million; 
Cumulative schedule variance: -$0.78 million. 

July; 
Cumulative cost variance: $0.227 million; 
Cumulative schedule variance: -$0.77 million. 

August; 
Cumulative cost variance: $0.762 million; 
Cumulative schedule variance: -$0.55 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

Factors behind the positive cost variance include an accounting lag on 
the invoicing and payment process associated with the procurements, 
which results in the appearance of cost efficiency. This issue has 
been occurring for several months and is causing the EVM metrics to be 
skewed to reflect false positive cost variances. The variance analysis 
reports do not give any explanation for why there is a negative 
schedule variance situation as of August 2011. As a result of the 
positive cost variance, we are forecasting a positive variance at 
completion ranging from $1 million to $7 million. 

Figure 13 below illustrates that as of August 2011 the Langmuir Probe 
and Waves portion of the MAVEN project was reporting a negative 
cumulative cost variance of $0.6 million and negative cumulative 
schedule variance of $0.3 million. 

Figure 13: GAO Earned Value Analysis of Langmuir Probe and Waves Data: 

[Refer to PDF for image: line graph] 

2010: 

November; 
Cumulative cost variance: $0.004 million; 
Cumulative schedule variance: -$0.17 million. 

December; 
Cumulative cost variance: -$0.009 million; 
Cumulative schedule variance: -$0.25 million. 

2011: 

January; 
Cumulative cost variance: -$0.0006 million; 
Cumulative schedule variance: -$0.24 million. 

February; 
Cumulative cost variance: -$0.041 million; 
Cumulative schedule variance: $0.04 million. 

March; 
Cumulative cost variance: -$0.142 million; 
Cumulative schedule variance: $0.04 million. 

April; 
Cumulative cost variance: -$0.23 million; 
Cumulative schedule variance: $0.03 million. 

May; 
Cumulative cost variance: -$0.27 million; 
Cumulative schedule variance: $0.02 million. 

June; 
Cumulative cost variance: -$0.48 million; 
Cumulative schedule variance: -$0.03 million. 

July; 
Cumulative cost variance: -$0.53 million; 
Cumulative schedule variance: -$0.25 million. 

August; 
Cumulative cost variance: -$0.59 million; 
Cumulative schedule variance: -$0.29 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

Reasons for the negative cost and schedule variances are due to costs 
for outside services and materials being more than planned as well as 
additional work required to troubleshoot problems and mitigate risks. 
Due to these problems, we are forecasting a negative variance at 
completion ranging from $2 million and $3 million. According to NASA, 
the project is not overrunning its commitment because the EVM baseline 
does not include unallocated future expenses held at the project and 
headquarters level. 

Orbiting Carbon Observatory 2: 

[Figure: artist depiction] 
* Project formulation start: March 2010; 
* Project confirmation date: September 2010; 
* Initial total project cost: $349.9 million; 
* Current total project cost: $477.2 million; 
* Launch readiness date: July 2014. 

Source: NASA. 

[End of figure] 

NASA's Orbiting Carbon Observatory 2 (OCO-2) is designed to enable 
more reliable predictions of climate change and is based on the 
original OCO mission that failed to reach orbit in 2009. It will make 
precise, time-dependent global measurements of atmospheric carbon 
dioxide. These measurements will be combined with data from a ground-
based network to provide scientists with the information needed to 
better understand the processes that regulate atmospheric carbon 
dioxide and its role in the carbon cycle. NASA expects enhanced 
understanding of the carbon cycle will improve predictions of future 
atmospheric carbon dioxide increases and the potential impact on the 
climate. 

The OCO-2 mission consists of a dedicated spacecraft with a single 
instrument, flying in a near-polar, sun-synchronous orbit. The Jet 
Propulsion Laboratory (JPL) has overall responsibility for project 
management. The OCO-2 spacecraft is being built by Orbital Sciences 
Corporation while the instrument is being built in-house at JPL. 
Orbital Sciences Corporation submits spacecraft effort EVM data 
monthly and the project incorporates that data into the overall 
project EVM report. The project is facing a launch delay because the 
Taurus XL launch vehicle failed on the Glory Mission, and the contract 
was terminated. The project will be rebaselined as a result of NASA 
having to select a new launch vehicle. The current $477.2 million 
total project cost is a preliminary amount pending the outcome of the 
rebaseline process. 

Table 11: Assessment of OCO-2 EVM Practices[A]: 

Dollars in millions. 

Project; 
Used a certified EVM system compliant with ANSI standard: Fully met 
this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $129; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: -$1 to -$26. 

Spacecraft[B]; 
Used a certified EVM system compliant with ANSI standard: Fully met 
this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $30; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: NA[C]. 

Source: GAO analysis of JPL and contractor EVM data as of July 2011. 

[A] All the budget and variance data reported is prior to the 
rebaselineing resulting from changing the launch vehicle. 

[B] DCMA certified Orbital Sciences Corporation's EVM system in 
January 2012. 

[C] NA means that there was not enough EVM data available for us to 
forecast an estimate at completion. 

[End of table] 

Project Using a Certified EVM System Compliant with the ANSI/EIA 
Standard: 

The OCO-2 project met all three fundamental ANSI/EIA-748 practices 
necessary for a reliable EVM system. JPL has a certified EVM system 
that complies with the ANSI/EIA standard. However, the implementation 
of its EVM system is questionable based on our findings below. We 
assessed the JPL EVM data against three ANSI/EIA guidelines. These 
guidelines state that the authorized work elements for the project 
should be defined typically using a WBS that has been tailored to the 
project and the WBS is the same for the cost estimate, schedule, and 
EVM. We found that the project had a work breakdown structure that was 
consistent with the WBS used for the EVM data. The ANSI/EIA guidelines 
also state that projects should have a schedule that describes the 
sequence of work by listing activities in the order in which they are 
to be carried out and identifying significant task interdependencies 
required to meet project requirements. Our review found some 
sequencing issues in the schedule. In particular, 14 percent of 
predecessor and successor tasks were not linked to one another, which 
is necessary for properly sequencing work so that it will update in 
response to changes. We also found 15 percent of the remaining 
activities were constrained, which also hinders the schedule from 
responding dynamically to changes and can portray an artificial or 
unrealistic view of the project plan. These sequencing issues and 
constraint dates within the schedule affect the reliability of the 
overall network and the schedule's ability to correctly calculate 
float values and the critical path. Project officials said the missing 
dependencies are mainly handoffs produced by level of effort (LOE) 
activities. Officials said these activities are not necessary for 
valid schedule network logic, and under no circumstances do these 
activities drive the critical path. They also said the constrained 
activities are largely composed of mandated delivery dates, which JPL 
uses as control points to manage subsystem schedule performance prior 
to assembly, test, and launch operations delivery, and coordinate 
major meeting logistics. Finally, the ANSI/EIA guideline states that a 
project should establish and maintain a time-phased budget baseline to 
track cost and schedule variances in an EVM system. Though resource 
loading the schedule is not required to meet the ANSI guidelines, it 
is a best practice and therefore resources should be accounted for in 
the schedule in order to develop this baseline, according to the GAO 
cost guide. To develop the baseline, resources must be accounted for 
in the schedule and our review found that the schedule was resource 
loaded. 

Project Conducted an Integrated Baseline Review: 

The project conducted an integrated baseline review in March 2011. 

EVM Surveillance Is Being Performed: 

JPL also has a formal surveillance plan in place for monitoring the 
EVM data. In particular, project officials said each month detailed 
earned value data analysis is performed on each cost account and each 
work package so that conclusive understanding of the performance 
status can be reached and communicated within the project team. In 
addition, during the project monthly management reviews, both cost and 
schedule variances and reasons causing them are presented. 

Data Resulting from the EVM System Are Reliable: 

We reviewed contractor management reports from December 2010 to July 
2011. Though we found no major data reliability issues when we tried 
to map the EVM data in the lower level spacecraft report to the JPL 
project level report, we could not understand how the costs tracked 
from one report to another. In a subsequent interview, officials 
explained, with supporting documentation, how the lower level Orbital 
Sciences budget at completion mapped to the budget at completion found 
in the JPL Project level EVM report. Though we appreciate the 
explanations provided by project officials regarding the differences 
between the two reports, the issue remains that without additional 
documentation and explanations by project officials, GAO or another 
independent party could not have reconciled the data. 

Figure 14 below illustrates that as of July 2011, the project was 
reporting a positive cumulative cost variance of $0.6 million and a 
negative schedule variance of $8 million as seen in the graph below. 
According to project officials, the schedule variance was being caused 
by spectrometer slit instability and an incompatible memory chip in 
the Remote Electronics Module. 

Figure 14: GAO Earned Value Analysis of OCO-2 Project Data: 

[Refer to PDF for image: line graph] 

2010: 

December; 
Cumulative cost variance: $2.204 million; 
Cumulative schedule variance: -$0.72 million. 

2011: 

January; 
Cumulative cost variance: $0.181 million; 
Cumulative schedule variance: -$4.273 million. 

February; 
Cumulative cost variance: $4.019 million; 
Cumulative schedule variance: -$1.814 million. 

March; 
Cumulative cost variance: $3.443 million; 
Cumulative schedule variance: -$2.649 million. 

April; 
Cumulative cost variance: $3.26 million; 
Cumulative schedule variance: -$3.942 million. 

May; 
Cumulative cost variance: $1.279 million; 
Cumulative schedule variance: -$5.433 million. 

June; 
Cumulative cost variance: $1.36 million; 
Cumulative schedule variance: -$6.637 million. 

July; 
Cumulative cost variance: $0.605 million; 
Cumulative schedule variance: -$8.087 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

Due to the positive cost variance and negative schedule variance we 
are forecasting a negative variance at completion ranging from $1 
million to $26 million. Project officials said the project is not 
overrunning its commitment because the EVM baseline provided to GAO 
does not include unallocated future expenses held at the project and 
headquarters level. 

Spacecraft Contractor Has a Certified EVM System Compliant with ANSI/ 
EIA Standard: 

The spacecraft contractor, Orbital Sciences Corporation, met two of 
the three fundamental ANSI/EIA-748 practices necessary for a reliable 
EVM System. At the time of our review, the Orbital Sciences did not 
have a certified EVM system that complied with the ANSI/EIA-748 
standard. In January 2012, DCMA certified the Orbital Sciences EVM 
System. Though the contractor now has a certified system, the 
implementation of that system is questionable based on our findings 
below. We assessed how well the contractor's EVM system was meeting 
three ANSI/EIA guidelines. These guidelines state that the authorized 
work elements for the project should be defined typically using a WBS 
that has been tailored to the project and the WBS is the same for the 
cost estimate, schedule, and EVM. We found that the spacecraft's WBS 
in the schedule was not consistent with the WBS used for the EVM data. 
The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. We found some sequencing issues in the spacecraft 
contractor schedule. We found that 27 percent of the remaining 
activities in the spacecraft's schedule were missing predecessor and 
successor links, which are necessary for properly sequencing work so 
that the schedule will update in response to changes. In addition, 24 
percent of the remaining activities had date constraints, which also 
hinder the schedule's ability to respond dynamically to status updates 
resulting in an artificial or unrealistic view of the project plan. 
These sequencing issues and constraint dates within the schedule 
affect the reliability of the overall network and the schedule's 
ability to correctly calculate float values and the critical path. 

Spacecraft Contractor Conducted an Integrated Baseline Review: 

Orbital Sciences conducted an integrated baseline review of the 
spacecraft effort in March 2011. 

EVM Surveillance Is Not Being Performed: 

While formal surveillance is not occurring, project officials stated 
that EVM performance data is reviewed during the monthly status 
reviews. 

Data Resulting from the EVM System Are Reliable: 

We reviewed contract performance reports from April 2011 to July 2011. 
Because we had only 4 months of data to review, we were unable to 
forecast estimate variance at completion. As of July 2011, the project 
was reporting a positive cumulative cost variance of $0.4 million and 
a negative cumulative schedule variance of $2 million as seen in the 
graph below. The positive cost variance was being driven by lower than 
expected burden rates and less staff supporting project management, 
flight assurance, and systems engineering efforts. 

Figure 15: GAO Earned Value Analysis of the Spacecraft Data: 

[Refer to PDF for image: line graph] 

2011: 

April; 
Cumulative cost variance: $0.184 million; 
Cumulative schedule variance: -$0.717 million. 

May; 
Cumulative cost variance: $0.277 million; 
Cumulative schedule variance: -$1.129 million. 

June; 
Cumulative cost variance: $0.31 million; 
Cumulative schedule variance: -$1.639 million. 

July; 
Cumulative cost variance: $0.375 million; 
Cumulative schedule variance: -$1.942 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The positive cost variance was being driven by lower than expected 
burden rates and less staff supporting project management, flight 
assurance, and systems engineering efforts. 

Radiation Belt Storm Probes: 

[Figure: artist depiction] 
* Project formulation start: June 2005; 
* Project confirmation date: December 2008; 
* Initial total project cost: $685.8 million; 
* Current total project cost: $686.0 million; 
* Launched on: August 30, 2012. 

Source: 2010. The Johns Hopkins University/Applied Physics Laboratory; 
All Rights Reserved. 

[End of figure] 

The Radiation Belt Storm Probes (RBSP) mission will explore the Sun's 
influence on the Earth and near-Earth space by studying the planet's 
radiation belts at various scales of space and time. This insight into 
the physical dynamics of the Earth's radiation belts will provide 
scientists with data to make predictions of changes in this little 
understood region of space. Understanding the radiation belt 
environment has practical applications in the areas of spacecraft 
system design, mission planning, spacecraft operations, and astronaut 
safety. The RBSP project built two spacecraft that will be used to 
measure the particles, magnetic and electric fields, and waves that 
reside in the Van Allen radiation belts. RBSP launched on August 30, 
2012 on a two-year prime mission. 

The RBSP spacecrafts and ground system are being designed, developed, 
and tested by the John Hopkins University's Applied Physics 
Laboratory.[Footnote 37] The RBSP project office is collecting EVM 
data at the project level. 

Table 12: Assessment of RBSP EVM Practices: 

Dollars in millions. 

Project; 
Used a certified EVM system compliant with ANSI standard: Agency did 
not meet this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $291; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: -$40 to -$41. 

Source: GAO analysis of contractor EVM data as of April 2011. 

[End of table] 

Project Does Not Have a Certified EVM System Compliant with the ANSI/ 
EIA Standard: 

At the project level, the Applied Physics Laboratory only fully met 
one and partially met another of the three fundamental practices 
necessary for a reliable EVM system. RBSP is the first full NASA 
mission to use EVM at the Applied Physics Laboratory. According to the 
RBSP project manager, the RBSP project implemented a limited earned 
value management system in Phase B as a risk mitigation activity for 
Phase C/D. This early implementation was a risk mitigation activity, 
which allowed the project's control account manager, instrument 
provider, and project office to better understand the reporting 
process and the use of the EVM system. The use of EVM during this 
phase was also intended to allow for timely, accurate, and useful EVM 
reporting during the formal reporting in later phases of the project. 
Since then Applied Physics Laboratory has made good progress and is in 
the process of meeting the intent of being compliant with the 32 
ANSI/EIA-748 guidelines. 

We assessed how well the Applied Physics Laboratory was meeting the 
three ANSI/EIA guidelines. These guidelines state that the authorized 
work elements for the project should be defined typically using a WBS 
that has been tailored to the project and the WBS is the same for the 
cost estimate, schedule, and EVM. Our review, we found that the WBS 
used in the schedule was not consistent with the WBS used for the EVM 
data. Project officials said that the project utilizes the Applied 
Physics Laboratory WBS for all earned value management activity, which 
is then mapped to the NASA WBS for reporting to the sponsor. Also, the 
Applied Physics Laboratory WBS is uniformly utilized and consistent 
across all control accounts in the EVM system. This internal WBS ties 
into both the contract performance report and the integrated master 
schedule utilized on the project. 

The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. Our review found some sequencing issues in the schedule. 
For example, 23 percent of the remaining activities were missing 
predecessor and successor links, which are necessary for properly 
sequencing work so that the schedule will update in response to 
changes. We also found that 29 percent of the remaining activities had 
date constraints, which also hinder the schedule's ability to respond 
dynamically to status updates resulting in an artificial or 
unrealistic view of the project plan. When schedules are not sequenced 
properly, float values and the calculated critical path will not be 
valid. Project officials explained that sequencing issues were a 
result of the project consciously including constrained instrument 
deliveries and deliverables, level of effort activities and material 
and subcontractor expenditures in the integrated master schedule, and 
though the RBSP integrated master schedule had these issues it was 
able to monitor the critical and near critical paths of all spacecraft 
systems and subsystems. Finally, the ANSI/EIA guidelines state that a 
project should establish and maintain a time-phased budget baseline to 
track cost and schedule variances in an EVM system. Though resource 
loading the schedule is not required to meet the ANSI/EIA guidelines, 
it is a best practice and therefore resources should be accounted for 
in the schedule in order to develop this baseline, according to the 
GAO cost guide. Our review found that the schedule was resource loaded. 

Project Conducted an Integrated Baseline Review: 

NASA conducted an integrated baseline review in August 2009. Of the 
seven IBR objectives identified, two were partially met and five were 
met. In December 2010, the IBR deputy chief notified the Applied 
Physics Laboratory that all areas of concern had been closed. As a 
result, the overall consensus of the government review team was that 
the integrated baseline review was successful. 

EVM Surveillance Is Not Being Performed: 

While formal surveillance is not being performed, the project office 
reviews the EVM data on a monthly basis. Although they do not perform 
formal surveillance, an Applied Physics Laboratory official said that 
they performed additional monthly independent reviews of the RBSP EVM 
system throughout Phase C/D. 

Data Resulting from the EVM System Are Reliable: 

We reviewed contract performance reports from August 2010 to April 
2011 and July 2011 to August 2011. We were not provided reports for 
May and June 2011. This was due to a re-plan determined necessary by 
the Living with a Star Program Office and NASA headquarters in May 
2011 due to changes in the launch manifest. Figure 16 below 
illustrates that as of August 2011, the project was reporting a 
negative cumulative cost variance of approximately $32 million and a 
negative cumulative schedule variance of $3 million. 

Figure 16: GAO Earned Value Analysis of Project Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: -$22.70 million; 
Cumulative schedule variance: -$12.88 million. 

September; 
Cumulative cost variance: -$22.89 million; 
Cumulative schedule variance: -$12.67 million. 

October; 
Cumulative cost variance: -$23.24 million; 
Cumulative schedule variance: -$12.57 million. 

November; 
Cumulative cost variance: -$24.27 million; 
Cumulative schedule variance: -$12.37 million. 

December; 
Cumulative cost variance: -$24.96 million; 
Cumulative schedule variance: -$12.39 million. 

2011: 

January; 
Cumulative cost variance: -$26.90 million; 
Cumulative schedule variance: -$12.78 million. 

February; 
Cumulative cost variance: -$28.16 million; 
Cumulative schedule variance: -$12.52 million. 

March; 
Cumulative cost variance: -$28.86 million; 
Cumulative schedule variance: -$11.42 million. 

April; 
Cumulative cost variance: -$29.59 million; 
Cumulative schedule variance: -$10.07 million. 

July; 
Cumulative cost variance: -$31.66 million; 
Cumulative schedule variance: -$2.94 million. 

August; 
Cumulative cost variance: -$31.92 million; 
Cumulative schedule variance: -$3.04 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The negative cumulative cost variance was caused by sustained effort 
on the radio frequency communications, as well as by work on the 
avionics equipment, and ground system software launch and post launch 
components being behind schedule. The schedule variance is minimal 
since the project is nearing completion. Due to the negative cost 
variance we are forecasting a negative variance at completion from $40 
million to $41 million. Project officials noted the forecasted 
variance at completion is below the revised contract value of $351.1 
million, although above the project's estimated budget at completion. 
In addition, officials said this is a project-level variance, and does 
not account for the application of unallocated future expenses to fund 
the movement of the launch date and to keep the project on track. When 
the project launched in August 2012, its estimate at completion was 
below the $351.1 million budget at completion. 

Stratospheric Observatory for Infrared Astronomy: 

[Figure: photograph] 
* Project formulation start: October 1991; 
* Project confirmation date: July 2007; 
* Initial total project cost: $2,954.5 million; 
* Current total project cost: $3,002.9 million; 
* Launch readiness date: December 2014. 

Source: NASA. 

[End of figure] 

The Stratospheric Observatory for Infrared Astronomy (SOFIA) is a 
joint project between NASA and Deutsche Zentrum für Luft-und Raumfahrt 
(DLR), the German space agency, to install a 2.5 meter telescope, as 
well as other scientific instruments capable of infrared and sub-
millimeter observations, in a specially modified Boeing 747SP 
aircraft. This airborne observatory is designed to provide routine 
access to the visual, infrared, far-infrared, and sub-millimeter parts 
of the electromagnetic spectrum. Its mission objectives include 
studying many different kinds of astronomical objects and phenomena, 
including star birth and death; the formation of new solar systems; 
planets, comets, and asteroids in our solar system; and black holes at 
the center of galaxies. Currently, five U.S. and two German funded 
interchangeable instruments for the observatory are being developed to 
allow a range of scientific measurement to be taken by SOFIA. 

The SOFIA project office is using EVM at the project level as well as 
collecting EVM data from the L-3 Communications Integrated Systems 
L.P. (L-3), which is responsible for the airborne system platform 
effort. The EVM data provided to the project for the airborne 
observatory platform effort is a subset of the overall SOFIA project 
level EVM report. The German component of the SOFIA project does not 
generate earned value data and are not part of the project's budget 
baseline. The University Space Research Association (USRA) has a 
support contract to help the Ames Research Center manage SOFIA's 
science and mission operations in cooperation with the Deutsches SOFIA 
Institute. The USRA contract was established before NASA began 
requiring earned value management compliance. As a result, they are 
not required to generate earned value data. However, all of these 
components are subsets of the overall SOFIA project. 

Table 13: Assessment of SOFIA EVM Practices: 

Dollars in millions. 

Project level; 
Used a certified EVM system compliant with ANSI standard: Agency did 
not meet this criterion; 
Conducted an integrated baseline review: Agency did not meet this 
criterion; 
EVM system surveillance is being performed: Agency partially met this 
criterion; 
Budget at completion: $384; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO variance at completion: -$1.4 to -$76. 

Airborne system; 
Used a certified EVM system compliant with ANSI standard: Fully met 
this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $38; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO variance at completion: $3 to $4. 

Source: GAO analysis of EVM data as of August 2011 for the project and 
July 2011 for the contractor data. 

[End of table] 

Project does not have a Certified EVM System Compliant with the ANSI/ 
EIA Standard: 

SOFIA project did not meet any of the three fundamental ANSI/EIA-748 
practices necessary for a reliable EVM system. Project officials said 
in-house projects are not required to have certified EVM systems. 
Though the SOFIA project does not have a certified system, we assessed 
how well the project was meeting three ANSI/EIA guidelines. These 
guidelines state that the authorized work elements for the project 
should be defined typically using a WBS that has been tailored to the 
project and the WBS is the same for the cost estimate, schedule, and 
EVM. We found that the WBS used in the SOFIA schedule was consistent 
with the WBS used for the EVM data. The ANSI/EIA guidelines also state 
that projects should have a schedule that describes the sequence of 
work by listing activities in the order in which they are to be 
carried out and identifying significant task interdependencies 
required to meet project requirements. Our review found some 
sequencing issues in the schedule. For example, 16 percent of the 
remaining activities had open ended logic links, and 1 percent had 
date constraints, among other things. SOFIA project officials 
explained that the majority of the missing links were due to using 
hammock activities to resource load the schedule, which resulted in 
missing successors.[Footnote 38] Out of all of the activities missing 
successors, only 11 were not hammocked activities, which accounted for 
a very small amount. In addition, SOFIA project officials said that 
constraints in the schedule were justified since they represented 
external deliveries or fiscal year funding availability. They added 
that the one hard constraint, Must Start On, was used to represent a 
fixed date for an international visit. While the majority of these 
explanations seem reasonable, when schedules are not sequenced 
properly, the slack values and the calculated critical path will not 
be valid. 

Project Did Not Conducted an Integrated Baseline Review: 

An IBR was not conducted at the project level. Project officials said 
SOFIA, an in-house project, did not begin collecting EVM data until 
very late in the development phase, and because of this an IBR was not 
conducted. Officials added although SOFIA did not conduct a project-
level IBR, the EVM baseline was established concurrently with an 
Agency approved re-plan and joint confidence level analysis in the 
2009/2010 time frame, and was reviewed by a Standing Review Board. 
Though the Standing Review Board review satisfied some of the 
objectives of an IBR, including confirmation of the schedule and 
budget baselines (e.g., schedule review, risk review, key milestones 
identified), it did not address the full IBR checklist (e.g. work 
authorizations, control account plans, EVM system description). 

EVM Surveillance Is Not being Performed: 

While formal surveillance is not occurring at the project level, EVM 
data assurance reviews are being performed monthly and quarterly by 
the SOFIA project office representatives. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to August 
2011. We found various issues that bring into question the reliability 
of the SOFIA Project EVM data. For example, we found negative values 
due to an "over-reporting" of progress in previous months that was 
caused by a problem with translating percent complete progress from 
the University Space Research Association schedule to the SOFIA 
integrated master schedule as well as corrections/modifications in 
costs posted for support service contracts. In addition, we found 
other anomalies in the data that SOFIA project officials explained 
were most likely due to mischarges by employees, delayed cost 
postings, or employees continuing to use charge codes inappropriately. 
While the cost impact of these problems was not that large for any one 
WBS element, each of these issues causes us to question the 
reliability of the data. According to project officials, the variances 
that caused these anomalies did not meet the reporting threshold. A 
variance analysis report provides a detailed, narrative report 
explaining significant cost and schedule variances and other contract 
problems and topics. Without this information, management cannot 
understand the reasons for variances and the contractor's plan for 
fixing them. When variance analysis reports are not produced, the EVM 
data will not be meaningful or useful as a management tool. 

Figure 17 below illustrates that as of August 2011, the SOFIA project 
was reporting a positive cumulative cost variance of $3.5 million and 
a negative cumulative schedule variance of $6.6 million. 

Figure 17: GAO Earned Value Analysis of the SOFIA Project Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: $6.2 million; 
Cumulative schedule variance: $0.4 million. 

September; 
Cumulative cost variance: $5.5 million; 
Cumulative schedule variance: $0.1 million. 

October; 
Cumulative cost variance: $3.6 million; 
Cumulative schedule variance: -$2.0 million. 

November; 
Cumulative cost variance: $6.8 million; 
Cumulative schedule variance: -$2.4 million. 

December; 
Cumulative cost variance: $4.1 million; 
Cumulative schedule variance: -$3.4 million. 

2011: 

January; 
Cumulative cost variance: $3.3 million; 
Cumulative schedule variance: -$4.6 million. 

February; 
Cumulative cost variance: $6.5 million; 
Cumulative schedule variance: -$1.9 million. 

March; 
Cumulative cost variance: $2.8 million; 
Cumulative schedule variance: -$6.1 million. 

April; 
Cumulative cost variance: $1.4 million; 
Cumulative schedule variance: -$7.9 million. 

May; 
Cumulative cost variance: $5.1 million; 
Cumulative schedule variance: -$5.4 million. 

June; 
Cumulative cost variance: $7.9 million; 
Cumulative schedule variance: -$2.0 million. 

July; 
Cumulative cost variance: $5 million; 
Cumulative schedule variance: -$4.7 million. 

August; 
Cumulative cost variance: $3.5 million; 
Cumulative schedule variance: -$6.6 million. 

Source: GAO analysis of NASA and contractor data. 

[End of figure] 

Due to the cost and schedule variances we are forecasting a negative 
variance at completion ranging from $1.4 million to $76 million. 

Airborne System Platform Contractor Has a Certified EVM System 
Compliant with the ANSI/EIA Standard: 

The contractor met all three fundamental ANSI/EIA-748 practices 
necessary for a reliable EVM system. In 2002, DCMA certified that the 
contractor has an EVM system compliant with the ANSI/EIA standard. 

Project Conducted an Integrated Baseline Review of Airborne System 
Platform Effort: 

The project conducted an IBR in November 2007. 

EVM Surveillance Is Being Performed: 

It is unclear if EVM surveillance is being performed on the L-3 EVM 
data. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We reviewed contract performance reports from August 2010 to July 
2011. We found various data anomalies that call into question the 
reliability of the contractor's EVM data for the contract performance 
reports we reviewed. For example, we found negative values, actual 
cost of work performed being reported without work being scheduled 
and/or performed. Project officials stated that these anomalies were 
caused by performance being taken in later months and also because 
these issues did not trip the threshold reporting. 

Figure 18 below illustrates that as of July 2011, the airborne system 
platform contractor was reporting a positive cumulative cost variance 
of $3.2 million and a negative schedule variance of $0.6 million. 

Figure 18: GAO Earned Value Analysis of the SOFIA Airborne System Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: $2.10 million; 
Cumulative schedule variance: -$0.38 million. 

September; 
Cumulative cost variance: $2.12 million; 
Cumulative schedule variance: -$0.29 million. 

October; 
Cumulative cost variance: $2.09 million; 
Cumulative schedule variance: -$0.32 million. 

November; 
Cumulative cost variance: $2.04 million; 
Cumulative schedule variance: -$0.60 million. 

December; 
Cumulative cost variance: $2.45 million; 
Cumulative schedule variance: -$0.72 million. 

2011: 

January; 
Cumulative cost variance: $2.45 million; 
Cumulative schedule variance: -$0.72 million. 

February; 
Cumulative cost variance: $2.66 million; 
Cumulative schedule variance: -$0.51 million. 

March; 
Cumulative cost variance: $2.71 million; 
Cumulative schedule variance: -$0.61 million. 

April; 
Cumulative cost variance: $3.01 million; 
Cumulative schedule variance: -$0.56 million. 

May; 
Cumulative cost variance: $3.21 million; 
Cumulative schedule variance: -$0.38 million. 

June; 
Cumulative cost variance: $3.13 million; 
Cumulative schedule variance: -$0.60 million. 

July; 
Cumulative cost variance: $3.15 million; 
Cumulative schedule variance: -$0.63 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The favorable cost variance is due to efficiencies in project 
oversight and engineering. The unfavorable cumulative schedule 
variance is being driven by a lack of government furnished equipment 
supposed to be received from NASA that is causing a backlog of tasks. 
Due to the cost and schedule variances we are forecasting a positive 
variance at completion ranging from $3 million to $4 million. 

Tracking and Data Relay Satellite Replenishment: 

[Figure: artist depiction] 
* Project formulation start: February 2007; 
* Project confirmation date: July 2009; 
* Initial total project cost: $451.3 million; 
* Current total project cost: $434.1 million; 
* Launch readiness date: 
K- December 2012; 
L- February 2014. 

Source: Boeing. 

[End of figure] 

The Tracking and Data Relay Satellite (TDRS) System consists of in-
orbit communication satellites stationed at geosynchronous altitude 
coupled with two ground stations located in New Mexico and Guam. The 
satellite network and ground stations provide mission services for 
near-Earth user satellites and orbiting vehicles. TDRS-K and L are the 
11th and 12th satellites, respectively, to be built for the TDRS 
system. They will contribute to the existing network by providing 
continuous high bandwidth digital voice, video, and mission payload 
data, as well as health and safety data relay services to Earth-
orbiting spacecraft such as the International Space Station and the 
Hubble Space Telescope. NASA is planning to launch TDRS-K in December 
2012 followed by the TDRS-L launch in February 2014. 

NASA is collecting EVM data from both the spacecraft and sustainment 
efforts. In December 2007, NASA awarded a fixed price incentive 
contract to design, develop, fabricate, integrate, test, ship, provide 
launch support, conduct on-orbit checkout operations and provide 
sustaining engineering support for two spacecraft, TDRS-K and TDRS-L, 
to Boeing Satellite Systems, Inc (Boeing). 

Table 14: Assessment of TDRS EVM Practices: 

Dollars in millions. 

Spacecraft; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $698; 
Data resulting from the EVM system are reliable: Fully met this 
criterion; 
GAO Variance at completion: -$152 to -$185. 

Sustainment; 
Used a certified EVM system compliant with ANSI/EIA standard: Fully 
met this criterion; 
Conducted an integrated baseline review: Fully met this criterion; 
EVM system surveillance is being performed: Fully met this criterion; 
Budget at completion: $5; 
Data resulting from the EVM system are reliable: Agency partially met 
this criterion; 
GAO Variance at completion: NA[A]. 

Source: GAO analysis of contractor data as of August 2011. 

[A] NA - Not applicable means there was not enough data to forecast an 
estimate at completion. 

[End of table] 

Spacecraft Contractor Has a Certified EVM System Compliant with the 
ANSI/EIA Standard: 

The spacecraft contractor, Boeing, met the three fundamental ANSI/EIA-
748 practices necessary for a reliable EVM system. Boeing has a 
certified EVM system that complies with the ANSI/EIA EVM standard. 
Though the contractor has a certified system, the implementation of 
that system is questionable based on our findings below. We also 
assessed whether the spacecraft contractor's EVM data against three 
ANSI/EIA guidelines. These guidelines state that the authorized work 
elements for the project should be defined typically using a WBS that 
has been tailored to the project and the WBS is the same for the cost 
estimate, schedule, and EVM. We found that the WBS used in the 
spacecraft integrated master schedule was consistent with the WBS used 
for the EVM data. 

The ANSI/EIA guidelines also state that projects should have a 
schedule that describes the sequence of work by listing activities in 
the order in which they are to be carried out and identifying 
significant task interdependencies required to meet project 
requirements. We found some sequencing issues in the contractor's 
schedule. For example, 13 percent of the remaining activities were 
constrained. When schedules are not sequenced properly, float values 
and the calculated critical path will not be valid. Project officials 
acknowledged the constraints and said they are a result of having to 
adjust support activities due to spacecraft integration and test 
delays as well as alignments to the current/actual manifest dates, 
which differ from the Boeing contractual and launch readiness dates. 
In addition, critical path metrics are generated and analyzed monthly 
to track the Boeing performance against the critical path activities. 
Finally, the ANSI/EIA guidelines state that a project should establish 
and maintain a time-phased budget baseline to track cost and schedule 
variances in an EVM system. Though resource loading the schedule is 
not required to meet the ANSI/EIA guidelines, it is a best practice 
and therefore resources should be accounted for in the schedule in 
order to develop this baseline, according to the GAO cost guide. We 
found that the schedule was not resource loaded. Project officials 
said the integrated master schedule is resource loaded but not inside 
the Microsoft Project schedule because it does not directly interface 
with the Boeing financial/EVM system. They stated that the integrated 
master schedule is produced using Microsoft Project and imported into 
a planning software tool. After the resource loading effort is 
performed, the planning data is transferred into the EVM system. The 
tool integrates the Microsoft Project schedule with the Boeing 
financial system and the resource allocations for each task. When 
adjustments are required Boeing again utilizes the planning tool. 

Project Conducted an Integrated Baseline Review: 

An integrated baseline review was conducted in 2008 where 211 issues 
were raised. All of these issues have since been resolved and closed. 

EVM Surveillance Is Being Performed: 

DCMA prepares a monthly EVM analysis report and performs continuous 
surveillance of Boeing's EVM implementation by sampling various 
control account managers for interviews about the process. 

Data Resulting from the EVM System Are Somewhat Reliable: 

We examined contract performance reports from August 2010 through 
August 2011. Figure 19 illustrates that as of August 2011, the project 
was reporting a negative cumulative cost variance of approximately 
$131 million and a negative cumulative schedule variance of $7 million. 

Figure 19: GAO Earned Value Analysis of the TDRS Spacecraft Project 
Data: 

[Refer to PDF for image: line graph] 

2010: 

August; 
Cumulative cost variance: -$0.096 million; 
Cumulative schedule variance: -$0.039 million. 

September; 
Cumulative cost variance: -$0.099 million; 
Cumulative schedule variance: -$0.030 million. 

October; 
Cumulative cost variance: -$0.104 million; 
Cumulative schedule variance: -$0.028 million. 

November; 
Cumulative cost variance: -$0.108 million; 
Cumulative schedule variance: -$0.016 million. 

December; 
Cumulative cost variance: -$0.108 million; 
Cumulative schedule variance: -$0.012 million. 

2011: 

January; 
Cumulative cost variance: -$0.110 million; 
Cumulative schedule variance: -$0.008 million. 

February; 
Cumulative cost variance: -$0.112 million; 
Cumulative schedule variance: -$0.008 million. 

March; 
Cumulative cost variance: -$0.115 million; 
Cumulative schedule variance: -$0.008 million. 

April; 
Cumulative cost variance: -$0.118 million; 
Cumulative schedule variance: -$0.004 million. 

May; 
Cumulative cost variance: -$0.122 million; 
Cumulative schedule variance: -$0.004 million. 

June; 
Cumulative cost variance: -$0.126 million; 
Cumulative schedule variance: -$0.005 million. 

July; 
Cumulative cost variance: -$0.128 million; 
Cumulative schedule variance: -$0.006 million. 

August; 
Cumulative cost variance: -$0.131 million; 
Cumulative schedule variance: -$0.007 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

The negative cumulative cost variance was being driven by higher 
staffing levels to support integration, the Preliminary Design Review 
and Critical Design Review, as well as an incorrect assessment of 
project requirements and the inability to use heritage specifications. 
Labor costs were also higher than expected due to part failures and 
the late completion of component qualifications. Extended test 
activities also contributed to the cumulative negative cost variance. 
Finally, more than expected resources were needed to complete board 
and slice designs, generate drawings, and assemble and test components 
because of the complexity of the design. Due to the negative cost and 
schedule variance we are forecasting a variance at completion ranging 
from $152 million to $185 million. Project officials said GAO's 
independent variance at completion gives the impression that NASA may 
request additional funding to complete the TDRS K and L. Because this 
is a fixed price, incentive fee contract, NASA officials said the 
agency is obligated only to pay up to the price ceiling of the 
contract. 

TDRS Sustainment Effort EVM Summary: 

Boeing is the contractor for both the spacecraft and sustainment 
efforts. As noted above, Boeing met all three fundamental practices 
for a reliable EVM system. 

In addition, the 2008 integrated baseline review was conducted for 
both the spacecraft and sustainment effort. As well, the formal 
surveillance performed by Boeing and the Defense Contract Management 
Agency applies to the sustainment effort: 

We reviewed contract performance reports from April 2011 to August 
2011. Because we had only 5 months of data, we were not able to 
forecast an independent estimate at completion. Figure 20 below 
illustrates that as of August 2011, the project was reporting a 
positive cumulative cost variance of $0.08 million. Because there were 
no variance analysis reports accompanying the sustaining effort, we 
were unable to determine what was causing the positive cost variance. 
Project officials stated that the Performance Measurement Baseline for 
this effort was almost entirely level of effort, so minimal variances 
would be occurring. They added that during the August 2011 time 
period, the contract reflected an April 2012 launch date even though 
the launch was being delayed. Consequently, Boeing's reports were 
reflecting work scheduled to occur in support of the earlier launch 
date when in fact very little effort was being done. As a result, 
since minimal costs were incurred, this resulted in a positive 
cumulative cost variance. 

Figure 20: GAO Earned Value Analysis of the TDRS Sustainment Data: 

[Refer to PDF for image: line graph] 

2011: 

April; 
Cumulative cost variance: $0.011 million; 
Cumulative schedule variance: -$0.003 million. 

May; 
Cumulative cost variance: $0.022 million; 
Cumulative schedule variance: -$0.003 million. 

June; 
Cumulative cost variance: $0.033 million; 
Cumulative schedule variance: -$0.003 million. 

July; 
Cumulative cost variance: $0.054 million; 
Cumulative schedule variance: -$0.006 million. 

August; 
Cumulative cost variance: $0.078 million; 
Cumulative schedule variance: -$0.006 million. 

Source: GAO analysis of contractor data. 

[End of figure] 

[End of section] 

Appendix IV: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration: 
Headquarters: 
Washington, DC 20546-0001: 

November 5, 2012: 

Reply to Attn: Office of the Chief Engineer: 

Ms. Cristina Chaplain: 
Director: 
Acquisition and Sourcing Management: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Ms. Chaplain: 

The National Aeronautics and Space Administration (NASA) appreciates 
the opportunity to review and comment on the Government Accountability 
Office (GAO) draft report entitled, "Earned Value Management 
Implementation Across Major Spaceflight Projects is Uneven" (GAO-13-
22). 

The Agency appreciates the thoroughness of the report: particularly 
the level of detail in describing situations where good Earned Value 
Management (EVM) practices were in evidence, those situations where 
partial or incomplete data was available, as well as shortcomings in 
regard to implementation. NASA agrees with GAO that EVM is one of many 
valuable tools used to assess and manage performance and values the 
GAO's feedback in this area. Additionally, NASA worked closely with 
GAO to provide technical corrections and clarifications to improve the 
accuracy of the GAO analysis. While we are not always fully in 
agreement, we appreciate the opportunity for open dialogue and the 
recommendations to improve NASA's EVM practices. NASA realizes that 
the GAO's work was based upon data that is more than a year old and 
understandably may not fully capture the significant progress that the 
Agency has made since then. 

In the draft report, GAO addresses four recommendations to the NASA 
Administrator. To improve NASA management and oversight of its 
spaceflight projects, GAO recommends that the NASA Administrator 
direct the appropriate offices to take the following actions: 

Recommendation 1: Establish a timeframe by which all new spaceflight 
projects will be required to implement NASA's newly developed EVM 
system, unless the project is proposing to use a certified system, to 
ensure that in-house efforts are compliant with ANSI/EIA-748. The 
timeframe selected should take in to account the need to increase 
NASA's institutional capability for conducting EVM and analyzing and 
reporting the data. 

Management's Response: Partially Concur. NASA already requires major 
spaceflight programs and projects to perform EVM with an ANSI/ETA 748 
compliant EVM system and acknowledges that not all projects are fully 
compliant with this requirement. To facilitate full compliance, NASA 
has developed an ANSI/E1A compliant Agency system and has made it 
available to any project that does not currently have a compliant 
system. A phased approach to rolling out the Agency EVM system based 
upon available resources, budgetary constraints, and institutional and 
project needs has been developed and supported by Agency leadership. 
NASA does not agree in the value proposition associated with forcing a 
project into applying the Agency system if the project has
skilled EVM workforce and an ANST/EIA compliant system in place. NASA 
will assess updating the requirement to apply the Agency EVM system to 
new projects as part of the normal revision process for NPR 7120.5, 
NASA Spaceflight Program and Project Management Requirements, in 2017. 
This will allow NASA the opportunity to continue the phased methodical 
implementation strategy with a target of having a cadre of expertise 
developed at all Centers responsible for managing major spaceflight 
acquisitions within a five-year horizon, assuming funding and 
resources are available to support the phased roll out. Once the 
institutional capability is in place at each Center, all Center 
projects will benefit. 

Recommendation 2: Conduct an EVM skills gap analysis to identify areas 
requiring augmented capability across the agency. Based on the results 
of the assessment, develop a workforce training plan to address any 
deficiencies. 

Management's Response: Concur. NASA concurs that the availability of 
experienced EVM experts across the Agency is limited. A skills gap 
assessment will be conducted, and the Agency EVM training program will 
be augmented to address the gaps. 

Recommendation 3: Develop an EVM change management plan to assist 
managers and employees throughout the agency with accepting and 
embracing earned value techniques while reducing the operational 
impact on the agency. The plan should include a strategy for having 
the agency's senior leadership communicate their commitment to 
implementation of EVM. 

Management's Response: Concur. NASA will develop a change management 
strategy to include training, communication, and other support 
materials targeted at all levels of the workforce from project team 
members to Agency leadership. 

Recommendation 4: To improve the reliability of project EVM data, NPR 
7120.5 should be modified to require projects to implement a formal 
surveillance program that: 

* Ensures anomalies in contractor-delivered monthly earned value 
management reports are identified and explained, and report 
periodically to the mission directorate's leadership on relevant 
trends in the number of unexplained anomalies. 

* Ensures consistent use of WBSs for both the EVM report and the 
schedule. 

* Ensures that lower level EVM data reconcile to project level EVM 
data using the same WBS structure. 

* Improves underlying schedules so that they are properly sequenced 
using predecessor and successor dependencies and are free of 
constraints to the extent practicable so that the EVM baseline is 
reliable. 

Management's Response: Partially Concur. NASA will not implement a 
formal surveillance program due to resource constraints, However, NASA 
concurs with the intent of the GAO recommendation and is taking action 
to improve the reliability and utility of the EVM data by establishing 
a surveillance process, expanding the workforce's EVM skills, and 
providing analytical tools consistent with those used by GAO in the 
audit. It is not appropriate to incorporate the surveillance 
requirement in NPR 7120.5 because it is inconsistent with the level of 
detail associated with all other requirements in the NPR. As part of 
the restructure of NPR 7120.5 in the current release, NASA made a 
concerted effort to place detailed information and guidance 
documentation in handbooks. 

NASA is developing an EVMS Acceptance and Surveillance Guide, as well 
as enhancing its analysis tools in response to the GAO recommendation.
NASA is very appreciative of the GAO's concerted effort, the 
professionalism of the team, and the attention to detail associated 
with the development of this report. We are committed to continuous 
improvement in order to explore and utilize space in an affordable way 
for the benefit of the Nation. To this end, NASA will use the findings 
identified in this report to improve Agency EVM implementation, and we 
look forward to continuing to work with the GAO to improve our 
performance and management practices. 

Thank you for the opportunity to comment on this draft report. If you 
have any questions or require additional information, please contact 
Sandra Smalley at (202) 358-4731. 

Sincerely, 

Signed by: 

Michael G. Ryschkewitsch: 
Chief Engineer: 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Shelby S. Oakley and Karen 
Richey (Assistant Directors), Greg Campbell; Jennifer K. Echard; Tisha 
D. Derricotte; Laura Greifner; Kristine R. Hassinger; Ben Jaskiewicz; 
William Laing; Richard Lee; Eric Lofgren; Kenneth E. Patton; Jose A. 
Ramos; Carrie W. Rogers; Stacey L. Steele; Roxanna T. Sun; and Umesh 
Thakkar made key contributions to this report. 

[End of section] 

Footnotes: 

[1] GAO, NASA: Assessments of Selected Large-Scale Projects, 
[hyperlink, http://www.gao.gov/products/GAO-12-207SP] (Washington, 
D.C.: March 1, 2012). Each of these major spaceflight projects have a 
lifecycle cost of $250 million or more. Since we began our review, 
five of the 21 projects have launched, one was canceled, and two other 
projects have entered the portfolio. 

[2] GAO, High-Risk Series: An Update, [hyperlink, 
http://www.gao.gov/products/GAO-11-278] (Washington, D.C.: Feb. 2011). 

[3] The NASA Inspector General report notes that NASA Policy Directive 
9501.3, Earned Value Performance Measurement, issued in February 1997, 
established the basis for applying EVM to its contracts. NASA 
considered a contract significant if it was a production contract with 
an estimated value of $250 million or more or an RDT&E contract with 
an estimated value of $60 million or more with a period of performance 
greater than 1 year. In March 2006, this policy was canceled and was 
replaced by NASA Procedural Requirements 7120.5. 

[4] NASA IG-99-058, Earned Value Management at NASA, September 30, 
1999; GAO, NASA: Lack of Disciplined Cost Estimating Processes Hinders 
Effective Program Management; [hyperlink, 
http://www.gao.gov/products/GAO-04-642] (Washington, D.C., May 28, 
2004); and GAO, Information Technology: Agencies Need to Improve the 
Implementation and Use of Earned Value Techniques to Help Manage Major 
System Acquisitions; [hyperlink, http://www.gao.gov/products/GAO-10-2] 
(Washington, D.C., Oct. 8, 2009). 

[5] NASA's life cycle for flight systems is defined by two phases--
formulation and implementation. The implementation phase, preceded by 
the formulation phase, is defined as the execution of approved plans 
for the development and operation of the program or project, and the 
use of control systems to ensure performance to approved plans and 
requirements and continued alignment with the agency's strategic 
goals. NASA Procedural Requirements 7120.5E, NASA Space Flight Program 
and Project Management Requirements, figure 2-4 and paragraph 1.3.1(c) 
(Aug. 14, 2012).(Hereinafter cited as NPR 7120.5E (Aug. 14, 2012.)) 

[6] GAO, GAO Cost Estimating and Assessment Guide, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2009). 

[7] The Program Management Council has the responsibility for 
periodically evaluating the technical, safety, and programmatic 
performance (including cost, schedule, and risk) and content of a 
program or project under their purview. These evaluations focus on 
whether the program or project is meeting its commitments to the 
agency. 

[8] [hyperlink, http://www.gao.gov/products/GAO-04-642]. 

[9] [hyperlink, http://www.gao.gov/products/GAO-10-2]. As part of the 
audit we selected NASA projects that rely on various elements of 
information technology and reviewed both the agency's spaceflight and 
information technology specific guidance. 

[10] A rebaseline is when cost, resources, and/or schedule commitments 
are revised. NASA projects are rebaselined when their estimated 
development cost exceeds NASA's baseline commitment development cost 
by 30 percent or more and Congress has reauthorized the project; 
events external to NASA make a rebaseline appropriate; or a NASA 
Associate Administrator determines that the project's scope changed 
from the approved project baseline. 

[11] The second EVM pilot on the Constellation Extra-Vehicular 
Activity Project was not completed due to the termination of the 
Constellation program. 

[12] EVM data are typically provided to NASA in the form of a standard 
report called the contract performance report. This report is the 
primary source of cost and schedule status and provides the 
information needed for effective program control. The contract 
performance report provides cost and schedule variances, based on 
actual performance against the plan, which can be further examined to 
understand the causes of any differences. 

[13] The critical path method is used to derive the critical 
activities--that is, activities that cannot be delayed without 
delaying the end date of the program. The amount of time an activity 
can slip before the program's end date is affected is known as "total 
float." 

[14] Constraints are generally used to demonstrate an external event's 
effect on the schedule. However, because they prevent activities from 
responding dynamically to network logic, including actual progress and 
availability of resources, they can affect float calculations and the 
identification or continuity of the critical path and can mask actual 
progress or delays in the schedule. Date constraints should be 
minimized because they restrict the movement of activities and can 
cause false dates in a schedule. 

[15] NASA Inspector General, Final Memorandum on Audit of the 
Stratospheric Observatory for Infrared Astronomy (SOFIA) Program 
Management Effectiveness, IG-09-013, (Washington, D.C., Mar. 27, 2009). 

[16] The Joint Cost and Schedule Confidence Level is a cost estimation 
tool that involves a probabilistic analysis of cost, schedule, and 
risk inputs to arrive at development cost and schedule estimates 
associated with various confidence levels. 

[17] An estimate at completion is an independent assessment of the 
cost to complete authorized work based on the project's historical EVM 
performance. Budget at completion is the sum of all estimated budgets, 
representing at the project level the cumulative value of the budgeted 
cost of the work scheduled over the life of the project or at lower 
levels, such as a control account or work breakdown structure element. 

[18] Variance thresholds try to quantify an acceptable range of 
deviation; those that do not exceed a threshold are usually not a 
concern while those that do are worthy of further inspection to 
determine the best course of action to minimize any negative impacts 
to the cost and schedule objectives. 

[19] GPM collects EVM data at the project level, however, this data 
was provided to GAO in a format we could not analyze. We included GPM 
in the 7 projects that had project level data. 

[20] As part of the initiative, the Office of the Chief Financial 
Officer piloted an approach for collecting cost at the level necessary 
to manage the work performed to support the work elements for 
effective project EVM. 

[21] FAR § 34.201(b). 

[22] The seven EVM principles as defined by ANSI/EIA-748 entail 
planning and integrating work scope, actual costs, and schedule into a 
performance management baseline against which accomplishments may be 
measured and assessed and the resulting information used to inform 
management's decision-making. 

[23] Additional, detailed, guidance is provided in an expanded NASA 
EVM Implementation Handbook. For example, the handbook describes roles 
and responsibilities of key project officials with respect to EVM, 
direction on how to implement EVM on the agency's in-house developed 
projects, and guidance on how EVM surveillance is to be conducted, as 
well as other important considerations. Projects are not required to 
follow the guidance in the handbook. 

[24] These surveys review center processes and infrastructure for 
compliance with Office of the Chief Engineer requirements, policy, 
procedures, processes, statutes, and regulations. A sample of projects 
is selected across NASA's centers every 2 years for review. The 
results are used to give centers feedback on areas where NASA policy 
and requirements should be modified and to respond to OMB and GAO 
concerns over the implementation of requirements. 

[25] GAO, Results-Oriented Cultures: Implementation Steps to Assist 
Mergers and Organizational Transformations, [hyperlink, 
http://www.gao.gov/products/GAO-03-669] (Washington, D.C.: July 2, 
2003). 

[26] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 
2009), and GAO Schedule Assessment Guide: Best Practices for Project 
Schedules--Exposure Draft, [hyperlink, 
http://www.gao.gov/products/GAO-12-120G] (Washington, D.C.: May 2012). 

[27] The three fundamental EVM practices we assessed were using a 
certified EVM system, conducting integrated baseline reviews, and 
performing surveillance of the EVM system. 

[28] [hyperlink, http://www.gao.gov/products/GAO-09-03SP]. 

[29] The agency baseline commitment is the integrated set of 
requirements, cost, schedule, technical content, and an agreed-to 
joint confidence level that forms the basis for NASA's commitment with 
OMB and Congress. NPR 7120.5E, appendix A (Aug. 14, 2012). 

[30] Unallocated future expenses are costs that are expected to be 
incurred but cannot yet be allocated to a specific work breakdown 
structure subelement of a project's plan. Management control of some 
unallocated future expenses may be retained above the level of the 
project (i.e., Agency, Mission Directorate, or program). The 
management agreement defines the parameters and authorities over which 
the program or project manager has management control. 

[31] The Landsat Data Continuity Mission (LDCM) and James Webb Space 
Telescope (JWST) projects do not collect project level data and GPM 
did not provide project level data in a format we could analyze. 
Therefore, for these projects, we were only able to analyze EVM data 
for segments of the project that reported EVM. Also, these projects 
had project segments, such as instruments, that did not produce any 
EVM data. 

[32] NASA was also developing a second Microwave Imager instrument for 
flight on another spacecraft--GMI-2--that was descoped from the 
project in 2012. However, the science requirements for GPM can still 
be met without flying the GMI-2 instrument. 

[33] Year-end indirect rate adjustments often happen when financial 
accounting assumptions about forward pricing do not hold true. When a 
company reconciles its books at the year end, it often finds that 
during the year it forecasted a larger business base than the actual 
business base, and at the end of the year, indirect rate adjustments 
have to be made, which can result in additional cost variances for all 
contracts. 

[34] Variance at completion is calculated by taking the difference 
between the estimate at completion and the budget at completion. 

[35] The Lunar Laser Communications Demonstration instrument is being 
funded by NASA's Human Exploration and Operations Mission Directorate 
at a cost of approximately $65 million. This cost is not included in 
the $262.9 million LADEE cost baseline. 

[36] The Lunar Quest Program consists of flight missions and 
instruments for lunar missions of opportunity, as well as research and 
analysis efforts. 

[37] The Applied Physics Laboratory is organized as a Limited 
Liability Company, wholly owned by Johns Hopkins University and 
operated as a division of the university. 

[38] A hammock activity is a way to represent level of effort 
activities in the schedule. Having no set duration of its own, the 
hammock activity duration is determined by the number of days between 
the beginning of the first activity and the end of the last activity 
in the group-not the sum of the activities' durations. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]