Information Technology: Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System Acquisitions
GAO-10-2
Published: Oct 08, 2009. Publicly Released: Nov 06, 2009.
Skip to Highlights
Highlights
In fiscal year 2009, the federal government planned to spend about $71 billion on information technology (IT) investments. To more effectively manage such investments, in 2005 the Office of Management and Budget (OMB) directed agencies to implement earned value management (EVM). EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of anticipated costs at completion. GAO was asked to assess selected agencies' EVM policies, determine whether they are adequately using earned value techniques to manage key system acquisitions, and eval- uate selected investments' earned value data to determine their cost and schedule performances. To do so, GAO compared agency policies with best practices, performed case studies, and reviewed documenta- tion from eight agencies and 16 major investments with the highest levels of IT development-related spending in fiscal year 2009.
While all eight agencies have established policies requiring the use of EVM on major IT investments, these policies are not fully consistent with best practices. In particular, most lack training requirements for all relevant personnel responsible for investment oversight. Most policies also do not have adequately defined criteria for revising program cost and schedule baselines. Until agencies expand and enforce their EVM policies, it will be difficult for them to gain the full benefits of EVM. GAO's analysis of 16 investments shows that agencies are using EVM to manage their system acquisitions; however, the extent of implementation varies. Specifically, for 13 of the 16 investments, key practices necessary for sound EVM execution had not been implemented. For example, the project schedules for these investments contained issues--such as the improper sequencing of key activities--that undermine the quality of their performance baselines. This inconsistent application of EVM exists in part because of the weaknesses contained in agencies' policies, combined with a lack of enforcement of policies already in place. Until key EVM practices are fully implemented, these investments face an increased risk that managers cannot effectively optimize EVM as a management tool. Furthermore, earned value data trends of these investments indicate that most are currently experiencing shortfalls against cost and schedule targets. The total life-cycle costs of these programs have increased by about $2 billion. Based on GAO's analysis of current performance trends, 11 programs will likely incur cost overruns that will total about $1 billion at contract completion--in particular, 2 of these programs account for about 80 percent of this projection. As such, GAO estimates the total cost overrun to be about $3 billion at program completion (see figure). However, with timely and effective management action, it is possible to reverse negative trends so that the projected cost overruns may be reduced.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Agriculture | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In response to our recommendation, the Department of Agriculture developed and, in March 2013, held a training course for executives on using earned value management (EVM) for decision making. According to the course's materials, the training was intended to provide executives with knowledge to improve risk management and decision making as it relates to performance trends from both a schedule and cost perspective. In addition, in January 2010, the department revised its EVM policy to include rebaselining criteria, including describing reasons when a rebaseline is warranted, describing the process for developing a new baseline, and requiring validation and management review of the new baseline. As a result, Agriculture is better positioned to implement EVM across the agency, and use it to more effectively manage its projects.
|
| Department of Homeland Security | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
Consistent with our recommendation, in May 2013, Department of Homeland Security (DHS) issued its Cost Estimating Handbook, which included a standard work breakdown structure for DHS information technology programs. The handbook notes that work the breakdown structure should be mappable to the program's earned value management (EVM) system. In addition, in October 2011, the issued the Acquisition Management Instruction/Guidebook and the Acquisition Program Baseline, which described the acceptable reasons for rebaselining, the process for developing a new baseline, and the requirements for validation and management review of the new baseline. As a result, DHS is better positioned to implement EVM across the agency, and use it to more effectively manage its projects.
|
| Department of Veterans Affairs | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In March 2010, the Department of Veterans Affairs addressed the weaknesses we identified by establishing its Program Management and Accountability System which included clear criteria for implementing the process on IT projects, established a standard structure for defining work products, and identified rebaselining criteria.
|
| Department of Commerce | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The Department of Commerce's Field Data Collection Automation (FDCA) program improved its ability to assess and to make timely decisions about contractor performance by integrating earned value data analysis into key program management tools. As a result, by July 2010, the FDCA program had reversed negative performance trends and had outperformed it planned cost target by about $5 million, thus mitigating the potential cost overruns expected on this key system acquisition program.
|
| Department of Commerce | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
As of July 2014, the Department of Commerce had not updated its policies governing earned value management to require the use of standard product-oriented work breakdown structures. According to the Commerce's Office of the Chief Information Officer, due to the diversity in the department's goals, missions, and programs, it defers to the operating units and programs in setting any detailed policies. However, Commerce did not provide evidence that its key operating units had updated their policies to require the use of standard work breakdown structures.
|
| Department of Homeland Security | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The Department of Homeland Security's Automated Commercial Environment (ACE) program switched from using a cost-plus-fixed-fee contract structure, where costs can be adjusted as a result of changes in the work to be performed, to using a firm-fixed-price structure with its major contractor. Consistent with this approach, in June 2010, the ACE program agreed to a firm-fixed-price task order with its major contractor that included the remaining implementation work through program completion. Because a firm-fixed-price contract places maximum risk on the contractor to provide the deliverable at the time, place, and price negotiated by the contractor and the government, the ACE program has reduced overall program risk and mitigated the potential for cost overruns (including those associated with schedule delays) on this key system acquisition program.
|
| Department of Justice | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In November 2009, the Department of Justice updated its earned value management (EVM) policy to require training for all personnel with investment oversight and program management responsibilities. In addition, the updated policy requires the use of a standard structure for defining work products. As a result, the Department of Justice has improved its ability to implement EVM across the agency.
|
| Department of Defense | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In August 2012, the Department of Defense's Office of Performance Assessments and Root Cause Analyses (PARCA) completed a review of the department's earned value management (EVM) training curricula. Subsequently, the Defense Acquisition University, working with PARCA, developed a new EVM training course. As of January 2014, this training course was incorporated into the curricula and required for program management personnel as part of the Defense Acquisition Workforce Improvement Act's Program Management Certification. As a result, the Department of Defense is better positioned to implement EVM across the agency, and use it to more effectively manage its projects.
|
| National Aeronautics and Space Administration | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The National Aeronautics and Space Administration's James Webb Space Telescope (JWST) program and Mars Science Laboratory (MSL) program took several actions to implement key earned value management practices. Specifically, the JWST program established a work breakdown structure dictionary, instituted schedule health checks, and increased the frequency of variance analysis. Additionally, the MSL program also instituted schedule health checks and began analyzing root causes and corrective actions for variances.
|
| Department of Commerce | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The Department of Commerce's Field Data Collection Automation (FDCA) program improved its ability to establish a comprehensive earned value management (EVM) system by implementing metrics to analyze the program schedule and determine if actions are needed to address scheduling issues, such as the improper sequencing of tasks. Further, the program improved its ability to ensure reliable data by integrating EVM data analysis into tools used for monitoring contractor performance. Lastly, the FDCA program has taken steps to improve its use of earned value data to manage the acquisition. For example, the program took action to mitigate risks by integrating risk review and analysis into the Decennial Census performance dashboard.
|
| Department of Veterans Affairs | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The Department of Veterans Affairs formulated policies as part of its Program Management and Accountability System that incorporate critical practices of earned value management. Based on these new policies, the department found the VistA-FM program to be poorly planned and managed, areas of which we also reported as having weaknesses. As a result of this evaluation and our recommendation, the department eliminated the program.
|
| Department of Homeland Security | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The Department of Homeland Security's Automated Commercial Environment (ACE) and Integrated Deepwater System--Common Operational Picture (COP) programs took actions to implement key earned value management (EVM) practices. Specifically, in June 2011, the ACE program conducted an integrated baseline review to validate its cost and schedule performance measurement baseline. During this review, the program assessed the quality of the integrated master schedule, including that all tasks were included and that resources had been assigned to these tasks, thus positioning the program to better execute planned work activities. In addition, the Deepwater COP program took actions to improve its ability to schedule and execute work, analyze EVM performance data, and execute management actions to mitigate risks. Specifically, the COP program increased its oversight of the contractor's EVM system though monthly reviews of earned value data in order to more closely monitor the program's schedule, analyze earned value performance data, and coordinate corrective actions--areas of which we also reported as having weaknesses.
|
| Department of Transportation | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The Department of Transportation's En Route Automation Modernization program took several actions to implement key earned value management (EVM) practices. Specifically, in August 2012, the program conducted an integrated baseline review--a key practice needed to effectively determine the objective measure of earned value and develop and validate the performance measurement baseline. In addition, the program took actions to improve the reliability of the program's schedule, including conducting schedule reviews on a weekly basis and using performance metrics to track the success of dependent activities, thus positioning the program to better execute planned work activities.
|
| Department of Agriculture | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
As of July 2014, the Department of Agriculture's Farm Program Modernization (MIDAS) program had not taken action to implement key earned value management practices. According to Agriculture's Office of the Chief Information Officer, there has not been any recognizable improvement in the earned value management practices and execution of the MIDAS program.
|
| Department of Defense | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report. |
The Department of Defense's Warfighter Information Network-Tactical (WIN-T) and the Joint Tactical Radio System-Handheld, Manpack, Small Form Fit (JTRS-HMS) programs took actions to implement key earned value management (EVM) practices. For example, in May 2010, the WIN-T program conducted an integrated baseline review and identified a list of action items intended to improve, among other things, the measurement of earned value and the quality of the program's cost and schedule baselines. The program also improved its ability to use EVM data for decision-making purposes by updating the performance baseline as program changes occurred. Specifically, the program management team documented baseline changes requests to identify work activities that had increased or decreased in value and the rationale for the adjustments, which were subsequently approved by the WIN-T Program Manager. In addition, the JTRS-HMS program sought assistance from the Defense Contract Management Agency to evaluate the program's master schedule on a monthly-basis to identify weaknesses for corrective action.
|
| Department of Defense | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The Department of Defense's Warfighter Information Network-Tactical (WIN-T) program and the Joint Tactical Radio System-Handheld, Manpack, Small Form Fit (JTRS-HMS) programs took actions to improve their negative performance trends. For example, in May 2010, the WIN-T program conducted a review of the performance baseline and identified a list of action items intended to improve, among other things, the measurement of earned value and the quality of the program's schedule baseline. The WIN-T Program Office and prime contractor also conducted comprehensive "deep dive" reviews at a major subcontractor where schedule performance data for planned work were discussed. By January 2011, the WIN-T program had reduced its schedule variance by approximately $4.5 million. In addition, the JTRS-HMS program restructured the contractor management team overseeing system development efforts by replacing key personnel, such as the JTRS Program Director, and reorganizing staff to better align with specific work products. Further, the program identified significant schedule drivers and took action to address them by, for example, revising the program schedule to focus on the delivery of smaller incremental capabilities. By April 2011, the JTRS-HMS program had reduced its schedule variance by approximately $6.3 million. As a result, the Department of Defense has improved schedule performance and reduced the potential for cost overruns associated with schedule delays that likely would have been incurred on these key system acquisition programs.
|
| Department of Veterans Affairs | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The Department of Veterans Affairs formulated policies as part of its Program Management and Accountability System that incorporated critical practices of earned value management. Based on these new policies, the department found the VistA-FM program to be poorly planned and managed, areas of which we also reported as having weaknesses. As a result of this evaluation and our recommendation, the department eliminated the program.
|
| Department of Agriculture | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
As of July 2014, the Department of Agriculture's Farm Program Modernization (MIDAS) program had not taken action to reverse negative performance trends to mitigate potential cost and schedule overruns. According to Agriculture's Office of the Chief Information Officer, there has not been any recognizable improvement in the earned value management practices and execution of the MIDAS program.
|
| Department of Justice | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
In August 2010, the Department of Justice's Next Generation Identification (NGI) program established an Executive Change Control Board to better prevent contract scope increases. This board is responsible for, among other things, reviewing and approving any change requests that affect the scope, milestones, and/or cost of the NGI development contract. In addition, the department established an independent verification and validation contractor with responsibility for independently reporting on a monthly basis the status of the NGI program, including earned value data. As a result, by February 2011, NGI had reduced the program's cost variance to approximately 1.6 percent of the cumulative cost of work performed, thus improving cost performance and reducing the potential cost overruns expected on this key system acquisition program.
|
| Department of Transportation | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In July 2010, the Department of Transportation updated its earned value management (EVM) policy to include consistent requirements for complying with industry EVM standards. In addition, the updated policy requires EVM training for executive-level staff and the use of a standard structure for defining work products. As a result, the Department of Transportation has improved its ability to implement EVM across the agency.
|
| National Aeronautics and Space Administration | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report. |
In December 2011, the National Aeronautics and Space Administration updated its project management guidance for IT projects to define and require the use of a standard work breakdown structure. In addition, the updated guidance includes criteria for rebaselining and a description of the required steps to be taken in the event of a rebaseline.
|
| Department of Transportation | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The Surveillance and Broadcast System program reported on earned value data trends to an executive-level council on a monthly basis and, by July 2010, had reduced its negative schedule variance. These actions have reduced the potential schedule delays expected on this key system acquisition program.
|
| National Aeronautics and Space Administration | In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns. |
The National Aeronautics and Space Administration's (NASA) James Webb Space Telescope (JWST) program and Mars Science Laboratory (MSL) program took several actions to mitigate potential cost and schedule overruns. For the JWST program, this included rebaselining its prime contract in May 2011 and improving its earned value management (EVM) practices, while the MSL program also took steps to improve its EVM practices, particularly by analyzing the root causes and corrective actions for negative variances shown in the EVM data. These actions helped NASA to mitigate the potential for cost and schedule overruns on these key system acquisition programs. Specifically, as of August 2012, the JWST program had achieved cost and schedule performance in-line with its May 2011 baseline, while the MSL program launched on schedule in November 2011, when measured against its fiscal year 2010 baseline.
|
Full Report
Public Inquiries
Topics
Best practicesCost analysisCost overrunsEarned value management systemsInformation technologyInvestmentsIT acquisitionsProgram managementReporting requirementsRequirements definitionSchedule slippagesInformation managementCost estimatesPolicies and procedures