Information Technology Cost Estimation: Agencies Need to Address Significant Weaknesses in Policies and Practices
GAO-12-629
Published: Jul 11, 2012. Publicly Released: Jul 11, 2012.
Skip to Highlights
Highlights
What GAO Found
While the eight agencies GAO reviewed—the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Labor, and Veterans Affairs, and the Environmental Protection Agency—varied in the extent to which their cost-estimating policies and procedures addressed best practices, most had significant weaknesses. For example, six of the eight agencies had established a clear requirement for programs to develop life-cycle cost estimates. However, most of the eight agencies’ policies lacked requirements for cost-estimating training, a standard structure for defining work products, and a central, independent cost-estimating team, among other things. The weaknesses in agencies’ policies were due, in part, to the lack of a priority for establishing or enhancing department or agency-level cost-estimating functions. Until agencies address weaknesses in their policies, it will be difficult for them to make effective use of program cost estimates for informed decision making, realistic budget formation, and meaningful progress measurement.
The 16 major acquisition programs had developed cost estimates and were using them, in part, to support program and budget decisions. However, all but 1 of the estimates were not fully reliable—meaning that they did not fully reflect all four characteristics of a reliable cost estimate identified in the GAO cost-estimating guide: comprehensive, well-documented, accurate, and credible. For example, the estimates for many of these investments did not include all life-cycle costs, such as costs for operating and maintaining the system; did not adequately document the source data and methodologies used to develop the estimate; were not regularly updated so that they accurately reflected current status; and lacked credibility because they were not properly adjusted to account for risks and uncertainty. The inadequate implementation of cost-estimating best practices was largely due to weaknesses in agencies’ policies. Until cost-estimating best practices are fully implemented, these programs face an increased risk that managers will not be able to effectively use their cost estimates as a sound basis for informed program and budget decision making.
Why GAO Did This Study
The federal government plans to spend at least $75 billion on information technology (IT) investments in fiscal year 2012. The size of this investment highlights the importance of reliably estimating the costs of IT acquisitions. A reliable cost estimate is critical to the success of any IT program, providing the basis for informed decision making and realistic budget formation. Without the ability to generate such estimates, programs risk missing their cost, schedule, and performance targets.
GAO was asked to (1) assess selected federal agencies’ implementation of cost-estimating policies and procedures, and (2) evaluate whether selected IT investments at these agencies have reliable cost estimates to support budget and program decisions. To do so, GAO compared policies and procedures to best practices at eight agencies. GAO also reviewed documentation supporting cost estimates for 16 major investments at these eight agencies—representing about $51.5 billion of the planned IT spending for fiscal year 2012.
Recommendations
GAO is recommending that the selected agencies modify cost-estimating policies to be consistent with best practices and update future cost estimates of the selected acquisition programs to address identified weaknesses. The seven agencies that commented on a draft of this report generally agreed with GAOs results and recommendations, although the Environmental Protection Agency disagreed with the assessment of one of its investments. However, GAO stands by its assessment.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Agriculture | To address weaknesses identified in the Department of Agriculture's policies for cost estimating, we recommend that the Secretary of Agriculture direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Department of Agriculture (USDA) has implemented this recommendation. USDA modified its cost estimating policy by creating a departmental regulation (Departmental Regulation (DR) 3130-012, dated March 4, 2016) as policy for governing information technology (IT) cost estimating procedures. The regulation requires the use of the GAO Cost Estimating and Assessment Guide and its 12-step process for all IT investments. The GAO guide, which represents a comprehensive presentation of GAO identified best practices that are essential for effective cost estimating guidance, is now a principal USDA IT cost estimating policy element. As a result, USDA has addressed the cost estimating policy weaknesses that we identified in our report. For example, USDA policy now includes ensuring that estimates establish and comply with cost-estimating best practices; and requiring that estimates be reviewed and approved by management, among other practices. The use of these practices will lead to cost estimates that are more reliable with more comprehensive, well-documented, accurate, and credible attributes.
|
| Department of Commerce | To address weaknesses identified in the Department of Commerce's policies for cost estimating, we recommend that the Secretary of Commerce direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Department of Commerce has implemented this recommendation by modifying its policy governing cost estimates. In June 2014, the department issued a revised policy on cost estimation and independent cost estimates; and in May 2015, Commerce issued a Departmental Administrative Order entitled Acquisition Project Management (DAO 208-16). These policy documents presented a clear requirement for cost estimating; stated that cost estimates use GAO best practices; and specified that the GAO Cost Estimating and Assessment Guide is a key resources for use in cost estimating activities. The policy requires that cost estimates contain attributes that allow them to be reliable, comprehensive, well-documented, accurate and credible as proscribed in our report. Additionally, the May 2015 Administrative Order further improved cost estimating policy by addressing specific weaknesses such as management review and training requirements that we identified in our report. The use of this modified policy and the related practices will lead to cost estimates that are more reliable with more comprehensive, well-documented, accurate, and credible attributes.
|
| Department of Homeland Security | To address weaknesses identified in the Department of Homeland Security's policies for cost estimating, we recommend that the Secretary of Homeland Security direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Department of Homeland Security has addressed our recommendation. As of October 2014, the department issued an updated Cost Estimating Handbook. The Handbook includes policies addressing the weaknesses we identified in our report. For example, it established requirements for cost estimating training, a standard structure for defining work products, and a process to collect and store cost-related data. The updated policies established in the handbook provide department officials with sound program cost estimates that may be used for informed decision making on how to most effectively manage their portfolios of projects.
|
| Department of Labor | To address weaknesses identified in the Department of Labor's policies for cost estimating, we recommend that the Secretary of Labor direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Department of Labor has generally implemented this recommendation by updating its policies governing cost estimates and establishing tools for defining work products and collecting cost related data. For example, Labor's updated System Development Life-cycle guide requires the use of standard work beak-down structures (WBS) at the beginning of the Conceptual Planning Phase and updated as necessary for each of the remaining life cycle phases. The guidance also requires the project managers to create a WBS dictionary that defines the work activities, deliverables and estimated resources needed to complete the work activities which help the department track cost and schedule by defined deliverables, and results in more consistent cost estimates. In addition, project managers are required to conduct a gate review at the end of each phase to determine whether all required WBS tasks have been completed successfully. Labor also established a cost benefit analysis tool which provides the department a standardized approach for performing quantitative analysis of the life cycle costs and benefits associated with each alternative to its IT investments. Additionally, Labor's IT Cost Estimate guide which is based on the GAO Cost Estimating and Assessment Guide, establishes a process for collecting and storing cost-related data which allows the cost data to be available to support future estimates. Labor, however, only partially addressed the policy on providing cost-estimating training. Labor's policy provides cost-estimating training and a mechanism to track participation, but the department did not provide evidence that its policy addressed providing training to personnel with investment oversight responsibility. In addition, while the guidance calls for the project manager (PM) to have the estimates independently reviewed by another PM, the guidance does not require an independent cost estimate conducted by a group outside the acquiring organization. Nevertheless, fully implementing its cost estimating guidance should provide Labor a basis for effective program cost estimating and result in more reliable cost estimates that management can use for making informed decisions.
|
| Department of Veterans Affairs | To address weaknesses identified in the Department of Veterans Affairs' policies for cost estimating, we recommend that the Secretary of Veterans Affairs direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Department of Veterans Affairs has addressed our recommendation. Specifically, in response to our recommendation, the department informed GAO that it has updated its Cost Estimating Guide. The cost estimating policies and procedures in the updated guide addressed the weaknesses we found with its cost estimating policy. For example, the guide established a process to collect and store cost related data. Specifically, all program cost estimates must be uploaded to a registry that is used to establish a central repository and knowledge base to inform future cost analysis. Establishing an agency-wide process to collect and store cost related data will improve the data available to all programs and increase the efficiency of developing cost estimates. As a result, program officials across the department will have reliable cost estimates to make informed program management decisions reducing the likelihood of cost overruns, missed deadlines, and performance shortfalls.
|
| Department of Justice | To address weaknesses identified in the Department of Justice's policies for cost estimating, we recommend that the Attorney General direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
A management analyst in the Department of Justice's Internal Review and Evaluation Office, Justice Management Division, responded orally that the department had no comments on our report. We have contacted Justice to obtain supporting information on its actions to address our recommendation. However, Justice was unable to provide sufficient evidence that the agency modified its policies governing cost estimating to ensure that it addressed the weaknesses that we identified. We continue to believe that until Justice addresses weaknesses in the department's cost-estimating policies and practices, it will be difficult for the agency to make effective use of program cost estimates for informed decision making, realistic budget formation, and meaningful progress measurement.
|
| Environmental Protection Agency | To address weaknesses identified in the Environmental Protection Agency's policies for cost estimating, we recommend that the Administrator of the Environmental Protection Agency direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified. |
The Environmental Protection Agency (EPA) has implemented this recommendation by modifying its cost estimating policy. Specifically, EPA updated its System Life Cycle Management Procedure document that defines the requirements and processes for implementation of the Agency system life cycle management (SLCM) policy; and establishes Agency practices for information technology systems and applications. It directs EPA Project Managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities; and specifies that the GAO Guide is part of EPA related policy, standards and guidance. The GAO guide specifically addresses the EPA cost estimating weaknesses described in our report, including establishment of an agency-wide process to collect and store cost-related data; a system to enforce training requirements for personnel with program management and investment oversight responsibilities; and a standard structure for defining work products. Consequently, usage of the best practice GAO guide will reduce the likelihood that the EPA cost estimates will only partially reflect key activities for developing a comprehensive, well-documented, and credible estimate. As a result, the risks of cost overruns, missed deadlines, and performance shortfalls will be significantly reduced.
|
| Department of Agriculture | To address weaknesses identified in the Department of Agriculture's practices for cost estimating, we recommend that the Secretary of Agriculture direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department of Agriculture has implemented this recommendation by directing that a departmental regulation requiring use of the GAO cost estimating guide and its 12-step process be utilized for all IT cost estimating activities. The GAO guide, which represents a comprehensive presentation of GAO identified best practices that are essential for effective cost estimating guidance and served as criteria for our assessment of program cost estimates, is now a principal feature of USDA IT cost estimating activities. By specifying that the GAO Guide be used, the USDA regulation presents the best practices governing characteristics of a reliable cost estimate. For example, among other practices, the GAO Guide and the Agriculture regulation require that program cost estimates be updated to reflect actual costs and cost changes. As a result of this action, responsible USDA officials are positioned to update future life-cycle cost estimates using cost-estimating practices that will lead to cost estimates that have more comprehensive, well-documented, accurate, and credible attributes and program officials will have a more reliable basis for increasing the likelihood that program cost estimates account for all possible costs.
|
| Department of Commerce | To address weaknesses identified in the Department of Commerce's practices for cost estimating, we recommend that the Secretary of Commerce direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department of Commerce has implemented this recommendation by directing program managers to follow GAO cost estimating principles and best practices and by specifying that the GAO Cost Estimating Guide be used as a key resource governing cost estimating activities. This action applies to all acquisition programs, projects and cost estimating activities throughout the Department; addresses the need to regularly update estimates; and requires that cost estimates contain attributes that allow them to be reliable, comprehensive, well-documented, accurate and credible as described in our report. By incorporating all practices advocated by GAO, including updating of future cost estimations to reflect actual costs and cost changes, future life-cycle cost estimates of the system acquisition programs discussed in our report will use cost-estimating practices that will improve the basis for budgetary decisions.
|
| Department of Homeland Security | To address weaknesses identified in the Department of Homeland Security's practices for cost estimating, we recommend that the Secretary of Homeland Security direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department of Homeland Security has addressed our recommendation. Specifically, in response to our recommendation, in December 2014, we confirmed that DHS officials directed IPAWS and the Rescue 21 management to use the revised DHS Cost Estimating Handbook for future cost estimations. This updated handbook established practices governing the characteristics of a reliable cost estimate and addressed the weaknesses of the IPAWS and Rescue 21 program cost estimates we identified in our report. For example, the handbook establishes the required practices of identifying data and data sources, and normalizing the data to improve its consistency and comparability, among others. In addition, the IPAWS program office updated its cost estimate in December 2015 and included improvements such as having 10 years of operations and maintenance costs beyond the end of development ensuring a more comprehensive cost estimate. As a result, program officials have a more robust basis for informed budget making decisions reducing the likelihood that IPAWS and Rescue 21 will have cost overruns.
|
| Department of Labor | To address weaknesses identified in the Department of Labor's practices for cost estimating, we recommend that the Secretary of Labor direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department of Labor has generally addressed this recommendation. As of May 2016, the Department revised its cost estimating guidance and practices used when updating the life-cycle estimate to address weaknesses we identified. For example, the guidance requires the cost estimate to include both government and contractor costs of the program over its full life cycle. The cost estimate should also be based on a work breakdown structure that allows a program to track cost and schedule by defined deliverables which helps ensure a comprehensive and well documented cost estimate. In addition, Labor's revised cost benefit analysis tool guidance requires that cost estimates be updated regularly so that they are accurate and reflect current status. Further, Labor's revised guidance requires risk analysis and sensitivity analysis to determine the level of risk associated with the cost estimate and requires risk adjusted figures to be included in the estimate which helps ensure that the estimate is credible. However, while the guidance calls for the project manager (PM) to have the estimates independently reviewed by another PM who is experienced with similar projects, the guidance does not require an independent cost estimate conducted by a group outside the acquiring organization. Nevertheless, fully implementing the guidance should help Labor address weaknesses we identified and guide consistent and reliable cost estimates that management can use as a basis for making informed decisions.
|
| Department of Veterans Affairs | To address weaknesses identified in the Department of Veterans Affairs' practices for cost estimating, we recommend that the Secretary of Veterans Affairs direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department of Veterans Affairs (VA) has implemented this recommendation by creating a VA Cost Estimating Guide in response to our report. The VA Guide directs project managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities. VA specifies that the GAO Guide, which includes practices addressing the need to regularly update estimates to reflect actual costs and cost changes; is part of VA intentions to establish requirements and standards for its cost estimating activities. Consequently, the VA Guide, which is intended for use throughout the Department, will guide development of future cost estimations. As a result of this action, responsible VA officials are positioned to update future life-cycle cost estimates of the system acquisition programs discussed in our report using cost-estimating practices that will allow estimates to be more comprehensive, well-documented, accurate and credible; and reduce the likelihood of inaccurate or unreliable cost estimates.
|
| Department of Justice | To address weaknesses identified in the Department of Justice's practices for cost estimating, we recommend that the Attorney General direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
A management analyst in the Department of Justice's Internal Review and Evaluation Office, Justice Management Division, responded orally that the department had no comments on our report. We have contacted Justice to obtain supporting information on its actions to address our recommendation. However, Justice was unable to provide sufficient evidence that the department directed responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in our report using cost-estimating practices that address the detailed weaknesses that we identified. We continue to believe that until Justice fully implement cost-estimating best practices, the department's system acquisition programs face an increased risk that managers will not be able to effectively use their cost estimates as a sound basis for informed program and budget decision making.
|
| Environmental Protection Agency | To address weaknesses identified in the Environmental Protection Agency's practices for cost estimating, we recommend that Administrator of the Environmental Protection Agency direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Environmental Protection Agency (EPA) has implemented this recommendation by directing project managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities; and specifying that the GAO Guide is part of EPA related policy, standards and guidance. The EPA guidance, which incorporates all practices advocated by GAO, is intended for use throughout the department including usage in developing future cost estimations. As a result of this action, responsible EPA officials are positioned to update future life-cycle cost estimates of the system acquisition programs discussed in our report to make them reflect actual cost and cost changes and other relevant cost-estimating practices that will make cost estimates more comprehensive, well-documented, accurate and credible.
|
| Pension Benefit Guaranty Corporation | To address weaknesses identified in the Pension Benefit Guaranty Corporation's practices for cost estimating, we recommend that the Director of the Pension Benefit Guaranty Corporation direct responsible officials to update future life-cycle cost estimates of the system acquisition program discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified. |
The Pension Benefit Guaranty Corporation has addressed our recommendation. As of October 2014, the agency revised its cost estimating practices used when updating the life-cycle cost estimate to address the weaknesses we identified. For example, the program office now tracks IT program data centrally ensuring the formation of a well documented, accurate, and credible cost estimate. As a result, program officials have a sound basis for informed budget making decisions and reducing the likelihood of underfunding and cost overruns.
|
| Department of Defense | To address weaknesses identified in the Department of Defense's practices for cost estimating, we recommend that the Secretary of Defense should direct responsible officials to update future life-cycle cost estimates of the Tactical Mission Command program using cost-estimating practices that address the detailed weaknesses that we identified. |
The Department does not plan to implement this recommendation. The Department agreed with the provisions in the GAO Cost Estimating and Assessment Guide used as criteria and the methodological basis for assessing cost estimating practices of its Tactical Mission Command program. However, our recommendation to update future life-cycle cost estimates is specific to the Tactical Mission Command program and only its future life-cycle cost estimates. This program is in the system deployment phase of its acquisition lifecycle, with segments nearing lifecycle end, being migrated out of the Army Architecture; and the program has no significant life-cycle cost changes anticipated. Consequently, there is no plan to update the Tactical Mission Command life-cycle cost estimate.
|
Full Report
GAO Contacts
Office of Public Affairs
Topics
IT investmentsCost estimatesInformation technologyFederal agenciesBest practicesDecision makingPolicies and proceduresLife cycle costsHomeland securityEnvironmental protection