This is the accessible text file for GAO report number GAO-14-470 entitled 'DOD Business Systems Modernization: Additional Enhancements Are Needed for Army Business System Schedule and Cost Estimates to Fully Meet Best Practices' which was released on October 30, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: September 2014: DOD Business Systems Modernization: Additional Enhancements Are Needed for Army Business System Schedule and Cost Estimates to Fully Meet Best Practices: GAO-14-470: GAO Highlights: Highlights of GAO-14-470, a report to congressional requesters. Why GAO Did This Study: DOD officials have stated that the implementation of enterprise resource planning systems, such as GCSS-Army, is critical to the department's goal of correcting financial management deficiencies and ensuring that its financial statements are validated as audit ready by September 30, 2017, as called for by the National Defense Authorization Act for Fiscal Year 2010. GAO was asked to review the schedule and cost estimates for selected DOD systems. This report addresses the extent to which the schedule and cost estimates for GCSS-Army were prepared consistent with GAO's Schedule and Cost Guides. The schedule and cost estimates are designed to cover GCSS-Army implementation through 2017. GAO assessed the schedule and cost estimates that supported DOD's December 2012 full deployment decision, which granted approval for GCSS-Army to be deployed for operational use to all remaining locations. GAO also met with GCSS-Army program officials, including lead schedulers and cost estimators. What GAO Found: The Army made some improvements to its schedule and cost estimates that supported the December 2012 full deployment decision for the Global Combat Support System-Army (GCSS-Army); however, the schedule and cost estimates did not fully meet best practices. GAO found that the schedule substantially met the credibility and control characteristics for developing a high-quality and reliable schedule. For example, the schedule was horizontally integrated, which means that it links products and outcomes with other associated sequenced activities. In addition, the GCSS-Army program management officials followed general guidelines for updating the schedule on a regular basis. GAO found that the schedule partially met the comprehensiveness and construction characteristics for a reliable schedule. Specifically, resources were not assigned to specific activities, and the schedule lacked a valid critical path, preventing management from focusing on the activities most likely to have detrimental effects on key program milestones if not completed as planned. By incorporating best practices for developing a reliable schedule, the Department of Defense (DOD) would increase the probability of completing the GCSS-Army program by the projected date. Table: Extent to Which Global Combat Support System-Army Schedule Met Best Practices: Characteristic: Comprehensive; Assessment: Partially met. Characteristic: Well-constructed; Assessment: Partially met. Characteristic: Credible; Assessment: Substantially met. Characteristic: Controlled; Assessment: Substantially met. Source: GAO analysis based on information provided by the Army. GAO-14-470. [End of table] GAO found that the GCSS-Army cost estimate fully or substantially met the comprehensiveness, documentation, and accuracy characteristics of a high-quality and reliable cost estimate. For example, the cost estimate included both government and contractor costs for the program over its life cycle, provided documentation that substantially described detailed calculations used to derive each element's cost, and was adjusted for inflation. In addition, GAO found that the cost estimate partially met the credibility characteristic of a reliable cost estimate. Although program management officials provided a cost model that discussed a limited risk analysis, the results of the risk and uncertainty analysis were not documented. Incorporating best practices would help ensure that DOD has a reliable cost estimate that provides the basis for effective resource allocation, proactive course correction when warranted, and accountability for results. Table: Extent to Which Global Combat Support System-Army Cost Estimate Met Best Practices: Characteristic: Comprehensive; Assessment: Fully met. Characteristic: Well-constructed; Assessment: Substantially met. Characteristic: Credible; Assessment: Substantially met. Characteristic: Controlled; Assessment: Partially met. Source: GAO analysis based on information provided by the Army. GAO-14-470. [End of table] What GAO Recommends: GAO is making two recommendations aimed at improving the Army's implementation of schedule and cost best practices for GCSS-Army. DOD concurred, but the completed actions it described related to the cost estimate were not fully responsive to GAO's recommendation. GAO continues to believe that fully incorporating best practices in the cost estimate would help improve its reliability. View [hyperlink, http://www.gao.gov/products/GAO-14-470]. For more information, contact Asif A. Khan at (202) 512-9869 or khana@gao.gov or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contents: Letter: Scope and Methodology: Background: GCSS-Army Program Schedule and Cost Estimates Did Not Fully Meet Best Practices: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Assessment of the Global Combat Support System-Army Program's Schedule Estimate: Appendix II: Assessment of the Global Combat Support System-Army Program's Cost Estimate: Appendix III: Comments from the Department of Defense: Appendix IV: GAO Contacts and Staff Acknowledgments: Tables: Table 1: Extent to Which Global Combat Support System-Army Schedule Met Best Practices: Table 2: Extent to Which Global Combat Support System-Army Cost Estimate Met Best Practices: Table 3: Assessment of Global Combat Support System-Army (GCSS-Army) Schedule Estimate Compared to Best Practices: Table 4: Assessment of Global Combat Support System-Army (GCSS-Army) Cost Estimate Compared to Best Practices: Abbreviations: DASA-CE: Deputy Assistant Secretary of the Army for Cost and Economics: DOD: Department of Defense: ERP: enterprise resource planning: GCSS-Army: Global Combat Support System-Army: IMS: integrated master schedule: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: September 30, 2014: Congressional Requesters: The Department of Defense (DOD) invests billions of dollars annually to develop and implement enterprise resource planning (ERP) systems, which it considers critical to transforming the department's business operations and addressing some of its long-standing weaknesses, including those related to financial management and business systems modernization.[Footnote 1] DOD officials have stated that the implementation of the ERPs, such as the Global Combat Support System- Army (GCSS-Army), is a key component to the department's goal of correcting financial management deficiencies and ensuring that its financial statements are validated as audit ready by September 30, 2017,[Footnote 2] as called for by the National Defense Authorization Act for Fiscal Year 2010.[Footnote 3] DOD's business systems[Footnote 4] modernization program has been on GAO's high-risk list since 1995 because of the size, complexity, and significance of the related efforts.[Footnote 5] In October 2010, we reported that the Army did not fully follow best practices in developing a reliable schedule or cost estimate for GCSS- Army.[Footnote 6] Specifically, the Army had not developed a fully integrated master schedule that reflected all activities, including government and contractor activities. Having such a schedule is crucial to the Army's ability to reliably estimate the program completion date. Also, the Army did not perform a sensitivity analysis for the cost estimate, which would have helped decision makers in determining how changes to assumptions or key cost drivers could affect the credibility of the cost estimate. To support Congress's continuing oversight of DOD's progress in implementing its ERP systems, you asked us to review the schedule and cost estimates for selected DOD ERP systems. The objective of this review was to determine the extent to which the schedule and cost estimates for GCSS-Army were prepared consistent with GAO's Schedule and Cost Guides.[Footnote 7] We reviewed the most current GCSS-Army schedule and cost estimates available at the time of our review, which supported DOD's December 2012 full deployment decision.[Footnote 8] Scope and Methodology: We assessed the GCSS-Army schedule[Footnote 9] that supported DOD's December 2012 full deployment decision using the GAO Schedule Guide to determine whether it was comprehensive, well-constructed, credible, and controlled.[Footnote 10] To assess the schedule, we obtained and reviewed documentation, including the integrated master plan, work breakdown structure, and statement of work.[Footnote 11] To assess the program's cost estimate, we used the GAO Cost Guide to evaluate the GCSS-Army Program Management Office's estimating methodologies, assumptions, and results to determine whether the cost estimate was comprehensive, well-documented, accurate, and credible. We obtained and reviewed documentation, including the program office estimate, software cost model, independent cost estimate, and risk and uncertainty analysis.[Footnote 12] We also met with key program officials, such as the program manager, lead schedulers, and cost estimators to present the preliminary results of our assessment of the program's schedule and cost estimates best practices and obtained explanations and clarifications. We conducted this performance audit from October 2011 to September 2014 in accordance with generally accepted government auditing standards.[Footnote 13] Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: GCSS-Army was initiated in December 2003 and is intended to provide all active Army, National Guard, and Army Reserve tactical units with the capability to track supplies, spare parts, and organizational equipment. The system is also intended to track unit maintenance, total cost of ownership, and other financial transactions related to logistics for all Army units--about 160,000 users. GCSS-Army is intended to integrate approximately 40,000 local supply and logistics databases into a single, enterprise-wide system. In December 2012, the Under Secretary of Defense for Acquisition, Technology and Logistics granted full deployment decision approval for GCSS-Army to be deployed to all remaining locations beyond the limited fielding locations of the life-cycle acquisition process.[Footnote 14] DOD officials reported that the GCSS-Army full deployment will be completed by the fourth quarter of fiscal year 2017. GCSS-Army program functionality is intended to be implemented across the Army in two waves--the first is to include two releases and is to provide supply (warehouse) and financial reporting capabilities, and the second is to include one release, which is to provide property book[Footnote 15] and maintenance capabilities. DOD has approved the funding for the Army to proceed with the deployment of the GCSS-Army functionality to all intended locations. This funding is approximately $3.7 billion. The Army reported that it had spent about $1.6 billion as of June 30, 2014. In October 2010, we reported that the Army did not fully follow best practices in developing a reliable schedule and cost estimate for implementing GCSS-Army.[Footnote 16] In particular, the Army had not developed a fully integrated master schedule that reflected all government and contractor activities and had not performed a sensitivity analysis for the cost estimate. We recommended that the Army develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically,[Footnote 17] establishing the critical path for all activities, and conducting a schedule risk analysis. In addition, we recommended that the Army update the cost estimate by using actual costs and preparing a sensitivity analysis. DOD concurred with our recommendations, and this report provides the status of the department's efforts to address our prior recommendations. GAO's Cost and Schedule Guides: In March 2009, we published the Cost Guide to address a gap in federal guidance about processes, procedures, and practices needed to ensure reliable cost estimates.[Footnote 18] The Cost Guide provides a consistent methodology based on best practices that can be used across the federal government to develop, manage, and evaluate capital program cost estimates. The methodology is a compilation of characteristics and associated best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of an acquisition program. In May 2012, we issued an exposure draft of the Schedule Guide as a companion to the Cost Guide.[Footnote 19] A consistent methodology for developing, managing, and evaluating capital program cost estimates includes the concept of scheduling the necessary work to a timeline, as discussed in the Cost Guide. Simply put, schedule variances are usually followed by cost variances. Because some program costs, such as labor, supervision, rented equipment, and facilities, cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. In addition, management tends to respond to schedule delays by adding more resources or authorizing overtime. Further, a schedule risk analysis allows for program management to account for the cost effects of schedule slippage when developing the life-cycle cost estimate. A cost estimate cannot be considered fully credible if it does not account for the cost effects of schedule slippage. GCSS-Army Program Schedule and Cost Estimates Did Not Fully Meet Best Practices: We found that the program schedule and cost estimates for the GCSS- Army did not fully meet best practices. Specifically, the GCSS-Army schedule supporting the December 2012 full deployment decision partially met the comprehensiveness and construction characteristics and substantially met the credibility and control characteristics for developing a high-quality and reliable schedule. In addition, the cost estimate fully met the comprehensiveness characteristic, substantially met the documentation and accuracy characteristics, and partially met the credibility characteristic for developing a high-quality and reliable cost estimate. It is important that the schedule and cost estimates are continually updated throughout the program's life cycle so that management has the best information available to make decisions. By incorporating best practices for developing reliable schedule and cost estimates, DOD would increase the probability of GCSS-Army successfully achieving full deployment by the fourth quarter of fiscal year 2017 to provide needed functionality for financial improvement and audit readiness. Program Schedule Did Not Fully Meet Best Practices: Our analysis found that the GCSS-Army program substantially met two and partially met the other two characteristics of a reliable schedule estimate and therefore did not provide the information needed to support the December 2012 full deployment decision (see table 1). Appendix I contains our detailed analysis of the GCSS-Army schedule estimate. The success of any program depends on having a reliable schedule of the program's work activities that will occur, how long they will take, and how the activities are related to one another. As such, the schedule not only provides a road map for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability. Table 1: Extent to Which Global Combat Support System-Army Schedule Met Best Practices: Characteristic: Comprehensive; Assessment: Partially met. Characteristic: Well-constructed; Assessment: Partially met. Characteristic: Credible; Assessment: Substantially met. Characteristic: Controlled; Assessment: Substantially met. Source: GAO analysis based on information provided by the Army. GAO-14-470. Note: GAO's methodology includes five levels of compliance with its best practices. "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. [End of table] Comprehensive. A schedule should reflect all activities as defined in the program's work breakdown structure, including activities to be performed by the government and the contractor; the resources (e.g., labor, materials, and overhead) needed to complete each activity; and how long each activity will take. We found that the GCSS-Army schedule partially met the comprehensive characteristic. The schedule used to support the full deployment decision reflected all activities to be performed by both the government and contractor for the program. However, resources were not loaded into the schedule software and were not assigned to specific activities in the schedule. GCSS-Army program management officials told us that the contractor used a separate system outside the schedule to manage the resources needed for the program. Information on resource needs and availability in each work period assists the program office in forecasting the likelihood that activities will be completed as scheduled. If the current schedule does not allow insight into the current or projected allocation of resources, the risk of the program's schedule slipping is significantly increased. Our analysis also determined that activity durations were not manageable and reasonably estimated in the schedule. We found that 30 percent of the remaining activities in the schedule exceeded the standard best practice for activity duration, which should be shorter than approximately 44 working days, or 2 working months. For example, audit support activities had durations over 100 working days. Durations should be as short as possible to facilitate the objective measurement of accomplished effort. If activities are too long, the schedule may not have enough detail for effective progress measurement and reporting. Well-constructed. A schedule should be planned so that critical project dates can be met. To meet this objective, all activities should be logically sequenced--that is, listed in the order in which they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities), as well as activities that cannot begin until other activities are completed (i.e., successor activities), should be identified and their relationships established. The schedule should identify the project's critical path.[Footnote 20] Establishing a valid critical path is necessary for examining the effects of any activity slipping along this path. The calculation of a critical path determines which activities drive the project's earliest completion date. The schedule should also identify total float so that the schedule's flexibility can be accurately determined.[Footnote 21] We found that the GCSS-Army schedule was partially well-constructed. The majority of logic used to sequence the activities within the schedule was generally error free, clearly indicating to program management the order of activities that must be accomplished. However, the schedule's critical path was not valid because it included level of effort activities and date constraints.[Footnote 22] Level of effort activities, such as program management, should not define the critical path because they are nondiscrete support activities that do not produce a definite end product; therefore, level of effort activities cannot determine the length of the project. In addition, date constraints prevent the critical path from being a continuous sequence of events from the current to finish dates of the project. Rather than relying on such constraints, the schedule should use logic and durations in order to reflect realistic start and completion dates for activities. Successfully identifying the critical path relies on several factors, such as capturing all activities; properly sequencing activities; and assigning resources, which, as noted earlier, had not been completely done. Without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliverables if they slip. Further, our analysis found that 28 percent of remaining schedule activities had more than 100 working days of total float, meaning that those activities could slip almost 5 working months and not affect the estimated finish date of the program. Based on the remaining duration of the program, 100 working days of float would not appear to be reasonable. The GCSS-Army Program Management Office stated that total float was not reliable at the time of the full deployment decision because the schedule was being updated to reflect a modification to the system. Without accurate values of total float for a program activity, management cannot determine the flexibility of tasks and therefore cannot properly reallocate resources from tasks that can safely slip to tasks that cannot slip without adversely affecting the estimated program completion date. Credible. A schedule should be horizontally and vertically integrated. A horizontally integrated schedule links products and outcomes with other associated sequenced activities, which helps verify that activities are arranged in the right order to achieve aggregated products or outcomes. A vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. A schedule risk analysis should also be performed using statistical techniques to predict the level of confidence in meeting a program's completion date. We found that the GCSS-Army schedule was substantially credible. The schedule was substantially horizontally integrated, which means that outcomes were aligned with sequenced activities. The schedule was also substantially vertically integrated; we were able to trace varying levels of activities and supporting subactivities. Such mapping or alignment among subsidiary schedules enables different groups--such as government teams and contractors--to work to the same master schedule, and provides assurance that the representation of the schedule to different audiences is consistent and accurate. However, our analysis found that a schedule risk analysis had not been fully conducted. GCSS- Army program management officials provided documentation for a schedule risk analysis, but we noted that risk analyses were not performed for all supporting activities because program management officials stated that the program fielding schedule was not finalized at the time of the full deployment decision. If a schedule risk analysis is not conducted, program management cannot determine (1) the likelihood that the project completion date will occur, (2) how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, (3) risks most likely to delay the project, (4) how much contingency reserve each risk requires, and (5) the activities that are most likely to delay the project.[Footnote 23] Controlled. A schedule should be continually updated using logic, durations, and actual progress to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule time frame. The schedule should be analyzed continually for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that affect activities identified as being in a program's critical path and that can affect a scheduled completion date. A baseline schedule should be used to manage the program scope, the time period for accomplishing it, and the required resources. We found that the GCSS-Army schedule was substantially controlled. GCSS-Army program management officials stated that they met weekly to discuss proposed schedule changes and update the schedule's progress, and management also prepared a schedule narrative document that contained a list of custom fields and assumptions. In addition, we found no anomalies throughout the schedule (e.g., activities with planned start dates scheduled to occur in the past and activities with actual finish dates scheduled to occur in the future). However, we found that there was not a documented baseline schedule to measure program performance against, which would allow management to monitor any schedule variances that affect the completion of work. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance. In our October 2010 report, we recommended that the Army develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically, establishing the critical path for all activities, and conducting a schedule risk analysis.[Footnote 24] The Army's December 2012 GCSS-Army schedule used to support the full deployment decision addressed several of the best practices that were an issue in our prior report, including capturing all activities, sequencing all activities, and integrating activities horizontally and vertically. However, as discussed, we continued to identify several best practices that were not yet fully addressed and also identified several new areas where the 2012 schedule did not incorporate best practices, such as activity durations and baseline schedule. Although GCSS-Army is in full deployment, without fully addressing best practices for scheduling, program managers will not have the best information available to make decisions related to issues such as the sequencing of activities and the flexibility of the schedule according to available resources. Program Cost Estimate Did Not Fully Meet Best Practices: We found that the GCSS-Army program fully met one, substantially met two, and partially met one of the characteristics of a reliable cost estimate and therefore did not provide the information needed to support the full deployment decision, as shown in table 2. Appendix II contains our detailed analysis of the GCSS-Army cost estimate. A reliable cost estimate is critical to the success of any program and is updated continually throughout its life cycle. Such an estimate provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction when warranted, and accountability for results. Table 2: Extent to Which Global Combat Support System-Army Cost Estimate Met Best Practices: Characteristic: Comprehensive; Assessment: Fully met. Characteristic: Well-constructed; Assessment: Substantially met. Characteristic: Credible; Assessment: Substantially met. Characteristic: Controlled; Assessment: Partially met. Source: GAO analysis based on information provided by the Army. GAO-14-470. Note: GAO's methodology includes five levels of compliance with its best practices. "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. [End of table] Comprehensive. A cost estimate should include costs of the program over its full life cycle, provide a level of detail appropriate to ensure that cost elements are neither omitted nor double-counted, and document all cost-influencing ground rules and assumptions. The cost estimate should also completely define the program and be technically reasonable. We found that the cost estimate for GCSS-Army was fully comprehensive. The cost estimate included both government and contractor costs of the program over its life cycle--from the inception of the program through design, development, deployment, and operation and maintenance. The cost estimate also included an appropriate level of detail, which provided assurance that cost elements were neither omitted nor double- counted, and included documentation of all cost-influencing ground rules and assumptions. The cost estimate documentation included the purpose of the cost estimate, a technical description of the program, and technical risks (e.g., the resolution for any identified deficiencies). Well-documented. A cost estimate should be supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. The documentation should capture such things as the source data used, the calculations performed, the results of the calculations, the estimating methodology used to derive each work breakdown structure element's cost, and evidence that the estimate was approved by management. The documentation should discuss the technical baseline description, and the data in the technical baseline should be consistent with the cost estimate. We found that the cost estimate for GCSS-Army was substantially well- documented. The cost estimate captured such things as the calculations performed to derive each element's cost and the results of the calculations. The documentation also included a technical baseline description that provided data consistent with the cost estimate. Further, the GCSS-Army Program Management Office presented evidence of receiving approval of the estimate through briefings to management. Although program management officials did not provide us with written documentation of the source data, the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (DASA-CE) did provide us with a full deployment decision briefing,[Footnote 25] which showed each major cost element and listed the methodology and sources of the data. However, the briefing documents included a limited amount of the actual source data, and we could not determine their reliability. [Footnote 26] Without sufficient background information about the source data and reliability of the data, the GCSS-Army cost estimator cannot know with any confidence whether the data collected can be used directly or need to be modified before use in the cost estimate. Accurate. A cost estimate should provide for results that are unbiased, are not overly conservative or optimistic, and contain no major mistakes. A cost estimate should be based on an assessment of most likely costs (adjusted properly for inflation), updated to reflect significant changes and grounded in a historical record of cost estimating and actual experiences on other comparable programs. In addition, variances between planned and actual costs should be documented, explained, and reviewed, and estimating techniques for each cost element should be used appropriately. We found that the cost estimate for GCSS-Army was substantially accurate. The GCSS-Army cost model detailed the inflation indexes and properly applied the indexes to each relevant cost element and included time phasing of the costs. The GCSS-Army cost model did not include any major mistakes, and all its cost elements summed up properly and were consistent with the cost estimate. In addition, the estimating techniques (i.e., engineering build-up[Footnote 27]) used to create the estimate were used appropriately. The cost model documentation did not explain whether the cost estimate was updated to reflect changes in technical or program assumptions. The program management officials provided documentation that reflected the technical changes for the major deployment decisions, but the documentation did not include details on how the costs were updated. Unless such documentation is available to verify that the cost estimate is properly updated on a regular basis, management will not have reasonable assurance that the cost estimate provides accurate information to make informed decisions about the program. Credible. A cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. The cost estimate should include a sensitivity analysis that identifies a range of possible costs based on varying major assumptions and data. A risk and uncertainty analysis should be conducted to determine the level of risk associated with the cost estimate and identify the effects of changing key cost driver assumptions and factors. In addition, the estimate's results should be cross-checked and reconciled to an independent cost estimate to determine whether other estimating methods produce similar results. We found that the cost estimate was partially credible. The Army Cost Review Board developed an independent cost estimate that was reconciled to the program management officials' cost estimate. The program management officials' cost estimate mentioned results of a risk analysis; however the risk and uncertainty analysis was not documented. Further, since the cost estimate that was provided discussed risk only at a summary level, it is unclear how management considered risk related to the program. Without a fully documented risk and uncertainty analysis, the estimate will lose credibility and management's decision-making ability will be impaired because it will not know the level of confidence associated with achieving the cost estimate. In addition, program management officials provided a cost estimate that identified major cost drivers, including system deployment and training. The cost estimate documentation contained a reference that a sensitivity analysis was completed on these cost drivers, but results of this analysis were not documented. As a result, the GCSS-Army cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which cost driver most affects the cost estimate. Further, GCSS-Army program officials provided us with one example of evidence that indicated that some cross-checking was performed using cost models; however, the results of this cross- checking were not documented. The purpose of cross-checking is to determine whether alternative methods would produce similar results, which would increase the credibility of the estimate. In our October 2010 report, we recommended that the Army update the GCSS-Army cost estimate by using actual costs and preparing a sensitivity analysis.[Footnote 28] For the 2012 cost estimate, we found that the Army had made progress, but we continued to identify deficiencies in documentation related to the sensitivity analysis, risk and uncertainty analysis, and cross-checking of major cost elements for reasonableness. Conclusions: While the Army made some improvements to the schedule and cost estimates that supported the full deployment decision, the Army did not fully meet best practices in developing schedule and cost estimates for the GCSS-Army program. The Army made progress in incorporating schedule best practices, such as capturing and sequencing all activities and integrating activities horizontally and vertically, but we identified other deficiencies in schedule and cost best practices. For example, GCSS-Army did not meet best practices related to schedule durations, a valid critical path, and a cost sensitivity analysis. It is critical to correct the deficiencies identified with the schedule and cost estimates to help ensure that the projected spending for this program is being used in the most efficient and effective manner. By incorporating best practices for developing reliable schedule and cost estimates, DOD would increase the probability of GCSS-Army successfully achieving full deployment by the fourth quarter of fiscal year 2017 to provide needed functionality for financial improvement and audit readiness. Recommendations for Executive Action: To help improve the implementation of GCSS-Army, we recommend that the Secretary of the Army take the following two actions: * Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to develop an updated schedule that fully incorporates best practices, including: - assigning resources to all activities, - establishing durations of all activities, - confirming that the critical path is valid, and: - ensuring reasonable total float. * Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to update the cost estimate to fully incorporate best practices by documenting the results of: - a risk and uncertainty analysis, - the cross-checking of major cost elements to see if results are similar, and: - a sensitivity analysis. Agency Comments and Our Evaluation: We provided a draft of this report to DOD for review and comment. In its written comments, reprinted in appendix III, DOD concurred with our recommendation to update the schedule to fully incorporate best practices and described planned and ongoing actions that the department is taking to address the recommendation. In particular, DOD indicated that the Army has taken steps to help ensure that (1) all activities are assigned resources in the schedule software, (2) all schedule activities with long durations have been detailed, (3) level of effort activities and date constraints have been removed from the schedule so that they do not define the critical path, and (4) the majority of the schedule activities associated with high total float have been removed. If effectively implemented, these actions should address the intent of our recommendation. DOD also concurred with our recommendation to update the cost estimate to fully incorporate best practices by documenting the results of a risk and uncertainty analysis, the cross-checking of major cost elements to see if results are similar, and a sensitivity analysis. DOD described completed actions that the department has taken to address the recommendation. DOD stated that GCSS-Army achieved Milestone C in August 2011 and a full deployment decision in December 2012, and that it prepared a cost estimate per DOD acquisition rules and guidelines. DOD also stated that the Army (1) followed all Army directed best practices and approvals from the Office of the Deputy Assistant Secretary of the Army for Cost and Economics and (2) prepared a sensitivity analysis, a risk analysis, and cross-checked major cost elements for similar results, but that those documented analyses and results were not included in the formal cost estimates as directed by the Army. DOD commented that these documented analyses and results are part of the formal working papers and were provided to GAO in February 2013. However, these actions do not fully address the intent of our recommendation. As stated in our report, we focused on the extent to which GCSS-Army's schedule and cost estimates were prepared consistent with GAO's Schedule and Cost Guides. We reviewed the cost estimate documentation provided by the Army in February 2013 and additional information provided in February 2014 and determined that the documentation did not fully meet best practices for a risk and uncertainty analysis, a sensitivity analysis, and cross-checking of major cost elements for similar results. As stated in our report, GCSS- Army program management officials provided a cost estimate that mentioned the results of a risk and uncertainty analysis, and contained a reference that a sensitivity analysis was completed. Also, GCSS-Army program management officials provided us with one example of evidence that indicated some cross-checking was performed using cost models. However, the results of the risk and uncertainty and sensitivity analyses, as well as the cross-checking were not documented consistent with best practices. As stated in our report, incorporating best practices for a reliable cost estimate would help ensure that DOD has a reliable cost estimate that provides the basis for effective resource allocation, proactive course correction when warranted, and accountability for results. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Secretary of Defense; the Secretary of the Army; the Assistant Secretary of Defense (Acquisition); the Acting Deputy Chief Management Officer; the Under Secretary of Defense (Comptroller); the Under Secretary of the Army, in his capacity as the Chief Management Officer of the Army; and the Program Manager for GCSS-Army. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staffs have any questions about this report, please contact Asif A. Khan at (202) 512-9869 or khana@gao.gov or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix IV. Signed by: Asif A. Khan: Director: Financial Management and Assurance: Signed by: Nabajyoti Barkakati: Chief Technologist: Applied Research and Methods Center for Science, Technology, and Engineering: List of Requesters: The Honorable Thomas R. Carper: Chairman: The Honorable Tom Coburn, M.D. Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Claire McCaskill: Chairman: Subcommittee on Financial and Contracting Oversight: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable John McCain: Ranking Member: Permanent Subcommittee on Investigations: Committee on Homeland Security and Governmental Affairs: United States Senate: [End of section] Appendix I: Assessment of the Global Combat Support System-Army Program's Schedule Estimate: This appendix provides the results of our analysis of the extent to which the Global Combat Support System-Army schedule supporting the December 2012 full deployment decision met the characteristics of a high-quality, reliable schedule. Table 3 provides the detailed results of our analysis. GAO's methodology includes five levels of compliance with its best practices.[Footnote 29] "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. Table 3: Assessment of Global Combat Support System-Army (GCSS-Army) Schedule Estimate Compared to Best Practices: Characteristic: Comprehensive; Best practice: The schedule should reflect all activities as defined in the project's work breakdown structure, which defines in detail the work necessary to accomplish a project's objectives, including activities to be performed by both the government and contractors. The schedule should reflect the resources (e.g., labor, materials, and overhead) needed to do the work, whether those resources will be available when needed, and whether any funding or time constraints exist. The schedule should realistically reflect how long each activity will take, allowing for discrete progress measurement with specific start and finish dates. Assessment: Partially met; Rationale for assessment: The schedule reflected all activities to be performed by both the government and the contractor for the program. To be comprehensive, the program schedule should reflect all efforts necessary to successfully complete the program. However, no resources were loaded in the schedule software or assigned to specific activities in the schedule. Program management officials told us that the contractor managed resources by using a separate system outside the schedule. The amount of available resources affects estimates of work and its duration and resources that will be available for subsequent activities. Information on resource needs and availability in each work period assists the program office in forecasting the likelihood that activities will be completed as scheduled. Because the current schedule did not allow insight into the current or projected allocation of resources, the risk of program slippage could be significantly increased. Several of the activity durations in the schedule did not meet the standard best practice for activity duration (i.e., shorter than approximately 44 working days or 2 working months). Specifically, about 30 percent of the remaining activities were greater than 44 working days. When durations are not based on the effort required to complete an activity, the resources available, resource efficiency, and other factors such as previous experience on similar activities, then there is little confidence in meeting the target deliverable date. Because the estimates of work required and duration for an activity are tied to the availability of resources, the lack of such information could hinder management's ability to compute total labor and equipment hours, calculate total project and per-period cost, resolve resource conflicts, and establish the reasonableness of the plan. Characteristic: Well-constructed; Best practice: The schedule should be planned so that critical project dates can be met. All activities should be logically sequenced--that is, listed in the order in which they are to be carried out--with predecessor and successor logic. Date constraints and lags should be minimized and justified to help ensure that the interdependence of activities that collectively lead to the completion of events or milestones can be established and used to guide work and measure progress. The schedule should identify the program critical path--the path of longest duration through the sequence of activities--to determine the program's earliest completion date. The schedule should identify total float--the amount of time by which a predecessor activity can slip before the delay affects the program's estimated finish date--so that the schedule's flexibility can be accurately determined. As a general rule, activities along the critical path have the least amount of float. Assessment: Partially met; Rationale for assessment: The majority of logic used to sequence the activities within the schedule was generally error free, clearly indicating the order of activities that must be accomplished. The majority of the activities in the schedule exhibited a straightforward sequencing of logic--finish-to-start logic relationship. Our analysis found that the GCSS-Army schedule's critical path was not valid because it included level of effort activities and date constraints. Level of effort activities, such as program management, should not define the critical path because they are nondiscrete support activities that do not produce a definite end product; therefore, level of effort activities cannot determine the length of the project. In addition, date constraints prevent the critical path from being a continuous sequence of events from the current to finish dates. Rather than relying on constraints, the schedule should use logic and durations in order to reflect realistic start and completion dates for activities. Until the schedule can produce a true critical path, the program office will not be able to provide reliable timeline estimates or identify when problems or changes may occur and their effect on downstream work. Also, without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliveries if they slip. Our analysis found that float calculations within the schedule were not reliable because the schedule was being updated to reflect a modification to the system. The schedule could not be updated until the modification was formally approved, and this caused the negative float values in the schedule. Twenty-eight percent of the remaining schedule activities had total float greater than 100 working days, meaning they could slip almost 5 working months and not affect the estimated finish date of the program. In addition, 11 activities had negative float, ranging from -1 to -10 days. Negative float stems from constraining one or more activities or milestones in the network. Negative float indicates critical path effort that may require management action, such as overtime or resequencing of work. The constraint should be examined and justified, and the resulting negative float should be evaluated for reasonableness. Without reasonable float estimates, management may be unable to identify activities that could be permitted to slip and thus release and reallocate resources to activities that require more resources to be completed on time. Characteristic: Credible; Best practice: The schedule should be horizontally integrated, meaning that it should link products and outcomes associated with other sequenced activities. These links are commonly referred to as "handoffs" and serve to verify that activities are arranged in the right order to achieve aggregated products or outcomes. The schedule should also be vertically integrated, meaning that the dates for starting and completing activities in the integrated master schedule should be aligned with the dates for supporting tasks and subtasks. Such mapping or alignment among levels enables different groups to work to the same master schedule. A schedule risk analysis should be performed using statistical techniques to predict the level of confidence in meeting a program's completion date, determine the time contingency needed for a level of confidence, and identify high-priority risks and opportunities. The analysis should focus not only on critical path activities but also on activities near the critical path, since they can affect the program's status. Assessment: Substantially met; Rationale for assessment: The schedule for GCSS-Army appropriately exhibited horizontal and vertical integration. However, the program office had not fully conducted a schedule risk analysis. Although the program office provided documentation for a schedule risk analysis performed by the contractor, the documentation did not show that a schedule risk analysis was performed for all supporting activities. Risk assessments were only provided for 5 of the 11 supporting activities in the schedule. In addition, there was no mention of correlation between activity durations, and no data were provided that would show which schedule activities were given high, medium, or low risk factors. If a schedule risk analysis is not conducted, (1) program management cannot determine the likelihood of meeting the program's completion date, (2) how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, (3) risks most likely to delay the program, (4) how much contingency reserve each risk requires, and (5) the paths or activities that are most likely to delay the program. Unless a statistical simulation is run, calculating the completion date from schedule logic and the most likely duration distributions will tend to underestimate the program's overall critical path duration. Characteristic: Controlled; Best practice: The schedule should be continually updated using logic and durations to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule timeframe. The schedule should be analyzed continually for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that impact activities identified as being on a program's critical path and that can impact a scheduled completion date. A baseline schedule is the basis for managing the program scope, the time period for accomplishing it, and the required resources. The baseline schedule is designated the target schedule and is subject to a configuration management control process against which program performance can be measured, monitored, and reported. A corresponding baseline document explains the overall approach to the program; defines custom fields in the schedule file; details ground rules and assumptions used in developing the schedule; and justifies constraints, lags, long activity durations, and any other unique features of the schedule. Assessment: Substantially met; Rationale for assessment: GCSS-Army program management officials stated that they met weekly to discuss proposed schedule changes and update the schedule's progress, and management officials prepared a schedule narrative document that contained a list of custom fields and assumptions. In addition, we found no anomalies throughout the schedule (e.g., activities with planned start dates scheduled to occur in the past and activities with actual finish dates scheduled to occur in the future). Although there was not a documented original baseline schedule against which program performance could be measured, program officials updated the schedule status weekly, performed trend analysis and met regularly to track progress. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance. A well-documented schedule is essential for validating and defending a baseline schedule, analyzing changes in the program schedule, and identifying the reasons for variances between estimates. Source: GAO analysis based on information provided by the Army. GAO-14-470. [End of table] [End of section] Appendix II: Assessment of the Global Combat Support System-Army Program's Cost Estimate: This appendix provides the results of our analysis of the extent to which the Global Combat Support System-Army cost estimate supporting the December 2012 full deployment decision met the characteristics of a high-quality cost estimate. Table 4 provides the detailed results of our analysis. GAO's methodology includes five levels of compliance with its best practices.[Footnote 30] "Not met" means the program provided no evidence that satisfies any of the criterion. "Minimally met" means the program provided evidence that satisfies a small portion of the criterion. "Partially met" means the program provided evidence that satisfies about half of the criterion. "Substantially met" means the program provided evidence that satisfies a large portion of the criterion. "Fully met" means the program provided evidence that completely satisfies the criterion. Table 4: Assessment of Global Combat Support System-Army (GCSS-Army) Cost Estimate Compared to Best Practices: Characteristic: Comprehensive; Best practice: The cost estimate includes all life-cycle costs. The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable. The cost estimate work breakdown structure is product oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted. The estimate documents all cost-influencing ground rules and assumptions. Assessment: Fully met; Rationale for assessment: The GCSS-Army cost estimate included both government and contractor costs of the program over its life cycle. The documentation provided a description of the purpose of the estimate, a technical description of the program and technical risks (e.g., the resolution for any identified deficiencies). The work breakdown structure outlined the end product and major work to be performed. The program provided supporting documentation that showed the ground rules and assumptions, such as hardware requirements. The estimate was based on a standard cost element structure as stated in the Department of Defense Automated Information Systems Economic Analysis Guide. Characteristic: Well-documented; Best practice: The documentation should capture the source data used, the reliability of the data, and how the data were normalized. The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each work breakdown structure element's cost. The documentation describes step- by-step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. The documentation discusses the technical baseline description and the data in the baseline are consistent with the estimate. The documentation provides evidence that the cost estimate was reviewed and accepted by management. Assessment: Substantially met; Rationale for assessment: The documentation for the cost estimate captured such things as the calculations performed to derive each element's cost and the results of the calculations. The documentation also included a technical baseline description that provided data consistent with the cost estimate. The GCSS-Army Program Management Office presented evidence of receiving approval of the estimate through briefings to management. However, documentation to capture the source data used and the reliability of the data for the cost estimate was limited. The program management officials said that the source data used for the cost estimate are not documented within the GCSS-Army cost model because of size limitations of the software, but that the source data are stored by the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (DASA-CE). The DASA-CE provided us with a full deployment decision briefing given to Army's Cost Review Board Working Group, which showed each major cost element and listed the methodology and sources of the data. However, this briefing document displayed a limited amount of the source data, and we could not determine the reliability of the source data. Without sufficient background information about the source data and their reliability, the GCSS-Army cost estimator cannot know with any confidence whether the data collected can be used directly or need to be modified before use in the cost estimate. Characteristic: Accurate; Best practice: The cost estimate results are unbiased, not overly conservative or optimistic and based on an assessment of most likely costs. The estimate has been adjusted properly for inflation. The estimate contains few, if any, minor mistakes. The cost estimate is regularly updated to reflect significant changes in the program so that it is always reflecting current status. The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Variances between planned and actual costs are documented, explained, and reviewed. Assessment: Substantially met; Rationale for assessment: The GCSS-Army cost model detailed the inflation indexes supporting the estimated costs and did not include any major mistakes. All its cost elements summed up properly and were consistent with the cost estimate. In addition, the estimating techniques used to create the cost estimate were used appropriately. However, the amount of risk and contingency was not documented, so it is unclear whether the cost estimate is unbiased, not overly conservative or optimistic, or based on the assessment of most likely costs. Unless the cost estimate is based on an assessment of the most likely costs and reflects the degree of uncertainty given all of the risks considered, management will not be able to make good program decisions. The cost estimate documentation did not explain whether the cost estimate was updated to reflect changes in technical or program assumptions. The program management officials provided documentation that reflected the technical changes for the major deployment decisions, but the documentation did not include details on how the costs were updated. Unless properly updated on a regular basis, the cost estimate cannot provide management with accurate information for assessing alternative decisions. Certain elements in the cost estimate appeared to be current and based on actual historical information, such as hourly rates and costs per license; but there is a lack of documentation regarding the source of the data that does not allow us to determine the reliability of the data or if they were appropriate for this estimate. The cost estimate does not include any discussion of variances between planned and actual costs, which would enable GCSS-Army cost estimators to assess how well they are estimating program costs and identifying lessons learned. However, program officials told us that they tracked the differences between the funds planned and expended for the major deployment decisions. Without a documented comparison between the current estimate, which is updated with actual cost, and the old estimate, management cannot see how well it is estimating program costs and how the program is changing over time. Characteristic: Credible; Best practice: A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors. Major cost elements were cross-checked to see whether results were similar. An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs. Assessment: Partially met; Rationale for assessment: The Army Cost Review Board developed an independent cost estimate that was reconciled to the program management officials' cost estimate. Program management officials provided a cost model that mentioned results of a risk and uncertainty analysis, but the results of this analysis were not documented. Since the cost estimate discussed risk only at a summary level, it is unclear how management considered risk related to the program. Without a fully documented risk and uncertainty analysis, the estimate will lose credibility and management's decision-making ability will be impaired because it will not know the level of confidence associated with achieving the cost estimate. GCSS-Army program management officials provided us with one example of evidence that showed some cross-checking was performed using cost models; however, the results of this analysis were not documented. The purpose of cross-checking is to determine whether alternative methods would produce similar results, which would increase the cost estimate's credibility. Program management officials provided a cost estimate that identified major cost drivers (e.g., system deployment and training). The cost estimate documentation contained only a reference that a sensitivity analysis was completed on these cost drivers, but results of this analysis were not documented. As a result, the GCSS-Army cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which cost driver most affects the cost estimate. Source: GAO analysis based on information provided by the Army. GAO-14-470. [End of table] [End of section] Appendix III: Comments from the Department of Defense: Assistant Secretary Of Defense: Acquisition: 3015 Defense Pentagon: Washington, DC 20301-3015: September 17, 2014: Mr. Asif A. Khan: Director: Financial Management and Assurance and Applied Research and Methods: U.S. Government Accountability Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Khan: This is the Department of Defense (DoD) response to the GAO Draft Report, GAO-14-470, "DOD Business Systems Modernization: Additional Enhancements Are Needed for Army Business System Schedule and Cost Estimates to Fully Meet Best Practices," dated August 6, 2014 (GAO Code 197112). Detailed comments on the report recommendations are enclosed. Sincerely, Signed by: Katrina McFarland: Assistant Secretary of Defense (Acquisition); Enclosure: As stated: GAO Draft Report Dated August 6, 2014: GAO-14-470 (GAO CODE 197112): “DOD Business Systems Modernization: Additional Enhancements Are Needed For Army Business System Schedule And Cost Estimates To Fully Meet Best Practices” Department Of Defense Comments To The GAO Draft Report: Recommendation 1: To help improve the implementation of GCSS-Army, the Government Accountability Office (GAO) recommends that the Secretary of the Army take the following two actions: Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to develop an updated schedule that fully incorporates best practices, including: * assigning resources to all activities, * establishing durations of all activities, * confirming that the critical path is valid, and, * ensuring reasonable total float. DoD Response: Concur. Assigning resources to all activities: GCSS-Army has worked through the LSI and DCMA to make sure all tasks are resource loaded in the MPM tool the LSI uses. Establishing durations of all activities: Early on in the project some tasks with long durations may have made it through the rolling wave planning window. GCSS-Army has since worked on making sure all scheduled work in the planning window is detailed out. Confirming that the critical path is valid: There have been several steps we have taken to make sure this has occurred. With permission from DCMA we have removed LOE out of the IMS. Therefore, the critical path will only go through measurable work activities. Second, the program exercises options on the contract one at a time. In the current option, the Government PMO knows the milestones that need to occur on time as to not affect the next option. Those are the release milestones. The LSI and Government Program Management Office met and agreed GCSS-Army would have critical paths leading to each release in the current option. When the next option is exercised, we will be able to link to these milestones and see new critical paths through to the next option. GCSS-Army has also removed the date constraints throughout the IMS. Ensuring reasonable total float: The LSI has removed license renewals and maintenance upgrades from the IMS. These tasks were able to slip to the end of the option period because of the type of work, without affecting the critical path. These tasks represented a majority of the tasks with high total float. GCSS-Army still has tasks in IMS that will lead to future work in options yet to be exercised. Those tasks still have high floats. However, when these tasks are late to their baseline, the LSI reports them weekly to the Government PMO so that actions can be taken in case the Government PMO foresees the tasks affecting the exercising of future options. Recommendation 2: To help improve the implementation of GCSS-Army, the Government Accountability Office (GAO) recommends that the Secretary of the Army take the following two actions: Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to update the cost estimate to fully incorporate best practices, by documenting the results of: * a risk and uncertainty analysis, * the cross checking of major cost elements to see if results are similar, and, * a sensitivity analysis. DoD Response: Concur and complete. As part of the GCSS-Army achieving a Milestone C in August 2011 and a Full Deployment Decision in December 2012, it prepared a cost estimate per DoD acquisition rules and guidelines. GCSS-Army followed all Army directed best practices and approvals from the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (ODASA-CE). GCSS-Army did prepare a sensitivity analysis, risk analysis and cross-checked major cost elements for similar results, however those documented analyses and results were not included in the formal cost estimates as directed by the Army. These documented analyses and results are part of the formal working papers of the program and ODASA-CE, and were provided to GAO in February 2013 during the audit process. [End of section] Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Asif A. Khan, (202) 512-9869 or khana@gao.gov: Nabajyoti Barkakati, (202) 512-4499 or barkakatin@gao.gov: Staff Acknowledgments: In addition to the contacts named above, Arkelga Braxton (Assistant Director), Karen Richey (Assistant Director), Beatrice Alff, Tisha Derricotte, Jennifer Echard, Emile Ettedgui, Patrick Frey, and Jason Lee made key contributions to this report. [End of section] Footnotes: [1] An ERP system is an automated system using commercial off-the- shelf software consisting of multiple, integrated functional modules that perform a variety of business-related tasks, such as general ledger accounting, payroll, and supply chain management. [2] For a complete list of the ERP systems that DOD has identified as essential to transforming its business operations, see GAO, DOD Financial Management: Reported Status of Department of Defense's Enterprise Resource Planning Systems, [hyperlink, http://www.gao.gov/products/GAO-12-565R] (Washington, D.C.: Mar. 30, 2012). [3] Pub. L. No. 111-84, div. A, § 1003(a), (b) (Oct. 28, 2009). Further, the National Defense Authorization Act for Fiscal Year 2014, Pub. L. No. 113-66, div. A, § 1003 (Dec. 26, 2013), also mandates a full audit of DOD's fiscal year 2018 financial statements, and that those results be submitted to Congress by March 31, 2019. [4] DOD's business systems are information systems, including financial and nonfinancial systems that support DOD business operations, such as civilian personnel, finance, health, logistics, military personnel, procurement, and transportation. [5] GAO, High-Risk Series: An Update, [hyperlink, http://www.gao.gov/products/GAO-13-283] (Washington, D.C.: February 2013). [6] GAO, DOD Business Transformation: Improved Management Oversight of Business System Modernization Efforts Needed, [hyperlink, http://www.gao.gov/products/GAO-11-53] (Washington, D.C.: Oct. 7, 2010). [7] GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules--Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-12-120G] (Washington, D.C.: May 2012), and GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [8] A full deployment decision is the decision, following completion of operational testing of the program, to scale up production and/or fielding. [9] Although the Army provided us with its integrated master schedule (IMS) for the GCSS-Army program, we determined that it did not meet all of the characteristics of an IMS in accordance with GAO's Schedule Guide. Specifically, an IMS is a document that integrates the planned work by the government, contractor, and other key parties; the resources necessary to accomplish that work; and the associated budget. For purposes of this report, we use "schedule" to refer to the Army's IMS. [10] A schedule is controlled if it is updated periodically by trained schedulers to realistically forecast dates for program activities and compared against a designated baseline schedule to measure, monitor, and report the project's progress. [11] An integrated master plan provides an event-based hierarchy of program events, with each event supported by accomplishments and each accomplishment associated with specific criteria to be satisfied for its completion. The plan is normally part of the contract and is therefore contractually binding. A program's work breakdown structure defines in detail the work necessary to accomplish a program's objectives, including activities to be performed by both the government and contractors. A statement of work accounts for the detailed technical tasks and, when completed, facilitates communication between the customer and supplier on cost, schedule, technical information, and the progress of the work. [12] An independent cost estimate is another estimate based on the same technical information that is used to validate and cross-check the cost estimate, but is prepared by a person or organization that has no stake in the approval of the program. A risk and uncertainty analysis recognizes the potential for error and attempts to quantify it by identifying the effects of changing key cost drivers (e.g., costs associated with personnel or software development). [13] When we initiated this work in October 2011, the U.S. Army Program Management Office was unable to provide us with a current GCSS- Army cost estimate or schedule for our review. We therefore suspended the engagement from May 2012 to February 2013 to give the U.S. Army Program Management Office more time to produce the GCSS-Army cost estimate and schedule. The Program Management Office provided us with these in February 2013. [14] Limited fielding is the deployment of a capability to a limited number of users to test the capability in an operational environment. [15] Property book is a formally designated set of property records used to account for organizational and installation property in a using unit. [16] [hyperlink, http://www.gao.gov/products/GAO-11-53]. [17] A horizontally integrated schedule links products and outcomes with other associated sequenced activities, and a vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. [18] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [19] [hyperlink, http://www.gao.gov/products/GAO-12-120G]. [20] A critical path is the sequence of activities that represents the longest path from the project's start and finish dates. [21] Float is the amount of time by which a predecessor activity can slip before the delay affects the program's estimated finish date. [22] Level of effort activities are typically related to management and other oversight that continue until the detailed work activities they support have been completed. A constraint predefines the start, finish, or both dates of an activity. [23] Contingency represents time reserve held at or above the government program management office for "unknown unknowns" that are outside a contractor's control. In this context, schedule contingency is added to a schedule to allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and for which experience implies likely delays. Schedule reserve, in contrast, is for "known unknowns" that are tied to the contract's scope and managed at the contractor level. [24] [hyperlink, http://www.gao.gov/products/GAO-11-53]. [25] The DASA-CE provides the Army decision makers with cost, performance and economic analysis in the form of expertise, models, data, estimates and analyses at all levels. [26] GCSS-Army: Full Deployment Decision Recommended Army Cost Position Briefing, presented to Stephen Barth, Office of the Deputy Assistant Secretary of the Army for Cost and Economics on October 11, 2012, and GCSS-Army: Army Enterprise Systems Integration Program Independent Cost Estimate Results Briefing, presented to the Cost Review Board Working Group in September 2012. [27] The engineering build-up method develops the cost estimate by rolling up detailed estimates prepared at lower levels of the work breakdown structure. [28] [hyperlink, http://www.gao.gov/products/GAO-11-53]. [29] GAO, GAO Schedule Assessment Guide: Best Practices for Project Schedules--Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-12-120G] (Washington, D.C.: May 2012). [30] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]