DOD Business Systems Modernization:

Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas

GAO-08-896: Published: Sep 8, 2008. Publicly Released: Sep 8, 2008.

Additional Materials:

Contact:

Valerie C. Melvin
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Defense (DOD) has long been challenged in implementing key information technology (IT) management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk. One of the larger business system investments is the Department of the Navy's Enterprise Resource Planning (Navy ERP) program. Initiated in 2003, the program is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its useful life and be fully deployed in fiscal year 2013. GAO was asked to determine whether key IT management controls are being implemented on the program. To do this, GAO analyzed, for example, requirements management, economic justification, earned value management, and risk management.

DOD has implemented key IT management controls on its Navy ERP program to varying degrees of effectiveness. To its credit, the control associated with managing system requirements is being effectively implemented. In addition, important aspects of other controls have at least been partially implemented, including those associated with economically justifying investment in the program and proactively managing program risks. Nevertheless, other aspects of these controls, as well as the bulk of what is needed to effectively implement earned value management, which is a recognized means for measuring program progress, have not been effectively implemented. Among other things, these control weaknesses have contributed to the more than 2-year schedule delay and the almost $600 million cost overruns already experienced on the program since it began, and they will likely contribute to future delays and overruns if they are not corrected. Examples of the weaknesses are: (1) Investment in the program has been economically justified on the basis of expected benefits that far exceed estimated costs ($8.6 billion versus $2.4 billion over a 20-year life cycle). However, important estimating practices, such as using historical cost data from comparable programs and basing the cost estimate on a reliable schedule baseline were not employed. While these weaknesses are important because they limit the reliability of the estimates, GAO's analysis shows that they would not have altered the estimates to the point of not producing a positive return on investment. (2) Earned value management has not been effectively implemented. To its credit, the program office has elected to implement program-level earned value management. In doing so, however, basic prerequisites for effectively managing earned value have not been executed. In particular, the integrated master schedule was not derived in accordance with key estimating practices, and an integrated baseline review has not been performed on any of the first increment's releases. (3) A defined process for proactively avoiding problems, referred to as risk management, has been established, but risk mitigation strategies have not been effectively implemented for all significant risks, such as those associated with data conversion and organizational change management, as well the risks associated with the above-cited weaknesses. The reasons that program management and oversight officials cited for these practices not being executed range from the complexity and challenges of managing and implementing a program of this size to limitations in the program office's scope and authority. Notwithstanding the effectiveness with which important aspects of several controls have been implemented, the above-cited weaknesses put DOD at risk of investing in a system solution that does not optimally support corporate mission needs and mission performance, and meet schedule and cost commitments.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: DOD has taken actions that address the intent of our recommendation. First, the program began using metrics to track and logically link most of the program activities. Second, the schedule now accounts for resources at the lowest level of detailed activity, and the department reported that, through mitigation planning, it ensured that resources were allocated to the critical activities. Third, the department has replaced its release development schedules with an integrated master schedule for the remaining deployment and sustainment activities. Fourth, the department reported, in July 2010, that its actions to ensure the schedule allocates reserve for high-risk activities include mitigation planning through a trade-off process that ensures resources are allocated to critical activities. With regards to the fifth element, Navy's Center for Earned Value Management stated that because development tasks were complete as of March 2010, and the remaining tasks are focused on deployment and sustainment, additional schedule risk analyses were not necessary. Collectively, the actions taken should provide the department with the means by which to estimate costs, gauge progress, identify and address potential problems, and promote accountability.

    Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to increase the quality of the program's integrated master schedule, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the schedule (1) includes the logical sequencing of all activities, (2) reflects whether all required resources will be available when needed, (3) defines a critical path that integrates all three releases, (4) allocates reserve for the high-risk activities on the entire program's critical path, and (5) incorporates the results of a schedule risk analysis for all three releases and recalculates program cost and schedule variances to more accurately determine a most likely cost and schedule overrun.

    Agency Affected: Department of Defense

  2. Status: Closed - Not Implemented

    Comments: The department has taken actions to address the first element of the recommendation. Specifically, to its credit, in August 2008, the program office conducted an integrated baseline review of its second release, Release 1.1, which resulted in recommendations to mature and implement EVM processes. (Note: the third release, Release 1.2, is no longer part of the program.) However, it has yet to address the remaining two elements associated with verifying that the program is in compliance with the 32 accepted industry EVM practices and having an independent organization perform surveillance of the program's EVM system. In July 2011, the Department of the Navy's Center for Earned Value Management stated that because the program completed the development phase in March 2010 and the remaining work has been focused on deployment and sustainment, additional surveillance reviews, which were planned to include review of compliance against the 32 EVM practices, were not necessary. However, as we reported, according to DOD policy and guidance, independent surveillance of EVM should occur over the life of the program to guarantee the validity of the performance data and ensure that EVM is being used effectively to manage cost, schedule and technical performance.

    Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to enhance Navy ERP's use of earned value management (EVM), the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) an integrated baseline review on the last two releases of the first increment is conducted, (2) compliance against the 32 accepted industry EVM practices is verified, and (3) a plan to have an independent organization perform surveillance of the program's EVM system is developed and implemented.

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: The department has taken actions to address the intent of our recommendation. Specifically, the July 2011 economic analysis for the program's full deployment decision applied uncertainty analyses in estimating benefits, including those benefits associated with legacy system retirements. Further, the economic analysis used a risk-adjusted life-cycle cost estimate. In addition, the department reported that its cost estimate used over 5 years of Navy ERP historical data, including detailed cost information for deployment to four system commands, and over 3 years of sustainment cost data.

    Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to improve the reliability of Navy ERP benefit estimates and cost estimates, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that future Navy ERP estimates include uncertainty analyses of estimated benefits, reflect the risks associated with not having cost data for comparable ERP programs, and are otherwise derived in full accordance with the other key estimating practices, and economic analysis practices discussed in this report.

    Agency Affected: Department of Defense

  4. Status: Closed - Implemented

    Comments: The department has taken actions to address the intent of this recommendation. First, the program office reevaluated its plans for mitigating risks associated with data conversion and adopting new business processes. Second, the program manager and System Command officials report monthly to the Navy Enterprise Resource Planning (ERP) Systems Integration Board (NESIB) on performance, and periodically brief oversight and approval authorities on the implementation of risk mitigation plans. Third, the NESIB requires actionable reporting on performance by the program manager and System Command officials, and the program manager is to report to the Milestone Decision Authority on implementation of risk mitigation strategies. Fourth, the program's risk inventory has been updated to include risks related to adopting new business processes and data conversion.

    Recommendation: To strengthen Navy ERP management control and better provide for the program's success, and to improve Navy ERP's management of program risks, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) the plans for mitigating the risks associated with converting data from legacy systems to Navy ERP and positioning the commands for adopting the new business processes embedded in the Navy ERP are re-evaluated in light of the recent experience with Naval Air Systems Command (NAVAIR) and adjusted accordingly, (2) the status and results of these and other mitigation plans' implementation are periodically reported to program oversight and approval authorities, (3) these authorities ensure that those entities responsible for implementing these strategies are held accountable for doing so, and (4) each of the risks discussed in this report are included in the program's inventory of active risks and managed accordingly.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Oct 16, 2014

Oct 10, 2014

Oct 8, 2014

Sep 30, 2014

Sep 25, 2014

Sep 23, 2014

Sep 19, 2014

Sep 18, 2014

Sep 10, 2014

Sep 9, 2014

Looking for more? Browse all our products here