DOD Systems Modernization:

Planned Investment in the Naval Tactical Command Support System Needs to be Reassessed

GAO-06-215: Published: Dec 5, 2005. Publicly Released: Dec 5, 2005.

Additional Materials:

Contact:

Randolph C. Hite
(202) 512-6256
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Because it is important that the Department of Defense (DOD) adheres to disciplined information technology (IT) acquisition processes to successfully modernize its business systems, GAO was asked to determine whether the Naval Tactical Command Support System (NTCSS) is being managed according to important aspects of DOD's acquisition policies and guidance, as well as other relevant acquisition management best practices. NTCSS was started in 1995 to help Navy personnel effectively manage ship, submarine, and aircraft support activities. To date, about $1 billion has been spent to partially deploy NTCSS to about one-half its intended ashore and afloat sites.

The Department of the Navy has not managed its NTCSS program in accordance with key aspects of the department's policies and related guidance, including federal and recognized best practice guidance. Collectively, these policies and guidance are intended to reasonably ensure that investment in a given IT system represents the right solution to fill a mission need and, if it is, that acquisition and deployment of the system are handled in a manner that maximizes the chances of delivering defined system capabilities on time and within budget. In the case of NTCSS, neither of these outcomes is being realized. The Navy has not economically justified its ongoing and planned investment in NTCSS. Specifically, it (1) has not reliably estimated future costs and benefits and (2) has not ensured that independent reviews of its economic justification were performed to determine its reliability. The Navy has not invested in NTCSS within the context of a well-defined DOD or Navy enterprise architecture, which is necessary to guide and constrain NTCSS in a way that promotes interoperability and reduces redundancy with related and dependent systems. The Navy has not effectively performed key measurement, reporting, budgeting, and oversight activities. In particular, earned value management, which is a means for determining and disclosing actual performance against budget and schedule estimates, has not been implemented effectively, and oversight entities have not had the visibility into the program needed to affect its direction. The Navy has not adequately conducted requirements management and testing activities. For example, requirements were neither prioritized nor traced to related documentation to ensure that the system delivers capabilities that meet user needs. This contributed to failures in developmental testing that have prevented the latest component of NTCSS from passing operational testing twice over the last 4 years. Reasons the Navy cited for not following policies and guidance ranged from their not being applicable to the NTCSS program, to lack of time available to apply them, to plans for strengthening system practices not being applied retroactively. Nevertheless, the Navy has begun taking steps and is considering other steps intended to address some of the above problems. Until program management improves, NTCSS will remain a risky program.

Recommendations for Executive Action

  1. Status: Closed - Not Implemented

    Comments: In July 2006, the Office of the Assistant Secretary of Defense (Networks and Information Integration) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. The OIPT decided to continue investment in NTCSS. According to agency officials, a post implementation review (PIR) was performed on the deployed system, which showed that the central design agency and the developmental test organization had adequate people, processes and tools to effectively execute their respective roles and responsibilities. However, while the scope of the PIR was to demonstrate benefits, the PIR results do not report on the adequacy of people, processes, and tools. The Central Design Agency, however, did receive a CMMI level 3 certification demonstrating that it has the people, processes and tools to effectively execute its roles and responsibilities in the several areas for which it was assessed. Agency officials, nevertheless, did not provide evidence that other key stakeholders, such as the developmental testing organization, have the people, processes, and tools to effectively execute their roles and responsibilities and stated that no further actions are planned to address the recommendation.

    Recommendation: If--based on reliable data--a decision is made to continue the NTCSS program, the Secretary of Defense should direct the Secretary of the Navy to ensure that key stakeholders, such as the central design agency and the developmental testing organization, have the people, processes, and tools to effectively execute their respective roles and responsibilities.

    Agency Affected: Department of Defense

  2. Status: Closed - Not Implemented

    Comments: In 2006, agency officials reported that the program milestone decision authority requested the Navy Inspector General review NTCSS program management activities and make recommendations for improvement consistent with current best acquisition practices. However, agency officials told us that during discussions between Navy acquisition and IG officials, it was determined that an outside activity review of NTCSS was not the appropriate method to determine program management effectiveness. Instead, they stated that during a review chaired by the Assistant Secretary of the Navy (Research, Development and Acquisition) in May 2008, program management activities, including earned value management (EVM), requirements development and management, and test management, were deemed to be effective. However, according to the meeting minutes, formal EVM had not been invoked on the program, rather "EVM-like metrics" were used for the OOMA development. Further, the minutes did not cite a basis for deeming the program management activities effective. Agency officials also reported that the NTCSS Central Design Agency (CDA) obtained a full SEI Capability Maturity Model-Integration maturity level 3 rating on several processes, including requirements development and management, and test management. While the CDA received a level 3 rating in requirements management and aspects of test management, it was not appraised for EVM. Agency officials stated that no further actions are planned to address the recommendation.

    Recommendation: If--based on reliable data--a decision is made to continue the NTCSS program, the Secretary of Defense should direct the Secretary of the Navy to ensure that the NTCSS program implements effective program management activities, including earned value management, requirements development and management, and test management.

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: According to DOD, the Office of the Assistant Secretary of Defense, Networks and Information Integration, chaired an Overarching Integrated Product Team (OIPT) on July 27, 2006, to reach a departmental decision on appropriate changes to the program. According to program officials, the OIPT decided that the NTCSS program was executing well, was needed, and was making progress. Therefore, the program was allowed to proceed.

    Recommendation: In addition, the Secretary of Defense should direct the Secretary of the Navy to present the results of these analyses to the Deputy Secretary of Defense, or his designee, and seek a departmental decision on how best to proceed with the program. Until this is done, the Secretary of Defense should direct the Secretary of the Navy to halt further deployment of NTCSS and to limit future investment in already deployed applications to essential operation and maintenance activities and only developmental activities deemed essential to national security needs.

    Agency Affected: Department of Defense

  4. Status: Closed - Implemented

    Comments: In its response to our report, DOD stated that it would continue to work to ensure that NTCSS is properly aligned to the DOD and Navy business architectures. In this regard, NTCSS was subsequently certified by the Defense Business Systems Management Committee as, among other things, being aligned to DOD's business enterprise architecture (BEA). This certification was based on the program office's certification, and as we reported in GAO-08-972, the Navy had performed its BEA compliance assessments in accordance with DOD guidance and using the DOD compliance tool. While we also reported that the Navy's compliance assessments did not, for example, include all relevant architecture products and address compliance with the Department of Navy's enterprise architecture, the Navy's actions for NTCSS have nevertheless largely met the intent of our recommendation.

    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to determine if continued investment in NTCSS, as planned, represents a prudent use of the department's limited resources. To accomplish this, the Secretary of the Navy should direct the program office to collaborate with the Undersecretary of Defense for Acquisition, Technology, and Logistics and the Under Secretary of Defense (Comptroller) to ensure that NTCSS is adequately aligned with evolving DOD and Navy enterprise architectures.

    Agency Affected: Department of Defense

  5. Status: Closed - Not Implemented

    Comments: In its response to our report, DOD stated that, in order to demonstrate the benefits realized by NTCSS, it will develop measures of effectiveness and conduct a post implementation review (PIR) of the deployed NTCSS capabilities and of the Optimized Organizational Maintenance Activity (OOMA). In July 2006, the Office of the Assistant Secretary of Defense (Networks and Information Integration) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. The OIPT decided to continue investment in NTCSS based on analysis that was presented at the meeting. However, program officials told us that the analysis used to support the decision does not comply with cost estimating best practices and relevant OMB cost benefit guidance.

    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to determine if continued investment in NTCSS, as planned, represents a prudent use of the department's limited resources. To accomplish this, the Secretary of the Navy should direct the program office to ensure that development of this economic analysis (1) complies with cost estimating best practices, including recognition of costs to resolve open trouble reports and change proposals, and relevant OMB cost benefit guidance and (2) incorporates available data on whether deployed NTCSS capabilities are actually producing benefits.

    Agency Affected: Department of Defense

  6. Status: Closed - Not Implemented

    Comments: In its response to our report, DOD committed to coordinating with the Office of Program Analysis and Evaluation in conducting an analysis of the relationship of the Naval Tactical Command Support System with the Navy Enterprise Resource Planning Program (ERP), and other programs that may provide similar functionality. The Office of the Assistant Secretary of Defense, Networks and Information Integration has since chaired an Overarching Integrated Product Team (OIPT) on July 27, 2006, to review NTCSS. The OIPT decided to continue investment in NTCSS based, in part, on the Navy's analysis regarding the relationship of NTCSS, Navy ERP and other programs. However, a formal analysis was not performed, nor is one planned. According to DOD and Navy officials, one was not performed because the OIPT agreed that continued investment in NTCSS, as the combat support solution afloat and aviation units ashore, was warranted because Navy ERP had no plan to develop an afloat logistics solution. Since the OIPT, the Navy continued to develop its NTCSS Optimized Organizational Maintenance Activity. Until DOD conducts a complete and disciplined analysis of the entire NTCSS program (reviewed and approved by PA&E and the Naval Cost Analysis Division) and provides this analysis to all key stakeholders, the Navy's investment decisions will continue to be made without complete and reliable data.

    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to determine if continued investment in NTCSS, as planned, represents a prudent use of the department's limited resources. To accomplish this, the Secretary of the Navy should direct the program office to collaborate with the Office of the Assistant Secretary of Defense for Networks and Information Integration/Chief Information Officer, the Office of Program Analysis and Evaluation, and the Naval Cost Analysis Division to prepare a reliable economic analysis that encompasses all viable alternatives, including the Navy's recent enterprise resource planning program.

    Agency Affected: Department of Defense

  7. Status: Closed - Not Implemented

    Comments: In July 2006, the Office of the Assistant Secretary of Defense for Networks and Information Integration(ASD(NII/CIO)) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. At the end of the meeting, the OIPT decided not to reestablish the ASD(NII/CIO) as the milestone decision authority but rather, accepted the Navy's recommendation that the milestone decision authority be retained with the Deputy Assistant Secretary of the Navy - C4I and Space.

    Recommendation: The Secretary of Defense should reestablish the Office of the Assistant Secretary of Defense for Networks and Information Integration/Chief Information Officer as the milestone decision authority and direct the Secretary of the Navy to take steps to ensure that Navy oversight entities fulfill their roles and responsibilities on NTCSS, including ensuring that reliable program reporting occurs and is acted upon.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Dec 19, 2014

Dec 16, 2014

Dec 12, 2014

Dec 11, 2014

Dec 1, 2014

Nov 21, 2014

Nov 20, 2014

Nov 19, 2014

Nov 18, 2014

Looking for more? Browse all our products here