DOD Systems Modernization: Planned Investment in the Naval Tactical Command Support System Needs to be Reassessed

In its response to our report, DOD committed to coordinating with the Office of Program Analysis and Evaluation in conducting an analysis of the relationship of the Naval Tactical Command Support System with the Navy Enterprise Resource Planning Program (ERP), and other programs that may provide similar functionality. The Office of the Assistant Secretary of Defense, Networks and Information Integration has since chaired an Overarching Integrated Product Team (OIPT) on July 27, 2006, to review NTCSS. The OIPT decided to continue investment in NTCSS based, in part, on the Navy's analysis regarding the relationship of NTCSS, Navy ERP and other programs. However, a formal analysis was not performed, nor is one planned. According to DOD and Navy officials, one was not performed because the OIPT agreed that continued investment in NTCSS, as the combat support solution afloat and aviation units ashore, was warranted because Navy ERP had no plan to develop an afloat logistics solution. Since the OIPT, the Navy continued to develop its NTCSS Optimized Organizational Maintenance Activity. Until DOD conducts a complete and disciplined analysis of the entire NTCSS program (reviewed and approved by PA&E and the Naval Cost Analysis Division) and provides this analysis to all key stakeholders, the Navy's investment decisions will continue to be made without complete and reliable data.

In its response to our report, DOD stated that, in order to demonstrate the benefits realized by NTCSS, it will develop measures of effectiveness and conduct a post implementation review (PIR) of the deployed NTCSS capabilities and of the Optimized Organizational Maintenance Activity (OOMA). In July 2006, the Office of the Assistant Secretary of Defense (Networks and Information Integration) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. The OIPT decided to continue investment in NTCSS based on analysis that was presented at the meeting. However, program officials told us that the analysis used to support the decision does not comply with cost estimating best practices and relevant OMB cost benefit guidance.

In 2006, agency officials reported that the program milestone decision authority requested the Navy Inspector General review NTCSS program management activities and make recommendations for improvement consistent with current best acquisition practices. However, agency officials told us that during discussions between Navy acquisition and IG officials, it was determined that an outside activity review of NTCSS was not the appropriate method to determine program management effectiveness. Instead, they stated that during a review chaired by the Assistant Secretary of the Navy (Research, Development and Acquisition) in May 2008, program management activities, including earned value management (EVM), requirements development and management, and test management, were deemed to be effective. However, according to the meeting minutes, formal EVM had not been invoked on the program, rather "EVM-like metrics" were used for the OOMA development. Further, the minutes did not cite a basis for deeming the program management activities effective. Agency officials also reported that the NTCSS Central Design Agency (CDA) obtained a full SEI Capability Maturity Model-Integration maturity level 3 rating on several processes, including requirements development and management, and test management. While the CDA received a level 3 rating in requirements management and aspects of test management, it was not appraised for EVM. Agency officials stated that no further actions are planned to address the recommendation.

In July 2006, the Office of the Assistant Secretary of Defense (Networks and Information Integration) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. The OIPT decided to continue investment in NTCSS. According to agency officials, a post implementation review (PIR) was performed on the deployed system, which showed that the central design agency and the developmental test organization had adequate people, processes and tools to effectively execute their respective roles and responsibilities. However, while the scope of the PIR was to demonstrate benefits, the PIR results do not report on the adequacy of people, processes, and tools. The Central Design Agency, however, did receive a CMMI level 3 certification demonstrating that it has the people, processes and tools to effectively execute its roles and responsibilities in the several areas for which it was assessed. Agency officials, nevertheless, did not provide evidence that other key stakeholders, such as the developmental testing organization, have the people, processes, and tools to effectively execute their roles and responsibilities and stated that no further actions are planned to address the recommendation.

In July 2006, the Office of the Assistant Secretary of Defense for Networks and Information Integration(ASD(NII/CIO)) held an Overarching Integrated Product Team (OIPT) meeting to review NTCSS. At the end of the meeting, the OIPT decided not to reestablish the ASD(NII/CIO) as the milestone decision authority but rather, accepted the Navy's recommendation that the milestone decision authority be retained with the Deputy Assistant Secretary of the Navy - C4I and Space.