Army Networks: Opportunities Exist to Better Utilize Results from Network Integration Evaluations
Highlights
What GAO Found
Since 2011, the Army has conducted five Network Integration Evaluations (NIE), which have provided extensive information and insights into current network capabilities and potential solutions to fill network capability gaps. According to senior Department of Defense (DOD) test officials, the NIE objective to test and evaluate network components together in a combined event is sound, as is the opportunity to reduce overall test and evaluation costs by combining test events. Further, the NIEs offer the opportunity for a more comprehensive evaluation of the broader network instead of piecemeal evaluation of individual network components. However, the Army is not taking full advantage of the potential knowledge that could be gained from the NIEs, and some resulting Army decisions are at odds with knowledge accumulated during the NIEs. For example, despite poor results in developmental testing, the Army moved forward to operational testing for several systems during the NIEs and they demonstrated similarly poor results. Yet the Army plans to buy and field several of these systems. Doing so increases the risk of poor performance in the field and the need to correct and modify deployed equipment. On the other hand, the Army has evaluated many emerging network capabilities--with generally favorable results--but has bought very few of them, in large part because it did not have a strategy to buy these promising technologies. Army officials have stated that the success of network modernization depends heavily on industry involvement but, with few purchases, it is unclear whether industry will remain interested. Finally, the Army has not yet developed metrics to determine how network performance has improved over time, as GAO recommended in an earlier report.
The Army has several actions under way or planned to enhance the NIE process and has further opportunities to collaborate with the test community. The Army has identified issues in the NIE process and its network modernization strategy that were causing inefficiencies or less-than-optimal results and has begun implementing actions to mitigate some of those issues. For example, the Army has begun performing technology evaluations, and integration of vendor systems in a lab environment to weed out immature systems before they get to the NIE. The Army has also developed a strategy and has an acquisition plan to address requirements, funding, and competition issues that will help enable it to buy emerging capabilities rapidly. However, the Army will need to validate the new strategy and plan and provide results to industry, which could help to manage industry expectations about how many of and how quickly it can buy these capabilities. DOD has started to identify and evaluate network metrics and to re-focus NIEs to gather additional data and insights. Taking these actions will ultimately allow the periodic review and evaluation of the actual effectiveness of network capabilities and the likely effectiveness of proposed investments. The test community has worked closely with the Army on the NIEs but has also voiced various concerns about the NIEs including their being a schedule-driven event. Tension between the acquisition and test communities has been long-standing. Additional opportunities exist for Army leadership and the test community to work together to further improve NIE execution and results and to reduce tensions between the two communities. A good starting point for the Army would be to take a fresh look at the test community observations and recommendations from previous NIEs.
Why GAO Did This Study
In 2011, the Army began a major undertaking to modernize its tactical network to improve communication and provide needed information to soldiers on the battlefield. The Army has identified the network as its number one modernization priority requiring approximately $3 billion per year indefinitely. NIEs provide semi-annual assessments of newly developed systems. Given the importance of the network, GAO was asked to examine elements of the process the Army is using to acquire network capabilities. This report examines (1) the results of the NIEs conducted to date and the extent to which the Army has procured and fielded network solutions, and (2) Army actions to enhance the NIE process. To conduct this work, GAO analyzed key documents, observed testing activities, and interviewed acquisition and testing officials.
Recommendations
To improve outcomes for the Armys network modernization strategy, GAO recommends that the Secretary of Defense direct the Army to (1) require successful developmental testing before moving to operational testing at an NIE, (2) correct issues identified during testing at NIEs prior to buying and fielding systems, (3) provide results to industry on Armys efforts to rapidly acquire emerging capabilities, and (4) pursue additional opportunities for collaboration with the test community on the NIEs. DOD agreed with the recommendations to varying degrees, but generally did not offer specific actions to address them. GAO believes all recommendations remain valid.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | To improve outcomes for its entire network modernization strategy, the Secretary of Defense should direct the Secretary of the Army to require that network systems from major defense acquisition programs obtain a positive Assessment of Operational Test Readiness (now called a Developmental Test and Evaluation Assessment) recommendation before being scheduled for operational testing during the NIE. |
In providing comments on this report, DOD partially concurred with the recommendation. The Army later stated that, as of September 11, 2017, a positive Developmental Test and Evaluation assessment is not required prior to operational testing of major defense acquisition programs involving Army networks. The Army stated that while a positive assessment is not required, an independent government developmental test report that formally certifies that the system has met maturity and performance criteria to proceed to the operational test is prepared.
|
| Department of Defense | To improve outcomes for its entire network modernization strategy, the Secretary of Defense should direct the Secretary of the Army to correct network system performance and reliability issues identified during the NIEs before moving to buy and field these systems. |
In providing comments on this report, DOD partially concurred with the recommendation. In 2017 the Army stated that several NIE events subsequent to our 2013 report have provided information used to assess and correct issues prior to procurement or fielding of systems. For example, tests of the WIN-T Increment 2 led to changes in user interfaces and software that simplified system operations and provided more robust protection against unplanned network shutdowns among other improvements. These improvements were validated at a second NIE and supported the program's full rate production decision in June 2015. As a result of NIE 16.2, the Army is fielding command post computing environment hardware, but delayed fielding of related software applications and architecture due to problems encountered in testing. Instead the Army opted to field a version more reflective of NIE results. The Army also refined the full mission weight and battery life requirements for the Manpack Radio. These revised requirements were directly influenced by the NIE.
|
| Department of Defense | To improve outcomes for its entire network modernization strategy, the Secretary of Defense should direct the Secretary of the Army to provide results to industry on the Army's actual experience in buying and fielding successfully demonstrated systems under evaluation and the length of time it has taken to date. |
In providing comments on this report, DOD concurred with the recommendation, stating that unless prohibited by regulatory and/or statutory requirements results can be provided to industry. Subsequent to our report, DOD stated that the Army communicates with industry via direct contact, industry days, newsletters, social media, and websites. The Army provides reports to industry participants within 60 days following each NIE, including a full assessment of system performance. The Army also provides individual letters to industry notifying them of the Army's disposition decisions based on NIE analysis and Army priorities.
|
| Department of Defense | To improve outcomes for its entire network modernization strategy, the Secretary of Defense should direct the Secretary of the Army to collaborate with all network stakeholder organizations to identify and correct issues that may result in improved network outcomes, including addressing the observations and recommendations of the test community related to the NIEs. |
In providing comments on this report, DOD concurred with the recommendation. Subsequent to our report, the Army reported in 2017 that it has leveraged test and evaluation community observations and recommendations in defining NIE requirements and metrics, as well as in improving the conditions and scenarios for operational tests. These efforts at additional operational realism are reflected in the significant reduction of contractor field service representatives (FSR). The Army has also improved data collection efforts at NIEs, in accordance with DOT&E recommendations. While, DOT&E's annual reports on NIEs include language complimentary of the Army's efforts to improve realism in operational scenarios and threat operations, they have objected to some changes the Army has made in conducting exercises. This includes the replacement of dedicated test units with new formations for each event and a need for further improvements to instrumentation and data collection procedures at test ranges.
|