Weapon System Requirements: Joint Staff Lacks Reliable Data on the Effectiveness of Its Revised Joint Approval Process
Fast Facts
DOD has faced challenges acquiring new weapon systems—such as aircraft, ships, and satellites—on time and on budget.
In 2018, DOD revised its process for reviewing and approving "capability documents," which identify new or enhanced capabilities in weapon systems.
But DOD's information system for tracking these documents doesn't have reliable data about how many programs have gone through the revised process, or how long it takes to review documents.
We recommended ensuring this system collects data that could help DOD evaluate its capability document process.
Highlights
What GAO Found
The Department of Defense (DOD) sets the foundation of its weapon system acquisitions in documented requirements for new or enhanced capabilities. DOD's Joint Staff uses the Joint Capabilities Integration and Development System (JCIDS) process to manage the review and approval of capability requirements documents. The Joint Requirements Oversight Council (JROC) oversees the process. At congressional direction, the Joint Staff revised the process in November 2018, reducing the JROC's role to focus on documents addressing requirements of multiple departments, while increasing the role of military departments for their unique capability documents.
GAO found that the Joint Staff lacks reliable data on the total number of programs that have completed the revised process. In addition, GAO found that Joint Staff data for the time to validate selected capability documents were also unreliable. Capability documents move through the JCIDS process in the Joint Staff's Knowledge Management and Decision Support (KM/DS) information system. GAO found discrepancies between KM/DS data and data from those that submit documents, known as sponsors. Joint Staff officials stated that deficiencies with the KM/DS system are at the root of its data issues. A detailed plan addressing these deficiencies will better position the Joint Staff to assess if the revised process is achieving stated JCIDS objectives. See figure below.
The Joint Staff cannot assess the JCIDS process because it lacks reliable data and a baseline to measure timeliness. Joint Staff guidance provides a notional length of time of 103 days to review documents in the JCIDS process, but this is not evidence-based. Joint Staff officials stated they have not measured the actual length of time that documents take to go through the JCIDS process. GAO analysis and sponsor officials confirmed that none of the selected capability documents completed the process within 103 days. Sponsor officials noted that certain issues can add time to the review process and emphasized document quality over fast review and approval. However, without a data-driven baseline that reflects issues that affect the length of time to validate capability documents, Joint Staff officials are not able to assess JCIDS' efficiency and effectiveness.
Discrepancies between Joint Staff and Sponsor Validation Timeline Data
Note: One selected program is not included in the figure because the sponsor withdrew it from the process.
Why GAO Did This Study
In the National Defense Authorization Act for Fiscal Year 2017, Congress mandated revisions to the JCIDS process by modifying the scope of the JROC's responsibilities. The accompanying Senate Armed Services Committee report noted that these changes were, in part, to improve the timeliness of the JCIDS process.
House Armed Services Committee report 116-120 included a provision for GAO to review the revisions to the JCIDS process. This report examines (1) key revisions to the process, (2) how many programs have been through the revised process and how long it took, and (3) the Joint Staff's ability to assess the timeliness of the process.
GAO reviewed JCIDS policies and guidance, and interviewed relevant DOD officials. GAO also selected a nongeneralizable sample of 12 capability documents from across the Air Force, Army, and Navy. GAO analyzed data associated with these documents from the Joint Staff's KM/DS information system and compared it to data provided by military department officials to determine the Joint Staff's ability to assess the timeliness of the document review process.
Recommendations
GAO recommends that DOD (1) develop a plan for resolving the JCIDS information system operational deficiencies, and (2) establish a baseline based on reliable data and issues that affect the length of time to validate capability documents. DOD concurred with the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | The Secretary of Defense should ensure that the Vice Chairman of the Joint Chiefs of Staff develop a plan for resolving the deficiencies of the information system that facilitates the operation of the Joint Capabilities Integration and Development System (JCIDS), to include updating guidance for (a) assessing the system's effectiveness in providing reliable system data, (b) accounting for programs that have entered and exited the process, and (c) collecting and analyzing JCIDS' performance and timeliness metrics. (Recommendation 1) |
DOD concurred with our recommendation. In July 2024, Joint Staff officials provided us with an update to their corrective action plan to address the recommendation. As part of implementing the corrective action plan, Joint Staff officials reported awarding a contract to update the information system that facilitates the operation of the Joint Capabilities Integration and Development System (JCIDS). They also stated that the Joint Staff has assigned additional technical personnel to assist in the update effort. Joint Staff officials noted that efforts are currently underway to develop a data analytics application within the system. The Joint Staff anticipates the full implementation of system improvements in September 2025. They believe that at full implementation, the system will address the various aspects of our recommendation.
|
| Department of Defense | After resolving the information system data reliability issues for the Joint Capabilities Integration and Development System (JCIDS), the Secretary of Defense should ensure that the Vice Chairman of the Joint Chiefs of Staff establish a baseline based on reliable data and relevant issues for determining a length time to validate capability documents and a process for using that baseline to assess JCIDS' efficiency and effectiveness in meeting management objectives and making improvements. (Recommendation 2) |
DOD concurred with our recommendation. In July 2024, Joint Staff officials provided us with an update to their corrective action plan to address the recommendation. Joint Staff officials acknowledged that with the implementation of a more reliable supporting information system from the actions related to Recommendation 1, the current baseline associated with the Joint Capabilities Integration and Development System (JCIDS) will be adjusted as appropriate to provide more accurate and meaningful metrics. They noted that updating the JCIDS information system will address establishing a baseline and build accountability for compliance. Joint Staff officials also stated that the statement of work for the information system support contract reinforces the requirement to collect the metrics that are outlined in the JCIDS manual and provides resources to incorporate these capabilities into the software. The Joint Staff estimates completion of these efforts in September 2025.
|