Information Technology:

DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best Practices and Controls

GAO-04-722: Published: Jul 30, 2004. Publicly Released: Jul 30, 2004.

Additional Materials:

Contact:

Randolph C. Hite
(202) 512-6256
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The way in which the Department of Defense (DOD) has historically acquired its business systems has been cited as a root cause for its limited success in delivering promised system capabilities and benefits on time and within budget. In response, DOD recently revised its systems acquisition policies and guidance to incorporate best practices, including those pertaining to business systems. GAO was asked to determine whether DOD's revised systems acquisition policies and guidance (1) are consistent with industry best practices, including those pertaining to commercial component-based systems, and (2) provide the necessary controls to ensure that DOD component organizations adhere to the practices.

DOD's revised policies and guidance largely incorporate 10 best practices for acquiring any type of information technology (IT) business system. For example, the revisions include the requirement that acquisitions be economically justified on the basis of costs, benefits, and risks. However, the revisions generally do not incorporate 8 best practices relating to the acquisition of commercial component-based systems. For example, they do not address basing any decision to modify commercial components on a thorough analysis of the impact of doing so or on preparing system users for the business process and job roles and responsibilities changes that are embedded in the functionality of commercial IT products. In total, GAO found that DOD's acquisition policies and guidance fully incorporate 8 of the 18 best practices that they were evaluated against, partially incorporate 5 practices, and do not incorporate the remaining 5 practices. DOD intends to expand its acquisition guidance to incorporate additional best practices by September 30, 2004, but department officials cite other priorities as a reason why they have not been able to complete this effort and could not provide a plan specifying how this will be accomplished. Until DOD's revised policies and guidance incorporate key systems acquisition best practices, the risk that system investments will not consistently deliver promised capabilities and benefits on time and within budget is increased. DOD's revised policies also do not contain sufficient controls to ensure that DOD components appropriately follow the best practices that are incorporated in its policies and guidance. According to acquisition best practices experts, as well as GAO's internal control guidance, controls are effective if they are backed by measurements that are verified. Although the revised policies and guidance require acquisition managers to examine and, as appropriate, adopt best practices, they do not cite what that examination entails. DOD believes existing controls are sufficient, even though these controls do not provide for measuring and validating the practices' use. Without specific requirements to measure and validate the use of best practices, the risk that they will not be followed increases, which, in turn, increases the risk that system investments will not meet expectations.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In response to our report, DOD agreed that incorporation of additional best practices in its acquisition policies and guidance should be undertaken or considered. Further, DOD has updated the acquisition guidebook to incorporate most of the missing practices that we identified in our report and the following 12 recommendations address. DOD's actions are generally consistent with our recommendation.

    Recommendation: To improve DOD's ability to acquire business systems, the Secretary of Defense should direct the Under Secretary for Acquisition, Technology, and Logistics, in collaboration with the Assistant Secretary for Networks and Information Integration and the Director, Operational Test and Evaluation, to develop and implement an explicit plan for incorporating into the 5000 series the best practices and associated activities currently missing from the series. The plan should specify tasks to be performed, resources needed and assigned, and milestones for completing tasks.

    Agency Affected: Department of Defense

  2. Status: Closed - Implemented

    Comments: DOD has developed an Enterprise Integration tool kit that endorses and supports the best practice of reusing reports, interfaces, and conversions that have been built or acquired by other programs. According to the Software Engineering Institute, system reuse is a key part of the product line requirements best practice. Under DOD's planned approach to consider product line requirements in acquisition, Enterprise Integration toolkit reuse covers such items as software modules or components, which developers are encouraged to use. DOD's actions are largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Product line requirements--rather than just the requirements for the system being acquired--are an explicit consideration in each acquisition.

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: DOD agreed that communication of acquisition management activities to all stakeholders is a best practice. DOD 5000 acquisition policies and procedures and related guidance now provide for program managers and acquisition officials to conduct acquisition oversight through integrated product teams as a way to communicate information. These policies and procedures are largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Acquisition project management activities are communicated to all stakeholders.

    Agency Affected: Department of Defense

  4. Status: Closed - Implemented

    Comments: DOD has updated the DOD 5000 acquisition policies and procedures and related guidance to include acquisition risk management, including references to a risk management guide that provides for identifying the status of risks at each phase and milestone of the acquisition process. Further, it describes what is to be done to manage and mitigate the risks. This is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Acquisition reviews include the status of identified risks.

    Agency Affected: Department of Defense

  5. Status: Closed - Implemented

    Comments: DOD has added this best practice activity to its DOD 5000 Acquisition guidance. In the guidebook, DOD added a section entitled, "Modifying Commercial off-the-Shelf Software (COTS)" which discusses ways to avoid modifying the COTS product. For example, the guidebook notes that modifying the core code of a COTS product should be avoided. Further, it adds that the business processes inherent in the COTS product should be adopted, not adapted, by the organization implementing the product. DOD not only discouraged the modification of commercial components but also noted that should the product need to be modified, a business process re-engineering approach should be considered. This is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Modification of commercial components is discouraged and allowed only if justified by a thorough analysis of life-cycle costs and benefits.

    Agency Affected: Department of Defense

  6. Status: Closed - Implemented

    Comments: DOD agreed that a best practice to modify deployed versions of system components should be incorporated. Accordingly, DOD's 5000 acquisition guidance cites the application of configuration management and incorporates configuration change controls. However, the guidance does not address whether the modifications are centrally controlled and whether unilateral user release changes are precluded.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Modification or upgrades to deployed versions of system components are centrally controlled, and unilateral user release changes are precluded.

    Agency Affected: Department of Defense

  7. Status: Closed - Implemented

    Comments: According to DOD, acquisition decisions about commercial components based on research, analysis, and evaluation of components interdependencies is considered in the Joint Capabilities Integration and Development System (JCIDS) process and is to be addressed in the Information Support Plan (ISP). Further, DOD has incorporated a section on the ISP in the DOD 5000 acquisition guidebook. Specifically, the guidebook notes that the ISP is to identify and resolve implementation issues concerning acquisition programs, as well as provide managers with a method of identifying information-related dependencies. This is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Acquisition decisions about commercial components are based on deliberate and thorough research, analysis, and evaluation of the components' interdependencies.

    Agency Affected: Department of Defense

  8. Status: Closed - Implemented

    Comments: DOD agreed that change management activities, such as those described in this recommendation, are best practices and it points to its Enterprise Integration tool kit as containing a change management roadmap that addresses organizational change, readiness, and preparing users for changes to their roles and responsibilities. This tool kit provides relevant change management information for the technology development phase of the acquisition process. As such, it is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Acquisition plans provide for preparing users for the impact that the business processes embedded in the commercial components will have on their respective roles and responsibilities.

    Agency Affected: Department of Defense

  9. Status: Closed - Implemented

    Comments: DOD agreed that change management activities, such as those described in this recommendation, are best practices and it points to its Enterprise Integration tool kit as containing a change management roadmap that addresses organizational change, readiness, and preparing users for using the system to execute their jobs. This tool kit provides relevant change management information for the technology development phase of the acquisition process. As such, it is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Changes affecting how users will be expected to use the system to execute their jobs are actively managed.

    Agency Affected: Department of Defense

  10. Status: Closed - Implemented

    Comments: DOD 5000 acquisition policies and procedures and related guidance now cite and reference the use of information on contractors' past performance, which can include the ability to implement commercial components. This is largely consistent with our recommendation.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and that the plan, at a minimum, should incorporate systems best practices activities. Systems integration contractors are explicitly evaluated on their ability to implement commercial components.

    Agency Affected: Department of Defense

  11. Status: Closed - Implemented

    Comments: DOD has incorporated a section on systems engineering in its DOD 5000 acquisition guidebook that discusses tradeoffs throughout a system's life cycle. In addition, the Joint Capabilities Integration and Development System require that three analyses be conducted before a capability is acceptable to enter the technology development phase of the acquisition process. These analyses are to be used at investment decision points in making tradeoffs among various available capabilities (including to those available in commercial components) relative to the architectural environment in which the system is to operate, defined system requirements, and existing cost/schedule constraints.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Investment decisions throughout a system's life cycle are based on a continuous set of tradeoffs among capabilities available in commercial components (current and future), the architectural environment in which the system is to operate, defined system requirements, and existing cost/schedule constraints.

    Agency Affected: Department of Defense

  12. Status: Closed - Not Implemented

    Comments: DOD stated that no changes were necessary to the DOD 5000 policies or guidebook because they call for a capability development document that describes system requirements. However, they do not cite the need to consider vendor/commercial product characteristics.

    Recommendation: The progress against this plan should be tracked and reported as appropriate, and the plan, at a minimum, should incorporate best practices activities. Evaluation criteria are established for selecting among commercial component options that include both defined system requirements and vendor/commercial product characteristics.

    Agency Affected: Department of Defense

  13. Status: Closed - Not Implemented

    Comments: According to DOD, its existing oversight process includes the necessary compliance activities because statutory and regulatory compliance are referenced in DOD's integrated process team and milestone decision point process components of its DOD 5000 acquisition policies and guidance. However, they do not require measurement and verification. Rather, it makes such reviews discretionary.

    Recommendation: To ensure that the best practices provided for in DOD acquisition policies and guidance are appropriately followed, the plan should incorporate steps to include in DOD's acquisition policies a provision for measurement and verification of best practices.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Sep 10, 2014

Sep 9, 2014

Sep 8, 2014

Jul 31, 2014

Looking for more? Browse all our products here