Skip to main content

Information Technology: DHS Needs to Improve Its Independent Acquisition Reviews

GAO-11-581 Published: Jul 28, 2011. Publicly Released: Jul 28, 2011.
Jump To:
Skip to Highlights

Highlights

Since its creation in 2003, the Department of Homeland Security (DHS) has been developing new information technology (IT) systems to perform both mission-critical and support functions; however, it has faced challenges in developing these systems. One way to manage the inherent risks of developing and acquiring systems is through independent verification and validation (IV&V)--a process conducted by a party independent of the development effort that provides an objective assessment of a project's processes, products, and risks throughout its life cycle and helps ensure that program performance, schedule, and budget targets are met. GAO was asked to determine (1) how DHS's IV&V policies and procedures for IT acquisitions compare with leading practices and (2) the extent to which DHS has implemented IV&V on its large IT system acquisitions. To do so, GAO assessed DHS's policy against industry standards and leading practice guides, as well as analyzed how eight selected IT programs had implemented IV&V.

DHS recognizes the importance of IV&V and recommends its use on major IT programs. Nevertheless, its acquisition policy does not address the elements of leading practices for IV&V. Specifically, the department has not established risk-based decision making criteria for determining whether, or the extent to which, programs should utilize IV&V. In addition, department policy does not define the degree of independence required of agents and does not require that programs determine and document the planned scope of their efforts, including the program activities subject to review; the resources required; roles and responsibilities; and how the results will be reported and acted upon. Moreover, the policy does not address overseeing DHS's investment in IV&V. Thus, officials were unaware of the extent to which it was being used on major IT acquisition programs, associated expenditures, or if those expenditures are producing satisfactory results. Absent such policy elements and more effective oversight, the department's investments in IV&V efforts are unlikely to provide optimal value for the department and, in some cases, may even fail to deliver any significant benefits. Many large IT acquisition programs across DHS reported using IV&V as part of their acquisition and/or development processes. Nevertheless, the eight major IT acquisition programs that GAO analyzed did not consistently implement the elements of leading practice. For example, the eight did not fully apply a structured, risk-based decision making process when deciding if, when, and how to utilize IV&V. In part, these weaknesses are due to the lack of clear departmentwide guidance regarding the use of such practices. As a result, the department's IV&V efforts may not consistently contribute toward meeting IT acquisition cost, schedule, and mission goals. GAO recommends that DHS (1) update its acquisition policy to reflect elements of effective IV&V, (2) monitor and ensure implementation of this policy on applicable new and ongoing IT programs, and (3) collect data on IV&V usage and use it to evaluate the effectiveness of these investments. DHS concurred with GAO's recommendations and described actions planned or under way to address them.

Recommendations

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Homeland Security To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department Chief Information Officer (CIO) and Chief Procurement Officer (CPO) to revise DHS acquisition policy such that it establishes (1) risk-based criteria for (1) determining which major and other high- risk IT acquisition programs should conduct IV&V and (2) selecting appropriate activities for independent review of these programs; (2) requirements for technical, financial, and managerial independence of agents; (3) standards and guidance for defining and documenting plans and products; (4) controls for planning, managing, and overseeing efforts; (5) mechanisms to ensure that plans and significant findings inform DHS acquisition program reviews and decisions, including those of the DHS's Acquisition Review Board (ARB); and (6) mechanisms to monitor and ensure implementation of this policy on applicable new IT acquisition programs.
Closed – Implemented
The Department of Homeland Security (DHS) concurred with our recommendation. In June 2016, DHS provided evidence that it had generally addressed the weaknesses identified in our report. Specifically, DHS developed an Independent Verification and Validation (IV&V) Annex guide in support of the department's System Engineering Life Cycle guidebook. The guidance provides risk based criteria for selecting activities for independent review of IT acquisition programs and guidance for defining and documenting plans and products for IV&V activities. In addition, the guidance addresses technical, financial, and managerial independence of agents, controls for managing IV&V activities and mechanisms to ensure that plans and significant findings inform DHS decisions. Further, the guidance calls for the completion of an IV&V template that provides an overview of IV&V activities across the entire lifecycle which helps ensure implementation of its IV&V guidance on applicable IT acquisition programs. Fully implementing the guidance should help guide consistent and effective execution of IV&V at DHS and contribute toward meeting the schedule and mission goals of its IT acquisition programs.
Department of Homeland Security To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department CIO and CPO to reevaluate the approach to IV&V for ongoing programs (including the eight programs featured in this report) and ensure that appropriate actions are taken to bring each of them into alignment with the elements of leading practice.
Closed – Implemented
The DHS concurred with our recommendation. In response, in June 2016 DHS provided updated guidance for its IV&V activities that was issued in January 2015. DHS reported that it has monitored and re-evaluated the planning, requirements analysis, design, integration, and testing activities of the eight programs featured in our report and provided high level summary information on the results of its evaluation of these eight programs, as of 2016. Because the IV&V guidance was developed in January 2015, the extent to which the department's IV&V approach was reevaluated to bring the eight assessed programs into alignment with elements of leading IV&V practices is unclear. Nevertheless, DHS updated its IV&V guidance in line with elements of leading IV&V practices. Fully implementing the guidance should help guide consistent and effective execution of IV&V at DHS and contribute toward meeting the schedule and mission goals of its IT acquisition programs.
Department of Homeland Security To help guide consistent and effective execution of IV&V at DHS, the Secretary of Homeland Security should direct the department CIO and CPO to collect and analyze data on IV&V efforts for major IT acquisition programs to facilitate the development of lessons learned and evaluation of the effectiveness of DHS's investments, and establish a process that uses the results to inform the department's IT investment decisions.
Closed – Implemented
The Department of Homeland Security (DHS) concurred with our recommendation. In June 2016 DHS provided updated guidance for its IV&V activities and documentation on its process for the collection and analysis of IV&V data to improve the monitoring of acquisition program activities. As part of this process, DHS stated that the results of IV&V analyses are incorporated into the agency's Strategic Sourcing EAGLE II Metrics report which is produced quarterly and maintained by the DHS Strategic Sourcing Program Office. According to DHS, the program IV&V testing activities are monitored by the department's Program Accountability and Risk Management and Enterprise Business Management Office. In addition, DHS reported that for all major IT investments, IV&V reports are reviewed by the program's oversight and governance bodies and actionable items are tracked to completion. Further, IV&V reports are archived and used to facilitate the development of lessons learned which can be leveraged and referenced in the execution of future oversight and governance boards. Taken together, these actions address the intent of our recommendation.

Full Report

Office of Public Affairs

Topics

Agency missionsBest practicesData collectionDecision makingEvaluation criteriaEvaluation methodsIndependent research and developmentInformation technologyInternal controlsIT acquisitionsIT investment managementLessons learnedOperational testingPolicy evaluationProcurement planningProcurement policyProgram evaluationProgram managementRisk managementSoftware verification and validationSystems designSystems development life cycleTestingAssessmentsPolicies and proceduresProgram implementationSystems development