Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk
Highlights
The Department of Homeland Security's (DHS) Secure Border Initiative Network (SBInet) is a multiyear, multibillion dollar program to deliver surveillance and decision-support technologies that create a virtual fence and situational awareness along the nation's borders with Mexico and Canada. Managed by DHS's Customs and Border Protection (CBP), SBInet is to strengthen CBP's ability to identify, deter, and respond to illegal breaches at and between border points of entry. Because of the program's importance, cost, and risks, GAO was asked to, among other things, determine (1) whether SBInet testing has been effectively managed, including the types of tests performed and whether they were well planned and executed, and (2) what the results of testing show. To do this, GAO reviewed test management documentation, including test plans, test cases, test procedures, and results relative to federal and related guidance, and interviewed program and contractor officials.
DHS has not effectively managed key aspects of SBInet testing. While DHS's approach appropriately consists of a series of progressively expansive developmental and operational events, the test plans, cases, and procedures for the most recent test events were not defined in accordance with important elements of relevant guidance. For example, while plans for component and system testing included roles and responsibilities for personnel involved in each of ten test events that GAO reviewed, none of the plans adequately described risks and only two of the plans included quality assurance procedures for making changes to the plans during their execution. Similarly, while GAO's analysis of a random probability sample of test cases showed that a large percentage of the cases included procedures and expected outputs and behaviors, a relatively small percentage described the inputs and the test environment (e.g.,facilities and personnel to be used). Moreover, even though the test cases largely included procedures, a large percentage were changed extemporaneously during execution in order to fulfill the purpose of the test. While some of the changes were minor, others were more significant, such as rewriting entire procedures and changing the mapping of requirements to cases. Further, these changes to procedures were not made in accordance with documented quality assurance processes, but rather were based on an undocumented understanding that program officials said they established with the contractor. Compounding the number and significance of changes are questions raised by the SBInet program office and a support contractor about the appropriateness of some changes. For example, a program office letter to the prime contractor stated that changes made to system qualification test cases and procedures appeared to be designed to pass the test instead of being designed to qualify the system. Program officials attributed these weaknesses to time constraints and guidance limitations. Because of these issues, the risk that testing has not sufficiently supported objectives, exercised program requirements, and reflected the system's ability to perform as intended is increased. From March 2008 through July 2009, about 1,300 SBInet defects have been found, with the number of new defects identified generally increasing faster than the number being fixed--a trend that is not indicative of a system that is maturing. Further, while the full magnitude of these unresolved defects is unclear because the majority were not assigned a priority for resolution, several of the defects that have been found have been significant. Although DHS reports that these defects have been resolved, they have caused delays, and related problems have surfaced that continue to impact the program's schedule. Further, an early user assessment raised concerns about the performance of key system components and the system's operational suitability. Program officials attributed limited prioritization of defects to a lack of defect management guidance. Given that key test events have yet to occur and will likely surface other defects, it is important for defect management to improve. If not, the likelihood of SBInet meeting user expectations and mission needs will be reduced.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To improve DHS's management of SBInet testing, including the risk-based resolution of current and to-be-detected system problems, the Secretary of Homeland Security should direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, to revise the SBInet Test and Evaluation Master Plan to include (1) explicit criteria for assessing the quality of test documentation, including test plans and test cases, and (2) a process for analyzing, prioritizing, and resolving program defects. |
In January 2010, the Secretary of Homeland Security initiated an assessment of the SBInet program. The assessment was motivated in part by continuing delays in the development and deployment of SBInet capabilities and concerns that the SBInet system had not been adequately justified by a quantitative assessment of cost and benefits. Between January 2010 and January 2011, we reported on a variety of management weaknesses that contributed to the Secretary's concerns, including inadequate acquisition management practices--test management, requirements management, and risk management--shrinking system capabilities, and the lack of effective contractor oversight and economic justification for the program; and we made associated recommendations with which DHS largely agreed. In January 2011, DHS completed its assessment and decided to end SBInet as originally conceived. We consider DHS's decision to discontinue the SBInet program a prudent course of action, and responsive to the many recommendations we made to address the wide array of identified weaknesses.
|
| Department of Homeland Security | To improve DHS's management of SBInet testing, including the risk-based resolution of current and to-be-detected system problems, the Secretary of Homeland Security should direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, to ensure that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised Test and Evaluation Master Plan. |
In January 2010, the Secretary of Homeland Security initiated an assessment of the SBInet program. The assessment was motivated in part by continuing delays in the development and deployment of SBInet capabilities and concerns that the SBInet system had not been adequately justified by a quantitative assessment of cost and benefits. Between January 2010 and January 2011, we reported on a variety of management weaknesses that contributed to the Secretary's concerns, including inadequate acquisition management practices--test management, requirements management, and risk management--shrinking system capabilities, and the lack of effective contractor oversight and economic justification for the program; and we made associated recommendations with which DHS largely agreed. In January 2011, DHS completed its assessment and decided to end SBInet as originally conceived. We consider DHS's decision to discontinue the SBInet program a prudent course of action, and responsive to the many recommendations we made to address the wide array of identified weaknesses.
|
| Department of Homeland Security | To improve DHS's management of SBInet testing, including the risk-based resolution of current and to-be-detected system problems, the Secretary of Homeland Security should direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, to ensure that sufficient time is provided for reviewing and approving test documentation prior to beginning a given test event. |
In January 2010, the Secretary of Homeland Security initiated an assessment of the SBInet program. The assessment was motivated in part by continuing delays in the development and deployment of SBInet capabilities and concerns that the SBInet system had not been adequately justified by a quantitative assessment of cost and benefits. Between January 2010 and January 2011, we reported on a variety of management weaknesses that contributed to the Secretary's concerns, including inadequate acquisition management practices--test management, requirements management, and risk management--shrinking system capabilities, and the lack of effective contractor oversight and economic justification for the program; and we made associated recommendations with which DHS largely agreed. In January 2011, DHS completed its assessment and decided to end SBInet as originally conceived. We consider DHS's decision to discontinue the SBInet program a prudent course of action, and responsive to the many recommendations we made to address the wide array of identified weaknesses.
|
| Department of Homeland Security | To improve DHS's management of SBInet testing, including the risk-based resolution of current and to-be-detected system problems, the Secretary of Homeland Security should direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, to triage the full inventory of unresolved system problems, including identified user concerns, and periodically report the status of the highest priority defects to Customs and Border Protection and Department of Homeland Security leadership. |
In January 2010, the Secretary of Homeland Security initiated an assessment of the SBInet program. The assessment was motivated in part by continuing delays in the development and deployment of SBInet capabilities and concerns that the SBInet system had not been adequately justified by a quantitative assessment of cost and benefits. Between January 2010 and January 2011, we reported on a variety of management weaknesses that contributed to the Secretary's concerns, including inadequate acquisition management practices--test management, requirements management, and risk management--shrinking system capabilities, and the lack of effective contractor oversight and economic justification for the program; and we made associated recommendations with which DHS largely agreed. In January 2011, DHS completed its assessment and decided to end SBInet as originally conceived. We consider DHS's decision to discontinue the SBInet program a prudent course of action, and responsive to the many recommendations we made to address the wide array of identified weaknesses.
|