This is the accessible text file for GAO report number GAO-10-158 
entitled 'Secure Border Initiative: DHS Needs to Address Testing and 
Performance Limitations That Place Key Technology Program at Risk' 
which was released on March 18, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 
GAO: 

January 2010: 

Secure Border Initiative: 

DHS Needs to Address Testing and Performance Limitations That Place 
Key Technology Program at Risk: 

Secure Border Initiative: 

GAO-10-158: 

GAO Highlights: 

Highlights of GAO-10-158, a report to congressional requesters. 

Why GAO Did This Study: 

The Department of Homeland Security’s (DHS) Secure Border Initiative 
Network (SBInet) is a multiyear, multibillion dollar program to 
deliver surveillance and decision-support technologies that create a 
virtual fence and situational awareness along the nation’s borders 
with Mexico and Canada. Managed by DHS’s Customs and Border Protection 
(CBP), SBInet is to strengthen CBP’s ability to identify, deter, and 
respond to illegal breaches at and between border points of entry. 
Because of the program’s importance, cost, and risks, GAO was asked 
to, among other things, determine (1) whether SBInet testing has been 
effectively managed, including the types of tests performed and 
whether they were well planned and executed, and (2) what the results 
of testing show. To do this, GAO reviewed test management 
documentation, including test plans, test cases, test procedures, and 
results relative to federal and related guidance, and interviewed 
program and contractor officials. 

What GAO Found: 

DHS has not effectively managed key aspects of SBInet testing. While DHS
’s approach appropriately consists of a series of progressively 
expansive developmental and operational events, the test plans, cases, 
and procedures for the most recent test events were not defined in 
accordance with important elements of relevant guidance. For example, 
while plans for component and system testing included roles and 
responsibilities for personnel involved in each of ten test events 
that GAO reviewed, none of the plans adequately described risks and 
only two of the plans included quality assurance procedures for making 
changes to the plans during their execution. Similarly, while GAO’s 
analysis of a random probability sample of test cases showed that a 
large percentage of the cases included procedures and expected outputs 
and behaviors, a relatively small percentage described the inputs and 
the test environment (e.g., facilities and personnel to be used). 
Moreover, even though the test cases largely included procedures, a 
large percentage were changed extemporaneously during execution in 
order to fulfill the purpose of the test. While some of the changes 
were minor, others were more significant, such as rewriting entire 
procedures and changing the mapping of requirements to cases. Further, 
these changes to procedures were not made in accordance with 
documented quality assurance processes, but rather were based on an 
undocumented understanding that program officials said they 
established with the contractor. Compounding the number and 
significance of changes are questions raised by the SBInet program 
office and a support contractor about the appropriateness of some 
changes. For example, a program office letter to the prime contractor 
stated that changes made to system qualification test cases and 
procedures appeared to be designed to pass the test instead of being 
designed to qualify the system. Program officials attributed these 
weaknesses to time constraints and guidance limitations. Because of 
these issues, the risk that testing has not sufficiently supported 
objectives, exercised program requirements, and reflected the system’s 
ability to perform as intended is increased. 

From March 2008 through July 2009, about 1,300 SBInet defects have 
been found, with the number of new defects identified generally 
increasing faster than the number being fixed—a trend that is not 
indicative of a system that is maturing. Further, while the full 
magnitude of these unresolved defects is unclear because the majority 
were not assigned a priority for resolution, several of the defects 
that have been found have been significant. Although DHS reports that 
these defects have been resolved, they have caused delays, and related 
problems have surfaced that continue to impact the program’s schedule. 
Further, an early user assessment raised concerns about the 
performance of key system components and the system’s operational 
suitability. Program officials attributed limited prioritization of 
defects to a lack of defect management guidance. Given that key test 
events have yet to occur and will likely surface other defects, it is 
important for defect management to improve. If not, the likelihood of 
SBInet meeting user expectations and mission needs will be reduced. 

What GAO Recommends: 

GAO is making four recommendations to DHS related to the content, 
review, and approval of test planning documentation and the analysis, 
disclosure, and resolution of system problems. DHS agreed with three 
and partially agreed with one of the recommendations, and it described 
actions under way and planned to address them. 

View GAO-10-158 or key components. For more information, contact 
Randolph C. Hite, (202) 512-3439 or hiter@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DHS's Management of SBInet Testing Has Not Been Effective and Has 
Increased Program Risks: 

SBInet Test Results Have Identified a Growing Number of System 
Performance and Quality Problems: 

DHS Science and Technology Directorate's Testing Process Is Being Used 
to Leverage Maturing Technologies for SBInet: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Homeland Security: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Overview of Key Test Events: 

Table 2: Components Tested during CQT: 

Figures: 

Figure 1: Potential Long-Term SBInet Concept of Operations: 

Figure 2: TUS-1 and AJO-1 Locations Along the Southwest Border: 

Figure 3: Extent to Which Component and System Qualification Test 
Plans Satisfied Relevant Guidance: 

Figure 4: Estimated Percentages of CQT Test Cases Containing Key 
Elements Based on a Sample of 60: 

Figure 5: Excerpt from NOC/SOC As-Run Log Showing Remapped 
Requirements and Rewritten Procedures: 

Figure 6: Trend in the Number of Open Defects from March 2008 to July 
2009: 

Figure 7: Percentage of SBInet Defects with and without Assigned 
Priorities: 

Abbreviations: 

AJO-1: Ajo Border Patrol Station: 

C3I: command, control, communications, and intelligence: 

CBP: U.S. Customs and Border Protection: 

COMM: communications component: 

COP: common operating picture: 

CQT: component qualification testing: 

DHS: Department of Homeland Security: 

DOD: Department of Defense: 

DT&E: developmental test and evaluation: 

EOIR: Electro-Optical/Infrared: 

IT: information technology: 

IT&E: independent test and evaluation: 

IV&V: independent verification and validation: 

MSS: Mobile Surveillance Systems: 

NOC/SOC: Network Operations Center/Security Operations Center: 

OT&E: operational test and evaluation: 

RTU: Remote Terminal Unit: 

SAT: system acceptance testing: 

SBInet: Secure Border Initiative Network: 

S&T: Science and Technology Directorate: 

SBI: Secure Border Initiative: 

SPO: SBInet System Program Office: 

SQT: system qualification testing: 

TEMP: Test and Evaluation Master Plan: 

TUS-1: Tucson Border Patrol Station: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

January 29, 2010: 

The Honorable Bennie G. Thompson: 
Chairman: 
Committee on Homeland Security: 
House of Representatives: 

The Honorable Christopher P. Carney: 
Chairman: 
The Honorable Gus M. Bilirakis: 
Ranking Member: 
Subcommittee on Management, Investigations, and Oversight: 
Committee on Homeland Security: 
House of Representatives: 

The Honorable Mike Rogers: 
Ranking Member Subcommittee on Emergency: 
Communications, Preparedness, and Response: 
Committee on Homeland Security: 
House of Representatives: 

Securing the 6,000 miles of international borders that the contiguous 
United States shares with Canada and Mexico is a challenge and a 
mission imperative to the Department of Homeland Security (DHS). 
Although hundreds of thousands of illegal aliens are prevented from 
entering the country illegally each year, many more are not detected. 
To enhance border security and reduce illegal immigration, DHS 
launched its multiyear, multibillion dollar Secure Border Initiative 
(SBI) program in November 2005. Through SBI, DHS intends to enhance 
surveillance technologies, raise staffing levels, increase domestic 
enforcement of immigration laws, and improve the physical 
infrastructure along the nation's borders. 

Within SBI, Secure Border Initiative Network (SBInet) is a 
multibillion dollar program that includes the acquisition, 
development, integration, deployment, and operation and maintenance of 
surveillance technologies to create a "virtual fence" along the 
border, as well as command, control, communications, and intelligence 
(C3I) technologies to create a picture of the border in command 
centers and vehicles. Managed by DHS's Customs and Border Protection 
(CBP), SBInet is to strengthen the ability of CBP to detect, identify, 
classify, track, and respond to illegal breaches at and between ports 
of entry.[Footnote 1] 

In September 2008, we reported that SBInet was at risk because of a 
number of acquisition management weaknesses. Accordingly, we made a 
number of recommendations to address the weaknesses, which DHS largely 
agreed with and committed to addressing. Among these weaknesses were 
several associated with system testing. As we have previously 
reported, testing occurs throughout a system's life cycle and is 
essential to knowing whether the system meets defined requirements and 
performs as intended.[Footnote 2] In light of SBInet's high cost and 
risks, and because of the importance of effective testing to the 
program's success, you asked us to review SBInet testing. Our 
objectives were to determine (1) the extent to which testing has been 
effectively managed, including identifying the types of tests 
performed and whether they were well planned and executed; (2) what 
the results of testing show; and (3) what processes are being used to 
test and incorporate maturing technologies into SBInet. 

To accomplish our objectives, we reviewed key testing documentation, 
including processes, test plans, test cases, execution logs, and 
results, as well as system requirements to be tested and the overall 
test management approach. We also analyzed a random probability sample 
of system test cases, and we interviewed program officials about test 
planning, execution, and management. We then compared this information 
to relevant guidance to identify any deviations and interviewed 
program officials as to the reasons for any deviations. 

We conducted this performance audit from December 2008 to January 2010 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. Further 
details of our objectives, scope, and methodology are included in 
appendix I. 

Background: 

SBInet includes the acquisition, development, integration, deployment, 
and operations and maintenance of a mix of surveillance technologies, 
such as cameras, radars, sensors, and C3I technologies. The initial 
focus of the program has been on addressing the requirements of CBP's 
Office of Border Patrol, which is responsible for securing the borders 
between the ports of entry. Longer-term, the program is to address the 
requirements of CBP's Office of Field Operations, which controls 
vehicle and pedestrian traffic at the ports of entry, and its Office 
of Air and Marine Operations, which operates helicopters, fixed-wing 
aircraft, and marine vessels used in securing the borders. Figure 1 
shows the potential long-term SBInet concept of operations. 

Figure 1: Potential Long-Term SBInet Concept of Operations: 

[Refer to PDF for image: illustration] 

The following are depicted on the illustration: 
* U.S.-Mexico border; 
* Port of entry; 
* Unattended ground sensors; 
* Radar/camera tower; 
* Mobile surveillance system; 
* Agent vehicles with mobile data terminal; 
* Command Center: 
- Border patrol sector headquarters; 
- Border patrol station; 
* Unmanned aerial surveillance; 
* Air support; 
* Marine support. 

Sources: GAO analysis of DHS data, Art Explosion (clip art). 

[End of figure] 

Surveillance technologies are to include a variety of sensor systems. 
Specifically, unattended ground sensors are to be used to detect heat 
and vibrations associated with foot traffic and metal associated with 
vehicles. Radar mounted on fixed and mobile towers is to detect 
movement, and cameras on fixed and mobile towers are used by operators 
to identify and classify items of interest detected and tracked by 
ground sensors and radar. Aerial assets are also to be used to provide 
video and infrared imaging to enhance tracking targets. 

C3I technologies (software and hardware) are to produce a common 
operating picture (COP)--a uniform presentation of activities within 
specific areas along the border. Together, the sensors, radar, and 
cameras are to gather information along the border and transmit this 
information to COP terminals located in command centers and agents' 
vehicles which, in turn, are to assemble it to provide CBP agents with 
border situational awareness. Among other things, COP hardware and 
software are to allow agents to (1) view data from radar and sensors 
that detect and track movement in the border areas, (2) control 
cameras to help identify and classify illegal entries, (3) correlate 
entries with the positions of nearby agents, and (4) enhance tactical 
decision making regarding the appropriate response to apprehend an 
entry, if necessary. 

Overview of SBInet Acquisition Strategy: 

In September 2006, CBP awarded a 3-year contract to the Boeing Company 
for SBInet, with three additional 1-year options. As the prime 
contractor, Boeing is responsible for designing, producing, testing, 
deploying, and sustaining the system. In September 2009, CBP extended 
its contract with Boeing for the first option year. 

CBP's acquisition strategy entails delivering the system incrementally 
in a series of discrete units of capability (blocks), in which an 
initial system capability is to be delivered based on a defined subset 
of the total requirements. Block capabilities generally include the 
purchase of commercially available surveillance systems, development 
of customized COP systems and software, and use of existing CBP 
communications and network capabilities. Subsequent blocks of SBInet 
capabilities are to be delivered based on feedback, unmet 
requirements, and the availability of new technologies. Such an 
incremental approach is a recognized best practice for acquiring large-
scale, complex systems because it allows users access to new 
capabilities and tools sooner, thus permitting both their operational 
use and evaluation. 

CBP's SBI Program Executive Office is responsible for managing key 
acquisition functions associated with SBInet, including tracking and 
overseeing the prime contractor.[Footnote 3] It is organized into four 
areas: SBInet System Program Office (SPO), Systems Engineering, 
Business Management, and Operational Integration, and it is staffed 
with a mix of government personnel and contractor support staff. In 
addition, the SPO is leveraging other DHS resources, such as the DHS 
Science and Technology Directorate (S&T). 

Recent GAO Work on SBInet Identified Management Weaknesses and Program 
Risks: 

In July 2008, we briefed CBP and DHS officials on the results of our 
then-ongoing review of SBInet, and in September 2008, we reported that 
important aspects of SBInet were ambiguous and in a continuous state 
of flux, making it unclear and uncertain what technology capabilities 
were to be delivered when.[Footnote 4] For example, we reported that 
the scope and timing of planned SBInet deployments and capabilities 
had continued to change since the program began and remained unclear. 
Further, the SPO did not have an approved integrated master schedule 
to guide the execution of the program, and our assimilation of 
available information indicated that key milestones continued to slip. 
This schedule-related risk was exacerbated by the continuous change in 
and the absence of a clear definition of the approach used to define, 
develop, acquire, test, and deploy SBInet. Accordingly, we concluded 
that the absence of clarity and stability in these key aspects of 
SBInet impaired the ability of Congress to oversee the program and 
hold DHS accountable for results, and it hampered DHS's ability to 
measure the program's progress. 

Furthermore, we reported that SBInet requirements had not been 
effectively defined and managed. While the SPO had issued guidance 
that defined key practices associated with effectively developing and 
managing requirements, such as eliciting user needs and ensuring that 
different levels of requirements and associated verification methods 
are properly aligned with one another, the guidance was developed 
after several key activities had been completed. In the absence of 
this guidance, the SPO had not effectively performed key requirements 
development and management practices. For example, it had not ensured 
that different levels of requirements were properly aligned, as 
evidenced by our analysis of a random probability sample of component 
requirements showing that a large percentage of them could not be 
traced to higher-level system and operational requirements. Also, some 
operational requirements, which are the basis for all lower-level 
requirements, were found to be unaffordable and unverifiable, thus 
casting doubt on the quality of the lower-level requirements that had 
been derived from them. As a result, we concluded that the risk of 
SBInet not meeting mission needs and performing as intended was 
increased, as were the chances of the system needing expensive and 
time-consuming rework. 

We also reported that SBInet testing was not being effectively 
managed. For example, the SPO had not tested the individual system 
components to be deployed to initial locations, even though the 
contractor had initiated integration testing of these components with 
other system components and subsystems in June 2008. Further, while a 
test management strategy was drafted in May 2008, it had not been 
finalized and approved, and it did not contain, among other things, a 
clear definition of testing roles and responsibilities; a high-level 
master schedule of SBInet test activities; or sufficient detail to 
effectively guide project-specific planning, such as milestones and 
metrics for specific project testing. We concluded that without a 
structured and disciplined approach to testing, the risk that SBInet 
would not satisfy user needs and operational requirements was 
increased, as were the chances of needed system rework. 

To address these issues, we recommended that DHS assess and disclose 
the risks associated with its planned SBInet development, testing, and 
deployment activities; and that it address the system deployment, 
requirements management, and testing weaknesses that we had 
identified. DHS agreed with all but one of our eight recommendations 
and described actions completed, under way, and planned to address 
them. We plan to issue another report in early 2010 that, among other 
things, updates the status of DHS's efforts to implement our prior 
recommendations. 

More recently, we reported that delays in deploying SBInet 
capabilities have required Border Patrol agents to rely on existing 
technology for securing the border that has performance shortfalls and 
maintenance issues.[Footnote 5] For example, on the southwest border, 
agents rely on existing cameras mounted on towers that have 
intermittent problems, including signal loss. 

Overview of Deployment Plans: 

In August 2008, the DHS Acquisition Review Board[Footnote 6] decided 
to delay the initial system deployment, known as Block 1, so that 
fiscal year 2008 funding could be reallocated to complete physical 
infrastructure projects. According to program officials, this decision 
also allowed more time for needed testing. In addition, the board 
directed the SPO to deliver a range of program documentation, 
including an updated Test and Evaluation Master Plan (TEMP), detailed 
test plans, and a detailed schedule for deploying Block 1 to the 
initial two sites following completion of all integration and field 
testing. In February 2009, the board approved the SPO's updated 
documentation. A key result was a revised timeline for deploying Block 
1, first to the Tucson Border Patrol Station (TUS-1), in April 2009, 
and then to the Ajo Border Patrol Station (AJO-1) in June 2009, both 
of which are located in the Tucson Sector of the Southwest border. 
Together, these two deployments cover 53 miles of the 1,989-mile-long 
southern border. Figure 2 shows the TUS-1 and AJO-1 areas of 
deployment. 

Figure 2: TUS-1 and AJO-1 Locations Along the Southwest Border: 

[Refer to PDF for image: Southwest Border map] 

The following are depicted on the map: 
* Yuma sector; 
* Tucson sector: 
- TUS-1; 
- AJO-1; 
* Area of deployment for Block 1. 

Sources: GAO analysis of DHS data, MapArt (map). 

[End of figure] 

The capabilities that are to be part of the Block 1 deployments at TUS-
1 and AJO-1 include towers with radar and cameras mounted on them, 
unattended ground sensors, communications towers, and a command and 
control facility. 

As of July 2009, the TUS-1 system was scheduled for government 
acceptance in February 2010. AJO-1 acceptance is to be in July 2010. 
Longer-term plans call for Block 1 deployments throughout the Tucson 
and Yuma border sectors by the summer of 2011, with additional 
deployments throughout the southwest border between 2011 and 2016. 

DHS's Management of SBInet Testing Has Not Been Effective and Has 
Increased Program Risks: 

DHS has not effectively managed key aspects of the testing of SBInet 
components and the integration of these components. While DHS's 
testing approach appropriately consists of a series of test events, 
some of which have yet to be completed, plans for key events that have 
been performed were not defined in accordance with relevant guidance. 
For example, none of the plans for tests of system components 
addressed testing risks and mitigation strategies. Also, over 70 
percent of the procedures for the key test events were rewritten 
extemporaneously during execution because persons conducting the tests 
determined that the approved procedures were not adequate. 
Collectively, these and other limitations have increased the risk that 
the deployed system will not perform as intended. 

SBInet Testing Appropriately Consists of a Series of Developmental and 
Operational Test Events, but Some Have Yet to Be Completed: 

According to relevant leading industry practices and government 
guidance,[Footnote 7] system testing should be progressive, meaning 
that it should consist of a series of test events that build upon and 
complement previous events in the series. These tests should first 
focus on the performance of individual system components, then on the 
performance of integrated system components, followed by system-level 
tests that focus on whether the system (or major system increments) is 
acceptable, interoperable with related systems, and operationally 
suitable to users. 

To its credit, the SPO has defined and is conducting a series of Block 
1 tests that are progressively broader in scope and that are to verify 
first that individual system parts meet specified requirements, and 
then that these combined parts perform as intended as an integrated 
and operational system. These tests began with contractor-performed 
component characterization testing, which is informal (nongovernment 
witnessed) testing on the selected commercial components to verify the 
vendors' representation of product specifications. 

According to the SBInet TEMP,[Footnote 8] dated November 24, 2008, the 
program's formal test events fall into two major phases: developmental 
test and evaluation (DT&E) and operational test and evaluation (OT&E). 
DT&E is to verify and validate the system's engineering process and 
provide confidence that the system design satisfies the desired 
capabilities. It consists of four test events--integration testing, 
component qualification testing (CQT), system qualification testing 
(SQT), and system acceptance testing (SAT). OT&E is to ensure that the 
system is effective and suitable in its operational environment with 
respect to key considerations, including reliability, availability, 
compatibility, and maintainability. It consists of three test events-- 
user assessment, operational test, and follow-on operational test and 
evaluation. (See table 1 for each test event's purpose, responsible 
parties, and location.) 

Table 1: Overview of Key Test Events: 

DT&E Events: 

Test events: Integration testing; 
Purpose: Demonstrate interoperability among system components and 
ensure the proper functioning of individual component hardware and 
software interfaces; 
Party responsible: Contractor performs and SPO witnesses; 
Location: Laboratory and field test site. 

Test events: Component qualification testing; 
Purpose: Verify the functional performance of individual components 
against component requirements; 
Party responsible: Contractor performs and SPO witnesses; 
Location: Laboratory and field test site. 

Test events: System qualification testing; 
Purpose: Verify that the system design satisfies system-level 
requirements; 
Party responsible: Contractor performs and SPO witnesses; 
Location: Field test site and deployment site. 

Test events: System acceptance testing; 
Purpose: Verify that the deployed system is built as designed and 
performs as predicted in the deployed environment; 
Party responsible: Contractor performs and SPO witnesses; 
Location: Deployment site. 

OT&E Events: 

Test events: User assessment; 
Purpose: Identify potential operational problems and progress toward 
meeting desired operational effectiveness and suitability capabilities 
prior to deployment using the version of the system tested during 
system qualification testing; 
Party responsible: Border Patrol executes in coordination with the 
SPO. U.S. Army Independent Test & Evaluation Team observes; 
Location: Field test site. 

Test events: Operational test; 
Purpose: Determine whether the system meets defined key performance 
parameters in its operational environment; 
Party responsible: Border Patrol executes with support from U.S. Army 
Independent Test & Evaluation Team; 
Location: Deployment site. 

Test events: Follow-on operational test and evaluation; 
Purpose: Evaluate changes or updates made to the system and ensure 
that it continues to meet operational needs; 
Party responsible: U.S. Army Independent Test & Evaluation Team 
performs; 
Location: Deployment site. 

Source: GAO analysis of DHS data. 

[End of table] 

Regression testing, which is testing to ensure that changes made to 
correct problems found during a test event did not introduce 
unintended problems, may also be performed as part of each event. 

As of October 2009, the SPO reported that three of the four DT&E test 
events had been completed or were under way: 

* Integration testing was conducted from roughly June to October 2008. 

* CQT was conducted from roughly October to December 2008.[Footnote 9] 
CQT regression testing was conducted from February to September 2009. 
The tested components are shown in table 2. 

Table 2: Components Tested during CQT: 

Name: C3I COP; 
Description: The hardware and software to produce a uniform picture of 
activities within specific areas along the border. 

Name: NOC/SOC; 
Description: The Network Operations Center (NOC) and Security 
Operations Center (SOC) monitors networked equipment, provides alerts 
for network problems, protects equipment from network-based attacks, 
and provides user authentication. 

Name: COMM; 
Description: The microwave radio communications component that 
transmits and receives voice, surveillance, and command and control 
data. 

Name: UGS; 
Description: The unattended ground sensors (UGS) that detect heat and 
vibrations associated with foot traffic and metal associated with 
vehicles. 

Name: Power; 
Description: The source of continuous power to the communication and 
sensor sites. 

Name: EOIR; 
Description: The electro-optical/infrared (EOIR) tower-mounted day 
camera and thermal camera, which are used to help identify and 
classify objects. 

Name: Network; 
Description: The hardware and software that transports information 
between system components. 

Name: Radar; 
Description: The devices that track multiple objects simultaneously 
and provide near-real-time information on the location and movement of 
objects. 

Name: RTU; 
Description: The devices that monitor, collect, and send health and 
status information from the towers and sensors to the COP or local 
users, and send command and control information from the COP or users 
to the sensors. They act as a data integration center for remote 
sensor data. 

Source: GAO analysis of DHS data. 

Note: In CBP's technical comments on a draft of this report, it stated 
that the Fixed Tower System was also a component tested during CQT. 
However, we did not receive documentation related to testing for this 
component, therefore it was not included as part of our analysis. 

[End of table] 

* SQT was conducted from December 2008 to January 2009. SQT regression 
testing is ongoing; the first round of regression testing began in 
February 2009, and a second round began in October 2009 and is still 
under way. 

* The first OT&E event, the user assessment, was conducted from late 
March to early April 2009. 

* The final DT&E test event, SAT, has not yet begun. As of October 
2009, it was scheduled to start in late December of 2009 for TUS-1. 
The final two OT&E test events are to occur after SAT has been 
completed and DHS accepts the system. 

Test Plans Were Not Well Defined: 

Effective testing includes developing well-defined test plans. 
According to the relevant guidance,[Footnote 10] test plans should 
specify each of the following key elements: 

* Roles and responsibilities: Identifies individuals or groups that 
are to perform each aspect of the specific test event, such as test 
operators and witnesses, and the functions or activities they are to 
perform. 

* Environment and infrastructure: Identifies the physical facilities, 
hardware, software, support tools, test data, personnel, and anything 
else necessary to support the test event. 

* Tested items and approach: Identifies the object of testing (such as 
specific software or hardware attributes or interfaces) and describes 
the method used to ensure each feature of these objects is tested in 
sufficient detail. 

* Traceability matrix: Consists of a list of the requirements that are 
being tested and maps each requirement to its corresponding test 
case(s), and vice versa. 

* Risk and mitigation strategies: Identifies issues that may adversely 
impact successful completion of testing, the potential impact of each 
issue, and contingency plans for mitigating or avoiding these issues. 

* Testing schedule: Specifies milestones, duration of testing tasks, 
and the period of use for each testing resource (e.g., facilities, 
tools, and staff). 

* Quality assurance procedures: Defines a process for ensuring the 
quality of testing, including steps for recording anomalies or defects 
that arise during testing and steps for making changes to approved 
procedures. 

The plans for the nine CQT events and for the one SQT event were 
written by the prime contractor and reviewed by the SPO. These 10 
plans largely satisfied four of the seven key elements described above 
(see fig. 3 below). Specifically, each plan included the roles and 
responsibilities for personnel involved in the specific test event, 
and most of the ten plans described the environment and infrastructure 
necessary for testing, explicitly identified the tested items and 
approach, and included traceability matrices. However, two of the 
plans' mappings of requirements to test cases contained gaps or 
errors. Specifically, the NOC/SOC plan mapped 28 out of 100 
requirements to incorrect test cases. Further, the RTU plan did not 
map 6 out of 365 requirements to any test case, making it uncertain 
if, or how, these requirements were to be tested. 

Figure 3: Extent to Which Component and System Qualification Test 
Plans Satisfied Relevant Guidance: 

[Refer to PDF for image: illustrated table] 

Key Element: Roles and responsibilities; 
CQT: C31 COP: Satisfied all aspects of the key element; 
CQT: NOC/SOC: Satisfied all aspects of the key element; 
CQT: COMM: Satisfied all aspects of the key element; 
CQT: UGS: Satisfied all aspects of the key element; 
CQT: POWER: Satisfied all aspects of the key element; 
CQT: EOIR: Satisfied all aspects of the key element; 
CQT: NETWORK: Satisfied all aspects of the key element; 
CQT: RADAR: Satisfied all aspects of the key element; 
CQT: RTU: Satisfied all aspects of the key element; 
SQT: Satisfied all aspects of the key element. 

Key Element: Environmental and infrastructure; 
CQT: C31 COP: Satisfied all aspects of the key element; 
CQT: NOC/SOC: Satisfied some but not all aspects of the key element; 
CQT: COMM: Satisfied all aspects of the key element; 
CQT: UGS: Satisfied all aspects of the key element; 
CQT: POWER: Satisfied all aspects of the key element; 
CQT: EOIR: Satisfied all aspects of the key element; 
CQT: NETWORK: Satisfied all aspects of the key element; 
CQT: RADAR: Satisfied all aspects of the key element; 
CQT: RTU: Satisfied all aspects of the key element; 
SQT: Satisfied all aspects of the key element. 

Key Element: Tested items and approach; 
CQT: C31 COP: Satisfied all aspects of the key element; 
CQT: NOC/SOC: Satisfied some but not all aspects of the key element; 
CQT: COMM: Satisfied all aspects of the key element; 
CQT: UGS: Satisfied all aspects of the key element; 
CQT: POWER: Satisfied all aspects of the key element; 
CQT: EOIR: Satisfied all aspects of the key element; 
CQT: NETWORK: Satisfied all aspects of the key element; 
CQT: RADAR: Satisfied all aspects of the key element; 
CQT: RTU: Satisfied some but not all aspects of the key element; 
SQT: Satisfied all aspects of the key element. 

Key Element: Traceability Matrix; 
CQT: C31 COP: Satisfied all aspects of the key element; 
CQT: NOC/SOC: Satisfied some but not all aspects of the key element; 
CQT: COMM: Satisfied all aspects of the key element; 
CQT: UGS: Satisfied all aspects of the key element; 
CQT: POWER: Satisfied some but not all aspects of the key element; 
CQT: EOIR: Satisfied all aspects of the key element; 
CQT: NETWORK: Satisfied all aspects of the key element; 
CQT: RADAR: Satisfied all aspects of the key element; 
CQT: RTU: Satisfied some but not all aspects of the key element; 
SQT: Satisfied all aspects of the key element. 

Key Element: Risk and mitigations; 
CQT: C31 COP: Did not satisfy any aspects of the key element; 
CQT: NOC/SOC: Did not satisfy any aspects of the key element; 
CQT: COMM: Satisfied some but not all aspects of the key element; 
CQT: UGS: Satisfied some but not all aspects of the key element; 
CQT: POWER: Satisfied some but not all aspects of the key element; 
CQT: EOIR: Satisfied some but not all aspects of the key element; 
CQT: NETWORK: Satisfied some but not all aspects of the key element; 
CQT: RADAR: Satisfied some but not all aspects of the key element; 
CQT: RTU: Did not satisfy any aspects of the key element; 
SQT: Satisfied some but not all aspects of the key element. 

Key Element: Testing schedule; 
CQT: C31 COP: Satisfied all aspects of the key element; 
CQT: NOC/SOC: Did not satisfy any aspects of the key element; 
CQT: COMM: Did not satisfy any aspects of the key element; 
CQT: UGS: Did not satisfy any aspects of the key element; 
CQT: POWER: Did not satisfy any aspects of the key element; 
CQT: EOIR: Did not satisfy any aspects of the key element; 
CQT: NETWORK: Did not satisfy any aspects of the key element; 
CQT: RADAR: Did not satisfy any aspects of the key element; 
CQT: RTU: Did not satisfy any aspects of the key element; 
SQT: Did not satisfy any aspects of the key element. 

Key Element: Quality assurance procedures; 
CQT: C31 COP: Did not satisfy any aspects of the key element; 
CQT: NOC/SOC: Satisfied all aspects of the key element; 
CQT: COMM: Satisfied some but not all aspects of the key element; 
CQT: UGS: Satisfied some but not all aspects of the key element; 
CQT: POWER: Satisfied some but not all aspects of the key element; 
CQT: EOIR: Satisfied some but not all aspects of the key element; 
CQT: NETWORK: Satisfied all aspects of the key element; 
CQT: RADAR: Satisfied some but not all aspects of the key element; 
CQT: RTU: Satisfied some but not all aspects of the key element; 
SQT: Satisfied some but not all aspects of the key element. 

Did not satisfy any aspects of the key element; 
Satisfied some but not all aspects of the key element; 
Satisfied all aspects of the key element; 

Source: GAO analysis of DHS data. 

[End of figure] 

Further, most test plans did not fully satisfy any of the three other 
key elements. First, the plans did not describe any risks that would 
have adversely impacted the successful completion of specific tests, 
other than noting the safety risk to participants, even though program 
officials acknowledged such risks existed. For example, program 
officials told us that the NOC/SOC component was tested on a nonsecure 
laboratory network that does not resemble the environment in which it 
is to be fielded, which increases the risk that testing results will 
not accurately demonstrate the component's performance when deployed. 
Nonetheless, this test plan did not identify this risk and provide 
mitigation steps. Program officials agreed that the plans did not 
document testing risks, but stated that risks were generally discussed 
and documented during test readiness reviews, including a discussion 
of potential risks, and actions to mitigate them or a decision to 
accept them. However, our analysis of test readiness review documents 
for the SQT and CQT events showed that only the SQT documents 
discussed risk; the nine CQT documents did not. 

Second, nine test plans did not include or reference a schedule for 
activities that showed the estimated time necessary to perform each 
testing task or the required period of use for testing resources. This 
increases the risk that resources necessary for testing may not be 
available when needed. According to the SBInet independent 
verification and validation (IV&V)[Footnote 11] contractor, this 
actually was experienced. For example, the contractor reported that 
groups performing unrelated activities required access to the same 
resources needed for RTU testing, thus causing the test event to be 
interrupted several times because of competing demands for these 
resources. 

Lastly, the test plans did not adequately define quality assurance 
procedures. For example, while nine plans described procedures for 
recording anomalies or defects, eight plans did not include procedures 
for making changes to the plan during test execution. This is 
important because, as discussed below, test operators made numerous 
changes to the procedures in every test plan that we reviewed. 

Program officials told us they, too, had concerns about the quality 
and rigor of tests and documentation that was created by the prime 
contractor. Based on these concerns, they tasked the IV&V contractor 
with assessing the quality of the test documentation prepared by the 
prime contractor. Among other things, the IV&V contractor reported 
major deficiencies in the test plans, including requirements that had 
not been accurately mapped to test cases. Further, the IV&V contractor 
reported that certain tests did not fully exercise all requirements. 
For example, SQT did not include any security-related NOC/SOC 
requirements.[Footnote 12] 

These limitations in the test plans are attributable to a number of 
factors. Program officials told us that they did not have detailed 
guidelines or criteria for assessing the quality of the prime 
contractor's test-related deliverables, and they had insufficient time 
and resources to review these deliverables. Specifically, neither the 
TEMP nor the task orders sufficiently describe required content for 
test plans. Absent such guidance and resources, reviewers told us that 
they assessed testing documents based on their own knowledge and 
experiences as subject matter experts. Further, the IV&V contractor 
reported, and program officials confirmed, that the government waived 
the contractually required review period for one test plan, resulting 
in the SPO not reviewing the plan before the test event. Program 
officials told us that they also skipped other preparation activities, 
such as testing dry runs,[Footnote 13] due to time pressure. 

As discussed next, such limitations resulted in demonstrated problems 
during test execution. Further, they increased the risk that SBInet 
program testing was not sufficient to fully assess system performance. 

Test Cases Were Not Well Defined: 

According to relevant guidance,[Footnote 14] test cases are used to 
guide the execution of each test event. Among other things, well- 
defined test cases are to include the following: 

* Objective: The purpose of the test case. 

* Outcomes: The outputs and expected behaviors produced by executing 
the procedure, including exact values. 

* Procedure: An ordered description of the steps that must be taken to 
execute each test case. 

* Environment: The conditions needed for test setup; execution; and 
results recording, such as hardware and software items and their 
characteristics and configuration; the facilities to be used; or 
personnel requiring specific training. 

* Inputs: The information required to execute each test case, such as 
specific input values, files, tables, or databases. 

* Dependencies: The sequence in which test cases need to be executed, 
if applicable. 

Program officials told us that SQT and CQT test cases were developed 
by the contractor and reviewed by the SPO. Of the 12 SQT test cases, 
each one contained all applicable key elements. However, all 12 test 
cases required changes to the procedures in order to adequately 
exercise and verify the requirements being tested, as discussed in the 
next section of the report. 

In contrast to the SQT test cases, the 251 CQT test cases[Footnote 15] 
largely satisfied three of the six key elements, while less than one- 
half satisfied the remaining three elements (see figure 4). We 
estimate that 92 percent[Footnote 16] of component test cases listed 
the objective of the test case, and 90 percent[Footnote 17] of the 
test cases described the specific outputs and expected behavior of the 
test case. In addition, an estimated 97 percent[Footnote 18] of test 
cases included an ordered description of the procedure. However, as 
discussed in the next section of this report, over 70 percent of these 
procedures had to be revised during execution in order to fulfill the 
purpose of the test due to errors, omissions, or other problems. 

Figure 4: Estimated Percentages of CQT Test Cases Containing Key 
Elements Based on a Sample of 60: 

[Refer to PDF for image: horizontal bar graph] 

Key element: Objective: 92%; 
95 percent confidence level: 83%-97%. 

Key element: Outcomes: 90%; 
95 percent confidence level: 81%-96%. 

Key element: Procedure: 97%; 
95 percent confidence level: 90%-99%. 

Key element: Environment: 37%; 
95 percent confidence level: 26%-47%. 

Key element: Inputs: 32%; 
95 percent confidence level: 21%-42%. 

Key element: Dependencies: 8%; 
95 percent confidence level: 3%-17%. 

Source: GAO analysis of DHS data. 

[End of figure] 

Additionally, we estimate that only 37 percent[Footnote 19] of CQT 
test cases contained a complete description of the necessary 
environment, including the conditions specific to the test case's 
execution, such as the configuration of the component(s) being tested. 
Without a clear description of the test environment, the risk of 
procedures being executed in a manner that will not fulfill the test 
objective is increased. In fact, the IV&V contractor identified 
several RTU test cases without the initial component configuration 
documented that failed during testing and had to be redone. 

We also estimate that only 32 percent[Footnote 20] of CQT test cases 
included the inputs required to execute the test procedures. Not 
documenting the inputs to testing, such as specific input values, 
files, tables, or databases, makes it more difficult to reproduce test 
results, determine the root cause of related anomalies, and recreate 
successful test conditions. 

Further, we estimate that only 8 percent[Footnote 21] of CQT test 
cases identified other test cases upon which they were dependent. 
According to program officials, such dependencies may not have existed 
for all test cases. The IV&V contractor reported instances where the 
lack of clearly defined dependencies led to test cases either failing 
the first time and needing to be rerun, or test cases needing to be 
modified in order to proceed, thus resulting in unnecessary rework. 

As with program test plans, SBInet officials attributed limitations in 
the program's test cases to a lack of detailed guidelines in the TEMP, 
or other criteria for assessing the quality of the prime contractor's 
test-related deliverables, and to having insufficient time and 
resources to conduct their reviews. Additionally, by skipping testing 
dry runs for some test events, problems that would have been found 
during dry runs were not identified or corrected. For example, program 
officials told us that a tracking system had not been tested during 
dry runs, and as a result, this system failed during the execution of 
an SQT test case, requiring the test case to be rerun. Ultimately, 
these limitations increase the risk of not discovering system issues 
during testing and not demonstrating the system's ability to perform 
as intended when deployed. 

Testing Was Largely Not Executed According to Plans, and Changes to 
Plans Were Numerous, Extensive, and Not Always Appropriate: 

According to relevant guidance,[Footnote 22] effective testing 
includes, among other things, executing approved test procedures as 
written. If necessary, this guidance states that changes to such plans 
should be made in accordance with documented quality assurance 
procedures. 

In the case of SBInet, test procedures[Footnote 23] were largely not 
executed as written. Further, changes to these procedures were not 
made according to a documented quality assurance process. Rather, they 
were made based on an undocumented understanding that program 
officials said they established with the contractor. More 
specifically, all 12 of the SQT test cases, as well as 211 of the 299 
[Footnote 24] CQT test cases (a combined 72 percent) were not executed 
as planned. While some of these changes were relatively minor, such as 
changing the color of an indicator light or adding a step to "click 
OK" in a dialog box, others were more significant. For example, 
changes to the NOC/SOC or the RTU test procedures included the 
following: 

* rewriting the entire procedure to demonstrate installation of remote 
software updates (RTU); 

* rewriting the entire procedure to demonstrate the collection of 
network security logs (NOC/SOC); 

* crossing out a step confirming that sensor detections had been 
received and handwriting that these sensors had not been tested (RTU); 
and: 

* changing the mapping of requirements to test cases (NOC/SOC). 

Figure 5 on the following three pages illustrates the significant 
changes made to test cases using an example from the as-run log for 
the NOC/SOC test. As shown in the figure, test operators added 
requirements (as annotated in the B2 Specification Reference) and 
completely eliminated verification steps 8 through 11 and replaced 
them with 2 pages of handwritten verification steps. 

Figure 5: Excerpt from NOC/SOC As-Run Log Showing Remapped 
Requirements and Rewritten Procedures: 

[Refer to PDF for image: illustration] 

Boeing: 

D333-414082-1, New: 

6.2.4 Account Management - Group Policy. 

Test Case Number: 11.2.4. 

Engineer/SME: [Redacted]. 

Golden Bucket: Authentication. 

Test Objective Title: Group Policy - Session Lock. 
	
Test Objective Description: Demonstrate that the Group Policy for 
NOC/SOC workstations and servers lock sessions after 20 minutes of 
inactivity. 

B2 Specification Reference: NOCSOCB2-398; NOCSOCB2-410; NOCSOCB2-402. 

Function/Feature to Demonstrate: The test shall demonstrate the 
following account management functions:	
* Group Policy is set to initiate a session timeout after 15 minutes 
of system inactivity; 
* The ability to change session timeout. 
	
Resources Required: [Blank]. 

Success Criteria: 
The test will be considered successful when session timeout setting in 
the Group Policy is valid and session timeout occurs after 15 minutes. 

Test Procedure: 

Verify that the Group Policy is set: 
1. Log on with administrator credentials and open Group Policy 
Management. 
2. Navigate to Start: Control Panel: Admin Tools: Group Policy 
Management. 
3. Select and expand [forest]. 
4. Select default domain policy. 		
5. Right-click, then select edit. 			
6. Verify that the Group Policy Editor window opens. 
7. User config: Admin template: Control Panels: Display. 
8. Verify Screen Saver setting is enables. 
9. Verify Screen Saver executable name is set by double left-clicking 
to view name; c:\windows\system32\scrnsave.scr. 
10. Verify Password Protect Screen Saver setting is enabled. 
11. Verify Screen Saver Timeout is enabled. Double left-click to open 
and verify setting is 180 seconds (3 minutes). 
12. Close window and Do Not Touch keyboard or mouse for 15 minutes. 
13. Verify that the screen is locked after 15 minutes as indicated by 
the computer locked dialog box. This step must be visually witnessed 
as no screens can be seen. 
14. Press Control-Alt-Delete to unlock the computer. 
15. Enter password at prompt. 
16. Verify successful authentications. 
17. Initiate a manual session lock by pressing Control-Alt-Delete. 
18. Select Lock Computer and hit enter. 
19. Verify session was locked as indicated by Computer Locked dialog 
box. 

Test Conductor: [Redacted]. 

QA: [Redacted]. 

Government Witness: [Redacted]. 

Source: DHS. 

Note: Figure 5 has been redacted to, among other things, remove the 
names of individuals involved with SBInet testing activities. 

[End of figure] 

These numerous and extensive changes to test cases were not made based 
on documented quality assurance procedures, in part, because the SPO 
did not establish programwide guidance describing such procedures. 
Instead, program officials told us that they "have an understanding" 
with the contractor governing how such changes are to be made. 
[Footnote 25] According to the officials, this process allows test 
conductors to change the established procedures in order to make 
typographical corrections, insert omitted steps, or make technical 
adjustments as long as the contractor's testers and quality assurance 
officer and the government witness approve the changes before they are 
executed. Based on our review of executed procedure logs, this process 
was generally followed. 

Despite this general adherence to the process for approving test 
procedure changes, not all of the changes that were made and approved 
appear to have been appropriate, thus casting doubt on the quality of 
the tests and the reliability of the results. For example, in a letter 
to the prime contractor, dated April 2009, the SPO stated that SQT 
test cases required significant rework during execution, and the 
changes made to procedures appeared to be designed to pass the test 
instead of designed to qualify the system.[Footnote 26] Similarly, the 
IV&V contractor reported instances during CQT execution where 
procedures had been changed in order to expedite passing the test. For 
example, it reported that in an RTU component test case, testers 
reduced the time and area scanned by radar to cover only a small, 
predetermined test area, ignored other areas, and passed the test case 
based on this reduced scope and result. 

According to program officials, the numerous and extensive changes 
were due, in large part, to the program's aggressive schedule--
limiting the amount of time available to review and approve test 
cases--and the lack of testing dry runs performed. Additionally, 
officials stated that ambiguities in requirements caused testers to 
rewrite steps during execution based on interpretations of what they 
thought the requirements meant, which differed from that of those who 
wrote the original test procedures. 

In CBP's technical comments on a draft of this report, it acknowledged 
that the changes made to test procedures were excessive, but it 
characterized the SQT changes as largely "procedural steps" that "did 
not affect the verification activity." In our view, the volume and 
nature of the changes made to CQT and SQT test procedures, in 
conjunction with the lack of a documented quality assurance process, 
increases the risk that the procedures did not always support test 
objectives, exercise program requirements, or reflect the system's 
ability to perform as intended. This means that system problems may 
not be discovered until later in the sequence of testing, such as 
during acceptance or operational testing. 

SBInet Test Results Have Identified a Growing Number of System 
Performance and Quality Problems: 

Since March 2008, the number of new SBInet defects has increased 
faster than the number of defects that have been fixed, which is not a 
trend that is indicative of a maturing system. In addition, weaknesses 
in the SPO's management of defects make their full magnitude unclear. 
Nevertheless, problems have already been identified that were 
significant enough to delay the system's schedule and necessitate 
system rework. Further, users recently identified concerns about the 
system's operational suitability. 

Trends in the Number of Unresolved Defects Show That the System Is Not 
Maturing and Is Potentially at Risk: 

As previously reported,[Footnote 27] testing is intended to identify 
and resolve system quality and performance problems as early as 
possible in the system life cycle. SBInet testing has identified a 
range of problems. For example, 1,333 defects were recorded from March 
2008 to July 2009. During this time, the number of defects that have 
been resolved has generally been outpaced by the number of defects 
that have been identified. Figure 6 shows the trend in the number of 
open defects from March 2008 to July 2009. 

Figure 6: Trend in the Number of Open Defects from March 2008 to July 
2009: 

[Refer to PDF for image: line graph] 

Date: March 2008; 
Open defects: 24. 

Date: April 2008; 
Open defects: 63. 

Date: May 2008; 
Open defects: 69. 

Date: June 2008; (Begin Integration testing]; 
Open defects: 106. 

Date: July 2008; 
Open defects: 149. 

Date: August 2008; 
Open defects: 168. 

Date: September 2008; 
Open defects: 139. 

Date: October 2008; (End Integration testing, begin CQT); 
Open defects: 103. 

Date: November 2008; 
Open defects: 105. 

Date: December 2008; (End CQT, begin SQT); 
Open defects: 233. 

Date: January 2009; (End SQT); 
Open defects: 278. 

Date: February 2009; (Begin SQT regression, ongoing); (Begin CQT 
regression, ongoing); 
Open defects: 247. 

Date: March 2009; 
Open defects: 330. 

Date: April 2009; 
Open defects: 253. 

Date: May 2009; 
Open defects: 219. 

Date: June 2009; 
Open defects: 302. 

Date: July 2009; 
Open defects: 240. 

Source: GAO analysis of DHS data. 

[End of figure] 

As we have previously reported,[Footnote 28] such an upward trend is 
indicative of an immature system and can indicate a failure to meet 
system specifications. This is particularly problematic for SBInet 
because DHS expects final system acceptance in early February 2010. 

Significant Problems Discovered during Testing Have Caused Schedule 
Delays and Users Have Raised Other System Concerns: 

Defects found during development and testing have been significant 
enough to delay both the deployment of Block 1 system components to 
TUS-1 and AJO-1 and completion of SQT. Although the SPO reports that 
these problems have been resolved, they have caused lengthy program 
delays, and other problems have surfaced that continue to impact the 
program's schedule. Further, an early user assessment of Block 1 
operational suitability has raised additional concerns. 

Among the significant defects that have been identified to date, five 
that surfaced during initial SQT prompted the DHS Acquisition Review 
Board in February 2009 to postpone the installation of sensor and 
communication equipment on towers at TUS-1 and to delay site 
preparation and installation at AJO-1 until the problems were 
corrected. The five defects were: (1) the radar circuit breaker 
frequently tripped when the radar dish rotated beyond its intended 
limits, (2) COP workstations frequently crashed, (3) towers swayed 
beyond tolerable limits during adverse weather conditions, (4) radar 
clutter (i.e., false detections) occurred in adverse weather 
conditions, and (5) blurry camera images. As a result, sensor 
installation was delayed at TUS-1 by about 1 month. In May 2009, 
program officials reported to the Acquisition Review Board that they 
had either resolved or had installed operational workarounds for these 
five problems. Subsequently, installation resumed at TUS-1.[Footnote 
29] 

While program officials have characterized the defects and problems 
found during development and testing as not being "show 
stoppers,"[Footnote 30] they have nevertheless caused delays, extended 
testing, and required time and effort to fix. Moreover, the SPO and 
its contractor have continued to find problems that further impact the 
program's schedule. For example, the radar problems mentioned 
previously were addressed by installing a workaround that included a 
remote ability to reactivate the circuit breaker via software, which 
alleviated the need to send maintenance workers out to the tower to 
manually reset the circuit. However, this workaround did not fully 
resolve the problem, and program officials said that root cause 
analysis continues on related radar power spikes and unintended 
acceleration of the radar dish that occasionally render the system 
inoperable. While program officials recently told us that they believe 
that they have found a solution to this problem, as of October 2009, 
the solution was still being tested and verified. One factor that has 
contributed to the time and resources needed to resolve this radar 
problem, and potentially other problems, is the ability of the prime 
contractor to effectively determine root causes for defects. According 
to program officials, including the SBI Executive Director, the 
contractor's initial efforts to isolate the cause of the radar 
problems were flawed and inadequate. Program officials added, however, 
that they have seen improvements in the contractor's efforts to 
resolve technical issues. These radar problems have already caused SQT 
regression delays of about 4 months. 

Further, Border Patrol operators have identified a number of concerns 
with SBInet during an early user assessment. Specifically, from March 
27 to April 3, 2009, CBP conducted a user assessment to give Border 
Patrol operators a chance to interact with Block 1 functionality and 
offer feedback based on their experiences. During the assessment, 
Border Patrol operators compared the performance capabilities of 
existing technology--Project 28[Footnote 31] and Mobile Surveillance 
Systems (MSS)[Footnote 32]--to those of Block 1. The user assessment 
took place in a field test environment. According to the assessment 
report, dated July 1, 2009, weather conditions during the assessment 
were described as "favorable," including typical wind conditions, yet 
the report describes a number of less than optimal results. For 
example, while Border Patrol operators noted that Block 1 offered 
functionality above what Project 28 radar offers, this functionality 
was not adequate for optimal operational effectiveness when detecting 
items of interest. Moreover, users raised concerns about the accuracy 
of Block 1's radar, and they characterized the range of Block 1 
cameras as being operationally insufficient. Specifically, Block 1 
cameras were assessed as having one-half the range of MSS's cameras, 
and significantly less range than the Project 28 cameras. Further, 
Block 1's video quality was assessed as being inconsistent. Regarding 
the COP, operators considered Block 1's capabilities to be a 
significant improvement over both the Project 28 and MSS. However, 
they also raised concerns about the COP's accuracy and the need for a 
number of relatively "small, but critical enhancements." 

Program officials stated that some of the problems identified by users 
were attributable, in part, to the users' insufficient familiarity 
with Block 1. However, Border Patrol officials reported that the 
agents who participated in the assessment had experience with the MSS 
and/or Project 28 and that the agents received 2 days of training 
prior to the assessment. As a result, the Border Patrol reported that 
the issues and concerns generated should be considered operationally 
relevant. 

System Defects Not Being Prioritized and Thus Full Magnitude of 
Program Risks Is Not Clear: 

As we have previously reported,[Footnote 33] having complete and 
accurate defect information is necessary to adequately understand 
system maturity and to make informed decisions about how to best 
allocate limited resources to meet competing demands for resolving 
them. According to relevant guidance,[Footnote 34] effective defect 
management includes having a defined process that, among other things, 
assigns priority to each defect and ensures that the more severe 
defects are given priority attention. 

The SPO does not have a documented approach for prioritizing and 
managing the disposition of defects. Instead, program officials stated 
that they rely on the prime contractor to do so. However, under this 
approach, system defects have not been consistently assigned 
priorities, and in fact, a majority of them have not been prioritized. 
Specifically, when unplanned anomalies occur during development or 
testing, they are documented in contractor databases and classified as 
development incidents, test incidents, or "nonconformances."[Footnote 
35] The anomalies or defects are then reviewed to determine if they 
are related to hardware or software problems or if they are deviations 
from engineering or design specifications and managed separately. Of 
the three types of defects--hardware, software, and nonconformances--
only software defects are regularly assigned a priority. Each software 
defect's priority is assessed on a scale of 1 to 5. In general, 
category 1 and 2 software defects are those that impact the system's 
operation and thus must be resolved prior to beginning a subsequent 
test event and are likely to affect system testing. Categories 3 
through 5 software defects are those that have an available workaround 
or have very little or no impact on system performance or testing. 
Although hardware incidents and nonconformances are not assigned a 
priority under this approach, program officials referred to 
nonconformances as "must-fixes" because the contractor must address 
deviations from the way the system was designed. 

As a result of this approach, about 60 percent (or 801 of 1,333) of 
Block 1 defects identified from March 2008 to July 2009 were not 
assigned a priority (see fig. 7). Further, contrary to the program's 
stated approach of not assigning priorities to hardware and 
nonconformances, 104 hardware defects and 77 nonconformances were 
assigned priorities. 

Figure 7: Percentage of SBInet Defects with and without Assigned 
Priorities: 

[Refer to PDF for image: three pie-charts] 

Total defects: 
* Without priorities: 60%: Of the 60%: 
- Hardware: 49%; 
- Software: less than 1%; 
- Nonconformances: 50%; 
* With priorities: 40%; Of the 40%: 
- Hardware: 20%; 
- Software: 66%; 
- Nonconformances: 14%. 

Source: GAO analysis of DHS data. 

[End of figure] 

The lack of defect prioritization is partially attributable to the 
fact that the SPO has not defined a process governing how defects are 
to be prioritized and managed. Officials acknowledged this and stated 
that they intend to have the contractor prioritize all defects in 
advance of future test readiness reviews. They added that they will 
use these reviews to ensure all priority 1 and 2 defects associated 
with a given test event are corrected prior to beginning the event. 

Until system defects are managed on a priority basis, the SPO cannot 
fully understand Block 1's maturity or its exposure to related risks, 
nor can it make informed decisions about how best to allocate limited 
resources to address existing and future system defects. 

DHS Science and Technology Directorate's Testing Process Is Being Used 
to Leverage Maturing Technologies for SBInet: 

The SPO does not have its own process for testing SBInet-relevant 
technologies that are maturing or otherwise available from industry or 
other government entities, including the Department of Defense (DOD). 
Instead, the SPO relies on DHS's Science and Technology Directorate 
(S&T), whose mission is to provide technology solutions that assist 
DHS programs in achieving their missions.[Footnote 36] To accomplish 
its mission, S&T relies on an undocumented process. According to the 
S&T sensors and surveillance program manager, the following process is 
followed: S&T first interacts with its customers to develop system 
component thresholds, requirements, and metrics (e.g., the maximum 
false alarm rates or minimum range that a technology must have to be 
useful). Based on this information, the program manager said that S&T 
then develops a test plan and performs initial laboratory testing in 
conjunction with the source of the potential technology. If 
successful, testing is moved to an environment configured to replicate 
the given program's target operational environment. The final step in 
the process, as explained by the S&T program manager, is to perform a 
field-based operational evaluation in which users evaluate the 
technology. The program manager stated that if operational evaluation 
is successful, the technology is typically judged to be sufficiently 
mature to transition to the program office to incorporate into the 
system solution. 

To leverage S&T, CBP signed a multiyear Interagency Agreement with the 
directorate in August 2007. According to this agreement, S&T is to, 
among other things, research, develop, assess, test, and report on 
available and emerging technologies that could be incorporated into 
the SBInet system solution. S&T has, to date, focused its efforts on 
potential technologies to fill known performance gaps or to improve 
upon technology choices that were already made. One area of focus has 
been gaps in the radar system's ability to distinguish true radar 
hits, such as a person crossing the border, from false alarms, such as 
those caused by clutter[Footnote 37] during adverse weather 
conditions. In this regard, S&T reports that it has worked with 
industry to research and assess technology techniques for reducing 
clutter, as well as technology to help pinpoint the location of an 
item of interest along the border (e.g., laser illuminators). 
According to program officials, S&T is working with a contractor to 
develop enhanced imagery techniques to address camera performance gaps 
that testing found to occur during adverse weather conditions. 

When researching and assessing emerging or available border security 
technologies, S&T officials told us that they interact with DOD 
components and research entities, such as the Army's Night Vision and 
Electronic Sensors Directorate,[Footnote 38] the Institute for Defense 
Analysis,[Footnote 39] and the Massachusetts Institute of Technology's 
Lincoln Laboratory. However, these officials stated that defense- 
related technologies are not always a good fit with SBInet because DOD 
operations are very different from those of DHS. For instance, they 
stated that DHS systems are designed for fixed, permanent locations, 
and thus need to have a long life span with minimal maintenance. 
Conversely, according to S&T, DOD systems are designed for short-term, 
changing missions and thus need to be adaptable, and disposable. 
Additionally, the S&T program manager told us that adapting some DOD 
sensor technology for DHS use can be constrained by the fact that this 
equipment can be classified, and some Border Patrol agents do not have 
the necessary security clearance to use it. 

In addition, program officials told us that they too interact with 
several DOD components and research entities, including the Army's 
Night Vision and Electronic Sensors Directorate. As a result of the 
combined efforts of S&T and the SPO, these officials told us that the 
current SBInet system solution leverages some technologies that are 
used in DOD systems. For example, they said that the COP software was 
based on a software product developed by DOD, and DOD uses the same 
radar system as SBInet. 

Conclusions: 

Effective testing is integral to successfully acquiring and deploying 
a large-scale, complex system of system components, like SBInet. As 
such, it is important that testing of SBInet components and their 
integration be managed with the rigor and discipline embodied in 
relevant guidance. To do less unnecessarily increases the risk of 
problems going undetected until late in the system's life cycle, such 
as when it is being accepted for use, or even worse, after it becomes 
operational. When this happens, the result is either incurring 
expensive and time-consuming rework to bring the system's capabilities 
and performance up to the point that it meets user expectations, or 
accepting and using a system that cannot adequately support mission 
operations. 

SBInet testing has not been managed in a manner to adequately ensure 
that Block 1 will perform as intended. While aspects of SBInet test 
management were positive--such as its provision for testing system 
components and subsystems first, followed by a series of test events 
aimed at their integration and the overall system's ability to 
function as intended in settings that are increasingly more 
representative of the target operational environment--other aspects of 
test management were not. These test planning, execution, and 
recording weaknesses can, in large part, be attributed to a lack of 
guidance in the program's Test Evaluation Master Plan for assessing 
test documentation, including test plans and test cases, and 
sufficient time for reviewing and approving test documentation. 
Collectively, they have increased the risk that promised system 
capabilities and performance parameters have not been sufficiently 
tested to the point of providing reasonable assurance that the system 
will perform as intended prior to its acceptance and operational use. 

This risk is compounded by the trend in the number of unresolved 
system problems that the test events performed to date have 
identified. Moreover, the full magnitude of the existing inventory of 
system problems is not clear because the Test and Evaluation Master 
Plan does not include clear and sufficient guidance, and contractor 
direction has not required that all problems be categorized by 
severity, thus precluding adoption of a truly prioritized and risk-
based approach to resolving them. Given that the number of unresolved 
problems is well into the hundreds, and that problems that have been 
identified have contributed to lengthy program delays, a full 
understanding of the relative significance of all problems is critical 
to DHS's efforts to successfully deliver SBInet. 

Furthermore, the chances of identifying additional problems, and thus 
requiring extra time and effort to resolve them, are high because key 
events, such as system acceptance testing and operational test and 
evaluation, have yet to occur. It is thus vital for DHS to immediately 
strengthen its management of SBInet testing to include defect tracking 
and resolution. If it does not, further program delays can be 
expected, and the likelihood of Block 1 meeting user expectations and 
mission needs will be reduced. 

Recommendations for Executive Action: 

To improve DHS's management of SBInet testing, including the risk-
based resolution of current and to-be-detected system problems, we 
recommend that the Secretary of Homeland Security direct the 
Commissioner of the U.S. Customs and Border Protection to have the SBI 
Executive Director, in collaboration with the SBInet Program Director, 
take the following four actions: 

* Revise the SBInet Test and Evaluation Master Plan to include (1) 
explicit criteria for assessing the quality of test documentation, 
including test plans and test cases, and (2) a process for analyzing, 
prioritizing, and resolving program defects. 

* Ensure that test schedules, plans, cases, and procedures are 
adequately reviewed and approved consistent with the revised Test and 
Evaluation Master Plan. 

* Ensure that sufficient time is provided for reviewing and approving 
test documentation prior to beginning a given test event. 

* Triage the full inventory of unresolved system problems, including 
identified user concerns, and periodically report the status of the 
highest priority defects to Customs and Border Protection and 
Department of Homeland Security leadership. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Director, 
Departmental GAO/Office of Inspector General Liaison and reprinted in 
appendix II, DHS stated that our report is factually sound, and it 
acknowledged the management and system engineering challenges 
discussed in the report. Further, the department agreed with our last 
three recommendations and partially agreed with the first 
recommendation. In this regard, it described actions under way and 
planned to address them and provided milestones for doing so. In 
addition, it referenced technical comments that were separately 
provided, which we have incorporated in the report, as appropriate. 

Regarding its partial agreement with our first recommendation, DHS 
agreed that program documentation should include the information that 
we recommended, but did not agree that this information should be 
included in the Test and Evaluation Master Plan. Instead, the 
department stated that it plans to include the information in other 
program documents, such as the System Engineering Plan, Quality 
Management Plan, Configuration Management Plan, and detailed test 
plans. We support the department's decision to include this important 
information in key program documentation, and believe that its plans 
to do so are consistent with the intent of our recommendation. 
However, we would add that it is important that the Test and 
Evaluation Master Plan, which is intended to describe the overall test 
and evaluation approach, at least reference the documents that include 
the recommended information. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to interested congressional committees and other parties. We will also 
send copies to the Secretary of Homeland Security, the Commissioner of 
the U.S. Customs and Border Protection, and the Director of the Office 
of Management and Budget. In addition, this report will be available 
at no cost on the GAO Web site at [hyperlink, http://www.gao.gov]. 

Should you or your offices have any questions on matters discussed in 
this report, please contact me at (202) 512-3439 or at hiter@gao.gov. 
Contact points for our Office of Congressional Relations and Public 
Affairs may be found on the last page of this report. Key contributors 
to this report are listed in appendix III. 

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine (1) the extent to which testing has 
been effectively managed, including identifying the types of tests 
performed and whether they were well planned and executed; (2) what 
the results of testing show; and (3) what processes are being used to 
test and incorporate maturing technologies into SBInet. 

To identify the types of tests performed, we reviewed the program's 
overall test management approach as described in relevant 
documentation, such as the SBInet Test and Evaluation Master Plan 
dated November 2008; program management review documentation; 
component and system qualification test readiness reviews; and 
contract task orders. We also interviewed key program officials, 
including the Secure Border Initiative (SBI) Executive Director, the 
SBInet Program Director, the SBInet Director of Architecture and 
System Design, the SBInet Test Lead, and the SBInet Director of 
Business Operations. 

To determine if SBInet tests were well planned, we focused on the test 
events associated with the two types of testing that had recently been 
completed or were under way--the nine component qualification test 
events and the system qualification test event. For these events, we 
reviewed relevant documentation, such as test plans and test cases, 
and we compared them to relevant guidance. More specifically, we 
compared the plan for each of the 10 test events against key elements 
described in Institute of Electrical and Electronics Engineers test 
documentation guidance. We also reviewed a randomly selected sample of 
60 test cases from a universe of 251 component qualification test 
cases, and we reviewed all 12 of the system qualification test cases 
(a total of 263). In doing so, we compared these test cases to six key 
aspects of well-defined test cases, as specified in relevant guidance. 
We used statistical methods appropriate for audit compliance testing 
to estimate 95 percent confidence intervals for the test cases in our 
sample. Because we followed a probability procedure based on random 
selection, we are 95 percent confident that each of the confidence 
intervals in this report will include the true values in the study 
population. 

To determine if SBInet tests were well executed, we reviewed each of 
the 299 as-run procedures that had been executed as part of component 
qualification testing (CQT) plus the 13 that were executed as part of 
system qualification testing (SQT) to determine how many included 
changes made by test conductors to the printed procedures, and whether 
these changes had been approved by the contractor's quality assurance 
officer. This number differs from the number of planned test cases we 
reviewed (263) because some test cases had been added or deleted and 
some were executed multiple times. In addition, we interviewed program 
officials and prime contractor representatives to better understand 
the scope and content of the test cases and reasons for any 
deviations. We also interviewed officials from the independent 
verification and validation contractor and reviewed related reports to 
determine their role in test planning and execution. Further, we 
visited the prime contractor's site in Huntsville, Alabama, to 
interview program and contractor officials to understand how system 
requirements related to test cases and procedures. 

To determine what the results of SBInet testing show, we reviewed 
relevant documentation, such as component and system qualification 
test reports, program management reviews, program office briefings, 
and Department of Homeland Security (DHS) Acquisition Review Board 
decision memoranda. We also reviewed and analyzed program data about 
open (i.e., unresolved) system problems (i.e., hardware and software 
defects and nonconformances) for the 17-month period beginning on 
March 2008 and ending July 2009 to determine the trend in the number 
of problems. Further, we analyzed the program's approach to managing 
these defects, including how they were prioritized for resolution. In 
addition, for those problems that were reported to the DHS Acquisition 
Review Board as significant, we analyzed the nature of the defects and 
tracked steps taken to address them and the extent to which these and 
other problems affected the program's schedule. We also interviewed 
program officials and prime contractor representatives to better 
understand the scope and nature of the selected problems and their 
impacts on system performance and program progress. To assess the 
reliability of the defect data that we used, we reviewed quality and 
access controls over the automated tools used to manage defect data 
(i.e., TipQA and ClearQuest). We determined that the data were 
sufficiently reliable for the purposes of this report. 

To determine the process used to test and incorporate maturing 
technologies into SBInet, we reviewed relevant SBInet documentation, 
such as a contract task order, the fiscal year 2009 SBI Expenditure 
Plan, and the Interagency Agreement between the program office and 
DHS's Science and Technology Directorate (S&T). In addition, we 
reviewed documentation provided by the directorate related to SBInet 
and border surveillance technologies, such as technical reviews of 
cameras conducted by the U.S. Army's Night Vision and Electronic 
Sensors Directorate and S&T briefings on border technology. We also 
interviewed directorate and program officials to better understand the 
processes being followed, and to identify examples of where 
technologies used by the Department of Defense had been incorporated 
into SBInet. 

We performed our work at the U.S. Customs and Border Protection 
headquarters, DHS Science and Technology Directorate, and prime 
contractor facilities in the Washington, D.C., metropolitan area and 
at a prime contractor facility in Huntsville, Ala. We conducted this 
performance audit from December 2008 to January 2010 in accordance 
with generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings 
and conclusions based on our audit objectives. We believe that the 
evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Homeland Security: 

U.S. Department of Homeland Security: 
Washington, DC 20528: 

December 28, 2009: 

Mr. Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Hite: 

Re: Draft Report GAO-10-158, Secure Border Initiative: DHS Needs to 
Address Testing and Performance Limitations that Place Key Technology 
Program at Risk (GAO Job Code 310674). 

The Department of Homeland Security (DHS) appreciates the opportunity 
to review and comment on the U.S. Government Accountability Office's 
(GAO's) draft report referenced above. GAO was asked to, among other 
tasks, determine (1) whether SBInet testing has been effectively 
managed, including the types of tests performed and whether they were 
well planned and executed and (2) what the results of testing show. 
GAO reviewed test management documentation, including test plans, test 
cases, test procedures, and results relative to federal and related 
guidance, and interviewed program and contractor officials to meet its 
objectives. 

GAO recommends that the Secretary direct the U.S. Customs and Border 
Protection (CBP) Commissioner to have the Secure Border Initiative 
(SBI) Executive Director, in collaboration with the SBInet Program 
Director, take four actions to improve DHS's management of SBInet 
testing, including the risk-based resolution of current and to-be-
detected system problems. CBP concurs with the last three 
recommendations and partially concurs with the first one. 

CBP acknowledges that the GAO draft report overall is factually sound. 
However, some aspects of various sections were clarified through 
technical comments that have been separately provided for the purpose 
of correctness and completeness. The draft report also highlights 
several management and systems engineering challenges that CBP is 
working to resolve as expeditiously as possible. To address the 
challenges described in the draft report, CBP has initiated a series 
of actions to improve both the management and systems engineering 
processes for SBInet. 

The GAO appropriately acknowledges SBInet efforts over the past year 
to implement a progressive test approach. CBP believes the SBInet test 
philosophy that highlights a confidence building approach is sound and 
in accordance with best practices. The testing regime starts with 
component-level testing, followed by testing of subsystems, followed 
by end-to-end system level testing under lab conditions, followed by 
end-to-end system level testing in an operational environment, and 
ends with an operational test in the end-user environment. The test 
regime has proven to be successful in identifying component and system 
deficiencies and limitations that have been addressed by the 
contractor and will undergo regression testing before the system is 
accepted. 

Recommendation 1: 

Revise the SBInet Test and Evaluation Master Plan to include (a) 
explicit criteria for assessing the quality of test documentation, 
including test plans and test cases, and (b) establish a process for 
analyzing, prioritizing, and resolving program defects. 

Response: 

CBP partially concurs with the recommendation. While CBP agrees that 
the specific data described should be included, CBP contends that the 
Test and Evaluation Master Plan is not a detailed test plan and should 
not contain the specific data described within the recommendation. 
That information will be provided in other program documentation, such 
as the Systems Engineering Plan, the Quality Assurance Plan, the 
Configuration Management Plan, and the Detailed Test Plans that will 
document the specific processes used to assess test documentation and 
defect management. The estimated time for completion of agreed upon 
corrective action is June 30, 2010. 

Recommendation 2: 

Ensure that test schedules, plans, cases, and procedures are 
adequately reviewed and approved consistent with the revised Test and 
Evaluation Master Plan. 

Response: 

CBP agrees with the recommendation. Test artifacts are reviewed both 
in advance and at each test event's Test Readiness Review (TRR) to 
ensure alignment with the Program's Integrated Master Schedule, which 
incorporates changes to the approved Test and Evaluation Master Plan 
schedule. The SBI Systems Engineering (SE) Directorate was created to 
ensure systems engineering processes are being properly defined, 
executed, and monitored, the Technical Management Division within the 
SE Directorate is the entity responsible for ensuring verification and 
validation processes are being executed and system engineering best 
practices are being incorporated. CBP officials stated that the due 
date at System Acceptance Test (SAT) TRR is estimated to be by 
September 2010. 

Recommendation 3: 

Ensure that sufficient time is provided for reviewing and approving 
test documentation prior to beginning a given test event. 

Response: 

CBP agrees with the recommendation. Test documentation from the prime 
contractor (Boeing) is required to be delivered to SBInet 15 days 
prior to the event's TRR. The Technical Management Division also 
ensures strategic technical issues/tenets drive the program management 
structure and decision framework. The Technical Management Division is 
responsible for ensuring Institute of Electrical and Electronics 
Engineers or equivalent Systems Engineering and Software Development 
standards are used in developing the SBI product suite. CPB officials 
noted that the due date at System Acceptance Test TRR is estimated to 
be by September 2010. 

Recommendation 4: 

Triage the full inventory of unresolved system problems, including 
identified user concerns and periodically report the status of the 
highest priority defects to CBP and DHS leadership. 

Response: 

CBP agrees with the recommendation. Unresolved system problems are 
addressed as part of the program's risk management process and 
biweekly risk review boards. The key risks and issues are also 
identified to the program's governance bodies via monthly program 
status briefings to the CBP Executive Steering Committee and monthly 
reports to DHS using the Next Generation Periodic Reporting System. 
Additionally, SBInet is planning to conduct an assessment of existing 
capability limitations as defined by open Non-Conformance Reports, 
Design Incident Reports, and Test Incident Reports within the TipQa 
database as part of the analysis that will be used to determine what 
capabilities or improvements should be made to the system baseline. 
Corrective action should be completed by June 30, 2010. 

Sincerely, 

Signed by: 

Michael E. McPoland, for: 

Jerald E. Levine: 
Director: 
Departmental GAO/OIG Liaison Office: 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439 or hiter@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Deborah Davis (Assistant 
Director), Carl Barden, James Crimmer, Neil Doherty, Lauren Giroux, 
Nancy Glover, Dan Gordon, Lee McCracken, Sushmita Srikanth, and 
Jennifer Stavros-Turner made key contributions to this report. 

[End of section] 

Footnotes: 

[1] At a port of entry location, CBP officers secure the flow of 
people and cargo into and out of the country, while facilitating 
travel and trade. 

[2] See, for example, GAO, Best Practices: A More Constructive Test 
Approach Is Key to Better Weapon System Outcomes, [hyperlink, 
http://www.gao.gov/products/GAO/NSIAD-00-199] (Washington, D.C.: July 
31, 2000); Office of Personnel Management: Improvements Needed to 
Ensure Successful Retirement Systems Modernization, [hyperlink, 
http://www.gao.gov/products/GAO-08-345] (Washington, D.C.: Jan. 31, 
2008); and Secure Border Initiative: DHS Needs to Address Significant 
Risks in Delivering Key Technology Investment, [hyperlink, 
http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22, 
2008). 

[3] In addition to the SBI Program Executive Office, the SBI 
Acquisition Office is responsible for performing contract 
administration activities. 

[4] [hyperlink, http://www.gao.gov/products/GAO-08-1086]. 

[5] GAO, Secure Border Initiative: Technology Deployment Delays 
Persist and the Impact of Border Fencing Has Not Been Assessed, 
[hyperlink, http://www.gao.gov/products/GAO-09-896] (Washington, D.C.: 
Sept. 9, 2009). 

[6] The DHS Acquisition Review Board is the departmental executive 
board that reviews programs with life cycle costs of $300 million or 
more for proper management, oversight, accountability, and alignment 
to strategic functions of the department. The board reviews 
investments before approving them to proceed to the next phase of 
acquisition. 

[7] See, for example, Office of the Under Secretary of Defense for 
Acquisition, Technology, and Logistics, Department of Defense 
Instruction 5000.02 (Arlington, Va., Dec. 8, 2008); Department of 
Homeland Security, Acquisition Instruction/Guidebook #102-01-001: 
Appendix B, Interim version 1.9 (Nov. 7, 2008); Software Engineering 
Institute, Capability Maturity Model" Integration for Acquisition, 
version 1.2 (Pittsburgh, Penn., November 2007); Institute of 
Electrical and Electronics Engineers, Inc., Standard for Software 
Verification and Validation, IEEE Std. 1012-2004 (New York, N.Y., June 
8, 2005); Defense Acquisition University, Test and Evaluation 
Management Guide, 5th ed. (Fort Belvoir, Va., January 2005); and GAO, 
Year 2000 Computing Crisis: A Testing Guide, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-10.1.21] (Washington, D.C.: 
November 1998). 

[8] The TEMP defines the program's integrated test and evaluation 
approach, including the scope of testing and the staff, resources 
(equipment and facilities), and funding requirements associated with 
testing. 

[9] In CBP's technical comments on a draft of this report, it 
characterized integration testing and CQT as being conducted in 
parallel and not sequentially. 

[10] Institute of Electrical and Electronics Engineers, Inc., Standard 
for Software and System Test Documentation, IEEE Std. 829-2008 (New 
York, N.Y., July 18, 2008). 

[11] In 2008, the SPO contracted with an IV&V agent to, among other 
things, further review the program's test documentation, execution, 
and related processes. Generally, the purpose of IV&V is to 
independently ensure that program processes and products meet quality 
standards. The use of an IV&V function is recognized as an effective 
practice for large and complex system development and acquisition 
programs, like SBInet, as it provides objective insight into the 
program's processes and associated work products. 

[12] In CBP's technical comments on a draft of this report, it stated 
that NOC/SOC security requirements were included in a subsequent phase 
of SQT testing and that testing of these requirements is also to occur 
as part of TUS-1 security certification and accreditation. 

[13] Testing dry runs are conducted to ensure the test setup and 
procedures are mature enough to proceed into formal test events. 

[14] Institute of Electrical and Electronics Engineers, Inc., IEEE 
Std. 829-2008. 

[15] We reviewed a randomly selected sample of 60 of the 251 component 
qualification test cases. 

[16] The 95 percent confidence interval for this estimate is 83 
percent to 97 percent. 

[17] The 95 percent confidence interval for this estimate is 81 
percent to 96 percent. 

[18] The 95 percent confidence interval for this estimate is 90 
percent to 99 percent. 

[19] The 95 percent confidence interval for this estimate is 26 
percent to 47 percent. 

[20] The 95 percent confidence interval for this estimate is 21 
percent to 42 percent. 

[21] The 95 percent confidence interval for this estimate is 3 percent 
to 17 percent. 

[22] Software Engineering Institute, Capability Maturity Model" 
Integration (CMMI) for Acquisition, version 1.2 (November 2007) and 
Institute of Electrical and Electronics Engineers, Inc., IEEE Std. 829-
2008. 

[23] As mentioned previously, test procedures are an ordered 
description of the steps that must be taken by each participant. 

[24] We stated earlier in this report that there were 263 test cases-- 
12 from SQT and 251 from CQT. The difference between the 299 test 
cases cited here and the 251 figure cited previously is due to, among 
other things, test cases that were executed multiple times. 

[25] During development testing on SBInet, the contractor's staff 
performed the actual tests. The contractor also provided quality 
assurance staff and an on-site test director. Government staff served 
as on-site witnesses during CQT and SQT events. 

[26] As stated previously, system qualification is designed to verify 
that the system design satisfies system-level requirements. 

[27] GAO, Customs Service Modernization: Automated Commercial 
Environment Progressing, but Further Acquisition Management 
Improvements Needed, [hyperlink, 
http://www.gao.gov/products/GAO-03-406] (Washington, D.C.: Feb. 28, 
2003) and Best Practices: A More Constructive Test Approach Is Key to 
Better Weapon System Outcomes, [hyperlink, 
http://www.gao.gov/products/GAO-NSIAD-00-199] (Washington, D.C.: July 
31, 2000). 

[28] [hyperlink, http://www.gao.gov/products/GAO-08-345]. 

[29] Installation at AJO-1 was expected to begin in late 2009. 

[30] David Aguilar, Chief of the U.S. Border Patrol, testified to the 
extent of problems found during testing to the House of 
Representatives Committee on Homeland Security, Subcommittee on 
Border, Maritime, and Global Counterterrorism on September 17, 2009. 

[31] The first SBInet capabilities were deployed under a pilot or 
prototype effort known as Project 28. 

[32] The Mobile Surveillance Systems are the current technology 
systems used to supplement fixed surveillance assets (such as towers 
and cameras) to help detect, classify, and track items of interest 
along the border. 

[33] [hyperlink, http://www.gao.gov/products/GAO-08-345]. 

[34] DHS, Acquisition Instruction/Guidebook #102-01-001: Appendix B, 
Interim version 1.9, (November 7, 2008) and Institute of Electrical 
and Electronics Engineers, Inc., IEEE Std. 829-2008. 

[35] According to contractor documentation, a development incident is 
a system incident (software), test equipment issue (hardware), 
requirement failure, or engineering unit failure identified during 
system development or informal (nongovernment witnessed) testing; a 
test incident is a system incident, hardware or software problem, test 
equipment issue, or requirement failure identified during formal 
testing; and nonconformances, which can be identified at any time, are 
problems with the reliability or quality of system hardware or 
software components, or any other issues that cause the system or 
component to not meet engineering or design specifications. 

[36] The SPO originally relied on its prime contractor to leverage 
such technologies. According to the program management task order, 
which was executed in April 2008, the prime contractor was to (1) 
collaborate with CBP and the SPO to develop an SBInet technology road 
map; (2) assess system architecture capabilities and recommend future 
technological enhancements; (3) identify evolving system requirements, 
gaps in capabilities, and potential solutions; and (4) identify new 
commercial products for program use. Program officials stated that 
this task order was modified, in part, to eliminate funding for these 
activities in order to focus the contractor on current development and 
deployment issues. 

[37] Clutter refers to items such as blowing trees, weather, or 
general interference, which radar or cameras may interpret as items of 
interest. 

[38] The Night Vision and Electronic Sensors Directorate is the Army's 
research and development laboratory for night vision and other 
advanced sensor technologies. 

[39] The Institute for Defense Analysis is a nonprofit corporation 
that administers three federally funded research and development 
centers to provide objective analyses of national security issues. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: