This is the accessible text file for GAO report number GAO-11-6 entitled 'Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor' which was released on October 18, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: October 2010: Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor: GAO-11-6: GAO Highlights: Highlights of GAO-11-6, a report to congressional requesters. Why GAO Did This Study: The Department of Homeland Security’s (DHS) Secure Border Initiative Network (SBInet) is to place surveillance systems along our nation’s borders and provide Border Patrol command centers with the imagery and related tools and information needed to detect breaches and make agent deployment decisions. To deliver SBInet, DHS has relied heavily on its prime contractor. Because of the importance of effective contractor management and oversight to SBInet, GAO was asked to determine the extent to which DHS (1) defined and implemented effective controls for managing and overseeing the prime contractor and (2) effectively monitored the contractor’s progress in meeting cost and schedule expectations. To do this, GAO analyzed key program documentation against relevant guidance and best practices, and interviewed program officials. What GAO Found: DHS has largely defined but has not adequately implemented the full range of controls that is reflected in relevant guidance and related best practices and is needed to effectively manage and oversee its SBInet prime contractor. To the department’s credit, it has defined a number of key policies and procedures for verifying and accepting contract deliverables and conducting technical reviews, such as the criteria that need to be met before commencing and concluding a critical design review. Moreover, it has implemented some of these defined practices, such as those associated with training key contractor management and oversight officials. However, DHS has not effectively implemented other controls. For example, it has not adequately documented its review of contract deliverables and communicated to the prime contractor the basis for rejecting certain deliverables. Further, it has not ensured that key documentation satisfied relevant criteria before concluding key technical reviews. These weaknesses can be attributed in part to limitations in the defined verification and acceptance deliverable process, a program office decision to exclude certain deliverables from the process, and insufficient time to review technical review documentation. All told, DHS has not effectively managed and overseen its SBInet prime contractor, thus resulting in costly rework and contributing to SBInet’ s well-chronicled history of not delivering promised capabilities and benefits on time and within budget. DHS has not effectively monitored the SBInet prime contractor’s progress in meeting cost and schedule expectations. While DHS has used earned value management (EVM), which is a proven management approach for understanding program status and identifying early warning signs of impending schedule delays and cost overruns, it has not ensured that its contractor has effectively implemented EVM. In particular, DHS has not ensured that validated performance baselines (the estimated value of planned work against which performance is measured) were timely, complete, and accurate. For example, the contractor was allowed to perform work on task orders for several months without a validated baseline in place. Further, not all work to be performed was included in the baselines that were eventually established, and the schedules for completing this work did not substantially comply with six of the eight key practices that relevant guidance states are important to having a reliable schedule. Also, DHS regularly received incomplete and anomalous EVM data from the prime contractor, which it had to rely on to measure progress and project the time and cost to complete the program. As a result, DHS has not been able to gain meaningful and proactive insight into potential cost and schedule performance shortfalls, and thus take corrective actions to avoid shortfalls in the future. Program officials attributed these weaknesses in part to the instability in the scope of the work to be performed, and the contractor’s use of estimated, rather than actual, costs for subcontractor work and the subsequent adjustments that are made when actual costs are received. This inability has contributed to the program’s failure to live up to expectations and to it costing more and taking longer than was necessary. What GAO Recommends: GAO is making recommendations to DHS aimed at revising and implementing policies and procedures related to contractor deliverables and technical reviews, and improving EVM baselines and data. DHS agreed with GAO’s recommendations and described actions to address them, but took exception with selected findings and conclusions regarding EVM implementation. GAO stands by these findings and conclusions for reasons discussed in the report. View [hyperlink, http://www.gao.gov/products/GAO-11-6] or key components. For more information, contact Randolph C. Hite at (202) 512-3439 or hiter@gao.gov. [End of section] Contents: Letter: Background: DHS Has Largely Defined but Has Not Effectively Implemented the Full Range of Needed Contractor Management and Oversight Controls: DHS Has Not Effectively Monitored the Contractor's Progress in Meeting Cost and Schedule Expectations: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Homeland Security: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Summary of SBInet Task Orders as of May 2010: Table 2: Percentage of Performance Measurement Baseline Work Tasks Categorized as Level-of-Effort for Six Baselines: Figures: Figure 1: Partial High-Level Depiction of SBI Program Organization: Figure 2: Partial High-Level Depiction of SBI Contracting Division Organization: Figure 3: Summary of Contract Deliverables That Did and Did Not Follow the Defined Process: Figure 4: Summary of Time Frames With and Without Validated Performance Baselines from June 2008 through February 2010: Figure 5: Summary of IBR Schedules' Satisfaction of Schedule Estimating Practices: Figure 6: Percentage of Work Breakdown Structure Elements with EVM Data Anomalies by Month for Each of the Four Task Orders: Figure 7: Total Number of Data Anomalies and Number of Explained Anomalies in the Monthly EVM Reports across the Four Task Orders: Abbreviations: ADTO: Arizona Deployment Task Order: AJO-1: Ajo Border Patrol Station: ANSI: American National Standards Institute: CBP: U.S. Customs and Border Protection: CDR: Critical Design Review: CMMI®: Capability Maturity Model Integration®: COP: common operating picture: COTR: contracting officer's technical representative: C3I: command, control, communications, and intelligence: DCMA: Defense Contract Management Agency: DTO: Design Task Order: DHS: Department of Homeland Security: EVM: earned value management: FAC-C: Federal Acquisition Certification in Contracting: IBR: integrated baseline review: IDIQ: Indefinite Delivery/Indefinite Quantity: IV&V: independent verification and validation: JPMR: Joint Program Management Review: NOC/SOC: Network Operations Center/Security Operations Center: SEP: Systems Engineering Plan: SBI: Secure Border Initiative: SBInet: Secure Border Initiative Network: SEI: Software Engineering Institute: SPO: SBInet System Program Office: STO: System Task Order: TUS-1: Tucson Border Patrol Station: WBS: work breakdown structure: [End of section] United States Government Accountability Office: Washington, DC 20548: October 18, 2010: The Honorable Bennie G. Thompson: Chairman: Committee on Homeland Security: House of Representatives: The Honorable Christopher P. Carney: Chairman: The Honorable Gus M. Bilirakis: Ranking Member Subcommittee on Management, Investigations, and Oversight: Committee on Homeland Security: House of Representatives: The Honorable Mike Rogers: Ranking Member: Subcommittee on Emergency Communications, Preparedness, and Response: Committee on Homeland Security: House of Representatives: To enhance border security and reduce illegal immigration, the Department of Homeland Security (DHS) launched its multiyear, multibillion dollar Secure Border Initiative (SBI) program in November 2005. Through SBI, DHS intends to enhance surveillance technologies, increase staffing levels, enforce immigration laws, and improve the physical infrastructure along our borders. Within SBI, the Secure Border Initiative Network (SBInet) is a multibillion-dollar program that involves the acquisition, development, integration, deployment, and operation and maintenance of surveillance technologies to create a "virtual fence" along the border as well as command, control, communications, and intelligence (C3I) technologies to create a picture of the border in command centers and vehicles. DHS's strategy is to deliver SBInet capabilities incrementally. In doing so, the department has relied heavily on its prime contractor-- the Boeing Company. Because of the importance of effective contractor management and oversight to the successful deployment and operation of SBInet capabilities, you asked us to determine to what extent DHS (1) has defined and implemented effective controls for managing and overseeing the SBInet prime contractor and (2) is effectively monitoring the prime contractor's progress in meeting cost and schedule expectations. To accomplish our objectives, we focused on DHS's management and oversight of four key task orders associated with designing, developing, and deploying the first increment of SBInet, known as Block 1, to its initial deployment sites in Arizona--the Tucson Border Patrol Station (TUS-1) and the Ajo Border Patrol Station (AJO-1). In doing so, we reviewed DHS and program guidance, policies, and plans for managing and overseeing the contractor; task orders and associated modifications; contract deliverable review comments; contractor schedules; the work breakdown structure governing all task orders; integrated baseline review documents for each task order; and task order contract cost performance reports. We also analyzed a nonprobability, random sample of 28 contract deliverables; a nonprobability, random sample of 8 technical reviews conducted with the contractor; and a nonprobability sample of 11 action items identified during program management reviews.[Footnote 1] In addition, we interviewed program officials about contractor management and oversight activities, such as verifying and accepting contract deliverables and conducting technical and management reviews. We also interviewed program officials about the prime contractor's cost and schedule performance. We then compared this information to relevant guidance and leading industry practices on contractor management and oversight, schedule estimating, and earned value management (EVM), [Footnote 2] such as the Software Engineering Institute's (SEI) Capability Maturity Model Integration® (CMMI®) for Acquisition, GAO's Cost Estimating and Assessment Guide, and the American National Standards Institute (ANSI) standard.[Footnote 3] We conducted this performance audit from June 2009 to October 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Further details of our objectives, scope, and methodology are in appendix I. Background: Managed by U.S. Customs and Border Protection (CBP), SBInet is to strengthen DHS's ability to detect, identify, classify, track, and respond to illegal breaches at and between ports of entry. It includes the acquisition, development, integration, deployment, and operations and maintenance of a mix of surveillance technologies, such as cameras, radars, sensors, and C3I technologies. Surveillance technologies include unattended ground sensors that can detect heat and vibrations associated with foot traffic and metal associated with vehicles, radar mounted on fixed towers that can detect movement, and cameras mounted on fixed towers that can be used to identify and classify items of interest detected and tracked by ground sensors and radar. These technologies are generally commercial off-the-shelf products. C3I technologies include customized software development to produce a common operating picture (COP)--a uniform presentation of activities within specific areas along the border--as well as the use of CBP network capabilities. Together, the surveillance technologies are to gather information along the border and transmit this information to COP terminals located in command centers, which are to assemble it to provide CBP agents with border situational awareness, to, among other things, enhance agents' tactical decisionmaking regarding potential apprehensions. Since fiscal year 2006, DHS has received about $4.4 billion in appropriations for SBI, including about $2.5 billion for physical fencing and related infrastructure, about $1.5 billion for virtual fencing (e.g., surveillance systems) and related infrastructure (e.g., towers), and about $300 million for program management.[Footnote 4] As of May 2010, DHS had obligated about $1.02 billion for SBInet. Program and Acquisition Organizational Structure: The SBI Program Management Office, which is organizationally within CBP, is responsible for managing key acquisition functions associated with SBInet, including prime contractor management and oversight. It is organized into four components: the SBInet System Program Office (SPO), Operational Integration Division, Business Operations Division, and Systems Engineering Division.[Footnote 5] The SPO is responsible for such key contractor oversight activities as verifying and accepting contract deliverables and conducting contractor management and technical reviews. In addition, the Business Operations Division has primary responsibility for monitoring the prime contractor's cost and schedule performance. As of May 2010, the SBI Program Management Office was staffed with 194 people--105 government employees, 78 contractor staff, and 11 detailees. (See figure 1 for a partial SBI program organization chart.) Figure 1: Partial High-Level Depiction of SBI Program Organization: [Refer to PDF for image: organization chart] Top level: Program Management Office: * Executive Director; * Deputy Executive Director (reports to Executive Director). Second level, reporting to Executive Director: * Operational Integration Division; * Business Operations Division; * Systems Engineering Division. Second level, reporting to Executive Director: SBInet System Program Office: * Executive Director; * Deputy Director (reports to Executive Director). Third level, reporting to Deputy Director: * Systems Solution Division; * Functional Engineering and Environmental Resources Division; * Southwest Border Deployment Division; * Northern Border Division. Source: GAO analysis based on DHS data. [End of figure] In addition, CBP has engaged the Defense Contract Management Agency to, among other things, perform surveillance of the prime contractor's EVM, systems engineering, hardware and software quality assurance, and risk management. Further, the SBI Contracting Division, which is organizationally within the CBP Office of Administration Procurement Directorate's Enterprise Contracting Office, is responsible for performing contract administration activities, such as maintaining the contract file and notifying the contractor in writing of whether deliverables are accepted or rejected. The SBI Contracting Division is organized into three branches: SBI Contracting, SBInet System Contracting, and SBInet Deployment Contracting. As of May 2010, the SBI Contracting Division was staffed with 22 people--18 government employees and 4 contractor staff. (See figure 2 for a partial SBI Contracting Division organization chart.) Figure 2: Partial High-Level Depiction of SBI Contracting Division Organization: [Refer to PDF for image: organization chart] Top level: * Office of Administration Procurement Directorate: - Executive Director. Second level, reporting to Office of Administration Procurement Directorate: * Enterprise Contracting Office: - Deputy Executive Director. Third level, reporting to Enterprise Contracting Office: * SBI Contracting Division; - Division Director. Fourth level, reporting to SBI Contracting Division: * SBI Contracting Branch; * SBInet System Contracting Branch; * SBInet Deployment Contracting Branch. Source: GAO analysis based on DHS data. [End of figure] Overview of SBInet Acquisition Strategy and Program Status: In September 2006, CBP awarded an Indefinite Delivery/Indefinite Quantity (IDIQ)[Footnote 6] prime contract to the Boeing Company. The contract was for 3 base years with three additional 1-year options for designing, producing, testing, deploying, and sustaining SBI. In September 2009, CBP exercised the first option year. Under the prime contract, CBP has issued 11 task orders that relate to SBInet, covering for example, COP design and development, system deployment, and system maintenance and logistics support. As of May 2010, 5 of the 11 task orders were complete and 6 were ongoing. (See table 1 for a summary of the SBInet task orders.) Table 1: Summary of SBInet Task Orders as of May 2010: Completed task orders: Task order: Program Management; Description: Completed task orders: Mission engineering, facilities and infrastructure, systems engineering, test and evaluation, and program management services; Period of performance: Completed task orders: Sept. 2006-Apr. 2008; Approximate contract value: $146.9 million; Approximate contract obligation: $146.9 million; Contract type: Cost-plus-fixed-fee[A]. Task order: Project 28; Description: Completed task orders: Prototype along 28 miles of the border in the Tucson sector; Period of performance: Completed task orders: Oct. 2006-Feb. 2008; Approximate contract value: $20.7 million; Approximate contract obligation: $20.7 million; Contract type: Firm-fixed-price. Task order: Project 28 Contractor Maintenance Logistics and Support; Description: Completed task orders: Project 28 operational maintenance and logistics support; Period of performance: Completed task orders: Dec. 2007-Dec. 2009; Approximate contract value: $10.6 million; Approximate contract obligation: $10.6 million; Contract type: Cost-plus-fixed-fee. Task order: Design for Buffalo Sector; Description: Completed task orders: SBInet design of remote video surveillance system capability for the Buffalo sector; Period of performance: Completed task orders: Feb. 2009-July 2009; Approximate contract value: $0.6 million; Approximate contract obligation: $0.6 million; Contract type: Firm-fixed-price[B]. Task order: System; Description: Completed task orders: Program management and systems engineering activities required to integrate all task orders; Period of performance: Completed task orders: Apr. 2008-May 2010; Approximate contract value: $222.1 million; Approximate contract obligation: $215.9 million; Contract type: Cost-plus-award-fee. Ongoing task orders: Task order: Command, Control, Communications, and Intelligence (C3I) Common Operating Picture (COP); Description: Completed task orders: Design, development, demonstration, and operations and maintenance of a functional C3I/COP system; Period of performance: Completed task orders: Dec. 2007-June 2010; Approximate contract value: $70.5 million[C]; Approximate contract obligation: $79.3 million; Contract type: Cost-plus-award-fee/cost-plus-fixed-fee/firm-fixed- price[D]. Task order: Design; Description: Completed task orders: Design of deployment solution, environmental clearance support, and other location-related work for the Tucson sector; Period of performance: Completed task orders: Aug. 2007-July 2010; Approximate contract value: $115.0 million; Approximate contract obligation: $115.0 million; Contract type: Cost-plus-fixed-fee. Task order: Arizona Deployment; Description: Completed task orders: Deployment to two sites covering approximately 53 miles of the southwest border in the Tucson sector; Period of performance: Completed task orders: June 2008-May 2011; Approximate contract value: 148.5; Approximate contract obligation: 148.5; Contract type: Cost-plus-incentive-fee/cost-plus-award-fee[E]. Task order: Integrated Logistics Support; Description: Completed task orders: Maintenance logistics and operational support; Period of performance: Completed task orders: Aug. 2008-Sept. 2010; Approximate contract value: $61.6 million; Approximate contract obligation: $61.6 million; Contract type: Cost-plus-fixed-fee[F]. Task order: Northern Border Project; Description: Completed task orders: Design, installation, and deployment of surveillance capabilities in the Detroit and Buffalo sectors; Period of performance: Completed task orders: Mar. 2009-Dec. 2010; Approximate contract value: $22.7 million; Approximate contract obligation: $22.7 million; Contract type: Fixed-price[B]. Task order: Systems Support; Description: Completed task orders: Program management and systems engineering activities required to integrate all task orders; C3I/COP maintenance; Period of performance: Completed task orders: May 2010-Mar. 2011; Approximate contract value: $28.6 million; Approximate contract obligation: $28.6 million; Contract type: Cost-plus-award-fee. Total: Approximate contract value: $847.80 million; Approximate contract obligation: $850.40 million. Source: GAO analysis of DHS data. Note: "Fixed-price" types of contracts provide for a firm price or, in appropriate cases, an adjustable price; "firm-fixed-price" contracts provide for a price not subject to adjustment on the basis of the contractor's experience in performing the contract; "cost-plus- incentive-fee" contracts provide for the reimbursement of allowable costs plus an initially negotiated fee, to be adjusted later based on the relationship of total allowable costs to total target costs; "cost- plus-award-fee" contracts provide for the reimbursement of allowable costs plus a base fee fixed at the contract's inception (which may be zero) and an award amount that the government determines to be sufficient to motivate excellence in performance; and "cost-plus-fixed- fee" contracts provide for the reimbursement of allowable costs plus a negotiated fee fixed at the inception of the contract. [A] The initial contract type of the task order was a cost-plus-award- fee. A final award fee determination did not take place because of scope and schedule changes. In lieu of the final award fee determination, the contract type was changed to a cost-plus-fixed-fee. [B] The travel component of the task order is cost reimbursable. [C] According to SBI Contracting Division officials, the contract value with the cost overruns is $79.3 million. [D] The initial contract type of the task order was cost-plus-award- fee. On July 31, 2009, additional development work was definitized as a cost-plus-fixed-fee structure. Further, on September 30, 2009, the software operations and maintenance component of the task order was changed to a firm-fixed-price structure. [E] The initial contract type of the task order was a cost-plus- incentive-fee. On November 20, 2009, the performance and schedule incentives component of the task order was changed to a cost-plus- award-fee. The cost incentives component remains a cost-plus-incentive- fee structure. [F] The initial contract type of the task order was cost-plus- incentive-fee. On November 6, 2009, future work under the task order was changed to a cost-plus-fixed-fee structure. [End of table] Through the task orders, CBP's strategy is to deliver SBInet capabilities incrementally. To accomplish this, the program office adopted an evolutionary system life cycle management approach in which system capabilities are to be delivered to designated locations in a series of discrete subsets of system functional and performance capabilities that are referred to as blocks. The first block (Block 1) includes the purchase of commercially available surveillance systems, development of customized COP systems and software, and use of existing CBP communications and network capabilities. According to program officials, as of July 2010, the Block 1 design was deployed to TUS-1 and was being deployed to AJO-1, both of which are located in CBP's Tucson Sector of the southwest border. Together, these two deployments cover 53 miles of the 1,989-mile-long southern border. Also, according to program officials, as of July 2010, TUS-1 and AJO-1 were to be accepted in September 2010 and December 2010, respectively. In January 2010, the Secretary of Homeland Security ordered an assessment of the SBI program. In addition, on March 16, 2010, the Secretary froze fiscal year 2010 funding for any work on SBInet beyond TUS-1 and AJO-1 until the assessment is completed. Also at that time, the Secretary redirected $35 million that had been allocated to SBInet Block 1 deployment to other tested and commercially-available technologies, such as truck-mounted cameras and radar, called Mobile Surveillance Systems, to meet near-term needs.[Footnote 7] According to the SBI Executive Director, the department's assessment would consist of a comprehensive and science-based assessment to determine if there are alternatives to SBInet that may more efficiently, effectively, and economically meet U.S. border security needs. Further, the Executive Director stated that if the assessment suggests that the SBInet capabilities are worth the cost, DHS will extend its deployment to sites beyond TUS-1 and AJO-1. However, if the assessment suggests that alternative technology options represent the best balance of capability and cost-effectiveness, DHS will redirect resources to these other options. According to program officials, the initial phase of the assessment, which addresses the Arizona border, was completed in July 2010, and the results are currently being reviewed by senior DHS management. Officials were unable to provide a date for completion of the review. Use of Acquisition Management Disciplines Is Important to Programs Like SBInet: Our research and evaluations of information technology programs have shown that delivering promised system capabilities and benefits on time and within budget largely depends on the extent to which key acquisition management disciplines are employed. These disciplines include, among others, requirements management, risk management, and test management. As is discussed in the following section, we have previously reported on the extent to which these disciplines have been employed on SBInet. Contractor management and oversight, which is the focus of this report, is another acquisition management discipline. Among other things, this discipline helps to ensure that the contractor performs the requirements of the contract and the government receives the service or product intended. DHS acquisition policies and guidance, along with other relevant guidance, recognize the importance of these management and oversight disciplines. As we have previously reported, not employing them increases the risk that system acquisitions will not perform as intended and will require expensive and time-consuming rework.[Footnote 8] GAO Has Previously Reported on Numerous SBInet Acquisition Management Weaknesses and Risks: Since 2007, we have identified a range of management weaknesses and risks facing SBInet, and we have made a number of recommendations to address them that DHS has largely agreed with, and to varying degrees, taken actions to address. For example, in February 2007, we reported that DHS had not fully defined activities, milestones, and costs for implementing the program or demonstrated how program activities would further the strategic goals and objectives of SBI.[Footnote 9] Further, we reported that the program's schedule contained a high level of concurrency among related tasks and activities, which introduced considerable risk. Accordingly, we recommended that DHS define explicit and measurable commitments relative to, among other things, program capabilities, schedules, and costs, and re-examine the level of concurrency in the schedule and adjust the acquisition strategy appropriately. In September 2008, we reported that a number of program management weaknesses put SBInet at risk of not performing as intended and taking longer and costing more to deliver than was necessary.[Footnote 10] Specifically, we reported the following: * Important aspects of SBInet were ambiguous and in a continued state of flux, making it unclear and uncertain what technology capabilities were to be delivered when. For example, the scope and timing of planned SBInet deployments and capabilities had continued to change since the program began and remained unclear. Further, the SPO did not have an approved integrated master schedule to guide the execution of the program, and our assimilation of available information indicated that key milestones continued to slip. This schedule-related risk was exacerbated by the continuous change in and the absence of a clear definition of the approach used to define, develop, acquire, test, and deploy SBInet. Accordingly, we concluded that the absence of clarity and stability in these key aspects of SBInet impaired the ability of Congress to oversee the program and hold DHS accountable for results, and it hampered DHS's ability to measure the program's progress. * SBInet requirements had not been effectively defined and managed. While the SPO had issued guidance that defined key practices associated with effectively developing and managing requirements, the guidance was developed after several key activities had been completed. In the absence of this guidance, the SPO had not effectively performed key requirements development and management practices, such as ensuring that different levels of requirements were properly aligned. As a result, we concluded that the risk of SBInet not meeting mission needs and performing as intended was increased, as were the chances of the system needing expensive and time-consuming rework. * SBInet testing was not being effectively managed. For example, the program office had not tested the individual system components to be deployed to initial locations, even though the contractor had initiated integration testing of these components. Further, its test management strategy did not contain, among other things, a clear definition of testing roles and responsibilities or sufficient detail to effectively guide planning for specific test events. To address these issues, we recommended that DHS assess and disclose the risks associated with its planned SBInet development, testing, and deployment activities and that it address the system deployment, requirements management, and testing weaknesses that we had identified. DHS largely agreed to implement our recommendations. In September 2009, we reported that SBInet had continued to experience delays.[Footnote 11] For example, deployment to the entire southwest border had slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1 had slipped from November 2009 and March 2010 to December 2009 and June 2010, respectively. We did not make additional recommendations at that time. In January 2010, we reported that SBInet testing was not being effectively managed.[Footnote 12] Specifically, we reported the following: * Test plans, cases, and procedures for the most recent test events were not defined in accordance with important elements of relevant guidance. Further, a large percentage of the test cases[Footnote 13] for these events were changed extemporaneously during execution. While some of the changes were minor, others were more significant, such as rewriting entire procedures and changing the mapping of requirements to test cases. Moreover, these changes to procedures were not made in accordance with documented quality assurance processes, but rather were based on an undocumented understanding that program officials said they established with the contractor. Compounding the number and significance of changes were questions raised by the SPO and a support contractor about the appropriateness of some changes, noting that some changes made to system qualification test cases and procedures appeared to be designed to pass the test instead of being designed to qualify the system. * About 1,300 SBInet defects had been found from March 2008 through July 2009, with the number of new defects identified during this time generally increasing faster than the number being fixed--a trend that is not indicative of a system that is maturing and ready for deployment. While the full magnitude of these unresolved defects was unclear because the majority were not assigned a priority for resolution, some of the defects that had been found were significant. Although DHS reported that these defects had been resolved, they had nevertheless caused program delays, and related problems had surfaced that continued to impact the program's schedule. In light of these weaknesses, we recommended that DHS (1) revise the program's overall test plan; (2) ensure that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised test plan; (3) ensure that sufficient time is provided for reviewing and approving test documents prior to beginning a given test event; and (4) triage the full inventory of unresolved system problems, including identified user concerns, and periodically report on their status to CBP and DHS leadership. DHS fully agreed with the last three recommendations and partially agreed with the first. In May 2010, we raised further concerns about the program and called for DHS to reconsider its planned investment in SBInet.[Footnote 14] Specifically, we reported the following: * DHS had defined the scope of the first incremental block of SBInet capabilities that it intended to deploy and make operational; however, these capabilities and the number of geographic locations to which they were to be deployed continued to shrink. For example, the geographic "footprint" of the initially deployed capability has been reduced from three border sectors spanning about 655 miles to two sectors spanning about 387 miles. * DHS had not developed a reliable integrated master schedule for delivering the first block of SBInet. Specifically, the schedule did not sufficiently comply with seven of nine key practices that relevant guidance states are important to having a reliable schedule. For example, the schedule did not adequately capture all necessary activities, assign resources to them, and reflect schedule risks. As a result, it was unclear when the first block was to be completed, and continued delays were likely. * DHS had not demonstrated the cost-effectiveness of Block 1. In particular, it had not reliably estimated the costs of this block over its entire life cycle. Further, DHS had yet to identify expected quantifiable and qualitative benefits from this block and analyze them relative to costs. Without a meaningful understanding of SBInet costs and benefits, DHS lacks an adequate basis for knowing whether the initial system solution is cost effective. * DHS had not acquired the initial SBInet block in accordance with key life cycle management processes. While processes associated with, among other things, requirements development and management and risk management, were adequately defined they were not adequately implemented. For example, key risks had not been captured in the risk management repository and thus had not been proactively mitigated. As a result, DHS is at increased risk of delivering a system that does not perform as intended. We concluded that it remains unclear whether the department's pursuit of SBInet is a cost-effective course of action, and if it is, that it will produce expected results on time and within budget. Accordingly, we recommended that DHS (1) limit near-term investment in the first incremental block of SBInet, (2) economically justify any longer-term investment in SBInet, and (3) improve key program management disciplines. DHS largely agreed with our recommendations, and noted ongoing and planned actions to address each of them. In June 2010, we reported several technical, cost, and schedule challenges facing SBInet, such as problems with radar functionality, and noted that the program office was staffed substantially below planned levels for government positions.[Footnote 15] We did not make any additional recommendations at that time. DHS Has Largely Defined but Has Not Effectively Implemented the Full Range of Needed Contractor Management and Oversight Controls: Federal regulations and relevant guidance recognize effective contractor management and oversight as a key acquisition management discipline. In addition, they describe a number of practices associated with it, including training persons who are responsible for contractor management and oversight, verifying and deciding whether to accept contract deliverables, conducting technical reviews with the contractor to ensure that products and services will satisfy user needs, and holding management reviews with the contractor to monitor contractor performance.[Footnote 16] To DHS's credit, it has largely defined key practices aimed at employing each of these contractor management and oversight controls. Moreover, it has implemented some of them, such as training key contractor management and oversight officials and holding management reviews with the prime contractor. However, it has not effectively implemented others, such as documenting contract deliverable reviews and using entry and exit criteria when conducting technical reviews. Reasons for these weaknesses include limitations in the defined verification and acceptance deliverable process and a SPO decision to exclude some deliverables from the process, and insufficient time to review technical review documentation. Without employing the full range of practices needed to effectively manage and oversee the prime contractor, DHS is limited in its ability to know whether the contractor is meeting performance and product expectations. Moreover, it increases the chances that SBInet will not function as intended and will take more time and resources than necessary to deliver, which we have previously reported is the case for Block 1. SBInet Contractor Management and Oversight Officials Have Been Properly Trained: Training supports the successful performance of relevant activities and tasks by helping to ensure that the responsible people have the necessary skills and expertise to perform the tasks. According to relevant guidance, organizations should define training expectations and should ensure that these expectations are met for individuals responsible for contractor management and oversight.[Footnote 17] DHS's acquisition management directives define training requirements for, among others, program managers, contracting officers, and contracting officer's technical representatives (COTR).[Footnote 18] Specifically: * Program managers must be certified at a level commensurate with their acquisition responsibilities. For a Level 1 information technology program,[Footnote 19] like SBInet, the designated program manager must be certified at a program management Level 3.[Footnote 20] * Contracting officers must be certified at the appropriate level to support their respective warrant authority.[Footnote 21] * COTRs must be trained and certified within 60 days of appointment to a contract or task order. The minimum mandatory requirements are the completion of 40 hours of COTR training and a 1-hour procurement ethics training course. For SBInet, CBP has ensured that people in each of these key positions have been trained in accordance with DHS requirements. Specifically, the two program managers associated with the development and deployment of SBInet Block 1--the SBI Executive Director and the SPO Executive Director--were issued training certifications that qualify each as an Acquisition Professional. Further, each contracting officer responsible for executing actions on the Arizona Deployment Task Order (ADTO), the Design Task Order (DTO), the C3I/COP task order, and the System Task Order (STO) between June 2008 and February 2010 was certified commensurate with his or her respective warrant authority. In addition, for the same time period, each COTR assigned to each of the four task orders received DHS-issued certificates indicating that each had met the minimum training requirements before being assigned to a task order. According to CBP officials, DHS leadership has made contractor management and oversight training a high priority, which helped to ensure that key officials were trained. By doing so, CBP has established one of the key controls to help ensure that prime contractor products and services meet expectations. DHS Has Not Effectively Verified and Accepted All Contract Deliverables: Effectively implementing a well-defined process for verifying and accepting prime contractor deliverables is vital to SBInet's success. DHS has a defined process for verifying and accepting contract deliverables, but this process does not ensure that deliverable reviews are sufficiently documented. Further, while the SPO has followed key aspects of this process, it has not effectively documented its review of certain deliverables and has not effectively communicated to the prime contractor the basis for rejecting all deliverables. Reasons for not doing so include limitations in the defined verification and acceptance process and a SPO decision to exclude some deliverables from the process. Without documenting all its reviews and effectively communicating the review results to the contractor, the SPO has increased the chances of accepting deliverables that do not meet requirements and having the contractor repeat work to correct deliverable problems. CBP Has a Defined Process for Verifying and Accepting SBInet Contract Deliverables: The purpose of contractor deliverable verification and acceptance is to ensure that contractor-provided products and services meet specified requirements and otherwise satisfy the terms of the contract. According to relevant guidance, organizations should have written policies and procedures for verifying and accepting deliverables that, among other things, (1) assign roles and responsibilities for performing verification and acceptance tasks and (2) provide for conducting and documenting deliverable reviews and for effectively communicating to the contractor deliverable acceptance and rejection decisions.[Footnote 22] To its credit, CBP has defined policies and procedures for verifying and accepting SBInet deliverables. Specifically, it issued its Deliverable Review and Approval Process in July 2007, which specifies how the SPO is to receive, review, and respond to all contract deliverables. Among other things, this guide assigns verification and acceptance roles and responsibilities to key program management positions. For example, it assigns the project manager[Footnote 23] responsibility for overseeing the review process and determining the acceptability of the deliverables, designates reviewers[Footnote 24] responsibilities for examining the deliverable, and assigns the contracting officer responsibility for communicating the decision to accept or reject the deliverable to the contractor. In addition, it provides for conducting deliverable reviews, which according to program officials, involves comparing the deliverable to the requirements enumerated in the applicable task order statement of work (typically within the Data Item Description). The process further specifies that the decision to accept or reject the deliverable is to be communicated in writing to the contractor. However, the process does not state that the results of all reviews are to be documented. Instead, its states that a deliverable review comment form is to be prepared only when deficiencies or problems exist. If the deliverable is acceptable, the form does not need to be prepared. Program officials could not explain why review documentation was not required for acceptable deliverables. As a result, and as discussed in the next section of this report, the SPO cannot demonstrate its basis for accepting a number of deliverables, which in turn has increased the risk of, and actually resulted in, deliverables being accepted that do not meet requirements. CBP Has Not Fully Followed its Defined Process for Verifying and Accepting SBInet Deliverables: The SPO followed its process for verifying and accepting SBInet- related deliverables about 62 percent of the time, based on the 29 deliverables that we reviewed.[Footnote 25] Specifically, the process was fully followed for 18 of the deliverables: (1) 6 that were accepted without documented review comments, (2) 5 that were accepted with documented review comments, and (3) 7 that were rejected with documented review comments. In addition, the acceptance or rejection of all 18 deliverables was communicated in writing to the contractor. For example, the ADTO Security Plan Addendum was delivered to the SPO in August 2008. The SPO reviewed the plan and documented its review comments, which included a determination that the plan did not address all required items specified in the task order's Data Item Description. The CBP contracting officer subsequently notified the contractor in writing that the plan was rejected for this and other reasons. In February 2009, the contractor resubmitted the plan, the SPO reviewed the plan and documented its comments, and the contracting officer again notified the contractor in writing that the plan was rejected. The contractor resubmitted the plan in May 2009, and program officials documented their review of the deliverable. The contracting officer subsequently communicated the deliverable's acceptance in writing to the contractor. (Figure 3 summarizes the number of contract deliverables that did and did not follow the defined process.) Figure 3: Summary of Contract Deliverables That Did and Did Not Follow the Defined Process: [Refer to PDF for image: 3 pie-charts] Deliverables following process: 18; Accepted with comments and with letter: 5; Accepted without comments and with letter: 6; Rejected with comments and with letter: 7. Deliverables not following process: 11; Accepted without comments and without letter: 5; Rejected without comments and without letter: 3; Decision not made, no comments, and no letter: 3. Source: GAO analysis of DHS data. [End of figure] The remaining 11 deliverables, however, did not fully adhere to the defined process. Of these, five were accepted without any documented review comments and without communicating the acceptance in writing to the contractor. The following are examples of these five and reasons for not adhering to the process. * Three of the five deliverables related to the C3I/COP task order did not require government approval. However, the Deliverable Review and Approval Process document does not provide for such treatment of these deliverables, and thus this practice is not in accordance with the process. Program officials told us that they have since recognized that this was a poor practice, and they have modified the task order to now require approval of all C3I/COP task order deliverables. * One of the five deliverables relates to the STO and is for the C2 Component Qualification Test package, which includes, among other things, the test plan and test cases and procedures for conducting the C2 qualification test event. In this case, the SPO accepted the test plan because, according to program officials, several days had passed since the deliverable was received, and they had not received any comments from the reviewers. They said that they therefore accepted the deliverable on the basis of not receiving any review comments to the contrary, but did not notify the contractor in writing of the acceptance. The 11 deliverables also include 3 that were rejected without documented review comments and without the rejection being communicated to the contractor in writing. The following are examples of these three and reasons for not adhering to the process are discussed below. * One of the three deliverables relates to the C3I/COP task order and is for the Network Operations Center/Security Operations Center (NOC/ SOC)[Footnote 26] Test Plan/Procedures/Description. According to program officials, the contractor did not submit the plan on time, thus requiring them to review it during the readiness review.[Footnote 27] Based on this review, the plan was rejected, which was communicated verbally to the contractor during the review. Despite rejecting the plan, the program office began testing the NOC/SOC component on the day of the review, without a revised plan being submitted, reviewed, and approved. According to program officials, this occurred in part because of insufficient time and resources to review contractor test-related deliverables. * Another one of the three deliverables also relates to the C3I/COP task order, and is for the Release 0.5 Software Test Plan/Procedures/ Description. According to program officials, the contractor submitted the plan late. The program office rejected the plan and provided oral comments during a teleconference prior to the review. Nevertheless, the test event again occurred without a revised plan being submitted, reviewed, and accepted. According to program officials, this was also due to insufficient time and resources to review the test plan. In this case, the plan was approved in late April 2009, which was 5 months after the test event was conducted. The 11 deliverables also include 3 for which a decision to accept or reject the deliverable was not made. See the following examples: * One of the three relates to the C3I/COP task order, and is for the NOC/SOC Interface Control Document. For this deliverable, review comments were not documented and no written communication with the contractor occurred. The deliverable was subsequently submitted three times and ultimately accepted. However, program officials could not explain whether the initial deliverable was accepted or rejected, or why the deliverable was submitted multiple times. * Another one of these relates to STO, and is for the Dynamic Object- Oriented Requirements System.[Footnote 28] For this deliverable, review comments were not documented, but CBP communicated in writing to the contractor that it was withholding comment on this submission of the deliverable and was to provide a consolidated set of comments on the subsequent submission. Subsequently, the contractor resubmitted the deliverable, and because it was accepted, the review was not documented. The contracting officer communicated the deliverable's acceptance in writing to the contractor. By not effectively verifying and accepting contractor deliverables, the SPO cannot ensure that the deliverables will satisfy stated requirements, thus increasing the risk of costly and time-consuming rework. For example, we recently reported that contractor-delivered test plans were poorly defined and resulted in problems during testing. In particular, NOC/SOC testing was hampered by requirements incorrectly mapped to test cases, did not provide for testing all requirements, and required significant extemporaneous changes to test cases during the test events.[Footnote 29] As a result of the testing problems, the SPO had to conduct multiple test events. Key Contract Technical Reviews Have Not Been Adequately Conducted: Technical reviews are performed throughout the project life cycle to confirm that products and services being produced by the contractor provide the desired capability and ultimately satisfy user needs. To its credit, DHS has defined a process for conducting technical reviews, but it has not effectively implemented it. In particular, the SPO did not ensure that all key documentation was reviewed and relevant criteria were satisfied before concluding key technical reviews. Program officials attributed these limitations to the program's aggressive schedule, which resulted in insufficient time to review relevant documentation. Concluding technical reviews without adequate justification has resulted in schedule delays and costly rework. CBP Has a Defined Process for Conducting Technical Reviews: According to relevant guidance, organizations should have written policies and procedures for conducting technical reviews that, among other things, (1) assign roles and responsibilities for performing the specific technical review tasks and (2) establish entry and exit criteria to determine the readiness of the technical solution to proceed to the technical review and to demonstrate and confirm completion of required accomplishments.[Footnote 30] To its credit, DHS has policies and guidance for conducting technical reviews. Specifically, DHS's Systems Engineering Life Cycle outlines the key reviews to be performed as well as how these reviews are aligned with the department's governance process.[Footnote 31] In addition, DHS guidance defines expectations for technical review exit criteria, stating that compliance with exit criteria is based upon the satisfaction of the content of the criteria, and not upon only the delivery of specified documents. To augment DHS policy and guidance, the SBInet Systems Engineering Plan (SEP), dated November 2008, identifies and describes the technical reviews to be conducted. They include, for example: * Requirements Review, which is to ensure that requirements have been completely and properly identified and are understood by the SPO and the contractor. Documentation associated with this review is to include, among other things, a requirements traceability matrix (i.e., a tool for demonstrating that component-level requirements are traceable to higher-level system level requirements). * Critical Design Review (CDR), which is to (1) demonstrate that the designs are complete and baselined and (2) ensure that the solution is ready for fabrication, coding, assembly, and integration. Documentation for this review is to include, among other things, (1) baselined requirements, (2) interface descriptions, and (3) identified risks and mitigation plans. * Test Readiness Review, which is to assess the readiness of the system solution to begin formal testing. Documentation for this review is to include, among other things, (1) test plans that include test cases and procedures and (2) a traceability matrix that maps each requirement to be tested to a corresponding test case. In addition, the SBInet SEP describes high-level roles and responsibilities for performing these reviews, and establishes entry and exit criteria for each. For example, it states that the SPO program manager and the chief engineer are responsible for leading the reviews. Further, the SEP defines entry and exit criteria for the CDR. For example: * Entry. System-level requirements should be traceable to component- level requirements; system internal and external interfaces should be defined. * Exit. Design baseline should be established and balanced across cost, schedule, performance, and risk considerations over the investment's life cycle; system risks should be identified and mitigation plans should be in place. Contractor Technical Reviews Have Not Been Performed Adequately: The SPO did not follow the defined process for conducting technical reviews. Instead, program officials told us that they used the requirements defined in the respective task orders to guide each review. However, the task orders do not define entry and exit criteria. Rather, they list a set of documents that the contractor is to provide and the SPO is to review. For example, for the Block 1 CDR, the relevant task order requires that the contractor deliver, among other documents, (1) baselined component and system requirements, (2) interface descriptions (i.e., descriptions of the data to be exchanged and the protocols used to exchange the data), and (3) all identified risks and mitigation plans for those risks. However, the task orders do not associate these documents with either entry or exit criteria, and they do not specify characteristics or qualities that the documents are to satisfy. Without explicit entry and exit criteria, the basis for beginning and ending the technical reviews is unclear, thus increasing the risk that a program will be allowed to proceed and begin the next phase of development before it is ready to do so. In fact, this risk was realized for SBInet. Technical reviews were concluded without adequate justification, which ultimately resulted in problems that required additional time and resources to fix. For example: * NOC/SOC Requirements Review. At this review, the contractor did not deliver a requirements traceability matrix, as required by the relevant task order, until almost a month after the review was completed. Nonetheless, program officials stated that they concluded the review in June 2008, without knowing whether the applicable higher- level system requirements were fully satisfied. * Block 1 CDR. For this review, the contractor delivered (1) the baselined component and system requirements, (2) the interface descriptions, and (3) all identified risks and mitigation plans for those risks. However, these deliverables did not demonstrate that all component- level requirements were baselined and interface descriptions were understood. As we previously reported, baselined requirements associated with the NOC/SOC were not adequately defined at the time of the CDR, as evidenced by the fact that they were significantly changed 2 months later.[Footnote 32] Program officials stated that while they knew that requirements were not adequately baselined at this review, they believed that the interface requirements were understood well enough to begin system development. However, this was also not the case. Specifically, 39 of 90 NOC/SOC interface requirements were removed from the baseline, and 2 new interface requirements were added after CDR. Further, all relevant risks were not identified, and not all identified risks had mitigation plans. Specifically, 7 of 31 identified risks did not have mitigation plans, including risks associated with poorly established requirements traceability and inadequately defined requirements for integration suppliers. Moreover, the risks identified were as of May 2008, prior to the beginning of CDR, and did not include four risks identified between June and October 2008, when CDR was concluded. For example, a risk associated with the instability of the C3I/COP software was not addressed during CDR. Without properly baselined requirements (including interfaces) and proactive mitigation of known risks, system performance shortfalls are likely. To illustrate, we previously reported that ambiguities in requirements actually forced testers to rewrite test steps during execution based on interpretations of what they thought the requirements meant, and they required the SPO to incur the time and expense of conducting multiple events to test NOC/SOC requirements. [Footnote 33] * NOC/SOC Component Qualification Test Readiness Review. The SPO did not ensure that a well-defined test plan was in place, to include, among other things, test cases and procedures and a traceability matrix that maps each requirement to be tested to a corresponding test case. Specifically, the contractor delivered the test plan on the day of the review, rather than 10 days prior to the review, as required by the relevant task order. Nevertheless, the SPO concluded the review based on its review of the plan during the test readiness review. In this regard, we previously reported problems with the NOC/SOC test plan, noting that the plan mapped 28 out of 100 requirements to incorrect test cases. Program officials attributed the test plan limitations to, among other things, insufficient time and resources to review the deliverables. The SBInet independent verification and validation (IV&V) contractor [Footnote 34] also identified weaknesses within technical reviews. Specifically, the IV&V contractor reported that the SPO was not provided with documentation, including the test plan, early enough for the NOC/SOC test readiness review to allow sufficient time for review. Moreover, in December 2009, the program identified technical oversight of technical review milestones as a major risk to the cost, schedule, and performance goals of the program. According to program officials, they are developing a technical review manual that is to supplement the SEP and provide detailed guidance for conducting technical reviews. In commenting on a draft of this report, DHS stated that it plans to complete and implement its technical review guide by December 2010. Program officials attributed the limitations with the technical reviews, in large part, to the program's aggressive schedule, which resulted in insufficient time to review relevant documentation. As discussed above, these limitations have resulted in schedule delays and costly rework. Contractor Management Reviews Were Conducted as Planned, but Were Limited by Unreliable Contractor Performance Data: Management reviews help to ensure that the contractor's interpretation and implementation of the requirements are consistent with those of the program office. According to relevant guidance, organizations should have written policies and procedures for conducting management reviews that, among other things, (1) involve relevant stakeholders; (2) assign roles and responsibilities for performing management review tasks; (3) communicate project status information, including cost and schedule information, and risks; and (4) identify, document, and track action items to closure.[Footnote 35] CBP policy also recognizes the importance of these reviews by requiring the conduct of management reviews.[Footnote 36] Further, the SBInet Program Management Plan identifies the types of management reviews that are to be conducted with the contractor. For SBInet, the primary management review is known as the Joint Program Management Review (JPMR). The plan also identifies, for example, the stakeholders that are to participate in the reviews, including the program manager, project managers, program control staff, and the risk management team; and it specifies the topics that are to be discussed at the reviews, such as project status, cost and schedule performance, and risks. To its credit, the program office has held monthly JPMRs, and these reviews include the prime contractor, the SBI Executive Director, the SBInet Program Manager, program control staff, risk management team, border patrol agents, DHS representatives, and the Defense Contract Management Agency (DCMA). Further, review documentation shows that JPMRs addressed cost and schedule performance to include EVM performance data, project milestones status and potential delays, and risks. Review documentation also shows action items were identified, documented, and tracked to closure, as part of these reviews. For example, during the May 2009 JPMR, an action to review the risk management process--including roles, responsibilities, and decision authority--was identified, and this action was documented in Boeing's Management Emphasis Tracking database.[Footnote 37] To address and close the action item, the program subsequently completed a review of the SBInet risk management process and tool, including reviewing lessons learned from other programs. The results of the review were presented during a February 2010 briefing, and the action item was closed. Effectively conducting management reviews has helped to provide program leadership with an understanding of the contractor's progress and the program's exposure to risks so that appropriate corrective action can be taken and the chances of delivering a system solution that meets mission needs within budget are enhanced. However, as discussed in the next section, the EVM performance data presented at these management reviews were not reliable, thus rendering those reviews, at best, limited in the extent to which they disclosed the true status of the program. DHS Has Not Effectively Monitored the Contractor's Progress in Meeting Cost and Schedule Expectations: Measuring and reporting progress against cost and schedule expectations (i.e., baselines) is a vital element of effective contractor management and oversight. As noted earlier, EVM provides a proven means for measuring progress against cost and schedule commitments and thereby identifying potential cost overruns and schedule delays early, when the impact can be minimized. However, DHS has not ensured that its prime contractor's EVM system, which was certified as meeting relevant standards, has been effectively implemented on SBInet. In particular, it has not ensured that performance measurement baselines were validated in a timely manner, that established baselines were complete and realistic, and that contractor-provided cost and schedule data were reliable. Reasons cited by program officials for these weaknesses include the instability in the scope of the work to be performed, an unexpected temporary stop in Block 1 design and deployment work when SBInet funding was redirected, and the contractor's use of estimated, rather than actual, costs for subcontractor work, which are subsequently adjusted when actual costs are received. Without effectively implementing EVM, DHS has not been positioned to identify potential cost and schedule problems early, and thus has not been able to take timely actions to correct problems and avoid program schedule delays and cost increases. Contractor EVM System Has Been Certified: In August 2005, the Office of Management and Budget issued guidance [Footnote 38] that, among other things, directs agencies to ensure that EVM systems are compliant with the American National Standards Institute (ANSI) standard.[Footnote 39] The ANSI standard consists of 32 guidelines associated with a sound EVM system that are intended to ensure that data are reliable and can be used for informed decision- making. The program office relies on the prime contractor's EVM system to provide cost and schedule performance data. This system was certified in April 2005 by DCMA as being compliant with the ANSI standard. DCMA certified the contractor's EVM system again in February 2009. Notwithstanding these certifications, DCMA identified a number of issues with the contractor's implementation of its EVM system. [Footnote 40] In particular, in January 2010, DCMA reported that the SBInet prime contractor's implementation of EVM was not consistent with all of the 32 ANSI guidelines. Specifically, DCMA identified concerns with the quality of scheduling and reporting, and the identification of significant differences between planned and actual cost and schedule performance, as well as reasons for those differences. Program Office Has Not Established Timely Performance Baselines: According to relevant guidance, the performance measurement baseline, which is the foundation of an EVM system and the estimated cumulative value of planned work, serves as the value against which performance is measured for the life of the program or task order.[Footnote 41] As such, it should be established as early as possible after contract or task order award, or whenever a major contract modification or baseline change occurs. DHS guidance further states that a baseline should be validated within 90 days of the contract or task order award.[Footnote 42] However, the program office validated a performance measurement baseline within 90 days for only two of the six baselines that we reviewed (see figure 4). For the other four, the length of time to establish a validated baseline ranged from 5 to 10 months. For example, the program office issued the ADTO in June 2008, and it did not establish a validated baseline until 10 months later in April 2009. Similarly, in February 2009, the program office modified the scope of the STO and extended the period of performance, but it did not validate the revised baseline to include the additional scope and time until 7 months later in September 2009. Figure 4 summarizes the periods of time during which earned value was, and was not, measured against a validated baseline. Figure 4: Summary of Time Frames With and Without Validated Performance Baselines from June 2008 through February 2010: [Refer to PDF for image: illustration] Design Task Order: June 2008 and earlier through March, 2009: Earned value data was not measured against a government-validated baseline. March 2009 through August 2009: (IBR 3/09) Earned value data was measured against a government-validated baseline. August 2009 through December 2009: Earned value data was not measured against a government-validated baseline. December 2009 and beyond: Earned value data was measured against a government-validated baseline. C3I/COP: IBR (2/08): through March 2009: Earned value data was measured against a government-validated baseline. System Task Order: IBR (5/08): through March 2009: Earned value data was measured against a government-validated baseline. March 2009 through September 2009: Earned value data was not measured against a government-validated baseline. IBR (9/09): September 2009 and beyond: Earned value data was measured against a government-validated baseline. Arizona Deployment Task Order: Contract award: June 2008 through April 2009: Earned value data was not measured against a government-validated baseline. IBR (4/09): April 2009 and beyond: Earned value data was measured against a government-validated baseline. Source: GAO analysis of SBInet data. Notes: Generally, the government established baselines through the period of performance for the task order at the time of the Integrated Baseline Review (IBR). The government had not completed all work for the DTO and STO within the original period of performance, and as such, extended the period of performance, and in some cases, established a new baseline. According to program officials, they held an IBR for the C3I/COP task order in March 2009. However, they did not provide sufficient documentation to support this statement. [End of figure] According to program officials, the delays in validating performance baselines were due to instability in the work to be performed, and the need to temporarily stop Block 1 design and deployment work between September 2008 and January 2009 because of DHS's decision to redirect funds from SBInet to the physical infrastructure.[Footnote 43] Without validated baselines, DHS was not positioned to identify potential cost and schedule problems early and to take timely corrective actions to mitigate those problems. DHS Has Not Established Reliable Performance Measurement Baselines: An integrated baseline review (IBR) is used to validate the performance measurement baseline. This review is intended to verify that the baseline is realistic and ensure that the contractor and the government mutually understand scope, schedule, and risks for a given task order before a substantial amount of work is performed. According to relevant guidance, establishing a complete and realistic performance measurement baseline includes (1) assigning responsibility for managing, tracking, and reporting earned value data for work performed; (2) estimating needed resources (i.e., budgets and staff), including management reserve, for performing assigned tasks; (3) defining a product-oriented description of all work to be performed; [Footnote 44] (4) scheduling all work in a time-phased sequence that reflects the duration of the program's activities; and (5) establishing objective performance measures for each task.[Footnote 45] In validating the performance measurement baselines for the four task orders that we reviewed, the program office implemented two of the above elements, but it did not implement the other three. Specifically, for each of the six baselines associated with the task orders, the program office (1) assigned responsibility for managing, tracking, and reporting earned value data associated with each work breakdown structure element and (2) estimated a time-phased budget, including the anticipated staff needed, for each work breakdown structure element, and established a management reserve. However, as discussed in the following section, the program office did not (1) define a product-oriented description of all work to be performed, (2) reliably estimate schedule baselines, and (3) adequately measure earned value performance. Program officials attribute these limitations in establishing comprehensive baselines to instability in the nature of the work to be performed and the prime contractor's method for determining subcontractor performance. Nevertheless, without complete and realistic baselines, the SPO has been hampered in its ability to conduct meaningful measurement and oversight of the prime contractor's status and progress, as well as holding the contractor accountable for results. More importantly, the lack of meaningful measurement and oversight has contributed to program cost overruns and schedule delays. Work Breakdown Structure Was Not Product Oriented and Did Not Include All Work: According to relevant guidance, a work breakdown structure deconstructs a program's end product into successively smaller levels until the work is subdivided to a level suitable for management control.[Footnote 46] Further, a work breakdown structure should be product oriented and include all work to be performed. The work breakdown structure that was used to define each of the task order baselines was not product oriented. Instead, it was defined in terms of functions that span multiple system products, such as systems engineering, system test and evaluation, and program management. Additionally, the work breakdown structure did not reflect all work to be performed. Specifically, for four of the six performance measurement baselines, the work breakdown structure did not include all work described in the corresponding task order's statement of work. For example, the work breakdown structure used to define the May 2008 STO baseline did not include the work associated with identifying and selecting components that meet system requirements and program security. Similarly, DCMA reported in June 2008 that the work breakdown structure included in this baseline did not account for all work identified in the system task order. Estimated Baseline Schedules Were Not Reliable: A reliable schedule provides a road map for systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine best practices associated with developing and maintaining a reliable schedule: (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating activities horizontally and vertically, (6) establishing the critical path[Footnote 47] for all activities, (7) identifying reasonable "float"[Footnote 48] between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations.[Footnote 49] The six task order baselines were not reliable because they substantially complied with only two of the eight key schedule estimating practices, and they did not comply with, or only partially or minimally complied with, the remaining six practices.[Footnote 50] (See figure 5 for a summary of the extent to which each of the baseline schedules met each of the eight practices.) Figure 5: Summary of IBR Schedules' Satisfaction of Schedule Estimating Practices: [Refer to PDF for image: illustrated table] Schedule practice: Capturing all activities: Task order schedule: C3I/COP February 2008: Minimally met; STO May 2008: Minimally met; DTO March 2009: Partially met; ADTO April 2009: Minimally met; STO September 2009: Minimally met; DTO December 2009: Minimally met. Schedule practice: Sequencing all activities: Task order schedule: C3I/COP February 2008: Substantially met; STO May 2008: Substantially met; DTO March 2009: Substantially met; ADTO April 2009: Substantially met; STO September 2009: Substantially met; DTO December 2009: Substantially met. Schedule practice: Assigning resources to all activities: Task order schedule: C3I/COP February 2008: Minimally met; STO May 2008: Minimally met; DTO March 2009: Minimally met; ADTO April 2009: Partially met; STO September 2009: Minimally met; DTO December 2009: Partially met. Schedule practice: Establishing the duration of all activities: Task order schedule: C3I/COP February 2008: Substantially met; STO May 2008: Substantially met; DTO March 2009: Substantially met; ADTO April 2009: Substantially met; STO September 2009: Substantially met; DTO December 2009: Substantially met. Schedule practice: Integrating activities horizontally and vertically: Task order schedule: C3I/COP February 2008: Partially met; STO May 2008: Partially met; DTO March 2009: Partially met; ADTO April 2009: Partially met; STO September 2009: Partially met; DTO December 2009: Partially met. Schedule practice: Establishing the critical path for all activities: Task order schedule: C3I/COP February 2008: Partially met; STO May 2008: Partially met; DTO March 2009: Partially met; ADTO April 2009: Partially met; STO September 2009: Partially met; DTO December 2009: Partially met. Schedule practice: Identifying reasonable float between activities: Task order schedule: C3I/COP February 2008: STO May 2008: Partially met; DTO March 2009: Minimally met; ADTO April 2009: Partially met; STO September 2009: Partially met; DTO December 2009: Partially met. Schedule practice: Conducting a schedule risk analysis: Task order schedule: C3I/COP February 2008: Not met; STO May 2008: Not met; DTO March 2009: Not met; ADTO April 2009: Not met; STO September 2009: Not met; DTO December 2009: Not met. Met = DHS provided complete evidence that satisfied the criteria. Substantially Met = DHS provided evidence that satisfied more than one- half of the criteria. Partially Met = DHS provided evidence that satisfied about one-half of the criteria. Minimally Met = DHS provided evidence that satisfied less than one- half of the criteria. Not Met = DHS provided no evidence that satisfied any of the criteria. Source: GAO analysis of DHS data. [End of figure] * Capturing all activities. The six schedules did not capture all activities defined in the task order baseline. Specifically, five of the six schedules did not reflect the work to be performed across the four task orders (i.e., integrated master schedule). Further, as previously mentioned, four of six work breakdown structures were missing elements defined in the respective task order statements of work. Moreover, two of the six schedules did not reflect all work that was defined in the work breakdown structure. For example, the December 2009 DTO schedule omitted efforts associated with design work for TUS- 1 and AJO-1. * Sequencing all activities. The six schedules substantially met this practice. Each of the schedules identified almost all of the predecessor and successor activities. However, each contained improper predecessor and successor relationships. For example, the May 2008 STO baseline included 52 of 538 activities (about 10 percent) with improper predecessor and successor relationships. Additionally, many activities in four of the schedules were constrained by "start no earlier than" dates. For example, as previously reported, the September 2009 baseline schedule contained 403 of 1,512 activities (about 27 percent) with "start no earlier than" constraints, which means that these activities are not allowed to start earlier than their assigned dates, even if their respective predecessor activities have been completed. * Assigning resources to all activities. Two of the six schedules partially met this practice. Specifically, two schedules included resources; however, those resources were allocated to less than 15 percent of the activities identified in each schedule. Moreover, the remaining four schedules did not include estimated resources. Instead, resources for all six schedules were maintained separately as part of the contractor's earned value system and only available to DHS upon request. * Establishing the duration of all activities. Each of the six baseline schedules substantially met this practice. Specifically, each schedule established the duration of key activities and included baseline start and end dates for most of the activities. Further, reasonable durations were established for the majority of the activities in the schedules, meaning that the durations established were less than 44 days. Nevertheless, each of the schedules included activities that were not of short duration, that is, more than 44 days. For example, the April 2009 ADTO baseline included 29 of 1,009 activities with durations ranging from 45 days to 352 days. * Integrating activities horizontally and vertically. Each of the schedules partially met this practice. As mentioned previously, the six schedules did not capture all activities defined in the task order baseline. Further, four of six work breakdown structures were missing elements defined in respective task order statements of work. Additionally, five of six schedules did not reflect the work performed across the four task orders (i.e., integrated master schedule), and each had improper predecessor and successor relationships. * Establishing the critical path for all activities. Each of the six schedules partially met this practice. Specifically, four of six work breakdown structures were missing elements defined in the respective task order statements of work. Additionally, four of the six schedules were missing predecessor and successor activities, and each of the schedules included improper predecessor and successor relationships. Further, five of the six schedules did not reflect the work to be performed across the four task orders (i.e., integrated master schedule). Unless all activities are included and properly linked, it is not possible to generate a true critical path. * Identifying reasonable float between activities. Each of the schedules identified float; however, the amount of float was excessive. For example, the February 2008 C3I/COP task order baseline included 259 of 294 activities (about 88 percent) with float greater than 100 days and 189 of the 259 (about 73 percent) with float in excess of 200 days. * Conducting a schedule risk analysis. DHS did not conduct a risk analysis of any of the schedules. Earned Value Performance Was Not Adequately Measured: According to the ANSI standard for EVM systems,[Footnote 51] only work for which measurement is impractical may be classified as "level-of- effort."[Footnote 52] Our research shows that if more than 15 percent of a program's budget is measured using level-of-effort, then that amount should be scrutinized because it does not allow schedule performance to be measured (i.e., performance equals planned work). [Footnote 53] However, the six baselines had between 34 and 85 percent of the baseline dollar value categorized as level-of-effort, including four with more than 50 percent (see table 2). Moreover, for five of the six baselines, program documentation showed that the program office did not identify any action items during the respective IBRs related to the high use of level-of-effort. According to program officials, the STO, which categorized between 70 and 85 percent of the baseline dollar value as level-of-effort, includes many program management activities (e.g., cost, schedule, and subcontractor management). Nevertheless, they recognized that the level-of-effort for this task order was high, and in November 2009, they directed the contractor to minimize the use of level-of-effort for STO. According to program officials, the high level-of-effort was due, in part, to the prime contractor's use of this measurement for subcontractor work. In November 2009, DCMA stated that the SPO's use of level-of-effort activities was high, noting that this could be masking true contractor performance. Table 2: Percentage of Performance Measurement Baseline Work Tasks Categorized as Level-of-Effort for Six Baselines: Task order IBR: C3I/COP task order February 2008; Level-of-effort percentage: 46 percent. Task order IBR: STO May 2008; Level-of-effort percentage: 85 percent. Task order IBR: DTO March 2009; Level-of-effort percentage: 53 percent. Task order IBR: ADTO April 2009; Level-of-effort percentage: 56 percent. Task order IBR: STO September 2009; Level-of-effort percentage: 70 percent. Task order IBR: DTO December 2009; Level-of-effort percentage: 34 percent. Source: GAO analysis of DHS data. [End of table] Unreliable EVM Data Limit DHS's Ability to Measure Contractor Performance: If performed properly, EVM can provide an objective means for measuring program status and forecasting potential program cost overruns and schedule slippages so that timely action can be taken to minimize their impact. To do so, however, the underlying EVM data must be reliable, meaning that they are complete and accurate and all data anomalies are explained. In the case of SBInet, the EVM data provided by the prime contractor for the 21-month period ending in February 2010 have not been reliable, as evidenced by numerous and unexplained anomalies in monthly EVM reports. Reasons for the anomalies include the contractor's use of estimated, rather than actual, costs for subcontractor work, which are subsequently adjusted when actual costs are received. Without reliable performance data, the true status of the SBInet program is unclear, thus limiting the SPO's ability to identify potential cost and schedule shortfalls. EVM Data Can Provide Objective Insight into Program Cost and Schedule Performance: EVM is a proven program measurement approach that, if implemented appropriately, can create a meaningful and coherent understanding of a program's true health and status. As a result, the use of EVM can alert decision makers to potential program problems sooner than is possible by using actual versus planned expenditure alone, and thereby reduce the chance and magnitude of program cost overruns and schedule slippages. Simply stated, EVM measures the value of completed work in a given period (i.e., earned value) against (1) the actual cost of work completed for that period (i.e., actual cost) and (2) the value of the work that is expected to be completed for that period (i.e., planned value). Differences in these values are referred to as cost and schedule variances, respectively. * Cost variances compare the value of the work completed with the actual cost of the work performed. For example, if a contractor completed $5 million worth of work and the work actually cost $6.7 million, there would be a negative $1.7 million cost variance. * Schedule variances are also measured in dollars, but they compare the value of the work completed with the value of the work that was expected to be completed. For example, if a contractor completed $5 million worth of work at the end of the month but was expected to complete $10 million worth of work, there would be a negative $5 million schedule variance. Positive variances indicate that activities are costing less or are being completed ahead of schedule. Negative variances indicate activities are costing more or are falling behind schedule. To determine both cost and schedule variances, all three values are necessary. SBInet EVM Data Have Not Been Reliable: According to relevant guidance, EVM data should be valid and free from unexplained anomalies (e.g., missing or negative values) because they can limit program management's ability to identify potential cost and schedule shortfalls.[Footnote 54] Therefore, anomalies should be minimized for each of the three values--earned value, planned value, and actual cost. Moreover, all anomalies should be identified, and the reason for each should be fully explained in the monthly EVM reports. To do less limits the completeness and accuracy of these values, and thus makes the resulting variance determinations unreliable. While an industry standard for what constitutes an acceptable volume of anomalies does not exist, EVM experts in the public and private sectors that we interviewed stated that the occurrence of EVM data anomalies should be rare. Of these experts, some agreed that an anomaly should occur in no more than 5 percent of the work breakdown structure elements for a given contract or task order, while some of these advocated an occurrence percentage of no more than 1-2 percent. However, the EVM data that the prime contractor delivered to the SPO from June 2008 through February 2010 (21 months) contained numerous, unexplained anomalies. Specifically, the monthly EVM reports for all four task orders that we reviewed showed one or more anomalies (e.g., missing or negative values for earned value, planned value, and actual cost) in each of the months that had a validated performance measurement baseline. More specifically, the average number of work breakdown structure elements across the four task orders that had data anomalies during this 21-month period ranged from 11 percent to 41 percent. For the C3I/COP task order in particular, the monthly percentage of work breakdown structure elements with anomalies ranged between 25 and 67 percent over the 21 months. (See figure 6 for the percentage of work breakdown structure elements with anomalies by month for each of the four task orders.) Figure 6: Percentage of Work Breakdown Structure Elements with EVM Data Anomalies by Month for Each of the Four Task Orders: [Refer to PDF for image: vertical bar graph] 2008: June: Arizona Deployment: 0; Design: 0; C3I/COP: 33%; System: 7%. July: Arizona Deployment: 0; Design: 0; C3I/COP: 33%; System: 4%. August: Arizona Deployment: 0; Design: 0; C3I/COP: 33%; System: 13%. September: Arizona Deployment: 0; Design: 0; C3I/COP: 42%; System: 22%. October: Arizona Deployment: 0; Design: 0; C3I/COP: 42%; System: 18%. November: Arizona Deployment: 0; Design: 0; C3I/COP: 58%; System: 9%. December: Arizona Deployment: 0; Design: 0; C3I/COP: 67%; System: 16%. 2009: January: Arizona Deployment: 0; Design: 0; C3I/COP: 33%; System: 13%; February: Arizona Deployment: 0; Design: 0; C3I/COP: 25%; System: 2. March: Arizona Deployment: 0; Design: 0; C3I/COP: 0; System: 0. April: Arizona Deployment: 0; Design: 14%; C3I/COP: 0; System: 0. May: Arizona Deployment: 11 Design: 10 C3I/COP: 0; System: 0. June: Arizona Deployment: 7%; Design: 10%; C3I/COP: 0; System: 0. July: Arizona Deployment: 11%; Design: 10%; C3I/COP: 0; System: 0. August: Arizona Deployment: 15%; Design: 0; C3I/COP: 0; System: 0. September: Arizona Deployment: 22%; Design: 0; C3I/COP: 0; System: 0. October: Arizona Deployment: 11%; Design: 0; C3I/COP: 0; System: 24%. November: Arizona Deployment: 19%; Design: 0; C3I/COP: 0; System: 29%. December: Arizona Deployment: 41%; Design: 0; C3I/COP: 0; System: 64%. 2010: January: Arizona Deployment: 41%; Design: 10%; C3I/COP: 0; System: 22%. February: Arizona Deployment: 56%; Design: 14%; C3I/COP: 0; System: 20%. Source: GAO analysis of DHS data. Note: We only analyzed EVM reports for those months with a validated baseline for any of the four task orders. There were no validated baselines for any of the four task orders in March 2009. [End of figure] The October 2009 STO monthly EVM report illustrates how the anomalies can distort the contractor's performance. According to this report, about $13,000 worth of work was planned to be completed on integration and management, approximately $13,000 worth of work was performed, and actual costs were about negative $550,000. Thus, the report erroneously suggests that the contractor performed $13,000 worth of work, and actually saved about $550,000 in doing so. Similarly, the September 2009 ADTO monthly report showed that about $200,000 worth of work was planned to be completed on tower sites and infrastructure, and $25,000 worth of work was performed, but that no costs were incurred, suggesting that the work was performed for free. Exacerbating the large percentage of monthly data anomalies across the four task orders is the fact that in most cases the reasons for the anomalies were not explained in the monthly EVM variance analysis reports. Specifically, about 79 percent of all anomalies across all four task orders during the 21-month period were not explained. In particular, 82 of 119 (or about 69 percent) of all data anomalies for the STO task order were not explained in the monthly reports, and none of the anomalies were explained for DTO. (See figure 7 for the total number of data anomalies and the number that were explained in the monthly reports across the four task orders.) Figure 7: Total Number of Data Anomalies and Number of Explained Anomalies in the Monthly EVM Reports across the Four Task Orders: [Refer to PDF for image: vertical bar graph] 2008: June: Total EVM data anomalies: 7; Total EVM data anomalies with explanations: 0. July: Total EVM data anomalies: 6; Total EVM data anomalies with explanations: 0. August: Total EVM data anomalies: 10; Total EVM data anomalies with explanations: 3. September: Total EVM data anomalies: 15; Total EVM data anomalies with explanations: 4. October: Total EVM data anomalies: 13; Total EVM data anomalies with explanations: 2. November: Total EVM data anomalies: 11; Total EVM data anomalies with explanations: 2. December: Total EVM data anomalies: 15; Total EVM data anomalies with explanations: 1. 2009: January: Total EVM data anomalies: 10; Total EVM data anomalies with explanations: 2. February: Total EVM data anomalies: 4; Total EVM data anomalies with explanations: 2. March: Total EVM data anomalies: 0; Total EVM data anomalies with explanations: 0. April: Total EVM data anomalies: 3; Total EVM data anomalies with explanations: 0. May: Total EVM data anomalies: 5; Total EVM data anomalies with explanations: 1. June: Total EVM data anomalies: 4; Total EVM data anomalies with explanations: 1. July: Total EVM data anomalies: 5; Total EVM data anomalies with explanations: 1. August: Total EVM data anomalies: 4; Total EVM data anomalies with explanations: 1. September: Total EVM data anomalies: 6; Total EVM data anomalies with explanations: 1. October: Total EVM data anomalies: 14; Total EVM data anomalies with explanations: 2. November: Total EVM data anomalies: 18; Total EVM data anomalies with explanations: 0. December: Total EVM data anomalies: 40; Total EVM data anomalies with explanations: 26. 2010: January: Total EVM data anomalies: 23; Total EVM data anomalies with explanations: 1. February: Total EVM data anomalies: 27; Total EVM data anomalies with explanations: 1. Source: GAO analysis of DHS data. Note: As mentioned previously, there were no validated baselines for any of the four task orders in March 2009. [End of figure] Program officials acknowledged problems with the EVM data and stated that they meet with the prime contractor each month to discuss the EVM reports, including the reliability of the data. According to program officials, limitations in the EVM data are due, in part, to the contractor's use of estimated, rather than actual, costs for subcontractor work, which are subsequently adjusted when actual costs are received. Officials further stated that they have been working with the contractor to reduce the volume of unexplained anomalies, and they believe that the reliability of the data has improved since February 2010. However, program officials did not provide any documentation to support this statement. Without reliable EVM data, the program office is unable to identify actual cost and schedule shortfalls, which along with the other contractor tracking and oversight weaknesses discussed in this report, has limited its ability to effectively minimize program cost increases and schedule delays. Conclusions: Effective management and oversight of a program's prime contractor is essential to successfully acquiring and deploying a system like SBInet. Integral to accomplishing this is defining and implementing a range of contractor management and oversight controls (e.g., processes and practices) that reflect relevant federal guidance and best practices. To do less increases the chances that contractor-delivered products and services will not satisfy stated requirements and will not meet customer expectations. The result is incurring the additional time and expense to redo or rework contractor deliverables, accepting products and services that do not perform as intended and do not meet mission needs, or both. Overall, DHS has not done an effective job of managing and overseeing its prime contractor, including monitoring the contractor's performance. DHS has largely defined key management and oversight processes and practices that it should have followed, and it implemented a number of these processes and practices. However, several key management and oversight controls were not adequately defined, and essential controls were not implemented. Most significantly, DHS did not adequately document deliverable reviews and communicate the basis for rejecting certain deliverables in writing to the contractor, which contributed to deliverables that did not live up to expectations and necessitated rework and caused later problems. Further, technical reviews were not grounded in explicit criteria for determining when reviews should begin and conclude, which also contributed to contract deliverables requiring costly and time- consuming rework. In addition, the cost and schedule baselines for measuring the contractor's performance were frequently validated too late and without sufficient accuracy and completeness to provide a meaningful basis for understanding performance, which precluded DHS from taking timely action to correct unfavorable results and trends. Compounding these serious baseline limitations was contractor-provided data about actual performance that were replete with unexplained anomalies, thus rendering the data unfit for effective contractor management and oversight. Notwithstanding of a number of contractor management and oversight definition and implementation efforts that DHS executed well, such as defining key processes and practices and training key staff, these above-cited weaknesses collectively mean that DHS's management and oversight of its prime contractor has been a major contributor to the SBInet program's well-chronicled history of not delivering promised system capabilities on time and on budget. These limitations can be attributed to a number of factors, including gaps in how certain processes and practices were defined, as well as not enforcing other processes and practices that were defined and applicable and not taking sufficient time to review deliverables that were submitted late. The limitations can be further attributed to the fact that SBInet has from its outset lacked clear definition and stability, and thus experienced continuous change in scope and direction--an issue that we have previously reported and made recommendations to address. Collectively, these factors have helped to create a contractor management and oversight environment, which, when combined with the many other acquisition management weaknesses that we have previously reported about and made recommendations to address, have produced a program that to date has not been successful, and if not corrected, can become worse. Recommendations for Executive Action: To improve DHS management and oversight of the SBInet prime contractor, we recommend that the Secretary of Homeland Security direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, take the following four actions: * Revise and implement, as applicable, contractor deliverable review processes and practices to ensure that (1) contractor deliverables are thoroughly reviewed and are not constrained by late contractor deliverables and imposed milestones, (2) the reviews are sufficiently documented, and (3) the acceptance or the rejection of each contractor deliverable is communicated in writing to the contractor, to include explicit explanations of the basis for any rejections. * Ensure that applicable entry and exit criteria for each technical review are used and satisfied before initiating and concluding, respectively, a given review. * Establish and validate timely, complete, and accurate performance measurement baselines for each new task order or major modification of an existing task order, as appropriate, to include, but not be limited to, ensuring that (1) the work breakdown structure includes all work to be performed, (2) baseline schedules reflect the key schedule estimating practices discussed in this report, and (3) level-of-effort performance measurement in excess of 15 percent is scrutinized, justified, and minimized. * Ensure that all anomalies in contractor-delivered monthly earned value management reports are identified and explained, and report periodically to DHS acquisition leadership on relevant trends in the number of unexplained anomalies. Because we have already made recommendations in prior reports to address the other management and oversight weaknesses discussed in this report, such as those related to requirements management, risk management, and Systems Engineering Plan implementation, we are not making any additional recommendations at this time. Agency Comments and Our Evaluation: In written comments on a draft of this report, signed by the Director, Departmental GAO/OIG Liaison Office and reprinted in appendix II, DHS agreed with our four recommendations and described actions under way or planned, which we summarize below, to address them. * With respect to our recommendation to revise and implement the contractor deliverable review process, DHS stated that it is updating the process to require written documentation of each review and the communication to the contractor of review results. * With respect to our recommendation to ensure that entry and exit criteria are used to initiate and conclude each technical review, DHS stated that it has established an SBI Systems Engineering Directorate to focus on technical oversight of the acquisition process, adding that the Directorate is developing a technical review guide that describes in detail the review process and the relevant entry and exit criteria for each technical review. * With respect to our recommendation to establish and validate timely, complete, and accurate performance measurement baselines, DHS stated that it is mindful of the need to establish and maintain current performance baselines, and to plan and implement baseline updates as completely and promptly as practicable, which it indicated is done through IBRs. DHS also noted that while scheduling practices remain a challenge, it continues to make improvements to its process, including implementing scheduling tools and templates. * With respect to our recommendation to identify and explain all anomalies in monthly EVM reports, and to report periodically relevant trends to DHS acquisition leadership, DHS acknowledged the need to correctly document anomalies in the monthly EVM reports, and stated that it is working with DCMA to improve contractor quality control issues and the content of the monthly EVM reports. It also stated that it is augmenting the reports with routine conversations between contractor and project management staff. The department also committed to advising the appropriate acquisition leaders through established reporting and oversight opportunities as issues arise with contractor performance or reporting. Notwithstanding its agreement with our recommendations, the department also commented that it took exception to selected findings and conclusions regarding the program's implementation of EVM. A summary of DHS's comments and our responses are provided below. * The department stated that it took exception to our finding that it did not ensure performance measurement baselines were validated in a timely manner, and said that it was not accurate to conclude that the lack of validated baselines precluded the program office from identifying cost and schedule problems and taking corrective action. In support of these positions, the department made the following three points, which our response addresses. First, the department stated that the SBInet program office delayed formal IBRs until it had finalized negotiated modifications to the task orders, and in doing so, was able to complete an IBR within 90 days of each major task order modification. We do not question whether the program office held IBRs within 90 days of final negotiation of major task order modifications. Our point is that DHS did not validate task order performance measurement baselines (i.e., hold IBRs) within 90 days of task order award, which is what DHS guidance states should occur. As our report states, the program office only met this 90-day threshold for two of the six baselines that we reviewed. Further, the length of time to validate the performance baselines for the four task orders far exceeded 90 days (5 to 10 months), during which time DHS reports show that significant work was performed and millions of dollars were expended. In fact, the DHS reports show that most of the planned work for some of these task orders had already been performed by the time the IBR was held and the baseline was validated. As we state in our report, and DHS acknowledged in its comments, the purpose of an IBR is to verify that the performance baseline is realistic and that the scope, schedule, and risks are mutually understood by the contractor and the government before a substantial amount of work is performed. Second, DHS commented that the program office maintained what it referred to as "interim" performance measurement baselines during the period of major program scope, schedule, and budget changes. We acknowledge that in some cases the program office had these "interim" baselines. However, these baselines are the contractor-provided baselines, meaning that the program office and the contractor had not mutually agreed to the scope, schedule, and risks associated with the work to be performed. Moreover, for two of the task orders, the program office did not have an "interim" baseline, even though the contractor performed significant work under these task orders. [Footnote 55] Third, the department stated that program leadership reviewed the contractor's technical and financial performance information relative to performance measurement baselines and implemented management actions as needed. We do not question whether program leadership reviewed contractor-provided performance information or whether actions to address problems may have been taken. However, our report does conclude, as is discussed later in this section, that the combination of the EVM weaknesses that our report cites, to include unreliable performance baselines and contractor-provided performance data, did not allow the program office to identify performance problems early and to take timely actions to avoid the well-documented schedule delays and cost increases that the program has experienced. * The department expressed two concerns with how it said our report characterized and quantified EVM anomalies. First, the department stated that our report failed to distinguish between factual errors and legitimate monthly accounting adjustments. We agree that our report does not distinguish between the two types of anomalies, and would add that this was intentional because making the distinction was not relevant to our finding. Specifically, our finding is that the reasons for the numerous anomalies were not explained in the monthly EVM variance analysis reports, therefore making the true status of the program unclear. Second, DHS stated that we incorrectly concluded that both errors and adjustments are problematic, distort cost performance, and limit management insight. In response, we did not conclude that all errors and adjustments have these impacts, but rather that the lack of explanation associated with such a large volume of anomalies made the true status of the program unclear, thus limiting the program office's ability to identify actual cost and schedule shortfalls, which is certainly problematic. Further, our report cites examples of cost performance data that provide a distorted picture of actual performance vis-à-vis expectations. Accordingly, the correct characterization of the report's conclusion concerning the reliability of EVM data is that the lack of explanation of the numerous anomalies in monthly reports is problematic, provides a distorted picture of cost performance, and limits management insight. To this very point, DHS acknowledged in its comments the importance of explaining the reason for anomalies in the monthly variance reports, regardless of whether they are due to factual errors or accounting adjustments. * The department stated that it took exception to our conclusion that the program office's lack of validated baselines in particular, and EVM shortcomings in general, contributed to cost and schedule growth and made it unable to identify cost and schedule problems early and take corrective actions to avoid them. In response, we did not conclude that the lack of validated baselines alone had either of these impacts. However, we did conclude that the collection of EVM weaknesses discussed in our report, to include untimely validated baselines, incomplete and unreliable baselines, and unreliable performance data, together precluded the program office from identifying problems early and taking corrective action needed to avoid the program's well-chronicled history of schedule delays and cost increases. In support of this conclusion, we state in the report, for example, that the performance measurement baselines that we reviewed understated the cost and time necessary to complete the work because they did not capture all work in the task orders' statements of work and because they were not grounded in a range of scheduling best practices. Given that cost and schedule growth is a function of the baseline against which actual cost and schedule performance is measured, it follows logically that an understated baseline would produce actual cost overruns and schedule delays. In addition, we would note that beyond these EVM shortcomings, our report also recognizes other contract tracking and oversight, test management, and requirements management weaknesses that have collectively contributed to the program's cost, schedule, and performance shortfalls. In addition to the above points, DHS provided technical comments, which we have incorporated in the report as appropriate. We are sending copies of this report to the Chairmen and Ranking Members of the Senate and House Appropriations Committees and other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We will also send copies to the Secretary of Homeland Security, the Commissioner of U.S. Customs and Border Protection, and the Director of the Office of Management and Budget. In addition, the report will be available at no cost on the GAO Web site at [hyperlink, http://www.gao.gov]. Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Signed by: Randolph C. Hite: Director, Information Technology Architecture and Systems Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives were to determine the extent to which the Department of Homeland Security (DHS) (1) has defined and implemented effective controls for managing and overseeing the Secure Border Initiative Network (SBInet) prime contractor and (2) is effectively monitoring the prime contractor's progress in meeting cost and schedule expectations. To accomplish our objectives, we focused on four key task orders--the Arizona Deployment Task Order, the Design Task Order, the System Task Order, and the Command, Control, Communications, and Intelligence/Common Operating Picture Task Order--that are integral to the design, development, and deployment of the first increment of SBInet, known as Block 1. We also focused on the SBInet System Program Office's (SPO) contracting and oversight activities that occurred from June 2008 through February 2010. To determine the extent to which DHS has defined and implemented effective controls for managing and overseeing the SBInet prime contractor, we focused on training, verifying and accepting contract deliverables, conducting technical reviews, and conducting management reviews. We chose these four because each is important to tracking and overseeing contractor performance for the four task orders. * Training. We compared relevant DHS contractor management and oversight training requirements to the program managers', contracting officers', and contracting officer's technical representatives' training certifications. * Verifying and accepting contract deliverables. We compared U.S. Customs and Border Protection's (CBP) process for verifying and accepting contract deliverables to leading industry practices[Footnote 56] to identify any variances. We then assessed a nonprobability, random sample of 28 contract deliverables that the SPO identified as being delivered between June 1, 2008, and August 31, 2009.[Footnote 57] We also judgmentally selected one additional review that was conducted between September 1, 2009, and February 28, 2010.[Footnote 58] For each of the 29 deliverables, we reviewed relevant documentation, such as the contract deliverables, review comment forms, and documented communications with the prime contractor indicating acceptance or rejection of the deliverable, and compared them to the CBP process and leading industry practices to determine what, if any, deviations existed. * Technical reviews. We compared relevant DHS and CBP guidance and entry and exit criteria in the task order Data Item Descriptions to leading industry practices to identify any variances. We assessed a nonprobability, random sample of technical reviews. Specifically, we assessed a technical review from each of the eight unique combinations of task orders and review types held between June 1, 2008, and August 31, 2009. We also judgmentally selected one additional review that was conducted between September 1, 2009, and February 28, 2010.[Footnote 59] For each of the nine reviews, we compared the package of documentation prepared for and used during these reviews to the criteria defined in the relevant task orders to determine the extent to which the reviews satisfied the criteria. * Management reviews. We compared relevant CBP guidance to leading industry practices to identify any variances. We then compared relevant documentation prepared for and used during monthly joint program management reviews to determine the extent to which the reviews addressed cost, schedule, and program risks. We also assessed a nonprobability sample of 11 action items that were identified during the reviews held from October 2008 through October 2009,[Footnote 60] and assessed relevant documentation to determine the extent to which they were tracked to closure. To determine the extent to which DHS is effectively monitoring the prime contractor's progress in meeting cost and schedule expectations, we focused on the program's implementation of earned value management (EVM) because it was the tool used to monitor the contractor's cost and schedule performance. Specifically, we analyzed the six performance measurement baselines, and the associated integrated baseline review documentation, such as briefings, the work breakdown structure (WBS) governing all task orders, task order statements of work, schedules, monthly contract performance reports, control account plans, and responsibility assignment matrixes. In doing so, we compared this documentation to EVM and scheduling best practices as identified in our Cost Estimating and Assessment Guide.[Footnote 61] Specifically, for each of the six baselines: * We reviewed control account plans and responsibility assignment matrixes to determine the period of performance and scope of work for each baseline, compared the work described in the respective task order statements of work to the work described in the responsibility assignment matrix, and reviewed the control account plans to determine the extent to which the level-of-effort measurement method was to measure contractor performance. * We analyzed the schedule presented at each baseline review against eight key schedule estimating practices in our Cost Estimating and Assessment Guide.[Footnote 62] In doing so, we used commercially available software tools to determine whether each schedule, for example, included all critical activities, a logical sequence of activities, and reasonable activity durations. Further, we characterized the extent to which the schedule met each of the practices as either not met, minimally met, partially met, substantially met, or met. * We analyzed the contract performance reports for each of the four task orders for each month that there was a validated baseline. Specifically, we identified instances of the following: (1) negative planned value, earned value, or actual cost; (2) planned value and earned value without actual cost; (3) earned value and actual cost without planned value; (4) actual cost without planned value or earned value; (5) earned value without planned value and actual cost; (6) inconsistencies between the estimated cost at completion and the planned cost at completion; (7) actual cost exceeding estimated cost at completion; and (8) planned or earned values exceeding planned cost at completion. To determine the number of anomalies, we identified each WBS element that had one or more of the above anomalies. Then, we identified the number of WBS elements at the beginning and the end of the baseline period of performance, and calculated the average number of WBS elements. We used this to determine the percentage of WBS elements with anomalies for each task order and for each month for which there was a validated baseline. * To support our work across this objective, we interviewed officials from the Department of Defense's Defense Contract Management Agency (DCMA), which provides contractor oversight services to the SPO, including oversight of EVM implementation, and prime contractor officials. We also reviewed DCMA monthly status reports and corrective action reports. For both objectives, we interviewed program officials to obtain clarification on the practices, and to determine the reasons for any deviations. To assess the reliability of the data that we used to support the findings in this report, we reviewed relevant program documentation to substantiate evidence obtained through interviews with knowledgeable agency officials, where available. We determined that the data used in this report are sufficiently reliable. We have also made appropriate attribution indicating the sources of the data. We performed our work at the CBP headquarters and prime contractor facilities in the Washington, D.C., metropolitan area and with DCMA officials from Huntsville, Alabama. We conducted this performance audit from June 2009 to October 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Homeland Security: Department of Homeland Security: Washington, DC 25528: September 23, 2010: Mr. Randolph C. Hite: Director, Information Technology Architecture and Systems Issues: U.S. Government Accountability Office: Washington, D.C. 20548: Dear Mr. Hite: Thank you for the opportunity to review and offer comment on the Government Accountability Office (GAO) draft report entitled, "Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of its Prime Contractor," GAO-11-6, dated October 2010. GAO was asked to determine the extent to which the Department of Homeland Security (OHS) (I) defined and implemented effective controls for managing and overseeing the SBInet prime contractor and (2) effectively monitored the prime contractor's progress in meeting cost and schedule expectations, To do this, GAO analyzed key performance documentation against relevant guidance and industry best practices, and interviewed program officials. GAO acknowledges that OHS has largely defined key management and oversight processes and procedures for verifying and accepting contract deliverables and conducting technical reviews. However, the draft report identifies measures that the U.S. Customs and Border Protection (CBP) SBInet can take to further strengthen contractor management and oversight. GAO made four recommendations to improve DHS management and oversight of the SBInet prime contractor. CBP concurs with the four recommendations. However, not withstanding our concurrence, we take exception to several assertions regarding aspects of the reliability of the Secure Border Initiative (SB1) program's Earned Value Management (EVM) practices, and government oversight of contractor activities. The comments below address our specific concerns. First, GAO asserts the program office did not ensure EVM performance measurement baselines (PMB) were "validated" in a timely manner, and concludes that the program office's EVM shortcomings led to cost and schedule growth in the program. We firmly take exception to these assertions. As background to many of the EVM findings and recommendations, GAO acknowledges throughout the report the significant amount of program change occurring with the SBInet program in late 2008-2009. These positive changes followed a major program re-baselining (Acquisition Program Baseline) with the Department as well as SBInet final system design updates after Critical Design Review. The program baseline change drove major scope, schedule, and budget changes across all of the Boeing task orders and associated performance measurement baselines. GAO concluded that without a "validated" baseline through the conduct of an Integrated Baseline Review (IBR), the program office was unable to identify cost and schedule problems early and to take timely corrective actions to mitigate those problems. The report also concluded that the program's lack of a "validated" baseline contributed to cost overruns and schedule delays. We believe these conclusions are inaccurate, and the causal assertion is not supported by any analysis presented in the report. Of significance is the fact that the program office maintained interim performance measurement baselines throughout the transition period with the best available task plan information, as required by EVM best practices. Program leadership also reviewed in various forum, the contractor's technical and financial performance information relative to the established PMBs and implemented rigorous management actions as needed. The program office delayed formal IBR(s) until completing final, negotiated task order modifications, and in each case the IBR was completed within the timeframe specified in DEIS directives (less than 90 days after the major contract modification). Certainly during a period of significant program changes. earned value management is challenged, but the program office believes prudent and compliant EVM measures were executed. GAO documented other EVM implementation concerns reported by the program office, specifically the prime contractor's over-reliance on the "level of effort" (LoE) EVM measure, and recurring quality control lapses, or "anomalies," contained in the contractor's monthly reports. For the LoE concern, the program office and the prime contractor initiated several actions to improve task order work planning by creating discreet work packages, enabling improved performance management and issue identification. However, given the stage of the program today, we will continue to see high levels of effort in the task order baselines due to maintaining sizeable contractor program management "overhead" (typically LoE), while transitioning from construction and testing completion into a sustaining system and software engineering mode (typically LoE), as well as performing a variety of system maintenance functions (also typically LoE). Secondly. we are concerned with GAO's characterization and quantification of "anomalies." The SBI program office demands accurate contractor performance reports, and is continuously engaged with the contractor to fix underlying problems. The draft report fails to distinguish between factual errors versus legitimate monthly accounting adjustments. For example: * Errors, and the resultant corrections, occur when the contractor collects and reports costs for one activity when the costs actually apply to another activity, and might include incorrect contract charge numbers issued to subcontractors or other simple administrative mistakes. Errors distort cost performance status however the SBI program office and the contractor promptly identify and take corrective actions to correct identified problems and improve quality controls for final monthly reports. * Adjustments routinely occur when the prime contractor reports estimated costs for a subcontractor (while awaiting a subcontractor to submit an invoice), and then in a subsequent report, the prime reconciles previously reported estimated costs with the final, invoiced actual costs. Adjustments may also be necessary to reflect contractor rate changes. These adjustments improve the accuracy and reliability of the cumulative cost performance data, yet may appear anomalous for the current month. The prime contractor's adjustments are completed in accordance with the contractor's approved system guide and established best practices. The GAO report combines both errors and adjustments when quantifying "anomalies" and incorrectly concludes that both errors and adjustments are problematic, distorting cost performance, and limits management insight. Regardless of error corrections or adjustments, the contractor is required to document such actions in the monthly cost performance report (Format 5). Over the past eight months, the contractor implemented quality control measures, and has significantly improved the process for monthly reporting of both errors and anomalies as well as significant accounting adjustments. The recommendations and CBP's corrective actions to address the recommendation are described below. Recommendation 1: Revise and implement, as applicable, contractor deliverable review processes and practices to ensure that (I) contactor deliverables are thoroughly reviewed and are not constrained by late contractor deliverables and imposed milestones, (2) the reviews are sufficiently documented, and (3) the acceptance or the rejection of each contractor deliverable is communicated in writing to the contractor, to include explicit explanations of the basis for any rejections. CBP Response: CBP concurs with the recommendation. Since the draft report was issued, we have taken several major steps to improve the SBInet program management structure, capabilities, procedures, and processes. This includes significant improvement in SBInet execution of the Contract Deliverable Requirements List (CDRL) review process. The program has commenced reviewing and updating the current CDRL, review policy. The updated CDRL policy will document the process and procedures for program technical reviews, and written documentation to the contractor regarding results of the review to include reviewer comment forms and formal communication of the acceptance, conditional acceptance, or rejection of the contract deliverable. In implementing the updated CDRL policy, contract deliverable review participants will be trained to ensure that reviewers are technically qualified and possess the procedural knowledge to effectively review and provide concise and thorough, review comments and that the process is fully aligned with the approved CDRL review policy. Due Date: Policy update expected to be completed by December 2010. Recommendation 2: Ensure that applicable entry and exit criteria for each technical review are used and satisfied before initiating and concluding, respectively, a given review. CLIP Response: CBP concurs with the recommendation. We have taken several major steps to improve the SBInet program management structure, capabilities, procedures, and processes. This has led to a more rigorous application of review entry and exit criteria. In addition, the SBI Systems Engineering (SE) Directorate was established to ensure that systems engineering capabilities are focused on providing the technical oversight required to support knowledge-based decision making throughout the acquisition process. The SE Directorate is incorporating the DIN Engineering Life Cycle best practices into a Technical Review Guide (TRG) that describes in detail the SE technical review process and the relevant entry and exit criteria for each technical review. The draft TRG is currently under review and when implemented will serve as the roadmap used by the SBI System Program Office to ensure that all relevant entry and exit criteria are understood and satisfied in a manner consistent with programmatic cost, schedule, and performance risk considerations. Due Date: Technical Review Guide expected to be completed and implemented by December 2010. Recommendation 3: Establish and validate timely, complete, and accurate performance measurement baselines for each new task order or major modification of an existing task order, as appropriate, to include, but not be limited to, ensuring that (1) the work breakdown structure includes all work to be performed; (2) baseline schedules reflect the key schedule estimating practices discussed in this report; and (3) level-of-effort performance measurement in excess of 15 percent is scrutinized, justified, and minimized. CRP Response: CBP concurs with the recommendation. The SBI program office is mindful of the need to establish and maintain current Performance Measurement Baselines (PMB), and to plan and implement baseline updates as completely and promptly as practicable. For new task orders and significant modifications. this is done through Integrated Baseline Reviews (IBRs). The purpose of the IBR is to confirm that the PMB covers the scope of work; the work is realistically and accurately scheduled; the proper mix of resources and budgets are assigned; the appropriate management processes are in place; and the risks associated with the PMB are identified, documented, and managed. Prior to an IBR, the program office routinely evaluates the control accounts, and the integrated PMB, for consistency, traceability, and completeness relative to the work breakdown structure (WBS), to include a cross-check of the WBS back to the control accounts. Also, the program office documents any adverse findings and resolves such with the prime contractor as part of the 1BR proceedings. As described above, the program office scrutinizes, and with the prime contractor, minimizes level of effort tasks in the PMB. Scheduling practices remain a challenge, and as reported in our response to prior GAO reviews, we continue to upgrade our processes. The program office is revising our scheduling standards to reflect procedural improvements for schedule development, management and surveillance. We have created and put into practice the use of 26 diagnostic schedule filters to analyze the health of our schedules. The filters fully leverage GAO's scheduling best practices. We have also implemented improvements to our practices for developing Integrated Master Plan/Integrated Master Schedule that utilizes newly revised planning and schedule templates based on the OHS Systems Engineering Lifecycle. Due Date: Ongoing corrective actions are in place, and the program office continues to assess and manage contractor progress and improvements. Recommendation 4: Ensure that all anomalies in contractor-delivered monthly earned value management reports are identified and explained, and report periodically to DHS acquisition leadership on relevant trends in the number of unexplained anomalies. CRP Response: CBP concurs with the recommendation. The monthly contract performance report "anomalies" (which include a variety of legitimate monthly updates to actual costs, rate changes, etc.) need to be documented correctly each month. The SBI program office shared with the GAO significant, recent activities and progress with associated corrective actions: * Fixing prime contractor systemic and quality control issues, with the assistance of the Defense Contract Management Agency, through active participation in program EVM surveillance; and; * Improving discipline and content of Format 5 reporting narrative, augmented with routine management discussions with contractor and government project management staff. Our focus remains to ensure we are able to determine program status relative to well-defined performance baselines. As issues arise with either contractor performance or reporting, we will advise the appropriate acquisition leaders through established reporting and oversight opportunities. Due Date: Ongoing; corrective actions are in place, and the program office continues to assess and manage contractor progress and improvements. Sincerely, Signed by: Jerald E. Levine: Director: Departmental GAO/OIG Liaison Office: [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Randolph C. Hite, (202) 512-3439 or hiter@gao.gov: Staff Acknowledgments: In addition to the contact named above, Deborah A. Davis (Assistant Director), Tisha Derricotte, Neil Doherty, Kaelin Kuhn, Lee McCracken, Jamelyn Payan, Karen Richey, Matt Snyder, Sushmita Srikanth, Stacey Steele, and Matthew Strain made key contributions to this report. [End of section] Footnotes: [1] We also selected one additional contract deliverable and one additional technical review for review. [2] EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of a program's total costs. [3] SEI, CMMI® for Acquisition, Version 1.2 (Pittsburgh, Pa., November 2007); and GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). Also, recognizing the importance of ensuring quality earned value data, ANSI and the Electronic Industries Alliance (EIA) jointly established a national standard for EVM systems in May 1998 (ANSI/EIA-748-A-1998). This standard, commonly called the ANSI standard, is comprised of guidelines on how to establish a sound EVM system. This document was updated in July 2007 and is referred to as ANSI/EIA-748-B. [4] The remaining $126 million was for upgrading CBP telecommunications links. [5] The physical infrastructure (e.g., physical fencing) portion of SBI is managed on a day-to-day basis by CBP's Office of Finance Facilities Management and Engineering division. [6] IDIQ contracts are used when the government cannot predetermine, above a specified minimum, the precise quantities of supplies or services that it will require during the contract period. Under IDIQ contracts, the government can issue delivery orders (for supplies) and task orders (for services). [7] The $35 million was part of American Recovery and Reinvestment Act of 2009 funding. Pub. L. No. 111-5, 123 Stat. 115, 162 (Feb. 17, 2009). [8] See for example, GAO, Secure Border Initiative: DHS Needs to Reconsider Its Proposed Investment in Key Technology Program, [hyperlink, http://www.gao.gov/products/GAO-10-340] (Washington, D.C.: May 5, 2010); Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk, [hyperlink, http://www.gao.gov/products/GAO-10-158] (Washington, D.C.: Jan. 29, 2010); and DOD Business Systems Modernization: Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas, [hyperlink, http://www.gao.gov/products/GAO-08-896] (Washington, D.C.: Sept. 8, 2008). [9] GAO, Secure Border Initiative: SBInet Expenditure Plan Needs to Better Support Oversight and Accountability, [hyperlink, http://www.gao.gov/products/GAO-07-309] (Washington, D.C.: Feb. 15, 2007). [10] GAO, Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering Key Technology Investment, [hyperlink, http://www.gao.gov/products/GAO-08-1086] (Washington, D.C.: Sept. 22, 2008). [11] GAO, Secure Border Initiative: Technology Deployment Delays Persist and the Impact of Border Fencing Has Not Been Addressed, [hyperlink, http://www.gao.gov/products/GAO-09-896] (Washington, D.C.: Sept. 9, 2009). [12] [hyperlink, http://www.gao.gov/products/GAO-10-158]. [13] A test case is a set of test inputs, execution conditions, and expected results developed for a particular objective, such as to verify compliance with a specific requirement. [14] [hyperlink, http://www.gao.gov/products/GAO-10-340]. [15] GAO, Department of Homeland Security: Assessments of Selected Complex Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-10-588SP] (Washington, D.C.: June 30, 2010). [16] See, for example, Federal Acquisition Regulation, part 46, quality assurance; and SEI, CMMI® for Acquisition. [17] SEI, CMMI® for Acquisition. [18] Training requirements for key officials are defined in three DHS policies: (1) Acquisition Certification Requirements for Program Manager, Management Directive (MD) 0782 (May 26, 2004); (2) Contracting Officer Warrant Program (Apr. 16, 2007); and (3) COTR Certification, Appointment & Responsibilities, MD 0780.1 (Dec. 20, 2004). [19] DHS has categorized major information technology investments as Levels 1, 2, and 3. A Level 1 major investment is defined in DHS Acquisition Directive 102-01 as program at or more than $1 billion in life cycle costs. [20] DHS's Acquisition Certification Requirements for Program Manager, MD 0782 (May 26, 2004) establishes three levels of certification: Level 1 establishes the foundation for managing increasingly complex investments or portfolios, Level 2 includes knowledge of strategic and capital planning, and Level 3 represents mastery of the knowledge and skills associated with the complexities of managing DHS's most strategic and sophisticated acquisitions. [21] For warrant authority up to $100,000, the contracting officer must have a Federal Acquisition Certification in Contracting (FAC-C) Program Level I certification; for warrant authority up to $25 million, the contracting officer must have a FAC-C Level II certification; and for warrant authority more than $25 million, including unlimited warrant authority, the contracting officer must have a FAC-C Level III certification. [22] SEI, CMMI® for Acquisition. [23] A project manager is responsible for each of the task orders. These project managers have overall responsibility and accountability for achieving the task order's scope, cost, schedule, and technical objectives. [24] According to the SPO, subject matter experts review the deliverables based on the contractual requirements that are outlined in the Data Item Descriptions, as well as their own expertise and knowledge about the deliverables. [25] These 29 deliverables actually resulted in 46 separate submissions because, for example, an initial submission was rejected and had to be resubmitted. [26] The NOC/SOC monitors networked equipment, provides alerts for network problems, protects equipment from network-based attacks, and provides user authentication. [27] According to the C3I/COP task order statement of work, the plan was to be submitted 10 days before the test readiness review. However, the contractor submitted the plan the day of this review. [28] This system is the program office's requirements management database. [29] [hyperlink, http://www.gao.gov/products/GAO-10-158]. [30] SEI, CMMI® for Acquisition. [31] DHS, Acquisition Instruction/Guidebook, 102-01-001, Appendix B, Systems Engineering Life Cycle, Interim Version 1.9 (Nov. 7, 2008). [32] [hyperlink, http://www.gao.gov/products/GAO-10-340]. [33] [hyperlink, http://www.gao.gov/products/GAO-10-158]. [34] In 2008, the SPO contracted with an IV&V agent to, among other things, review the program's test documentation, execution, and related processes. Generally, the purpose of IV&V is to independently determine the extent to which program processes and products meet quality standards. The use of IV&V is recognized as an effective practice for large and complex system development and acquisition programs, like SBInet. [35] SEI, CMMI® for Acquisition. [36] CBP, System Life Cycle Handbook, Version 1.2 (Sept. 30, 2008). [37] This database is Boeing-managed and tracks SBInet action items throughout their lifecycle. The database captures such information as individual action assignments, status and updates, and relevant comments. [38] Office of Management and Budget Memorandum, M-05-23 (Aug. 4, 2005). [39] ANSI/EIA-748-B. [40] As mentioned previously, DCMA conducts periodic surveillance reviews of the prime contractor's EVM system implementation. [41] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 227-229, and Department of Defense, Earned Value Management Implementation Guide (October 2006). [42] DHS, Department of Homeland Security Acquisition Manual, Section 3034.202 Integrated Baseline Reviews (October 2009). Integrated baseline reviews are intended to verify that the baseline is realistic and the scope, schedule, and risks are mutually understood by the contractor and the government. [43] Physical infrastructure includes physical fencing and roads. [44] The scope of the work is generally defined in a work breakdown structure, which is a hierarchical structure that shows how work tasks relate to one another as well as to the overall end product. [45] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 215-231. [46] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 215. [47] The critical path represents the chain of dependent activities with the longest total duration in the schedule. [48] Float is the amount of time a task can slip before affecting the critical path. [49] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 218-224. [50] We did not assess the ninth practice because the schedules that were reviewed at the IBRs were the initial schedules for each task order, and as such, were not updated. [51] ANSI/EIA-748-B. [52] Level-of-effort is unmeasured effort of a general or supportive nature which does not produce definitive end products (e.g., program administration). [53] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 226. [54] [hyperlink, http://www.gao.gov/products/GAO-09-3SP], 226. [55] The program office did not have a baseline for ADTO and DTO, from July 2008 through October 2008 and from June 2008 through January 2009, respectively. [56] Software Engineering Institute, Capability Maturity Model Integration® for Acquisition, Version 1.2 (Pittsburgh, Pa., November 2007). [57] We conducted a nonprobability sample because we determined that the SPO's contract deliverable log was not sufficiently reliable for purposes of identifying all required deliverables during our time frames, and thus we did not select a sample that could be projected to the total population. [58] We selected one additional deliverable that was associated with the technical review noted below to provide a more current assessment of the program's adherence to its process. [59] We selected one additional technical review based on the combinations of task orders and review types in our existing sample to provide a more current assessment of the program's implementation of established criteria. [60] We selected one action item for each month in which a joint program management review was held and action items were documented. [61] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009), 215 steps 1-6. [62] We did not assess the ninth practice--updating the schedule using logic and durations--because the schedules that were reviewed at the integrated baseline reviews were the initial schedules for each task order, and as such, were not updated. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: