This is the accessible text file for GAO report number GAO-11-596 entitled 'DHS Science and Technology: Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met' which was released on July 14, 2011. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: June 2011: DHS Science and Technology: Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met: GAO-11-596: GAO Highlights: Highlights of GAO-11-596, a report to congressional requesters. Why GAO Did This Study: In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate’s (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs—that is, those with over $300 million in life-cycle costs—to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights. What GAO Found: TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components’ documents describing the programs’ performance requirements, as required by the T&E directive. DHS senior level officials considered TES’s T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components’ test agents-—a government entity or independent contractor carrying out independent operational testing for a major acquisition-—or document its review of other component acquisition documents, such as those establishing programs’ operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES’s decision-making process and ensure that it is overseeing components’ T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program’s operational requirements—the key performance requirements that must be met for a program to achieve its intended goals—can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness. What GAO Recommends: GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO’s recommendations. View [hyperlink, http://www.gao.gov/products/GAO-11-596] or key components. For more information, contact David Maurer, (202) 512- 9627, maurerd@gao.gov. [End of section] Contents: Letter: Background: TES Met Some Oversight Requirements for T&E of Acquisition Programs Reviewed; Additional Steps Needed to Ensure That All Requirements Are Met: TES and Component Officials Cited Challenges in Coordinating and Overseeing T&E across DHS; Efforts Are Underway to Address Some Challenges: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Descriptions of Selected Major Acquisitions Programs: Appendix II: DHS's Fiscal Year 2010 Major Acquisition Programs: Appendix III: Comments from the Department of Homeland Security: Appendix IV: GAO Contact and Staff Acknowledgments: Tables: Table 1: Summary of TES's Key Responsibilities Outlined in the T&E Directive: Table 2: DHS Fiscal Year 2010 Major Acquisition Programs: Figures: Figure 1: TES Key Responsibilities as Part of the DHS Acquisition Process: Figure 2: Selected Major Acquisition Programs and Their Components and Acquisition Phases as of April 2011: Figure 3: TES's Involvement in Reviewing and Approving Test and Evaluation Master Plans for 11 Selected Acquisition Programs as of April 2011: Figure 4: TES's Involvement in Reviewing and Approving Operational Test Plans for 11 Selected Acquisition Programs as of April 2011: Figure 5: TES's Involvement in Reviewing and Approving Operational Test Agents for 11 Selected Acquisition Programs as of April 2011: Abbreviations: ADE: Acquisition Decision Event: ARB: Acquisition Review Board: ASP: Advanced Spectroscopic Portal program: AT-2: Advanced Technology 2 program: CBP: U.S. Customs and Border Protection: DHS: Department of Homeland Security: DNDO: Domestic Nuclear Detection Office: DOD: Department of Defense: ICE: U.S. Immigration and Customs Enforcement: IRDS: Interoperable Rapid Deployment Systems: MCS: Interoperable Rapid Deployment Systems Mobile Communication Systems: OHA: Office of Health Affairs: P25: Project 25-compliant systems: S&T: Science and Technology Directorate: SBInet: Secure Border Initiative Network: TEMP: test and evaluation master plan: TACCOM: Atlas Tactical Communications program: T&E: Test and Evaluation: TES: Test & Evaluation and Standards Office: TSA: Transportation Security Administration: USCIS: U.S. Citizenship and Immigration Service: [End of section] United States Government Accountability Office: Washington,DC 20548: June 15, 2011: The Honorable Joseph I. Lieberman: Chairman: The Honorable Susan M. Collins: Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Daniel K. Akaka: Chairman: Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia: Committee on Homeland Security and Governmental Affairs: United States Senate: Department of Homeland Security (DHS) acquisitions represent hundreds of billions of dollars in life-cycle costs to support a wide range of missions, including securing the nation's borders, mitigating natural disasters, screening airline passengers and baggage, and investigating security threats. DHS acquisition spending has increased by 50 percent from $9.1 billion in fiscal year 2004 to $13.6 billion in fiscal year 2010.[Footnote 1] A key goal of DHS's acquisitions process is ensuring that programs and technologies meet technical and performance specifications and are tested in DHS's operational environment, and that the results of these tests are evaluated before these programs and technologies are allowed to progress toward purchase and deployment. The Homeland Security Act of 2002 created DHS and, within it, established the Science and Technology Directorate (S&T).[Footnote 2] The act provided S&T with responsibility for conducting national research, development, test and evaluation (T&E), and procurement of technology and systems for, among other things, detecting, preventing, protecting against, and responding to terrorist attacks.[Footnote 3] S&T's Test & Evaluation and Standards Office[Footnote 4] (TES) is responsible for, among other things, conducting oversight over T&E activities across all of DHS's components to ensure that major acquisitions--those with life-cycle costs of over $300 million dollars [Footnote 5]--being considered by DHS are appropriately tested and evaluated by DHS component agencies prior to their purchase and deployment. We have previously reported on several major DHS acquisitions that were deployed before appropriate T&E was successfully completed. For example, in October 2009, we reported that the Transportation Security Administration (TSA) procured and deployed explosives trace portal equipment--a machine which detects traces of explosives on airline passengers by using puffs of air to dislodge particles from the passengers' body or clothing into an analyzer--even though TSA officials were aware that earlier tests did not demonstrate reliable performance in an airport environment.[Footnote 6] We recommended that TSA conduct an evaluation and determine whether it was cost effective to continue to use these machines. TSA concurred with this recommendation and later halted further deployment of these machines due to performance, maintenance, and installation problems. As of April 2011, TSA reported that it had removed all 101 machines that it had deployed from airports. Furthermore, in January 2010, we reported that DHS had not effectively managed key aspects of the testing of Customs and Border Protection's (CBP) Secure Border Initiative Network (SBInet), a multibillion dollar program to deliver surveillance and decision-support technologies along the U.S. border with Mexico and Canada.[Footnote 7] Among other things, we reported that test procedures were largely not executed as written and changes to these procedures were not made according to a documented quality assurance process which increases the risk that the procedures would not support test objectives or reflect the system's ability to perform as intended. We made a number of recommendations related to the content, review, and approval of test planning documentation and resolution of system problems, with which DHS generally concurred. In January 2011, the Secretary of Homeland Security directed CBP to end SBInet as originally conceived and to develop a new border technology deployment plan.[Footnote 8] These past problems highlight the importance that T&E plays in the successful development of major acquisition programs and technologies, as well as the importance of overseeing the T&E efforts of DHS's components. In May 2009, DHS issued a T&E directive requiring components to, among other things, ensure adequate and timely T&E is performed to support informed acquisition decision making and also requires that S&T's TES oversee these activities by reviewing component T&E activities and documentation, and approving those T&E activities and documentation related specifically to operational testing. You asked us to evaluate S&T's efforts to oversee T&E across DHS and identify any challenges that it faces in performing this mission. Specifically, this report addresses: (1) the extent to which TES oversees T&E of selected DHS major acquisition programs throughout the system acquisition process; and: (2) what challenges, if any, TES and component officials report facing in coordinating and overseeing T&E across DHS acquisition programs. To address our objectives, we reviewed DHS departmental and component- level policies and guidance, such as DHS's acquisition directive and guidebook, T&E directive, and our past reports. We also reviewed relevant program documentation, including memoranda of acquisition decision events in the acquisition process. Further, we conducted interviews with relevant DHS and component officials involved in the acquisition process including T&E of programs and technologies. We focused our review generally on the period after May 2009 when DHS issued its T&E directive since there were no specific T&E requirements prior to its issuance.[Footnote 9] We focused our review on TES's T&E staff and activities and did not review the efforts of TES staff who were engaged in developing national standards to meet homeland security mission needs or TES staff in two testing facilities, since developing standards and conducting tests did not relate to overseeing T&E across DHS components and thus, were outside the scope of our review. Regarding our objective to determine the extent to which TES oversees T&E of selected DHS major acquisition programs throughout the system acquisition process, we initially selected a nonprobability sample of 12 programs of the 86 on DHS's major acquisitions list for fiscal year 2010 and, for 11 of the 12 programs, we analyzed related acquisition and T&E documentation, such as the program's test plans, as well as memoranda documenting approval of operational test agents (test agents), and interviewed DHS component officials to help determine the extent that TES reviewed these plans and agents and documented its approval during fiscal year 2010.[Footnote 10] Selections were based on three factors--programs which had undergone a senior DHS management review in fiscal year 2010, programs representing different DHS components, and programs which were overseen by all nine of TES's Test Area Managers. For 11 programs, we conducted semistructured interviews and collected information from component-level officials involved in the development of T&E documentation. The results of these analyses cannot be generalized to TES's efforts to oversee all major acquisition programs, but they provide informative examples of the extent to which TES carried out its oversight responsibilities. We reviewed documentation of TES's oversight efforts and compared them to the requirements in the DHS acquisition and T&E directive to determine the extent to which TES fulfilled its responsibilities. We also compared TES efforts to document their oversight to Standards for Internal Control in the Government.[Footnote 11] Further, we conducted site visits to S&T's Transportation Security Laboratory and TSA's Transportation Security Integration Facility to obtain an overview of testing in general and to specifically observe testing of transportation security technologies. See appendix I for a description of the 11 DHS major acquisition programs selected for our analysis. To determine what challenges, if any, TES and component officials report facing in coordinating and overseeing T&E of major DHS acquisition programs, we interviewed TES and component officials responsible for T&E from the 11 selected programs and conducted semistructured interviews with all nine TES Test Area Managers who were primarily responsible for conducting T&E oversight in fiscal year 2010. We analyzed and categorized the challenges that these officials said they faced in conducting T&E of their programs. We discussed the major challenges we identified with TES officials, who agreed with our evaluation. We also reviewed various TES and DHS initiatives to address these challenges. For example, we reviewed minutes from TES's T&E Council, which was formed to promote T&E best practices and lessons learned in establishing consistent T&E policies and processes for use in acquisition programs throughout DHS. In addition, we reviewed minutes of T&E Council working groups, which were formed to address particular challenges, such as testing and evaluating information technology acquisitions. We also reviewed documents and interviewed TES and component officials with regard to T&E training and certification efforts. Further, we interviewed the Under Secretary for S&T and other S&T executives regarding S&T's T&E efforts and the challenges that S&T and TES face in overseeing T&E across DHS. Finally, we reviewed our past work related to these challenges. We conducted this performance audit from May 2010 through June 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Overview of DHS Acquisition Process: DHS acquisitions support a wide range of missions and investments including ships and aircraft, border surveillance and screening equipment, nuclear detection equipment, and systems to track the department's financial and human resources. In support of these investments, DHS, in 2003, established an investment review process to help reduce risk and increase the chances for successful acquisition outcomes by providing departmental oversight of major investments throughout their life cycles and to help ensure that funds allocated for investments through the budget process are being spent wisely, efficiently, and effectively. Our work over the past several years has consistently pointed to the challenges DHS has faced in effectively managing and overseeing its acquisition of programs and technologies. * In November 2008, we reported that DHS had not effectively implemented its investment review process, and as a result, the department had not provided the oversight needed to identify and address cost, schedule, and performance problems for its major acquisitions.[Footnote 12] Specifically, we reported that of the 48 major investments reviewed requiring milestone or annual reviews, 45 were not reviewed in accordance with the departments' investment review policy, and 18 were not reviewed at all. Four of these investments had transitioned into a late acquisition phase--production and deployment--without any required reviews. We recommended and DHS concurred that DHS identify and align sufficient management resources to implement oversight reviews in a timely manner throughout the investment life cycle. * In June 2010, we reported that over half of the 15 DHS programs we reviewed awarded contracts to initiate acquisition activities without component or department approval of documents essential to planning acquisitions, setting operational requirements, and establishing acquisition program baselines.[Footnote 13] Our work noted that without the development, review, and approval of these key acquisition documents, agencies are at risk of having poorly defined requirements that can negatively affect program performance and contribute to increased costs. In January 2011, DHS reported that it has begun to implement an initiative to assist programs with completing departmental approval of acquisition program baselines. * In our February 2011 biennial update of the status of high-risk areas needing attention by Congress and the executive branch, we continued to designate DHS's implementation and transformation, which includes the department's management functions, as a high-risk area. [Footnote 14] For example, because of acquisition management weaknesses, major programs, such as SBInet, have not met capability, benefit, cost, and schedule expectations. Further, DHS had not fully planned for or acquired the workforce needed to implement its acquisition oversight policies as we previously recommended.[Footnote 15] As of January 2011, DHS reported that it had increased its acquisitions management staffing and planned to hire more staff to develop cost estimates. DHS has taken several actions to address these recommendations and implement more discipline and rigor in its acquisition processes. Specifically, DHS created the Acquisition Program Management Division in 2007 to develop and maintain acquisition policies, procedures, and guidance as a part of the system acquisition process.[Footnote 16] DHS also issued an interim acquisition directive and guidebook in November 2008 for programs to use in preparing key documentation to support component and departmental making.[Footnote 17] In January 2010, DHS finalized the acquisition directive which established acquisition life- cycle phases and senior-level approval of each major acquisition program at least three times at key acquisition decision events during a program's acquisition life-cycle.[Footnote 18] This directive established the acquisition life-cycle framework with four phases: (1) identify a capability need (need phase); (2) analyze and select the means to provide that capability (analyze/ select phase); (3) obtain the capability (obtain phase); and: (4) produce, deploy, and support the capability (produce/deploy/support phase). Each acquisition phase culminates in a presentation to the Acquisition Review Board (ARB), which is to review each major acquisition (that is, those designated as level 1 or level 2 programs) at least three times at key acquisition decision events during a program's acquisition life cycle.[Footnote 19] The acquisition decision authority--the Chief Acquisition Officer or other designated senior- level official--is to chair the ARB and decide whether the proposed acquisition meets certain requirements necessary to move on to the next phase and eventually to full production. The directive outlines the extent and scope of required program, project, and service management; level of reporting requirement; and the acquisition decision authority based on whether the acquisition is classified as level 1, 2, or 3. The acquisition decision authority for major acquisitions--level 1 and level 2--is to be at the department or component level and the acquisition decision authority for nonmajor acquisitions--level 3--is to be at the component level.[Footnote 20] An acquisition may be raised to a higher level acquisition level by the ARB.[Footnote 21] The ARB supports the acquisition decision authority in determining the appropriate direction for an acquisition at key Acquisition Decision Events. Following an ARB meeting, the Acquisition Program Management Division is to prepare an acquisition decision memorandum as the official record of the meeting to be signed by the acquisition decision authority. This memo is to describe the approval or other decisions made at the ARB and any action items to be satisfied as conditions of the decision. The ARB reviews are to provide an opportunity to determine a program's readiness to proceed to the following life-cycle phase. However, we reported in March 2011 that the ARB had not reviewed most of DHS's major acquisition programs by the end of fiscal year 2009 and programs that were reviewed had not consistently implemented action items identified as part of the review by established deadlines.[Footnote 22] Our prior work has shown that when these types of reviews are skipped or not fully implemented, programs move forward with little, if any, early department-level assessment of the programs' costs and feasibility, which contributes to poor cost, schedule, and performance outcomes.[Footnote 23] As a part of its responsibilities, the Acquisition Program Management Division has identified major DHS acquisition programs, projects, or services for oversight through the ARB process. According to Acquisition Program Management Division officials, beginning in fiscal year 2009, the list was to be updated on a yearly basis through interviews with and documentation from component program offices. In May 2010, the Undersecretary for Management identified 86 programs on DHS's major oversight list for fiscal year 2010, 62 of which TES and component officials determined required T&E oversight--that is programs that were in an acquisition phase where T&E was being planned or conducted.[Footnote 24] Several of the 62 programs consisted of multiple subprojects, such as TSA's Passenger Screening Program. For more information on these 86 major acquisition programs, see appendix II. DHS's 2010 acquisition directive also includes guidance for preparing documentation to support component and departmental decision making and specifies requirements for developmental and operational T&E as a part of the acquisition review process.[Footnote 25] Developmental T&E may include a variety of tests, such as system qualification testing, system acceptance testing, and software testing. Developmental testing may be carried out by the user and may be conducted in simulated environments, such as laboratories, test facilities, or engineering centers that might or might not be representative of the complex operational environment. Operational T&E is a field test, performed under realistic conditions by actual users in order to determine the operational effectiveness and suitability of a system, and the corresponding evaluation of the data resulting from the test. TES's Role in Overseeing Component Testing and Evaluation: To carry out its responsibilities for overseeing T&E, S&T established TES in 2006 and created the position of Director of TES in June 2007. [Footnote 26] TES's mission is to establish and manage DHS T&E policies and procedures and to oversee and coordinate T&E resources to verify attainment of technical performance specifications and operational effectiveness and suitability. To carry out its T&E oversight, in fiscal year 2010, TES had a budget of about $23 million and as of February 2011 had a staff of 26, which includes the TES Director, 19 staff dedicated to T&E activities, and 6 dedicated to developing standards. In May 2009, DHS issued a delegation which specified the responsibilities and duties of the Director of Operational Test & Evaluation.[Footnote 27] The TES Director and Director of Operational Test and Evaluation, while distinct positions in the T&E directive, share some advisory, review, and oversight responsibilities. For example, both are responsible for advising program managers in developing T&E documentation and approving test and evaluation master plans. The TES Director is responsible for developing DHS T&E policy and the Director of Operational Test and Evaluation is to approve operational test plans and report to the ARB after assessing operational test reports. Since May 2009, the Director of Operational Test and Evaluation position has not been continuously filled according to the current TES Director.[Footnote 28] In a November 2010 memo, the Under Secretary for Science and Technology designated one person as both the director of TES and the Director of Operational Test and Evaluation until further notice. The T&E directive outlines the responsibilities of the TES Director and the Director of Operational Test and Evaluation. According to the directive, the TES Director is to establish the department's testing and evaluation policies and processes and the Director of Operational Test and Evaluation is to administer those policies and processes. The directive also outlines TES's responsibilities in overseeing T&E across DHS components and its role in the acquisition review process. Table 1 describes TES's T&E responsibilities as outlined in the T&E directive for all level 1, level 2, and special oversight acquisition programs.[Footnote 29] Table 1: Summary of TES's Key Responsibilities Outlined in the T&E Directive: TES responsibility: Review the mission need statement (Need phase); Description: The mission need statement is to identify the functional capabilities required to fill an identified gap. TES responsibility: Review the concept of operations (Analyze/Select phase); Description: The concept of operations document is to describe how the preferred solution from the Analysis of Alternatives will be used operationally, and thus confirming the solution's effectiveness and suitability in a "real-world" operational environment. TES responsibility: Review the integrated logistic support plan (Analyze/Select); Description: The integrated logistic support plan is to define the support and sustainability capabilities necessary to assure the defined system/solution is operationally supportable. TES responsibility: Approve the operational test agent (Analyze/Select phase); Description: An operational test agent is a government agency or independent contractor that is to plan, conduct, and report independent operational T&E on selected DHS programs. The operational test agent is identified and approved as early as possible in the acquisition process. TES responsibility: Review the operational requirements document (Analyze/Select phase); Description: The operational requirements document is to provide performance parameters that need to be met by the program to insure a useful capability is delivered to the user to close the capability gap identified in the mission need statement. The operational requirements document is to provide key performance parameters, which are the most important requirements the system must meet to fulfill its fundamental purpose, that are quantifiable, measurable, and testable. TES responsibility: Approve the test and evaluation master plan (TEMP) (Obtain phase); Description: The TEMP is to be the basic "top-level" planning document for T&E related activities for major acquisition programs. The TEMP is to describe the necessary developmental T&E and operational T&E that need to be conducted to determine system technical performance, operational effectiveness/suitability, and limitations. The TEMP is to identify all critical issues and describe the objectives, responsibilities, resources, and schedules for all completed and planned T&E, including modeling and simulation tools used in the T&E process. The TEMP is to be updated if significant changes in the T&E strategy, schedule, or resource requirements occur and the approval for the updated TEMP is to be the same as the original. TES responsibility: Review the developmental test report (Obtain phase); Description: The development test report is to report the results of testing that is concerned chiefly with validating the contract requirements and the attainment of engineering design goals and manufacturing processes. Developmental testing is to be carried out by the user and may be conducted in simulated environments, such as laboratories, test facilities, or engineering centers that might or might not be representative of the complex operational environment. TES responsibility: Approve the operational test plan (Obtain phase); Description: The operational test plan documents are to be specific test cases, sites, and design for operational testing. TES responsibility: Participate in operational test readiness reviews (Obtain phase); Description: Operational test readiness reviews are to be process assessments to ensure that the system can proceed into Initial operational test and evaluation with a high probability of success. More than one operational test readiness review may be conducted prior to initial operational test and evaluation. TES responsibility: Observe operational test(s) (Obtain phase); Description: Operational testing is to be the field test, under realistic conditions, of any system or component, for determining that system or component's overall effectiveness and suitability for use before deployment of the system or component. Operational testing is to provide information for the overall assessment of how well a system will satisfy the mission need when operated by typical users in the expected environment. TES responsibility: Review the operational test and evaluation reports (Obtain phase); Description: An operational test and evaluation report is to be produced by the operational test agent after the completion of operational testing. TES responsibility: Write a letter of assessment of the operational test and evaluation report (Obtain phase); Description: The letter of assessment is to be an assessment of the adequacy of the operational test, a concurrence or nonconcurrence on the operational test agent's evaluation of operational suitability and operational effectiveness, and any further independent analysis identified by TES. The letter of assessment is to be written within 30 days of receiving the operational test and evaluation report and is to be provided to the Chair of the ARB. Source: GAO analysis of DHS Directive Number 026-06 Test and Evaluation and DHS Acquisition Instruction/Guidebook 102-01, Interim Version 1.9 (Nov. 7, 2008). Note: Review is defined as TES examining documents as required by the T&E directive. Analysis of Alternatives is defined as identifying alternative solutions and analyzing/comparing the alternatives based on cost, risk, and capability. [End of table] The T&E directive requires TES to review and approve required component acquisition documentation before an ARB meets for an acquisition decision event. These documents are meant to be reviewed and, if required, approved in a sequential order associated with the acquisition phase, because these documents build upon one another. Figure 1 presents TES's responsibilities throughout the four DHS acquisition phases as defined in the acquisition directive. Figure 1: TES Key Responsibilities as Part of the DHS Acquisition Process: [Refer to PDF for image: illustration] Need: Define the problem: * Review mission need statement. ADE 1. Analyze/Select: Identify the alternatives and resource requirements: * Review concept of operations; * Review integrated logistics support plan; * Review operational requirements document; * Approve operational test agent. ADE 2A. Obtain: * Approve test and evaluation master plan. ADE 2B. Develop and evaluate capabilities: * Approve operational test plan; * Observe operational test; * Review operational test & evaluation report; * Write letter of assessment. ADE 3. Produce/Deploy/Support: Sources: GAO analysis of DHS Directive Number 026-06 Test and Evaluation; and Art Explosion (clipart). Legend: ADE - Acquisition Decision Event. Note: Acquisition Decision Events occur when the ARB meets to determine whether a program has all of the necessary acquisition documents (in addition to other DHS requirements not illustrated in this figure) to move to the next phase in the acquisition process. The ARB can identify a program as having received an ADE 1, 2A, 2B, and 3. The T&E directive states that the operational test agent is to be identified and approved as early as possible in the acquisition process. Little to no T&E occurs in the Produce/Deploy/Support phase. [End of figure] To carry out these responsibilities for the 62 acquisition programs under its oversight in fiscal year 2010, TES has test area managers who assist component officials in fulfilling their T&E responsibilities and provide guidance and clarification in regard to the requirements in the T&E directive. According to TES, each major acquisition program is assigned a test area manager and as of February 2011, TES employed nine test area managers. TES Met Some Oversight Requirements for T&E of Acquisition Programs Reviewed; Additional Steps Needed to Ensure That All Requirements Are Met: TES met its oversight requirements when approving test plans and test reports in accordance with DHS acquisition and T&E directives for the 11 major acquisition programs we selected for review. However, TES did not consistently document its review and approval of operational test agents or its review of other required acquisition documentation, which could provide more assurance that components were meeting T&E directives when TES reviewed these documents. Further, TES does not plan an independent assessment of TSA's Advanced Spectroscopic Portal's operational test results, as required by the T&E directive. TES Oversight of Components' Test Plans and Test Reports: TES is to oversee T&E of major DHS acquisition programs by ensuring that the requirements set forth in the T&E directive are met and by working with component program officials to develop T&E documentation, such as test and evaluation master plans, as required by DHS's acquisition directive. TES's T&E oversight responsibilities set forth in the T&E and acquisition directives pertain to programs primarily in the analyze/select and obtain phases of the acquisition process because most testing and evaluation efforts occur in these phases. As a result, the requirements of the T&E directive and TES's oversight vary depending on when a program progresses through certain phases of the acquisition process.[Footnote 30] For example, when a program is in the produce/deploy/support phase there is usually little to no T&E activity, so TES's involvement is limited. We reviewed TES's T&E oversight efforts for 11 DHS programs and found that TES had conducted oversight of components' test plans and test reports, as set forth in the acquisition and T&E directives, as it asserted. The 11 programs, each managed by different DHS components, were in one phase of the acquisition process or had two or more subprojects simultaneously in different phases of the acquisition process. For example, Coast Guard's H-65 helicopter program has 6 discrete subprojects, each with its own completion schedule, including 4 subprojects in the Produce/Deploy/Support phase and 2 subprojects in the Obtain phase. Acquisition Program Management Division, TES, and component officials determine if subprojects need to develop separate sets of acquisition documents as they progress through the acquisition process. Figure 2 provides an overview of these programs and their associated acquisition phases. Additional details on these programs can be found in appendix I. Figure 2: Selected Major Acquisition Programs and Their Components and Acquisition Phases as of April 2011: [Refer to PDF for image: illustration] Phases: 1) Need: ADE 1. 2) Analyze/Select: ADE 2A. 3) Obtain: ADE 2B. ADE 3. 4) Produce/Deploy/Support: Programs and progress through phases: Advanced Spectroscopic Portal - Domestic Nuclear Detection Office: Need, Analyze/Select; Obtain. Atlas Tactical Communications,[A][B] - U.S. Immigration and Custom Enforcement: Need, Analyze/Select; Obtain, Produce/Deploy/Support. BioWatch Generation-3 - Office of Health Affairs: Need, Analyze/Select. H-65 Conversion Sustainment Project[B] - U.S. Coast Guard: Need, Analyze/Select; Obtain. Information Integration and Transformation[B] - U.S. Secret Service: Need, Analyze/Select; Obtain (to ADE 2B). National Cybersecurity and Protection System[B] - National Protection and Programs Directorate: Need, Analyze/Select; Obtain. National Security System Program[B] - Office of Intelligence & Analysis: Need, Analyze/Select. Passenger Screening Program, Advanced Technology-2 - Transportation Security Administration: Need, Analyze/Select; Obtain, Produce/Deploy/Support. Secure Border Initiative Network Block 1 - U.S. Customs and Border Protection: Need, Analyze/Select; Obtain. Transformation and Systems Consolidation - DHS Office of the Chief Financial Office: Need, Analyze/Select. Transformation - U.S. Citizenship and Immigration Services: Need, Analyze/Select. Sources: GAO analysis of interviews and data collected from DHS components. Legend: ADE - Acquisition Decision Event. [A] Atlas Tactical Communications program made procurements for subprojects in the amount of about $27 million which the ARB approved without requiring acquisition documentation, such as test plans. According to Acquisition Program Management Division officials, the ARB made the decision due to the low dollar threshold of the project and the need to use American Recovery and Reinvestment Act funds within a certain time frame. See Pub. L. No. 111-5, 123 Stat. 115 (2009). [B] Programs which have two or more subprojects simultaneously in different acquisition phases. The acquisition phase illustrated represents those subprojects which were involved in T&E activities . [C] National Cybersecurity Protection System program Block 2.1 is in the Obtain phase and block 3.0 is in the Analyze/Select phase. Note: Acquisition phases for programs represented were as of April 2011. For a description of the programs and subprojects, see appendix I. [End of figure] As shown in figure 3, for the 11 selected DHS programs, TES reviewed and approved test and evaluation master plans for 6 of the 7 programs that were required to develop such plans by the T&E and acquisition directives and had documented their approval of these plans. For the one program that was in the phase that required such a plan--ATLAS Tactical Communications--the program had not yet drafted its test and evaluation master plan. The remaining 4 programs had plans in draft form that had not yet been submitted to TES for review. As a result, TES was not yet required to review these plans. Figure 3: TES's Involvement in Reviewing and Approving Test and Evaluation Master Plans for 11 Selected Acquisition Programs as of April 2011: [Refer to PDF for image: illustrated table] Program: Advanced Spectroscopic Portal; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: Atlas Tactical Communications[A]; Has a test and evaluation master plan been developed? N/A; TES involvement in the development of the test and evaluation master plan: N/A; TES approval of the test and evaluation master plan is documented: N/A. Program: BioWatch Gen-3; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: H-65 Conversion/Sustainment Project; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: Information Integration and Transformation[B]; Has a test and evaluation master plan been developed? In-draft; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: No. Program: National Cybersecurity and Protection System; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: National Security Systems Program; Has a test and evaluation master plan been developed? In-draft; TES involvement in the development of the test and evaluation master plan: N/A; TES approval of the test and evaluation master plan is documented: N/A. Program: Passenger Screening Program, Advanced Technology - 2; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: SBInet Block 1; Has a test and evaluation master plan been developed? Yes; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: Yes. Program: Transformation and Systems Consolidation; Has a test and evaluation master plan been developed? In-draft; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: N/A. Program: Transformation; Has a test and evaluation master plan been developed? In-draft; TES involvement in the development of the test and evaluation master plan: Yes; TES approval of the test and evaluation master plan is documented: N/A. Sources: GAO analysis of interviews and data collected from DHS components and TES officials. N/A: Not applicable because program has not reached a state where document is required. [A] Atlas Tactical Communications program made procurements for subprojects which the ARB approved in March 2010 without requiring acquisition or T&E documentation, such as test plans. According to Acquisition Program Management Division officials, the ARB made the decision due to the low dollar threshold of the project and the need to use $20 million in American Recovery and Reinvestment Act funds within a certain time frame. See Pub. L. No. 111-5, 123 Stat. 115 (2009). [B] In February 2011, the ARB granted an acquisition decision event 2A and acquisition decision event 2B decision for one segment of the Information Integration and Transformation program without an approved test and evaluation master plan and for the remaining three segments, the ARB granted an acquisition decision event 2A decision. The ARB, among other things, directed program officials to work with TES to complete the test and evaluation master plan by May 2011. Note: The status of the test and evaluation master plan for the overall program is represented as of April 2011. Programs may have two or more subprojects simultaneously in different acquisition phases and these subprojects may have separate subsets of T&E documentation, such as an addendum to the test and evaluation master plan. Also, test and evaluation master plans are to be reviewed and updated throughout the obtain phase if significant changes in T&E strategy, schedule, or resource requirements occur. See appendix I for detailed information. [End of figure] Component officials from each of these six programs stated that TES provided input to the development of the test and evaluation master plans. For example, Office of Health Affairs officials stated that TES officials suggested that the BioWatch Gen-3 program office incorporate an additional test event to ensure that the program was tested under specific environmental conditions described in the program's operational requirements document, which resulted in more tests. In addition, U.S. Customs and Border Protection (CBP) officials stated that TES participated in a line-by-line review of the SBInet test plan and provided detailed suggestions. Further, TES suggested that the criteria used for operational testing in the test and evaluation master plan needed to be expanded, and that an update may be required for SBInet to progress to the next acquisition phase. All of the component program officials who had undergone TES review or approval told us that TES test area managers provided their input in a variety of ways, including participating in T&E working groups, in specific meetings to discuss T&E issues, or by providing written comments culminating in TES's approval of the plan.[Footnote 31] After the test and evaluation master plan is developed, the test agent is to develop operational test plans, which detail field testing of the system under realistic conditions for determining that the system's overall effectiveness and suitability for use before deployment of the system. As shown in figure 4, of the 11 selected acquisition programs, TES reviewed and approved operational test plans for the 4 programs that were required to develop such plans by the acquisition directive and documented their approval of these plans. Figure 4: TES's Involvement in Reviewing and Approving Operational Test Plans for 11 Selected Acquisition Programs as of April 2011: [Refer to PDF for image: illustrated table] Program: Advanced Spectroscopic Portal; Has an operational test plan been developed? Yes; TES involvement in the development of the operational test plan: Yes; TES approval of the operational testlan documented: Yes. Program: Atlas Tactical Communications[A]; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Program: BioWatch Gen-3; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Program: H-65 Conversion/Sustainment Project[B]; Has an operational test plan been developed? No; TES involvement in the development of the operational test plan: No; TES approval of the operational testlan documented: No. Program: Information Integration and Transformation; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Program: National Cybersecurity and Protection System; Has an operational test plan been developed? Yes; TES involvement in the development of the operational test plan: Yes; TES approval of the operational testlan documented: Yes. Program: National Security Systems Program; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Program: Passenger Screening Program, Advanced Technology - 2; Has an operational test plan been developed? Yes; TES involvement in the development of the operational test plan: Yes; TES approval of the operational testlan documented: Yes. Program: SBInet Block 1; Has an operational test plan been developed? Yes; TES involvement in the development of the operational test plan: Yes; TES approval of the operational testlan documented: Yes. Program: Transformation and Systems Consolidation; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Program: Transformation; Has an operational test plan been developed? N/A; TES involvement in the development of the operational test plan: N/A; TES approval of the operational testlan documented: N/A. Sources: GAO analysis of interviews and data collected from DHS components and TES officials. N/A: Not applicable because program has not reached a state where document is required. [A] Atlas Tactical Communications program made procurements for subprojects which the ARB approved in March 2010 without requiring acquisition or T&E documentation, such as test plans. According to Acquisition Program Management Division officials, the ARB made the decision due to the low dollar threshold of the project and the need to use $20 million in American Recovery and Reinvestment Act funds within a certain time frame. See Pub. L. No. 111-5, 123 Stat. 115 (2009). [B] According to Coast Guard and TES officials, the H-65 program has two segments which are in the obtain phase, of which one segment will not have a test plan (segment 5) due to questions related to system effectiveness and for the other segment, the test plan is to be developed (segment 6). Note: Operational test plans for the overall program are represented as of April 2011. Programs may have two or more subprojects simultaneously in different acquisition phases and these subprojects may have separate subsets of T&E documentation, including operational test plans. See appendix I for detailed information. [End of figure] Component officials from these 4 programs said that TES provided input into their test plans. For example, National Protection and Programs Directorate officials from the National Cybersecurity Protection System program stated that TES had significant comments on their operational test plan, such as including confidence levels associated with the system's key performance requirements and helping program officials select a sample size necessary to measure statistically significant results. In addition, TES officials requested that the plan include different testing scenarios in order to demonstrate a varied use of the system.[Footnote 32] In addition, officials from the Transportation Security Administration's (TSA) Advanced Technology-2 program indicated that TES provided significant input to their plan through a working group. The remaining 7 of the 11 programs had not yet begun to develop their operational test plan. At the conclusion of operational testing, the test agent is to write a report on the results of the test. The T&E directive specifies that TES is to receive the operational test report, which is to address all the critical issues and provide an evaluation of the operational suitability and operational effectiveness of the system. After reviewing the operational test report, TES then is to write a letter of assessment--which is an independent assessment of the adequacy of the operational test and provides TES's concurrence or nonconcurrence on the test agent evaluation of operational suitability and operational effectiveness. TES is to provide the letter of assessment to the ARB as it is determining whether a program should progress to the production and deployment phase. Of the 11 programs we selected to review, TES developed a letter of assessment for the 1 program--TSA's Advanced Technology 2 --that had completed operational testing and had a written operational T&E report on the results. The assessment concluded that while the T&E activities were adequate to inform the ARB as to system performance, TES did not concur with TSA's test agent's assessment as to system effectiveness because the system did not achieve a key performance parameter during testing. The ARB considered the letter of assessment and TES's input and granted TSA permission to procure and deploy a limited number of screening machines.[Footnote 33] TSA will have to go before the ARB again to determine if full-scale production can proceed after TSA has provided the ARB with a business case and risk mitigation plan related to testing issues. The remaining 10 selected programs had not completed operational testing and thus, were not ready for letters of assessment.[Footnote 34] In addition to letters of assessment, TES officials told us that they regularly discuss T&E issues and concerns either verbally or through e- mails with Acquisition Program Management Division officials, who are responsible for organizing ARB meetings. For example, Acquisition Program Management Division officials stated that they rely on TES to provide candid information about the suitability of various programs' T&E and whether these issues impact their program's readiness to go before the ARB. Further, the officials told us that TES's input at the ARBs, if any, is to be documented in acquisition decision memorandums. Acquisition Program Management Division officials also noted that TES's input may be used in making the decision about when to hold an ARB for a particular program. T&E input from TES is one of many factors the ARB uses in overseeing acquisitions. For example, according to S&T officials, the ARB considers the current threat assessments and the extent to which the program, if implemented sooner, would help to address that threat. The ARB also considers factors such as the cost of the program and potential costs of conducting more testing and whether the results of operational testing were sufficient to achieve the intended benefits of the program. As a result, the ARB may accept a higher level of risk and allow a program to proceed even if testing concerns have been raised, if it determines that other reasons for quicker implementation outweigh these concerns. TES officials also stated that they work extensively with components prior to ARB meetings to ensure that T&E issues are addressed, with the goal to address these issues before going before the ARB. TES meets with component officials during regular acquisition review team meetings to resolve various issues before ARB meetings are convened. For example, due to concerns about the results of system qualification tests, TES recommended to SBInet program and ARB officials that the program should not proceed to the next milestone--site preparation, tower construction, and sensor and communication equipment installation at the Ajo, Arizona test site--until after operational testing was completed at the Tucson, Arizona test site. In May 2009, the ARB authorized SBInet to proceed with plans for the Ajo, Arizona site despite TES's advice to the contrary, and directed TES to work with component officials to revise test plans, among other things. [Footnote 35] Additional Steps Needed to Ensure that T&E Requirements Are Met: While TES's oversight of the test plans and reports for major acquisition programs selected for review is in accordance with provisions in the T&E directive, it did not consistently document its review and approval of certain acquisition documentation or document the extent to which certain requirements in the T&E directive were met. TES Did Not Consistently Document the Extent to which Criteria Used in Its Approval of Operational Test Agents Were Met: The T&E directive requires that an operational test agent--a government agency or independent contractor carrying out independent operational testing for major acquisition programs--is to meet certain requirements to be qualified and approved by TES, but does not specify how TES's approval is to be documented. According to the T&E directive, the test agent may be within the same component, another government agency, or a contractor, but is to be independent of the developer and the development contractor. Because the responsibilities of a test agent are significant throughout the T&E process, this independence is to allow the agent to present objective and unbiased conclusions regarding the system's operational effectiveness and suitability to DHS decision makers, such as the ARB. For example, some the test agent's responsibilities in the T&E directive include: * Being involved early in the acquisition cycle by reviewing draft requirements documents to help ensure that requirements are testable and measurable. * Assisting the component program manager in the preparation of the test and evaluation master plan. * Planning, coordinating, and conducting operational tests, and preparing the operational T&E report. * Reporting operational test results to the program manager and TES. According to TES officials, the test agent is also to meet other requirements in order to be approved by TES, such as having the expertise or knowledge about the product being tested and having the capacity and resources to execute the operational tests. To ensure that criteria for test agents are met, the T&E directive requires TES to approve all agents for major acquisition programs. As shown in figure 5, of the 11 programs we reviewed, 8 programs had selected a test agent and the others were in the process of selecting a test agent. TES provided documentation, such as memoranda, of its approval for 3 of these 8 programs. For the remaining 5 programs, there was no documentation of the extent to which these test agents had met the criteria and that TES had approved them. According to TES officials, they did not have a mechanism in place requiring a consistent method for documenting their review and approval of component agents or the extent to which criteria used in reviewing these agents were met. Figure 5: TES's Involvement in Reviewing and Approving Operational Test Agents for 11 Selected Acquisition Programs as of April 2011: [Refer to PDF for image: illustrated table] Program: Advanced Spectroscopic Portal[A]; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: Yes; Operational test agent approval documented by TES in a memo: No. Program: Atlas Tactical Communications[B]; Has an operational test agent been selected? Operational test agent signed the test and evaluation master plan: N/A; Operational test agent approval documented by TES in a memo: N/A. Program: BioWatch Gen-3; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: Yes; Operational test agent approval documented by TES in a memo: No. Program: H-65 Conversion/Sustainment Project; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: Yes; Operational test agent approval documented by TES in a memo: No. Program: Information Integration and Transformation; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: N/A; Operational test agent approval documented by TES in a memo: Yes. Program: National Cybersecurity and Protection System; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: No; Operational test agent approval documented by TES in a memo: No. Program: National Security Systems Program; Has an operational test agent been selected? No; Operational test agent signed the test and evaluation master plan: N/A; Operational test agent approval documented by TES in a memo: N/A. Program: Passenger Screening Program, Advanced Technology - 2; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: Yes; Operational test agent approval documented by TES in a memo: Yes. Program: SBInet Block 1; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: Yes; Operational test agent approval documented by TES in a memo: No. Program: Transformation and Systems Consolidation; Has an operational test agent been selected? No; Operational test agent signed the test and evaluation master plan: N/A; Operational test agent approval documented by TES in a memo: N/A. Program: Transformation; Has an operational test agent been selected? Yes; Operational test agent signed the test and evaluation master plan: N/A; Operational test agent approval documented by TES in a memo: Yes. Sources: GAO analysis of interviews and data collected from DHS components and TES officials. N/A: Not applicable because test agent has not been selected or a test and evaluation master plan has not been developed. [A] In a May 2008 memorandum of understanding among the Science and Technology Directorate, U.S. Customs and Border Protection, DHS Management Directorate, and Domestic Nuclear Detection Office, TES was designated to carry out the responsibilities of the test agent for the Advanced Spectroscopic Portal program. [B] Atlas Tactical Communications program made procurements for subprojects in March 2010 which the ARB approved without requiring acquisition or T&E documentation, such test plans. According to Acquisition Program Management Division officials, the ARB made the decision due to the low dollar threshold of the project and the need to use $20 million in American Recovery and Reinvestment Act funds within a certain time frame. See Pub. L. No. 111-5, 123 Stat. 115 (2009). Note: Operational test agents for the overall program are represented as of April 2011. Programs may have two or more subprojects simultaneously in different acquisition phases and these subprojects may use the same or different test agents. See appendix I for detailed information. [End of figure] In the absence of such a mechanism in fiscal year 2010, TES's approval of test agents was not consistently documented. TES and component officials stated that the approval for the five programs was implicit or provided verbally without documentation regarding whether the test agent met the T&E directive requirements. The T&E directive states that the test agent is to be identified and approved as early as possible in the acquisition process to, among other things, assist the component program officials in developing the test and evaluation master plan and review draft requirements documents to provide feedback regarding the testability of proposed requirements. TES and component officials stated that they assumed that test agents were approved using various approaches. Specifically, of the five programs that had test agents sign the test and evaluation master plan, one program had documented approval from TES. For example, Coast Guard and Office of Health Affairs officials stated that they did not have explicit documentation of TES's approval of their agents; however, they believed that TES's approval was implicit when TES approved their test and evaluation master plan since the test agent and TES are both signatories on the plan. CBP and National Protection and Programs Directorate officials told us that TES provided verbal approval for their test agents. Since there is no mechanism requiring TES to document its approval of the agent, and approval was granted verbally, there is no institutional record for DHS or an independent third party to validate whether TES followed its criteria when approving these test agents and whether the test agent was identified and approved before the test and evaluation master plan and requirements documents were finalized, as outlined in the T&E directive. With regard to the three programs in which TES had documented its approval in memoranda, these memoranda detailed TES's agreement or nonagreement with a particular agent and highlighted whether the agent met the criteria outlined in the T&E directive. For example, TES provided interim approval to all three of the programs with the conditions that the programs prove at a later date that the test agents met all the requirements. For example: * In April 2010, TES wrote a memo and granted interim approval with "serious reservations" for 1 year to TSA's test agent for the Passenger Screening program.[Footnote 36] In the memo, TES cited concerns about the organizational structure and the lack of independence of the test agent since the test agent was part of the same TSA office responsible for managing the program. The memo outlined several steps that TSA should take, including the implementation of interim measures, such as new procedures, to ensure the necessary independence critical to testing and evaluation efforts as required by DHS directives. TES officials told us that by documenting TES's interim approval in a memo, they were able to communicate their concerns about the test agent's independence to TSA and DHS decision makers and set forth interim measures that TSA needed to address regarding their concerns. * In July 2010, TES granted conditional approval to the test agent for the U.S. Citizenship and Immigration Services' (USCIS) Transformation program's test agent. TES made its approval contingent on the program developing a plan to ensure that the test agent was familiar with the component's business practices. According to TES officials, after component officials gave a briefing to TES, they determined that the test agent met the requirements and it was approved. * In January 2011, TES granted conditional approval for the U.S. Secret Service's Information Integration and Transformation program to bring its selected test agent on board. TES's final approval will be given after program officials brief TES on the test agent's operational testing approach, which is to demonstrate that the test agent has knowledge of the product and has the capacity to execute the tests. TES officials told us that they do not have approval memos for all of the test agents that have been hired by program offices since the T&E directive was implemented in May 2009. Because TES did not consistently document their approvals of test agents, it is unclear whether TES has ever disapproved a test agent. TES officials acknowledged that they did not consistently document that the test agents met T&E requirements and did not document their approval of test agents. TES officials said that it would be beneficial to do so to ensure that agents met the criteria required in the T&E directive. In addition, Standards for Internal Control in the Federal Government and associated guidance state that agencies should document key decisions in a way that is complete and accurate, and that allows decisions to be traced from initiation, through processing, to after completion.[Footnote 37] These standards further state that documentation of key decisions should be readily available for review. Without a mechanism for documenting its review and approval of test agents for major acquisition programs, it will be difficult for DHS or an independent third party to validate TES's decision-making process to ensure that it is effectively overseeing component testing. Moreover, it will be difficult for TES to provide reasonable assurance that these agents met the criteria outlined in the T&E directive, such as the requirement that they be independent of the program being tested. TES Did Not Consistently Document the Extent to Which Certain Acquisition Documents Met T&E Criteria: In addition to reviewing and approving test plans, under the T&E directive, TES is required to review certain component acquisition documents, including the mission need statements, operational requirements document, concept of operations, and developmental test reports, amongst others. These documents, which are required at the need, Analyze/Select, and Obtain phases of the acquisition process, are to be reviewed by TES to assist component program managers in identifying and resolving technical, logistical, and operational issues early in the acquisition process and to ensure that these documents meet relevant criteria. Specifically, as outlined in the T&E directive, TES is to review the mission need statement to establish awareness of the program and help ensure that the required standards are developed and that the component has identified the appropriate resources and support needed to conduct testing. TES is also to review the operational requirements document, including the key performance parameters and critical operational issues that specify the operational effectiveness and operational suitability issues that the test agent is to examine in order to assess the system's capability to perform the mission. Further, TES is to review the concept of operations, since this document describes how the technology or equipment will be used in an operating environment. TES is to review the developmental test reports to maintain knowledge of contractor testing and to assist in its determination of the program's readiness to progress to operational testing. We have previously reported that inadequate attention to developing requirements results in requirements instability, which can ultimately cause cost escalation, schedule delays and fewer end items. [Footnote 38] Further, we reported that without the required development and review of key acquisition data, DHS cannot provide reasonable assurance that programs have mitigated risks to better ensure program outcomes. TES officials stated that they do not have a mechanism to document or track those that they did review, what criteria they used when reviewing these documents, and the extent to which the documents reviewed met those criteria. For the 11 DHS programs that we reviewed, 8 programs had component-approved mission need statements; 2 programs, Atlas Tactical Communications and Transformation, had not yet completed such statements; and 1 program, the initial SBInet program, had completed a mission need statement in October 2006 before the T&E directive was issued and did not develop a separate mission need statement for the Block 1 increment of the program. Of the 8 programs that had mission need statements, 6 components told us that they did not have evidence that TES reviewed the mission need statement in accordance with the T&E directive. Further, TES could not demonstrate that it had received or reviewed these documents. Since TES did not have documentation of its review, it is difficult to determine the extent to which the documents were reviewed and the extent to which these documents met the review criteria. TES officials told us that they do not usually provide substantial input into the mission need statements and that they receive these documents to establish awareness of a new program. Further, while one TES test area manager told us that he reviews all developmental test reports, another test area manger told us that some programs do not routinely send him developmental test reports. Also, for example, Secret Service officials said that for the Information Integration and Transformation program they provided the operational requirements document, concept of operations, and integrated logistic support plan to TES. Specifically, the officials said that TES officials were very helpful in providing input on draft documents and made improvements to the documents by suggesting, for example, that the tests be more realistic by including personnel from field offices, headquarters, and external agencies in the live/ production test environment. In contrast, officials from TSA stated that while they provided their mission need statement, concept of operations, integrated logistics support plan, and acquisition program baseline documents for the Advanced Technology 2 (AT-2) program to TES, TES officials did not provide input or comments on any of those documents. TES officials told us that the AT-2 program was initiated and developed some acquisition documentation prior to May 2009 when the T&E directive was issued. Specifically the operational requirements document was approved and finalized by TSA in June 2008 prior to the T&E directive and provided later to TES in February 2010 when the program was being reviewed. When TES reviewed the operational requirements document along with other documents such as the test and evaluation master plan, TES wrote a memo to TSA in March 2010 requesting that detection performance requirements be clarified and that users concur with the requirements. After several months of discussion, TSA and TES agreed on an approach which was used as the basis for initial operational T&E. Standards for Internal Controls in the Federal Government, as outlined earlier, state that agencies should document key decisions, and further that documentation of key decisions should be readily available for review.[Footnote 39] TES officials stated that they do not have a mechanism requiring that they document their review of certain acquisition documentation or the extent to which the document met the criteria used in reviewing these documents, and recognized that doing so would be beneficial. Developing a mechanism for TES to document its review of key acquisition documents could better position TES to provide reasonable assurance that it is reviewing key documentation and providing input that is important for determining the outcome of future testing and evaluation efforts, as required by the T&E directive. Moreover, such a policy could help to ensure that an institutional record exists for DHS or an independent third party to use in determining whether TES is effectively overseeing component T&E efforts and assisting in managing DHS major acquisition programs. TES Does Not Plan an Independent Assessment of ASP's Operational Test Results as Required by the T&E Directive: According to the T&E directive, TES is to conduct an independent assessment of the adequacy of an operational test, provide a concurrence or nonconcurrence on the test agent's evaluation of operational suitability and operational effectiveness, and provide any further independent analysis it deems necessary for all major DHS acquisition programs. TES is to document this independent assessment by writing a letter of assessment within 30 days of receiving the operational test report from the components' test agent and provide the letter of assessment to the ARB, who then uses the assessment in making its determination of whether the program can proceed to purchase and implementation. While TES has developed a letter of assessment for the two other programs undergoing an ARB decision to enter into the production and deployment phase since the T&E directive was issued in May 2009, TES officials told us that they do not plan to write such an assessment for the Advanced Spectroscopic Portal (ASP) program[Footnote 40] because they are the test agent for ASP and thus, are not in a position to independently assess the results of testing that they conducted. In April 2008, over a year before the T&E directive was issued, senior level executives from DHS, S&T, CBP, and the Domestic Nuclear Detection Office (DNDO) signed a memorandum of understanding regarding arrangements for ASP operational testing. The memo designated Pacific Northwest National Lab, a U.S. Department of Energy laboratory, as the test agent. However, the memo also outlined the roles and responsibilities of TES, many of which reflected the duties of a test agent, such as developing and approving all operational test plans, responsibility for the management of testing and field validation, and developing and approving operational test reports. TES officials told us that they were using Pacific Northwest National Lab staff to carry out the operational tests, but are acting, for all intents and purposes, as the test agent for ASP. TES and DNDO officials told us that this arrangement was made after repeated testing issues arose with the ASP program. In September 2008, we reported that ASP Phase 3 testing by DNDO provided little information about the actual performance capabilities of ASP and that the resulting test report should not be used in determining whether ASP was a significant improvement over currently deployed equipment.[Footnote 41] Specifically, we found that the ASP Phase 3 test results did not help determine an ASP's "true" level of performance because DNDO did not design the tests to assess ASP performance with a high degree of statistical confidence. In response to our report, DHS convened an independent review team to assist the Secretary in determining whether he should certify that there will be a significant increase in operational effectiveness with the procurement of the ASP system.[Footnote 42] The independent review team found that the test results and measures of effectiveness were not properly linked to operational outcomes. In May 2009, we reported that DHS had increased the rigor of ASP testing in comparison with previous tests.[Footnote 43] For example, DNDO mitigated the potential for bias in performance testing (a concern we raised about prior testing) by stipulating that there would be no ASP contractor involvement in test execution. However, the testing still had limitations, such as a limited set of scenarios used in performance testing to conceal test objects from detection. Moreover, we also reported that TES was to have the lead role in the final phase of ASP testing. As of February 2011, TES officials told us that the final phase of testing, consisting of 21 days of continuous operation, had not yet been scheduled. With TES acting as the test agent, it is not in a position to exercise its responsibilities during the operational testing phase, such as approving the operational test plan or writing a letter of assessment of the final results of operational testing. As it has done for two other recent DHS acquisition programs, TES was able to confirm through its independent assessment whether the test agent conducted operational testing as described in the test and evaluation master plan and operational test plan. For example, TES outlined concerns in its letter of assessment to the ARB that the AT-2 system did not meet a stated operational requirement key performance parameter--a throughput measure of bags per hour--for the majority of the time under test which resulted in a "not effective" determination by TES. TES officials recognized that, as the test agent, they are not in a position to conduct an independent assessment of operational test results and write a letter of assessment for ASP and that they are the highest level organization within DHS for both T&E oversight and operational test expertise. They further stated that the decision to have TES serve as the test agent was made prior to the issuance of the T&E directive and that it was too late in the program's development to go back and select another agent. Nevertheless, TES officials recognized that this one-time situation would result in the lack of an independent assessment of ASP test results and there were no plans to conduct or contract for such an independent assessment. While we acknowledge that this decision was made prior to the T&E directive and the requirement that TES write a letter of assessment of all major acquisition programs, it is nonetheless important that ASP undergo an independent assessment of its test results since its operational test plan, which was developed by TES, was not subject to oversight. Because ASP has faced testing issues, many of which we have reported on in past years, it is important that this program undergo oversight to help avoid similar problems from reoccurring.[Footnote 44] Without an independent assessment of ASP's operational test results, it will be difficult to ensure that operational testing was properly planned, conducted, and that the performance results are useful. In addition, arranging for an independent assessment of operational tests results could provide the ARB with critical information on testing and evaluation efforts to help it determine whether ASP should be approved for purchase and implementation. TES and Component Officials Cited Challenges in Coordinating and Overseeing T&E across DHS; Efforts Are Underway to Address Some Challenges: TES and component officials reported challenges faced in coordinating and overseeing T&E across DHS components that fell into four primary categories: (1) ensuring that a program's operational requirements-- the key requirements that must be met for a program to achieve its intended goals--can be effectively tested; (2) working with DHS component program staff that have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time and resources for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, but it is too early to determine their effectiveness. Ensuring Operational Requirements Can Be Tested and Are Suitable to Meet Mission Needs: Both TES and component officials stated that one of their challenges is developing requirements that are testable, consistent, accurate, and complete. Specifically, six of the nine TES test area managers told us that working with DHS components to ensure that operational requirements can be tested and are suitable to meet mission needs is important because requirements development is one of the biggest challenges facing DHS. For example, one TES test area manager described the difficulty in drafting a test and evaluation master plan if operational requirements are not testable and measurable. Another TES test area manager indicated that programs' operational requirements documents often do not contain user needs or operational requirements for system performance. This leads to difficulties in testing those requirements later. Further, six of the nine TES test area managers cited that some components' operational requirements are difficult to test as written, which results in delays in drafting T&E documents as well as impacting the program cost and schedule parameters. Our prior work has found that program performance cannot be accurately assessed without valid baseline requirements established at the program start. According to DHS guidance, the baseline requirements must include a threshold value that is the minimum acceptable value which, in the user's judgment, is necessary to satisfy the need. In June 2010, we reported that if threshold values are not achieved, program performance is seriously degraded, the program may be too costly, or the program may no longer be timely.[Footnote 45] In addition, we reported that inadequate knowledge of program requirements is a key cause of poor acquisition outcomes, and as programs move into the produce and deploy phase of the acquisition process, problems become much more costly to fix. To help remedy these issues, we have made a number of recommendations to address them. DHS has generally agreed with these recommendations and, to varying degrees, has taken actions to address them. For example: * In May 2010, we reported that not all of the SBInet operational requirements that pertain to Block 1--a surveillance, command, control, communications, and intelligence system being fielded in two portions of the international border in Arizona--were achievable, verifiable, unambiguous, and complete. For example, a November 2007 DHS assessment determined that 19 operational requirements, which form the basis for the lower-level requirements used to design and build the system, were not complete, achievable, verifiable, or affordable. Further, the DHS assessment noted that a requirement that the system should provide for complete coverage of the border was determined to be unverifiable and unaffordable because defining what complete coverage meant was too difficult and ensuring complete coverage, given the varied and difficult terrain along the border, was cost prohibitive. To address these issues, we recommended that the currently defined Block 1 requirements, including key performance parameters, are independently validated as complete, verifiable, and affordable and any limitations found in the requirements are addressed.[Footnote 46] Furthermore, CBP program officials told us that they recognized the difficulties they experienced with requirements development practices with the SBInet program. Within CBP, the Office of Technology, Innovation, and Acquisition has responsibility for managing the SBInet program. Office of Technology, Innovation, and Acquisition officials told us that their office was created to strengthen expertise in acquisition and program management of SBInet. * In May 2009, we reported that ASP testing uncovered multiple problems in meeting the requirements for successful integration into operations at ports of entry.[Footnote 47] As a result, we recommended that DHS assess ASPs against the full potential of current equipment and revise the program schedule to allow time to conduct computer simulations of ASP's capabilities and to uncover and resolve problems with ASPs before full-scale deployment. * We also reported that other TSA technology projects were delayed because TSA had not consistently communicated clear requirements in order to test the technologies.[Footnote 48] We recommended that TSA evaluate whether current passenger screening procedures should be revised to require the use of appropriate screening procedures until it is determined that existing emerging technologies meet their functional requirements in an operational environment. In March 2011 testimony, the Under Secretary for S&T stated that S&T had begun working with the DHS Under Secretary for Management to use their collective expertise and resources to better address the "front end" of the acquisition cycle, namely, the translation of mission needs into testable requirements.[Footnote 49] Further, in response to this challenge, S&T has reorganized and established an Acquisition Support and Operations Analysis Group, which is to provide a full range of coordinated operations analysis, systems engineering, T&E, and standards development support for DHS components. In addition, TES's T&E Council is currently focusing on the challenges related to requirements development. Specifically, TES test area managers have presented specific briefings to component officials at council meetings which provide information on how to better generate requirements.[Footnote 50] Further, in response to our previously mentioned report designating DHS on the high-risk list, DHS developed a strategy to, among other things, strengthen its requirements development process.[Footnote 51] DHS's January 2011 strategy describes the establishment of a capabilities and requirements council to evaluate and approve operational requirements early in the acquisition process.[Footnote 52] Specifically, the capabilities and requirements council is to, among other things, reconcile disagreements across program offices and approve analyses of alternatives and operational requirement documents. We stated in a March 2011 response to DHS on its strategy that it was unclear how the introduction of new governance groups will streamline the process and address previously identified issues because it appeared that the governance groups are chaired by the Deputy Secretary and have many of the same participants. Since the S&T reorganization has only recently taken place and the T&E Council and the department's strategy have only recently begun to address the challenge of requirements generation, it is too soon to determine the effectiveness of these actions in addressing this challenge. T&E Experience and Expertise within DHS Components Varies: TES officials told us that T&E experience and expertise within DHS components varies, with some components possessing staff with extensive T&E experience and expertise and others having relatively little. For example, TES officials noted that the Coast Guard and TSA have T&E policies and procedures in place, as well as staff with extensive T&E experience, which limited their dependence on TES for T&E expertise. Other components in DHS told us they rely more on TES or contractors for T&E expertise. For the 11 DHS programs we reviewed, officials from components which do not have many acquisition programs, such as the Office of Intelligence and Analysis, reported needing more assistance from TES in identifying and selecting appropriate and qualified test agents, for example. Conversely, components with more acquisition programs, such as the Coast Guard, told us that they have well-established test agents and procedures in place, and require little guidance from TES. For example, we reported in April 2011 that most Coast Guard major acquisition programs leverage Navy expertise, in some way, to support a range of testing, engineering, and other program activities.[Footnote 53] Furthermore, CBP recently established a new office whose goal is to strengthen expertise in acquisition and program management, including T&E, and ensure that CBP's technology efforts are focused on its mission and integrated across the agency. In response to this challenge, TES has worked with DHS's Acquisition Workforce Office to develop T&E certification requirements and training for components. TES officials told us that they have worked with the Acquisition Workforce Branch and developed pilot courses on T&E for component T&E staff, including Fundamentals of Test and Evaluation, Intermediate Test and Evaluation, and Advanced Test and Evaluation. In April 2010, DHS issued an acquisition workforce policy which establishes the requirements and procedures for certification of DHS T&E managers.[Footnote 54] The policy allows T&E managers to be certified at a level that is commensurate with their education, training, and experience. Component staff from 6 of the 11 programs we reviewed said they participated in TES's certification training program and believed that the training would assist them in carrying out their T&E responsibilities. In addition, TES is in the process of hiring four additional staff to assist the test area managers in their T&E oversight responsibilities and hoped to have the additional staff hired by the end of fiscal year 2011. Lack of DHS staff to conduct acquisition oversight, including T&E, is a departmentwide challenge. In our previous reports, DHS acquisition oversight officials said that funding and staffing levels have limited the number of programs they can review.[Footnote 55] We recommended that DHS identify and align sufficient management resources to implement oversight reviews in a timely manner. DHS generally concurred with the recommendation and, as of January 2011, has reported taking action to address it by identifying needed capabilities and hiring staff to fill identified gaps. Further, to address this challenge, in 2009 and 2010, T&E Council representatives from the Acquisition Workforce Branch made presentations at council meetings to update members on the status of various acquisition workforce issues, including T&E certification. For example, presenters asked T&E Council members to inform their respective components about new T&E certification courses and to provide information on how to sign up for the courses. In 2010, the Acquisition Workforce Policy was implemented by DHS, which allowed the department to begin to certify T&E acquisition personnel. While DHS has undertaken efforts to help address these challenges, it is too soon to evaluate the impact that these efforts will have in addressing them. Using Existing T&E Directives and Guidance to Oversee Information Technology Acquisitions: Effectively managing IT acquisitions is a governmentwide challenge. TES and component officials we interviewed told us that T&E guidance, such as specific guidance for integrating developmental testing and operational testing, may not be sufficient for the acquisition of complex IT systems.[Footnote 56] Specifically, component officials stated that the assessment of risks and environmental factors are different for IT programs than other acquisitions and that conducting testing in an operational environment may not be necessary for IT programs because the operational environment is no different than the test environment. In addition, four of the nine test area managers told us that aspects of the existing T&E guidance may not directly apply to IT acquisitions. The department is in the process of making modifications to its acquisitions process to better accommodate information technology acquisitions. According to the previously mentioned January 2011 strategy submitted to GAO, DHS is piloting a new model for IT acquisitions. This model, which is to be consistent with the department's overall acquisition governance process, is to have many of the steps in the modified process that are similar or the same as what currently exists but time frames for different types of acquisitions would be instituted. For example, acquisition programs designated as IT programs may go through a more streamlined acquisition process that may better fit the rapidly changing IT environment, and the ARB would have the option to delegate oversight responsibilities to an executive steering committee. In other cases, TES and component officials are investigating the possibility of conducting integrated testing--the combination of developmental and operational testing--for some programs although this process may take longer to plan and pose greater risks because testing is being done simultaneously. Further, the T&E Best Practices Integrated Working Group, a subgroup of the T&E Council, including TES, Acquisition Program Management Division, and Office of Chief Information Officer officials, was working to identify and promote T&E best practices for IT system acquisition. This group drafted an operational test agent risk assessment process to validate the streamlining process approach while adhering to acquisition and T&E policy and directives, and as of March 2011, one component, USCIS, has made use of this process. Additionally, three other programs are investigating the possible use of this process and the possibility of tailoring or eliminating T&E deliverables or operational T&E requirements for IT programs, with the approval of TES. The group has identified three IT acquisition programs to serve as a pilot for this effort.[Footnote 57] As DHS considers modifications to its T&E process for IT programs, it also must consider the effect such a change could have on determining a system's technical performance and evaluating the system's operational effectiveness and suitability. For example, we have previously reported on testing problems with SBInet, a CBP program designated as an IT program. We found that SBInet testing was not performed in a manner that would adequately ensure that the system would perform as intended.[Footnote 58] Among the factors contributing to these problems was insufficient time for reviewing and approving test documentation, which in part, led to test plans and test cases not being well-defined. As a result, we recommended that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised test and evaluation master plan. Since the efforts DHS is taking to address this challenge have only recently been initiated, it is too early to tell what impact they will have on the overall challenges of T&E for IT programs. Allowing Appropriate Time for T&E within Program Cost and Schedule: Both TES and component officials stated that balancing the need to conduct adequate T&E within the confines of a program's costs and schedule is a recurring challenge, and a challenge that is difficult to solve. We have previously reported on the challenges associated with balancing the need to conduct testing within program cost and schedules. Our past review of the Department of Defense's (DOD) Director of Operational Test and Evaluation found that while the acquisition community has three central objectives--performance, cost, and schedule--the Director of Operational Test and Evaluation has but one--operational testing of performance.[Footnote 59] We reported that these distinct priorities can lead to testing disputes. We reported that these disputes encompassed issues such as (1) how many and what types of test to conduct; (2) when testing should occur; (3) what data to collect, how to collect them, and how best to analyze them; and (4) what conclusions were supportable, given the analysis and limitations of the test program. The foundation of most of these disputes laid in different notions of the costs and benefits of testing and the levels of risk that were acceptable when making full-rate production decisions. The DOD Director of Operational Test and Evaluation consistently urged more testing (and consequently more time, resources, and cost) to reduce the level of risk and number of unknowns before the decision to proceed to full-rate production, while the services consistently sought less testing and accepted more risk when making production decisions. These divergent dispositions frequently led to healthy debates about the optimal test program, and in a small number of cases, the differences led to contentious working relations. TES and DHS component officials expressed views similar to those expressed in our past work at DOD. Of the nine TES test area managers we talked with, four told us that allowing appropriate time and resources for T&E within program cost and schedule is a challenge. According to the test area manager's, component program management officials often do not incorporate sufficient time within their schedule for T&E or reduce the time allowed for T&E to save time and money. In one test area managers' view, doing so can reduce the effectiveness of testing or negatively impact the results of the tests. However, TSA officials told us that TES wanted to insert new test requirements for the AT-2 program--including the involvement of more TSA staff in the tests--after the program schedule was established and it was difficult to accommodate the changes and resulted in some delays. TES officials told us that these test requirements were in lieu of other planned field testing, which were not consistent with the program's concept of operations and that TSA officials agreed with the new test requirements. According to TES and component officials we spoke with, both the program officials and TES understand the views and perspectives of one another and recognize that a balance must be struck between effective T&E and managing programs within cost and schedule. As a result, TES is working with program officials through the T&E Council or T&E working groups to discuss these issues early in the acquisition cycle (before it is too late), particularly while developing the test and evaluation master plan, which outlines the time allowed for testing and evaluation. Conclusions: Timely and accurate information resulting from T&E of major acquisitions early in the acquisition process can provide valuable information to DHS's senior level managers to make informed decisions about the development, procurement, deployment, and operation of DHS's multibillion dollar portfolio of systems and services. Improving the oversight of component T&E activities is but one part of the significant challenges DHS faces in managing its acquisitions. Components themselves are ultimately responsible for the management and implementation of their programs and DHS senior level officials are responsible for making key acquisitions decisions which lead to production and deployment. TES helps support acquisition decisions by providing oversight over major acquisitions' T&E, which can help reduce, but not eliminate, the risk that new systems will not be operationally effective and suitable. Since the Homeland Security Act creating DHS was enacted in 2002, S&T has had the responsibility for overseeing T&E activities across the department. However, S&T did not have staff or the acquisition and T&E directives in place to conduct such oversight across DHS components until May 2009 when DHS issued its T&E directive. Since then, TES has implemented some of the requirements and overseen T&E of major acquisitions we reviewed, as well as provided independent assessments of operational test results to the ARB. However, TES has not consistently documented its compliance with the directives. Documenting that TES is fulfilling the requirements within DHS acquisition and T&E directives and the extent to which the criteria it is using to review and approve these documents are met, including approving operational test agents and reviewing key acquisition documentation, would assist TES in demonstrating that it is conducting T&E oversight and meeting requirements in these directives. Furthermore, without an independent assessment of operational test results for the Advance Spectroscopic Portal program, a key T&E oversight requirement in the T&E directive, the ARB will lack T&E oversight and input it needs to determine whether ASP is ready to progress toward production and deployment. This is especially important, given that program's troubled history, which we have highlighted in a series of prior reports. Recommendations for Executive Action: To better ensure that testing and evaluation requirements are met, we recommend that the Secretary of Homeland Security direct the Under Secretary for Science & Technology to take the following two actions: * Develop a mechanism to ensure that TES documents its approval of operational test agents and the extent that the test agents meet the requirements in the T&E directive, and criteria that TES use in reviewing these test agents for major acquisition programs. * Develop a mechanism to ensure that TES documents its required review of component acquisition documents, including the mission need statements, concept of operations, operational requirements documents, developmental test reports, test plans, and other documentation required by the T&E directive, the extent that these documents meet the requirements in the T&E directive, and criteria that TES uses in reviewing these documents. To ensure that the ARB is provided with an independent assessment of the operational test results of the Advanced Spectroscopic Portal program to help determine whether the program should be approved for purchase and implementation, we recommend that the Secretary of Homeland Security take the following action: * Arrange for an independent assessment, as required by the T&E directive, of ASP's operational test results, to include an assessment of the adequacy of the operational test and a concurrence or nonconcurrence on the operational test agent's evaluation of operational suitability and operational effectiveness. Agency Comments and Our Evaluation: We received written comments on a draft of this report from DHS on June 10 2011, which are reproduced in full in appendix III. DHS concurred with all three of our recommendations. DHS concurred with our first recommendation (1) that S&T develop a mechanism to ensure that TES documents its approval of operational test agents, (2) the extent that the test agents meet the requirements in the T&E directive, and (3) the criteria that TES uses in reviewing these test agents for major acquisition programs. Specifically, DHS stated that the Director of TES issued a memorandum to test area managers and TES staff regarding the operational test agent approval process which describes the responsibilities, considerations for selection, and the process necessary to select an operation test agent. In addition, DHS stated that TES is drafting memos approving operational test agents using the new test agent approval process. DHS also concurred with our second recommendation that S&T develop a mechanism to ensure that TES documents (1) its required review of component acquisition documents required by the T&E directive, (2) the extent that these documents meet the requirements in the T&E directive, and (3) the criteria that TES uses in reviewing these documents. DHS stated that the Director of TES issued a memorandum to test area managers and TES staff detailing the role of TES in the document review process and the process that TES staff should follow for submitting their comments to these documents. Finally, DHS concurred with our third recommendation that S&T arrange for an independent assessment of ASP's operational test results. DHS stated that the ASP program is under review and does not have an operational test scheduled. However, TES is investigating the option of using a separate test agent to conduct operational testing of ASP, which would allow TES to perform the independent assessment and fulfill its independent oversight role as outlined in DHS policy. Such actions, if taken, will fulfill the intent of this recommendation. DHS also provided technical comments on the report, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees and the Secretary of Homeland Security. The report will be available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you or your staff have questions regarding this report, please contact me at (202) 512-9627 or at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. Signed by: David C. Maurer: Director, Homeland Security and Justice Issues: [End of section] Appendix I: Descriptions of Selected Major Acquisitions Programs: [Refer to PDF for image: illustrated table: photo included for each program] Program: Advanced Spectroscopic Portal (ASP); Component: Domestic Nuclear Detection Office (DNDO); Type: Non-IT; Level: 1; Description: An effort to develop and deploy technologies to allow Customs and Border Protection to detect nuclear or radiological materials from conveyances, such as trucks, entering the United States at land and sea ports of entry; Phase as of April 2011: Obtain. Program: Atlas Tactical Communications (TACCOM); Component: U.S. Immigration and Customs Enforcement (ICE); Type: IT; Level: 1; Description: An effort to modernize ICE’s tactical communication systems and equipment, that Ice agents and officers use to support mission-critical communications from outdated analog systems to modern and standardized digital systems. Project 25 upgrade will modernize tactical communications and deploy site infrastructure and end-user subscriber radios. Interoperable Rapid Deployment Systems (IRDS) will outfit ICE with transportable communication systems to support rapid deployment requirements fro routine, emergency and disaster response, and special operations. The program is divided into six segments, including: (1) P25 upgrade for the Atlanta Region, (2) P25 upgrade for the Boston Region, (3) P25 upgrade for the Denver Region, (4) P25 upgrade for the central hub infrastructure, (5) IRDS mobile radio communication kits that support disaster and emergency response operations, and, (6) IRDS Mobile Communication Systems (MCS) mobile communication vehicles that support disaster and emergency response operations. In March 2011, the Component Acquisition Executive determined that the TACCOM program would be consolidated with other ICE infrastructure programs and that the program would be required to submit acquisition documentation to the ARB prior to August 2011. Phase as of April 2011: Mixed: Segment 1: Produce/Deploy/Support; Segments 2-6: In the process of being updated. Program: BioWatch Gen-3; Component: Office of Health Affairs (OHA); Type: Non-IT; Level: 1; Description: A nationwide, interoperating network of detectors/identifiers that is to provide autonomous air-sampling analysis of the environment for biological agents of concern. The system is to enable detection, identification, and reporting of recognized organisms within a 6-hour period; Phase as of April 2011: Obtain. Program: HH-65 Conversion/Sustainment Projects; Component: U.S. Coast Guard (USCG); Type: Non-IT; Level: 1; Description: The project is to upgrade 95 helicopters and procure 7 new helicopters to extend the helicopters’ service life in the fleet through 2025. There are six discrete segments: (1) Re-engine - Upgrade 95 helicopters with new and more powerful engines; (2) National Capital Region Air Defense (NCRAD) - Increase fleet size from 95 to 102; (3) Airborne Use of Force (AUF) – Increase communications capability and provide weapons and night vision; (4) Obsolete Component Modernization (OCM) – Replace obsolete components and subsystems; (5) Ship Helicopter Secure Traverse & System (SHSTS) – Provide the ability to automatically secure the aircraft to the flight deck and traverse it into the hanger; and, (6) Automatic Flight Control System (AFCS/Avionic) Modernize digital Common Avionics Architecture System (CAAS) common with the H-60T upgrade and digital Automatic Flight Control System; Phase as of April 2011: Mixed: Segment 1: Produce/Deploy/Support; Segment 2: Produce/Deploy/Support; Segment 3: Produce/Deploy/Support; Segment 4: Produce/Deploy/Support; Segment 5: Obtain; Segment 6: Obtain. Program: Information Integration and Transformation (IIT); Component: U.S. Secret U.S. Secret Service (USSS); Type: IT; Level: 2; Description: An effort to modernize Secret Service’s IT infrastructure, communications systems, applications, and processes. The program is divided into four discrete segments: (1) Enabling Capabilities: IT Infrastructure Modernization/Cyber Security/Database Architecture; (2) Communications Capabilities; (3) Control Capabilities; and, (4) Mission Support Capabilities. In February 2011, the ARB granted an acquisition decision event 2A and acquisition decision event 2B decision for the Enabling Capabilities segment. The remaining three segments remained in the Analyze/Select phase; Phase as of April 2011: Mixed: Segment 1: Obtain; Segment 2: Analyze/Select; Segment 3: Analyze/Select; Segment 4: Analyze/Select. Program: National Cybersecurity & Protection System (NCPS); Component: National Protection and Programs Directorate (NPPD); Type: IT; Level: 1; Description: An integrated system of intrusion detection, analytical, intrusion prevention, and information-sharing capabilities that are to be used to defend the federal civilian government’s information technology infrastructure from cyber threats. Includes the hardware, software, supporting processes, training, and services that are to be developed and acquired to support the mission. The initial system, known as Einstein, was renamed as Block 1.0 and includes capabilities such as centralized data storage. (1) Block 2.0 is to add an Intrusion Detection System (IDS) which is to assess network traffic for the presence of malicious activity; (2) Block 2.1 is to provide a Security Incident and Event Management which is to enable data aggregation, correlation, and visualization. (3) Block 3.0 is to provide an intrusion prevention capability; Phase as of April 2011: Mixed: Block 2.0: Produce/Deploy/Support; Block 2.1: Obtain; Block 3.0: Analyze. Program: National Security Systems Program (NSSP); Component: Office of Intelligence & Analysis (I&A); Type: IT; Level: 1; Description: A joint initiative between Intelligence and Analysis (I&A) and the Office of the Chief Information Officer which is to bring a unified, enterprise approach to the management of all classified information technology infrastructure including: (1) Homeland Secure Data Network (HSDN) for secret level communications infrastructure; (2) Homeland Top Secret Network (HTSN) for top secret communications infrastructure; and; (3) Homeland Secure Communications (HSC) for classified voice and video teleconference capabilities; Phase as of April 2011: Mixed: HSDN: Produce/Deploy/Support; HTSN: Analyze/Select; HSC: Analyze/Select. Program: Passenger Screening Program (AT-2); Component: Transportation Security Administration (TSA); Type: Non-IT; Level: 1; Description: A next generation of x-ray technology that is to complement the traditional x-ray technology and provide new technical capabilities, such as automated detection algorithms, threat image projection, alternate viewing station, bulk explosive algorithms, and expanded threat list that incorporates emerging threats to aviation security. A system which is to provide Transportation Security Officers capability to screen passengers' carry-on baggage at airports nationwide; Phase as of April 2011: Produce/Deploy/Support. Program: SBInet Block 1; Component: U.S. Customs and Border Protection (CBP); Type: IT; Level: 1; Description: A program which is to deliver surveillance and decision- support technologies that create a virtual fence and situational awareness along the U.S. borders with Mexico and Canada. The first SBInet deployment of the Block I system took place in the Tucson, Arizona station. The second deployment of the Block I system took place in the Ajo, Arizona station. In January 2011, the SBInet program ended as originally conceived; however, limited deployments of technology, including 15 sensor towers and 10 communication towers, remained deployed and operational in Arizona. The T&E results on these towers were to be reported sometime in April 2011; Phase as of April 2011: Obtain. Program: Transformation and Systems Consolidation (TASC); Component: DHS Office of the Chief Financial Officer (OCFO); Type: IT; Level: 1; Description: TASC is to develop and field an integrated financial management, asset management, and procurement management system solution. The program is to use standard business processes and a single line of accounting compliant with the common governmentwide accounting classification structure. The TASC Executive Steering Committee determined that the Federal Emergency Management Agency will be the first DHS component to migrate to TASC; Phase as of April 2011: Analyze/Select. Program: Transformation; Component: U.S. Citizenship and Immigration Services (USCIS); Type: IT; Level: 1; Description: An effort to move immigration services from a paper-based model to an electronic environment. The program is to deliver a simplified, Web-based system for benefit seekers to submit and track their applications. The new, account-based system is to provide customers with improved service; Phase as of April 2011: Analyze/Select. Source: GAO analysis of DHS component program descriptions, acquisition decision memoranda, and Acquisition Program Management Division data; and DHS and Flickr photo-images. [End of table] [End of section] Appendix II: DHS's Fiscal Year 2010 Major Acquisition Programs: In fiscal year 2010, there were 86 acquisition programs on the Acquisition Program Management Division's oversight list, which included the acquisition level and designation as an information technology acquisition. Table 2 lists information on these 86 acquisition programs, and in addition, includes information on the acquisition phase for each program as of April 2011 and whether the program was subject to the test and evaluation (T&E) directive. For example, some programs, such as Customs and Border Protection's acquisition of Border Patrol Facilities would not involve any T&E activities and therefore would not be subject to the requirements in the T&E directive or DHS Science and Technology Directorate's Test and Evaluation and Standards office (TES) oversight. Table 2: DHS Fiscal Year 2010 Major Acquisition Programs: Analysis and Operations (A&O): Component and acquisition program: Common Operational Picture (COP); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Homeland Security Information Network (HSIN); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Customs and Border Protection (CBP): Component and acquisition program: Advance Passenger Information (APIS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No. Component and acquisition program: Automated Commercial Environment (ACE); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Automated Targeting System (ATS) Maintenance; Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No. Component and acquisition program: Border Patrol Facilities; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: No. Component and acquisition program: Facilities Management &Engineering Tactical Infrastructure (FM&E TI); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: No[B]. Component and acquisition program: Fleet Management Program (FMP); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: Land Ports of Entry Modernization; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Non-Intrusive Inspection Systems Program (NII); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: SAP; Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Secure Border Initiative Network (SBInet); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Secure Freight Initiative; Acquisition level: 3; Information technology acquisition: IT; Acquisition phase as of April 2011: [C]; Subject to T&E oversight: No[C]. Component and acquisition program: Strategic Air and Marine Plan; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: Tactical Communication (TAC COM); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: TECS Modernization; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: Transportation; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[D]. Component and acquisition program: Western Hemisphere Travel Initiative (WHTI); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. DHS: Component and acquisition program: Chief Administrative Officer (CAO) Consolidated Mail System Program/Consolidated Remote Delivery (CRO); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: n/a; Subject to T&E oversight: No[D]. Component and acquisition program: Chief Administrative Officer (CAO) Electronic Records Management System (ERMS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Chief Financial Officer (CFO) Transformation and Systems Consolidation (TASC); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: St. Elizabeth's; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: n/a; Subject to T&E oversight: No[B]. Component and acquisition program: Chief Human Capital Officer (CHCO) Human Resource Information Technology (HR-IT); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Chief Information Officer (CIO) Homeland Secure Data Network (HSDN); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[A]. Component and acquisition program: CIO Infrastructure Transformation Program (ITP); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Domestic Nuclear Detection Office (DNDO): Component and acquisition program: Advanced Spectroscopic Portal (ASP); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Federal Emergency Management Agency (FEMA): Component and acquisition program: Grants Management Integrated Environment (GMIE); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Housing Inspection Services (HIS); Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: No[D]. Component and acquisition program: Risk Mapping, Analysis and Planning (Risk Map); Acquisition level: 1; Information technology acquisition: IT and Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Logistics Supply Chain Management System (LSCMS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Intelligence and Analysis (I&A): Component and acquisition program: National Security System Program (NSSP); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Online Tracking Information System (OTIS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: n/a; Subject to T&E oversight: No[D]. Immigration and Customs Enforcement (ICE): Component and acquisition program: Atlas; Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: Detention and Removal Operations (DROM); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Detention and Removal Operations (DRO); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: No[D]. Component and acquisition program: DRO Electronic Health Record System (EHR); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Federal Financial Management System (FFMS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[C]. Component and acquisition program: Student & Exchange Visitor Information System (SEVIS I and II); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Tactical Communications (TAC COM); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: TECS Modernization; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. National Protection and Programs Directorate (NPPD): Component and acquisition program: Federal Protective Services; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: n/a; Subject to T&E oversight: No[D]. Component and acquisition program: Infrastructure Information Collection Program & Visualization (IICVP); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: National Cybersecurity Protection System (2.1 and 3.0); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Next Generation Network (NGN); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: U.S. Visitor and Immigrant Status Indicator Technology (US VISIT); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: National Security Emergency Preparedness Priority Telecom Service (NS/EP PTS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[C]. Office of Health Affairs (OHA): Component and acquisition program: BioWatch Gen-3; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Science and Technology Directorate (S&T): Component and acquisition program: National Bio and Agro-Defense Facility (NBAF); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: National Biodefense Analysis and Countermeasures Center (NBACC); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Transportation Security Administration (TSA): Component and acquisition program: Electronic Baggage Screening Programs (EBSP); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Field Real Estate Management (FREM); Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[D]. Component and acquisition program: HRAccess; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[D]. Component and acquisition program: National Explosives Detection Canine Team Program System (K9); Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[E]. Component and acquisition program: Passenger Screening Programs (PSP); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: Screening Partnership Program; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: No[E]. Component and acquisition program: Secure Flight; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Security Technology Integrated Program (STIP); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Specialized Training; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[E]. Component and acquisition program: Transportation Worker Identification Credentialing (TWIC); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: TTAC Infrastructure Modernization Program (TIM); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Information Technology Infrastructure Program (ITIP); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. U.S. Coast Guard (USCG): Component and acquisition program: Core Accounting System (CAS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[F]. Component and acquisition program: HC-130H Conversion/Sustainment Projects; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: HC-130J Fleet Introduction (Missionization project); Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: HC-144A Maritime Patrol Aircraft (MPA); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: H-60 Conversion Projects; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Component and acquisition program: H-65 Conversion/Sustainment Projects; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Unmanned Aircraft Systems (UAS); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Need; Subject to T&E oversight: Yes. Component and acquisition program: C4ISR (Command, Control, Communication, Computers, Intelligence, Surveillance, and Reconnaissance); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Coast Guard Logistics Information Management System (CG LMIS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Interagency Operations Centers (IOC); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Nationwide Automatic Identification System (NAIS); Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Rescue 21; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Coastal Patrol Boat; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Cutter Boats; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: f; Subject to T&E oversight: No[F]. Component and acquisition program: Fast Response Cutter (FRC); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Medium Endurance Cutter Sustainment; Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: National Security Cutter (NSC); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Obtain; Subject to T&E oversight: Yes. Component and acquisition program: Offshore Patrol Cutter (OPC); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Patrol Boat Sustainment; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. Component and acquisition program: Response Boat - Medium (RB M); Acquisition level: 1; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: Yes. U.S. Citizenship and Immigration Services (USCIS): Component and acquisition program: Benefits Provision - Verification Information System (VIS); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. Component and acquisition program: Customer Service Contract Call Center Operations; Acquisition level: 2; Information technology acquisition: Non-IT; Acquisition phase as of April 2011: n/a; Subject to T&E oversight: No[D]. Component and acquisition program: Integration Document Production (IDP); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Produce/Deploy/Support; Subject to T&E oversight: No[A]. Component and acquisition program: Transformation; Acquisition level: 1; Information technology acquisition: IT; Acquisition phase as of April 2011: Analyze/Select; Subject to T&E oversight: Yes. U.S. Secret Service (USSS): Component and acquisition program: IT Modernization or Information Integration and Transformation (IIT); Acquisition level: 2; Information technology acquisition: IT; Acquisition phase as of April 2011: Mixed; Subject to T&E oversight: Yes. Source: GAO analysis of Acquisition Program Management Division and TES data. [A] Program was in the Produce/Deploy/Support phase and did not require T&E oversight. [B] Program was to acquire facilities and did not require T&E. [C] Program was later redesignated as level 3 acquisition which does not require T&E oversight. [D] Program was to acquire services and did not require T&E. [E] Program was to acquire training and did not require T&E. [F] Program was later delegated to the Component Acquisition Executive for oversight. n/a: Not available because DHS did not provide the acquisition phase for this program. Note: Acquisition level 1 includes programs with $1 billion or greater in estimated total life-cycle costs and level 2 includes programs with $300 million to $1 billion in estimated total life-cycle costs. Mixed acquisition phase represents programs with two or more subprojects which may simultaneously be in different acquisition phases. According to the TES Director, some programs in the Produce/Deploy/Support acquisition phase had limited T&E activity and were subject to the T&E directive, while others in this phase did not have any T&E activity and were not subject to the T&E directive. In January 2011, the SBInet program ended as originally conceived; however, limited deployments of technology, including 15 sensor towers and 10 communication towers, remained deployed and operational in Arizona. See GAO-11-448T. ICE and CBP TACCOM programs are to use compatible radio hardware; however, they are different programs. ICE's TACCOM program is to involve upgrading radios and attaching units to existing infrastructure, whereas CBP, in addition to updating existing infrastructure, will expand system coverage by incorporating new communication sites. [End of table] [End of section] Appendix III Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, D.C. 20528: June 10, 2011: David C. Maurer: Director: Homeland Security and Justice Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Re: Draft Report GAO-11-596, "DHS Science And Technology: Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met" Dear Mr. Maurer: Thank you for the opportunity to review and comment on this draft report. The U.S. Department of Homeland Security (DHS) appreciates the U.S. Government Accountability Office's (GAO's) work in planning and conducting its review and issuing this report. The Department is pleased to note the report's positive acknowledgment of actions DHS has taken to implement more discipline and rigor it its acquisition processes. The report also recognizes the importance the DHS Science and Technology Directorate's Test & Evaluation and Standards Office (TES) has in the successful development of major acquisition programs and technologies, as well as its importance overseeing the T&E efforts of DHS's components. Along these lines, DHS is proud that GAO found TES met its oversight requirements for approving test plans and test reports for all 11 (100 percent) of the major acquisition programs reviewed. We agree with GAO that the results of their analyses provide informative examples of the extent to which TES has carried out its major acquisition program oversight responsibilities. The draft report contained three recommendations directed to DHS. As discussed below, DHS concurs with all the recommendations. Specifically, to better ensure that testing and evaluation requirements are met, GAO recommended that the Secretary of Homeland Security direct the Under Secretary for Science & Technology to: Recommendation 1: Develop a mechanism to ensure that TES documents its approval of operational test agents and the extent that the test agents meet the requirements in the T&E directive, and criteria that TES use in reviewing these test agents for major acquisition programs. Response: Concur. On January 14, 2011, during GAO's review, the Director of TES issued a memorandum to the Test Area Managers and the TES Staff regarding the operational test agent approval process. This memorandum describes the responsibilities of, considerations for selection, and the process necessary to select an Operation Test Agent (OTA) in the conduct of Operational Test & Evaluation (OT&E) efforts in support of a DHS acquisition. TES is currently drafting formalized memorandums re-certifying OTAs according to the documented operational test agent approval process. Recommendation 2: Develop a mechanism to ensure that TES documents its required review of component acquisition documents, including the mission need statements, concept of operations, operational requirements documents, developmental test reports, test plans, and other documentation required by the T&E directive, the extent that these documents meet the requirements in the T&E directive, and criteria that TES use in reviewing these documents. Response: Concur. Shortly after a TES off-site in January 2010, before this GAO review was initiated, the Director of TES began working on formalizing internal processes. On April 15, 2011, the Director of TES issued a memorandum to the Test Area Managers and TES Staff regarding the document review process. This memorandum provides specific details on the definition of documents, the role of TES in the document review process, and how comments on these documents should be submitted. TES also participates on approximately 30 T&E Integrated Product Teams in which TES works closely with the Components during their document development to ensure T&E requirements are incorporated. To ensure that the Acquisition Review Board is provided with an independent assessment of the operational test results of the Advanced Spectroscopic Portal (ASP) Program to help determine whether the program should be approved for purchase and implementation, GAO recommended that the Secretary of Homeland Security: Recommendation 3: Arrange for an independent assessment, as required by the T&E directive, of ASP's operational test results, to include an assessment of the adequacy of the operational test and a concurrence or non-concurrence on the operational test agent's evaluation of operational suitability and operational effectiveness. Response: Concur. The ASP program is currently under review and does not have an operational test scheduled. TES however, is investigating the option of using a different OTA to conduct the ASP OT&E, and the DHS Director, Operational Test and Evaluation (DOT&E) will perform the independent oversight role outlined in DHS policy. Again, thank you for the opportunity to review and comment on this draft report. Technical comments were submitted under separate cover. We look forward to working with you on future Homeland Security issues. Sincerely, Signed by: Jim H. Crumpacker: Director: Departmental GAO/OIG Liaison Office: [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO, Test and Evaluation: Impact of DOD's Office of the Director of Operational Test and Evaluation, GAO/NSIAD-98-22 (Washington, D.C.: Oct. 24, 1997). GAO Contacts: David C. Maurer, (202) 512-9627 or maurerd@gao.gov: Staff Acknowledgments: In addition to the contact named above, Christopher Currie (Assistant Director), Nancy Kawahara, Bintou Njie, Melissa Bogar, Jessica Drucker, Caitlin White, Richard Hung, Michele Fejfar, Labony Chakraborty, Tracey King, Paula Moore, Dan Gordon, Michele Mackin, Molly Traci, and Sean Seales made significant contributions to this report. [End of section] Footnotes: [1] In 2010 constant dollars. [2] Pub. L. No. 107-296, 116 Stat. 2135 (2002). [3] Pub. L. No. 107-296, § 302(5), 116 Stat. 2135, 2163 (2002). [4] Previously known as Test & Evaluation and Standards Division (TSD). Under a March 2011 reorganization, TES is an office under the Acquisition Support and Operations Analysis Division in S&T and all T&E functions and personnel remained the same. [5] DHS defines these life-cycle costs in 2009 constant dollars. [6] GAO, Aviation Security: DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges, [hyperlink, http://www.gao.gov/products/GAO-10-128] (Washington, D.C.: Oct. 7, 2009). [7] GAO, Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations That Place Key Technology Program at Risk, [hyperlink, http://www.gao.gov/products/GAO-10-158] (Washington, D.C.: Jan. 29, 2010). [8] GAO, Border Security: Preliminary Observations on the Status of Key Southwest Border Technology Programs, [hyperlink, http://www.gao.gov/products/GAO-11-448T] (Washington, D.C.: Mar. 15, 2011). [9] DHS Directive Number 026-06, Test and Evaluation. [10] One of the 12 programs initially selected, FEMA's Grants Management Integrated Environment, was dropped from our initial selection because the program had been put on hold until fiscal year 2013 due to other departmental priorities and no substantive T&E activities had been initiated. We did not conduct an interview with program officials or analyze any program documentation. No other FEMA programs met our selection criteria. [11] GAO, Internal Control: Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [12] GAO, Department of Homeland Security: Billions Invested in Major Programs Lack Appropriate Oversight, [hyperlink, http://www.gao.gov/products/GAO-09-29] (Washington, D.C.: Nov. 18, 2008). [13] GAO, Department of Homeland Security: Assessments of Selected Complex Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-10-588SP] (Washington, D.C.: June 30, 2010). [14] GAO, High-Risk Series: An Update, [hyperlink, http://www.gao.gov/products/GAO-11-278] (Washington, D.C.: February 2011). [15] GAO, Homeland Security: Successes and Challenges in DHS's Efforts to Create an Effective Acquisition Organization, [hyperlink, http://www.gao.gov/products/GAO-05-179] (Washington, D.C.: Mar. 29, 2005). [16] System acquisition process means the sequence of acquisition activities starting from the agency's reconciliation of its mission need, with its capabilities, priorities, and resources, and extending through the introduction of a system into operational use or the otherwise successful achievement of program objectives. OMB Circular A- 109, Major System Acquisitions. [17] The department operated under the March 2006 Management Directive No. 1400, Investment Review Process, until November 2008 when DHS issued Acquisition Management Directive 102-01, Interim Version, which superseded Management Directive No. 1400. DHS Acquisition Instruction/ Guidebook 102-01, Interim Version 1.9 (Nov. 7, 2008). [18] DHS Management Directive No. 102-01, January 20, 2010. Acquisition Decision Events occur when the ARB meets to determine whether a program has all of the necessary acquisition documents and other DHS requirements to move to the next phase in the acquisition process. Programs receive an acquisition decision event 1, 2A, 2B, and 3. The 2A and 2B acquisition events may be combined into one acquisition decision event. [19] Levels are determined by the life-cycle cost of the program, not the procurement cost. Level 1 (major acquisition) life-cycle cost is identified at or above $1 billion dollars. Level 2 (major acquisition) life-cycle cost is identified as $300 million or more, but less than $1 billion dollars. Level 3 (nonmajor acquisition) life-cycle cost is identified as less than $300 million dollars. DHS Acquisition Directive 102-01 established the ARB. The ARB is the cross-component board within the department that determines whether a proposed acquisition has met the requirements of key phases in the acquisition life-cycle framework and is able to proceed to the next phase and eventual full production and deployment. The ARB is comprised of the acquisition decision authority (chair of the ARB), the Under Secretary for Management, the Under Secretary for Science and Technology, the Assistant Secretary for Policy, the General Counsel, the Chief Financial Officer, the Chief Procurement Officer, the Chief Information Officer, the Chief Human Capital Officer, the Chief Administrative Officer, the Chief Security Officer, user representatives from components sponsoring the capability, and other officials within the department determined to be appropriate to the subject matter by the acquisition decision authority. [20] While the directive gives the authority to delegate level 2 major acquisitions, Acquisition Program Management Division officials told us that no level 2 acquisitions have been delegated to the component level as of February 2011. [21] The ARB can raise an acquisition to a higher acquisition level if the ARB determines that its importance to DHS strategy and performance plans is disproportionate to its size; it has high executive visibility; it impacts more than one DHS component; it has significant program or policy implications; or the Deputy Secretary, Chief Acquisition Officer, or acquisition decision authority recommends an increase to a higher acquisition level. [22] [hyperlink, http://www.gao.gov/products/GAO-11-318SP]. [23] [hyperlink, http://www.gao.gov/products/GAO-10-588SP]. [24] The 24 major acquisition programs which did not require T&E oversight included 9 programs for services, such as Protective Services for National Protection and Programs Directorate; 5 programs which were in the Produce/Deploy/Support phase and did not require T&E oversight; 3 programs which involved training; 3 programs which were later redesignated to level 3 acquisitions; 2 programs which were for facilities; and, 2 which were delegated to the Component Acquisition Executive to manage. [25] DHS Directive Number 102-01 is designed in three parts. The first part describes the acquisition process and was signed by the Under Secretary for Management. It details the key acquisition documents such as the operational requirement documents that program offices need to provide to the ARB in order to receive an acquisition decision event. The second part provides guidance for conducting the acquisition process and was signed by the Chief Procurement Officer. This section provides specific guidance on how to complete all of the necessary steps for each acquisition decision event as well as the acquisition process. The third and final part provides templates for completing steps in the acquisition process. [26] DHS Directive Number 10100.1 "Organization of the Office of the Under Secretary for Science and Technology" established the Director of the Test and Evaluation and Standards Division in June 2007. [27] DHS Delegation 10003, issued in May 2009, specifies the responsibilities and duties of the Director of Operational Test & Evaluation. [28] For purposes in this report, we refer to TES Director and Director of Operational Test and Evaluation responsibilities generically as TES responsibilities, since some of their responsibilities overlap and the TES Director had been acting Director of Operational Test and Evaluation when the position was unfilled during fiscal year 2010. [29] A special oversight acquisition program is an acquisition that may be raised to a higher acquisition level by the Acquisition Review Board if (1) its importance to DHS's strategic and performance plans is disproportionate to its size; (2) it has high executive visibility; (3) it impacts more than one DHS Component; (4) it has significant program or policy implications; or (5) the Deputy Secretary, Chief Acquisition Officer, or acquisition decision authority recommends an increase to a higher acquisition level. [30] According to TES officials, DHS determined that some T&E documentation would not be required for older programs which were close to the Produce/Deploy/Support phase at the time the acquisition and T&E directives was issued. For example, TSA's Secure Flight program was not required to develop a TEMP because it was approved for the Produce/Deploy/Support phase in September 2009, 4 months after the T&E directive was issued. Secure Flight allows TSA to assume from air carriers the function of watch list matching. [31] A T&E working group is designed to discuss all aspects of a program's T&E efforts. The membership of the working group is to include all stakeholders and appropriate subject matter experts. The working group is to be established as early as possible and is the responsibility of the Program Manager. [32] The program's test director halted operational testing at the recommendation of the operational test agent 3 weeks after it commenced because there were not enough users involved and the tests were not producing sufficient data for analysis. As a result, changes were made in the system architecture. The operational test plan is being updated and resubmitted to TES for approval prior to resuming operational testing. [33] TSA was granted permission to deploy up to 700 machines at this ARB meeting. TSA plans to deploy up to a total of 2,325 machines, which would represent full operating capability, by fiscal year 2014. [34] Atlas Tactical Communications program made procurements for subprojects which the ARB approved in March 2010 without requiring acquisition or T&E documentation, such as test plans. According to Acquisition Program Management Division officials, the ARB made the decision due to the low dollar threshold of the project and the need to use $20 million in American Recovery and Reinvestment Act funds within a certain time frame. See Pub. L. No. 111-5, 123 Stat. 115 (2009). In March 2011, the ARB determined that Atlas Tactical Communications and Infrastructure programs should merge into one program for fiscal year 2012 and that the program complete acquisition documentation for an acquisition decision event 2B decision scheduled for August 2011. [35] In January 2011, the SBInet program ended as originally conceived; however, limited deployments of technology, including 15 sensor towers and 10 communication towers, remained deployed and operational in Arizona. These deployments are to be subject to further T&E, as DHS develops its new plan for border technology. [36] The memo raised concerns about the Passenger Screening Program and the Electronic Baggage Screening Program using the TSA's Office of Security Technology (OST) as their operational test agent, since OST was not organizationally independent of either program office. [37] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [38] [hyperlink, http://www.gao.gov/products/GAO-09-29]. [39] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [40] The ASP program is an effort by DHS to develop, procure, and deploy a successor to existing radiation detection portals. Radiation detection portals, also known as radiation portal monitors, are designed to detect the emission of radiation from objects that pass by them. The current portals are generally deployed at the U.S. land and sea borders by DHS's Domestic Nuclear Detection Office (DNDO) and operated by DHS's Customs and Border Protection (CBP). [41] GAO, Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results, [hyperlink, http://www.gao.gov/products/GAO-09-979] (Washington, D.C.: Sept. 30, 2008). [42] Congressional Research Service, The Advanced Spectroscopic Portal Program: Background and Issues for Congress, RL 34750 (Washington, D.C.: Dec. 30, 2010). [43] GAO, Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Portal Monitors, but Preliminary Results Show Limits of the New Technology, [hyperlink, http://www.gao.gov/products/GAO-09-655] (Washington, D.C.: May 21, 2009). [44] In 2007, we reported that DNDO's tests of ASP were not an objective and rigorous assessment of the ASP's capabilities. Specifically, we reported that tests conducted by DNDO at the Nevada Test Site from February to March 2007 used biased test methods and were not an objective assessment of the ASP's performance capabilities. Further, DNDO's test methods--specifically, conducting dry runs and dress rehearsals with contractors prior to formal testing--enhanced the performance of the ASPs beyond what they are likely to achieve in actual use. [45] [hyperlink, http://www.gao.gov/products/GAO-10-588SP]. [46] GAO, Secure Border Initiative: DHS Needs to Reconsider Its Proposed Investment in Key Technology Program, [hyperlink, http://www.gao.gov/products/GAO-10-340] (Washington, D.C.: May 2010). [47] [hyperlink, http://www.gao.gov/products/GAO-09-655]. [48] [hyperlink, http://www.gao.gov/products/GAO-10-128]. [49] Hearing before the U.S. House of Representatives, Committee on Appropriations, Subcommittee on Homeland Security, Mar. 30, 2011. [50] The issue of developing requirements is a challenge that spans across DHS's components. However, we did not identify or assess all efforts within each DHS component to help address this challenge. [51] [hyperlink, http://www.gao.gov/products/GAO-11-278]. [52] In January 2011, the DHS Under Secretary for Management submitted an updated Integrated Strategy for High Risk Management to GAO which outlines DHS's response to address our long-standing recommendations to improve departmental management and operations. [53] GAO, Coast Guard: Opportunities Exist to Further Improve Acquisition Management Capabilities, [hyperlink, http://www.gao.gov/products/GAO-11-480] (Washington, D.C.: Apr. 13, 2011). [54] Acquisition Workforce Policy 064-04-004, Acquisition Certification Requirements for DHS T&E Managers. [55] [hyperlink, http://www.gao.gov/products/GAO-10-588SP] and [hyperlink, http://www.gao.gov/products/GAO-11-318SP]. [56] The T&E directive states that where practicable without compromising either the developmental or operational test objectives, developmental and operational testing should be accomplished in an integrated fashion to conserve resources. Further, the directive does not specify under what conditions integrated testing is to take place and does not specify that IT programs should undergo different T&E than non-IT programs. [57] The three programs are: U.S. Secret Service's Integrated Information and Transformation (IIT), National Protection and Programs Directorate's National Cybersecurity Protection System (NCPS), and U.S. Citizenship and Immigration Services' Verification Information System (VIS). [58] [hyperlink, http://www.gao.gov/products/GAO-10-158]. [59] GAO, Test and Evaluation: Impact of DOD’s Office of the Director of Operational Test and Evaluation, [hyperlink, http://www.gao.gov/products/GAO/NSIAD-98-22] (Washington, D.C.: Oct. 24, 1997). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: