Data Mining:

DHS Needs to Improve Executive Oversight of Systems Supporting Counterterrorism

GAO-11-742: Published: Sep 7, 2011. Publicly Released: Oct 7, 2011.

Additional Materials:

Contact:

David A. Powner
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Data mining--a technique for extracting useful information from large volumes of data--is one type of analysis that the Department of Homeland Security (DHS) uses to help detect and prevent terrorist threats. While data-mining systems offer a number of promising benefits, their use also raises privacy concerns. GAO was asked to (1) assess DHS policies for evaluating the effectiveness and privacy protections of data-mining systems used for counterterrorism, (2) assess DHS agencies' efforts to evaluate the effectiveness and privacy protections of their data-mining systems, and (3) describe the challenges facing DHS in implementing an effective evaluation framework. To do so, GAO developed a systematic evaluation framework based on recommendations and best practices outlined by the National Research Council, industry practices, and prior GAO reports. GAO compared its evaluation framework to DHS's and three component agencies' policies and to six systems' practices, and interviewed agency officials about gaps in their evaluations and challenges..

As part of a systematic evaluation framework, agency policies should ensure organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency throughout the system's life cycle. While DHS and three of its component agencies--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and the U.S. Citizenship and Immigration Services--have established policies that address most of these key policy elements, the policies are not comprehensive. For example, DHS policies do not fully ensure executive review and transparency, and the component agencies' policies do not sufficiently require evaluating system effectiveness. DHS's Chief Information Officer reported that the agency is planning to improve its executive review process by conducting more intensive reviews of IT investments, including the data-mining systems reviewed in this report. Until such reforms are in place, DHS and its component agencies may not be able to ensure that critical data mining systems used in support of counterterrorism are both effective and that they protect personal privacy. Another aspect of a systematic evaluation framework involves ensuring that agencies implement sound practices for organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency and oversight throughout a system's life cycle. Evaluations of six data mining systems from a mix of DHS component agencies showed that all six program offices took steps to evaluate their system's effectiveness and privacy protections. However, none performed all of the key activities associated with an effective evaluation framework. For example, four of the program offices executed most of the activities for evaluating program privacy impacts, but only one program office performed most of the activities related to obtaining executive review and approval. By not consistently performing necessary evaluations and reviews of these systems, DHS and its component agencies risk developing and acquiring systems that do not effectively support their agencies' missions and do not adequately ensure the protection of privacy-related information. DHS faces key challenges in implementing a framework to ensure systems are effective and provide privacy protections. These include reviewing and overseeing systems once they are in operation, stabilizing and implementing acquisition policies throughout the department, and ensuring that privacy-sensitive systems have timely and up-to-date privacy reviews. The shortfalls GAO noted in agency policies and practices provide insight into these challenges. Until DHS addresses these challenges, it will be limited in its ability to ensure that its systems have been adequately reviewed, are operating as intended, and are appropriately protecting individual privacy and assuring transparency to the public. GAO is recommending that DHS executives address gaps in agency evaluation policies and that component agency officials address shortfalls in their system evaluations. DHS concurred with GAO's recommendations and identified steps it is taking to address selected recommendations. The department also offered technical comments, which GAO incorporated as appropriate.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to investigate whether the information sharing component of U.S. Immigration and Customs Enforcement (ICE) Pattern Analysis and Information Collection (ICEPIC), called the Law Enforcement Information Sharing Service, should be deactivated until a PIA that includes this component is approved.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: The DHS Privacy Office began a privacy compliance review of the ICEPIC program in September 2011. While the review was ongoing, the Chief Privacy Officer reviewed, approved, and in October 2011, published an updated privacy impact assessment of the program that included the Law Enforcement Information Sharing service. In the results of the privacy compliance review, published in December 2011, the office noted that by directing the expedited preparation and review of the PIA update, the DHS Privacy Office was able to bring the system into compliance with the E-Government Act and DHS policy (which obviated the need to shut down the program).

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to develop requirements for providing additional scrutiny of privacy protections for the sensitive information systems that are not transparent to the public through privacy impact assessments (PIAs).

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: In order to provide for additional scrutiny of the privacy protections for the sensitive information systems that do not have publicly-available privacy impact assessments, DHS noted in its agency comments letter that the DHS Privacy Office planned to include an annex on unreleased privacy impact assessments in its Annual Report to Congress that was marked and handled with the appropriate national security (or other sensitive, but unclassified) restrictions. The agency further noted that members of Congress could request the relevant documents or schedule a briefing with the agency after reviewing the annex. Subsequently, the Privacy Office included this annex in its recent report to Congress. By completing this activity, the DHS Privacy Office can better assure it is maintaining appropriate transparency for its systems and of its actions.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to identify steps to mitigate challenges related to the review and oversight of operational systems and to DHS's changing policy requirements and determine clear corrective actions, taking the impact on components and on individual program managers into account.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (OCIO) stated that it has taken several steps to improve the review and oversight of operational systems. For example, the OCIO reported that it is currently revising its Operational Analysis guidance revision, to incorporate component comments on the draft guidance, and expected the guidance to be approved by September 30, 2013. Once approved and implemented, the guidance indicates that operational assessments of steady state systems (i.e., systems that have been deployed and are in use) will be submitted as part of the systems' budgeting process. The OCIO also indicated it planned to identify effective methods for communicating policy changes to components in a timely manner to mitigate the challenges components experience in complying with DHS acquisition and governance policies. We will continue to monitor the department's actions in these areas.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to ensure the consistency of component agencies' policies with DHS policies and proposed improvements to those policies, including requiring data quality assessments, requiring re-evaluations of operational systems, and establishing investment review boards with clearly defined structures for system review.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (OCIO) reported that it is continuing to establish its process for reviewing component agencies' IT directives and policies to ensure alignment with those issued by the department. The office further noted that it has identified the component agencies' existing policies it needs to review to evaluate the policies' consistency with OCIO corporate policy. If the IT policies are inconsistent, the department plans to work with the component counterparts to resolve the discrepancy. Further, the OCIO stated that the office has drafted a Management and Integration Directive to ensure component policies are aligned with DHS corporate IT policies. The draft directive would require component CIOs to comply with all DHS IT policies, processes, standards, guidelines, and procedures and ensure all component IT policies are in compliance and alignment with DHS CIO directives and instructions. We will continue to monitor the agency's progress in finalizing and implementing the directive to address this recommendation.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the the appropriate component agency administrators to ensure that the system program offices for Analytical Framework for Intelligence (AFI), Automated Targeting System (ATS)-Passenger (ATS-P), Citizen and Immigration Data Repository (CIDR), Data Analysis and Research for Trade Transparency System (DARTTS), ICEPIC, and TECS Modernization (TECS-Mod) address the shortfalls in evaluating system effectiveness and privacy protections identified in this report, including shortfalls in applying acquisition practices, ensuring executive review and approval, and consistently documenting executive reviews.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: While we received updates on several of the systems we reviewed at the Department of Homeland Security (DHS) component agencies, none of the agencies have provided artifacts that demonstrate that they have addressed all the shortfalls we identified. We will continue to monitor DHS's activities in this area.

    Apr 7, 2014

    Mar 31, 2014

    Mar 28, 2014

    Mar 26, 2014

    Mar 12, 2014

    Mar 7, 2014

    Feb 27, 2014

    Feb 13, 2014

    Looking for more? Browse all our products here