Border Security: DHS Should Improve the Quality of Unlawful Border Entry Information and Other Metric Reporting
Fast Facts
The Department of Homeland Security is required by law to report annually on 43 specific measures of border security effectiveness.
DHS reported on 35 of those metrics in its 2017 report and generally used data and methods that should produce reliable results.
However the report disclosed some, but not all, of the data's limitations. For example, the report didn't indicate that different locations may have recorded some border-related data differently. Complete information about the data would provide context for Congress and the public to fully and appropriately understand the report.
We recommended ways to help DHS improve report quality.
Average wait time at land ports of entry is one of the 43 metrics.
Photo of vehicles lined up at a border crossing.
Highlights
What GAO Found
The Department of Homeland Security (DHS) reported on 35 of 43 metrics called for by the National Defense Authorization Act (NDAA) for Fiscal Year 2017 (see figure); it generally used quality information, but did not identify some data limitations. GAO found that about half of the 35 metrics generally included elements as called for by the NDAA, while 17 metrics differed, such as in scope or calculation. For example, DHS only provided information on the southwest border for some metrics, such as the estimate of undetected unlawful border crossers for which a methodology for estimating unlawful crossings for the northern border had not yet been completed. DHS components responsible for collecting the metric data generally have processes in place to ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations for some, but not all, of the data elements and metrics used. For example, GAO found that DHS did not disclose limitations on data related to apprehensions of individuals that were assisted by unmanned aerial systems. By developing and implementing a process to systematically review the reliability of the data and comprehensively identify and communicate limitations, DHS would improve the quality of the information provided.
Number of Metrics Included in the Department of Homeland Security's (DHS) Fiscal Year 2017 Border Security Metrics Report
DHS used a statistical model to estimate three metrics on unlawful border entries but did not validate some assumptions the model employs through sensitivity analyses and provide measures of statistical uncertainty in accordance with standards for federal agencies. For example, DHS's model assumes that 100 percent of families unlawfully crossing the border will be apprehended, but DHS did not provide information on the extent to which the assumption affected its metrics. DHS also did not provide information on the level of statistical uncertainty for the metrics, such as margins of error. Providing such information would allow Congress and the public to better understand the potential limitations and accuracy of these metrics of unlawful entry. Additionally, DHS's statistical model, which is based on Mexican adults not seeking asylum, represents a small and declining share of those apprehended at the border and DHS is developing a new model to account for current border conditions.
Why GAO Did This Study
According to DHS, the United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected and processed for entry. Securing U.S. border areas is a key part of DHS's mission, and the department's ability to measure its border security efforts is essential for it to manage its responsibilities effectively and efficiently.
The NDAA for Fiscal Year 2017 requires DHS to report annually on 43 border security metrics. DHS issued its first report in May 2018. The Act also includes a provision for GAO, within 270 days of receipt and biennially for the following 10 years, to review and report on the data and methodology contained in DHS's report. This report assesses the extent to which DHS: (1) reported metrics as outlined in the NDAA using quality information; and (2) validated assumptions and conveyed statistical uncertainty for unlawful entry metrics, among other objectives. GAO assessed the methodology and data in DHS's report, analyzed DHS's use of statistical models, and interviewed officials from DHS offices and components involved in developing the metrics.
Recommendations
GAO is making four recommendations, including that DHS develop and implement a process to systematically review the reliability of metric data, identify and communicate limitations of the metrics, and include the results of sensitivity analyses and measures of statistical uncertainty for metrics derived from statistical models. DHS concurred with the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | The Secretary of Homeland Security should develop and implement a process to systematically review the reliability of the data used in its Border Security Metrics Report and comprehensively identify any limitations with the data and methodologies that underlie its metrics. (Recommendation 1) |
DHS concurred with the recommendation, and in July 2021, DHS officials told us they intend to take steps to implement it when they have available resources. In July 2022, DHS officials stated that they did not yet have sufficient resources to implement the recommendation. In July 2023, DHS officials stated that they have ongoing conversations to examine their processes and determine how they can systematically review the reliability of the operational data. As of July 2024, DHS reported that it was in the process of completing its 2023 Border Security Metrics Report. When DHS finalizes the report, we will assess any steps DHS took to review the reliability of the data used in the report and identify any limitations with the data and methodologies that underlie its metrics.
|
| Department of Homeland Security | The Secretary of Homeland Security should ensure the communication of the limitations of the metrics identified through the systematic review in the department's annual Border Security Metrics Report. (Recommendation 2) |
In its comment letter, DHS concurred with the recommendation and requested that we consider it implemented because the department had detailed some of the limitations in its fiscal year 2017 Border Security Metrics Report and planned to continue to identify known limitations and the progress made to mitigate previously identified limitations in future reports. In its fiscal year 2019 report, DHS communicated additional data limitations. However, in that report, DHS did not disclose or clearly describe some or all of the limitations for 17 of the 21 metrics we identified in our March 2019 review. In July 2022, DHS officials stated that DHS had identified all known limitations in the 2021 Border Security Metrics Report. However, the officials also said in July 2022 that DHS had not yet implemented a process to systematically review the reliability of the data used in the reports. In July 2023, OIS officials stated that they have ongoing conversations to examine their processes and determine how they can systematically review the reliability of the operational data. As of July 2024, DHS reported that it was in the process of completing its 2023 Border Security Metrics Report. When DHS finalizes the report, we will assess the extent to which it communicated limitations of the metrics it identified by systematically reviewing the reliability of the data used in the report.
|
| Department of Homeland Security | The Under Secretary for the Office of Strategy, Policy, and Plans should include the results of sensitivity analyses to key assumptions in its statistical models of unlawful entry estimates in its annual Border Security Metrics Report. (Recommendation 3) |
On August 5, 2020, DHS issued its fiscal year 2019 Border Security Metrics Report. In the report, DHS included a sensitivity analysis that determined how three key assumptions of its statistical model affected the precision of the model-based apprehension rate of attempted unlawful border crossers and the total estimated number of unlawful entries in fiscal year 2018. By including the results of its sensitivity analysis in the fiscal year 2019 report, DHS met the intent of our recommendation. This sensitivity analysis will allow Congress, policymakers, and the public to better understand potential limitations of DHS's statistical model and more fully evaluate the validity of the border security metrics derived from the model.
|
| Department of Homeland Security | The Under Secretary for the Office of Strategy, Policy, and Plans should include measures of statistical uncertainty for all metrics based on estimates derived from statistical models in its annual Border Security Metrics Report. (Recommendation 4) |
DHS concurred with the recommendation, and in the fiscal year 2019 Border Security Metrics Report, DHS took steps toward implementing it by conveying statistical uncertainty for one metric based on estimates derived from its statistical model. In particular, DHS outlined how this uncertainty might affect the model-based apprehension rate in a report appendix. According to OIS officials, this uncertainty reflected sampling and prediction error. Further, in April 2022, DHS issued its fiscal year 2021 Border Security Metrics Report. In the report, DHS conveyed statistical uncertainty for all three metrics it reported based on estimates derived from its statistical model, including the model-based apprehension rate, the estimated number of undetected unlawful entries, and the probability of detection rate. Specifically, DHS noted the confidence interval level it used and displayed upper and lower bounds for the estimates. By including measures of statistical uncertainty for all metrics that rely on its statistical model in its fiscal year 2021 report, DHS met the intent of our recommendation. This additional information on statistical uncertainty will allow Congress, policy makers, and the public to more fully evaluate the validity of the border security metrics based on estimates derived from its statistical model.
|