This is the accessible text file for GAO report number GAO-11-375R entitled 'Aviation Security: TSA's Revised Cost Comparison Provides a More Reasonable Basis for Comparing the Costs of Private-Sector and TSA Screeners' which was released on March 9, 2011. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO -11-375R: United States Government Accountability Office: GAO: March 4, 2011: The Honorable John L. Mica: Chairman: Committee on Transportation and Infrastructure: House of Representatives: The Honorable Daniel E. Lungren: Chairman: Subcommittee on Cybersecurity, Infrastructure Protection, and Security Technologies: Committee on Homeland Security: House of Representatives: Subject: Aviation Security: TSA's Revised Cost Comparison Provides a More Reasonable Basis for Comparing the Costs of Private-Sector and TSA Screeners: As you requested, this letter discusses the Transportation Security Administration's (TSA) efforts to address our January 2009 recommendation to TSA to improve its methodology for comparing the cost and performance of screening services at commercial airports using private-sector screeners under the Screening Partnership Program (SPP airports) and at airports using federal screeners (non-SPP airports).[Footnote 1] Our January 2009 report identified some design strengths in TSA's analysis comparing the cost and performance of screening services at SPP and non-SPP airports, such as recognizing that cost savings would be limited by the mandated structure of the program, but also identified 10 limitations in TSA's methodology that could affect the accuracy and reliability of the cost and performance comparisons, and its usefulness in informing future management decisions. We recommended that if TSA plans to rely on its comparison of cost and performance of SPP and non-SPP airports for future decision making, the agency should update its analysis to address the limitations we identified. TSA generally concurred with our findings and recommendation. In summary, on January 4, 2011, TSA provided us with an update on the status of its efforts to address seven limitations related to cost we cited in our report, as well as a revised comparison of costs for screeners at SPP and non-SPP airports. This revised cost comparison generally addresses three of the seven limitations and provides TSA with a more reasonable basis for comparing the cost of SPP and non-SPP airports. However, TSA needs to take additional actions or provide additional documentation to address the remaining four limitations related to cost and the three limitations related to performance that we identified in our January 2009 report (see enclosure). In addition, TSA reported that it does not plan to rely solely on its cost and performance study for future management decisions related to the SPP. As requested, this letter focuses on the status of TSA's efforts related to its revised cost comparison of SPP and non-SPP airports. Enacted in response to the September 11, 2001, terrorist attacks, the Aviation and Transportation Security Act (ATSA) established TSA and charged it with responsibility for securing all modes of transportation, including civil aviation.[Footnote 2] Prior to ATSA and the establishment of TSA, passenger and baggage screening had been performed by private screening companies under contract to airlines. ATSA required TSA to create a federal workforce to assume the job of conducting passenger and checked-baggage screening at the nation's commercial airports.[Footnote 3] In accordance with ATSA, TSA also created the SPP to allow commercial airports an opportunity to apply to TSA to use private-sector screeners.[Footnote 4] These private- sector screeners are employed by TSA-approved private-sector screening contractors. TSA funds all passenger and baggage screening activities at SPP airports as well as airports with federal screeners. Federal screeners are employed by TSA and are referred to as transportation security officers. The observations in this letter are based on a review of TSA's updated January 4, 2011, cost analysis as well as discussions with agency officials. We compared TSA's updated study methodology as described by TSA officials to criteria in federal accounting standards for managerial cost accounting, GAO and Office of Management and Budget (OMB) guidance for estimating and managing program costs, and GAO's Standards for Internal Control in the Federal Government.[Footnote 5] We did not independently verify TSA's analysis for the revised cost comparisons. We performed this work from September 2010 through February 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and complete work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our review objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions. We reported in January 2009, among other things, that TSA had underestimated costs to the government for screeners at non-SPP airports because the agency did not include all of the costs associated with passenger and baggage screening services at these airports, such as workers' compensation and general liability insurance, and certain retirement benefits to be paid by the Office of Personnel Management to TSA retirees at non-SPP airports. Further, TSA did not reflect the revenue received by the government from corporate income taxes paid by SPP contractors. The omission of these factors reduced the reliability of TSA's 2009 cost estimate by increasing the costs for private-contractor screeners relative to federal screeners. According to Statement of Federal Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards, the full cost of programs is to be reported to assist Congress and executives in making informed decisions on program resources and to ensure that programs get expected and efficient results. We believe TSA has made progress in addressing three of seven limitations related to cost we identified in our January 2009 report and now has a more reasonable basis for comparing the cost of SPP and non-SPP airports. Specifically, TSA's updated cost analysis includes: * the potential cost impact of overlapping administrative personnel; [Footnote 6] * costs associated with passenger and baggage screening services at non-SPP airports, such as workers' compensation, general liability insurance, certain retirement costs, and a corporate tax adjustment to reflect revenue received by the government from corporate income taxes paid by SPP contractors; and: * cost comparisons for multiple fiscal years. In all, these adjustments narrowed the reported cost differences between SPP and non-SPP airports. Specifically, in 2007, TSA estimated that SPP airports would cost about 17 percent more to operate than airports using federal screeners. In its January 4, 2011, update, TSA estimated that SPP airports would cost 3 percent more to operate in 2011 than airports using federal screeners. TSA has also taken actions that partially address the four remaining limitations related to cost, but needs to do more to fully address them. These actions and remaining steps are discussed in greater detail in the enclosure. TSA reports that it is working with the Department of Homeland Security (DHS) to further refine its estimate, which will provide important context for any future discussions regarding the use and relative costs of screeners at SPP and non-SPP airports.[Footnote 7] We continue to encourage the agency to seek assistance from DHS's Cost Analysis Division as it considers any additional technical refinements to its SPP cost analysis. We will continue to follow up on the limitations related to screener performance that we cited in our January 2009 report and report on them as appropriate. Among other things, our report raised questions about the performance measures used to judge screener performance and compare the performance of screeners at SPP and non-SPP airports. For example, TSA's study design did not include confidence levels related to screening performance or account for factors, such as airport configuration or size, that could contribute to differences in performance between SPP and non-SPP airports. We requested comments on a draft of this letter from the Administrator of TSA. The agency did not provide written comments. However, in e- mails received on February 14 and 17, 2011, the TSA liaison stated that TSA has taken several steps to address the cost and performance limitations noted in our 2009 report and provided additional information. For example, the liaison stated that TSA has introduced several additional performance metrics to address the performance limitations noted in our report. These additional metrics are designed to help mitigate threats, assess employee performance and staffing, and ensure adherence to TSA headquarters programs. We have updated the draft enclosure to reflect the information the TSA liaison provided by modifying our assessment of TSA's actions to address the first performance limitation discussed in the enclosure. We also updated the enclosure based on additional information the liaison provided related to TSA's actions to address the sixth cost limitation and all performance limitations. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 5 days from the date of this report. At that time we will send copies to relevant House and Senate committees and subcommittees, the Secretary of the Department of Homeland Security, the TSA Administrator, and other interested parties. In addition, this report will be available at no charge on GAO's Web site at [hyperlink, http://gao.gov]. If you or your staff have any questions concerning this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Individuals making key contributions to this report were Glenn Davis, Assistant Director; Daren Sweeney, Analyst-in-Charge; Chuck Bausell; Wendy Dye; Stan Kostyla; Thomas Lombardi; Lara Miklozek; John Warner; and Yee Wong. Signed by: Stephen M. Lord, Director: Homeland Security and Justice Issues: Enclosure: [End of section] Enclosure: Status of TSA's Efforts to Address SPP Cost and Performance Limitations: Table 1 below provides information on the 10 limitations we cited in our January 2009 report regarding TSA's methodology for comparing the cost and performance of screening services at commercial airports using private-sector screeners under the Screening Partnership Program (SPP airports) and at airports using federal screeners (non-SPP airports).[Footnote 8] Table 1 also provides information on the status of TSA's actions to address these limitations. Specifically, TSA has generally addressed 3 of 4 cost limitations but needs to take additional actions or provide additional documentation to address the remaining 4 limitations related to cost and the 3 limitations related to performance that we identified in our January 2009 report. Table 1: Status of TSA's Efforts to Address SPP Cost and Performance Limitations: Cost limitations: 1. TSA's study design did not consider the impact of overlapping administrative personnel on the costs of SPP airports; Status: Generally addressed; Description of actions taken: TSA adjusted its cost estimates to more accurately account for overlapping contractor management and support staff. Cost limitations: 2. TSA underestimated costs to the government for non-SPP airports by not including all costs associated with providing passenger and baggage screening services; Status: Generally addressed; Description of actions taken: TSA adjusted its cost estimates to include indirect costs, such as workers' compensation, liability insurance, certain retirement costs, and revenue received by the government from corporate income taxes paid by SPP contractors. Cost limitations: 3. TSA's analysis included more uncertainty in the cost estimates for non-SPP airports than for SPP airports; Status: Partially addressed; Description of actions taken: TSA officials stated that the DHS Cost Analysis Division offered to help TSA develop a confidence interval to better identify the level of uncertainty of its cost estimates. We will continue to monitor TSA's efforts to develop additional documentation addressing this limitation. Cost limitations: 4. TSA's design did not include an analysis to determine how changes in underlying assumptions would affect the size of the estimate, and its cost comparison did not account for differences in screening performance or analyze the costs associated with a particular level of performance; Status: Partially addressed; Description of actions taken: TSA reported that it completed cost studies to reflect how changes in different assumptions affect the size of the estimate. While TSA officials noted challenges associated with estimating costs related to performance, TSA expressed interest in continuing to work with us to address this limitation. We will continue to monitor TSA's efforts to develop additional documentation addressing this limitation. Cost limitations: 5. TSA's comparison of SPP and non-SPP airports' costs was based on data from a single fiscal year: 2007. However, it is unclear how representative the costs for a single year may be; Status: Generally addressed; Description of actions taken: TSA officials revised its cost comparison to include data for multiple fiscal years. Cost limitations: 6. TSA did not ensure that cost data collected were reliable and did not prepare documentation of its costing methodology called for in federal accounting standards; Status: Partially addressed; Description of actions taken: TSA officials provided us with additional information and briefings on its methodology and conclusions, and reported that its estimates were reviewed by an independent group in DHS. However, TSA did not provide sufficient evidence related to data reliability and its costing methodology. Federal accounting standards require a reference guide documenting instructions for managerial cost accounting procedures and practices.[A] This documentation helps to ensure management's approval of the methodology and its consistent application over time, and provides a basis for review of the methodology by agency officials and independent evaluators as well as a tool for training. We will continue to monitor TSA's efforts to develop its guidance for addressing this limitation. Cost limitations: 7. Key assumptions and methods used were also not documented in sufficient detail to justify the reasonableness of costs; Status: Partially addressed; Description of actions taken: TSA officials provided us with additional information on its assumptions and methodology for its original cost estimate. However, TSA did not provide sufficient documentation to justify the reasonableness of key assumptions and costing methods. We will continue monitoring TSA's efforts to identify and justify the assumptions and methods used in its cost estimates. Performance limitations: 1. TSA's study design provided no evidence showing that the five performance measures are the most meaningful indicators of performance or that they provide a comprehensive basis for comparing the performance of SPP and non-SPP airports; Status: Partially addressed; Description of actions taken: TSA collects multiple performance measures which may be used to measure and compare performance of screeners between SPP and non-SPP airports, and reported implementing additional metrics since the completion of our 2009 study. However, we believe that the measures selected to compare performance should be those most closely linked to differences between SPP and non-SPP airports--specifically, differences that reflect greater management flexibilities that may exist at SPP airports. For example, TSA stated that it uses Threat Image Projection (TIP) scores to measure security effectiveness at airports. However, it is unclear the degree to which differences in TIP scores at SPP and non-SPP airports would be attributable to the SPP. Thus, we will continue monitoring TSA's efforts to select measures that capture differences in performance between SPP and non-SPP airports. Performance limitations: 2. In comparing the performance of SPP and non-SPP airports, TSA's design did not control or otherwise account for other possible factors such as airport configuration or size that could contribute to performance differences between SPP and non-SPP airports; Status: Generally not addressed; Description of actions taken: TSA officials stated that they manage SPP airports with the same standards and performance metrics as non- SPP airports. TSA also stated that one of the performance measures, Threat Image Projection (TIP), is not affected by airport configuration or size. However, TSA did not provide any analysis that observed performance differences between SPP and non-SPP airports may reflect factors unrelated to the Screening Partnership Program, such as the possibility that SPP airports may be different than non-SPP airports in configuration or size. We will continue to monitor TSA's efforts to develop additional documentation addressing this limitation. Performance limitations: 3. TSA's study design did not provide any confidence levels for estimates related to screening performance, consistent with generally accepted statistical practices. The study also did not address the procedures needed to ensure data reliability; Status: Generally not addressed; Description of actions taken: TSA has not provided confidence levels for estimates related to screening performance or addressed the procedures needed to ensure data reliability. TSA noted that one of its performance measures uses an algorithm that promotes data reliability. However, TSA's example does not directly address the limitation--that TSA needs to develop confidence intervals on its estimates of performance at SPP and non-SPP airports and also ensure the reliability of all data related to screening performance. We will continue to monitor TSA's efforts to develop additional documentation addressing this limitation. Source: GAO analysis of TSA's updated responses on steps taken to address limitations. [A] Federal Accounting Standards Advisory Board: Statement of Federal Accounting Standards No. 4, Managerial Cost Accounting Standards and Concepts (June 30, 2009). Legend: Generally addressed - TSA has taken key actions to address the design limitation noted in our prior report, GAO-09-27R, Airport Screening (January 2009). Partially addressed - TSA has taken some key actions to address the design limitation noted in our prior report, GAO-09-27R, Airport Screening (January 2009), but TSA needs to complete additional steps or provide additional documentation to fully address the design limitations. Generally not addressed - TSA has taken little or no significant actions to address the design limitation noted in our prior report, GAO-09-27R, Airport Screening (January 2009). [End of table] [End of section] Footnotes: [1] GAO, Aviation Security: TSA's Cost and Performance Study of Private-Sector Airport Screening, [hyperlink, http://www.gao.gov/products/GAO-09-27R] (Washington, D.C.: Jan. 9, 2009). [2] See Pub. L. No. 107-71, 115 Stat. 597 (2001). [3] See Pub. L. No. 107-71, § 110, 115 Stat. at 614-16. [4] See Pub. L. No. 107-71, § 108, 115 Stat. at 611-13 (codified as amended at 49 U.S.C. §§ 44919-20). TSA established the SPP after concluding a 2-year pilot program through which private screening companies performed screening operations at five commercial airports. [5] Statement of Federal Financial Accounting Standards No. 4, Managerial Cost Accounting Concepts and Standards; GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, [hyperlink, http://www.gao.gov/products/GAO-07-1134SP] (Washington, D.C.: July 2007); OMB Circular A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs; GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO-AIMD-00-21.3-1] (Washington, D.C.: November 1999); and OMB Circular A-76, Performance of Commercial Activities. [6] For example, TSA stated that it refined its Federal Security Director (FSD) staffing allocation models to more accurately account for contractor management of the workforce, to adjust FSD staffing to reflect the needs of airports, and to reduce redundancies. The Staffing Allocation Model seeks, within certain TSA constraints, to estimate the most efficient balance of screeners needed to ensure security and minimize wait times. See GAO, Aviation Security: TSA's Staffing Allocation Model Is Useful for Allocating Staff among Airports, but Its Assumptions Should Be Systematically Reassessed, [hyperlink, http://www.gao.gov/products/GAO-07-299] (Washington, D.C.: Feb. 28, 2007). [7] As noted in OMB Circular A-94, estimates of costs are typically uncertain because of imprecision in both underlying data and modeling assumptions. Because such uncertainty is basic to many estimates, its effects should be analyzed and reported. Useful information in such a report would include the key sources of uncertainty; expected value estimates of outcomes; and the sensitivity of results to important sources of uncertainty. [8] GAO, Aviation Security: TSA's Cost and Performance Study of Private-Sector Airport Screening, [hyperlink, http://www.gao.gov/products/GAO-09-27R] (Washington, D.C.: Jan. 9, 2009). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: