This is the accessible text file for GAO report number GAO-11-705R entitled 'Federal Protective Service: Actions Needed to Resolve Delays and Inadequate Oversight Issues with FPS's Risk Assessment and Management Program' which was released on August 15, 2011. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-11-705R: United States Government Accountability Office: Washington, DC 20548: July 15, 2011: The Honorable Bennie G. Thompson: Ranking Member: Committee on Homeland Security: House of Representatives: Subject: Federal Protective Service: Actions Needed to Resolve Delays and Inadequate Oversight Issues with FPS's Risk Assessment and Management Program: Dear Mr. Thompson: The Federal Protective Service (FPS), which is within the Department of Homeland Security's (DHS) National Protection and Programs Directorate (NPPD), is responsible for protecting the more than 1 million federal employees and members of the public who work in and visit the over 9,000 federal facilities owned or leased by the General Services Administration (GSA) from a potential terrorist attack or other acts of violence.[Footnote 1] To accomplish its facility protection mission, FPS has about 1,200 full-time employees and approximately 13,200 contract security guards. FPS has an annual budget of about $1 billion and receives its funding from the revenues and collections of security fees charged to tenant agencies for protective services such as facility security assessments (FSA) and providing contract security guard services. Since 2008, we have issued numerous reports that address major challenges FPS faces in protecting federal facilities. For example, in 2009 and 2010 we reported that FPS had problems completing high-quality FSAs in a timely manner and did not provide adequate oversight of its contract guard program.[Footnote 2] In September 2007, FPS decided to address the challenges with its legacy security assessment and guard management systems with a new system. On August 1, 2008, DHS's Immigration and Customs Enforcement (ICE) competitively awarded and FPS funded a $21 million, 7-year contract to develop and maintain the Risk Assessment and Management Program (RAMP) system.[Footnote 3] RAMP is a web-enabled risk assessment and guard management system, and its initial implementation was scheduled for July 31, 2009. Among other things, RAMP is intended to: * provide FPS with the capability to assess risks at federal facilities based on threat, vulnerability, and consequence, and track countermeasures to mitigate those risks; and: * improve the agency's ability to monitor and verify that its contract security guards are trained and certified to be deployed to federal facilities.[Footnote 4] In response to your request that we examine RAMP, this report addresses the following questions: 1. What is RAMP's current status, including whether it can be used as planned? 2. What are the factors that contributed to this status? 3. What are the actions FPS is taking to develop and implement RAMP? Scope and Methodology: To answer these questions, we reviewed documents from FPS and ICE including: RAMP's requirement and project management documents, cost estimates, FPS's risk calculator and template, DHS's security standards such as the National Infrastructure Protection Plan (NIPP) and the Interagency Security Committee's (ISC) Physical Security Criteria for Federal Facilities, and RAMP contract files. We reviewed FPS's and ICE's requirement and project documents to determine whether FPS complied with selected GAO and industry best practices in project management such as: managing changes in requirements and conducting user acceptance testing in developing and implementing RAMP.[Footnote 5] These practices were selected because they are critical in developing information technology systems. To understand how FPS is conducting risk assessments currently, we also reviewed FPS's risk calculator and FSA template. We reviewed the original and follow-on RAMP contracts and contract documentation files to determine if FPS and ICE complied with DHS's acquisition policy and the Federal Acquisition Regulation (FAR). In addition, we interviewed officials at FPS, ICE, DHS, NPPD, GSA; officials from 5 tenant agencies in GSA buildings; the primary RAMP contractor; and 7 of FPS's 37 contract guard vendors. We selected these contractors based on the number of guards they employed and geographic locations. We also visited 2 of FPS's 11 regions and interviewed regional directors, commanders, and inspectors about their use of RAMP and the FSA template and risk calculator, and observed guard post inspections. We selected these regions based on criteria such as: number of federal facilities in the region and their facility security levels, the number of contract guards in the region, and geographic dispersion. Our work is not generalizable to all FPS's regions and guard contractors. We conducted this performance audit from July 2010 through July 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: RAMP is over budget, behind schedule, and cannot be used to complete FSAs and reliable guard inspections as intended. RAMP's contract award amount totals $57 million, almost three times more than the $21 million original development contract amount. As of June 2011, FPS has spent almost $35 million of the $57 million to develop RAMP. RAMP's costs increased in part because FPS changed the original system requirements and the contractor had to add additional resources to accommodate the changes. FPS also has experienced delays in developing and implementing RAMP, as it is almost 2 years behind its original July 2009 implementation date. FPS cannot use RAMP to complete FSAs because the agency did not verify the accuracy of the federal facility data it obtained from GSA or include an edit feature in RAMP that would allow inspectors to edit these data when necessary. FPS is also experiencing difficulty using RAMP to ensure that its approximately 13,200 contract guards have met training and certification requirements to be deployed at federal facilities because it does not have a process for verifying this information before it is entered into RAMP. RAMP also does not yet fully incorporate certain government security standards. For example, according to an FPS official, RAMP does not support the April 2010 ISC Physical Security Criteria for Federal Facilities because FPS did not have time to incorporate it in the June 2010 version of RAMP.[Footnote 6] FPS is planning to incorporate these standards in the next version of RAMP. Several factors have contributed to FPS being unable to use RAMP as planned. Most importantly, FPS and ICE did not adequately follow GAO's project management best practices in developing and implementing RAMP. For example, FPS did not manage requirement changes or conduct user acceptance testing with its inspectors as part of RAMP's development. [Footnote 7] In addition, ICE did not always comply with DHS's acquisition policy and the FAR as we found that contractor performance evaluations were not completed. Contractor performance evaluations are important tools for ensuring that the contractor meets the terms of the contract. FPS is taking some steps to address RAMP's problems. Most notably, FPS has preliminarily decided to discontinue its current RAMP development contract and is considering using a new contractor to finish developing RAMP. FPS is also working to incorporate ISC's Physical Security Criteria for Federal Facilities into RAMP before the next version is implemented. Given the technological changes that may have occurred since FPS began developing RAMP 4 years ago, there could be alternative systems that would better meet FPS's needs. However, FPS has not evaluated whether further developing RAMP is the most cost- beneficial approach compared to possible alternatives. In addition, FPS has not developed a plan to address the problems we found with RAMP, for example, ensuring the accuracy of federal facility and contract guard data. See enclosure I for more information. Conclusions: After almost 4 years of effort and spending almost $35 million, FPS has not accomplished its goals of using RAMP to complete FSAs and reliable guard inspections. Consequently, until FPS resolves RAMP's problems, FPS will not have a comprehensive method of identifying risks to Federal facilities or a reliable method for overseeing its contract guard workforce. While FPS plans to take some actions, if it does not take additional steps to specifically address the problems we found, these problems are likely to continue. It is also crucial that FPS take immediate steps to follow project management best practices in further development of RAMP or any alternative system. Until FPS does so, it risks repeating some of the same mistakes it made during the last 4 years, which have resulted in significant expenditures on a risk assessment and management system that is not functional. Completing the required contractor performance evaluations and ensuring that contract files are maintained in accordance with DHS and FAR requirements is important. For example, completing the required contractor performance evaluations would have provided FPS and ICE officials with the ability to assess the contractor's performance during key phases of RAMP's development and the opportunity to take corrective action if necessary. Maintaining contract files that comply with DHS's acquisition policy and the FAR is also important because the contract files should contain information that explains the basis for key acquisition decisions. FPS's ongoing efforts to protect federal facilities should not be impeded by its decision to finish developing RAMP, particularly since the agency continues to charge GSA and tenant agencies millions of dollars to protect their facilities. Thus, it is important that FPS not only resolve the problems with RAMP but also, while doing so, continue to pursue interim measures to enhance the protection of the over 1 million government employees and members of the public that visit such facilities each year from a potential terrorist attack or other acts of violence. Finally, we agree with FPS that incorporating the ISC's Physical Security Criteria for Federal Facilities into RAMP is important and encourage FPS to continue its efforts to ensure that this happens before the next version of RAMP is rolled out. Recommendations for Executive Action: Given the challenges FPS faced thus far with developing RAMP, technological changes that may have occurred in the last 4 years, and to help guide and ensure the successful development and implementation of any risk assessment and contract guard management system, we recommend that the Secretary of Homeland Security direct the Under Secretary of NPPD and the Director of FPS to take the following four actions: * evaluate whether it is cost-beneficial to finish developing RAMP or if other alternatives for completing FSAs and managing security guards would be more appropriate, * increase the use of project management best practices by managing requirements and conducting user acceptance testing for any future RAMP development efforts, * establish a process for verifying the accuracy of federal facility and guard training and certification data before entering them into RAMP, and: * develop interim solutions for completing FSAs and guard inspections while addressing RAMP's challenges. To improve contract administration, we recommend that the Secretary of Homeland Security direct the Directors of ICE and FPS to complete contract performance evaluations for the current RAMP contractor, and ensure that the evaluations and other required documents are maintained in the contract file in accordance with DHS's acquisition policy and the FAR. Agency Comments and Our Evaluation: We provided a draft of this letter and attached enclosures to DHS for comment. DHS concurred with our recommendations and provided technical comments that we incorporated where appropriate. As agreed upon with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to appropriate congressional committees, the Secretary of Homeland Security, and the Director of the FPS. The report will be available at no charge on GAO's website at [hyperlink, http://www.gao.gov/]. If you or your staff members have any questions about this information, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Additionally, Tammy Conquest, Assistant Director; Greg Hanna; Alicia Loucks; Justin Reed; Amy Rosewarne; Susan Michal-Smith; and Frank Taliaferro made key contributions to this report. Sincerely yours, Signed by: Mark Goldstein: Director, Physical Infrastructure Issues: Enclosures - 4: [End of section] Enclosure I: RAMP Briefing Report: Federal Protective Service: Actions Needed to Resolve Delays and Inadequate Oversight Issues with FPS's Risk Assessment and Management Program: Briefing for the Ranking Member, Committee on Homeland Security, House of Representatives: For more information, contact Mark Goldstein, goldsteinm@gao.gov or 202-512-2834. Overview: * Introduction; * Background; * Objectives; * Summary of Results; * RAMP Is Over Budget, Behind Schedule, and Cannot Be Used to Complete FSAs and Reliable Guard Inspections; * FPS Did Not Follow Some Project Management Best Practices in Developing and Implementing RAMP; * FPS Is Taking Some Steps to Address RAMP's Problems; * Conclusions; * Recommendations for Executive Action; * Agency Comments and Our Evaluation; Introduction: The Federal Protective Service (FPS), which is within the Department of Homeland Security's (DHS) National Protection and Programs Directorate (NPPD), is responsible for protecting the more than 1 million federal employees and members of the public who work in and visit the over 9,000 federal facilities owned or leased by the General Services Administration (GSA) from a potential terrorist attack or other acts of violence'[Footnote 8] To accomplish its facility protection mission, FPS has about 1,200 full-time employees and approximately 13,200 contract security guards. FPS has an annual budget of about $1 billion and receives its funding from the revenues and collections of security fees charged to tenant agencies for protective services such as conducting facility security assessments (FSA) and providing contract guard services. Since 2008, GAO has issued numerous reports that discuss major challenges FPS faces in protecting these facilities. For example, in 2009 we reported that FPS had problems with completing high-quality FSAs in a timely manner and could not comprehensively assess risk across federal facilities.[Footnote 9] We also found in 2010 that FPS lacked adequate oversight of its contract guard program.[Footnote 10] Background: In September 2007, FPS decided to replace its legacy facility security assessment and guard management systems with a new system. On August 1, 2008, DHS's Immigration and Customs Enforcement (ICE) competitively awarded and FPS funded a $21 million, 7-year (1 base year and 6 option years) cost-plus fixed fee contract to develop and maintain the Risk Assessment and Management Program (RAMP) system. RAMP is a web-enabled risk assessment and guard management system and was to, among other things: * provide FPS with the capability to assess risks to federal facilities based on threat, vulnerability, and consequence, and track countermeasures to mitigate those risks; and; * improve FPS's ability to monitor and verify that its approximately 13,200 guards are trained and certified to be deployed to federal facilities.[Footnote 11] According to the original development contract, RAMP was to be designed, developed, and implemented in three phases and completed by July 31, 2011. * Phase 1 would create a system that would enable a user to conduct FSAs that would assess risks, calculate a risk score, and recommend countermeasures for facilities by July 31, 2009. * Phase 2 would add the capability to manage FPS's contract guard workforce, including monitoring whether individual guards were certified by July 31, 2010. * Phase 3 would add more functions to the system, such as providing FPS with the ability to modify imported GSA facility data and assessing risks across FPS's portfolio of federal facilities by July 31, 2011. FPS also developed RAMP to comply with government security standards, such as those outlined by DHS's National Infrastructure Protection Plan (NIPP) and the Interagency Security Committee (ISC), which were not incorporated in FPS's previous risk assessment system. * The NIPP sets forth DHS's coordinated approach to protect the nation's critical infrastructure and key resources to reduce vulnerability, deter threats, and minimize the consequences of attacks and other incidents. * The ISC Physical Security Criteria for Federal Facilities establishes a baseline set of countermeasures to be applied to all federal facilities based on their facility security level, and provides a framework for customizing security countermeasures to address the unique risks faced at each facility. Finally, during RAMP's initial development, FPS was part of ICE. ICE provided software development and project management technical expertise and was responsible for awarding, administering, and overseeing the contract. FPS funded RAMP's development and was responsible for defining RAMP's requirements. Objectives: Our objectives for this briefing are to discuss: (1) RAMP's current status, including whether it can be used as planned; (2) factors that contributed to this status; and; (3) actions FPS is taking to develop and implement RAMP. Summary of Results: RAMP is over budget, behind schedule, and cannot be used as intended. RAMP's contract award amount totals $57 million, almost three times more than the $21 million original development contract amount. As of June 2011, FPS has spent almost $35 million of the $57 million to develop RAMP. FPS also has experienced delays in developing and implementing RAMP, as it is almost 2 years behind its original July 2009 implementation date. FPS cannot use RAMP to complete FSAs because the agency did not verify the accuracy of the federal facility data it obtained from GSA or include an edit feature in RAMP that would allow inspectors to edit these data when necessary. FPS is also experiencing difficulty using RAMP to ensure that its approximately 13,200 contract guards have met training and certification requirements to be deployed at federal facilities because it does not have a process for verifying this information before it is entered into RAMP. RAMP also does not yet fully incorporate certain government security standards. For example, according to an FPS official, RAMP does not support the April 2010 ISC Physical Security Criteria for Federal Facilities because FPS did not have time to incorporate it in the June 2010 version of RAMP. FPS is planning to incorporate these standards in the next version of RAMP. Several factors have contributed to FPS being unable to use RAMP as planned. Most importantly, FPS and ICE did not adequately follow GAO's project management best practices in developing and implementing RAMP. [Footnote 12] For example, FPS did not manage requirement changes or conduct user acceptance testing with its inspectors as part of RAM P's development. In addition, ICE did not always comply with DHS's acquisition policy and the Federal Acquisition Regulation (FAR), as we found that contractor performance evaluations were not completed. Contractor performance evaluations are one of the most important tools for ensuring that the contractor meets the terms of the contract. FPS is taking some steps to address RAMP's problems. Most notably, FPS has preliminarily decided to discontinue its current RAMP development contract and is considering using a new contractor to finish developing RAMP. Given the technological changes that may have occurred since FPS began developing RAMP 4 years ago, there could be alternative systems that would better meet FPS's needs. However, FPS has not evaluated whether further developing RAMP is the most cost- beneficial option compared to possible alternatives. In addition, FPS has not developed a plan to address the problems we found with RAMP, for example ensuring the accuracy of federal facility and contract guard data. Objective 1: What is RAMP's current status? RAMP Is Over Budget, Behind Schedule, and Cannot Be Used to Complete FSAs and Reliable Guard Inspections: RAMP Is Over Budget: RAMP's potential costs have increased significantly from the initial award amount. RAMP's contract award amount totals $57 million, almost three times more than the $21 million original development contract amount. As of June 2011, FPS has spent almost $35 million of the $57 million to develop RAMP. RAMP's costs increased, in part, because: * FPS changed the original requirements and the contractor had to add additional resources to accommodate them, for example, FPS requested that RAMP operate independently of the web; and; * unanticipated costs associated with FPS needing to meet DHS's Office of Security requirement for a more secure laptop occurred. RAMP Is Behind Schedule: RAMP has been under development for almost 4 years and is currently almost 2 years behind its original July 2009 implementation date. FPS planned to have the FSA component of RAMP completed by July 31, 2009 and the contract guard inspection module completed by July 31, 2010 as well as provide the capability to modify imported GSA facility data and assess risk across FPS's portfolio of federal facilities completed by July 31, 2011. However, as of June 2011, FPS cannot reliably use RAMP to complete FSAs because the agency did not verify the accuracy of the federal facility data it obtained from GSA. See enclosure II for a timeline of RAMP's original and actual milestones. RAMP Cannot Be Used to Complete FSAs: One of the key functions of RAMP was to significantly improve how FPS completes FSAs. Specifically, with RAMP, FPS was supposed to be able to complete FSAs that were based on threat, vulnerability, and consequence. Moreover, FPS would be able to complete FSAs according to government security standards. However, FPS officials said RAMP cannot be used to complete FSAs because data for federal facilities (e.g., the address, government tenants, or the number of floors) obtained from GSA are either missing or unreliable. In addition, FPS did not design RAMP to allow inspectors to edit these data from GSA when necessary, which would have led to incomplete FSA reports. Although GSA officials informed FPS that the facility data had limitations and were not designed for FPS's purpose, an FPS official stated that the agency chose to use these data in RAMP because they were the best source available on federal facilities. However, FPS did not verify the completeness or accuracy of the data. We have reported that agencies should consider the level of risk associated with using data that have missing values in key elements.[Footnote 13] Instead of using RAMP to complete FSAs as planned, FPS inspectors are using a risk calculator spreadsheet and FSA template document. According to FPS guidance, inspectors are to use the calculator to determine threat, vulnerability, and consequence information for facilities. This information is then entered into the template and provided to tenant agencies as a report. There are several issues with these tools. First, according to an FPS official, the template does not meet the ISC standards because it should associate a facility's risks with appropriate countermeasures. Second, because these tools produce individual reports and FPS does not aggregate their results, the agency's ability to assess risk across its portfolio of federal facilities remains limited. Third, FPS stakeholders also raised concerns about the FSA risk calculator and template. For example, in December 2010, FPS training personnel at the Federal Law Enforcement Training Center identified problems with the risk calculator and decided not to teach new inspectors how to use the FSA risk calculator or template. An FPS area commander also said that to identify credible threats using the FSA template, inspectors are using the same subjective approach used in FPS's previous security assessment tool. As a result of the problems with this tool, FPS does not currently employ a comprehensive method for assessing risk to federal facilities but instead must rely on more manual methods until the permanent solution to the problem is implemented. RAMP Cannot Be Used to Complete Reliable Guard Inspections: FPS designed RAMP to help manage its contract guard workforce, including conducting guard post inspections, but the agency is experiencing difficulty using RAMP to ensure that its approximately 13,200 contract guards have the required training and certifications to be deployed at federal facilities. FPS is using RAMP to conduct guard post inspections to ensure that qualified guards are standing post, but neither FPS's guard training and certification information nor its method for determining the qualification status of contract guards in RAMP is reliable. FPS does not have reliable information on its contract guards, in part because it did not fully verify the accuracy of the guard training and certification information from its previous system before migrating it into RAMP, as we recommended in 2010.[Footnote 14] In addition, FPS relies on guard companies to electronically submit guard training and certification information and does not verify these data before they are uploaded into RAMP. As a result, some guards may be designated in RAMP as unqualified when they are qualified, or as qualified when they are unqualified. Furthermore, once guard training and certification information is uploaded into RAMP, FPS still cannot internally verify this information because it no longer maintains physical files. Also, inspectors cannot verify this information during guard post inspections because FPS no longer requires guards to carry certain physical credentials, such as a firearms qualification and training certificate. According to FPS headquarters officials, each region is required to audit 10 percent of each guard company's files each month to determine if they contain the required training and certification information. However, the process for selecting the 10 percent can vary by region and guard company, and FPS does not use the results of those audits to verify the information in RAMP. In addition to challenges with the reliability of its guard information, FPS is also experiencing difficulty using RAMP to determine whether a guard is qualified. For example, FPS did not design RAMP to: * take into account the differences in guard certification requirements specified in FPS's 119 contracts; * distinguish between newly hired guards in training and guards that are unqualified because they have not met training and certification requirements; or; * account for training and certification records when a guard works for more than one company. These factors contribute to FPS having limited assurance that RAMP can be used to determine whether or not a guard is qualified to stand post at a federal facility. We have previously reported that an agency must have reliable information relating to its mission on a real-time basis to effectively manage and control its operations, and should ensure that data validation is performed to identify erroneous data. [Footnote 15] FPS Has Experienced Difficulty Incorporating Certain Government Security Standards: FPS intended for RAMP to support government security standards, such as the NIPP, and to implement ISC security standards—both of which were lacking in the previous systems. Compliance with DHS's NIPP risk assessment framework is important because it ensures that FPS is calculating risk in a manner consistent with other agencies with federal protection responsibilities. Similarly, compliance with ISC standards provides agencies with federal protection responsibilities a consistent approach to mitigate risks at federal facilities. RAMP meets the NIPP's risk assessment framework by including questions to determine the threats, vulnerabilities, and consequences associated with a facility, and calculating an overall numerical risk score for the facility based on the product of these factors. However, according to an FPS official, RAMP does not yet support the April 2010 ISC Physical Security Criteria for Federal Facilities because FPS did not incorporate them in the June 2010 version of RAMP. FPS is planning to incorporate these standards in the next version of RAMP. Objective 2: What factors contributed to RAMP's current status? FPS Did Not Follow Some Project Management Best Practices in Developing and Implementing RAMP: GAO's project management best practices indicate that agencies should manage changes in requirements and conduct user acceptance testing. FPS did not follow these practices in developing RAMP. For example, while FPS originally planned for RAMP Phase 1 to focus on FSAs, FPS changed the requirements for this phase to include the development of the contract guard module. Additionally, FPS changed RAMP from requiring an Internet connection to a system that users could work on while not connected to the Internet and that would also meet ICE network security standards. The contractor informed FPS that these requirement changes were beyond the contract scope and would take more time and resources. FPS and ICE officials authorized the contractor to add staff to implement the changes requested by FPS and resulting additional work, but did not agree to extend the deadline for deploying RAMP. This authorization resulted in FPS spending the entire $21 million original contract amount by April 2010, as opposed to 2015 when the 7-year contract was supposed to end. However, this increase in resources was not effective, as the contractor could not deliver a functional RAMP on this schedule. Our prior work on information technology project management indicates that increasing staff to speed up work is generally not effective and can actually cause greater delays because of the need to coordinate the work and integrate new staff onto the project.[Footnote 16] Additionally, in order to deploy RAMP in November 2009, FPS and ICE did not conduct user acceptance testing with its inspectors, which is a GAO project management best practice. Although the contractor conducted limited system testing, FPS and ICE did not conduct user acceptance testing, which could have identified technical and design problems before RAMP was deployed.[Footnote 17] For example, during the initial rollout of RAMP, many inspectors had problems logging in and thus were not able to use it. In another example, once logged into RAMP, some inspectors experienced significant delays because RAMP downloaded training and certification information on approximately 13,200 guards although the inspector did not need all of this information. Our previous work indicates that user acceptance and system testing help programs meet technical requirements to deliver needed capabilities, and proceeding with acquisitions prior to the completion of testing can result in delays in achieving technical capability.[Footnote 18] As part of DHS, ICE and FPS are required to comply with DHS's Homeland Security Acquisition Regulation and the FAR. For example, DHS and the FAR require that a performance evaluation be completed annually and at the conclusion of the contract for those contracts exceeding $100,000. These evaluations are one of the most important tools for ensuring that the contractor meets the terms of the contract. DHS policy also requires contracting officials to consider past performance as one of several evaluation factors in awarding new contracts. However, when we reviewed the original RAMP contract file in March 2011, we did not find any performance evaluations for the RAMP contractor. According to ICE contracting officials, the performance evaluations were not completed because developing and implementing an initial version of RAMP was the higher priority. We also did not find any documentation in the contract files that ICE took action against the contractor for performance issues, although an ICE official provided us with a March 2010 memorandum to the contractor indicating performance issues. Specifically, the memorandum noted that, as of February 2010, RAMP was over budget, behind schedule, and not performing as expected. This memo is the first official indication that ICE was not satisfied with the contractor's performance. In response to this memo, the contractor provided a corrective action plan in April 2010 to address the performance issues. Finally, although DHS's acquisition policy and the FAR specify that the basis for changes to contracts should be documented, we found that key decisions regarding the change in RAMP's requirements were not documented in the contract files. For example, the justification for the decision to spend the entire $21 million of the original contract in less than 2Y ears was not documented in the contract files. According to ICE contracting officials, these documents were also not completed because developing and implementing the initial version of RAMP was a higher priority. Objective 3: What are the actions FPS is taking to develop and implement RAMP? FPS Is Taking Some Steps to Address RAMP's Problems: FPS is taking some steps to address RAMP's problems. For example, FPS's Director acknowledges that RAMP is not working, and that continuing the current course will not make it functional. As a result, FPS has preliminarily decided to discontinue its current development contract and is considering a new contractor to finish developing RAMP. According to FPS officials, this change will, among other things, reduce development costs and increase the functionality of RAMP. In addition, FPS plans to conduct user testing with its inspectors to ensure that the next version of RAMP functions, integrates stakeholder comments, and incorporates ISC standards. Given the technological changes that may have occurred since FPS began developing RAMP 4 years ago, there may be alternative FSA and guard management systems that would better meet FPS's needs. However, FPS has not evaluated whether further developing RAMP is the most cost- beneficial option or if alternative systems would better meet FPS's needs. In addition, FPS has not developed a plan to address all the problems we found with RAMP, such as ensuring the accuracy of federal facility and contract guard data. Conclusions: After almost 4 years of effort and spending almost $35 million, FPS has not accomplished its goals of using RAMP to complete FSAs and reliable guard inspections. Consequently, until FPS resolves RAMP's problems, FPS will not have a comprehensive method of identifying risks to federal facilities or a reliable method for overseeing its contract guard workforce. While FPS plans to take some actions, if it does not take additional steps to specifically address the problems we found, these problems are likely to continue. It is also important that FPS take immediate steps to follow project management best practices in further development of RAMP or any alternative. Until FPS does so, it risks repeating some of the same mistakes it made during the last 4 years, which have resulted in a risk assessment and management system that is not functional. Completing the required contractor performance evaluations and ensuring that contract files are maintained in accordance with DHS and the FAR is important. For example, completing the required contractor performance evaluations would have provided FPS and ICE officials with the ability to assess the contractor's performance during key phases of RAMP's development and the opportunity to take corrective action if necessary. Maintaining contract files that comply with DHS's acquisition policy and the FAR is also important because the contract files should contain information that explains the basis for key acquisition decisions. FPS's ongoing efforts to protect federal facilities should not be impeded by its decision to finish developing RAMP, particularly since the agency continues to charge GSA and tenant agencies millions of dollars to protect their facilities. Thus, it is important that FPS not only resolve challenges with RAMP but also concurrently pursue interim measures to enhance the protection of the over 1 million government employees and members of the public that visit such facilities each year from a potential terrorist attack or other acts of violence. Finally, we agree with FPS that incorporating the ISC Physical Security Criteria for Federal Facilities into RAMP is important, and encourage FPS to continue its efforts to ensure that this happens before the next version of RAMP is rolled out. Recommendations for Executive Action: Given the challenges FPS faced with developing RAMP, technological changes that may have occurred in the last 4 years, and to help guide and ensure the successful development and implementation of any future risk assessment and contract guard management system, we recommend that the Secretary of Homeland Security direct the Under Secretary of NPPD and the Director of FPS to take the following four actions: * evaluate whether it is cost-beneficial to finish developing RAMP or if other alternatives for completing FSAs and managing security guards would be more appropriate; * increase the use of project management best practices by managing requirements and conducting user acceptance testing for future RAMP development efforts; * establish a process for verifying the accuracy of federal facility and guard training and certification data before entering them into RAMP; and; * develop interim solutions for completing FSAs and guard inspections while addressing RAMP's challenges. To improve contract administration, we recommend that the Secretary of Homeland Security direct the Directors of ICE and FPS to complete contract performance evaluations for its current RAMP contractor and ensure that the evaluations and other required documents are maintained in the contract file in accordance with DHS's acquisition policy and the FAR. Agency Comments and Our Evaluation: We provided a draft of these briefing slides and enclosures to DHS for comment. DHS concurred with our recommendations and provided technical comments that we incorporated where appropriate. [End of briefing slides] Enclosure II: Risk Assessment and Management Program (RAMP) Detailed Timeline and Capability: [Refer to PDF for image: timeline] Original milestones: April 2007: Initial planning. August 1, 2008: RAMP development contract awarded. July 31, 2009: End of contract base year and original deadline for release of RAMP Phase 1. July 31, 2010: End of option year 1 and original deadline for release of RAMP Phase 2. July 31, 2011: End of option year 2 and original deadline for release of RAMP Phase 3. July 31, 2012: End of option year 3. July 31, 2013: End of option year 4. July 31, 2014: End of option year 5. July 31, 2015: End of option year 6 and original end of RAMP's life cycle. Actual schedule: April 2007: Initial planning. August 1, 2008: RAMP development contract awarded. March 10, 2009: Contract modification to provide funding for different laptops. May 27, 2009: Exercise option period in its entirety, increasing the contract amount. November 16, 2009: Federal Protective Service (FPS) launches initial version of RAMP. November 2009–April 2010: RAMP users encounter challenges with the log-in credential process and extensive processing time delays following the release of the initial version of RAMP. December 29, 2009: Exercise option years 4, 5, and 6 early. April 12, 2010: Release of next RAMP version to enhance processing capabilities. June 4, 2010: Signed 2-year follow-on contract, first year fully funded up front. June 28, 2010: Release of next RAMP version to enhance processing capabilities. July 26, 2010: Word Facility Security Assessment (FSA) template and Excel risk calculator issued to FPS inspectors to complete FSAs. January 2012 (anticipated milestone): Anticipated release for next RAMP version. September 20, 2022 (anticipated milestone): Current projected end of RAMP's life cycle. Source: GAO analysis of FPS data. [End of figure] [End of section] Enclosure III: Federal Protective Service's (FPS) Process for Entering Guard Training and Certification Information into Risk Assessment and Management Program (RAMP): Step 1: Within 7 days of a certifying event (e.g., completing cardiopulmonary resuscitation training), FPS requires guard companies to electronically submit guard training and certification information to FPS through an extensible markup language (XML) forms format, such as Microsoft InfoPath.[Footnote 19] Step 2: An FPS contractor uploads the XML forms into RAMP. FPS requires that the guard training and certification information be uploaded within 24 hours of submission from the guard company. Step 3: RAMP refreshes daily to include new uploads. During the refresh period, RAMP may reject a guard company's submission because of data input errors such as mismatched Social Security numbers or a misspelled name. The guard company then has to correct the guard information and resubmit it to FPS to be reuploaded into RAMP. Step 4: At this point in the process, guard training and certification information is available in RAMP for guard post inspections. Source: GAO analysis of FPS information. Enclosure IV: Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, DC 20528: July 8, 2011: Mark L. Goldstein: Director, Physical Infrastructure Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Re: Draft Report GAO-11-705R, "Federal Protective Service: Actions Needed to Resolve Delays and Inadequate Oversight Issues with FPS's Risk Assessment and Management Program" Dear Mr. Goldstein: Thank you for the opportunity to review and comment on this draft report. The U.S. Department of Homeland Security (DHS) appreciates the U.S. Government Accountability Office's (GAO's) work in planning and conducting its review and issuing this report on the Federal Protective Service's (FPS's) Risk Assessment and Management Program (RAMP). RAMP is intended to provide FPS personnel with a centralized source of information for Federal facilities they protect. The DHS National Protection and Programs Directorate (NPPD)/FPS is responsible for the safety of more than a million people who pass through our security portals each day. Our contracted Protective Security Officers (PSOs) conduct millions of inspections each year in pursuit of genuine security—and not just the illusion of it. More than 700,000 dangerous objects and contraband, including weapons, are confiscated each year from entrants at NPPD/FPS screening posts. Our NPPD/FPS officers and inspectors conduct facility security assessments (FSAs), cover more than 1,000 demonstrations and disturbances, and make more than 1,600 arrests annually. Addressing GAO recommendations is a top priority for NPPD/FPS., and work is under way to resolve the issues identified in this report, including GAO's determination that deficiencies in RAMP development may have impacted security at Federal facilities. As stated above, FPS conducts comprehensive FSAs to identify credible threats for each facility and assess specific vulnerabilities and likely consequences associated with those threats. It should be noted that FSAs are one piece of the protective services provided to the Federal community and FPS's other efforts, such as patrol and response, tenant awareness training, countermeasure testing, etc., are ongoing and have a direct bearing on the security of Federal facilities. The development of RAMP has been under way for nearly 4 years. Yet, after careful consideration and review, FPS has determined that RAMP development—as it was being pursued—was not cost-effective and has not fulfilled its original goals. However FPS has a continuing need for elements of RAMP and its basic functionality—which is discussed further in the Departmental response to GAO's specific recommendations. The draft report contained five recommendations, with which DHS concurs. Specifically, GAO recommended that the Secretary of Homeland Security direct the Under Secretary of NPPD and the Director of FPS take the following actions: Recommendation 1: Evaluate whether it is cost beneficial to finish developing RAMP or if other alternatives for completing FSAs and managing security guards would be more appropriate. Response: Concur. NPPD/FPS is revalidating RAMP requirements with its stakeholders and accessing next generation architecture to ensure that future RAMP investments deliver robust capability to the end user, and maximize network efficiencies and information sharing. NPPD/FPS has already begun carefully assessing alternative programs, such as the DHS Science and Technology Directorate's recommended Integrated Rapid Visual Screen solution and the NPPD/Office of Infrastructure Protection (IP) Infrastructure Survey Tool (1ST). Thus far, the results of our preliminary assessment indicate that, at a minimum, NPPD/FPS will gain efficiencies and improve RAMP capability by leveraging the IST that is housed on the Link Encrypted Network System (LENS), an NPPD/IP gateway. With this adjustment, NPPD/FPS will move toward greater collaboration and integration with other NPPD elements. While RAMP is re-engineered to incorporate threat level calculations, recommended countermeasures, and Interagency Security Committee standards, a version of 1ST was selected as an interim solution, enabling NPPD/FPS to continue processing credible FSAs. Placing RAMP on the same network backbone will enable information sharing between NPPD/IP and NPPD/FPS, which will further enhance the Department's ability to protect Federal facilities. Presently, the Department of Energy (DOE), Argonne National Laboratory (ANL) supports LENS. Further, our preliminary assessment also indicates that development of RAMP by DOE ANL for LENS would be more economical than our current approach. As a result, NPPD/FPS suspended RAMP development with our current contractor while DOE ANL and other alternatives are considered. Recommendation 2: Increase the use of project management best practices by managing requirements and conducting user acceptance testing for any future RAMP development efforts. Response: Concur. Additional RAMP development activities will incorporate project management best practices. NPPD requires all acquisition efforts to comply with the Acquisition Management Directive 102.1 (MD 102). Further. NPPD has implemented the NPPD Acquisition Instruction 102-01-01 describing our internal acquisition review process. The Directive was developed on the basis of project management best practices. NPPD/FPS's adherence to these documents will address GAO's recommendation that future RAMP development efforts have robust requirements development, change management, and user acceptance testing processes. Key to success will be to engage and involve the end users and any other stakeholders throughout the entire process to ensure the product delivered meets all expectations and requirements. Best management practices also include development and tracking project activities, milestones, costs, and deliverables through monthly cost reports, project schedule reviews, systems engineering lifecycle gate reviews, and weekly status reports. NPPD/FPS is now identifying stakeholders to fully develop a new integrated project team and also intends to hire a Program Manager immediately to oversee NPPD/FPS information technology (IT) projects, which will ensure compliance with DHS's Systems Engineering Lifecycle requirements and guidance. Lastly, NPPD/FPS will adhere to the Office of Management and Budget's recent "25 Point Implementation Plan to Reform Federal Information Technology Management" (December 9, 2010). This document mandates inclusion of value-added activities and requires Federal IT programs to be structured to deploy business functionality in predetermined release cycles, with initial deployment to end users not more than 18 months after the program begins. Recommendation 3: Establish a process for verifying the accuracy of federal facility and guard training and certification data before entering into RAMP. Response: Concur. With the further development of RAMP, NPPD/FPS intends to make improvements to the PS0 certification validation process, as well as the post-inspection and administrative audit processes. These improvements will focus on accountability for data integrity, metrics and trend analysis, and should also help identify and correct process deficiencies. Recommendation 4: Develop interim solutions for completing FSAs and guard inspections while addressing RAMP's challenges. Response: Concur. As an interim assessment solution, NPPD/FPS is utilizing the 1ST until the development, testing, training, and implementation of future RAMP capabilities have been completed. A modified version of the 1ST will replace the current Microsoft Excel Survey Tool template currently in use by NPPD/FPS. This modified 1ST will be incorporated into the future capabilities of RAMP. It will enable field-based inspectors to complete and file their assigned FSAs electronically in the on-line database, and provide supervisors the ability to approve or comment on the assessments electronically. Additionally, the data collected via the interim IST will ultimately be available in the shared risk assessment database. NPPD plans for the completed FSAs to become a part of the national critical infrastructure and key resources (CIKR) database, allowing NPPD the capability to view and share all CIKR assessments. The new PSO inspection process will focus on assessing the PSOs" knowledge of the post-orders and emergency preparedness and response measures specific to the facility they protect (e.g., Active Shooter, Code Adam, Occupant Emergency Plans, Shelter-in-Place, response to suspicious packages and bomb threats, etc.). NPPD/FPS will analyze data collected from PSO inspections to identify opportunities for remedial improvements. Finally, NPPD/FPS has established a policy to employ a common matrix to collect, categorize, and validate certification data and conduct trend analysis on inspection deficiencies. This common matrix will be designed so that monthly reporting on deficiencies can be incorporated in the Contractor Performance Appraisal Reporting System (CPARS) for guard services contracts. NPPD/FPS will ensure contractual actions taken by the contracting officer in response to performance problems are documented in the contract file. NPPD/FPS will conduct an ongoing assessment of the contractor's performance on the basis of regular inspections and will employ a common format for documenting and addressing performance problems. GAO also recommended that the Secretary of Homeland Security direct the Directors of U.S. Immigration and Customs Enforcement (ICE) and FPS to: Recommendation 5: Complete contract performance evaluations for its current RAMP contractor and ensure that the evaluations and other required documents are maintained in the contract file in accordance with DHS's acquisition policy and Federal Acquisition Regulations (FAR). Response: Concur. The DHS Office of Procurement Operations (OPO) now administers the current contract. OPO will ensure that the Contracting Officer's Technical Representative and the OPO contracting officer complete the required assessments in the CPARS and maintain this information in the contract files per DHS acquisition policy and FAR. NPPD/FPS and OPO are working to complete the contract performance evaluations on the existing RAMP contract. The first contractor performance evaluation under OPO administration is in progress and is due mid-October 2011. Prior to the transfer of contract administration to OPO, ICE had administered the contract. Since that time, ICE has made improvements to its contractor performance reporting program. ICE has created a permanent full-time position to manage CPARS. The manager will track and monitor performance reporting and provide hands-on training to CPARS users. The Head of Contracting Activity receives a monthly status report on CPARS compliance. Additionally, timely CPARS registration has been included in the employee performance plan for every contract specialist/contracting officer. Subsequent to the GAO review, ICE provided copies of monthly quality assurance evaluations that had previously been completed. As described earlier, performance assessments will be placed in CPARS for the expired RAMP contract. Again, we thank you for the opportunity to review and provide comment on this draft report. Sensitivity comments were submitted under separate cover. We look forward to working with you on future Homeland Security-related engagements. Sincerely, Signed by: Jim. H. Crumpacker: Director: Departmental GAO/OIG Liaison Office: [End of section] Footnotes: [1] We refer to property that is owned by the federal government and under the control and custody of the GSA as GSA-owned property. [2] GAO, Homeland Security: Greater Attention to Key Practices Would Improve the Federal Protective Service's Approach to Facility Protection, [hyperlink, http://www.gao.gov/products/GAO-10-142] (Washington, D.C.: Oct. 23, 2009) and GAO, Homeland Security: Federal Protective Service's Contract Guard Program Requires More Oversight and Reassessment of Use of Contract Guards, [hyperlink, http://www.gao.gov/products/GAO-10-341] (Washington, D.C.: Apr. 13, 2010). [3] During RAMP's initial development, FPS was part of ICE. ICE provided software development and project management technical expertise and was responsible for contract award and administration. [4] According to DHS, risk is influenced by the nature and magnitude of threats, the vulnerabilities to these threats, and the consequences that could result. [5] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: Mar. 2004) and Carnegie Mellon Software Engineering Institute, Capability Maturity Model® Integration for Acquisition (CMMI-ACQ), Version 1.2 (November 2007). [6] The ISC Physical Security Criteria for Federal Facilities establishes a baseline set of countermeasures to be applied to all federal facilities based on their facility security level, and provides a framework for customizing security countermeasures to address the unique risks faced at each facility. [7] Managing requirements entails managing the capabilities or conditions that a product is required to meet to satisfy an agreement or standard. User acceptance testing is conducted to ensure that a system meets contract requirements and performs satisfactorily for the users of the program. [8] We refer to property that is owned by the federal government and under the control and custody of the GSA as GSA-owned property. [9] GAO, Homeland Security: Greater Attention to Key Practices Would Improve the Federal Protective Service's Approach to Facility Protection, [hyperlink, http://www.gao.gov/products/GAO-10-142] (Washington, D.C.: Oct. 23, 2009). [10] GAO, Homeland Security: Federal Protective Service's Contract Guard Program Requires More Oversight and Reassessment of Use of Contract Guards, [hyperlink, http://www.gao.gov/products/GA0-10-341] (Washington, D.C.: Apr. 13, 2010). [11] According to DHS, risk is influenced by the nature and magnitude of threats, the vulnerabilities to these threats, and the consequences that could result. [12] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March 2004). [13] GAO, Assessing the Reliability of Computer-Processed Data, [hyperlink, http://www.gao.gov/products/GAO-09-365G] (Washington, D.C.: February 2009). [14] [hyperlink, http://www.gao.gov/products/GA0-10-341]. [15] [hyperlink, http://www.gao.gov/products/GA0-10-341]. See also GAO, Internal Control Management and Evaluation Tool, [hyperlink, http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 2001). [16] [hyperlink, http://www.gao.gov/products/GA0-04-394G]. [17] User acceptance testing is conducted to ensure that a product meets contract requirements and performs satisfactorily. [18] GAO, Department of Homeland Security: Assessments of Selected Complex Acquisitions, [hyperlink, http://www.gao.gov/products/GAO-10-588SP] (Washington, D.C.: June 30, 2010). [19] Microsoft InfoPath is an XML forms-creation and data-gathering tool that permits businesses to gather information without program coding. It requires manual data entry. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: