This is the accessible text file for GAO report number GAO-08-576R entitled 'Comments on the Office of Personnel Management's February 20, 2008 Report to Congress Regarding the Retirement Systems Modernization,' which was released on March 28, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-08-576R: United States Government Accountability Office: Washington, DC 20548: March 28, 2008: The Honorable Richard Durbin: Chairman: The Honorable Sam Brownback: Ranking Member: Subcommittee on Financial Services and General Government: Committee on Appropriations: United States Senate: The Honorable José E. Serrano: Chairman: The Honorable Ralph Regula: Ranking Member: Subcommittee on Financial Services and General Government: Committee on Appropriations: House of Representatives: Subject: Comments on the Office of Personnel Management's February 20, 2008 Report to Congress Regarding the Retirement Systems Modernization: The Office of Personnel Management (OPM) is modernizing the paper- intensive processes and antiquated information systems it uses to support the retirement of civilian federal employees through the Retirement Systems Modernization (RSM) program. In January 2008, we reported on the agency's management of this program, in which we noted concerns and made recommendations for improvement in four key areas: (1) system testing, (2) system defect resolution, (3) program cost estimation, and (4) program earned value management.[Footnote 1] The explanatory statement of the House Appropriations Committee regarding the fiscal year 2008 Consolidated Appropriations Act directed OPM to submit to your Committees and to GAO not later than February 20, 2008, a report of its actions on the four areas of concern that we identified.[Footnote 2] Further, the explanatory statement directed that GAO provide to your Committees and to OPM our comments on the agency's report. In response to this direction, our comments (including a summarization of our report findings and recommendations and OPM's reported actions for each area of concern) follow. System Testing: GAO Finding and Recommendation: Effective testing is an essential component of any program that includes system development. Generally, the purpose of testing is to identify defects or problems in meeting defined system requirements or satisfying system user needs[Footnote 3]. Our study determined that OPM's initial RSM system test results did not provide assurance that a major system component--the Defined Benefits Technology Solution (DBTS)--would perform as intended, and that the agency's compressed and concurrent testing schedule increased the risk that it would not have sufficient resources or time to verify that the system will work as intended. Accordingly, we recommended that OPM ensure that sufficient resources are provided to fully test functionality, actions for mitigating the risks inherent in concurrent testing are identified, test results verify that all system components perform as expected, and test activities and results are subjected to independent verification and validation. OPM Report to Congress: OPM's report summarized the results of certain user acceptance tests (UAT) that the agency has conducted as part of verifying that DBTS meets requirements such as accurately calculating retirement benefits. The agency reported test results as shown in table 1. Table 1: OPM Reported Test Results in Terms of Scenarios: Scenarios Tested: Scenarios Passed: Percent Passed: Test: UAT 5; Scenarios Tested: 192; Scenarios Passed: 142; Percent Passed: 74. Test: UAT 6; "Discontinued and reconstituted as UAT 6.5 to limit focus to high frequency functionality for Go Live". Test: UAT 6.5; Scenarios Tested: 57; Scenarios Passed: 51; Percent Passed: 89. Source: OPM. [End of table] GAO Comments: OPM limited its discussion of system testing to the UAT results. In this regard, the results shown for UAT 5, indicating that 74 percent of scenarios had passed testing, falls short of the agency's stated goal of 95 percent of scenarios passed. In addition, the statement that UAT 6 was "discontinued and reconstituted as UAT 6.5 to limit focus to high frequency functionality for Go Live" indicates that the scope of UAT 6.5 was reduced from the scope planned for UAT 6. Further, OPM's report did not address the results of other critical tests that the agency had planned to conduct starting in December 2007 and ending in February 2008 as illustrated in figure 1. Figure 1: RSM Test Schedule: [See PDF for image] This figure is a timeline of the RSM test schedule. The following items are displayed on the timeline: Test: UAT 1: Date: Mid-March 2007. Test: UAT 2 part 1; Date: Late May 2007. Test: UAT 2 part 2; Date: Early June 2007. Test: UAT 3 part 1; Date: Early August 2007. Test: UAT 3 part 2; Date: Mid-September 2007. Test: UAT 4; Date: Mid-October 2007. Test: UAT 5; Date: Mid-December 2007; Test: Parallel test; Date: Mid-to-late December 2007. Test: UAT 6; Date: Mid-January 2008. Test: Parallel test; Date: Mid-January to Mid-February 2008. Test: Integrated product test; Date: Late January to Mid-February 2008. Test: Performance test; Date: Late January to Mid-February 2008. Test: Business capacity release test; Date: February 2008. Increment 1 deployment: Date: End of February 2008. Source: GAO, based on OPM data. [End of figure] Specifically, the report did not discuss the results of the following tests: * Parallel test to verify that the new system produces the same results as existing systems. * Integrated product test to confirm that system components meet functional requirements (e.g., accurately calculate benefits). * Performance test to confirm that the new system meets performance requirements (e.g., processing volume and execution time). * Business capability test to confirm the operational readiness of the new system for end users. As our report noted, these tests are intended to verify that DBTS and other system components work together as expected when they are combined and that the complete system resulting from the RSM program satisfies all requirements (e.g., functional and performance) and is acceptable to end users. Finally, the report did not address the conduct of independent verification and validation of test activities and results. The purpose of independent verification and validation is to provide an independent review of system processes and products to ensure that quality standards are being met. System Defect Resolution: GAO Finding and Recommendation: Defects are system problems that require a resolution and can be due to a failure to meet the system specifications. Defects are often identified prior to and during system tests. As we have previously reported, having current and accurate defect information is necessary to adequately understand system maturity and to make informed decisions about how to best allocate limited resources to meet competing priorities.[Footnote 4] Our study of RSM determined that trends in identifying and resolving system defects indicated a growing backlog of problems to be resolved prior to deployment of the new system. These include urgent and high priority defects, which by OPM's definition are defects that either prevent progress or defects that need to be addressed in the current system phase prior to deployment. As a result, we recommended that the agency monitor and review DBTS defects to ensure that all urgent and high priority defects are resolved prior to system deployment and that the resolution of urgent and high priority defects is subjected to independent verification and validation. OPM Report to Congress: OPM's report summarized system defect resolution as shown in table 2. Table 2: OPM Reported Defects Status: Defect Severity Priority: Urgent; Number of defects, October 2007: 9; Number of defects, January 2008: 14; Projected number of defects at February 2008 deployment: 1. Defect Severity Priority: High; Number of defects, October 2007: 129; Number of defects, January 2008: 70; Projected number of defects at February 2008 deployment: 12. Defect Severity Priority: Medium/low; Number of defects, October 2007: 229; Number of defects, January 2008: 85; Projected number of defects at February 2008 deployment: 2. Defect Severity Priority: Total; Number of defects, October 2007: 367; Number of defects, January 2008: 168; Projected number of defects at February 2008 deployment: 15. Source: OPM. [End of table] In its report, OPM stated that the decreasing pattern of defects was consistent with the agency's expectation for the identification and resolution of defects. The report also stated that the one urgent priority defect that was projected to remain at the time of the February 2008 deployment was related to functionality referred to as "unpaid re-deposit" and is expected to be infrequent and easily identifiable. GAO Comments: With the exception of an increase in urgent priority defects from 9 to 14, OPM reported progress toward resolving defects between October 2007 and January 2008. However, the agency did not report actual defect resolution data for February 2008; instead it reported the projected defects it expected to remain at the time of system deployment, scheduled for February 25, 2008. If this projection was realized, it would represent continued defect resolution progress. Nevertheless, OPM's plan to deploy its system with a total of 13 unresolved urgent and high priority system defects increased risk and contradicted earlier assertions that the agency would resolve urgent and high priority defects prior to deployment. In addition, the report did not address conducting independent verification and validation of system defect resolution results. Cost Estimating: GAO Finding and Recommendation: A cost estimate is the summation of individual program cost elements, using established methods and valid data to estimate future costs. Credible cost estimates are produced by following rigorous steps and are accompanied by detailed documentation, including descriptions of the system under development, estimation methodology, ground rules and assumptions, and sensitivity and uncertainty analyses.[Footnote 5] Our study determined that the reliability of OPM's $421.6 million RSM life-cycle cost estimate was questionable because the agency could not support the estimate with a description of the system to be developed and a description of the methodology used to produce the estimate. Thus, we recommended that the agency develop a revised RSM cost estimate that addresses the weaknesses identified and task an independent verification and validation contractor with reviewing the process used to develop the estimate and assessing the reliability of the resulting estimate. OPM Report to Congress: Relative to this area of concern, OPM reported that 86 percent of program costs are associated with fixed price contracts and that remaining costs are predominantly for OPM staff and project management activities. Further, the agency asserted that most future program costs are known and do not require cost estimating. Additionally, the agency provided information on factors that contributed to an increase in the RSM life-cycle cost estimate from $371.2 million to $421.6 million. GAO Comments: The report did not provide new information or describe the progress the agency has made to address the weaknesses in the RSM cost estimate. Specifically, the agency did not provide documentation about the system under development (i.e., a technical baseline description), a cost estimating methodology, estimating ground rules and assumptions, and sensitivity and uncertainty analyses. Without such documentation, an assessment of the reliability of OPM's estimate cannot be made. Further, the extent to which OPM's estimate reflects changes to the program, including delayed deployment of certain functionality from February 2008 to August 2008, is not readily apparent. Finally, OPM's report did not address conducting independent verification and validation of its cost estimate and the process used to develop the estimate. Earned Value Management: GAO Finding and Recommendation: Earned value management (EVM) is a tool for measuring program progress by comparing the value of work accomplished with the amount of work expected to be accomplished. Such a comparison permits actual performance to be evaluated, based on variances from the planned cost and schedule, and future performance to be forecasted. Our study determined that OPM's reporting of program progress using EVM was unreliable because the agency did not establish and validate a meaningful performance measurement baseline. Accordingly, we recommended that the agency establish a basis for effective use of earned value management by validating the RSM performance measurement baseline through a program level integrated baseline review, and that it task an independent verification and validation contractor with reviewing the process used to develop the baseline and assessing the reliability of the performance measurement baseline. OPM Report to Congress: OPM stated in its report that the agency had established a performance measurement baseline, and had implemented EVM practices. Specifically, the agency stated that it had measured and reported performance on a monthly basis, used an agency-standard EVM tool set, and worked with its contractors to improve EVM reporting. Further, OPM characterized GAO's concerns about its use of EVM as being related to the proper inclusion of work to be performed by government personnel and of work related to a delay in deployment of planned functionality. GAO Comments: OPM's report did not provide new information or describe the progress the agency has made toward addressing the three specific weaknesses that we identified in its use of EVM. First, the report did not state whether OPM has established a performance measurement baseline that reflects the full scope of the RSM program. Second, the report did not say whether the agency has validated its performance measurement baseline in an integrated baseline review. Lastly, the report did not address whether the baseline against which the agency has measured and reported has been stabilized. In addition to these three weaknesses, the report also did not address conducting independent verification and validation of the performance measurement baseline and the process used to develop the baseline. Without addressing these weaknesses, OPM's use of EVM will continue to be unreliable. Agency Comments: In written comments on a draft of this letter, the Director of OPM stated that the agency successfully initiated rollout of its new retirement system to the approximately 26,000 employees in agencies serviced by the General Services Administration payroll processing center on February 25, 2008. Further, the Director stated that OPM's forthcoming formal response to GAO's January 2008 report on RSM will provide detailed information on the agency's progress in developing and implementing the new system. Finally, the Director commented that the short time between its February 20, 2008 report to Congress and system deployment necessitated that the report be brief and that its contents (which are summarized in this letter) be limited to official use only. However, in subsequent comments e-mailed to us, OPM's liaison to GAO stated that the report to Congress did not contain sensitive information. The agency's comments are reprinted in the enclosure. We are sending copies of this letter to the Director of the Office of Personnel Management and other appropriate congressional committees. We will make copies available to other interested parties upon request and will made it available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this correspondence, please contact me at (202) 512-6304 or melvinv@gao.gov. Key contributors to this letter were Mark T. Bird, Rebecca E. LaPaze, and Teresa M. Neven. Signed by: Valerie C. Melvin: Director, Human Capital and Management: Information Systems Issues: [End of correspondence] Enclosure: Comments from the Office of Personnel Management: The Director: United States Office Of Personnel Management: Washington, DC 20415: [hyperlink, http://www.opm.gov]: [hyperlink, http://www.usaJobs.gov]: "Our mission is to ensure the Federal Government has an effective civilian workforce." March 18, 2008: Mr. Gene L. Dodaro: Acting Comptroller General: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Dodaro: Thank you for the opportunity to comment on the Government Accountability Office's (GAO) draft response to the Office of Personnel Management's (OPM) February 20, 2008, Retirement Systems Modernization (RSM) Report to Congress. On February 25, 2008, OPM successfully initiated the rollout of its landmark strategic program that has modernized the business processes and supporting technology used to administer the Federal Government's defined retirement benefit plans. Approximately 26,000 employees in agencies serviced by the General Services Administration (GSA) payroll processing center are now eligible to retire under the new system, which is being called RetireEZ. These agencies include OPM, GSA, the National Archives and Records Administration, and the Railroad Retirement Board. Subsequent rollouts will cover the remainder of the Executive Branch, the U.S. Postal Service and the Legislative and Judicial Branches. OPM was tasked with responding to the Congressional request for a status report just three days before the launch of RetireEZ. The challenging timing of this request necessitated that our responses were briefer than we would have preferred. We believe the brevity of this report supports our position that this information — while accurate — should be limited to official use only. We believe the public deserves the benefit of a more comprehensive report. OPM is currently preparing our sixty day response to GAO's report entitled Office of Personnel Management: Improvements Needed to Ensure Successful Retirement Systems Modernization (GAO-08-345). This information, to be submitted to the United States House of Representatives and Senate Committees on Appropriations no later than March 31, 2008, will provide greater detail on OPM's progress in developing and implementing the new system, in addition to addressing the four recommendations in this particular GAO report. Thank you again for the opportunity to provide GAO with these comments. OPM continues to take the necessary steps to ensure that the Federal Government has a state-of-the-art retirement administration system for annuitants and employees. Sincerely, Signed by: Linda M. Springer: Director: [End of enclosure] Footnotes: [1] GAO, Office of Personnel Management: Improvements Needed to Ensure Successful Retirement Systems Modernization, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-345] (Washington, D.C.: January 31, 2008). [2] Consolidated Appropriations Act, 2008, P.L. 110-161 (Dec. 26, 2007); 153 Cong. Rec. H15741, H16056 (daily ed. Dec. 17, 2007). [3] GAO, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998); Information Technology: Customs Automated Commercial Environment Progressing, but Need for Management Improvements Continues, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-267] (Washington, D.C.: March 14, 2005); and Homeland Security: Visitor and Immigrant Status Program Operating, but Management Improvements Are Still Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T] (Washington, D.C.: January 25, 2006). [4] GAO, Customs Service Modernization: Automated Commercial Environment Progressing, but Further Acquisition Management Improvements Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-406] (Washington, D.C.: February 28, 2003) and Homeland Security: Visitor and Immigrant Status Program Operating, but Management Improvements Are Still Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T] (Washington, D.C.: January 25, 2006). [5] GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, Exposure Draft, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, D.C.: July 2007). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: