This is the accessible text file for GAO report number GAO-03-406 entitled 'Customs Service Modernization: Automated Commercial Environment Progressing, but Further Acquisition Management Improvements Needed' which was released on February 28, 2003. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to Congressional Committees: February 2003: Customs Service Modernization: Automated Commercial Environment Progressing, but Further Acquisition Management Improvements Needed: GAO-03-406: GAO Highlights: Highlights of GAO-03-406, a report to Congressional committees: Why GAO Did This Study: The U.S. Customs Service is conducting a multiyear, multibillion dollar project, the Automated Commercial Environment (ACE), a new trade processing system that is planned to support effective and efficient movement of goods into the United States. By congressional mandate, Customs’ expenditure plans for ACE must meet certain conditions, including being reviewed by GAO. This study addresses whether Customs’ latest plan satisfies these conditions and provides observations about the plan and Customs’ efforts to implement GAO’s open recommendations for improving ACE management. What GAO Found: Customs’ November 2002 ACE expenditure plan, the fourth in a series of legislatively required plans, provides for certain project management tasks as well as the definition, design, and development of the first release of the second of four planned ACE increments. GAO’s analysis of the plan shows that it meets the legislative conditions imposed by the Congress. In its series of reports on Customs’ management of ACE, GAO has made a number of recommendations, which Customs is currently addressing. However, Customs has been slow to correct weaknesses in two areas fundamental to effective acquisition management—people and processes. These weaknesses increase the risk that ACE will be late, cost more than necessary, and not perform as intended. * While the Customs Modernization Office (CMO) has developed a human capital strategy, neither the strategy nor supporting documentation identifies how the commitments made in the strategy will be met, including what steps will be taken and what resources are needed to execute the steps. Further, based on the strategy’s timeline, it will be over a year before the CMO fully implements the strategy. * Customs’ has made slow progress in implementing key acquisition practices, such as project office management and acquisition risk management, which GAO first recommended Customs do in 1999. Since that time many practices have been developed, but almost none of them have been implemented, as shown in the figure below. Additionally, Customs’ ACE contractor is conducting system tests in accordance with best practices, initial test results are positive, and indicators suggest the contractor is delivering a quality product. However, Customs is not employing independent verification and validation (IV&V) in overseeing ACE testing, which is one way to mitigate the acquisition weaknesses cited above. Without IV&V, Customs states that it is relying on the contractor’s reputation and maturity level as guarantors of system quality, which is not adequate for a complex and risky program like ACE. What GAO Recommends: To ensure that Customs is positioned to effectively manage the acquisition of ACE, GAO is making recommendations to the commissioner aimed at improving Customs’ acquisition management capabilities. Customs concurred with GAO’s recommendations and described specific actions that it is taking to respond to each. Contents: Letter: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendixes: Appendix I: Customs’ Fourth Automated Commercial Environment (ACE) Expenditure Plan: Appendix II: Comments from the U.S. Customs Service: Abbreviations: ACE: Automated Commercial Environment: CIO: Chief Information Officer: CMM: Capability Maturity Model: CMO: Customs Modernization Office: IV&V: independent verification and validation: OMB: Office of Management and Budget: SA-CMM: Software Acquisition Capability Maturity Model: SAT: system acceptance tests: SEI: Software Engineering Institute: February 28, 2003: The Honorable Ben Nighthorse Campbell Chairman The Honorable Byron L. Dorgan Ranking Minority Member Subcommittee on Treasury and General Government Committee on Appropriations United States Senate: The Honorable Ernest J. Istook, Jr. Chairman The Honorable Steny H. Hoyer Ranking Minority Member Subcommittee on Transportation, Treasury, Postal Service, and General Government Committee on Appropriations House of Representatives: In November 2002, the U.S. Customs Service submitted to Congress its fourth expenditure plan, seeking release of $314 million for its Automated Commercial Environment (ACE) project. ACE is to be Customs’ new trade processing system and the first project under the Customs Modernization Program. As required by Customs’ fiscal year 2002 appropriation and continuing resolutions,[Footnote 1] we reviewed the expenditure plan. Our objectives were (1) to determine whether the fourth ACE expenditure plan satisfies certain legislative conditions, (2) to determine whether the plan is consistent with our open ACE recommendations, and (3) to provide observations about the plan and Customs’ management of ACE. On December 27, 2002, we briefed your offices on the results of this review. This report transmits the results of our work. The full briefing, including our scope and methodology, is reprinted as appendix I. Concerning our first objective, Customs’ expenditure plan satisfies the legislative conditions specified in Customs’ appropriations act. That is, the plan provides for (1) meeting the capital planning and investment control review requirements of the Office of Management and Budget (OMB); (2) complying with Customs’ enterprise architecture;[Footnote 2] and (3) complying with federal acquisition rules, requirements, guidelines, and systems acquisition management practices. Further, the plan was reviewed and approved by the Joint Capital Investment Review Board[Footnote 3] and OMB. Concerning our second objective, Customs is making progress in addressing our open recommendations, which are as follows: * Justify and make investment decisions incrementally. * Before building each ACE release, certify to Customs’ appropriations subcommittees that Customs’ enterprise architecture has been sufficiently extended and updated. * Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating as defined in the Software Engineering Institute’s (SEI) institutional and project-specific estimating guidance. * Immediately develop and implement a human capital management strategy for the Customs Modernization Office (CMO). * Limit future expenditure plan requests for management reserve funds to 10 percent of the total funds requested for the program, or adequately justify any management reserve requests in excess of 10 percent. * Develop and implement process controls consistent with SEI’s Software Acquisition Capability Maturity Model (SA-CMM[Footnote 4]), and by September 30, 2002, assess and report to Customs’ appropriation subcommittees on the maturity of CMO’s software acquisition process. * Address the risks associated with the accelerated ACE acquisition strategy and report on the strategy going forward and plans for mitigating the risks associated with this strategy. While Customs has taken actions to address each of these recommendations, its actions to develop and implement a human capital strategy and acquisition management process controls were only partially consistent with our recommendations. Specifically, while the CMO developed a human capital strategy that specified high-level commitments and satisfied some best practices, there are some notable omissions, including how these commitments will be met. Further, based on the timeline in the strategy, it will be over a year before key practices are implemented. Also, while Customs reported to its House and Senate appropriations subcommittees on its development of SA-CMM processes, it did not report on implementation. Further, while many of the acquisition processes have been developed, Customs has not yet implemented most of them. Finally, we made the following observations: * System integration and acceptance testing performed by the ACE contractor is being conducted in accordance with best practices. The contractor has established a test organization and built an infrastructure for test activities. It has completed system integration and system acceptance tests (SAT) for ACE increment 1, release 1.1, and these test activities were consistent with test management best practices. * The results of SAT performed by the ACE contractor are positive. All but seven ACE increment 1, release 1.1 requirements successfully passed their test cases for SAT. For those that did not pass, only noncritical system defects resulted, and acceptable workarounds were identified. * Data provided by the ACE contractor on all unresolved system defects for ACE increment 1, release 1.1, show a recent downward trend, suggesting that the contractor is delivering a quality product. * Customs’ oversight of contractor testing does not provide for independent verification and validation (IV&V). Industry best practices include IV&V as a part of effective managerial oversight and control. * The potential for ACE infrastructure to support other Department of Homeland Security applications could affect both cost and schedule for ACE. Customs is one of several agencies to be merged into the Department of Homeland Security, and it has proposed that the ACE infrastructure be used by these agencies. Because a meaningful understanding of the effect of the new department’s use of the ACE infrastructure is not yet known, there is the potential for existing ACE cost and schedule commitments to change. Recommendations for Executive Action: To ensure that Customs has the requisite capability to manage its ACE acquisition, we recommend that the Customs Service Commissioner designate strengthening CMO human capital and acquisition processes as priority matters. To this end, we recommend that the Commissioner direct the Chief Information Officer (CIO) to immediately: * develop and implement each of the missing SEI SA-CMM practices for the key process areas discussed in this report, and until this is accomplished, report to its appropriations subcommittees quarterly on the progress of its efforts to do so; * develop and implement the missing human capital management practices discussed in this report, and until this is accomplished, report to its appropriations committees quarterly on the progress of its efforts to do so; and: * establish an IV&V function to assist Customs in overseeing contractor efforts, such as testing. Additionally, we recommend that the Commissioner take steps, as appropriate in light of Customs’ merger into the Department of Homeland Security, to have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. Agency Comments and Our Evaluation: In written comments on a draft of this report signed by the acting director, Office of Planning, Customs stated that it was pleased with the report, agreed with our recommendations, and described specific actions that are being taken to implement each. Customs’ comments are reprinted in appendix II. Customs also provided additional information regarding our recommendation that it establish an IV&V function to assist in overseeing contractor efforts, such as testing. Specifically, Customs stated that it wanted to modify its earlier statement that it relies on the contractor’s reputation and maturity level as guarantors of system quality. Customs said that its modified position is based on a number of practices that it believes are elements of an integrated approach to IV&V. However, the practices that Customs described as elements of IV&V, while additive to system quality, are not independent from program cost and schedule pressures, and therefore do not constitute IV&V. We are sending copies of this report to the Chairmen and Ranking Minority Members of other Senate and House committees and subcommittees that have authorization and oversight responsibilities for the Customs Service. We are also sending copies to the Secretary of Homeland Security, the Secretary of the Treasury, the Commissioner of the Customs Service, and the Director of OMB. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-3439. I can also be reached by E-mail at HiteR@gao.gov. Key contributors to this report were Mark Bird, Harold Brumm, Jr., Barbara Collier, Scott Farrow, Joanne Fiorino, Tamra Goldstein, Michael Holland, Freda Paintsil, Madhav Panwar, Karen Richey, Randolph Tekeley, and Aaron Thorne. Signed by Randolph C. Hite: Randolph C. Hite Director, Information Technology Architecture and Systems Issues: [End of section] Appendixes: [End of section] Appendix I: Customs’ Fourth Automated Commercial Environment (ACE) Expenditure Plan: [See PDF for image] - graphic text: [End of figure] - graphic text: [End of section] Appendix II: Comments from the U.S. Customs Service: U.S. Customs Service Memorandum: DATE: February 13, 2003: MEMORANDUM FOR RANDOLPH C. HITE: U.S. GENERAL ACCOUNTING OFFICE: FROM: Acting Director Office of Planning: SUBJECT: General Accounting Office (GAO) Draft Report: Customs Service Modernization Automated Commercial Environment (ACE) Progressing, but Further Acquisition Management Improvements Needed: Thank you for providing us with a copy of your draft report and the opportunity to discuss the issues in this report. Overall, we are pleased with the GAO report. GAO concludes that Customs has met the legislative requirements imposed by Congress and continues to progress in addressing prior GAO recommendations. GAO also reports a positive assessment of the testing practices and results of the ACE contractor. GAO characterizes progress in implementing a human capital strategy and key acquisition management practices as slow and recommends quarterly reporting of progress to the appropriations committees. While we certainly agree with the recommendations, we briefly note actions taken that show that Customs continues to work diligently to address these two critical program management dimensions. Customs continues to implement its Modernization Strategic Human Capital Management Plan (HCMP) in accordance with the GAO principles. Priority attention was given to gaining Commissioner approval for an expanded Customs Modernization Office (CMO) organizational structure and filling these positions. This has been accomplished, and a talented cadre of executives is trained and functioning to oversee the Modernization task orders. In addition, our quarterly report will show that we have taken responsive actions across the range of GAO principles, such as the following: * Integration of human capital functional staff into management teams through the inclusion of the OF Resources Management Group (RMG) in the RCMP implementation and through participation of the Director of the RMG and Customs Office of Human Resources Management in the Modernization Management Team. * Issuance of the new “Customs Leadership Development Guide” in December 2002 that clearly identifies a comprehensive leadership competency model, as well as a Customs Succession Management System. * Establishment of performance plans for the CMO staff in October 2002, and the inclusion of Modernization objectives in the performance plans of an array of OIT executives and managers. *CMO recognition of staff for their contributions at a quarterly ceremony with both cash and non- traditional awards. The CMO is also continuing to implement key acquisition practices. Its January assessment found substantial progress in implementing key practices, but Customs agrees that much more progress is needed. To ensure that progress is made: * CMO and other Customs managers and personnel have been assigned responsibilities to manage and implement the acquisition processes and have been trained on the process improvement goals, timeframes, and their roles in achieving these goals. * The CMO continues to internally assess progress against the CMO process improvement goals. Additionally, Customs is contracting with the Software Engineering Institute (SEI) to conduct both rated and non- rated Software Acquisition -Capability Maturity Model (SA-CMM) assessments to measure progress towards becoming a Level 2 organization. * Deficiencies identified in the process asset audit have been distilled into Corrective Action Requests (CARs) that are tracked in an OF database. The CMO is aggressively working to close these CARs and continues to move as rapidly as possible through process implementation. These are examples of the progress Customs will include in its quarterly report. OF also agrees to implement actions responsive to GAO’s recommendation relative to an independent verification and validation function (IV&V). However, OF wishes to modify the characterization of its IV&V function provided to GAO in the short timeframe we had for a response. To state that we rely upon the contractor’s reputation and maturity level as guarantors of system quality is not reflective of the range of practices the CMO has implemented as part of its integrated approach to IV&V. Such practices have included; * Independent review of the e-Customs Partnership (eCP) requirements products by end users. * Involvement of the OIT Software Development and Infrastructure Services Divisions in conducting IV&V of the technical design and implementation products. * Engagement of the MITRE Corporation, as the CMO’s Federally Funded Research and Development Center (FFRDC), to provide technically knowledgeable and independent reviews of all critical eCP products. These practices represent elements of an Integrated Independent Verification and Validation (12V2) function that enables us to accomplish the needs of a complex Modernization program. Customs recognizes that these functions could benefit from being formalized into a process. As a result, where appropriate, Customs will strengthen and formalize this process, as well as establish formal accountability for the 12V2 function, including formal sign-off. MITRE, as the Customs FFRDC, will take the lead in reviewing, documenting, and strengthening the 12V2 process. If you have any questions regarding these comments, please contact Ms. Michele Donahue at (202) 927-0957. Signed by Brenda B. Smith: Brenda B. Smith: Attachment: U.S. Customs Service General Accounting Office (GAO) Review of Fourth Automated Commercial Environment (ACE) Expenditure Plan: Recommendation 1: Develop and implement the missing human capital management, practices discussed in this briefing, and until this is accomplished, report to its (Customs) appropriations committees quarterly on the progress of its (Customs) efforts to do so. Response: Customs concurs with GAO’s recommendation to implement the actions identified in the Modernization Strategic Human Capital Management Plan (HCMP) and further develop the Customs Modernization Office (CMO) human capital practices. Customs also shares GAO’s desire to speed implementation and thereby mitigate program risks. Customs will report quarterly on its progress in implementing its HCMP. Milestone Date:March 31, 2003 for CMO concurrence on Job and Core Competencies: Recommendation 2: Develop and implement each of the missing Software Engineering Institute (SEI) Software Acquisition-Capability Maturity Model (SA-CMIM) practices for the key process areas discussed in this briefing, and until this is accomplished, report to its (Customs) appropriations subcommittees quarterly on the progress of its efforts to do so. Response: Customs agrees with GAO on the immediate need to develop and implement all of the SEI SA-CMM practices in the key process areas, and believes substantial progress is being made toward that end. Customs will provide quarterly progress reports to Customs appropriations committees in accordance with Congressional direction. Milestone Date:April 2003 for updated, process asset audit: Recommendation 3: Establish an independent verification and validation (IV&V) function to assist Customs in overseeing contractor efforts, such as testing. Response: The opportunity afforded by the comment period on the GAO draft report leads the Office of Information and Technology (OIT) to modify the position it took in responding to GAO’s short turnaround request of December 26, 2002 for a characterization of its IV&V activities. OIT believes it has in place a number of elements of an Integrated Independent Verification and Validation (12V2) function that conforms to contemporary practices and has provided value in terms of reduced risks. However, Customs recognizes that these functions could benefit from being formalized into a process. As a result, where appropriate, Customs will strengthen and formalize this process, as well as establish formal accountability for the 12V2 function, including formal sign-off. MITRE, as the Customs Federally Funded Research and Development Center FFRDC), will take the lead in reviewing, documenting, and strengthening the I V2 process. Milestone Date:May 30, 2003 for an 12V2 strategic plan July 31, 2003 for documented 12V2 process: Recommendation 4: Take steps, as appropriate in light of Customs merger into the Department of Homeland Security, to have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. Response: Customs recognizes the importance of apprising the appropriations committees of any changes in ACE plans that might arise from merger into the Department of Homeland Security. However, Customs questions the appropriateness of the GAO recommendation that “tentative proposals or plans” be included. Customs will continue its practice of providing the appropriations committees with expenditure plans that are well justified and are vetted and approved across the Executive Branch. When the ACE infrastructure is used or extended to support other Homeland Security applications, this will be noted in the appropriate Expenditure Plans. Milestone Date:August 2003 for release of Expenditure Plan to the Congress and GAO: FOOTNOTES [1] For example, P. L. 108-2 appropriated such amounts as may be necessary under the authority and conditions provided in the applicable appropriations act for fiscal year 2002 for continuing projects or activities that were conducted in fiscal year 2002, at a rate for operations not exceeding the current rate, and for which appropriations, funds, or other authority was made available in the Treasury and General Government Appropriations Act, 2002, P. L. 107-67, 115 Stat. 514, 520 (2001). These appropriations shall be available to the extent and in the manner which would be provided by the fiscal year 2002 Treasury Appropriations Act. [2] An enterprise architecture is an institutional blueprint for guiding and constraining investments in business process change and systems. [3] To expedite reviews of ACE expenditure plans, the Customs and Treasury Investment Review Boards were consolidated to form the Joint Capital Investment Review Board. [4] Capability Maturity Model (CMM) is a service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and Trademark Office. The SA-CMM identifies key process areas that are necessary to effectively manage software-intensive system acquisitions. Level 2 is the second level of the SA-CMM’s five-level scale; achieving this level means that an organization has the software acquisition rigor and discipline to repeat project successes. GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: