Information Technology:

Customs Automated Commercial Environment Program Progressing, but Need for Management Improvements Continues

GAO-05-267: Published: Mar 14, 2005. Publicly Released: Mar 14, 2005.

Additional Materials:

Contact:

Randolph C. Hite
(202) 512-6256
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Homeland Security (DHS) is conducting a multiyear, multibillion-dollar acquisition of a new trade processing system, planned to support the movement of legitimate imports and exports and strengthen border security. By congressional mandate, plans for expenditure of appropriated funds on this system, the Automated Commercial Environment (ACE), must meet certain conditions, including GAO review. This study addresses whether the fiscal year 2005 plan satisfies these conditions, describes the status of DHS's efforts to implement prior GAO recommendations for improving ACE management, and provides observations about the plan and DHS's management of the program.

The fiscal year 2005 ACE expenditure plan, including related program documentation and program officials' statements, largely satisfies the legislative conditions imposed by the Congress. In addition, some of the recommendations that GAO has previously made to strengthen ACE management have been addressed, and DHS has committed to addressing those that remain. However, much remains to be done before these recommendations are fully implemented. For example, progress has been slow on implementing the recommendation that the department proactively manage the dependencies between ACE and related DHS border security programs. Delays in managing the relationships among such programs will increase the chances that later system rework will be needed to allow the programs to interoperate. Among GAO's observations about the ACE program and its management are several regarding DHS's approach to addressing previously identified cost and schedule overruns. DHS has taken actions intended to address these overruns (such as revising its baselines for cost and schedule, as GAO previously recommended); however, it is unlikely that these actions will prevent future overruns, because DHS has relaxed system quality standards, meaning that milestones are being passed despite material system defects. Correcting such defects will require the program to use resources (e.g., people and test environments) at the expense of later system releases. Until the ACE program is held accountable not only for cost and schedule but also for system capabilities and benefits, the program is likely to continue to fall short of expectations. Finally, the usefulness of the fiscal year 2005 expenditure plan for congressional oversight is limited. For example, it does not adequately describe progress against commitments (e.g., ACE capabilities, schedule, cost, and benefits) made in previous plans, which makes it difficult to make well-informed judgments on the program's overall progress. Also, in light of recent program changes, GAO questions the expenditure plan's usefulness to the Congress as an accountability mechanism. The expenditure plan is based largely on the ACE program plan of July 8, 2004. However, recent program developments have altered some key bases of the ACE program plan and thus the current expenditure plan. In particular, the expenditure plan does not reflect additional program releases that are now planned or recent changes to the roles and responsibilities of the ACE development contractor and the program office. Without complete information and an up-to-date plan, meaningful congressional oversight of program progress and accountability is impaired.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Automated Commercial Environment (ACE) program has demonstrated the use of appropriate exit criteria for several key readiness milestones. Specifically, the system life cycle gate review process for Office of Information and Technology projects, including ACE, has been amended to include milestone readiness decisions based on an assessment and acceptance of risks (including the risks related to unresolved defects). Further, the ACE Risk and Issue Management Process guidance provides for the systematic identification, analysis, prioritization, planning, execution, evaluation, and documentation of program risks and issues and requires that any risks associated with going forward should be identified as part of each life cycle gate review. This process has been applied in the following two major gate reviews: Release A1 (ESAR) Critical Design Review (May 2006) and Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review (August 2006). In the case of A1, program officials reported that no severe defects existed, but that all known risks associated with proceeding past the milestone had been entered into the program's risk database and documented in the certification package submitted to the Department of Homeland Security's Chief Information Officer. Program officials reported these steps in the quarterly reports to Congress. Going forward, officials stated that similar assessments will be part of future gate reviews.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

  2. Status: Closed - Not Implemented

    Comments: Automated Commercial Environment (ACE) expenditure plans, including the fiscal year 2007 plan, have not adequately reported on progress against previous plan commitments. In May 2006, we reported that the fiscal year 2006 expenditure plan contained several such reporting gaps, such as whether funding amounts were actually obligated and expended as planned and whether releases met schedules. In particular, we reported that the plan did not address whether the design and development for Release 5 was actually accomplished as planned. Similarly, we reported the next year that the fiscal year 2007 expenditure plan did not adequately describe progress against commitments that were made in the fiscal year 2006 plan. For example, the 2007 expenditure plan did not: report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved; report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans; or address progress against the milestone dates for each stage of a release that was included in the prior year's plan. As a case in point, the fiscal year 2006 expenditure plan described specific functionality planned for Release 5/Drop A1, such as online registration for trade representatives, but the fiscal year 2007 expenditure plan does not include information on whether these functions were delivered. According to program officials, the quarterly congressional reports provide more current information on the program's progress against prior commitments. However, we found that these reports have also not fully addressed the commitments made in prior expenditure plans. For example, recent quarterly reports described progress against developmental milestones for each release, but did not report whether key functionality associated with each release was delivered. Our analysis of the fiscal year 2009 ACE expenditure plan shows that progress being reported against commitments in prior expenditure plans continues to be limited. For example, progress against cost and schedule commitments were not provided or were vague. Program officials stated that they intend to improve its progress reporting in their quarterly status reports.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures reporting in future expenditure plans of progress against commitments contained in prior expenditure plans.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

  3. Status: Closed - Implemented

    Comments: In May 2006, we reported that defects for the Automated Commercial Environment (ACE) program are tracked with two separate information systems: one for trouble tickets and another for problem trouble reports. Information about defects that are tracked in both systems may be inconsistent and needed to be manually reconciled to ensure consistency. As a result, the true status of ACE defects -- an important measure of system quality -- was not known. The Bureau of Customs and Border Protection has taken steps to ensure the reliability of defect data used to measure progress against quality commitments for ACE. Specifically, the ACE program office has integrated the two tools using a cross-referencing function and has instituted manual processes for reconciling the data in each tool. The December 2006 quarterly congressional report stated that these efforts have improved the ability to record, assess, and report on system quality and performance.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures reliability of data relevant to measuring progress against commitments.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

  4. Status: Closed - Not Implemented

    Comments: Automated Commercial Environment (ACE) program officials established an accountability framework that is used to inform both its annual expenditure plans and its quarterly congressional reports. However, as with prior expenditure plans, the fiscal year 2007/ 2008 expenditure plans did not reflect the most current program information. In particular, status information on milestones, earned value management (EVM) values, and risks was more than 3 months old when the plans were submitted to the appropriations committees. For example, the fiscal year 2007 expenditure plan listed the milestone for the Release 5/Drop A1 Production Readiness Review (PRR) as March 1, 2007, when this milestone had already slipped by more than 4 months (to July 12, 2007) a month before the plan was submitted to the appropriations committees. ACE program officials continue to use the quarterly ACE status reports to provide the appropriations committees with more current, relevant, and complete information about the program than could be provided in the expenditure plan. However, the quarterly reports were generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance. For example, the December 2006 quarterly report was not provided until February 26, 2007, and it contained the same outdated PRR milestone as the expenditure plan. Nevertheless, in its fiscal year 2008 ACE expenditure plan, DHS committed to providing future ACE quarterly reports within 45 days of the end of each quarter. Our review of the last three ACE quarterly reports showed that each was submitted more than 3 months after the end of the applicable quarter. A DHS official told us that intend to continue to work on implementing the recommendation.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures currency, relevance, and completeness of all such commitments made to the Congress in expenditure plans.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

  5. Status: Closed - Implemented

    Comments: Custom and Border Protection's Automated Commercial Environment (ACE) program has defined and implemented an accountability framework that covers program commitment areas and measures progress against costs, milestones, schedules, and risks for selected releases. In May 2007, we reported that the program office had developed an initial version of an accountability framework for measuring several program commitments, such as capabilities and milestones, but that other commitments, such as benefits, had not been as well defined. During the past year, the program office has completed its coverage of program commitments in key program documents--the ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE performance reference model, periodic Program Management Review (PMR) reports, and the fiscal year 2007 expenditure plan. The accountability framework now captures and integrates data on ACE functional and performance capabilities, benefits, and mission value, estimated and actual costs, milestones and schedule, accomplishments, and earned value management status. Further, the framework provides visibility into each ACE release at several levels of detail, and it is being used by the ACE Executive Director and others as the means for monitoring program progress, issues, and decisions. Data and examples from the accountability framework are included in both the periodic PMR reports and the fiscal year 2007 ACE expenditure plan.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures coverage of all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

  6. Status: Closed - Implemented

    Comments: The August 2006 version of the Automated Commercial Environment (ACE) program plan describes general roles and responsibilities for the government and the prime contractor. More detailed roles and responsibilities of Customs and Border Protection (CBP), the prime contractor, and the support contractors have been documented in a roles and responsibilities matrix that assigns primary responsibility for each activity, such as testing, training, and maintenance. In addition, the task orders describe the specific responsibilities and expectations of the contractors.

    Recommendation: To strengthen accountability for the ACE program and better ensure that future ACE releases deliver promised capabilities and benefits within budget and on time, the DHS Secretary, through the Under Secretary for Border and Transportation Security, should direct the Commissioner, Customs and Border Protection, to define and implement an ACE accountability framework that ensures clear and unambiguous delineation of the respective roles and responsibilities of the government and the prime contractor.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection

 

Explore the full database of GAO's Open Recommendations »

Sep 25, 2014

Sep 23, 2014

Jun 10, 2014

May 22, 2014

May 12, 2014

May 8, 2014

May 7, 2014

Apr 2, 2014

Looking for more? Browse all our products here