This is the accessible text file for GAO report number GAO-09-852R entitled 'Defense Logistics: Observations on Army's Implementation of the Logistics Modernization Program' which was released on July 8, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-09-852R: United States Government Accountability Office: Washington, DC 20548: July 8, 2009: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Ike Skelton: Chairman: The Honorable Howard McKeon: Ranking Member: Committee on Armed Services: House of Representatives: Subject: Defense Logistics: Observations on Army's Implementation of the Logistics Modernization Program: This letter formally transmits the enclosed briefing in response to House Report No. 110-652, which accompanied the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009 (Pub. L. No. 110-417). The House report directed the Secretary of the Army to submit a report on Logistics Modernization Program (LMP) implementation at Army depots and the expected end-state capabilities of LMP to the Senate Committee on Armed Services, the House Committee on Armed Services, and GAO by January 31, 2009. The House report also directed us to review the report submitted by the Secretary of the Army for completeness and to provide a report to the Senate Committee on Armed Services and the House Committee on Armed Services by March 31, 2009. On March 31, 2009, we provided the enclosed briefing to staff of your committees to satisfy this mandate. We are sending copies of this report to the appropriate congressional committees. We are also sending copies to the Secretary of Defense; the Deputy Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; and the Secretary of the Army. This report will also be available at no charge on our Web site at [hyperlink, http://www.gao.gov]. Please contact William M. Solis at (202) 512-8365 or solisw@gao.gov, Asif A. Khan at (202) 512-9859 or khana@gao.gov, or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov if you or your staff have questions on matters discussed in this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were J. Chris Martin, Senior Technologist; David Schmitt, Assistant Director; Evelyn Logue; Darby Smith; and Jim Melton. Signed by: William M. Solis: Director, Defense Capabilities and Management: Signed by: Asif A. Khan: Director, Financial Management and Assurance: Signed by: Nabajyoti Barkakati: Chief Technologist: Applied Research and Methods: Center for Technology and Engineering: Enclosure: [End of section] Enclosure: Army’s Implementation of the Logistics Modernization Program: Congressional Briefing: March 31, 2009: Introduction: In February 1998, the Army Materiel Command began the Logistics Modernization Program (LMP) to replace its existing material management systems—the Commodity Command Standard System and the Standard Depot System. These systems have been in operation for more than 30 years. LMP is intended to transform logistics operations in six core processes: order fulfillment, demand and supply planning, procurement, asset management, material management, and financial management. When fully implemented, LMP is expected to manage goods and services valued at about $40 billion, including 6 million Army inventory items. GAO has previously reported on problems the Army has encountered in its efforts to implement LMP.[Footnote 1] The specific problems related to: * data conversion errors that resulted in LMP containing inaccurate unit prices or units of issue for certain materials; * billing and collection errors whereby some customers were not billed for costs that should have been included in their bills and other customers were overbilled; * requirements management and testing process did not comply with best practices, and; * a lack of an independent verification and validation process. House Report 110-652 directed the Secretary of the Army to report to the House and Senate Armed Services Committees and GAO by January 31, 2009, on the Army’s efforts to implement LMP. The Army submitted its report to the House and Senate Armed Services Committees and GAO on October 21, 2008. In addition, the House report: * stipulated that the Army’s report on LMP address nine specific areas and; * directed GAO to review the Army’s report for completeness and to report to the House and Senate Armed Services Committees by March 31, 2009. The nine specific areas that the Army was to report upon were: 1) expected LMP capabilities at the levels of depot production, business operations and financial management, and Headquarters Army Materiel Command; 2) specific LMP capabilities implemented at each depot; 3) date of expected implementation at each depot; 4) description of how LMP will forecast future maintenance capacity and drive budgetary decisions; 5) percentage of workforce at each depot expected to be proficient on the system; 6) strategy to educate and train depot employees on system capabilities and the new business approach to resource planning and supply chain management as a result of LMP implementation; 7) detailed plan for ensuring that 100 percent of each depot’s operating files are loaded by the planned date of implementation at each depot; 8) leveraging of lessons learned from previous implementations; and; 9) detailed risk-mitigation strategy to support current production in the event that LMP implementation is not as successful as planned. Objectives: Our objectives were as follows: 1. Determine the extent to which the Army’s report addressed the nine specific areas identified in the House committee report. 2. Ascertain the specific actions the Army has taken, has under way, or has planned to address GAO’s prior recommendations pertaining to the Army’s implementation of LMP. 3. Evaluate the management processes and controls that the Army established to assess the feasibility of future LMP deployments. Summary of Observations: Our review and analysis of the Army’s October 21, 2008, report to the House and Senate Armed Services Committees found that the report provided the required information to address each of the nine specified areas and thereby satisfied the reporting requirement. (See page 10.) The Army has begun to implement virtually all of GAO’s prior recommendations related to LMP to address issues related to data conversion, billing and collection, requirements management and testing, and independent verification and validation. The effective implementation of these recommendations should help the Army to reduce its risk to acceptable levels. (See pages 11-15.) The Army has management processes and controls in place to assess the feasibility of future LMP deployments, enabling it to understand the risks associated with making a deployment decision and evaluate whether those risks are acceptable. (See pages 16-28.) * The Army recognizes that deploying a system such as LMP can never be a risk-free proposition and that problems will arise after deployment regardless of the actions taken prior to deployment. - Ability to go back to the legacy systems if needed. * The Army has a risk management approach in place to minimize the effects of those problems to provide reasonable assurance that depot operations in support of the warfighter are not significantly disrupted. - Enhanced help desk support to understand the “root cause” of problems identified by users and help those users immediately where possible. Background: The Army began efforts to implement LMP in 1999, with the intent of having the system fully deployed by June 2004. * LMP began operations in July 2003 at Tobyhanna Army Depot, the Communications-Electronics Command, the Army Materiel Command, and the Defense Finance and Accounting Service. No additional sites have been added since that time. * The Army will decide in April 2009 whether to have additional deployments of LMP. As of September 2008, the Army has obligated approximately $880 million for the development and implementation of LMP. Objective 1 Observations: Mandated Army Report Provided Required Information: Our review and analysis of the Army’s October 21, 2008, report to the House and Senate Armed Services Committees found that the report provided the required information to address each of the nine specified areas and thereby satisfied the reporting requirement. The Army’s report provided details on (1) expected LMP capabilities, (2) LMP capabilities implemented at depots, (3) date of expected implementation, (4) how LMP will forecast maintenance capacity and drive budgetary decisions, (5) percentage of workforce expected to be proficient, (6) strategy to educate employees on LMP capabilities, (7) a plan to ensure the depot’s operating files are loaded into LMP, (8) leveraging of lessons learned, and (9) a detailed risk-mitigation strategy in the event that LMP implementation is not successful. Objective 2 Observations: Prior GAO Recommendations: The Army has acted to implement virtually all of GAO’s prior recommendations related to LMP. The effective implementation of these recommendations should help the Army reduce its risk to acceptable levels. GAO made recommendations regarding problems we identified in the following areas: * Data conversion; * Billing and collection; * Requirements management and testing; * Independent verification and validation. Data conversion: * We previously reported that during the first deployment, LMP contained inaccurate unit price or unit of issue data for certain materials, resulting in excess material being ordered and incorrect prices being charged. Because of the inaccurate data, the Army received quantities of parts and supplies for use in repairing military assets that were far greater than intended. * When Tobyhanna converted from its legacy system to LMP in July 2003, the June 30, 2003, ending account balances in the legacy system did not reconcile to the beginning account balances in LMP. Accurate account balances are essential for financial reporting and preparation of future budgets. * The Army has since implemented a program to address the data conversion problems caused during the first deployment, and also has a process to evaluate whether its data conversion activities have reduced the risks associated with this critical factor to acceptable levels. Billing and collection: * We previously reported that shortly after LMP was implemented, Tobyhanna Army Depot began experiencing difficulty in accurately billing customers and recognizing revenue. * LMP’s billing problems resulted in (1) customers not being billed for costs incurred that should have been billed and (2) customers being billed too much. - As noted in a prior GAO report (GAO-07-860), 146 customer orders valued at approximately $5.4 million were not billed during the January 31, 2007, billing cycle. - Similarly, during the January 31, 2007, billing cycle about 308 customer orders amounting to about $5.8 million were shown as being over billed, and a credit should have been issued to the customer. - It is essential that revenue is properly recognized and customers are billed for work performed because these are the means by which the depots finance their day-to-day operations. Moreover, credits from the depots to the customers can be used by the customers to meet other funding demands. * In addition to improving its data conversion processes, the Army implemented technical and process improvements in December 2007. The Army reported that these improvements reduced the number of incorrect bills from 40 per cycle to zero. Because of time constraints, we did not independently verify these reported results. Requirements management and testing: * We previously reported that the Army did not have an adequate process for managing LMP requirements that complied with best practices. In 2007, we reported (in GAO-07-860) that the Army revised and improved this process. * We previously reported that the Army did not have an adequate testing process for LMP because the tests conducted by the Army were not tied to specific requirements, were not “end-to-end,” and were not performed by users of the system and because test results were neither properly documented nor verified and validated by an independent reviewer in accordance with best practices. * The Army has since revised its testing process, which we were unable to fully assess because of time constraints. However, we found that (1) users were conducting the test; (2) detailed test scripts were being used; (3) an automated system was capturing test results; and (4) test results were being evaluated daily by a group that consisted of users, the LMP program office, and an independent contractor, which increased the assurance that identified problems were properly evaluated. Independent verification and validation: * The Army has implemented an independent verification and validation process, which should help to identify to management the issues or weaknesses that increase the risks associated with the project. Objective 3 Observations: Improved Management Processes: The Army has implemented critical project management processes and controls to identify and understand the risks associated with making a deployment decision and evaluate whether those risks are acceptable before deploying LMP at Corpus Christi, Letterkenny, and Redstone Arsenal. These key processes were absent during the first deployment at Tobyhanna. * Improved training to address human capital needs; * Data conversion processes developed to provide reasonable assurance that critical data are properly converted; * Quantitative metrics to evaluate the effectiveness of the revised project management process have been developed; * “Go Live” processes developed to rapidly evaluate and respond to potential issues that will invariably arise during the second deployment. Figure: Employee Training Hours as of March 26, 2009: [Refer to PDF for image: stacked vertical bar graph] Actual: Corpus Christ Army Depot: 24,280 hours; Letterkenny Arsenal: 14,832 hours; Redstone Arsenal: 37,120 hours; Total: 76,232 hours. Future: Corpus Christ Army Depot: 20,556 hours; Letterkenny Arsenal: 19,704 hours; Redstone Arsenal: 71,592 hours; Total: 111,852 hours. Total: Corpus Christ Army Depot: 44,836 hours; Letterkenny Arsenal: 34,536 hours; Redstone Arsenal: 108,712 hours; Total: 188,084 hours. Source: GAO analysis of Army data. Notes: On March 19, 2009, LMP estimated that its total training hours would be greater than the amount shown in the above chart. This difference was caused by LMP reevaluating its course offerings and employee training assignments, primarily at Corpus Christi Army Depot. The LMP project office expects that the revised data will be loaded into the training system during the week ending April 3, 2009, and that the total number of employee training hours associated with the second deployment will be about 195,000 employee training hours. [End of figure] The majority of training (about 92 percent of the total employee hours) is expected to be delivered prior to deployment. Some training (e.g., budget stratification reporting) will occur at Redstone after deployment since the users will not be performing the functions designed to be addressed by the training until deployment has occurred, and Army officials explained that it is more effective to train an individual when he or she is close to performing the expected function. Training process includes a means to evaluate the effectiveness of the training program in delivering the expected skills. * Participants are tested and must achieve a 70 percent score to be considered as passing the class. * If a person does not achieve a 70 percent score, he or she may take other actions, such as taking the class again to achieve a passing score. - One-on-one help is also offered if the participant continues to experience problems. * Participants also provide feedback on whether they believe that the course has helped them do their jobs. - According to the Army, overall the participants have rated the training program as a 4.1 out of 5, with 5 being the highest score. LMP has conducted three data migration trial loads to identify issues associated with migrating data from the legacy systems to LMP. * At the end of February 2009, more than 95 percent, or about 12.5 million of the 13.1 million records, could be loaded into LMP without issue. * As of March 27, 2009, about 96 percent of the records loaded into LMP successfully processed without issue. For the remaining records, Army officials stated that they will either resolve the issues before deployment or take actions to mitigate any adverse effects. The Army recognizes that not all legacy data will be successfully loaded into LMP prior to deployment. * Historical data are needed for planning or other purposes. - A data warehouse is being created that contains the data in the legacy system that are not being converted--this is the same approach used at the initial deployment site and it has served the mission needs. - The legacy systems will remain available in an inactive state as long as the Army considers it necessary. * A review was made to determine which records were associated with work that is not expected to be performed at deployment locations during the next 3 years, and data conversion of these records will occur after deployment, if the need arises. * Data needed to support various LMP processes (commonly referred to as master data) are being carefully reviewed. - Preventive measures have been implemented to ensure that all master data items being loaded into the system have correct base unit of measure (BUOM) values. -- In the initial deployment, items were loaded into LMP with incorrect BUOM values. For example, part numbers were loaded into the system with a BUOM with a qualitative measure such as “box” rather than a quantitative measure such as “each.” - The Army developed a manual work-around process to provide reasonable assurance that items already contained in LMP with incorrect BUOM will not cause problems at the second deployment locations. -– LMP currently has about 329 items in its records that contain incorrect BUOM values. These errors caused reordering and billing problems at Tobyhanna: 121 of these items are used by Corpus Christi and Letterkenny; 72 are only used by Letterkenny; 25 are only used by Corpus Christi. -– Manual work-around processes developed at Tobyhanna have addressed the problems and will be used at the second deployment locations until a long-term system fix can be implemented. Objective 3 Observations: Improved Management Processes–Quantitative Metrics: The Army has defined the critical business processes that must be completed prior to making a decision on whether the deployment should “Go Live,” and has developed quantitative metrics to evaluate whether this has occurred. Examples of quantitative measures that are used include the following: * Validate that the critical business process testing has been completed and that any problems that have not been addressed have agreed-upon mitigation strategies. * Validate that the expert users can successfully execute the applicable processes--100 percent of the selected expert users are able to successfully execute their respective tasks. * Validate that end users have completed 90 percent of the required training courses. * Validate that the overall data accuracy rate is 95 percent, with each deployment location having accuracy of at least 90 percent. * Validate the interoperability, availability, and responsiveness of LMP. – Failures associated with exchanging data with other systems are less than 2 percent, with an objective of 1 percent; – Availability is at least 99.5 percent; – Average response time is less than 2 seconds for 80 percent of the transactions, with an objective of 2 seconds or less for 95 percent of the transactions. Objective 3 Observations: Improved Management Processes –“Post-Go Live” Processes: The Army recognizes that deploying a system such as LMP can never be a risk-free proposition and that problems will arise after deployment regardless of the actions taken prior to deployment. The Army has implemented processes designed to reduce the adverse impacts associated with those problems to acceptable levels. * End users experiencing problems will contact expert users at their location to help determine whether they are experiencing system problems or whether the users need help understanding how to perform a given task. - Help tickets are generated, regardless of the reason, to help the Army determine if additional training is needed or if the training needs to be modified. - When the training is modified, all users who have taken the course are notified of the changes. The Army has enhanced its help desk support to understand the “root cause” of problems identified by users and help those users immediately where possible. The Army is following best practices and will use a three-tier help desk system to address problems that cannot be addressed by the expert users at the deployment locations. * Tier 1: - Provides a single point of contact for end users; - Gathers the necessary data to assess the reported problem; - Determines if the problem can be resolved or escalates the problem to the next level; for example, the Tier 1 Help Desk (1) may be aware of a specific problem and can provide the user with the “work-around” that has been developed to address the problem or (2) can determine whether the system is operating as designed and provide the user with instructions on how to accomplish a specific transaction; - Verifies user satisfaction with the help desk process. * Tier 2 has more knowledgeable support personnel and determines if the problem can be resolved or escalates the problem to the next level. - May recommend changes to LMP based on the change management processes used by the program. * Tier 3 support is used for complex problems that cannot be addressed by the Tier 2 support or problems associated with vendor-supplied hardware and software. Agency Views: We provided a draft of these briefing slides to the Departments of Defense and the Army for their official review and comments. On March 30, 2009, we met with officials from the Defense Business Transformation Agency, the Office of the Secretary of Defense for Supply Policy, the LMP Program Office, and the Department of the Army to discuss our observations and obtain their comments on the briefing slides. The participants agreed with our observations and the information in the briefing slides. Methodology: To determine if the Army’s report addressed the nine specific areas identified in the House committee report, we reviewed the Army’s report and supporting documentation and compared them against the requirements. To ascertain the specific actions that the Army has taken to address GAO’s prior recommendations on the implementation of LMP, we reviewed past GAO reports related to LMP implementation. We also met with Army officials at the LMP Program Office to discuss the actions they either have implemented or have under way to address each recommendation, and we reviewed Army plans and policies that govern implementation. We also traveled to an Army depot to determine if the actions were being taken. To evaluate the management processes and controls that the Army has in place to ascertain if LMP should be deployed, we met with Army officials at the LMP Program Office and Corpus Christi Army Depot to determine the processes and controls they have in place. We also observed a training class being conducted for LMP users, and we analyzed Army data related to training. We conducted this performance audit from January 2009 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of enclosure] Footnotes: [1] GAO, DOD Business Systems Modernization: Billions Continue to Be Invested with Inadequate Management Oversight and Accountability, [hyperlink, http://www.gao.gov/products/GAO-04-615] (Washington, D.C.: May 27, 2004); Army Depot Maintenance: Ineffective Oversight of Depot Maintenance Operations and System Implementation Efforts, [hyperlink, http://www.gao.gov/products/GAO-05-441] (Washington, D.C.: June 30, 2005); and DOD Business Transformation: Lack of an Integrated Strategy Puts the Army’s Asset Visibility System Investments at Risk, [hyperlink, http://www.gao.gov/products/GAO-07-860] (Washington, D.C.: July 27, 2007). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: