This is the accessible text file for GAO report number GAO-13-231 entitled 'Acquisition Workforce: Federal Agencies Obtain Training to Meet Requirements, but Have Limited Insight into Costs and Benefits of Training Investment' which was released on April 16, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: March 2013: Acquisition Workforce: Federal Agencies Obtain Training to Meet Requirements, but Have Limited Insight into Costs and Benefits of Training Investment: GAO-13-231: GAO Highlights: Highlights of GAO-13-231, a report to congressional requesters. Why GAO Did This Study: The acquisition workforce manages and oversees billions of dollars in acquisition programs and contracts to help federal agencies get what they need, at the right time, and at a reasonable price; therefore, it is important that agencies provide adequate training to this workforce. In this review, GAO identified (1) the role of OFPP and the FAI in assisting agencies in meeting certification requirements; (2) agencies’ approaches to providing training; and (3) the extent to which agencies collect information on the costs and benefits of their acquisition training. To determine OFPP and the FAI roles, GAO analyzed relevant legislation. GAO obtained information from 23 federal agencies on their training approaches through a questionnaire, and selected 4 agencies—-the Departments of Education and the Treasury, DHS, and VA-—to provide illustrative examples. GAO used its questionnaire and a subsequent data call to obtain information on how agencies collect information on the costs and benefits of their training. What GAO Found: The Office of Federal Procurement Policy (OFPP) sets standards and policies for the federal acquisition workforce, and has established certification requirements, including minimal training, for the three main acquisition roles—-contracting staff, Contracting Officer’s Representatives, and Program/Project Managers—-to promote the development of government-wide core acquisition competencies and facilitate mobility across agencies. DOD follows separate certification standards. The Federal Acquisition Institute (FAI), which is responsible for fostering and promoting the training and development of the acquisition workforce, works closely with OFPP and has initiatives underway to improve the collection and management of training information, including cost data and course evaluations; streamline communication of acquisition training guidance; and coordinate efforts to leverage acquisition workforce training resources throughout the government. To support efforts for the acquisition workforce to attain and maintain federal certification requirements, most agencies (17 of 23) provide the majority of their acquisition training using external sources—-vendors, FAI, the Defense Acquisition University, or other agencies. The Departments of Homeland Security (DHS), the Treasury, and Veterans Affairs (VA) operate their own permanent centers with dedicated resources that train the agency’s acquisition workforce. Education reported using a different approach to providing acquisition training—it gets most of its training from other government entities. Federal agencies face similar challenges in providing training to their acquisition workforce. The top challenges reported by agencies in obtaining training for their acquisition workforce involved having sufficient resources. Twenty of 23 agencies identified obtaining adequate funding and 19 of 23 identified obtaining sufficient staff to manage training as challenging. In addition, almost half of the agencies reported that the fundamental step of identifying the acquisition workforce is a challenge especially when members of the workforce are involved in acquisitions as a secondary and not primary duty. The training cost data that agencies collect is not comparable and agencies have limited information on the benefits of their acquisition workforce training investments. Although almost all agencies provided some cost data in response to GAO’s questionnaire and subsequent data call, the agencies’ cost data did not allow for government-wide assessment of their training investment. To date, FAI’s efforts to collect training cost data has also met with limited success. Cost data collected in 2012 by GAO and FAI included different cost components-—such as facilities, travel, and instructors—-which do not allow for government-wide analysis. Having comparable training cost data are important to inform FAI efforts to establish government-wide contracts for training. As for determining benefits of training, 7 of 23 federal agencies reported having no metrics, not even basic end-of- course evaluations. Without basic data, agencies do not have insight into the benefits of their acquisition workforce training efforts. What GAO Recommends: GAO recommends that OFPP help ensure that agencies collect and report comparable cost data and assess the benefits of acquisition training by (1) providing further guidance on the cost data agencies are to report annually, and (2) requiring agencies to analyze course evaluations, at a minimum, to help assess the benefits of training investments. OFPP concurred with the recommendations and indicated it has started to take actions to provide additional guidance. View [hyperlink, http://www.gao.gov/products/GAO-13-231]. For more information, contact Belva M. Martin at (202) 512-4841 or martinb@gao.gov. [End of section] Contents: Letter: Background: OFPP and FAI Provide Policy, Guidance, and Assistance to Help Improve Government-wide Training Efforts: Agencies Generally Use External Sources for Acquisition Training and Face Similar Workforce Training Challenges: Lack of Comparable Cost Data and Limited Insights On Benefits of Training Hinder Efforts to Maximize Resources Government-wide: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Summary of Agencies' Response to GAO Questionnaire on Civilian Acquisition Workforce Training: Appendix II: Objectives, Scope, and Methodology: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Key Agency Officials Involved in Managing Acquisition Workforce Training: Table 2: OFPP's Legislative Responsibilities Related to Acquisition Workforce Training: Table 3: Percentage of Acquisition Workforce with Federal Acquisition Certifications by Category--Fiscal Year 2011: Table 4: Interagency Councils Involved in Oversight and Management of Acquisition Workforce Training: Table 5: Leading Training Investment Practices Agencies Should Implement: Figures: Figure 1: Agency-reported Sources Used for Acquisition Workforce Training in Fiscal Year 2011: Figure 2: Number of Agencies That Reported Training Source Selections Factors: Figure 3: Challenges Reported By Agencies in Obtaining Acquisition- Specific Training Resources: Figure 4: Challenges Reported by Agencies in Staff Attending Training: Abbreviations: ACMIS: Acquisition Career Management Information System: AHCP: Acquisition Human Capital Plan: CAO: Chief Acquisition Officer: COR: Contracting Officer's Representative: DAU: Defense Acquisition University: DHS: Department of Homeland Security: DOD: Department of Defense: FAC: Federal Acquisition Certification: FAI: Federal Acquisition Institute: FAITAS: FAI Training Application System: OFPP: Office of Federal Procurement Policy: P/PM: Program/Project Manager: VA: Department of Veterans Affairs: [End of section] United States Government Accountability Office: Washington, DC 20548: March 28, 2013: Congressional Requesters: Constrained budgets and the need to address gaps in critical federal workforce skills point to the need for federal agencies to ensure their training and development programs are effective, in order to provide personnel with the skills essential to fulfilling their agencies' missions. The shortage of trained acquisition personnel hinders agencies from managing and overseeing acquisition programs and contracts that have become more expensive and increasingly complex. As a result, the federal government is at risk for significant overcharges and wasteful spending of the billions of dollars it spends for goods and services each year. For fiscal year 2011, 23 Chief Financial Officers Act agencies had about 83,000 federal employees who managed and oversaw over $160 billion in acquisitions.[Footnote 1] The Office of Management and Budget's Office of Federal Procurement Policy (OFPP), the Federal Acquisition Institute (FAI), and the executive agencies play important roles in helping to ensure that acquisition workforce professionals meet federal training standards and that training dollars are used effectively. The acquisition workforce plays a key role in managing programs and overseeing contracts to help agencies get what they need, at the right time, and at a reasonable price. Accordingly, you asked us to evaluate approaches of federal agencies, other than the Department of Defense (DOD), to train the acquisition workforce. We reviewed the efforts of 23 federal agencies to provide training to the acquisition workforce, as well as the efforts of OFPP and FAI to support this training throughout the government. Specifically, this report addresses the (1) role of OFPP and FAI in promoting Federal Acquisition Certification standards and assisting agencies in meeting acquisition workforce training requirements, (2) approaches agencies use to provide training to their acquisition workforces, and (3) the extent to which agencies collect information on the costs and benefits of their acquisition training. You were also interested in the grants workforce, which we will report on separately. To identify the roles of OFPP and FAI in promoting Federal Acquisition Certification standards, we analyzed legislation on their authority and relevant guidance they provided to agencies, and interviewed their officials. To provide government-wide information on federal agencies' approaches to providing acquisition workforce training, we administered a questionnaire to 23 agencies from July to August 2012 to obtain data about their current acquisition-specific training approaches, including the sources used for courses (in particular whether courses were provided by the agency itself or by external sources, such as a commercial sector vendor, FAI, the Defense Acquisition University (DAU) or another agency), facilities, budgetary and staffing resources, methods of training, and challenges faced in providing acquisition training.[Footnote 2] We received questionnaire responses from all 23 agencies, although not every agency responded to every question. One agency provided responses from its headquarters procurement office and three bureaus. We consolidated the information from the four surveys to ensure the tabulations eliminated duplicative counting of the agency and to provide averaged data, as appropriate. A summary of the agencies' responses to each question is provided in appendix I. To determine the information agencies gather on the costs and benefits of their training approaches, we supplemented the questionnaire responses by soliciting cost and metrics data from the agencies using a data collection instrument from October to November 2012. We also reviewed the agencies' most recent Acquisition Human Capital Plans (AHCP), dated March 2012. To provide further insight into agencies' training approaches and tracking of cost and effectiveness data, we selected the Departments of Education (Education), Homeland Security (DHS), the Treasury (Treasury), and Veterans Affairs (VA) to provide illustrative examples. DHS, Treasury, and VA are agencies that operate dedicated acquisition workforce training facilities--permanent centers with dedicated resources that provide training specifically for the agency's acquisition workforce. Education is a smaller agency that has no dedicated training facility for its acquisition workforce. We also discussed the extent to which these agencies were aware of and had applied leading training investment practices reported in our prior work, which the Office of Personnel Management and experts had agreed should be implemented by agencies to support effective training investment decisions.[Footnote 3] Of the eight leading practices GAO identified, we focused on four dealing primarily with determining the costs and effectiveness of training: (1) identifying the appropriate level of investment to provide for training and development efforts and prioritizing funding so that the most important training needs are addressed first; (2) having criteria for determining whether to design training and development programs in-house or obtain these services from a contractor or other external source; (3) tracking the cost and delivery of training and development programs agency-wide; and (4) evaluating the benefits achieved through training and development programs, including improvements in individual and agency performance. Further details of our objectives, scope, and methodology are provided in appendix II. We conducted this performance audit from February 2012 to March 2013, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: The Office of Personnel Management, as the federal government's human capital manager, provides leadership and guidance on establishing and operating efficient federal training and development programs throughout the government. It provides advice and assistance to agencies on training and development programs so that those programs support strategic human capital investment. OFPP has specific responsibilities related to acquisition workforce training to help ensure that the skills needed to handle the complexities of acquisition programs and contracts are maintained. The Office of Federal Procurement Policy Act requires executive agencies and the OFPP Administrator to establish management policies and procedures for the effective management of the acquisition workforce-- including education, training, and career development.[Footnote 4] In 2005, OFPP defined the civilian acquisition workforce to include, at a minimum, professionals serving in the contracting series (GS-1102), Contracting Officers, Purchasing series (GS-1105), Program/Project Managers (P/PM), and Contracting Officer's Representatives (COR) or the equivalent, and additional positions identified by the agency. [Footnote 5] Participation in the acquisition workforce may be on a full-time, part-time, or occasional basis. OFPP, working through FAI, is also required to foster and promote the development of government-wide training standards and certifications for the acquisition workforce and to perform a range of activities supporting management of this workforce.[Footnote 6] FAI's activities are funded through the Acquisition Workforce Training Fund, which is financed by deposits in an amount equal to 5 percent of the dollar amount of acquisitions made by civilian agencies under the Federal Supply Schedules and Governmentwide Acquisition Contracts.[Footnote 7] FAI, under the guidance of its Board of Directors, works closely with OFPP as well as the DAU to collaborate on many issues involved with training and developing the federal acquisition workforce. The Defense Acquisition Workforce Improvement Act required the Secretary of Defense to establish and maintain DAU in order to provide training for the DOD acquisition workforce.[Footnote 8] FAI differs notably from DAU, however, in that it shares much of the responsibility for training the civilian acquisition workforce with the agencies and has fewer budgetary and staffing resources. For example, FAI currently has 12 staff, while DAU has over 700 faculty and staff. The Administrator of OFPP is also required to ensure that agencies collect and maintain standardized information on the acquisition workforce. Agencies, in consultation with OFPP, can structure their management and oversight of training for their acquisition workforce in such a manner as best supports the agency. Key agency positions involved in managing the acquisition workforce within the agencies are described in table 1. Table 1: Key Agency Officials Involved in Managing Acquisition Workforce Training: Position: Chief Acquisition Officer (CAO); Acquisition Training-related Responsibilities: Responsible for developing and maintaining an agency acquisition career management program to ensure an adequate professional acquisition workforce. Participates in the strategic planning and performance evaluation process to develop strategies and specific plans for training to rectify deficiencies in the skills and abilities of the acquisition workforce. Responsible, in consultation with the Acquisition Career Manager, for reporting acquisition workforce data to the Human Capital Office for the agency's human capital plan. Position: Senior Procurement Executive; Acquisition Training-related Responsibilities: Responsible for providing management direction of an agency's procurement system. In some agencies, CAOs oversee or assume this role to help ensure implementation of agency acquisition workforce policy. Responsible for identifying training and development needs of the acquisition workforce. Position: Acquisition Career Manager; Acquisition Training-related Responsibilities: Leads an agency's acquisition career management program and ensures that its acquisition workforce meets the skills and capabilities required in OMB, OFPP, and agency policies. Source: GAO summary of relevant sections of title 41 of the U.S. code and OMB Policy Letter 05-01, Developing and Managing the Acquisition Workforce. [End of table] Agencies have several methods to choose from when providing training courses for their acquisition workforce. In general, acquisition workforce training is categorized by: (1) agency or agency-sponsored training provided by the agency, which may be taught using in-house staff or contractor employees; (2) government entity training provided by another federal agency, FAI, or the DAU; and (3) vendor training provided by a commercial sector company, which is generally available to multiple agencies at the same time. For the purposes of this report we refer to government entity and vendor training as external sources. OFPP and FAI Provide Policy, Guidance, and Assistance to Help Improve Government-wide Training Efforts: OFPP sets standards and policies for the training and development of the federal acquisition workforce. Its efforts include strengthening workforce planning requirements and setting standards for core acquisition training by establishing certification requirements. FAI conducts activities that support and assist civilian agencies in training and development of their acquisition workforces. FAI efforts include: improving the collection and management of training information, including cost data and course evaluations; streamlining the communication of acquisition training guidance; and coordinating efforts to maximize acquisition workforce training investments government-wide. OFPP Promotes Government-wide Practices for Acquisition Workforce Training and Development: In carrying out its responsibilities, OFPP's activities have focused on improving the training and development of the acquisition workforce across the government. To promote the development of core acquisition competencies throughout the government and facilitate employee mobility across civilian agencies, OFPP, in 2005, required the development of Federal Acquisition Certification (FAC) requirements for acquisition professionals serving as contracting staff (including Contracting Officers), CORs, and P/PMs. The certification programs require a minimum set of career-specific courses, along with education and experience requirements, to obtain certification, and continuous learning requirements to ensure professionals remain current on acquisition policies and practices.[Footnote 9] In addition, since fiscal year 2010, OFPP has required agencies to complete an annual AHCP as part of their annual budget planning process to elevate acquisition workforce planning from a short-term activity to provide input into the agency's strategic plan to a continued effort to guide the overall growth and development of the acquisition workforce. OFPP officials stated that they use the data provided in the agencies' plans to develop a government-wide snapshot of the acquisition workforce and to identify practices for sharing throughout the government. Table 2 provides an overview of OFPP's legislative responsibilities related to acquisition workforce training. Table 2: OFPP's Legislative Responsibilities Related to Acquisition Workforce Training: Legislative Responsibilities Related to Training: * Issuing policies to promote uniform implementation by executive agencies of their acquisition workforce policies and procedures. * Recommending to the Administrator of General Services Administration that sufficient budget resources and authority go to FAI so it can support governmentwide training standards and certification requirements to enhance the mobility and career opportunities of the federal acquisition workforce. * Acting through the FAI, providing and updating governmentwide training standards and certification requirements. * Establishing a Board of Directors to provide oversight of FAI and its efforts to support the acquisition workforce. * Acting through the FAI, coordinating with the Office of Personnel Management on human capital efforts. Source: GAO summary of federal law. [End of table] FAI Activities Help Agencies Monitor and Train the Acquisition Workforce: FAI's mission is to foster and promote the development of a professional acquisition workforce government-wide. However, FAI was not established, nor does it have the budgetary or staffing resources, to meet the acquisition workforce training needs of the entire civilian acquisition workforce, according to FAI officials. FAI's activities during the last few years are aimed at improving the collection of training information to better inform government-wide oversight, streamlining the communication of acquisition guidance and updates on initiatives, and coordinating efforts to maximize acquisition workforce training investments throughout the government. Since FAI's initiatives are in various stages of implementation, we did not evaluate their effectiveness; however, in several instances, we note our initial observations of these efforts. The main efforts and initiatives FAI is pursuing to fulfill its training-related legislative responsibilities include: * Fostering and promoting the development of a professional acquisition workforce throughout the government[Footnote 10]. According to officials, FAI develops and provides courses, through commercial vendors, that support the FAC program requirements. It is coordinating with DAU to determine if existing defense acquisition course materials can be modified for training the civilian acquisition workforce. FAI reported that while its spending plan for fiscal year 2012 could provide $3 million for courses, training requests submitted by the agencies totaled in excess of $18 million. FAI is working with a number of vendors who provide acquisition workforce training to establish government contracts that would provide uniform costs for standardized training courses needed for the FAC programs. FAI plans to have contracts in place in 2013 that would be available for agencies to use to provide fiscal year 2014 training. * Assisting federal agencies with their acquisition human capital planning efforts[Footnote 11]. FAI offers guidance and direction on human capital planning through its instructions for agencies' annual AHCP submissions. Beginning in 2010, OFPP has required agencies to submit an annual AHCP each March that provides the agency's strategies and goals for increasing both the capacity and capability of the acquisition workforce. To assist the agencies in preparing each submission, FAI provides them with a report template that includes specific topics to be included. We observed that the fiscal year 2012 AHCP submissions provided information on the agencies' goals for strengthening the acquisition workforce, including certification goals, and planned training initiatives for the current and future fiscal years. FAI has an effort underway to develop an inventory of acquisition courses to identify duplicate courses provided by the agencies using the AHCP submissions. The guidance for the fiscal year 2012 AHCP included a request to complete a template with data, including cost, on each course provided or being developed by the agencies. For the fiscal year 2013 AHCP submissions, which are due March 2013, FAI officials distributed the report templates to coincide with the budget cycle to help agencies inform their budget process. * Collecting and analyzing acquisition workforce data from the Office of Personnel Management, heads of executive agencies, and individual employees[Footnote 12]. FAI recently upgraded the government-wide acquisition career information management system it maintains for all agencies. In 2011, FAI replaced the Acquisition Career Management Information System (ACMIS) with the FAI Training Application System (FAITAS), which allows registration for courses, tracking of individuals' training records, and other information management tools. For example, agencies can now manage the certification approval process, track an individual's continuous learning progress, and search for courses offered by FAI and other agencies that use FAITAS to enroll participants in their courses. FAI reports that five agencies currently use the FAITAS registration function to alert other agencies to unfilled seats within a specific agency's training course. Additional features of the system include a course scheduling module; a business intelligence tool for agencies to identify training locations and course availability; and a communication tool for broader outreach to the acquisition workforce. For example, an OFPP official explained that FAITAS can be used to send e-mail updates on OFPP policy, guidance, or other OFPP or FAI initiatives to all individuals registered in the system. FAI conducts periodic surveys of agencies to collect data about the acquisition workforce for OFPP. For example, FAI conducts the bi- annual Acquisition Workforce Competency Survey to collect information on agency acquisition workforces' skills and abilities and to identify competency gaps within and across agencies. FAI officials explained that the survey results provide a government-wide view of which courses are in demand that informs their efforts to achieve economies of scale when providing training. FAI reports an annual count of the acquisition workforce as it has done for more than the past decade; however, its count has been limited to professionals serving in four job series--General Business and Industry (GS-1101), Contracting (GS-1102), Purchasing (GS-1105), and Procurement Clerical and Assistance (GS-1106). FAI began gathering data on CORs and P/PMs as part of the acquisition workforce count in fiscal year 2007. In its recent report on the acquisition workforce count, FAI reported that 33,271 personnel were employed in the acquisition workforce's four main job series in fiscal year 2010, and an additional 47,959 and 4,186 personnel were identified as CORs and P/PMs, respectively, by the agencies in their AHCPs--which represents about 85,000 individuals in the civilian acquisition workforce for fiscal year 2010. FAI does not directly collect data for the workforce count, and has acknowledged that agencies have some problems identifying CORs and P/PMs but the data have been improving. * Facilitating interagency intern and training programs, as requested by agencies[Footnote 13]. According to officials, FAI also conducts outreach and leverages training courses government-wide. In 2012, FAI met with officials from each agency to discuss training efforts and locate training spaces that could be used by FAI and others when supplying courses through vendors. FAI officials reported that these agency visits provided insights regarding how the agencies' training programs varied in terms of their organization and resources and identified duplicative training efforts. In addition, FAI provides opportunities to share information across the acquisition workforce community. FAI began providing webinars on current acquisition issues in fiscal year 2012, and launched a newsletter for the acquisition workforce community in December 2012. According to officials, the newsletter will provide more in-depth information on policy changes, human capital initiatives, and tools and technology enhancements. FAI hosts, coordinates, and participates in roundtables at the Interagency Acquisition Career Management Committee meetings to foster discussion among agencies and share information on their training or educational challenges and needs. According to FAI officials, these meetings provide opportunities to share leading practices, identify challenges, and discuss potential initiatives. Recent discussion topics included plans to hold a competency-based certification workshop, roll out of additional FAITAS workforce management tools, and results of FAI's bi-annual Acquisition Workforce Competency Survey. In July 2012, FAI established the Federal Acquisition Council on Training to help share information on agencies' training efforts and standardize acquisition training throughout the government. According to FAI officials, the council's goal is to help ensure no acquisition workforce training seat, whether offered by FAI or another agency, goes unfilled. Agencies are asked to open up unfilled training seats to other agencies whenever available, and to use FAITAS as the official course registration system to communicate available training seats and enroll participants for courses to reduce overhead and administrative costs for agencies. * Periodically analyzing acquisition career fields and evaluating the effectiveness of training and career development programs for acquisition personnel[Footnote 14]. According to officials, FAI has efforts underway to standardize the end-of-course participant evaluations administered by the vendors who provide FAI-sponsored courses. According to FAI officials, course participants will not receive their course completion certifications until they complete the course evaluation. Currently FAI does not analyze course evaluations administered by the vendors; however, officials commented that the use of standardized evaluations could enable them to work with vendors to improve the consistency of the information provided in training courses. A pilot is underway using standard evaluations for FAI courses taught by vendors being provided to one agency, and FAI plans to extend the pilot to other agencies early in 2013. Agencies Generally Use External Sources for Acquisition Training and Face Similar Workforce Training Challenges: Most agencies approach acquisition workforce training through classroom courses taught by external sources--vendors, FAI, DAU, or other agencies. While all agencies have meeting spaces for training, three operate permanent centers with dedicated resources that train the agency's acquisition workforce. The agencies' current training focus is to provide courses through which their acquisition workforces may attain or maintain their FAC certifications. Agencies reported facing several challenges in providing acquisition-related training. The areas reported as being the most challenging are related to staffing and budgetary resources. Some agencies also reported challenges with the identification of their acquisition workforce, which is a fundamental step needed for managing the workforce and its training. Agencies also reported that additional assistance from OFPP and FAI would help their acquisition workforce training efforts. In addition, the agencies reported that their acquisition workforces are challenged in finding time in their workload to attend training. For more details on the approaches, budgetary and staffing resources, and other challenges faced by the agencies, see appendix I for a summary of the 23 agencies' responses to the questionnaire we administered. Agencies Generally Rely on External Training Sources: Agencies provide acquisition workforce training predominantly through external sources--government entities or vendors. In fact, 17 agencies reported that the majority of their acquisition workforce training comes from having their workforce attend training provided by external sources rather than from their agency. The other government entities may provide training to an agency without seeking additional reimbursement, as is the case with FAI, DAU, and some federal agencies; however, others, such as the VA, charge a fee to attend their training to recoup their costs.[Footnote 15] Of the remaining 6 agencies, 5 reported that they hold agency-sponsored courses to provide the majority, if not all, of their acquisition workforce training, and one agency did not report. Some agencies use contractor personnel to instruct a portion of their agency-sponsored training. Figure 1 illustrates the sources of training that agencies reported using to provide training to their acquisition workforce in fiscal year 2011. Figure 1: Agency-reported Sources Used for Acquisition Workforce Training in Fiscal Year 2011: [Refer to PDF for image: stacked horizontal bar graph] Agency: Agriculture; Agency: 5%; Vendor: 15%; Government entity (another agency, FAI, DAU): 80%. Agency: Commerce; Agency: 9%; Vendor: 40%; Government entity (another agency, FAI, DAU): 51%. Agency: Education; Agency: 10%; Vendor: 65%; Government entity (another agency, FAI, DAU): 25%. Agency: Energy; Agency: 0%; Vendor: 55%; Government entity (another agency, FAI, DAU): 45%. Agency: Environmental Protection Agency; Agency: 6%; Vendor: 7%; Government entity (another agency, FAI, DAU): 87%. Agency: General Services Administration; Agency: 51%; Vendor: 15%; Government entity (another agency, FAI, DAU): 34%. Agency: Health and Human Services; Agency: 10%; Vendor: 35%; Government entity (another agency, FAI, DAU): 55%. Agency: Homeland Security; Agency: 29%; Vendor: 0%; Government entity (another agency, FAI, DAU): 71%. Agency: Housing and Urban Development; Agency: 100;% Vendor: 0%; Government entity (another agency, FAI, DAU): 0%. Agency: Interior; Agency: 5%; Vendor: 18%; Government entity (another agency, FAI, DAU): 77%. Agency: Justice; Agency: 8%; Vendor: 22%; Government entity (another agency, FAI, DAU): 70%. Agency: Labor; Agency: 20%; Vendor: 5%; Government entity (another agency, FAI, DAU): 75%. Agency: National Aeronautics and Space Administration; Agency: 90%; Vendor: 0%; Government entity (another agency, FAI, DAU): 10%. Agency: Nuclear Regulatory Commission; Agency: 75%; Vendor: 15%; Government entity (another agency, FAI, DAU): 10%. Agency: Office of Personnel Management; Agency: 5%; Vendor: 10%; Government entity (another agency, FAI, DAU): 85%. Agency: Small Business Administration; Agency: 0%; Vendor: 9%; Government entity (another agency, FAI, DAU): 91%. Agency: Social Security Administration; Agency: 0%; Vendor: 90%; Government entity (another agency, FAI, DAU): 10%. Agency: State; Agency: 32%; Vendor: 45%; Government entity (another agency, FAI, DAU): 23%. Agency: Transportation; Agency: 14%; Vendor: 35%; Government entity (another agency, FAI, DAU): 68%. Agency: Treasury; Agency: 15%; Vendor: 40%; Government entity (another agency, FAI, DAU): 45%. Agency: U.S. Agency for International Development; Agency: 20%; Vendor: 20%; Government entity (another agency, FAI, DAU): 60%. Agency: Veterans Affairs; Agency: 100%; Vendor: 0%; Government entity (another agency, FAI, DAU): 0%. Source: GAO analysis of agencies' responses to questionnaire. Notes: Agency training includes courses taught by agency staff or contractor personnel. The National Science Foundation did not respond to this question in GAO's questionnaire. The Department of Transportation provided data that does not total 100 percent due to its averaging of the responses from its bureaus. The Department of Justice data is an average of the responses submitted by 4 procurement offices--headquarters and 3 bureaus. [End of figure] A leading training investment practice involves agencies taking steps to identify the appropriate level of investment to provide for training and development efforts and to prioritize funding so that the most important training needs are addressed first.[Footnote 16] The four agencies we selected illustrate that agencies use different approaches to manage and provide acquisition workforce training, due to such factors as the size of their workforce, the need for certification training, and the resources dedicated to training. Three agencies--DHS, Treasury, and VA--chose to have permanent training facilities dedicated to providing courses to the acquisition workforce. These three agencies differed, however, in the percentage of courses they provided to meet their acquisition workforce needs. DHS manages its entire acquisition workforce training at the department level and recently expanded the number of courses provided by the agency. However, part of DHS's strategy is to make use of DAU and FAI courses that address its needs, which can be obtained at no additional charge. VA also manages all of its training at the department level and in 2008 established a training academy, the VA Acquisition Academy, to provide all acquisition-related courses to agency personnel who need it. Alternately, Treasury built its training program on an existing training institute within the Internal Revenue Service, which employs the majority of Treasury's acquisition workforce, to provide agency courses available to all of its bureaus and to other agencies. Of the three dedicated training facilities, VA is the only one to provide agency-sponsored training for 100 percent of its acquisition training courses. VA, DHS, and Treasury all reported using contractors to instruct the agency-provided training at their respective facilities. Education does not have a dedicated facility and reported that it provides only about 10 percent of its acquisition workforce training via agency-sponsored courses, obtaining about 90 percent of needed training through external sources. The 23 agencies reported that several factors influence the selection of the sources of training they use. These include the effectiveness of the training course, the demand for specific courses, the level of budgetary resources available, and the availability of specific courses to meet training needs. In their responses to our questionnaire, the agencies did not explain how effectiveness is determined. As we discuss later in this report, based on a subsequent data call we found that the agencies have limited insight into the effectiveness of their acquisition workforce training courses. Figure 2 illustrates the factors and number of agencies that reported them as influencing their training source selections. Figure 2: Number of Agencies That Reported Training Source Selections Factors: [Refer to PDF for image: vertical bar graph] Effectiveness of training method: Number of agencies: 20. Cost effectiveness: Number of agencies: 15. Demand for specific courses: Number of agencies: 15. Budgetary resources: Number of agencies: 13. Course already available: Number of agencies: 12. Staffing resources: Number of agencies: 5. Source: GAO analysis of agencies' responses to questionnaire. Note: The 23 agencies could select more than one factor in response to our questionnaire. [End of figure] Agencies' current focus is to provide courses that allow their acquisition workforces to attain or maintain their FAC certification. In response to our questionnaire, most agencies--17 of 23--reported that they are able to find sufficient courses to do this. In their AHCP submission for fiscal year 2011, most agencies (15) reported certification rates of more than 75 percent for their contracting staff, but 3 agencies reported over 90 percent of their contracting staff were certified. The majority of the 23 agencies reported that more than 90 percent of their CORs and P/PM staff had obtained their basic FAC certifications--with many reaching 100 percent. Agency officials explained that the agency's certification rates fluctuate due to changes in the workforce--including experienced staff leaving the agency and less experienced staff being in the process of obtaining certifications. Table 3 shows the certification rates reported by agencies for fiscal year 2011. Table 3: Percentage of Acquisition Workforce with Federal Acquisition Certifications by Category--Fiscal Year 2011: Agriculture; Contracting: 84%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Commerce; Contracting: 62%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Education; Contracting: 95%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Energy; Contracting: not reported; Contracting Officer Representative: not reported; Program/Project Manager: not reported. Environmental Protection Agency; Contracting: 80%; Contracting Officer Representative: 100%; Program/Project Manager: 91%. General Services Administration; Contracting: 79%; Contracting Officer Representative: 94%; Program/Project Manager: 100%. Health and Human Services; Contracting: 40%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Homeland Security; Contracting: 86%; Contracting Officer Representative: 100%; Program/Project Manager: 94%. Housing and Urban Development; Contracting: 76%; Contracting Officer Representative: 96%; Program/Project Manager: 29%. Interior; Contracting: 77%; Contracting Officer Representative: 91%; Program/Project Manager: 7%[A]. Justice; Contracting: 74%; Contracting Officer Representative: 89%; Program/Project Manager: 57%. Labor; Contracting: 80%; Contracting Officer Representative: 89%; Program/Project Manager: 100%. National Aeronautics and Space Administration; Contracting: 84%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Nuclear Regulatory Commission; Contracting: 76%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. National Science Foundation; Contracting: 71%; Contracting Officer Representative: 100%; Program/Project Manager: n/a. Office of Personnel Management; Contracting: 95%; Contracting Officer Representative: 95%; Program/Project Manager: 100%. Small Business Administration; Contracting: 74%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Social Security Administration; Contracting: 76%; Contracting Officer Representative: 24%[A]; Program/Project Manager: 100%. State; Contracting: 84%; Contracting Officer Representative: 100%; Program/Project Manager: 100%. Transportation; Contracting: 91%; Contracting Officer Representative: 47%; Program/Project Manager: 58%. Treasury; Contracting: 65%; Contracting Officer Representative: 99%; Program/Project Manager: 100%. U.S. Agency for International Development; Contracting: 40%; Contracting Officer Representative: 95%; Program/Project Manager: 100%. Veterans Affairs; Contracting: 77%; Contracting Officer Representative: 5%[A]; Program/Project Manager: 100%. Source: Agency's Fiscal Year 2011 Acquisition Human Capital Plans. Note: [A] In discussions, VA officials stated that the 5 percent submission for the Acquisition Human Capital Plan was based on an estimate of the number of CORs at that time, and the agency has since improved its method of counting the number of CORs. Further, VA officials stated that the variation in certification rates among agencies may be reflecting different definitions of the workforce. Social Security Administration noted it began its COR training program 2011. Interior noted it began its P/PMs training program in 2012. [End of table] Officials from three of the four selected agencies--DHS, Treasury, and VA--explained that they prioritize their training funding based on the need for acquisition staff to be certified. Education, where the majority of the acquisition workforce has attained their certifications already, reported it is focusing on courses that meet staff needs to maintain certifications or develop expertise. Agencies Face Challenges with Having Adequate Training Resources, Identifying the Acquisition Workforce, and Staff Having Time to Attend Training: The top challenges reported by agencies in obtaining training for their acquisition workforces were having sufficient resources--both staffing to manage the training program and budgetary resources--to provide training. Specifically, 20 agencies reported that obtaining adequate funding is a challenge, and 19 reported that obtaining sufficient staff to manage the acquisition workforce training is a challenge. As for determining the appropriate level of investment, we have previously reported that when assessing opportunities for training, the agency should consider the competing demands confronting the agency, the limited resources available, and how these demands can best be met with available resources.[Footnote 17] Figure 3 provides a summary of the number of agencies that reported challenges involved in obtaining sufficient acquisition workforce training. Figure 3: Challenges Reported By Agencies in Obtaining Acquisition- Specific Training Resources: [Refer to PDF for image: horizontal bar graph] Challenge: Obtaining sufficient staffing to manage acquisition training; Extremely or Very Challenging: 10; Moderately or Somewhat Challenging: 9; Not at all challenging: 4. Challenge: Obtaining adequate level of budgetary resources; Extremely or Very Challenging: 8; Moderately or Somewhat Challenging: 12; Not at all challenging: 3. Challenge: Level of authority to control budgetary resources; Extremely or Very Challenging: 5; Moderately or Somewhat Challenging: 13; Not at all challenging: 5. Challenge: Federal acquisition certification (FAC) requirements; Extremely or Very Challenging: 4; Moderately or Somewhat Challenging: 9; Not at all challenging: 10. Challenge: Identifying members of the acquisition workforce; Extremely or Very Challenging: 2; Moderately or Somewhat Challenging: 9; Not at all challenging: 12. Challenge: Level of priority of the acquisition process in the agency; Extremely or Very Challenging: 1; Moderately or Somewhat Challenging: 12; Not at all challenging: 9. Source: GAO analysis based on agencies' responses to GAO questionnaire. Note: For "level of priority of the acquisition process in the agency, one agency reported "don't know." [End of figure] Although almost all agencies reported that obtaining an adequate level of funding is challenging, less than half--10 of 23--reported that their current acquisition training budgets are insufficient to meet their training needs. The acquisition workforce training budgets for fiscal year 2012 reported by 22 of the agencies ranged from $0 to $39,598,000 for an agency with its own acquisition workforce training facility. These budgets correspond to acquisition workforces that ranged in size from 186 to 11,867 employees for fiscal year 2010 that handled anywhere from about 900 to 1.4 million contracting actions with obligations ranging from $132 million to $25 billion in fiscal year 2011. Almost all the agencies--20 of 23--reported that they would like additional FAI assistance to help their acquisition workforce members attain federal certification by providing more course offerings or additional topics. In addition, to accommodate travel restrictions and budget constraints, some agencies would like FAI to provide more virtual courses for personnel that are geographically dispersed. FAI has some efforts underway that may somewhat address these issues, but their results are still uncertain as they are just beginning. FAI officials noted that while agencies want more offerings, FAI still has under-attended classes that have to be canceled. FAI officials also commented that developing on-line courses can also be very expensive and the cost to maintain currency of information is an often times overlooked cost. Some agencies reported challenges with a fundamental step towards managing the acquisition workforce--identifying the members of their acquisition workforce--in responding about the challenges faced with obtaining training for their acquisition workforce. Almost half of the agencies--11 of 23--reported this as challenging to some degree, although only 2 identified it as extremely or very challenging. In addition, FAI officials view identification of the acquisition workforce as the biggest challenge to providing training. FAI and officials at an agency that reported this as extremely challenging noted that some agencies are only beginning to train their workforce to meet certification requirements and may not recognize the identification of their workforce as a challenge until their acquisition workforce management practices become more mature. If members of the acquisition workforce are not identified as such, it is difficult to ensure their training and development needs are being met. Our prior work on acquisition personnel at the Department of Defense noted challenges with identifying the total workforce with acquisition-related responsibilities.[Footnote 18] Officials across DOD, including senior officials at DAU, told us that identifying some individuals with acquisition-related responsibilities, such as CORs, is challenging, in part because those personnel are dispersed throughout many organizations, come from a variety of career fields, and are often involved in acquisitions as a secondary and not a primary duty. The agencies' Acquisition Career Managers told us that they face similar challenges in identifying CORs and other personnel with acquisition-related responsibilities. Three of the four agencies selected for further insight--DHS, Education, and VA--acknowledged that they continue to be challenged by identifying some members of their acquisition workforces. A majority of the agencies--15 of 23--reported that they would like additional assistance from OFPP to improve their acquisition workforce training by either of two ways. Thirteen agencies supported the idea of OFPP creating separate job series for additional acquisition workforce categories, such as P/PMs. Twelve agencies supported the idea of OFPP maintaining a government-wide database to identify and track members. According to FAI and OFPP officials, they are sympathetic to agency concerns about identifying the total acquisition workforce. As for creating separate job series to more easily identify members of the acquisition workforce, OFPP notes that any changes would need to be made by the Office of Personnel Management and that some acquisition positions, such as CORs, may not lend themselves to becoming a unique job series because the work is performed as a collateral duty. As for maintaining a government-wide database to identify and track acquisition workforce members, FAI officials noted that agencies are encouraged to use the FAITAS registration system to track their workforce's individual training records and certifications. However, registration into the system is currently voluntary. Agencies also reported that their acquisition workforces are challenged in finding time in their workload to attend training. Most agencies--14 of 23--considered the ability to find time as extremely or very challenging. Figure 4 provides a summary of the number of agencies that reported challenges to their staff accessing and attending acquisition-related training. Figure 4: Challenges Reported by Agencies in Staff Attending Training: [Refer to PDF for image: horizontal bar graph] Challenge: Availability of time in the acquisition personnel’s workload; Extremely or Very Challenging: 14; Moderately or Somewhat Challenging: 8; Not at all challenging: 1. Challenge: Access to courses; Extremely or Very Challenging: 6; Moderately or Somewhat Challenging: 13; Not at all challenging: 4. Challenge: Availability of budget resources for training; Extremely or Very Challenging: 6; Moderately or Somewhat Challenging: 12; Not at all challenging: 4. Challenge: Priority level given to training by management; Extremely or Very Challenging: 1; Moderately or Somewhat Challenging: 13; Not at all challenging: 7. Challenge: Priority level given to training by acquisition personnel; Extremely or Very Challenging: 0; Moderately or Somewhat Challenging: 16; Not at all challenging: 7. Source: GAO analysis based on agencies' responses to GAO questionnaire. Note: For "availability of budget resources for training" one agency did not respond, and for "priority level given to training by management" one agency reported "not applicable" and another reported "don't know." [End of figure] Lack of Comparable Cost Data and Limited Insights On Benefits of Training Hinder Efforts to Maximize Resources Government-wide: Agencies collect some training cost data and limited information about the benefits of their acquisition workforce training. Based on responses to our questionnaire, a supplemental data request, and discussions with agencies' and FAI officials, we found that many agencies do not collect data on the costs of training provided to their acquisition workforce that can be used to inform agency and government-wide training resource investment decisions. In addition, some agencies do not have metrics to assess the effectiveness of their training. Government-wide Training Cost Data Reported Is Not Comparable: In response to our questionnaire, 16 of the 23 agencies provided cost data associated with their acquisition workforce training programs. After issuing a subsequent request for available cost information, 5 more agencies provided data related to the costs of individual acquisition workforce training courses. During our review, several agencies contacted us for clarification on the cost data we requested, noting that training cost data is not readily available within their agency. We have previously reported that training costs can include expenses for instructional development; participant and instructor attendance; facility, material, and equipment costs; and travel and per diem expenses.[Footnote 19] The data eventually provided by the agencies included different cost components, which did not lend themselves to comparative analysis. For example, some agencies provided data on the cost per seat of specific courses, while others provided the total costs for delivery of each course. Some agencies provided the costs associated with the development of an in-house course, while others included development costs with delivery costs as a total cost of obtaining the course from a vendor. Although the data do not allow direct comparisons among agencies, the data show a range of costs for similar courses. For example, five agencies provided a per seat cost for a COR Refresher course, which ranged from $97 to $363 per seat. In addition, five agencies provided a per seat cost for a Cost Analysis and Negotiation Techniques course, which ranged from $282 to $925 per seat. However, a number of factors can produce the variation in the agency-reported costs and these factors, along with the agencies' data collection methods, can limit the ability to make government-wide decisions on acquisition workforce training investments. According to agency officials, the costs of individual acquisition workforce training courses can vary greatly among agencies due to a variety of factors, including: the number of times per year the course is provided, actual attendance numbers, location of training, and whether courses are tailored with agency-specific information. FAI officials reported that they also received limited responses to their request for training cost data in the agencies' latest AHCPs, which made it impossible for them to complete their planned comparative analyses. The instructions for the FAI data request did not include definitions of the cost components to be reported, which are important to help ensure consistent reporting. Due to the limited response on cost data, FAI initiated a subsequent data call. As of November 2012, FAI had received responses from 20 of the 23 agencies, but again was unable to perform a comparative analysis of the cost data due to various limitations in the data it received. FAI officials noted that collecting cost data from agencies is an evolving process and that having comparable training cost data is important to help FAI in its efforts to maximize the use of acquisition workforce training dollars government-wide. FAI plans to request cost data annually with the AHCPs; however, the guidance for completing the next submission does not include definitions of the cost data to be reported, which may again yield data that cannot be compared across agencies. Improved reporting of cost data could assist FAI as it moves forward with its plans to award government-wide contracts for acquisition workforce training in 2014 by allowing FAI to obtain insight into the agencies' training costs when obtaining training from a variety of external sources. Agencies Gather Limited Information on Benefits of Acquisition Workforce Training Investment: About half of the agencies--12 of 23--did not have insight into whether their acquisition workforce training investment is improving individual skills or agency performance. In particular, in response to our questionnaire and subsequent data request 7 of 23 agencies reported having no metrics to monitor or assess the effectiveness of their acquisition workforce training efforts, a measure of whether the training investment is improving individual skills or agency performance. Five agencies did not provide information to support use of metrics to measure the training benefits, either improving individual skills or agency performance, of acquisition training in response to our supplemental data request. We have previously reported that training programs can be assessed by measuring (1) participant reaction to the training program, (2) changes in employee behavior or performance, and (3) the impact of the training on program or organizational results, which may include a return on investment assessment that compares training costs to derived benefits.[Footnote 20] Of the 11 agencies that provided information to support their use of metrics, 3 reported using end-of- course evaluations to measure participants' reaction, one reported using end-of-course tests to measure changes in employees' knowledge, and one reported using post-course surveys to supervisors or participants to measure if what was learned affected the participants' behavior. The other 6 agencies reported measures aimed at determining the impact of training on the agency's mission. Furthermore, DHS officials said that they plan to begin using post-course surveys of participants and supervisors in fiscal year 2013, and they are pursuing the development of additional measures to evaluate the impact of training on the agency. VA is also pursuing the development of additional metrics, such as proxy measures for return on investment, to evaluate the impact of training on the agency. Participant evaluations offer limited insight into improvements in individual and agency performance; however, we recognize that higher levels of evaluation (such as evaluating the impact on organizational performance or comparing training costs to derived benefits) can be challenging to conduct because of the difficulty and costs associated with data collection and the complexity in directly linking training and development programs to improved individual and organizational performance.[Footnote 21] Officials at some of the selected agencies told us that when the course is provided by an external source, such as FAI or DAU, they rely on the external source to provide end-of-course evaluations to the participants. As we noted earlier, 17 of 23 agencies obtain the majority of their acquisition training from external sources. In particular, some agencies noted that they do not collect or assess the end-of-course evaluations, relying solely on the external source to use the evaluations as they believe appropriate. FAI officials noted that if the agencies do not review the evaluations themselves, they are missing the opportunity to ensure that the courses are effective in training their acquisition personnel. We have previously reported that it is increasingly important for agencies to be able to evaluate their training programs and demonstrate how these efforts help develop employees and improve the agencies' performance because it can aid decision makers in managing scarce resources and provide credible information on how training has affected individual and organizational performance.[Footnote 22] Although FAI has an initiative underway to standardize the evaluations provided for its courses, its impact may be limited if agencies do not obtain the evaluation results and use them to evaluate the effectiveness of the training. Conclusions: Given the large acquisition investments the federal government makes each year, it is essential that the people in agencies who manage these procurements day-to-day--the acquisition workforce--be well- trained to handle their responsibilities. Fundamentally, agencies need key information to manage and oversee their acquisition workforce training investments. For example, agencies need to identify their acquisition workforce members, and measure how the training benefits the agencies by providing the workforce with the knowledge and skills to do their jobs effectively. At this point, OFPP, FAI, and federal agencies are taking steps that may potentially improve the effectiveness and efficiency of their training programs, but a number of challenges and limitations need to be overcome. Although FAI has various efforts underway--such as its efforts to ensure that every training seat is filled--to assist agencies in meeting OFPP requirements and leveraging federal government training resources, these efforts face significant obstacles. Agencies, FAI, and OFPP lack data on the acquisition workforce itself and the benefits of training, as well as cost data that can be used to make comparisons. These limitations hinder government-wide efforts to share information during these times of constrained budgets. OFPP, FAI, and the agencies need basic information on how much agencies are spending to train the acquisition workforce. Providing definitions and guidance about the elements of costs agencies should include in their annual acquisition human capital plans will help OFPP and FAI in their efforts to collect comparable data across agencies. FAI can then analyze and share information to help agencies make choices regarding how best to dedicate resources to effectively train the acquisition workforce. Additionally, comparable costs data will be helpful to inform FAI efforts to award government-wide contracts for standard acquisition workforce certification training. Currently, the main focus of monitoring and tracking acquisition workforce training efforts in agencies is on completion of training to attain and maintain the FAC certifications. Although the higher levels of evaluation (such as evaluating the impact on organizational performance or comparing training costs to derived benefits) can be challenging to conduct due to costs and complexity, agencies should, at a minimum, evaluate participant reaction to the training program through end of course evaluations. However, this is a basic measure that some agencies are lacking because the evaluations currently being administered go to the provider of the class and not the agency paying for the training. Therefore, these agencies have little insight into how the workforce perceives the training they received. We note that FAI has initiatives underway to standardize the course evaluations for the courses it provides and to make completion of the course evaluations mandatory to receive course credit. At present, this effort only applies to FAI-sponsored courses and will not include agencies that use other vendors, and does not guarantee that agencies will analyze the results; it is important for all agencies to collect and analyze this basic information. Recommendations for Executive Action: To help ensure that agencies collect and report comparable cost data and perform a minimal assessment of the benefits of their acquisition training investments to aid in the coordination and evaluation of the use of resources government-wide, we recommend the Director of the Office of Management and Budget direct the Administrator of the Office of Federal Procurement Policy, in consultation with the Director of the Federal Acquisition Institute, to take the following two actions: * provide further guidance, including definitions, on the types of costs that agencies should include in their Acquisition Human Capital Plan submission to help determine total training investment; and: * require all agencies, at a minimum, to collect and analyze participant evaluations of all acquisition workforce training as a first step to help assess the effectiveness of their training investment. Agency Comments and Our Evaluation: We provided a draft of this report to OFPP, FAI, and the selected agencies--DHS, Education, Treasury, and VA. OFPP and FAI concurred with our recommendations in oral comments and e-mail responses. Education and Treasury responded via e-mail with no comments. DHS and VA provided technical comments, which we incorporated as appropriate. In oral comments, OFPP and FAI agreed with our recommendations and noted that they have begun drafting guidance to federal agencies. Consistent with our report, OFPP and FAI emphasized the importance of acquisition workforce management tools that improve the government's ability to leverage acquisition resources especially during this time of budgetary constraints. These officials highlighted FAITAS, which can serve as a workforce management tool and training application system. Currently, use of this system is voluntary and some agencies use the system to enroll participants in FAI courses, communicate available training seats within a specific agency's training course, and track their workforce's individual training records and certifications. Recognizing the potential benefits of this system in helping to coordinate and evaluate the use of training resources government-wide, OFPP stated that it is considering making FAITAS reporting mandatory for civilian agencies. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Administrator of OFPP, the Director of FAI, and the Secretaries of DHS, Education, Treasury, and VA. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or MartinB@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Signed by: Belva M. Martin: Director, Acquisition and Sourcing Management: List of Requesters: The Honorable Thomas Carper: Chairman: The Honorable Tom Coburn: Ranking Member: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Darrell Issa: Chairman: The Honorable Elijah Cummings: Ranking Member: Committee on Oversight and Government Reform: United States House of Representatives: The Honorable Claire McCaskill: Chairman: Subcommittee on Financial and Contracting Oversight: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Susan Collins: United States Senate: [End of section] Appendix I: Summary of Agencies' Response to GAO Questionnaire on Civilian Acquisition Workforce Training: In July 2012, GAO administered a government-wide survey of the 23 civilian CFO Act agencies designed to gather information regarding the training approach and methods, staffing and budgetary resources, and costs associated with providing acquisition training—including the operation of facilities dedicated to training the acquisition workforce. We received responses from 100 percent of the agencies, although not every agency responded to each question. A summary of the number of agencies' who responded to each question is provided below. We removed the open-ended narrative responses, but included the ranges of open-ended data. Details of our survey methodology are provided in appendix II. Acquisition Workforce: 1. The Office of Federal Procurement Policy guidance provides minimum categories of positions that must be included in an agency's definition of its acquisition workforce, and allows agencies to also include other personnel who contribute substantially to successful acquisition outcomes. Does your agency include any additional positions in its definition of your acquisition workforce other than the Contracting series (1102), Contracting Officers, Purchasing series (1105), Program/Project Managers, and Contracting Officer's Representatives (CORs)/Contracting Officer's Technical Representatives (COTRs) or the equivalent? Yes: 11 No: 12; Skip to 2; Don't Know: 0; Skip to 2. 1a. (If "Yes" to 1) What additional personnel categories are included in your agency's definition of its acquisition workforce? (List all that apply) 2. What is the current size of your agency's workforce for each of the acquisition personnel groups? (Enter the number of people, including full-time and part-time, for each group.) a. Contracting Officers: Number of people: 21; Unable to identify: 1; Don't know: 1; Response range: 0-1,710 staff. b. Contracting series (1102) personnel other than Contracting Officers: Number of people: 23; Unable to identify: 0; Don't know: 0; Response range: 14-1,447 staff. c. Purchasing series (1105) personnel: Number of people: 23; Unable to identify: 0; Don't know: 0; Response range: 0-1,019 staff. d. Designated contract oversight officials (CORs, COTRs, or equivalent personnel): Number of people: 22; Unable to identify: 0; Don't know: 1; Response range: 125-7,499 staff. e. Program/project managers: Number of people: 21; Unable to identify: 2; Don't know: 0. Response range: 4-2,760 staff. f. Other (Please specify Number of people: 12; Unable to identify: 0; Don't know: 0. Response range: 0-1,780 staff. Training Resources: 3. What is your office's budget for providing dedicated training to the acquisition workforce for fiscal year 2012? Responses: 21; Response range: $0-$39.6 million. 4. What is your agency's overall training budget for fiscal year 2012? Responses: 13; Don't know: 10; Response range: $210,000-$81.2 million. 5. Does your office have responsibility for deciding how the acquisition workforce dedicated training budget is used to provide training to the acquisition workforce? Responses: Partial: 7; All: 11; Skip to 6; None: 5; Skip to 6; Don't know: 0; Skip to 6. 5a. (If "Partial" to 5) What portion of the acquisition workforce training budget do you have authority over? Responses: 6; Response range: $0 (decentralized to bureaus)-$45.2 million. 5b. What other units have authority over the remaining funds used to provide training to the acquisition workforce? Responses: 9. 6. In your opinion, does your office currently have sufficient budgetary resources to provide the acquisition-specific training to your acquisition workforce that is needed? Responses: Yes: 11; Skip to 7; No: 10; Don't know: 1; Skip to 7. 6a. (If "No" to 6) How much additional funding would your office require to be able to provide all the acquisition-specific training that your acquisition workforce needs? Responses: 9; Don't know: 1; Response range: $10,000-$3.5 million. 7. Are your Acquisition Career Manager duties performed as a full-time or part-time role? Responses: Full-time: 9; Skip to 8; Part-time: 14. 7a. (If "Part-time" to 7) What percentage of your time do you spend performing your Acquisition Career Manager duties? Responses: 14; Don't know: 0; Response range: 1-80%. 8. Does your office have dedicated staff to manage the provision of acquisition-specific training to the acquisition workforce? Responses: Yes: 11 No: 12; Skip to 9; Don't know: 0; Skip to 9. 8a. (If "Yes" to 8) How many staff manage the acquisition workforce training? a. Federal employees with full-time responsibility; Responses: 12. Response range: 0-42. b. Federal employees with part-time responsibility; Responses: 12. Response range: 0-4. c. Contractor personnel; Responses: 12. Response range: 0-30. 9. In your opinion, does your office have sufficient resources to manage the provision of acquisition-specific training to your acquisition workforce? Responses: Yes: 10; Skip to 10; No: 11; Don't know: 0; Skip to 10. 9a. (If "No" to 9) How many additional staff would your office require to be able to adequately oversee the provision of training to the acquisition workforce? Responses: staff with full-time responsibility: 10; Response range: 0-10; staff with part-time responsibility: 10; Response range: 0-5; Don't know: 1. 10. In your opinion, does your office have sufficient access to sources of acquisition-specific training courses needed to provide your acquisition workforce with the training it needs for fiscal year 2012? Yes: 17; Skip to 11; No: 6 Don't know: 0; Skip to 11. 10a. (if "No" to 10) For what percentage of the agency's acquisition- specific training needs can you not find sources? (Please provide an estimate if actual data is not available.) Responses: 4; Response range: 5-60%; Don't know: 0. 10b. (If "No" to 10) What acquisition-specific training is not accessible to your workforce at this time? 11. How much of a challenge do the following represent for obtaining resources to provide acquisition-specific training to your acquisition workforce? a. Identifying members of the acquisition workforce: Extremely challenging: 1; Very challenging: 1; Moderately challenging: 4; Somewhat challenging: 5; Not at all challenging: 12; Not applicable: 0; Don't know: 0. b. Level of priority of the acquisition process in the agency: Extremely challenging: 0; Very challenging: 1; Moderately challenging: 7; Somewhat challenging: 5; Not at all challenging: 9; Not applicable: 0; Don't know: 1. c. Level of authority to control budgetary resources: Extremely challenging: 2; Very challenging: 3; Moderately challenging: 7; Somewhat challenging: 6; Not at all challenging: 5; Not applicable: 0; Don't know: 0. d. Obtaining adequate level of budgetary resources: Extremely challenging: 3; Very challenging: 5; Moderately challenging: 7; Somewhat challenging: 5; Not at all challenging: 3; Not applicable: 0; Don't know: 0. e. Obtaining sufficient staffing to manage acquisition training: Extremely challenging: 6; Very challenging: 4; Moderately challenging: 3; Somewhat challenging: 6; Not at all challenging: 4; Not applicable: 0; Don't know: 0. f. Federal acquisition certification (FAC) requirements: Extremely challenging: 0; Very challenging: 4; Moderately challenging: 5; Somewhat challenging: 4; Not at all challenging: 10; Not applicable: 0; Don't know: 0. g. Other (Please specify the challenge below): Extremely challenging: 3; Very challenging: 2; Moderately challenging: 2; Somewhat challenging: 0; Not at all challenging: 0; Not applicable: 1; Don't know: 0. 12. Of the challenges you cited above, which one would you consider to be your biggest challenge in providing sufficient acquisition-specific training to your acquisition workforce? (Enter the letter designation (a-h) from question 11.) 13. What additional comments or information on acquisition training resources would you like to provide? (if you expand on any of your answers to the prior questions in this section, please reference the number of the question.) Acquisition Training Sources and Methods: 14. What percentage of the number of acquisition-specific training course offerings was provided via each of the following sources in FY2011? Percentage of total number of course offerings provided during the year: a. Agency-provided: 22%; Response range: 0-100%. b. Federal Acquisition Institute (FAI): 22%; Response range: 0-85%. c. Commercial sector courses other than those obtained through FAI (Please specify the main vendors below): 22%; Response range: 0-90%. d. Defense Acquisition University: 22%; Response range: 0-62%. e. Other non-defense federal agencies (Please specify the main agencies below): 22%; Response range: 0-25%. f. Other (Please specify the source below): 3%; Response range: 0-10%. g. Other (Please specify the source below): 0%; Response range: 0. Total: 100%. 15. What percentage of the number of acquisition-specific training courses was provided via each of the following delivery methods in FY2011? Percentage of total number of courses provided: a. Instructor-led classroom: 22; Response range: 10-100%. b. Instructor-led on-line: 22; Response range: 0-45%. c. Self-paced bookwork: 22; Response range: 0-5%. d. Self-paced on-line work: 22; Response range: 0-70%. e. On-the-job instruction: 22; Response range: 0-25%. f. Other (Please specify the delivery method below): 2 Response range: 0-20%. g. Other (Please specify the delivery method below): 0 Total: 100%. 16. For the most prevalent method of training used to deliver acquisition-specific training to your workforce, what is the main reason(s) this method of training is used? (Check all that apply.) a. Cost effectiveness: 15; b. Effectiveness of training method: 20; c. Staffing resources: 5; d. Budgetary resources: 13; e. Demand for specific courses: 15; f. Course already available: 12; g. Other (Please specify): 6. 17. What additional comments or information on the approaches or methods your agency is currently using to provide acquisition-specific training would you like to provide? (If you expand on any of your answers to the prior questions in this section, please reference the number of the question.) Acquisition Workforce Training and Certifications: 18. Does your agency have required acquisition-specific training for any members of the acquisition workforce beyond the new Federal Acquisition Certification (FAC) programs' requirements? Yes: 11 No: 12; Skip to 19; Don't Know: 0; Skip to 19. 18a. (If "Yes" to 18) How would you briefly describe the additional requirements for the respective acquisition workforce personnel categories? a. Contracting officers: b. Contracting series (1102) personnel other than Contracting Officers: c. Purchasing series (1105) personnel (i.e. Ordering Officer training): d. Designated contract oversight officials (CORS, COTRs, or equivalent personnel): e. Program/project managers: f. Other (Please specify the position below): g. Other (Please specify the position below): h. Other (Please specify the position below): 19. In your opinion, is the acquisition workforce able to obtain sufficient training to meet its needs (in terms of knowledge basics and continuing education)? Yes: 16; Skip to 20; No: 7; Don't Know: 0; Skip to 20. 19a. (If "No" to 19) What acquisition-specific training is your agency not able to obtain for your workforce at this time? a. Courses to meet Federal Acquisition Certification for Contracting (FAC-C): Yes: 4; No: 2; Don't know: 0. b. Courses to meet Federal Acquisition Certification for CORs (FAC- COR): Yes: 4; No: 2; Don't know: 0. c. Courses to meet Federal Acquisition Certification for Program Managers (FAC-PM): Yes: 4; No: 2; Don't know: 0. d. Continuous learning courses to maintain FAC-C certification: Yes: 3; No: 3; Don't know: 0. e. Continuous learning courses to maintain FAC-COR certification: Yes: 4; No: 2; Don't know: 0. f. Continuous learning courses to maintain FAC-PM certification: Yes: 6; No: 1; Don't know: 0. g. Other (Please specify): Yes: 2; No: 2; Don't know: 0. 20. Have any recent trends in federal contracting—such as increase in services contracting or information technology (IT) contracts—had an impact on your agency's acquisition-specific training approaches or methods? Yes: 15 No: 8; Skip to 21; Don't Know: 0; Skip to 21. 20a. (If "Yes" to 20) What actions did your agency take to address the impact of any trends? a. Services contracting—-additional guidance/training on contracting for services; Yes: 12; No: 1; Don't know: 0. b. Services contracting--Development of a services contracting cadre; Yes: 3; No: 9; Don't know: 0. c. IT contracting-—additional guidance/training for IT contracting; Yes: 10; No: 2; Don't know: 0. d. IT contracting—Development of an IT contracting cadre; Yes: 9; No: 5; Don't know: 0. e. Other (Please specify): Yes: 3; No: 0; Don't know: 0. 21. In your opinion, has your agency found sufficient sources for the specific courses needed to obtain the various federal acquisition certifications? Yes: 19; Skip to 22; No: 3; Don't know: 1. 21a. (If "No" to 21) What specific training courses are not available for your workforce at this time? 22. Are any of the following guidance or actions needed from the Federal Acquisition Institute to assist your agency in obtaining the federal acquisition certifications? a. Providing more course offerings each year; Yes: 17; No: 4; Don't know: 1. b. Providing additional training topics; Yes: 19; No: 3; Don't know: 0. c. Facilitating additional interagency coordination on courses available at agencies; Yes: 16; No: 5; Don't know: 2. d. Providing a government-wide resource (i.e., database) for identifying/tracking members of the acquisition workforce; Yes: 15; No: 4; Don't know: 2. e. Other (Please specify); Yes: 8; No: 0; Don't know: 1. 23. Are any of the following guidance or actions needed from the Office of Federal Procurement Policy to assist your agency in providing training to your acquisition workforce? a. Creating job series for additional acquisition workforce categories, such as Program Managers; Yes: 13; No: 7; Don't know: 3. b. Providing a government-wide resource (i.e., database) for identifying/tracking members of the acquisition workforce; Yes: 12; No: 8; Don't know: 2. c. Other (Please specify; Yes: 4; No: 0; Don't know: 1. 24. How much of a challenge do the following represent for your acquisition workforce to take the acquisition-specific training it needs (in terms of knowledge basics and continuing education)? a. Availability of budget resources for training; Extremely challenging: 2; Very challenging: 4; Moderately challenging: 4; Somewhat challenging: 8; Not at all challenging: 4; Not applicable: 0; Don't know: 0. b. Access to courses; Extremely challenging: 1; Very challenging: 5; Moderately challenging: 8; Somewhat challenging: 5; Not at all challenging: 4; Not applicable: 0; Don't know: 0. c. Priority level given to training by acquisition personnel; Extremely challenging: 0; Very challenging: 0; Moderately challenging: 6; Somewhat challenging: 10; Not at all challenging: 7; Not applicable: 0; Don't know: 0. d. Priority level given to training by management; Extremely challenging: 0; Very challenging: 1; Moderately challenging: 6; Somewhat challenging: 7; Not at all challenging: 7; Not applicable: 1; Don't know: 1. e. Availability of time in the acquisition personnel's workload; Extremely challenging: 7; Very challenging: 7; Moderately challenging: 5; Somewhat challenging: 3; Not at all challenging: 1; Not applicable: 0; Don't know: 0. f. Other (Please specify the challenge below); Extremely challenging: 1; Very challenging: 1; Moderately challenging: 0; Somewhat challenging: 0; Not at all challenging: 0; Not applicable: 0; Don't know: 1. g. Other (Please specify the challenge below); Extremely challenging: 0; Very challenging: 0; Moderately challenging: 0; Somewhat challenging: 0; Not at all challenging: 0; Not applicable: 0; Don't know: 0. 25. Of the challenges you cited above, which one would you consider to be the biggest challenge for the acquisition workforce to obtain needed acquisition-specific training? (Enter the letter designation (a-g) from 24.) 26. What additional comments or information on acquisition training courses and certification requirements would you like to provide? (If you expand on any of your answers to the prior questions in this section, please reference the number of the question.) Training Locations: 27. Did your agency use any of the following locations when providing training to your acquisition workforce? a. Your agency's conference moms or other meeting space; Yes: 23; No: 0; Don't know: 0. b. Vendor-provided locations; Yes: 19; No: 3; Don't know: 1. c. Another agency's training facility (Please provide the name of the other agencies below); Yes: 13; No: 8; Don't know: 1. d. Facilities rented by your agency (i.e., hotel conference room); Yes: 4; No: 17; Don't know: 0. e. Your own agency's dedicated training facility (Please note the name(s) and location(s) of facilities your agency); Yes: 14; No: 8; Don't know: 0. Agency-Operated Training Facilities: If your agency does not operate any dedicated training facilities, skip to 34 in the next section. 28. Are any of your own agency-operated training facilities dedicated to providing courses for only the acquisition workforce? Yes: 3; No: 14; Skip to 29; Don't know: 0; Skip to 29. 28a. (If "Yes" to 28) What are the names and locations of the facilities that provide training only for the acquisition workforce? 29. What were the aggregate costs of operating your agency's training facilities for FY2011 for each of the following cost categories? a. Personnel (dedicated to the facility) Response: 1; Total costs: $89,000; Costs specific to acquisition training (if available): Response: 2; Range: $0-89,000. b. Contractor personnel/consultants; Response: 0; Costs specific to acquisition training (if available): Response: 1; Range: $750,000. c. Property/rent/lease (if applicable); Response: 1; Total costs: $2,532,450; Costs specific to acquisition training (if available): Response: 2 Range: $468,000-$2,531,340. d. Facility operations (electricity heating, etc.); Response: 0; Costs specific to acquisition training (if available): Response: 1; Range: $0. e. Overhead; Response: 0; Costs specific to acquisition training (if available): Response: 0. f. Other (Please specify the type of cost below): supplies; Response: 0; Costs specific to acquisition training (if available): Response: 1 Range: $5,000. g. Other (Please specify the type of cost below): new center; Response: 0; Costs specific to acquisition training (if available): Response: 1; Response: $955,000 30. What is the yearly capacity of your agency-operated training facilities? (Please specify the number of seats, courses. or other measure of capacity that the facility can handle in a given year) a. Number of seats: Total annual volume: 2; Range: 6,195-17,136; Volume specific to acquisition training (if available): 2; Range: 6,195-17,136. b. Number of courses: Total annual volume: 2; Range: 221-672; Volume specific to acquisition training (if available): 2; Range: 221-672. c. Other (Please specify the type of measure below): Total annual volume: 0; Volume specific to acquisition training (if available): 0. 31. What level of capacity did your agency-operated training facilities function at in FY2011? 95% or greater: 1; Skip to 32; 85% or greater, but less than 95%: 2; Skip to 32; 75% or greater, but less than 85%: 0; Skip to 32; 50% or greater, but less than 75%: 0; 25% or greater, but less than 50%: 0; Less than 25%: 0; Don't Know: 0. 31a. What were the main reasons the training facilities were not used beyond the percentage indicated in 31? Don't know: 0. 32. What additional comments or information on the operation of your training facility would you like to provide? (If you expand on any of your answers to the prior questions in this section, please reference the number of the question) 33. Does your agency currently allow personnel from other federal agencies to take acquisition-specific courses from your agency- operated training facilities? Yes: 3; No: 0; Skip to 34; Don't Know: 0; Skip to 34. 33a. (If "Yes" to 33) To what classes are other agencies allowed to send participants? Don't know: 0. 33b. What agencies sent participants in the last two fiscal years or are sending participants during this fiscal year? Don't know: 0. 33c. How are other agencies notified of the availability of your agency's training classes? Don't know: 0. Other Issues: 34. Does your agency have training-related metrics that show the effectiveness of acquisition-specific training on the performance of the acquisition workforce? Yes: 10; No: 13; Skip to 35; Don't Know: 0; Skip to 35. 34a. (If "Yes" to 34) What metrics are used to show the effectiveness of acquisition-specific training on the performance of the acquisition workforce and how are they measured? 35. In your opinion, are any changes needed to improve your effectiveness in providing acquisition-specific training to your acquisition workforce? Yes: 16 No: 7; Skip to 36; Don't Know: 0; Skip to 36. 35a. (If "Yes" to 35) what changes do you feel are needed? 36. In your opinion, does your agency have any practices that have effectively addressed the challenges you face in managing acquisition- specific training that you believe would benefit other agencies? Yes: 12 No: 10; Skip to 37; Don't Know: 1; Skip to 37. 36a. (If "Yes" to 36) what are those practices? 37. What additional comments or information on the management and provision of acquisition-specific training would you like to provide? (If you expand on any of your answers to the prior questions in this section, please reference the number of the question.) [End of section] Appendix II: Objectives, Scope, and Methodology: The objectives of this review were to evaluate the training approaches of federal agencies, other than the Department of Defense, for the acquisition workforce. Specifically, this report addresses the (1) role of the Office of Federal Procurement Policy (OFPP) and the Federal Acquisition Institute (FAI) in promoting Federal Acquisition Certification (FAC) standards and assisting agencies in meeting acquisition workforce training requirements; (2) approaches agencies use to provide training to their acquisition workforces; and (3) the extent to which agencies track information on the costs and benefits of their acquisition training. To identify the roles of OFPP and FAI in promoting FAC standards, we analyzed legislation on their authority, the OFPP acquisition workforce strategic plan, and relevant guidance provided by them to agencies. We interviewed officials at OFPP and FAI about efforts to (1) provide oversight and assist agencies in meeting training requirements; (2) leverage government-wide resources; and (3) share leading practices. Since FAI's initiatives are in various stages of implementation, we did not evaluate their effectiveness; however, in several instances, we note our initial observations of the efforts. To obtain government-wide information on how agencies provide training to their acquisition workforce, we administered a questionnaire with both open and close-ended questions to 23 Chief Financial Officers Act agencies.[Footnote 23] These agencies are: the Departments of Agriculture, Commerce, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, the Treasury, and Veterans Affairs; the Environmental Protection Agency; General Services Administration; National Aeronautics and Space Administration; Nuclear Regulatory Commission; National Science Foundation; Office of Personnel Management; Small Business Administration; Social Security Administration; and the U.S. Agency for International Development. [Footnote 24] We used a standard set of questions to obtain data about their training approaches, including the sources for course offerings (in particular whether courses were provided by the agency, a commercial sector vendor, FAI, the Defense Acquisition University, or another agency), facilities, budgetary and staffing resources, methods of training, and challenges faced in providing acquisition training. Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such nonsampling error. After we drafted the questionnaire, we asked for comments from federal officials knowledgeable about acquisition workforce and training issues and from independent GAO survey professionals. We conducted pretests to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on agency officials, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. We chose the two pretest sites to include perspectives from agencies with robust acquisition workforce training programs and training facilities. We made changes to the content and format of the questionnaire after both reviews and after each of the pretests, based on the feedback we received. We sent the questionnaire by e-mail in an attached Microsoft Word form that respondents could return electronically after marking checkboxes or entering responses into open answer boxes on July 5, 2012. We sent the questionnaire jointly to the Chief Acquisition Officer, Acquisition Career Manager, and Chief Learning Officer of each agency, asking for a consolidated agency response. After two weeks, we sent a reminder to everyone who had not responded. All questionnaires were returned by August 10, 2012. We received questionnaire responses from 100 percent of the agencies, although not all agencies answered every question. The Department of Justice provided four responses--one from its headquarters procurement office and three from bureau procurement offices. We consolidated the information from the four surveys to ensure the tabulations eliminated duplicative counting of the agency and to provide averaged data, as appropriate. In questions regarding the agency's opinions regarding challenges faced, we deferred to the response of the headquarters procurement office. To confirm our understanding of the variety of issues addressed and obtain additional insights on the agencies' responses to our questionnaire, we discussed our preliminary results with representatives from the 23 agencies at meetings of the Chief Acquisition Officers Council, Interagency Acquisition Career Management Committee, and the Chief Learning Officers Council. Table 4 provides a summary of the functions of these councils as related to government-wide efforts for training the acquisition workforce. Table 4: Interagency Councils Involved in Oversight and Management of Acquisition Workforce Training: Council: Chief Acquisition Officers Council (CAOC); Charter date: June 2004; Main Purpose: Serves as the principal interagency forum for monitoring and improving the federal acquisition system; Training-related responsibilities: * Develops recommendations for the Director of the Office of Management and Budget (OMB) on acquisition policies and requirements; * Shares experiences, ideas, best practices, and innovative approaches; * Works with the Office of Personnel Management to assess and address acquisition training needs of the government; * OFPP, OMB, and DOD are represented on the council along with the Chief Acquisition Officers, including those from each of the 23 civilian CFO Act agencies. Council: Interagency Acquisition Career Management Committee (IACMC); Charter date: September 2005; Main Purpose: Supports the implementation of interagency plans and programs for improving and maintaining the competence of federal acquisition personnel--as such, it supports OFPP and FAI in workforce program implementation; Training-related responsibilities: * Identifies civilian agency training needs and assists FAI in developing the training delivery strategy for courses funded by the Acquisition Workforce Training Fund; * Communicates agency acquisition workforce training needs and priorities to FAI for core training, just-in-time classes, electives, and other classes; * Coordinates the development and implementation of acquisition career management plans and programs--including the federal acquisition certification programs; * Facilitates the exchange of information on agency acquisition career management programs covering personnel management; * Assists FAI in (1) developing recommendations for new initiatives and alternate methods of training delivery; (2) making subject matter specialists available to review and evaluate course material and participate in pilot courses; (3) ensuring that training meets civilian agencies' needs; and (4) ensuring that civilian agencies participate in decisions affecting long-range curriculum development. Council: Chief Learning Officers Council (CLOC); Charter date: 2005; Main Purpose: Community of practice composed of federal Chief Learning Officers (CLO) or their equivalents formed to act collectively to support and further learning and training efforts across the government; Training-related responsibilities: * Shares agencies' best practices and creates learning opportunities for agencies and organizations; * Officials who hold positions similar to a CLO in independent agencies, boards, commissions, and the intelligence community are also members. Source: GAO summary of council charters. [End of table] We also reviewed the agencies' annual Acquisition Human Capital Plans (AHCP) due March 31, 2012, to collect information on certification rates, and corroborate information agencies provided in response to the questionnaire on acquisition workforce size and training approaches. To determine the information agencies track on the costs and benefits of the training provided, we supplemented the questionnaire responses by soliciting additional cost and metrics data from the agencies using a data collection instrument. We developed our data collection instrument based on cost information provided from agencies in response to our questionnaire. We sent agency-tailored data collection instruments by e-mail in an attached Microsoft Excel spreadsheet that respondents could return electronically after updating or providing new data on October 15, 2012. We pre-populated each agency's spreadsheet with any data previously provided by the agency, and sent the data collection instrument to the Acquisition Career Manager and the official who provided the response to the questionnaire. After two weeks, we sent a reminder to everyone who had not responded. The data collection instruments were returned by November 28, 2012. We received data collection instruments from 91 percent of the agencies. To gather illustrative examples and more detailed explanations regarding training approaches and data tracked related to costs and benefits, we selected four agencies--the Departments of Education (Education), Homeland Security (DHS), the Treasury (Treasury), and Veterans Affairs (VA)--for further review. DHS, Treasury, and VA are agencies that operate dedicated acquisition workforce training facilities--permanent centers with dedicated resources that provide training specifically for the agency's acquisition workforce. Education, which is one of the smallest agencies, has no dedicated facility and has reported issues with access to acquisition workforce training courses. We interviewed agency officials to determine the extent to which they were aware of and using the leading training investment practices that the Office of Personnel Management and experts agreed should be implemented by agencies to support effective training investment decisions.[Footnote 25] Table 5 provides a summary of these leading practices. Table 5: Leading Training Investment Practices Agencies Should Implement: Agencies should implement: Practice 1; (a) Identify the appropriate level of investment to provide for training and development efforts and (b) prioritize funding so that the most important training needs are addressed first. Agencies should implement: Practice 2; Identify the most appropriate mix of centralized and decentralized approaches for its training and development programs. Agencies should implement: Practice 3; Consider governmentwide reports and other targeted initiatives to improve management and performance when planning its training and development programs. Agencies should implement: Practice 4; Have criteria for determining whether to design training and development programs in-house or obtain these services from a contractor or other external source. Agencies should implement: Practice 5; Compare the merits of difference delivery mechanisms (such as classroom or computer-based training) and determine what mix of mechanisms to use to ensure efficient and cost-effective delivery. Agencies should implement: Practice 6; Track the cost and delivery of its training and development programs agency-wide. Agencies should implement: Practice 7; Evaluate the benefits achieved through training and development programs, including improvements in individual and agency performance; (a) Has a formal process for evaluating employee satisfaction with training; (b) Has a formal process for evaluating improvement in employee performance after training; (c) Has a formal process for evaluating the impact of training on the agency's performance goals and mission. Agencies should implement: Practice 8; Compare training investments, methods, or outcomes with those of other organizations to identify innovative approaches or lessons learned. Source: GAO analysis based on prior GAO reports, other related expert studies, and federal training requirements, as reported in [hyperlink, http://www.gao.gov/products/GAO-12-878]. [End of table] Of the eight leading practices GAO identified, we focused on the four practices dealing primarily with determining costs and effectiveness of training: (practice 1) identifying the appropriate level of investment to provide for training and development efforts and prioritize funding so that the most important training needs are addressed first; (practice 4) having criteria for determining whether to design training and development programs in-house or obtain these services from a contractor or other external source; (practice 6) tracking the cost and delivery of its training and development programs agency-wide; and (practice 7) evaluating the benefits achieved through training and development programs, including improvements in individual and agency performance. We conducted this performance audit from February 2012 to March 2013, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Belva M. Martin, (202) 512-4841 or MartinB@gao.gov: Staff Acknowledgments: In addition to the contact name above, Penny Berrier (Assistant Director); R. Eli DeVan; Mya Dinh; Jean McSween; Aku Pappoe; Kenneth Patton; Carol D. Petersen; Sylvia Schatz; and Suzanne Sterling made key contributions to this report. [End of section] Footnotes: [1] The 23 civilian Chief Financial Officer Act agencies we included are the Departments of Agriculture, Commerce, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, the Treasury, and Veterans Affairs; the Environmental Protection Agency; General Services Administration; National Aeronautics and Space Administration; Nuclear Regulatory Commission; National Science Foundation; Office of Personnel Management; Small Business Administration; Social Security Administration; and the U.S. Agency for International Development. 31 U.S.C. § 901. [2] The Department of Defense is excluded from this review because we have previously reported on its acquisition workforce training and its follows separate acquisition workforce certification standards specified in the Defense Acquisition Workforce Improvement Act, Pub. L. No. 101-510, § 1202(a) (1990) (codified as amended at 10 U.S.C. §§ 1701-1764 (2013)). See GAO, Defense Acquisition Workforce: Better Identification, Development, and Oversight Needed for Personnel Involved in Acquiring Services, [hyperlink, http://www.gao.gov/products/GAO-11-892] (Washington, D.C.: Sept. 28, 2011) and Defense Acquisition Workforce: DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed, [hyperlink, http://www.gao.gov/products/GAO-11-22] (Washington, D.C.: Oct. 28, 2010). [3] GAO, Federal Training Investments: Office of Personnel Management and Agencies Can Do More to Ensure Cost-Effective Decisions, [hyperlink, http://www.gao.gov/products/GAO-12-878] (Washington, D.C.: Sept. 17, 2012). [4] 41 U.S.C. § 1703. [5] Office of Management and Budget, Policy Letter 05-01, April 2005. [6] 41 U.S.C. § 1703(c)(2). [7] The Federal Supply Schedules program consists of contracts awarded by the General Services Administration or the Department of Veterans Affairs for similar or comparable goods or services, established with more than one supplier, at varying prices. Federal Acquisition Regulation (FAR) § 8.401 and § 8.402. The Schedules offer a large group of commercial products and services ranging from office supplies to information technology services. Governmentwide Acquisition Contracts (GWAC) are considered multi-agency contracts but, unlike other multi-agency contracts, are not subject to the same requirements and limitations, such as documentation that the contract is in the best interest of the government as set forth under the Economy Act. GWACs are contracts for information technology established by one agency for government-wide use that are operated--(1) by an executive agent designated by the Office of Management and Budget pursuant to 40 U.S.C. § 11302(e); or (2) under a delegation of procurement authority issued by the General Services Administration (GSA). FAR § 2.101. The Clinger-Cohen Act of 1996 authorized GWACs to be used to buy information technology goods and services. Pub. L. No. 101-510, § 1202 (1996), (codified at 40 U.S.C. § 11314(a)(2)). All agencies can order goods and services directly through the Schedules contracts and GWACs. [8] 10 U.S.C. § 1746. [9] The Federal Acquisition Certification requirements were issued in 2005 for contracting professionals and 2007 for Contracting Officer's Representatives and Program/Project Managers. [10] 41 U.S.C. § 1201(a)(1). [11] 41 U.S.C. § 1201(a)(11). [12] 41 U.S.C. § 1201(a)(3). [13] 41 U.S.C. § 1201(a)(9). [14] 41 U.S.C. § 1201(a)(4) and 41 U.S.C. § 1201(a)(7). [15] Although FAI does not seek reimbursement for individual classes, civilian agencies "pay" for the courses since they provide the funding for FAI's operations through the mandatory fee paid equal to five percent of the dollar amount of acquisitions the agency makes through the Federal Supply Schedule and the Governmentwide Acquisition Contracts. [16] [hyperlink, http://www.gao.gov/products/GAO-12-878]. [17] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. [18] [hyperlink, http://www.gao.gov/products/GAO-11-892]. [19] [hyperlink, http://www.gao.gov/products/GAO-12-878] and [hyperlink, http://www.gao.gov/products/GAO-04-546G]. [20] Training program measures are generally associated with the Kirkpatrick model of evaluation, a popular model of evaluation, that includes measuring (1) participant reaction to the training program; (2) changes in employee skills, knowledge, or abilities; (3) changes in employee behavior or performance; and (4) the impact of the training on program or organizational results, which may be split into a fifth level that includes a return on investment assessment that compares training costs to derived benefits. See [hyperlink, http://www.gao.gov/products/GAO-12-878]. [21] [hyperlink, http://www.gao.gov/products/GAO-12-878]. [22] [hyperlink, http://www.gao.gov/products/GAO-12-878] and [hyperlink, http://www.gao.gov/products/GAO-04-546G]. [23] 31 U.S.C. § 901. [24] The Department of Defense is excluded from this review because we have previously reported on its acquisition workforce training and it follows separate acquisition workforce certification standards specified the Defense Acquisition Workforce Improvement Act, Pub. L. No. 101-510, § 1202(a) (1990). See GAO, Defense Acquisition Workforce: Better Identification, Development, and Oversight Needed for Personnel Involved in Acquiring Services, [hyperlink, http://www.gao.gov/products/GAO-11-892] (Washington, D.C.: Sept. 28, 2011) and Defense Acquisition Workforce: DOD's Training Program Demonstrates Many Attributes of Effectiveness, but Improvement Is Needed, [hyperlink, http://www.gao.gov/products/GAO-11-22] (Washington, D.C.: Oct. 28, 2010). [25] [hyperlink, http://www.gao.gov/products/GAO-12-878] [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]