This is the accessible text file for GAO report number GAO-12-878 
entitled 'Federal Training Investments: Office of Personnel Management 
and Agencies Can Do More to Ensure Cost-Effective Decisions' which was 
released on September 17, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

A Report to the Chairman, Subcommittee on Oversight of Government 
Management, the Federal Workforce, and the District of Columbia; 
Committee on Homeland Security and Government Affairs, United States 
Senate: 

September 2012: 

Federal Training Investments: 

Office of Personnel Management and Agencies Can Do More to Ensure Cost-
Effective Decisions: 

GAO-12-878: 

GAO Highlights: 

Highlights of GAO-12-878, a report to the Chairman, Subcommittee on 
Oversight of Government Management, the Federal Workforce, and the 
District of Columbia; Committee on Homeland Security and Governmental 
Affairs, United States Senate. 

Why GAO Did This Study: 

OPM and agency CHCOs play an important role in ensuring that federal 
training dollars are invested effectively. GAO was asked to review the 
extent to which: (1) CHCOs of selected federal agencies have 
established processes to set and prioritize training investments that 
are aligned with leading practices; and (2) OPM’s guidance and 
assistance for developing training investment strategies align with 
these leading practices. GAO obtained information from 27 CHCOs on 
their training investment practices through a questionnaire, and 
selected four agencies—-the Departments of Energy (DOE), Homeland 
Security (DHS), the Interior (DOI) and Veterans Affairs (VA)—-to 
provide illustrative examples. We compared both CHCO and OPM practices 
to leading practices, identified through past GAO and expert studies. 

What GAO Found: 

Many Chief Human Capital Officers (CHCOs) reported that they are 
implementing several leading practices important to making strategic 
decisions about training delivery, such as determining the best mix of 
decentralized and centralized training and considering government-wide 
reform when planning training. However, many CHCOs reported they are 
not implementing some practices that support making more cost-
effective training investment decisions, such as prioritizing training 
so that the most important needs are met first and evaluating the 
benefits of training. In addition, many CHCOs do not have information 
from component or sub-agency leaders regarding their level of 
investments and priorities. Consequently, some agencies are 
duplicating internal training investments and missing opportunities to 
leverage economies of scale across their agencies. Federal agencies 
also need reliable information on how much they spend on training and 
for what purposes. However, several CHCOs reported they do not 
completely and reliably track training costs agency-wide. 

The Office of Personnel Management (OPM) provides guidance and 
assistance to agencies on a number of the leading practices, such as 
evaluating the benefits of training in three of its guides and in 
workshops. In some practice areas that are challenges to agencies, 
such as prioritization of investments and determining whether to 
design training and development programs in-house or obtain these 
services from a contractor, guidance is minimal or absent. OPM also 
requires agencies to submit training investment data and provides 
guidance on how to do so, but considers this data to be unreliable 
because it is incomplete. However, OPM officials have not internally 
assessed improvements in the completeness of the data over the last 3 
years or the quality of the data in the six years that agencies have 
been required to submit it, and have only provided agencies with one 
summary of their data for correction. Agencies and OPM reported there 
are also opportunities for OPM to help agencies reduce duplicative 
investments across agencies. For example, currently, agencies 
independently purchase or develop training for the same mandated or 
common occupational training. Agency leaders and OPM recognize that 
this has led to redundant and inefficient federal training 
investments. According to OPM officials, HR University—which is a 
website currently administered by OPM to provide training for the HR 
community—has already resulted in a cost savings of $14.5 million as a 
result of sharing the best HR training government-wide. Several 
agencies and OPM officials reported that HR University could be 
expanded to provide mandatory training and serve as a model for 
centralizing training in other occupations or functional areas, which 
could save millions more and help standardize training.  

What GAO Recommends: 

GAO recommends, among other things, that OPM improve guidance and 
assistance to agencies in establishing a process for setting and 
prioritizing training investments; improve the reliability of agency 
training investment information; and identify the best existing 
courses that fulfill governmentwide training requirements, and offer 
them to all agencies through the HR University or other appropriate 
platforms. OPM fully or partially concurred with four recommendations 
and did not concur with a portion of another. OPM, DOI and VA provided 
technical comments, which GAO incorporated, as appropriate, into the 
report. DOE and DHS had no comments. 

View [hyperlink, http://www.gao.gov/products/GAO-12-878]. For more 
information, contact Yvonne J. Jones at (202) 512-2717 or 
Jonesy@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

CHCOs are Implementing Leading Practices Related to Training Delivery, 
but Not Practices that Support Making More Cost-Effective Training 
Investments: 

OPM Guidance and Assistance to Agencies on Federal Training 
Investments Addresses Many Leading Practices that We Identified, But 
Could be Improved in Areas That Are Challenges to Agencies: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Table of Statutes, Regulations, and Executive Orders 
Related to Leading Practices: 

Appendix III: Illustrations of Building a Business Case for Training 
from OPM's Guide to Strategically Planning Training and Measuring 
Results: 

Appendix IV: Comments from the Office of Personnel Management: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Leading Training Investment Practices: 

Table 2: OPM's Guidance Documents and Description: 

Table 3: The Leading Training Investment Practices CHCOs Reported 
Implementing: 

Table 4: Leading Training Investment Practices that Many CHCOs 
Reported Not Implementing Routinely: 

Table 5: Comparison of Agency Implementation of Leading Practices and 
OPM Guidance and Assistance: 

Table 6: Selection of Case Example Agencies: 

Figures: 

Figure 1: OPM illustration of questions used to develop a business 
case for training: 

Figure 2: OPM illustration of developing a business case for training: 

Abbreviations: 

HCO: chief human capital officer: 

CLO: chief learning officer: 

DHS: Department of Homeland Security: 

DOE: Department of Energy: 

EHRI: Enterprise Human Resources Integration: 

GPRA: Government Performance and Results Act: 

GEAR: Goals-Engagement-Accountability-Results: 

GPRAMA: GPRA Modernization Act of 2010: 

HR: human resources: 

DOI: Department of the Interior: 

OPM: Office of Personnel Management: 

TED: Training and Executive Development: 

VA: Department of Veterans Affairs: 

VALU: Veterans Affairs Learning University: 

VBA: Veterans Benefits Administration: 

VHA: Veterans Health Administration: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

September 17, 2012: 

The Honorable Daniel K. Akaka:
Chairman:
Subcommittee on Oversight of Government Management, the Federal 
Workforce, and District of Columbia:
Committee on Homeland Security and Governmental Affairs:
United States Senate: 

Dear Chairman Akaka: 

Constrained budgets and the need to address gaps in critical federal 
skills and competencies make it is essential that agencies identify 
the appropriate level of investment and establish priorities for 
employee training and development, so that the most important training 
needs are addressed first.[Footnote 1] The Office of Personnel 
Management (OPM) and agency chief human capital officers (CHCOs) play 
important roles in ensuring that federal training dollars are invested 
wisely. OPM provides leadership and guidance on the establishment and 
operation of efficient federal training and development programs 
government-wide and advises the President on means for furthering and 
strengthening federal training programs. CHCOs statutorily have the 
primary role in setting the workforce development strategies of their 
agencies, and advising and assisting their agency heads who are 
responsible for establishing priorities for needed training and 
development, and providing for the use of funds and resources in 
accordance with these priorities. The decisions that these key players 
make in determining how to invest federal training dollars will 
ultimately have an important impact on how well the federal government 
is equipped to address its current and future performance and fiscal 
goals and challenges. 

In order to better understand how federal training investment 
decisions are made and whether improvements are needed, you asked us 
to review the methods that agencies are using to establish their 
training investment strategies and OPM's training investment guidance 
to agencies. Accordingly, this review assesses the extent to which (1) 
CHCOs of selected federal agencies have established processes to set 
and prioritize training investments that are aligned with leading 
practices and (2) OPM's guidance and assistance for developing 
training investment strategies align with these leading practices. 

For the purposes of this review, we define the key terms "training", 
"development" and "agency-wide" in the following ways: 

* Training is making available to employees planned and coordinated 
educational programs of instruction in professional, technical, or 
other fields that are or will be related to the employee's job 
responsibilities. Training can be accomplished through a variety of 
approaches, such as classroom training, e-learning, and professional 
conferences that are educational or instructional in nature.[Footnote 
2] 

* Development is generally considered to include training, structured 
on-the-job learning experiences, and education. Developmental programs 
can include experiences such as coaching, mentoring, or rotational 
assignments. 

* Agency-wide includes all components, sub-agencies or offices within 
a cabinet department or independent agency. 

For both objectives of the review, we compared OPM and CHCO practices 
against eight federal training investment leading practices, which are 
based on our prior studies; other expert studies; and statutory, 
regulatory and executive order training requirements.[Footnote 3] (See 
table 1). OPM reviewed these criteria and agreed that they are 
practices that agencies should be implementing to support effective 
training investment decisions. While some of the leading practices we 
identified are related to statutory, regulatory and executive order 
training requirements, the leading practice questions we examined were 
not developed to assess whether agencies are compliant with these 
requirements.[Footnote 4] OPM officials agreed agency responses to our 
questions do not indicate whether or not agencies are in compliance 
with statutory, regulatory, and executive order training requirements. 
Therefore, we do not make determinations regarding compliance with 
these requirements in this review. 

Table 1: Leading Training Investment Practices: 

Agencies should implement: 

Practice 1: (a) Identify the appropriate level of investment to 
provide for training and development efforts and (b) prioritize 
funding so that the most important training needs are addressed first. 

Practice 2: Identify the most appropriate mix of centralized and 
decentralized approaches for its training and development programs. 

Practice 3: Consider government-wide reforms and other targeted 
initiatives to improve management and performance when planning its 
training and development programs. 

Practice 4: Have criteria for determining whether to design training 
and development programs in-house or obtain these services from a 
contractor or other external source. 

Practice 5: Compare the merits of different delivery mechanisms (such 
as classroom or computer-based training) and determine what mix of 
mechanisms to use to ensure efficient and cost-effective delivery. 

Practice 6: Track the cost and delivery of its training and 
development programs agency-wide. 

Practice 7: Evaluate the benefits achieved through training and 
development programs, including improvements in individual and agency 
performance: (a) Has a formal process for evaluating employee 
satisfaction with training; (b) Has a formal process for evaluating 
improvement in employee performance after training; (c) Has a formal 
process for evaluating the impact of training on the agency's 
performance goals and mission. 

Practice 8: Compare training investments, methods, or outcomes with 
those of other organizations to identify innovative approaches or 
lessons learned. 

Source: GAO analysis based on prior GAO reports, other related expert 
studies, and federal training requirements. 

[End of table] 

To obtain government-wide information on agency training investment 
practices, we obtained high-level information from a Council made up 
of CHCOs representing 27 agencies (known as the CHCO Council) through 
a questionnaire on their training investment practices and processes. 
[Footnote 5] To obtain additional perspective and insights on the 
training investment practices identified in the questionnaire, we 
discussed the responses with the CHCO Council and a Council made up of 
the Chief Learning Officers (CLO) from the 27 agencies (known as the 
CLO Council). We describe these Councils in greater detail in the 
background of this report. Based on the responses to the questionnaire 
and workforce size, we selected four agencies--the Department of 
Homeland Security (DHS), Department of Veterans Affairs (VA), 
Department of the Interior (DOI),[Footnote 6] and Department of Energy 
(DOE)--from which to obtain illustrative examples of how they 
implemented the training investment practices identified in the 
questionnaire. As part of our review of agency practices, we also 
obtained information on the steps that agencies are taking to identify 
and prioritize investment allocations for training required to 
implement the GPRA Modernization Act of 2010 (GPRAMA).[Footnote 7] 

To identify and assess OPM's guidance and assistance to agencies on 
training investment strategies, we reviewed OPM training guidance, and 
relevant documentation on forums, workshops or other assistance and 
activities. In addition, we interviewed officials from the OPM offices 
with primary responsibility for providing training policy guidance and 
technical assistance to agencies. We compared this information to the 
leading practices identified in table 1. We also identified and 
described the steps that OPM has taken to identify the skills and 
training needed to implement performance management improvements, such 
as those required by GPRAMA, as a foundation for future agency 
training investments. However, we did not assess the effectiveness of 
OPM's efforts to identify GPRAMA-related skills and actions to develop 
related training. 

Based on information obtained from agencies and OPM, we assessed which 
leading training investment practices were being implemented by 
agencies and addressed by OPM guidance and assistance. We also 
identified the challenges or limitations reported by agencies to 
implementing the practices, and opportunities for improvement in 
agency processes and related OPM guidance. 

We conducted this performance audit from December 2011 to September 
2012, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. Further 
details of our objectives, scope, and methodology are provided in 
appendix I. 

Background: 

The federal government has established a policy to develop its 
employees through training programs, to improve public service, 
increase efficiency and economy, and build and retain a force of 
skilled and efficient employees, among other things. In 1967, 
President Johnson signed Executive Order No. 11348, to provide agency 
heads and OPM with presidential direction on how training is to be 
carried out.[Footnote 8] Under Executive Order No. 11348, OPM is 
responsible for planning and promoting the development, improvement, 
coordination, and evaluation of training in accordance with chapter 41 
of title 5 of the U.S. Code and the established policy. Chapter 41 of 
title 5 sets forth the statutory framework for federal government 
training and development. The executive order further requires OPM to 
identify functional areas in which new or expanded interagency 
training activity is needed and either conduct such training or 
arrange for agencies having the substantive competence to do so; as 
well as to coordinate interagency training conducted by and for 
agencies. It also requires OPM to assist agencies in developing sound 
programs and financial plans for training and provide advice, 
information, and assistance to agencies on planning, programming, 
budgeting, operating, and evaluating training programs. In addition to 
these activities, OPM provides advice and assistance to agencies on 
training and development programs. OPM's Training and Executive 
Development (TED) group (a subcomponent within the Office of Executive 
Resources) is the primary office that provides policy direction and 
leadership to agencies in developing plans and strategies to implement 
training and development programs. It also provides agency guidance to 
ensure the government's training and development programs support 
strategic human capital investments. The TED group provides assistance 
through two main mechanisms: guidance documents and technical 
assistance. OPM has developed five guides that agencies can use as 
references for different aspects of making or reporting training 
investment decisions in the planning, designing, implementation, and 
evaluation phases of their training and development programs (see 
table 2). 

Table 2: OPM's Guidance Documents and Description: 

Guidance document: Guide to Human Resources Reporting[A] (March 2012); 
Description: This guide assists agencies with preparing and submitting 
human resources, payroll, and training data files to OPM's Enterprise 
Human Resources Integration (EHRI) data warehouse.[B] Agencies are 
required to provide updates of their training information to OPM's 
electronic data collection system, including training costs[C]. 

Guidance document: Training Evaluation Field Guide (January 2011)[D]; 
Description: This guide provides information on methodologies for 
conducting training evaluations with the goal of identifying cost 
effective training initiatives that maximize mission accomplishments. 
OPM developed this guide based, in part, on the statutory requirements 
related to agency evaluation of training programs under 5 U.S.C. § 
4103(c) as implemented by OPM regulations 5 CFR § 410.202. 

Guidance document: Draft Training Policy Handbook (December 2011); 
Description: This guide[E] provides guidance to agencies on 
implementing training activities required by laws and regulations on 
training and development in the federal government. 

Guidance document: Guide for Collection and Management of Training 
Information (August 2008); 
Description: This document provides guidance to assist agencies in 
complying with the legal and regulatory instructions for collecting 
and managing federal training information. Under 5 CFR § 410.601 
agencies are required to maintain and report to OPM government-wide 
Electronic Data Collection System training data on its employees' 
training and development activities. 

Guidance document: Guide to Strategically Planning Training and 
Measuring Results[F] (July 2000); 
Description: This document provides guidance to agencies in planning 
the allocation and use of training resources. Among many other things, 
it provides ideas and tools to help agencies identify potential 
training investments that will assist them in achieving their goals 
and accomplishing their mission, and provides approaches and models to 
assist agencies in evaluating training's contribution to the 
organization. 

Source: GAO analysis of OPM guidance documents. 

[A] OPM, Guide to Human Resources Reporting, accessed August 6, 2012, 
[hyperlink, http://main.opm.gov/feddata/ghrr/index.asp]. 

[B] The OPM's EHRI program's data warehouse is the government's 
premier source for integrated federal workforce information. The 
system currently collects, integrates, and publishes data for 2 
million executive branch employees on a biweekly basis, supporting 
agency and government-wide analytics. 

[C] 5 C.F.R. § 410.601. 

[D] This draft is an updated version of OPM's Training Policy 
Handbook: Authorities and Guidelines published in May 11, 2007. 

[E] OPM is currently updating the Training Policy Handbook to align it 
with the updated regulations on training (5 C.F.R. part 410) and 
management development (5 C.F.R. part 412). We obtained a draft of the 
updated Training Policy Handbook. 

[F] In July 2000, OPM developed this guide in response to Exec.Order 
No.13111, Using Technology to Improve Training Opportunities for 
Federal Government Employees, 64 Fed. Reg. 2793 (Jan. 12, 1999). 

[End of table] 

The TED group also provides technical assistance on agency training 
investments through facilitating discussions and forums, and providing 
training to agencies' human resources (HR) staff. For example, the TED 
group uses various web-based mechanisms--such as OPM's website, OPM 
LISTSERV, OPM Federal Training and Development web site and OPM 
Federal Training and Development Wiki--to facilitate discussions 
between agencies on training investments and to share guidance with 
agencies. In addition to these facilitated discussions and forums, the 
TED group provides training to federal HR professionals in various 
areas, including activities that support making training investment 
decisions. For example, OPM provides training to HR staff through its 
partnership with the CHCO Council to operate HR University, which OPM 
officials and the HR University website report is the federal 
government's single "one stop" training resource center for the HR 
professional throughout the federal government. HR University is an 
effort that is intended to achieve government-wide savings through 
pooling and sharing training resources and identifying the best HR 
training across government.[Footnote 9] 

Agencies have the primary responsibility for establishing, operating, 
maintaining, and evaluating their training programs in support of 
achieving their mission and goals. OPM regulations specify that agency 
employee developmental plans and programs should be designed to build 
or support an agency workforce capable of achieving agency mission and 
performance goals and facilitating continuous improvement of employee 
and organizational performance.[Footnote 10] Furthermore, Executive 
Order No. 11348[Footnote 11] states that agency heads must undertake 
several activities in support of developing employees, including: 

* Review periodically, but not less often than annually, the agency's 
program to identify training needed to bring about more effective 
performance at the least possible cost; 

* Conduct periodic reviews of individual employee's training needs as 
related to program objectives; 

* Conduct research related to training objectives required for program 
improvement and effectiveness; 

* Plan, program, and evaluate training for both short and long-range 
program needs by occupations, organizations, or other appropriate 
groups; 

* Establish priorities for needed training, and provide for the use of 
funds and man-hours in accordance with these priorities; 

* Establish training facilities and services as needed; 

* Extend agency training programs to employees of other agencies and 
assign his employees to interagency training whenever this will result 
in better training, improved service, or savings to the government. 

The CHCO Council, established under the Chief Human Capital Officers 
Act of 2002, provides assistance to OPM and agencies in accomplishing 
federal human capital goals.[Footnote 12] The 25-member CHCO Council 
is composed of the Director of the OPM, who serves as chairman; the 
Deputy Director for Management of the Office of Management and Budget 
(OMB), who acts as vice chairman; the CHCOs of the 15 executive 
departments; and the CHCOs of 8 additional agencies designated by the 
OPM Director.[Footnote 13] Additionally, the CHCO Council has an 
Executive Director from OPM who coordinates and oversees the 
activities of the council. The CHCO Council supports OPM in leading 
federal agencies in the strategic management of human capital, 
providing a forum for senior management officials to exchange HR best 
practices, and informing the dialogue on civil service reform in order 
to build and maintain an outstanding Federal workforce for the Nation. 
According to the CHCO Council's charter, among other purposes, the 
council is to: 

* advocate and assure a culture of continuous learning and high 
performance, developing and implementing effective strategies to 
attract, develop, manage, and retain employees with superior abilities; 

* identify human capital best practices and benchmarks, and apply 
those exemplars to their agencies and the federal government as a 
whole; and: 

* provide leadership in identifying and addressing the needs of the 
federal government's human capital community, including training and 
development. 

To help CHCOs implement their training goals, many of the 24 Chief 
Financial Officers Act agencies[Footnote 14] and smaller agencies 
established Chief Learning Officers (CLOs). These officers 
subsequently formed an informal Chief Learning Officers (CLO) Council, 
which is a community of practice composed of federal CLOs or their 
equivalents who meet periodically to share best practices and create 
learning opportunities for agencies and organizations.[Footnote 15] 
The purpose of the CLO Council is to provide a regular forum for CLOs 
to discuss and collaborate on high-level agency strategic and 
operational issues affecting the Federal learning and workforce 
development community of the federal government. These two Councils, 
in partnership with OPM are to play a key role in assisting agencies 
in the implementation of federal training and development efforts. 

CHCOs are Implementing Leading Practices Related to Training Delivery, 
but Not Practices that Support Making More Cost-Effective Training 
Investments: 

Many CHCOs reported that they are implementing leading practices we 
identified as being important to making strategic training and 
development investment decisions, especially regarding the delivery of 
training. These practices include determining the best mix of 
decentralized and centralized training, considering government-wide 
reforms when identifying their training needs, and measuring employee 
satisfaction with training, among other things. However, many CHCOs 
reported that they are not implementing the leading practices that 
would allow them to make more cost-effective training decisions, such 
as having an agency-wide process for prioritizing training investments 
so that the most important training needs are addressed first and 
comparing the merits of different delivery mechanisms (e.g. classroom 
or computer-based training) to determine what mix of mechanisms will 
be most efficient and cost-effective. All of these practices are 
important to ensuring that training investments will be both effective 
and efficient in equipping federal employees to accomplish their 
agencies' goals. 

CHCOs Reported Implementing Leading Practices that Support Making 
Strategic Training Delivery Decisions: 

Many CHCOs reported that they are implementing six of the leading 
practices that we identified as being important to making strategic 
training and development investment decisions, especially regarding 
the delivery of training, as shown in Table 3. However, regarding the 
leading practice related to tracking training investments agency-wide, 
we found that even those who reported that they track training agency-
wide did not do so completely or reliably. 

Table 3: The Leading Training Investment Practices CHCOs Reported 
Implementing: 

Leading training investment practices: Practice 2: Identify the most 
appropriate mix of centralized and decentralized approaches for its 
training and development programs; 
Number of agencies with the response: 
Yes: 27; 
No: 0. 

Leading training investment practices: Practice 3: Consider government-
wide reforms and other targeted efforts when planning its training and 
development programs; 
Number of agencies with the response: 
Yes: 23; 
No: 4. 

Leading training investment practices: Practice 4: Have criteria for 
determining whether to design training and development programs in-
house or obtain these services from a contractor or other external 
source; 
Number of agencies with the response: 
Yes: 15; 
No: 12. 

Leading training investment practices: Practice 6: Track the cost and 
delivery of its training and development programs agency-wide; 
Number of agencies with the response: 
Yes: 16; 
No: 11. 

Leading training investment practices: Practice 7: Evaluate the 
benefits achieved through training and development programs, including 
improvements in individual and agency performance: 

(a) Has a formal process for evaluating improvement in employee 
performance after training: (a) Has a formal process for evaluating 
employee satisfaction with training; 
Number of agencies with the response: 
Yes: 25; 
No: 1; 
Sometimes: (1. 

(b) Has a formal process for evaluating improvement in employee 
performance after training; 
Number of agencies with the response: 
Yes: 13; 
No: 12; 
Sometimes: 2. 

Leading training investment practices: Practice 8: Compare training 
investments, methods, or outcomes with those of other organizations to 
identify innovative approaches or lessons learned; 
Number of agencies with the response: 
Yes: 24; 
No: 3. 

Source: GAO summary of CHCO responses to a GAO questionnaire. 

[End of table] 

Identifying the Most Appropriate Mix of Centralized and Decentralized 
Approaches for its Training and Development Programs: 

All CHCOs reported that their agencies have implemented this practice. 
We have previously reported that, while neither approach fits every 
situation, agencies need to consciously think about the advantages and 
disadvantages of using centralized and decentralized approaches, 
particularly for the design of training and development programs. 
[Footnote 16] Centralizing design can enhance consistency of training 
content and offer potential cost savings. A decentralized approach to 
training design can enable agencies to tailor training programs to 
better meet local and organizational unit needs. Agencies with 
decentralized approaches often embed training representatives within 
their business lines and field structures to assist in coordination of 
training efforts, including design and development. Nineteen of the 27 
agencies reported that they have both centralized and decentralized 
training processes, while eight reported having completely 
decentralized training processes. Most of these agencies reported that 
their CHCOs or CHCO staff typically make centralized training 
decisions, while the leadership within the components, subagencies or 
offices make mission-specific training decisions. In the questionnaire 
responses, CHCOs identified a range of officials who are involved in 
making training investment decisions at the corporate and sub-agency 
level, including CHCOs and their staff, chief management officers, 
chief executive officers, budget officers, chief information officers, 
and others. A number of agencies also reported that advisory or 
oversight boards or training universities within their agency are 
involved in making training investment decisions. 

In the four agencies that we selected for review to obtain 
illustrative examples of how they implemented the training investment 
practices, the CHCOs or their representatives reported that their 
agencies made a decision to have both centralized and decentralized 
processes because they believe that the components or sub-agencies are 
more knowledgeable about their mission-specific training needs, while 
the central human capital staff can add the most value by managing 
investment decisions for more general training across the department. 
VA--which was one of the four agencies that we selected--established a 
corporate university known as the Veterans Affairs Learning University 
(VALU) to provide training to all VA employees. VALU provides training 
primarily in general areas such as leadership, management and 
supervision, as well some career and technical training. VALU offers 
training to the administrations and staff offices through a request 
process that is based on the training needs that the administrations 
and staff offices identify. Those training needs are required to be 
aligned to VA critical training areas.[Footnote 17] An Enterprise 
Training Advisory Board, established in April 2012, also advises the 
Dean of VALU on the impact of training, potential training 
development, and methods of delivery.[Footnote 18] However, another 
tier of training is also provided within VA's three administrations--
the Veterans Health Administration (VHA), Veterans Benefits 
Administration (VBA), and National Cemetery Administration. Each 
administration independently makes training investment decisions and 
provides training to its employees in mission-specific and some 
general and mandatory areas. The leadership of each administration 
makes decisions about the level and prioritization of these training 
investments. For example, at the VHA the Associate Deputy Under 
Secretary for Health (or equivalent) and, subsequently, the Deputy 
Under Secretary for Health assess the training requested by their 
offices against various criteria, including whether training requests 
are aligned with and support VA and VHA strategic goals, objectives, 
strategies, initiatives, and performance improvement goals. During 
each review these officials prioritize the requests through a voting 
process, and forward selected training to the next level. Ultimately, 
the training is sent for approval to the Under Secretary for Health 
and the VA Chief of Staff. The other three agencies that we met with 
also reported having both centralized and decentralized processes for 
making mission-specific training investment decisions. However, most 
often, decentralized training decisions were not required to be vetted 
with department level leadership for these three agencies. 

Considering Government-wide Reforms and Other Targeted Initiatives 
When Planning its Training and Development Programs: 

Nearly all CHCOs in our review reported that they have a process for 
these considerations. We have previously reported that when planning 
training and development efforts, agencies should look to the actions 
of the administration, Congress, and internal and external auditors by 
considering administration priorities, legislative reforms, and major 
management challenges that might shape agency priorities and 
strategies for training and development.[Footnote 19] As an 
administration focuses its efforts on addressing its priorities, 
agencies can benefit by having mechanisms or processes for considering 
whether and to what extent these initiatives could be linked to 
employees' skills and competencies and the related training and 
development approaches that might be needed. Twenty-three of the 27 
CHCOs who responded to our questionnaire reported having such a 
process in place. For example, 16 of the CHCOs reported that they are 
already setting investment allocations or training priorities to 
implement GPRAMA. 

At DOE, another agency we selected for review to obtain illustrative 
examples, officials reported that they have identified the training 
that the department currently offers and will need to offer to 
implement GPRAMA. The Secretary issued a memo to DOE employees on 
GRPAMA's implementation and is holding town hall meetings on improving 
organizational performance. Another effort that DOE expects to support 
the implementation of GPRAMA is the Goals-Engagement-Accountability-
Results (GEAR) model that OPM and OMB are helping to pilot in DOE and 
four other federal agencies, which includes efforts to improve 
employee performance, among other things.[Footnote 20] According to 
DOE officials and related documentation, DOE's GEAR implementation 
plan includes aligning employee performance management with 
organization performance management and developing training to support 
these goals, which along with initiating knowledge sharing activities, 
will promote improvement of DOE's organizational performance. 

Have Criteria for Determining Whether to Design Training and 
Development Programs in-house or to Obtain These Services from a 
Contractor or Other External Source: 

Many of the CHCOs in our review reported having criteria for this 
purpose. Training can be provided by the agency itself, another 
government agency, a school, a manufacturer, a professional 
association, or other competent persons or groups in or outside of 
government. To aid in making these decisions, agencies should try to 
develop clear criteria for determining when to contract for training 
and development services. We have previously stated that factors that 
agencies should consider in these decisions include the capability of 
in-house staff to develop and implement the training; the prior 
experience, capability, and stability of possible providers in the 
marketplace; and agency limitations on cost, time, and resources 
[Footnote 21] Of the 27 CHCOs included in our questionnaire, 15 
reported that they have criteria for determining whether to design 
training and development programs in-house or obtain these services 
from a contractor or other external source. 

One agency that we selected for review to obtain illustrative examples 
was DOI, which reported implementing this practice, however, the 
extent to which this decision-making process is implemented agency-
wide is unclear. In its questionnaire response, DOI's CHCO reported 
that DOI's Office of Strategic Employee and Organizational Development 
has responsibility for offering corporate training through DOI's 
university. This office decides whether to "make or buy" 
departmentwide training. When we met with DOI officials in the course 
of our review, they explained that although almost all courses are 
delivered by vendors because DOI has no internal trainers, they do 
have a small cadre of instructional designers who can develop some e-
Learning courses. Decisions on whether to develop the courses 
internally are based on various criteria, including whether a course 
can be developed quickly, does not require a significant amount of 
content development, and subject matter experts can be provided to 
support the course development. Although this department level process 
is useful, DOI officials did not know if the bureaus within the 
department consistently use a "make or buy" approach. They reported 
that the larger bureaus have some capacity for in-house development 
while the smaller bureaus do not have this capability. 

Track the Cost and Delivery of its Training and Development Programs 
Agency-wide: 

Many CHCOs reported that their agencies implement this practice, 
although most of the CHCOs who reported that they do not track 
training investments agency-wide were leaders of the agencies with the 
largest workforces.[Footnote 22] We have previously reported that to 
obtain a comprehensive determination of the costs of these 
initiatives, agencies need to find ways around barriers that prevent 
them from fully and accurately identifying the expenses associated 
with all components of their training and development processes. 
[Footnote 23] These costs can include expenses for instructional 
development; participant and instructor attendance; facility, 
material, and equipment costs, and travel and per diem expenses. To 
track the cost and delivery of training and development programs, 
agencies need credible and reliable data from learning management 
systems as well as accounting, financial, and performance reporting 
systems. To the extent possible, agencies also need to ensure data 
consistency across the organization (such as having data elements that 
are pulled from various systems representing the same type of 
information). Variation in the methods used to collect data can 
greatly affect the analysis of uniform, quality data on the cost and 
delivery of training and development programs. In response to our 
questionnaire, 16 CHCOs reported that they track training investments 
agency-wide. 

In our review of agencies to obtain illustrative examples of 
practices, two of the selected agencies, DOE and VA , were among the 
CHCOs who reported in their questionnaire that they track training 
investments agency-wide. However, our review found both these agencies 
may not do so completely or reliably. Both agencies identified the 
same limitations and challenges as the agencies whose CHCOs reported 
that they do not have a process to track training agency-wide. For 
example, DOE and VA officials reported experiencing challenges in 
tracking training costs agency-wide because of their multiple learning 
management systems and inconsistent practices in tracking training 
data.[Footnote 24] Officials from DHS and DOI, who reported in the 
questionnaire that they could not track training investments agency-
wide, described similar challenges to those experienced by DOE and VA. 
Moreover, officials from all four agencies were unaware of the total 
amount their agencies invest in federal training and stated that they 
could not provide reliable training data to OPM, which requests these 
data to address its government-wide training responsibilities. Under 
OPM regulations, agencies are required to maintain data on training 
activities and expenditures and submit these data to OPM. 

As an example of challenges tracking training investments, DHS 
reported that it is unable to track or better leverage training 
investments across the department because of the nine, major 
incompatible Learning Management Systems that it uses to track 
training throughout the agency. We highlighted these same challenges 
in a 2005 report on DHS training, noting that the lack of common 
management information systems and the absence of commonly understood 
training terminology across components, among other things, may impede 
the agency's ability to achieve its training goals.[Footnote 25] 
According to more recent documentation on the limitations of DHS' 
tracking systems, the components' disparate systems currently limit 
them from sharing useful training information across the department, 
effectively aggregating training data agency-wide, and reporting 
complete training investment information to OPM. As a result, DHS is 
seeking to purchase a single learning management system.[Footnote 26] 

Even when agencies had a single training information system, the 
components may not consistently use them to track training investments 
because of inconsistent coding schemes for tracking similar training 
activities. For example, even though DOI has a single system for 
tracking training information, officials reported that their human 
capital office must rely on employees or data stewards[Footnote 27] to 
input their training data and some cost data may not be included, such 
as training travel costs, or certain types of training may not be 
entered, such as conferences. Such costs are sometimes paid directly 
by an employee's immediate office using a government credit card and 
are not tracked as training. In addition, learning management system 
training data are often not reconciled with DOI's financial 
expenditure data because, until recently, their financial systems have 
not captured education tuition and training fees, and are still unable 
to track training travel costs. Therefore DOI's cost data are most 
likely incomplete. Similarly, officials from DOE, DHS, and VA reported 
that they are aware of some inconsistencies in whether some types of 
training, such as conferences, are entered into their learning 
management systems. They also stated that there are inconsistencies in 
how agency components capture and code workforce training into their 
system because they lack a common definition for what types of 
activities should be considered training or have varying coding 
schemes or tools for capturing the cost. For example, some 
organizations use a procurement request or obtain a contractor to 
deliver training and do not document these costs using the standard 
government form for tracking data or in their learning management 
systems. In other cases, training investment data are captured using 
different coding in various learning management systems and financial 
systems. Some officials report that reconciling these data would be 
difficult. For example, DOE's chief financial officer reported that it 
takes a couple of months to gather training investment data from DOE's 
various systems, partly because the systems have inconsistent coding 
for these data. DOE' officials reported that changing financial codes 
to reconcile training data would be time consuming and expensive 
because their financial systems are 20 years old. However, after years 
of highlighting this challenge, they are seeking approval to make such 
changes. Officials from the four agencies generally reported that, as 
a result of all of these factors, there is no overarching awareness or 
oversight of how much is spent on training investments and for which 
activities. 

Evaluate the Benefits Achieved Through Training and Development 
Programs, Including Improvements in Individual and Agency Performance: 

Nearly all CHCOs reported having a formal process to evaluate employee 
satisfaction with training, but fewer had processes to evaluate the 
impact of training on employees or agency performance. We have 
previously reported that it is increasingly important for agencies to 
be able to evaluate their training and development programs and 
demonstrate how these efforts help develop employees and improve the 
agencies' performance because it can aid decision makers in managing 
scarce resources, and provide credible information on how training and 
development programs have affected organizational 
performance.[Footnote 28] To do so agencies need to develop evaluation 
processes that systematically track the cost and delivery of training 
and development efforts and assess the benefits of these efforts. 
Training and development programs can be assessed by measuring (1) 
participant reaction to the training program, (2) changes in employee 
behavior or performance; and (3) the impact of the training on program 
or organizational results, which may include a return on investment 
assessment that compares training costs to derived benefits.[Footnote 
29] Some of these methods can help provide better value through 
identifying areas for continuous improvement in training programs. We 
consider the processes for conducting these evaluations to be formal 
when they are systematically conducted throughout the agency, have 
established guidelines and criteria that govern how they are 
implemented and are documented. However, CHCOs may also have other 
criteria for determining what is considered a formal process, based on 
their agencies' environment. We asked CHCOs about their formal 
processes for conducting the three levels of evaluation listed 
earlier, which are the common types of evaluations. Many CHCOs 
reported routinely implementing the first two, but not the third 
(which we discuss later in this report). Twenty-five of the 27 CHCOs 
included in our questionnaire reported that they measure employee 
satisfaction; and a little more than half reported that they measure 
improvement in employee performance. 

Officials from the four agencies that we interviewed reported that 
they all assess employee's reaction to training and sometimes assess 
changes in employee performance. For example, officials from DOE 
reported that they evaluate all the training that they offer by 
surveying participants' reactions to the training--which can include 
their feedback on the effectiveness of the instructor, the topics, the 
presentation style, the schedule, audiovisuals, and other subjects--
and use this information to make revisions to the program courses. 
Documents that we reviewed on training evaluations identified updates 
or revisions made to course materials and tests to improve their 
effectiveness, based on training feedback and policy updates. As an 
example of evaluating the impact of training on employee performance, 
DOI officials stated that, while they do not have an agency-wide 
process, some of their organizations--such as those within the Bureau 
of Land Management and National Park Service use an online evaluation 
tool to assess the impact of training courses on employees' abilities 
to perform tasks, about 6 weeks after a course has been completed. 
According to the official, at this time, the process is not used 
department-wide, but the agency is looking into how it may be able to 
do so starting in fiscal year 2013. According to DOI's CLO, 
establishing this link between training, employee competencies and 
mission critical occupation work is one that DOI is targeting for 
improvement. 

Compare training investments, methods, or outcomes with those of other 
organizations to identify innovative approaches or lessons learned: 

Nearly all CHCOs reported that they implement this practice. We have 
previously reported that there are many ways to help improve 
performance, so it is important for agencies to continually look to 
others to identify innovative approaches that may relate to their 
training and development efforts. Within the context of that agency's 
unique environment and situation, an agency can compare its 
investments, approaches, and outcomes with those of public and private 
organizations that are undertaking notably innovative and effective 
training and development efforts. Agencies can uncover weaknesses in 
their training and development strategies that need improvement and 
identify new ideas, mechanisms, and metrics that they could employ. 
[Footnote 30] Twenty-four of the 27 CHCOs included in our 
questionnaire reported that they compare training investments, 
methods, or outcomes with those of other organizations to identify 
innovative approaches or lessons learned. 

Officials from the two agencies we asked to provide examples of this 
practice described this process as occurring informally through 
interactions with other CLO Council members. For example, DHS 
officials that we met with reported that they meet with other agencies 
to share best practices and recommend vendors during breaks or after 
the CLO meetings. The officials said that examples of sharing ideas on 
new training programs included recent discussions by OPM and agencies 
on the GEAR pilot program lessons learned and new courses for 
developing supervisors. While two agencies reported having informal 
interactions with other agencies to share and compare training 
information, none of the agencies that we met with described efforts 
to benchmark their practices with agencies or other relevant entities. 
We have previously reported that benchmarking can help agencies 
determine who is the very best, who sets the standard, and what that 
standard is.[Footnote 31] 

Many CHCOs are Not Implementing the Leading Practices that Support 
More Cost-Effective Training Investment Decisions: 

Many CHCOs reported that they are not implementing the leading 
practices that would allow them to make more cost-effective training 
decisions, as shown in Table 4. 

Table 4: Leading Training Investment Practices that Many CHCOs 
Reported Not Implementing Routinely: 

Leading training investment practices: Practice (1a): Identify the 
appropriate level of investment to provide for training and 
development efforts; 
Number of CHCOs with the response: 
Yes: 11; 
No: 16. 

Leading training investment practices: Practice (1b): Prioritize 
funding so that the most important training needs are addressed first; 
Number of CHCOs with the response: 
Yes: 12; 
No: 15. 

Leading training investment practices: Practice 5: Compare the merits 
of different training delivery mechanisms and determine what mix to 
use to ensure efficient and cost-effective delivery; 
Number of CHCOs with the response: 
Yes: 11; 
No: 16. 

Leading training investment practices: Practice 7(c): Have a formal 
process for evaluating the impact of training on the agency's 
performance goals and mission[A]; 
Number of CHCOs with the response: 
Yes: 8; 
No: 8; 
Sometimes: 11. 

Source: GAO analysis of CHCO responses to a GAO questionnaire. 

[A] One agency that reported "I don't know" was included in the 
category "No". 

[End of table] 

Identify the Appropriate Level of Investment to Provide for Training 
and Development Efforts and Prioritize Funding So That the Most 
Important Training Needs are Addressed First: 

Many CHCOs included in our review reported that they have not 
implemented this practice. We have previously stated that, to 
determine the best ways to leverage investments and establish 
priorities, agencies can develop an annual training plan that targets 
developmental areas of greatest need and that outlines the most cost-
effective training approaches to address those needs.[Footnote 32] 
When assessing investment opportunities for its training plan, the 
agency ought to consider the competing demands confronting the agency, 
the limited resources available, and how those demands can best be met 
with available resources. If training is identified as a solution to 
improve agency performance, agencies can prioritize training using 
criteria, such as expected demand for the investment from internal 
sources, availability of resources to support the effort, potential 
for increased revenue, and risk of unfavorable consequences if 
investments are not made. Given current budget constraints, agencies 
may also want to prioritize training that has the potential to improve 
their efficiency. Developing a business case for training and 
development that includes this information sets forth the expected 
costs and benefits of the investments and provides decision makers 
with essential information they need to allocate necessary resources. 
Furthermore, under Executive Order No.11348 and OPM regulations, 
agencies are to establish training priorities, although agencies are 
not specifically instructed to establish an agency-wide process to do 
so.[Footnote 33] Of the 27 CHCOs questionnaire responses, 16 CHCOs 
reported that they do not set a level of investment agency-wide and 15 
CHCOs reported that they do not prioritize training agency-wide. 

In our meetings with officials from the DOE, DOI, DHS, and VA as well 
as the CLO council, agency officials cited several reasons for why 
they do not establish a level of training investment agency-wide or 
prioritize training agency-wide. Some of the reasons were described as 
purposeful decisions not to do so and other reasons were described as 
limitations in their ability to do so. First, CHCOs elect to establish 
and prioritize training investments for centralized training and are 
often not involved in the investment decisions made for specific 
training within the components or offices, as we previously described. 
In addition, large components or sub-agencies often have autonomy over 
their training budgets because the budgets are appropriated directly 
to them from Congress. As a result, CHCOs and their staff are often 
unaware of how much these components spend for training and do not 
have input into these decisions. Component and sub-agency heads often 
act autonomously and are not required to communicate with the CHCO 
about these decisions. Further, because of limitations in internal 
tracking systems for training (which we discussed earlier in this 
report), CHCOs do not have information on all of the training that is 
completed in their agency and the related costs. 

Officials from various agencies involved in the CLO Council and three 
of the four agencies that we individually met with reported having a 
lack of visibility into the prioritization and level of training 
investments throughout their agencies, which they reported limits 
their ability to better leverage and reduce duplication in training 
investments their agencies. Officials in the agencies that we met with 
reported that, although they believe that their components or 
organizational elements are more capable of making training decisions 
related to their specific missions, the lack of coordination and 
communication on training investments and priorities has led to some 
duplicative and ineffective training investments in their departments. 
For example, senior human capital officials in DOI reported that the 
department's leadership, including the CHCO are not aware of the 
department's overall training investments agency-wide and have no 
formalized mechanism for ensuring accountability for how the funds are 
used. They are aware that bureaus are buying duplicative training or 
offering similar training classes that are of varying effectiveness--
which is resulting in inefficient training investments. For example, 
one bureau recently independently contracted with an external provider 
for mid-level manager leadership training that was already offered at 
DOI's university and paid $50,000 more than DOI University charges. 
According to officials, this is a common problem. In addition to 
duplicative training courses, in some cases, bureaus are duplicating 
the creation of new training facilities. For example, a regional 
director of an DOI bureau built a training classroom with a computer 
lab, despite having access to existing computer labs within the 
complex where he worked and also at DOI facilities a few miles way. 
Further, according to the officials, because it is common practice for 
each bureau to independently secure training, there is no consistency, 
little quality control, and no maximization of procurement tools (such 
as blanket purchase agreements) across DOI.[Footnote 34] In order to 
address these challenges, DOI has formed a one-time departmentwide 
task force known as the Department Innovation and Efficiency Team for 
Training. This task force was expected to identify: potential 
duplication in training, funds expended in training delivery, and the 
cost of travel and facilities, among other things. In July 2012, the 
committee made recommendations to the CLO on opportunities to generate 
efficiencies and savings in training operations. DOI's Office of 
Strategic Employee and Organization Development is developing action 
plans to address the committee's recommendations. 

Officials from DHS also reported experiencing similar challenges with 
duplicative or ineffective training investments in their agencies. 
Some of these challenges are long standing. For example, seven years 
ago we reported that DHS's two-tier training process (component and 
departmentwide) and lack of communication throughout the department on 
the availability of some training programs and resources were 
challenges that could impede its ability to achieve departmental 
training goals and efficiencies.[Footnote 35] DHS is still taking 
steps to address this on-going challenge. In June 2005, DHS formally 
chartered a Training Leaders Council (TLC) and recently revised its 
charter in June 2011. The TLC is made up of senior training leaders 
from each component, and representatives from headquarters to serves 
as an advisory and collaborative community of practice to promote 
effective and efficient training, education, and professional 
development opportunities to DHS employees. According to DHS' Human 
Capital leaders, while this group does not set or prioritize training 
investments, it provides a forum for exchanging useful information 
about common challenges and training practices, which helps in making 
more efficient use of existing agency resources. DHS also established 
the Human Resource Information Technology Executive Steering 
Committee, made up of management chiefs and HR and information 
technology leadership across DHS in 2010, and included TLC leadership 
as members in July 2011. This group makes some funding decisions 
related to some training investments, such as their recent decision to 
fund the purchase of a single learning management system for the 
entire department. However, according to DHS officials, because DHS 
has multiple congressional committees and subcommittees from which the 
components receive funding and training direction, coordinating 
training investments remains challenging. 

Compare the Merits of Different Training Delivery Mechanisms and 
Determine What Mix to Use to Ensure Efficient and Cost-Effective 
Delivery: 

Many CHCOs that responded to our questionnaire reported that they do 
not compare the merits of different training delivery mechanisms. Our 
past research and that of others has shown that agencies should 
deliberatively consider the options for delivering training and 
consider essential issues, such as the goals and objectives for the 
training, the type of audience intended for the training, the nature 
of the training content, the availability of technology and resources, 
and the timing for delivering the training.[Footnote 36] Agencies can 
use a variety of instructional approaches to achieve learning--in the 
classroom, through distance learning, or in the workplace. When 
warranted, agencies should also consider blended learning that 
combines different teaching methods (e.g. Web-based and instructor-
led) within the same training effort and provide trainees with the 
flexibilities to choose among different training delivery methods 
while leveraging resources in the most efficient way possible. When 
assessing delivery options, agencies can try to achieve economies of 
scale and avoid duplication of effort by taking advantage of existing 
course content or training, such as sharable on-line courses or 
multiagency training programs. However, In the responses to our 
questionnaire, 16 of the 27 CHCOs reported that they do not compare 
the merits of the different training delivery mechanisms in their 
agency. 

In our meetings with DHS and VA to obtain illustrative examples, DHS 
officials reported that their current learning management systems do 
not allow them to mine information on the different delivery 
mechanisms used throughout the department and to assess and compare 
their effectiveness. According to the officials, they could obtain 
this information manually, but it would be a very labor intensive 
process. Therefore, it is not done. In contrast, VA officials informed 
us that they are assessing different delivery mechanisms for training 
and conferences offered by VALU because they recognize that 
opportunities exist to offer more efficient mechanisms (such as e-
learning). Moreover, VHA, which has the largest workforce in the 
department, builds into its initial investment decision-making process 
considerations of which delivery methods will be most effective and 
efficient, and subsequently evaluates employee satisfaction with the 
various delivery methods to inform future investment decisions. 
Without processes such as these, agencies that do not compare the 
merits of different training delivery mechanisms have limited 
information for determining what mix of methods provides the most 
efficient and effective delivery of federal training. 

Have a Formal Process for Evaluating the Impact of Training on the 
Agency's Performance Goals and Mission: 

Most CHCOs reported that their agencies do not have a routine formal 
process to implement this practice. As we previously mentioned, it is 
increasingly important for agencies to be able to evaluate their 
training and development programs and demonstrate how these efforts 
help to improve the agencies' performance, and to assist them in 
making more effective decisions about how to allocate scarce 
resources. Agencies are required by statute and OPM implementing 
regulations to evaluate how well training programs contribute to 
mission accomplishment and meet organizational performance goals. 
[Footnote 37] We have identified having a formal process for this 
evaluation as a leading practice. However, there are some 
understandable limitations to regularly and formally implementing this 
practice. For example, some agency officials that we met with reported 
that the cost and time required can be significant for obtaining 
results of evaluations of training that measure the impact on agency 
performance goals. As a result, they can only conduct this level of 
review for training that they identify as highly important to key 
areas of their mission. We have previously reported that not all 
training and development programs require, or are suitable for, higher 
levels of evaluation.[Footnote 38]For example, it may be ineffective 
to try to measure the impact of training in an area that is still 
undergoing other significant changes that could affect relevant 
performance goals, such as changes in related policy and management 
structure. We recognize that higher levels of evaluation (such as 
evaluating the impact on organizational performance or return on 
investment) can be challenging to conduct because of the difficulty 
and costs associated with data collection and the complexity in 
directly linking training and development programs to improved 
individual and organizational performance. Factors to consider when 
deciding the appropriate level of evaluation include estimated costs 
of the training effort, size of the training audience, management 
interest, program visibility, and the anticipated "life span" of the 
effort. Each agency will need to consider the feasibility and cost-
effectiveness of conducting these in-depth evaluations, along with 
budgetary and staffing circumstances that may limit the agency's 
ability to complete such evaluations. Given the current budget 
constraints that agencies face, making thoughtful tradeoffs regarding 
how to target costly evaluation reviews is a sensible approach. While 
it is important to prioritize reviews of training, 8 of the 27 CHCOs 
that responded to our questionnaire reported that they do not have a 
formal process for evaluating the impact of their training on their 
agency's performance. 

For example, the CHCO at DOE reported in our questionnaire that DOE 
does not implement this practice. We met with the CLO from DOE who 
informed us that DOE does not have a formal process for implementing 
this practice because the agency does not have a systematic documented 
approach for conducting this level of review. Moreover, evaluation 
data are not collected in a way that allows it to be aggregated into a 
comprehensive assessment of its impact on the agency's overall 
mission. For example, different organizations within DOE conduct 
reviews to assess the impact of training on their goals, but they are 
not captured in an automated system and the methodologies that DOE 
organizations use to conduct these reviews vary. As an illustration, 
DOE organizations that work with nuclear material evaluate the 
technical training that they provide to their employees against 
required certification and mission goals. However, the organizations 
conduct these evaluations differently, and because of these varied 
methodologies and lack of automated results data, it is difficult to 
aggregate the reviews into an assessment of how training has affected 
DOE's overall training and mission goals. Similarly, the CLO's office 
evaluates cross-cutting training for employee satisfaction and 
employee performance at DOE, but does not effectively or consistently 
evaluate its impact on agency goals. According to the CLO, to assist 
them in developing a more systematic formal process, they are 
participating in OPM training on developing training evaluations and 
in the GEAR pilot program--which is intended to better link employee 
performance to organizational goals. 

In contrast, VA's training review processes illustrate that agencies 
that have a formal process for assessing the impact of training on 
their performance mission and goals can use it to make better training 
investment decisions. VA recently assessed the return on investment of 
its corporate training and, the department's Administrations recently 
evaluated the impact of mission-specific training on their performance 
goals. In January 2012, VA evaluated the monetary and mission-related 
benefits of training that was implemented under its Human Capital 
Investment Plan.[Footnote 39] According to the return on investment 
assessment and report developed by VALU and VHA's National Center for 
Organization Development, VA' s two-year $577 million investment in 
training and development under VA's Human Capital Investment Plan has 
resulted in $604 million dollars in savings that are tied to 
reductions in costly VA turnovers, fewer overdue accounts renewable, 
and fewer equal employment opportunity complaints.[Footnote 40] The 
report also states that VA has gained non-financial returns, such as 
faster benefits processing, increased veteran hiring programs, and 
improved patient satisfaction. According to VALU officials, they have 
used details in this report along with other factors to make decisions 
about future training and development investments. Similarly, for its 
mission-specific training, VHA recently conducted an in-depth review 
of training provided to Patient Aligned Care Teams to improve their 
collaborative delivery of care to patients. The evaluation assessed 
the training participants' satisfaction, skill acquisition, 
application on the job, and impact on VHA's business. The assessment 
ultimately determined that the training was successful in addressing 
the desired behavior changes in the work place and that key 
organizational results were influenced by the training, but it also 
identified some improvements that VHA could make.[Footnote 41] 

OPM Guidance and Assistance to Agencies on Federal Training 
Investments Addresses Many Leading Practices that We Identified, But 
Could be Improved in Areas That Are Challenges to Agencies: 

OPM guidance and assistance to agencies on federal training 
investments are in line with five of the eight leading practices, but 
OPM lacks guidance and assistance in some areas that are challenges to 
agencies, as shown in table 5. 

Table 5: Comparison of Agency Implementation of Leading Practices and 
OPM Guidance and Assistance: 

Leading Practice: Practice 1: (a) Identify the appropriate level of 
investment to provide for training and development efforts and (b) 
prioritize funding so that the most important training needs are 
addressed first[A]; 
Not Addressed by OPM guidance or assistance. 

Leading Practice: Practice 2: Identify the most appropriate mix of 
centralized and decentralized approaches for its training and 
development programs; 
Not Addressed by OPM guidance or assistance. 

Leading Practice: Practice 3: Consider government-wide reforms and 
other targeted initiatives to improve management and performance when 
planning its training and development programs; 
Addressed by OPM guidance or assistance. 

Leading Practice: Practice 4: Have criteria for determining whether to 
design training and development programs in-house or obtain these 
services from a contractor or other external source; 
Not Addressed by OPM guidance or assistance. 

Leading Practice: Practice 5: Compare the merits of different delivery 
mechanisms (such as classroom or computer-based training) and 
determine what mix of mechanisms to use to ensure efficient and cost-
effective delivery; 
Addressed by OPM guidance or assistance. 

Leading Practice: Practice 6: Track the cost and delivery of its 
training and development programs; 
Addressed by OPM guidance or assistance. 

Leading Practice: Practice 7: Evaluate the benefits achieved through 
training and development programs, including improvements in 
individual and agency performance; (a) Has a formal process for 
evaluating employee satisfaction with training; (b) Has a formal 
process for evaluating improvement in employee performance after 
training; (c) Have a formal process for evaluating the impact of 
training on the agency's performance goals and mission; 
Addressed by OPM guidance or assistance. 

Leading Practice: Practice 8: Compare training investments, methods, 
or outcomes with those of other organizations to identify innovative 
approaches or lessons learned; 
Addressed by OPM guidance or assistance. 

Source: GAO analysis of OPM guidance and assistance tools. 

[End of table] 

OPM Provides Guidance or Assistance that Addresses Some Leading 
Training Investment Practices: 

OPM guidance or assistance to agencies on federal training investments 
addresses five of the eight leading practices for federal agency 
training investment decision-making processes. OPM's five primary 
guidance documents that relate to making training investment decisions 
include the Guide to Human Resources Reporting, Training Evaluation 
Field Guide, Draft Training Policy Handbook, Guide for Collection and 
Management of Training Information, and Guide to Strategically 
Planning Training and Measuring Results. (See table 2 for a brief 
description of these guides). In addition, OPM provides technical 
assistance to agencies via facilitated forums, discussion, and 
training. 

Consider Government-wide Reforms and Other Targeted Initiatives to 
Improve Management and Performance When Planning its Training and 
Development Programs: 

Our review of OPM's guidance documents and assistance shows that OPM 
has provided some technical assistance to agencies on this practice, 
although OPM does not have guidance documents that provide specific 
advice on this topic. For example, because of requirements in GPRAMA, 
OPM is providing assistance to agencies in considering this government-
wide reform when planning their training and development programs. 
GPRAMA required OPM to identify the competencies needed to perform the 
following three functions: developing goals, evaluating programs, and 
analyzing and using performance information for the purpose of 
improving government efficiency and effectiveness. OPM, working with 
subject matter experts developed a competency model for the three new 
roles required by GPRAMA--performance improvement officer, performance 
improvement staff, and goal leader. Earlier this year OPM advised 
agencies that it would provide guidance on how to incorporate the 
skills and competencies into these position descriptions, as specified 
in the GPRAMA. The Director of OPM stated that the agency would work 
with the CLOs to incorporate the key skills and competencies into 
agency training programs.[Footnote 42] 

OPM has begun providing this assistance to agencies by facilitating 
sessions for agencies to develop training requirements for 
implementing the new positions and roles required by GPRAMA. For 
example, OPM worked with OMB to gather information on existing 
training, provide learning opportunities, and consolidate new and 
existing training courses and materials to support this effort. Using 
this information, OPM and OMB led two working group meetings with 
agencies to discuss GPRAMA training needs and next steps. In a working 
group meeting in February 2012, OPM and agencies discussed which 
competencies identified in GPRAMA could be improved readily through 
training. OPM provided participants with a chart developed from an OPM 
and the Merit System Protection Board 2011 Trainability Study on which 
competencies for the three new roles required by GPRAMA were highly 
trainable versus those that were less trainable.[Footnote 43] After 
the discussion, OPM and participants identified the most critical and 
manageable next steps, including: 

* create a common competency assessment tool to assess competency gaps 
within agencies; 

* create a course on writing results-oriented goals and standards--
while also gathering existing training; 

* create a working group to assess the needs and create a solution to 
satisfy the training requirement for the Organizational Performance 
Analysis, Planning and Evaluating, and Performance Measurement 
competencies and to collect relevant case studies, as well as to 
identify opportunities to leverage agency resources; 

* identify existing subject matter experts in the agencies and create 
forums, workshops, training sessions, etc. where they can share their 
expertise and possibly engage in peer-to peer coaching; 

* create working groups where necessary; and: 

* consider the development of a career path after OPM's classification 
study. 

In addition to this assistance, although not specific to government-
wide reforms, OPM's Training Policy Handbook advises agencies to 
conduct a training needs assessment that includes an evaluation of 
organization needs, which should take into consideration changing 
demographics, political trends, technology, and the economy. 

Compare the merits of different delivery mechanisms (such as classroom 
or computer-based training) and determine what mix of mechanisms to 
use to ensure efficient and cost-effective delivery: 

OPM's TED group advises agencies in its Training Policy Handbook and 
2000 Guide to Strategically Planning Training and Measuring Results to 
use multiple delivery methods, or combine them, when providing 
training to employees. For example, the Training Policy Handbook 
maintains that agencies should decide which delivery option is best to 
achieve the instructional goals of the training, highlighting that 
some methods are more effective for certain courses. It states that a 
performance management course may include role play scenarios which 
may not be suited for an e-learning course. Further, the guide states 
that agencies need to develop training delivery mechanisms that 
effectively limit unnecessary overlap and duplication of efforts. 
Similarly, we have previously reported that agencies need to consider 
essential issues such as the goals and objectives for the training, 
the type of audience for which the training is intended, the nature of 
the training content, the availability of technology and resources, 
and the timing for delivering the training when identifying the most 
effective and efficient delivery mechanism.[Footnote 44] Agency 
officials who have implemented this practice reported seeing positive 
results. For example, VHA officials that we met with and agencies that 
have publicly discussed their efforts to assess the different delivery 
mechanisms at a March 2012 Partnership for Public Service Forum on: 
Going Virtual-Maximizing the Return On Investment of Online Training 
reported significant savings and increases in the effectiveness of 
their training by assessing and changing their training delivery 
mechanisms. Specifically, VHA officials reported achieving several non-
financial and financial benefits as a result of moving from in-person 
meetings and audio and video conferencing to providing training on-
line for one of its leadership training programs. According to a VHA 
assessment report, the benefits included: consistent curriculum across 
eight medical centers in three states; easier accessibility to course 
materials and job aids; immediate access to feedback on courses from 
learners; easier reproduction of courses for instructors; a return on 
investment of 140 percent since implementation; and $116,000 saved in 
travel costs, facilitation, and facilities; among other things. 

OPM guidance informs agencies that they should implement this 
practice. However, the guidance does not include methodologies for how 
to implement it. Officials from DHS--an agency that reported that it 
does not implement this practice--stated that tools provided by OPM 
could be strengthened to assist them in comparing training delivery 
mechanisms. For example, DHS officials reported that they have 
difficulty implementing this practice partly because their components 
do not track comparative data on the different delivery mechanisms. 
According to the DHS officials, the standard government form for 
tracking training data (Standard Form-182) does have a category for 
tracking training delivery type, but filling out this block is not 
mandatory and is often not used.[Footnote 45] The DHS officials 
reported that an OPM requirement to capture these data would improve 
their ability to gather the information needed from DHS components to 
effectively implement this practice. As noted earlier, 15 of the 27 
CHCOs included in our review, reported that they do not implement this 
practice, which indicates that they may also benefit from additional 
guidance and tools on ways to do so. 

Track the cost and delivery of its training and development programs: 

OPM's TED group provides guidance and assistance to agencies on 
tracking and reporting the cost and delivery of training and 
development programs in four of its five guides. For example, OPM's 
2000 Guide to Strategically Planning Training and Measuring Results 
advises agencies to calculate the cost of the expenses associated with 
designing, developing, implementing, and evaluating their training 
programs and provides a list of the most common types of training 
costs. OPM's Guide for Collection and Management of Training 
Information also outlines agency requirements to track various types 
of data training and provides a list of several data sources (e.g. 
Standard Form-182, agency personnel records, procurement documents, 
financial and performance records, training evaluation forms, etc.) 
that agencies could use to collect this information. Similarly, the 
Training Policy Handbook also incorporates guidance on tracking the 
cost and delivery of agencies' training and development. In the more 
recent 2012 Guide to Human Resources Reporting, OPM outlines 
requirements for agencies to track training data and describes the 
requirement to use certain standard tracking forms, such as the 
standard "Authorization, Agreement, and Certification of Training" 
(Standard Form-182) to track data. The guide also instructs agencies 
to provide all training information included in this form for 
submission to OPM's Enterprise Human Resources Integration (EHRI) 
database systems. 

Although OPM provides several guidance documents and assistance on 
tracking the cost and delivery of training, we found that this 
practice continues to be a challenge for many agencies to implement. 
Agency officials that we met with reported that they could benefit 
from additional assistance from OPM in developing a common definition 
of what should be tracked as training, developing policies to 
strengthen the utilization of Standard Form -182 to document and 
report all training costs, and encouraging agencies through guidance 
and technical assistance, to routinely report training cost data to 
agency learning management systems. 

In addition, to providing guidance on tracking data, OPM facilitates 
the collection of federal training data government-wide. Executive 
Order No. 11348 requires OPM to develop, install, and maintain a 
system to provide the training data needed to carry out its own 
functions and to provide staff assistance to the President. OPM's EHRI 
is the government-wide repository for these training data.[Footnote 
46] As noted above, agencies have been required since 2006 to report 
training data to OPM monthly via this system. However, according to 
OPM officials, they consider the data to be unreliable because they 
are incomplete. Therefore, OPM officials have not used it to inform 
their training guidance and assistance to agencies, to counsel heads 
of agencies and other agency officials regarding federal training 
needs or investments or to assist agencies in developing sound 
programs and financial plans for training programs. 

According to OPM officials and documents, OPM should assess EHRI 
training data for technical compliance and data quality validation. 
Technical compliance is the testing and approving of agency systems 
for data quality (i.e. correct formatting, adherence to edit rules). 
Once systems are technically compliant, agencies are required to send 
monthly data feeds of completed training events to OPM.[Footnote 47] 
Once agencies are reporting these data for all major components, all 
employees, all types of training (e.g. conferences, on-line, 
classroom), and training cost data, OPM are to evaluate the data 
quality to determine if it presents an accurate picture of all 
training in the agency. However, OPM officials told us that they have 
not assessed the quality of data or developed a report on its 
reliability because no agency is sending information on all training 
events. According to OPM officials, when agencies request assistance 
or when OPM finds that an agency has been grossly delinquent in 
providing data, OPM officials will inquire further and offer 
assistance to the agencies. However, they typically do not document 
reliability issues or the agreed upon action plans to address the 
problems. The officials agreed that this is a problem, but stated that 
they would need more staff resources to provide this level of 
assistance and oversight. 

We believe that the current reliability of agency training investment 
data is unknown because OPM officials have not internally assessed 
improvements in the completeness of the data over the last 3 years or 
the quality of the data in the six years that agencies have been 
required to submit it. The two internal reviews that OPM conducted of 
training data were in 2008 and September 2009. In the 2009 review, OPM 
reported that there was an increase from fiscal year 2008 in the 
amount of training data being reported by agencies, but that the 
quality of the data was still less than what was necessary to provide 
an accurate picture of federal training investments. According to the 
2009 report, over half of all agencies were reporting data for the 
entire agency, 86 percent were reporting on a regular basis, but only 
7 percent were reporting cost data. The report identified several of 
the same reasons that we previously described as limitations to 
agencies reporting training investment data. Although the report 
stated that OPM would continue to work with agencies to assess the 
quality/validity of training investment data and determine whether 
agencies are reporting all training events, as noted above, OPM 
officials informed us that they have not assessed the quality of the 
data because the data are not 100 percent complete. 

While it is important to have complete data, we do not believe that 
having incomplete data necessarily prevents OPM from assessing the 
overall reliability of the data, if it meets standards for 
sufficiency. In our guidance on assessing the reliability of computer 
based data, we have stated that agencies can assess data if it is 
sufficiently complete.[Footnote 48] Data are sufficiently reliable 
when testing and reviews of the existing information provide assurance 
that (1) the likelihood of significant errors or incompleteness is 
minimal and (2) the use of the data would not lead to an incorrect or 
unintentional message. Further, we consider the data not to be 
sufficiently reliable when there are significant errors or 
incompleteness in some of or all the key data elements and if using 
the data would probably lead to an incorrect or unintentional message. 
Because OPM has not conducted an assessment of improvements in agency 
training data in three years, it is unknown whether it is currently 
complete enough to test other aspects of its quality and reliability. 
According to the officials, although they have not conducted a formal 
review of the data, they are able to visually look at the EHRI data 
base and tell that the data are significantly more complete than in 
past years. 

OPM also previously identified several steps that its officials would 
take to assist agencies in improving their data, but have not yet 
implemented all of them. According to OPM's 2009 report assessing EHRI 
data, OPM planned to assist agencies in improving training investment 
data, by: (1) working with agencies to fully report all training 
investment data--including costs; (2) working with agencies to 
decrease errors in reporting; and (3) providing individual agencies 
with summary reports of the data that they submitted to OPM for their 
review and verification.[Footnote 49] We found that OPM has initiated 
some related efforts, but has not fully addressed two of these issues. 
In order to decrease errors in reporting, OPM officials and EHRI 
reports show that OPM has worked with agencies to identify technical 
errors in their training data submission. However, to improve 
reporting on cost data--which is currently a challenge for agencies--
OPM held one focus group with agencies in 2007, which it used to 
updated it's guidance on tracking training data in 2008.[Footnote 50] 
OPM also has not followed through on plans to annually provide 
agencies with reports of their training data for verification and 
correction. According to OPM, the purpose of the training data report 
is to (1) inform agencies of the training data OPM had received (2) 
offer them the opportunity to work closely with OPM in correcting any 
identified deficiencies and (3) to make note of the progress they have 
made in addressing OPM's training reporting requirement. OPM officials 
said that they sent one report to agencies (in fiscal year 2010) 
summarizing their training data and requesting verification and this 
report was provided in response to expectations that the data would be 
posted on the government-wide website Data.gov.[Footnote 51] In our 
review of examples of agency responses, we found that agencies 
identified important discrepancies in their data, including 
significant underestimates of the costs spent on training, and 
reported that they would take steps to address incorrect data. 
However, OPM officials informed us that they do not have a process for 
documenting whether agencies have taken steps to correct their data. 

Although OPM has only provided one summary of EHRI data to agencies, 
agency officials that we met with stated that they could benefit from 
using this type of data summary to improve their training data. 
Further, OPM officials stated that using these summaries to improve 
EHRI data could help agencies measure the return of investment on 
their training and assist agencies' stakeholders in making more 
informed decisions on the best use of training dollars. During our 
review, OPM officials reported that they began developing a report to 
send to agencies with fiscal year 2011 training data. 

Evaluate the benefits achieved through training and development 
programs, including improvements in individual and agency performance: 

The TED group provides guidance and assistance to agencies in 
evaluating training programs through three of its guides and in 
workshops. OPM's Training Evaluation Field Guide is the primary guide 
through which OPM advises agencies on how to evaluate training. The 
guide instructs agencies to define the results they want to achieve 
from training and to align training with the agency mission and goals. 
Further, the Guide discusses useful models for evaluating training and 
describes Return on Expectations[Footnote 52] as the ultimate 
indicator to demonstrate the training value to stakeholders.[Footnote 
53] The guide also provides information on evaluation requirements 
outlined in laws and regulations, and provides practical instruction 
by identifying common challenges and solutions related to identifying 
the most cost-effective methods to train and develop employees. In 
addition to this guidance, OPM's Training Policy Handbook instructs 
agencies to evaluate all training to determine whether or not it 
provides meaningful contributions to agency results. Similarly, in its 
Guide for Collection and Management of Training Information, OPM 
highlights the importance of collecting accurate, comprehensive 
training information and making it available to decision-makers and 
others who have a vested interest in the training activities of the 
federal government. This guide discusses the two basic types of 
performance measures for measuring training and development program 
effectiveness: process indicators and outcome indicators.[Footnote 54] 

In addition to its guides, OPM has made evaluation tools available to 
agencies on its website and held workshops on training evaluation in 
order help agencies identify and share best practices on evaluating 
training. For example, OPM's website contains a Training Evaluation 
Tool that describes the levels of training evaluation and provides 
agencies with evaluation questions to be answered in each of the four 
levels and the types of information typically collected. 

As previously mentioned, some agency officials reported that it is 
difficult to conduct these reviews because their cost and time demands 
can be significant. As a result, some agencies only conduct them for 
the most critical training and others reported that they do not have a 
formal process for conducting these reviews at all. While we agree 
that it is appropriate to target costly evaluations to the most 
important training, those who do not implement this practice at all 
could benefit from using OPM's comprehensive guidance and assistance 
on training evaluations. 

Compare training investments, methods, or outcomes with those of other 
organizations to identify innovative approaches or lessons learned: 

OPM does not have a guidance document that advises agencies on how to 
compare training investment methods and outcomes with other agencies, 
but provides some support to agencies in this area through technical 
assistance. For example, the TED group uses various web-based 
mechanisms such as OPM's website, OPM LISTSERV, OPM Federal Training 
and Development web site and OPM Federal Training and Development Wiki 
to facilitate discussions between agencies on training investments. We 
observed the exchange and sharing of information among agencies 
through OPM's LISTSERV, which is used by 950 employees from various 
federal agencies to share training practices and advice. At times, 
agencies requested and shared information with each other on the most 
effective or efficient ways to implement specific training programs or 
requested models for which to compare their activities. Similarly, 
OPM's wiki page contains examples and models of training programs for 
others to use when developing their training programs. According to 
OPM officials, the TED group also provides best practice forums on 
topics when they believe agencies need additional assistance. For 
example, OPM officials reported that they have held forums with 
agencies on the Training Evaluation Field Guide to share best 
practices and tools among agencies. OPM is working on a similar forum 
for developing supervisory training. 

OPM Does Not Provide Guidance and Assistance on Three Leading Training 
Investment Practices: 

OPM does not have guidance and assistance for three leading training 
investment practices, two of which are areas in which agencies 
reported experiencing challenges. We examined the five guidance 
documents that OPM provides related to making training investment 
decisions and documentation on OPM's technical assistance and did not 
find support for the following three practices. OPM officials 
confirmed that the agency does not provide direct guidance or 
assistance in some of these areas. 

Identify the appropriate level of investment to provide for training 
and development efforts and prioritize funding so that the most 
important training needs are addressed first: 

In our review of OPM guidance and documentation on its technical 
assistance, we found that OPM provides some guidance to agencies 
regarding steps to identify training and development investment needs 
and related training strategies, but we did not find guidance on 
prioritizing these investments so that the most important training 
needs are addressed first. According to TED officials, each agency 
must assess its own needs as the primary driver for investment 
determinations. To that end, OPM officials provide guidance and tools 
for conducting training needs assessments in OPM's training policy 
guide and on OPM's website and also direct agencies to review 
benchmarks in American Society for Training & Development's State of 
the Industry reports. OPM officials also reported that they use the 
Human Capital Assessment Accountability Framework and related efforts 
to emphasize the importance of considering training as a solution to 
addressing mission critical competencies and skill gaps, but 
acknowledged that OPM does not provide specific guidance on 
prioritizing training investments through these processes.[Footnote 55] 

OPM officials identified guidance that they believe addresses the 
leading practice of prioritizing training investments; however, we 
found that the guidance does not address prioritization. As we 
previously mentioned, it is a leading practice for agencies to 
prioritize their training investments using criteria, such as expected 
demand for the training from internal sources, availability of 
resources to support the effort, potential for increased revenue, and 
risk of unfavorable consequences if investments are not made, or their 
potential to improve their efficiency. TED officials identified OPM's 
2000 Guide to Strategically Planning Training and Measuring Results as 
the source of guidance to agencies on this practice.[Footnote 56] In 
our review of this guide, we did not find any explicit guidance to 
agencies on prioritizing training investments. Instead, the guide 
advises agencies to build a business case for their training 
strategies. OPM defines a business case as a method for projecting and 
documenting the benefits to be gained as a result of investing 
resources in a training intervention. The guide encourages agencies to 
consider questions that are important to building a business case and 
provides an example of how to build a business case using this 
information (see Appendix III for diagrams from the guide on building 
a business case for training). These steps are consistent with our 
identified leading practice. However, the guide does not take the 
additional step of advising agencies on how to prioritize training 
investments selected from their business case(s) relative to each 
other. We have previously reported that, when budgets are constrained, 
training is often one of first investments that agencies reduce. 
Therefore, it is increasingly important for agencies to prioritize 
their selected training activities, so that the most important 
training is identified. Moreover, they need to communicate those 
priorities agency-wide, in order to identify common needs and 
potential areas for consolidated investments. As previously noted in 
our review, this is a practice that most CHCOs reported that they do 
not implement, which, as illustrated by our case example agencies, has 
resulted in costly duplicative and inefficient training investments at 
some agencies. 

Identify the most appropriate mix of centralized and decentralized 
approaches for its training and development programs. 

In our review of OPM guidance and documentation on its technical 
assistance, we did not identify specific guidance or assistance to 
agencies on this practice. As we previously noted, while neither 
approach fits every situation, agencies need to consciously think 
about the advantages and disadvantages of using centralized and 
decentralized approaches, particularly for the design of training and 
development programs. Although OPM officials confirmed that they do 
not provide guidance and assistance to agencies in this area, OPM 
officials agreed with this leading practice. We found that most 
agencies included in our review reported that they already implement 
the practice, so additional guidance may not be necessary. 

Have criteria for determining whether to design training and 
development programs in-house or obtain these services from a 
contractor or other external source. 

TED officials agreed that they do not provide guidance or assistance 
to agencies on this practice. TED officials stated that agencies need 
to incorporate this leading practice into their training investment 
decision-making process. As we previously noted, once an agency has 
identified its training and development needs, it should make informed 
decisions about whether to design and develop training and development 
programs in-house or buy these services from a contractor or other 
external source. Factors that they should consider include the 
capability of in-house staff to develop and implement the training; 
the prior experience, capability, and stability of possible providers 
in the marketplace; and agency limitations on cost, time, and 
resources. As previously mentioned, 12 of the 27 CHCOs reported that 
they do not implement this leading practice, and our discussion with 
DOI officials, one of the four agencies that we interviewed for this 
review, illustrates that even those agencies that reported 
implementing this practice may not be doing so for all training in the 
agency. 

Agency Officials Identified a Need for OPM to Provide More Assistance 
in Improving the Effectiveness and Efficiency of Training and 
Development Investments Government-wide: 

Agency officials reported they could use more OPM assistance in 
leveraging federal training investments across the government. Part of 
OPM's role is to identify functional areas in which new or expanded 
interagency training activity is needed and either conduct such 
training or arrange for agencies having the substantive competence to 
do so; as well as to coordinate interagency training conducted by and 
for agencies. Members of the CLO council emphasized that they could 
benefit from more OPM assistance in achieving greater interagency 
collaboration on training to reduce duplicative training investments. 
All four agencies that we interviewed reported concerns similar to the 
Council's. For example, DOI officials noted that OPM's knowledge and 
expertise could help agencies identify one basic approach to 
competency management (e.g., establishing levels of proficiency and 
competency validation processes) that can be used across government 
rather than using multiple approaches at various agencies. At the 
present, each agency individually identifies training needed for these 
competencies, which results in duplication and variation in the 
quality of training provided throughout the government. DOI officials 
stated it could be more efficient if agencies would use a standard set 
of Knowledge, Skills, and Abilities to hire and identify training and 
development investment priorities. The officials also suggested that 
OPM's HR University could be used to provide training for other 
mission-critical occupations. Further, VA officials stated that it 
would assist agencies if OPM established government-wide courses for 
mandatory training and cross-cutting areas.[Footnote 57] As an 
example, the officials stated the federal government has 17 different 
versions of No Fear Act training.[Footnote 58] The officials suggested 
that OPM could establish one government wide training for such 
subjects which would help agencies save federal time and money. 
Officials from DHS and DOE expressed similar views. 

In contrast, the shared training efforts that are being implemented by 
the Federal Healthcare Training Partnership collaborative, which 
consists of 14 federal agencies that provide clinical health care or 
related training to support their mission, illustrate the potential 
magnitude of savings that could be achieved by leveraging training 
across agencies. The Federal Healthcare Training Partnership was 
created by its members to share training programs and resources across 
the agencies to speed up the provision of employee learning and reduce 
training costs. According to VA officials--who lead the effort, the 
agencies formed this group because they saw the unaddressed need and 
opportunity to save costs in common training areas. Documentation 
provided by VA on the collaborative group states that in fiscal 2011, 
Federal Healthcare Training Partnership partner organizations shared 
more than 2,300 programs, generating a total cost avoidance of more 
than $82 million.[Footnote 59] They did so by utilizing the partner 
organizations' existing learning systems to share training that was 
originally developed for a single agency's internal use and making it 
available to all federal learners, as well as by coordinating the 
joint development or purchase of training needed by two or more 
partner agencies. VA officials stated that, while this has been a 
valuable effort to specifically improve healthcare-related training 
investments for the agencies involved, all federal agencies would 
benefit from an expansion of leveraging training investments across 
the government. 

OPM officials agreed that increased coordination of mandatory and 
common training across the government could reduce duplication and 
improve the efficiency of federal training investments. The officials 
reported that OPM has already engaged in some efforts to partner with 
or support CLO and CHCO Council efforts to share specific training 
across agencies. For example, the officials worked with the Social 
Security Administration to share a "Plain-Language" writing course 
developed by the Social Security Administration with other agencies, 
by placing it on OPM's Training and Development Wiki page. In 
addition, OPM officials stated that in 2010, the CHCO Council and OPM 
collaborated to establish HR University, which is aimed at addressing 
the competency and skill gaps within the HR community and achieve 
savings government-wide by identifying and sharing the best HR 
training with all agencies.[Footnote 60] While the system was 
initially designed to provide training to the HR community, it has 
also been used to provide some mandatory training and HR training to 
other supervisors and managers. For example, officials reported that 
they recently added a mandated Uniformed Services Employment and 
Reemployment Rights Act training to HR University. The CHCO Council 
and OPM have also developed a formula to calculate cost savings 
resulting from the shared courses, which agencies can use to track 
their savings and return on investment. According to the OPM Executive 
Director of the CHCO Council and HR University's website, in its first 
year, HR University has saved the government $14.5 million as a result 
of the shared training, and OPM officials expect that it could produce 
significantly more savings, when other courses are added.[Footnote 61] 
According to OPM officials, while HR University primarily serves the 
needs of the HR community, OPM would support using the HR University 
model to centralize training in other occupations or functional areas. 

Conclusions: 

The federal government's efforts to build and retain a workforce of 
skilled and efficient employees are essential to addressing skill gaps 
in critical fields and effectively and efficiently deliver services to 
the public. Training and development programs play a vital role in 
fulfilling these goals. However, agency leaders need to be as 
strategic about how they invest resources in this area as they are in 
other key areas of agency operations. Training investment decisions 
should be based on an assessment of the appropriate level of training 
investments and the prioritization of those investments, as well as an 
evaluation of the most cost-effective delivery mechanisms, and the 
known costs and benefits of their training investments. CHCOs and OPM 
each play a vital role in ensuring that these investment decisions are 
effectively made. 

While CHCOs report that they are implementing leading practices that 
support the successful delivery of training, they could do more to 
ensure that these investments are more cost effective. Because many 
CHCOs do not have the information that they need from component or 
subagency leaders regarding the level of training investments and 
mechanisms for setting priorities agency-wide, their agencies are 
duplicating some internal training investments and missing 
opportunities to leverage economies of scale and share the most 
effective training across their agencies. Many CHCOs are also limiting 
their opportunities to make training more cost effective and 
accessible because they are not comparing the merits of different 
training delivery mechanisms. OPM's guidance and assistance in these 
three areas are minimal or absent and could be strengthened to assist 
agencies in implementing these leading practices. In addition to these 
limitations, some CHCOs do not have a formal process to evaluate the 
impact of training on their mission. While not all training and 
development programs require, or are suitable for, higher levels of 
evaluation, those who do not implement this practice for any training 
are missing information that could help them make more effective 
investment decisions and could benefit from using OPM's existing 
guidance and assistance on conducting such evaluations. 

Federal agencies and OPM also need reliable information on how much 
agencies spend on training and for what purposes, in order to make 
effective training investment decisions. However, CHCOs do not 
completely and reliably track training costs agency-wide and, 
therefore, are unable to provide OPM with the reliable information 
that it needs to more effectively guide government-wide training 
policies. OPM has responsibility for providing regulations for 
maintenance of agency training data, assessing the completeness and 
quality of those data when agencies submit it, and using it to target 
its assistance to agencies. But OPM does not know the extent of the 
reliability of federal training investment data because they have not 
compared improvements in the completeness of the data over last 3 
years and determined if it meets the standards of sufficiency for data 
assessment and have not assessed the quality of the data in the 6 
years that agencies have been required to submit it. 

Given the fiscal challenges facing the nation, the federal government 
needs to take advantage of every opportunity to better leverage 
resources and investments across agencies. However, at present many 
agencies independently purchase or develop training for the same 
government-wide mandated courses. OPM has an opportunity to reduce 
duplicative and inefficient training investments by leveraging 
existing training resources government-wide. Agency leaders and OPM 
recognize that this has led to redundant and inefficient federal 
training investments. HR University--the one-stop-shop training 
platform administered by OPM for many courses mostly related to the HR 
community--provides a model that can result in cost savings and help 
standardize some mandatory training courses across government. 

Recommendations for Executive Action: 

To improve federal training investment decision-making processes, the 
Director of OPM should take the following five actions: 

1. Include in existing or new OPM guidance or technical assistance 
additional information in the following areas: 

* Steps agencies should take and factors they should consider when 
prioritizing federal training investments agency-wide, including 
developing a process to rank training using criteria, such as expected 
demand for the investment from internal sources, availability of 
resources to support the effort, potential for increased revenue, and 
risk of unfavorable consequences if investments are not made. 

* Steps agencies should take and factors they should consider for 
comparing the merits of different delivery mechanisms and determining 
the mix of mechanisms to use, in order to ensure efficient and cost-
effective delivery of federal training. Such guidance could include 
requesting that agencies consistently utilize Standard Form-182 to 
document and report training costs associated with the different 
delivery mechanisms employed. 

2. In line with statutory and regulatory provisions on maintenance and 
reporting of training information, work with the CHCO Council to 
improve the reliability of agency training investment information by: 

* ensuring that agencies are familiar with and follow guidance 
outlined in OPM's Guide for the Collection and Management of Training 
Information regarding which training events should be documented as 
training and reported to OPM; 

* developing policies to strengthen the utilization of Standard Form-
182 to document and report training costs; 

* encouraging agencies through guidance and technical assistance, to 
develop policies that require consistent reporting of training data to 
their learning management systems; and: 

* encouraging each agency to assess its existing training information 
system(s) and identify whether it is providing complete and reliable 
data and, if not, to develop approaches to improve the system(s), in 
order to do so. 

3. Provide regular report summaries to agencies on EHRI training 
investment data and its reliability, in order to improve the 
transparency and reliability of federal training investment data. 

4. Once federal training data reliability has been sufficiently 
improved, consistent with Executive Order No. 11348, use EHRI data to: 

a) counsel heads of agencies and other agency officials on the 
improvement of training, and: 

b) assist agencies in developing sound programs and financial plans 
for training and provide advice, information, and assistance to 
agencies on planning and budgeting training programs. 

In collaboration with the CHCO and CLO Councils, identify the best 
existing courses that fulfill government-wide training requirements, 
such as mandatory Equal Employment Opportunity training, or training 
in common federal occupations, such as basic training in financial 
management, and offer them to all agencies through HR University or 
other appropriate platform to reduce costly and duplicative federal 
training investments. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the departments of OPM, DHS, 
DOE, DOI, and VA for review and comment. OPM commented on our five 
recommendations to their agency, concurring with one recommendation, 
partially concurring on three recommendations and not concurring with 
a portion of one recommendation. OPM's official comments are reprinted 
in appendix IV. OPM, DOI, and VA provided technical comments, which we 
incorporated into our report, as appropriate. DOE and DHS had no 
comments. 

OPM partially concurred with our first recommendation that it should 
provide in existing or new guidance information on prioritizing 
federal training investments agency-wide and factors agencies should 
consider for comparing the merits of different delivery mechanisms. 
OPM stated that its publications mentioned in our report already 
provide guidance on necessary steps and specific factors agencies 
should consider when prioritizing training investments. However, none 
of the guides that we obtained or that OPM provided for our review 
contain a specific discussion about ranking training investments based 
on key factors that should be considered, such as expected demand for 
the investment from internal sources, availability of resources to 
support the effort, potential for increased revenue, risk of 
unfavorable consequences if investments are not made or the potential 
to improve efficiency. OPM stated that, as part of its effort to 
revise the Human Capital Assessment Framework resources that it 
provides to agencies, OPM plans to include tools and guidance on steps 
agencies can take to prioritize learning investments as part of its 
strategic human capital planning. We did not change our 
recommendation, which is based on OPM's current guidance and 
assistance. OPM's reported future plan to provide more specific 
guidance on prioritization has the potential to address our 
recommendation, when implemented. OPM also agreed to provide further 
guidance regarding what steps agencies should take and what factors 
they should consider in comparing the merits of different delivery 
mechanisms and determining the mix of mechanisms to use to ensure 
efficient and cost-effective delivery of federal training. 

OPM did not concur with the portion of our second recommendation 
regarding working with the CHCO council to improve the reliability of 
agency training investments by developing a common definition of what 
should be documented as training. OPM stated that the definition of 
training is clearly stated in 5 U.S.C Chapter 41 and OPM's Draft 
Training Policy Handbook and Guide for the Collection and Management 
of Training Information outlines which training events should be 
documented as training and reported to OPM. Consequently, OPM 
recommended that we delete this task for OPM. OPM's Guide for the 
Collection and Management of Training Information states that all 
courses, workshops and conferences paid for by the government; all 
federally mandated training; and, all agency required training should 
be reported to OPM's EHRI system. It further states that agencies do 
not have to report training that occurs spontaneously or casually/ 
incidentally (e.g., reading a book, having a discussion, web casts, 
briefings, etc.); training that has no specified training goals; 
training where there are no ways to evaluate if the training improved 
knowledge, skills, abilities or competencies; and, training that was 
not paid for by the government. We agree that this guidance should 
assist agencies in knowing which training to track and report, and 
therefore have removed this task from the recommendation. However, 
given the concerns raised by officials in our case example agencies 
regarding inconsistencies in whether conferences and other trainings 
are actually tracked, and recent events regarding spending at such 
training, we modified our recommendation to suggest that OPM work 
closely with CHCOs to ensure that this guidance is followed as it 
addresses the other actions we recommend to improve reliable 
reporting. OPM concurred with the other actions identified in the 
recommendation which included working with the CHCO council to: 
develop policies to strengthen the utilization of Standard Form-182 to 
document and report training costs; encourage agencies through 
guidance and technical assistance, to develop policies that require 
consistent reporting of training data to their learning management 
systems; and encourage each agency to assess its existing training 
information system(s) and identify whether it is providing complete 
and reliable data and, if not, to develop approaches to improve the 
system(s), in order to do so. 

OPM partially concurred with our third recommendation that it should 
provide regular report summaries to agencies on EHRI training 
investment data and its reliability, in order to improve the 
transparency and reliability of federal training investment data. OPM 
stated that it will provide regular summaries to agencies on the 
training investment data submitted to OPM to improve transparency. 
However, OPM stated that these summaries will not directly lead to 
improved reliability of the data because agencies must take action to 
improve the data in order to have an effect on data reliability. OPM 
also noted that agencies currently have the option of working with OPM 
to secure a subscription to Business Objects--a reporting tool that 
will allow agencies to run reports of the data they have provided to 
OPM and determine whether those data accurately reflect what is 
occurring in their agencies. OPM recommended that we revise our 
recommendation to read, "Provide regular report summaries to agencies 
on EHRI training investment data in order to improve the transparency 
of federal training investment data." We agree that agencies are 
ultimately responsible for making changes to their data to improve its 
reliability. However, OPM plays an important role in the first step of 
that process by reporting the current information that it has, so that 
agencies can make corrections. We believe that this recommendation 
along with our prior recommendation on steps OPM and CHCOs can take to 
improve reliability will contribute to improving the transparency and 
reliability of agency training data. Therefore, we did not make 
changes to this recommendation. 

OPM concurred with our fourth recommendation that it should counsel 
heads of agencies and other agency officials on the improvement of 
training; assist agencies in developing sound programs and financial 
plans for training; and provide advice, information, and assistance to 
agencies on planning and budgeting training programs using EHRI data, 
once federal training data reliability has been sufficiently improved. 
OPM stated that it will consult with agencies on possible improvements 
and assistance on planning training programs once federal training 
data are reliable. 

OPM partially concurred with our fifth recommendation that it should, 
in collaboration with the CHCO and CLO Councils, identify the best 
existing courses that fulfill government-wide training requirements, 
such as mandatory Equal Employment Opportunity training, or training 
in common federal occupations, such as basic training in financial 
management, and offer them to all agencies through HR University or 
another appropriate platform to reduce costly and duplicative federal 
training investments. OPM stated that it agrees and is already 
collaborating with the CHCO and CLO Councils to identify, collect, and 
share existing mandatory courses that fulfill government-wide training 
requirements (e.g., Plain Writing, Telework. USERRA, Veterans 
Employment, Constitution Day) through HR University or on OPM's 
Federal Training and Development Wiki. Therefore, OPM recommended that 
we revise the recommendation to recognize that the expansion of 
mandatory training by HR University would be a continuation of efforts 
they have started. We have revised the recommendation to reflect this 
comment. 

We are sending copies of this report to the Director of OPM. In 
addition, this report will be available at no charge on the GAO 
website at [hyperlink, http://www.gao.gov]. If you have any questions 
about this report, please contact me at 202-512-2717 or 
jonesy@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. GAO staff who made key contributions to this report are listed 
in appendix V. 

Sincerely yours, 

Signed by: 

Yvonne D. Jones: 
Director: 
Strategic Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

To better understand how federal training investment decisions are 
made and whether improvements are needed, you asked us to review the 
methods that agencies are using to establish their training investment 
strategies and Office of Personnel Management's (OPM) training 
investment guidance to agencies. Accordingly, this review assesses the 
extent to which (1) chief human capital officers (CHCOs) of selected 
federal agencies have established processes to set and prioritize 
training investments that are aligned with leading practices; and (2) 
OPM's guidance and assistance for developing training investment 
strategies align with these leading practices. 

For the purposes of this review, we define the key terms "training", 
"development" and "agency-wide" in the following ways: 

* Training is making available to employees planned and coordinated 
educational programs of instruction in professional, technical, or 
other fields that are or will be related to the employee's job 
responsibilities. Training can be accomplished through a variety of 
approaches, such as classroom training, e-learning, and professional 
conferences that are educational or instructional in nature.[Footnote 
62] 

* Development is generally considered to include training, structured 
on-the-job learning experiences, and education. Developmental programs 
can include experiences such as coaching, mentoring, or rotational 
assignments. 

* Agency-wide includes all components, sub-agencies or offices within 
a cabinet department or independent agency. 

For both objectives of the review, we compared OPM and CHCO practices 
against eight federal training investment leading practices, which are 
based on our prior studies, other expert studies, and statutory, 
regulatory, and executive order training requirements.[Footnote 63] 
(See table 1 at the beginning of this report). OPM reviewed these 
criteria and agreed that they are practices that agencies should be 
implementing to support effective training investment decisions. They 
also informed us that, while some leading practices are related to 
training program requirements contained in statutory, regulatory, or 
executive order provisions, responses to our questions about the 
leading practices are not an indication of whether agencies are in 
compliance with these laws and regulations.[Footnote 64] 

To obtain government-wide information on agency training investment 
practices, through a questionnaire on their training investment 
practices and processes, we obtained high-level information from 
members of the 27 agencies represented on the CHCO Council.[Footnote 
65] We provided a standard set of questions to each CHCO to ensure we 
consistently captured their responses to our questions on their 
training investment practices. We then analyzed the results of the 
questionnaire to identify the main themes and develop summary 
findings. Two of our analysts conducted this analysis, placed CHCO 
responses into categories, and tallied the number of responses in each 
category. A third analyst traced the responses back to the original 
questionnaire and verified the appropriate categorization of CHCOs' 
responses. To characterize CHCOs views throughout this report, we 
defined modifiers (e.g., "many") to quantify users' views as follows: 

* "nearly all" users represents 23 to 27 users, 

* "most" users represents 18 to 22 users, 

* "many" users represents 13 to 17 users, 

* "several" users represents 8 to 12 users, 

* "some" users represents 3 to 7 users, and: 

* "few" users represents 0 to 3 users. 

To obtain additional perspective and insights on the training 
investment practices identified in the questionnaire, we discussed the 
responses with CHCO and chief learning officers (CLO) councils. In 
addition, based on the responses to the questionnaire and workforce 
size, we selected four agencies (the Department of Homeland Security, 
Department of Veterans Affairs, Department of the Interior, and 
Department of Energy) from which to obtain illustrative examples of 
how they implemented the training investment practices identified in 
the questionnaire. (See table 6 for selection traits). As part of our 
review of agency practices, we also obtained information on the steps 
that agencies are taking to identify and prioritize investment 
allocations for training required to implement the GPRA Modernization 
Act of 2010 (GPRAMA).[Footnote 66] 

Table 6: Selection of Case Example Agencies: 

Agency selected: DHS; 
Workforce size: Greater than 100,000; 
Training oversight structure[A]: None; 
Sets a specific level of training investment: No; 
Agency-wide process for prioritizing training investments: Yes; 
Centralized or decentralized training investment processes: both; 
Agency-wide process for tracking cost and delivery of training: No. 

Agency selected: VA; 
Workforce size: Greater than 100,000; 
Training oversight structure[A]: Oversight committee and learning 
board; 
Sets a specific level of training investment: Yes; 
Agency-wide process for prioritizing training investments: Yes; 
Centralized or decentralized training investment processes: both; 
Agency-wide process for tracking cost and delivery of training: Yes. 

Agency selected: DOE; 
Workforce size: Less than 100,000; 
Training oversight structure[A]: Oversight committee and learning 
board; 
Sets a specific level of training investment: No; 
Agency-wide process for prioritizing training investments: Yes; 
Centralized or decentralized training investment processes: 
decentralized; 
Agency-wide process for tracking cost and delivery of training: Yes. 

Agency selected: DOI; 
Workforce size: Less than 100,000; 
Training oversight structure[A]: Oversight committee; 
Sets a specific level of training investment: No; 
Agency-wide process for prioritizing training investments: No; 
Centralized or decentralized training investment processes: 
decentralized; 
Agency-wide process for tracking cost and delivery of training: No. 

Source: GAO analysis of CHCO responses to a GAO questionnaire. 

[A] During our meetings with DHS, VA, DOI and DOE, officials clarified 
or changed their answers to some of the other responses regarding 
their oversight structure as well as prioritizing and tracking 
training investments agency-wide. Similarly, VA, DOE, and DOI reported 
having decentralized processes in their questionnaire. However, when 
we met with officials in the agency, they described have both some 
centralized and decentralized training investment processes. 

[End of table] 

To identify and assess OPM's oversight and guidance to agencies on 
training investment strategies, we reviewed OPM training guidance, 
relevant documentation on forums, workshop or other assistance, and 
oversight activities. In addition, we interviewed officials from OPM 
offices with primary responsibility for providing training policy 
guidance and technical assistance to agencies. We compared this 
information to the leading practices identified in table 6. We also 
identified and described the steps, that OPM has taken to identify the 
skills and training needed to implement performance management 
improvements, such those required by GPRAMA, as a foundation for 
future agency training investments. However, we did not assess the 
effectiveness of OPM's efforts to identify GPRAMA-related skills and 
actions to develop related training: 

Based on information obtained from agencies and OPM, we assessed which 
leading training investment practices were being implemented by 
agencies and addressed by OPM guidance and assistance. We also 
identified the challenges or limitations reported by agencies to 
implementing the practices, and opportunities for improvement in 
agency processes and related OPM guidance. 

We conducted this performance audit from December 2011 to September 
2012, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Table of Statutes, Regulations, and Executive Orders 
Related to Leading Practices: 

Leading Training Investment Practices[A]: 1. Agencies should (a) 
Identify the appropriate level of investment to provide for training 
and development efforts and (b) prioritize funding so that the most 
important training needs are addressed first; 
Related Statutes, Executive Orders, and OPM Regulations: 
* Exec. Order No. 11348, section 303(e) requires agency heads to 
establish priorities for needed training, and provide for the use of 
funds and man-hours in accordance with these priorities; 
* 5 C.F.R. § 410.201(c), in implementing the E.O., requires agency 
heads (or designee(s)) to establish priorities for training employees 
and allocate resources according to those priorities. 

Leading Training Investment Practices[A]: 2. Agencies should identify 
the most appropriate mix of centralized and decentralized approaches 
for its training and development programs; 
Related Statutes, Executive Orders, and OPM Regulations: There are no 
statutory, regulatory, or Executive Order requirements directly 
related to this practice. 

Leading Training Investment Practices[A]: 3. Agencies should consider 
government-wide reforms and other targeted initiatives to improve 
management and performance when planning its training and development 
programs; 
Related Statutes, Executive Orders, and OPM Regulations: There are no 
statutory, regulatory, or Executive Order requirements directly 
related to this practice. 

Leading Training Investment Practices[A]: 4. Agencies should have 
criteria for determining whether to design training and development 
programs in-house or obtain these services from a contractor or other 
external source; 
Related Statutes, Executive Orders, and OPM Regulations: There are no 
statutory, regulatory, or Executive Order requirements directly 
related to this practice. 

Leading Training Investment Practices[A]: 5. Agencies should compare 
the merits of different delivery mechanisms (such as classroom or 
computer-based training) and determine what mix of mechanisms to use 
to ensure efficient and cost-effective delivery; 
Related Statutes, Executive Orders, and OPM Regulations: There are no 
statutory, regulatory, or Executive Order requirements directly 
related to this practice. 

Leading Training Investment Practices[A]: 6. Agencies should track the 
cost and delivery of its training and development programs; 
Related Statutes, Executive Orders, and OPM Regulations: 
* 5 U.S.C. 4118 authorizes OPM to prescribe regulations--and in doing 
so--to specifically provide for; 
* the maintenance of necessary information concerning the general 
conduct of the training activities of each agency, and such other 
information as is necessary "to enable the President and Congress to 
discharge effectively their respective duties for supervision, control 
and review of these training programs"; 
* The submission of reports by agencies on results and effects of 
training programs and plans and economies resulting therefrom, 
including estimates of costs of training; 
* 5 C.F.R. § 410.601(a) requires agencies to maintain records of 
training plans, expenditures, and activities in such form and manner 
as necessary to submit to OPM. Subsection (b) provides that beginning 
December 31, 2006, agencies are to report training data at such times 
and in such form as required for OPM's government-wide Electronic Data 
Collection System. 

Leading Training Investment Practices[A]: Agencies should evaluate the 
benefits achieved through training and development programs, including 
improvements in individual and agency performance; 
Related Statutes, Executive Orders, and OPM Regulations: 
* 5 U.S.C . § 4103(c) requires the head of an agency to evaluate, on a 
regular basis, each program or plan established, operated, or 
maintained under subsection (a)[B] with respect to accomplishing 
specific performance plans and strategic goals in performing the 
agency mission and modify such program or plan as needed to accomplish 
such plans and goals; 
* Exec. Order. No. 11348, section 303(a) requires agency heads to 
review at least annually the agency's program to identify training 
needed to bring about more effective performance at the least possible 
cost; 
* 5 C.F.R . § 410.202-provides that agencies must evaluate their 
training programs annually to determine how well such plans and 
programs contribute to mission accomplishment and meet organizational 
performance goals; 
* 5 C.F.R. 410.201(d)(4), requires heads of agencies (or their 
designee(s)) to develop and maintain plans and programs that assess 
periodically, but not less often than annually, the overall agency 
talent management program to identify training needs within the agency 
as required by section 303 of E.O. 11348. 

Leading Training Investment Practices[A]: Agencies should compare 
training investments, methods, or outcomes with those of other 
organizations to identify innovative approaches or lessons learned; 
Related Statutes, Executive Orders, and OPM Regulations: There are no 
statutory, regulatory, or Executive Order requirements directly 
related to this practice. 

Source: GAO Analysis of relevant laws, regulations, and Executive 
Orders. 

[A] These key training investment practices are part of the framework 
outlined in the GAO's guide GAO, Human Capital: A Guide for Assessing 
Strategic Training and Development Efforts for the Federal Government, 
GAO-04-546G (Washington, D.C.: March 2004). This guide summarizes 
attributes of effective training and development programs and it is 
based on the GAO analysis of prior work, other related expert studies, 
and federal training requirements. 

[B] 5 USC 4103(a) requires agency head to establish, operate, 
maintain, and evaluate a program or programs, and a plan or plans 
there under, for the training of employees. 

[End of table] 

[End of section] 

Appendix III: Illustrations of Building a Business Case for Training 
from OPM's Guide to Strategically Planning Training and Measuring 
Results: 

Figure 1: OPM illustration of questions used to develop a business 
case for training: 

[Refer to PDF for image: illustration] 

Tip: Building the Business Case: 

Step 2: Develop Training Strategies To Achieve Goals: 
After selecting the projected type of benefit and indicator, you 
should answer the following questions: 

Are the projected benefits consistent with the strategic and 
performance goals and measures?  

What is the value added from closing the competency gap? 

What are the consequences if training did not occur? 

Do the potential benefits outweigh the costs?  

Source: OPM. 

[End of figure] 

Figure 2: OPM illustration of developing a business case for training: 

[Refer to PDF for image: illustration] 

Step 2: Develop Training Strategies To Achieve Goals: 
 
The following example illustrates the process of developing a business 
case for training:  

Nontraining Solutions: 
* Automation enhancements; 
* Incentives. 

Competency Gap:     
All employees involved in importer compliance activities will need to 
be able to apply the new risk management approach. This process 
differs greatly from current practices. 

Training Strategy A: Classroom Training; 
Total Annual Costs: $300,000. 
  
Training Strategy B: CD-ROM interactive Lessons; 
Total Annual Costs: $250,000. 

Potential Benefits: 
* 10% increase in collection of import fees; 
* 20% decrease in compliance resources used. 

Projected Value of Benefits: $2.5 million; 
Projected Return-on-Investment Ratio: 10:1. 

It training does not occur it is unlikely that the risk management 
model will be implemented. Effective implementation of the model is 
needed in order to realize the increased collection rate. 
   
Intangible Benefits: 

(1) The risk management model can be applied to other job activities 
performed by the potential target audience. Applying the model to 
other work areas will enhance productivity.  

(2) Morale will improve because the risk management model allows 
individual performers to make decisions.  

Source: OPM. 

[End of figure] 

Appendix IV: Comments from the Office of Personnel Management: 

United States Office of Personnel Management: 
Employee Services: 
Washington, DC 20415: 

August 30. 2012: 

Ms. Yvonne J. Jones: 
Director, Strategic issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Jones: 

Thank you for providing the U.S. Office of Personnel Management (OPM) 
the opportunity to comment on the Government Accountability Office 
(GAO) draft report, "Federal Training investments: OPM and Agencies 
Can Do More to Ensure Cost-Effective Decisions." We appreciate the 
opportunity to provide you with comments about this report. 

Response to Recommendations: 

Recommendation: Include in existing or new OPM guidance or technical 
assistance additional information in the following areas: 

* Steps agencies should take and factors they should consider when 
prioritizing federal training investments agency-wide, including 
developing a process to rank training using criteria, such as expected 
demand for the investment from internal sources, availability of 
resources to support the effort, potential for increased revenue, and 
risk of unfavorable consequence if investments are not made. 

* Steps agencies should take and factors they should consider for 
comparing the merits of different delivery mechanisms and determining 
the mix of mechanisms to use. in order to ensure efficient and cost-
effective delivery of federal training. Such guidance could include 
requesting that agencies consistently utilize Standard Form-182 to 
document and report training costs associated with the different 
delivery mechanisms employed. 

We will address recommended actions separately. 

Management Response: 

* Steps agencies should take and factors they should consider when 
prioritizing federal training investments agency-wide, including 
developing a process to rank training using criteria, such as expected 
demand for the investment from internal sources, availability of 
resources to support the effort, potential for increased revenue, and 
risk of unfavorable consequence if investments are not made. 

* Partially concur. Throughout the OPM publications mentioned in this 
draft report OPM already provides guidance on necessary steps and 
specific factors agencies should consider when prioritizing training 
investments. Specifically, through OPM's Training Evaluation
Field Guide we encourage agencies to maximize training investment and 
demonstrate value by aligning training with organizational goals and 
assessing training impact up front. We clearly outline how agencies 
should assess impact by including leadership and using the process 
outlined in the field guide. When training impact and alignment with 
organizational goals is assessed during the up-front planning phase, 
it will offer new insight on where training budgets and resources 
should he allocated. 

In addition, GAO notes that OPM's Human Capital Assessment and 
Accountability Framework (HCAAF) requires agencies to strategically 
align learning investments with agency mission priorities including 
mission critical skills gaps and other organizational goals. OPM is 
currently revising the HCAAF', a decade after its initial design, to 
incorporate lessons learned and current research and best practice in 
strategic human capital management. As part of the rollout for this 
revitalized HCAAF. OPM is refreshing the HCAAF Resource Center to 
provide Federal agencies with updated tools and technical assistance 
to support implementation of strategic human capital management. The 
new HCAAF Resource Center will include tools and guidance on steps 
agencies can take to prioritize learning investments as part of its 
strategic human capital planning. 

Management Response: 

* Steps agencies should take and factors they should consider for 
comparing the merits of different delivery mechanisms and determining 
the mix of mechanisms to use. in order to ensure efficient and cost-
effective delivery of federal training. Such guidance could include 
requesting that agencies consistently utilize Standard Form-I 82 to 
document and report training costs associated with the different 
delivery mechanisms employed. 

We concur. The task of accurately determining the merits of different 
delivery mechanisms and the most appropriate mix in order to ensure 
efficient and cost-effective delivery is accomplished within the 
framework of Instructional Design. Having qualified instructional 
designers on staff to collect data on the vital information related to 
course content, audience. and environment to determine 
possible/plausible delivery options is ideal in ensuring greater 
training effectiveness. Nevertheless, knowing that this is not the 
reality in many training offices, OPM will provide further guidance on 
the use of the most appropriate training delivery methods. 

Recommendation: In line with statutory and regulatory provisions on 
maintenance and reporting of training information, work with the CHCO 
Council to improve the reliability of agency training investment 
information by: 

* Developing a common definition of what should be documented as 
training: 

* Developing policies to strengthen the utilization of Standard Form 
182 to document and report training costs; 

* Encouraging agencies through guidance and technical assistance, to 
develop policies that require consistent reporting of training data to 
their learning management systems; and; 

* Encouraging each agency to assess its existing training information 
system(s) and identify whether it is providing complete and reliable 
data and, if not, to develop approaches to improve the system(s), in 
order to do so. 

We will address recommended actions separately. 

Management Response: 

* Developing a common definition of what should be documented as 
training; 

We do not concur. The definition of training is clearly stated in 5 
U.S.0 Chapter 41. In addition, page 36 of OPM's Draft Training Policy 
Handbook and page 32 of OPM's Guide for the Collection and Management 
of Training Information outlines which training events should be 
documented as training and reported to OPM. Recommend deletion of this 
task for OPM. 

* Developing policies to strengthen the utilization of Standard Form 
182 to document and report training costs; 

* Encouraging agencies through guidance and technical assistance, to 
develop policies that require consistent reporting of training data to 
their [earning management systems: and; 

* Encouraging each agency to assess its existing training information 
system(s) and identify whether it is providing complete and reliable 
data and, if not, to develop approaches to improve the system(s), in 
order to do so. 

We concur. OPM will work with agency Chief Human Capital Officers to 
continue to strengthen the use of the SF 182 and increase reliability 
of their training data. 

Recommendation: Provide regular report summaries to agencies on EHRI 
training investment data and its reliability, in order to improve the 
transparency and reliability of federal training investment data. 

Management Response: 

We partially concur. OPM will provide regular summaries to agencies on 
the training investment data submitted to OPM to improve transparency. 
However, these summaries will not directly lead to improved 
reliability of the data. Agencies must take action to improve the data 
to positively impact data reliability. It should be noted, agencies 
currently have the option of working with OPM to secure a subscription 
to Business Objects—a reporting tool that will allow agencies to run 
reports of the data they have provided to OPM and determine if it 
accurately reflects what is occurring in the agency. Recommend 
revision of the task to read, "Provide regular report summaries to 
agencies on EHRI training investment data in order to improve the 
transparency of federal training investment data." 

Recommendation: Once federal training data reliability has been 
sufficiently improved, consistent with Executive Order No. 1 1348, use 
EHRI data to: 

a) Counsel heads of agencies and other officials on the improvement of 
training, and; 

b) Assist agencies in developing sound programs and financial plans 
for training and provide advice, information and assistance to 
agencies on planning and budgeting training programs. 

Management Response: 

We concur. Once the data is reliable OPM will consult with agencies on 
possible improvements and assistance on planning training programs. 

Recommendation: In collaboration with the CHCO and CLO Councils. 
identify the best existing courses that fulfill government-wide 
training requirements, such as mandatory Equal Employment Opportunity 
training, or training in common federal occupations, such as basic 
training in financial management, and offer them to all agencies 
through HRIJ or other appropriate platform to reduce costly and 
duplicative federal training investments. 

Management Response: 

We partially concur. In collaboration with the CHCO and CLO Councils 
OPM is already identifying, collecting, and sharing existing mandatory 
courses that fulfill government-wide training requirements (e.g., 
Plain Writing, Telework, USERRA, Veterans Employment, Constitution 
Day). Those courses are presently available on HR University 
[hyperlink,, http://www.hru.gov] and on OPM's Federal Training and 
Development Wiki. OPM will continue to work with the CHCO and CLO 
Councils to expand the suite of mandatory courses. However, while 
OPM's Federal Training and Development Wiki is tailored to all, HR 
University is not. 

The mission of HRU is to "Cultivate a culture of continuous learning 
and excellence for Federal HR professionals by providing career 
guidance. training and resources to achieve a consistently high level 
of technical, consulting. business and professional competency. While 
HRU has been enormously successful in capitalizing on centralized HR 
and manager/supervisor training across the federal government - and 
should absolutely be viewed as a model for other functional areas, the 
Chief Human Capital Officers Council created HRU to primarily serve 
the needs of the federal HR community, and any expansion of HRU must 
in all respects remain true to this mission. It is not the mission of 
HR University to be a centralized source of training and development 
across other functional or occupational areas. Prior expansion through 
the "Manager's Corner" was in support of a corollary MR function, 
Employee Relations, and supports the Employee Performance Management 
model. GEAR, and thus falls under the HR discipline. Non-HR training 
should be centralized, but FIR1) is not the vehicle for non-HR 
training, such as financial management. 

Therefore, we recommend the task be revised to read. "In collaboration 
with the CHCO and CL0 Councils. continue to identity existing courses 
that fulfill government-wide training requirements, such as mandatory 
Equal Employment Opportunity training, or training in common federal 
occupations, such as basic training in financial management, and using 
the HRU model, or other appropriate platform, to offer them to 
agencies to reduce costly and duplicative federal training 
investments." 

Technical comments to the draft report are enclosed. Unless otherwise 
noted, the suggested revisions arc meant to provide technical accuracy. 

Sincerely, 

Signed by: 

Angela Bailey: 
Associate Director: 
Employee Services: 

Enclosure: 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Yvonne J. Jones, Director (202) 512-2717 or Jonesy@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, William Doherty (Assistant 
Director), Latesha Love, Angela Leventis, and Karin Fangman made key 
contributions to this report. Also contributing to this report were 
Benjamin Crawford, Eric Gorman, and Natalie Maddox. 

[End of section] 

Footnotes: 

[1] In the February 2011 update of our High Risk Series, we reported 
that the most significant human capital challenge to the government is 
to close current and emerging critical skills gaps. GAO, High-Risk 
Series: An Update, [hyperlink, http://www.gao.gov/products/GAO-11-278] 
(Washington, D.C.: February 2011). 

[2] This definition of training includes mandatory training that 
governs how employees conduct themselves when carrying out their 
responsibilities, such as mandatory Equal Employment Opportunity 
training or Information Technology Security training. 

[3] All of these practices are described in GAO, Human Capital: A 
Guide for Assessing Strategic Training and Development Efforts for the 
Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: March 
2004). We also used several studies from the Corporate Leadership 
Council. 

[4] Appendix II includes a table that lists the laws, regulations, and 
Executive Orders that are related to the leading practices. 

[5] We sent a questionnaire to all 27 members of the CHCO Council and 
received responses from all 27 members of the Council. For a full list 
of Council members, see the full discussion of scope and methodology 
in Appendix I. 

[6] In a separate review, we are also assessing the Department of the 
Interior's human capital program, including training requirements for 
key oil and gas oversight positions and to what extent the department 
has developed and implemented plans to address ongoing training 
challenges. 

[7] GPRAMA mandated the Director of OPM to: (1) within a year of 
enactment, identify, in consultation with the Performance Improvement 
Council, the key skills and competencies needed by federal government 
personnel for developing goals, evaluating programs, and analyzing and 
using performance information for the purpose of improving government 
efficiency and effectiveness; (2) within 2 years of enactment, 
incorporate, as appropriate, key skills and competencies into relevant 
position classifications; and (3) within 2 years of enactment, work 
with each agency to incorporate the key skills identified into 
training for relevant employees at each agency Pub.L. No. 111-352, 124 
Stat. 3866 (Jan. 4, 2011) 

[8] Exec. Order No. 11348, "Further Training of Government Employees", 
32 Fed.Reg. 6335 (April 20, 1967), as amended by Exec. Order No. 
12107, 44 Fed. Reg.1055 (Dec 28, 1978). 

[9] HR University was established in 2010, in partnership with CHCO 
Council. 

[10] 5 C.F.R. § 410.201. 

[11] Exec. Order No. 11348, §§ 302 and 303 

[12] Pub. L. No. 107-296, Title XIII, 116 Stat. 2287, 2288-89, § 1303 
(Nov. 25, 2002) at 5 U.S.C. § 1401 note. 

[13] The CHCO Council was established with 25 members; however, the 
Director of OPM may designate other members to the council. Such 
additional members may include, but are not limited to: (1) the CHCOs 
of other executive agencies and (2) members who are designated on an 
ex officio basis and who may be invited to contribute to projects, as 
particular skills and expertise are needed. The Council currently 
includes a representative for Small Agency Council (Corporation for 
National and Community Service) and the Office of the Director or 
National Intelligence (ODNI), as designated by the OPM Director. 
Although the Office of the Director or National Intelligence is not 
generally covered under title 5 of the U.S. Code, according to OPM, 
they were still designated to join the Council and follow many of the 
same training practices employed by other agencies. 

[14] These 24 agencies are subject to the Chief Financial Officers 
Act. See listing of agencies at 31 U.S.C. § 901. 

[15] The CLO Council derives its authority from its members, not their 
respective agencies. Its members, act in a collective capacity and 
agree to support and further the ends of the CLO Council's vision and 
mission. CLO Council members are the most senior CLOs or equivalent of 
the federal agencies that make up the cabinet of the President and 
those who hold similar positions in independent agencies, boards, 
commissions, and the intelligence community. 

[16] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[17] On June 5, 2012 an enhanced training requirements process was 
launched with an emphasis on closing identified skill gaps based on 
training needs assessments. According to VA officials, this process 
incorporated lessons learned, best practices, and prioritization 
criteria to align training with transformational goals and the VA 
Strategic plan. 

[18] According to the Enterprise Training Advisory Board charter, 
which was signed on April 10, 2012, the group is an enterprise 
training and development management advisory board. It serves as a 
high-level executive review and recommendation group to advise the 
Dean of VALU on business requirements and on the impact of planned 
policies and procedures for leadership and mission-critical, non-
clinical career technical training and development. The Enterprise 
Training Advisory Board shall provide advice to the Dean of VALU the 
impact of training and support communication regarding requested or 
recommended training plans and projects. However, it does not have 
authority to approve training investment decisions. 

[19] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[20] Beginning in late May 2011, a workgroup of the National Council 
on Federal Labor-Management Relations (LMR) partnered with members of 
the CHCO Council to develop a new model of employee performance 
management, referred to as GEAR. GEAR focuses on articulating a high-
performance culture, aligning employee performance engagement with 
organizational performance management, implementing accountability at 
all levels, and create a culture of engagement. OPM is piloting GEAR 
at five agencies--the Housing and Urban Development, the DOE, the 
Coast Guard, OPM and VA. 

[21] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[22] The eleven agencies that reported that they do no track training 
investments agency-wide represented 65 percent of the federal 
workforce. This figure is based on CRS calculations of the size of 
each agency's workforce, as of December 2010. 

[23] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[24] A learning management system is a software application that 
automates the administration, tracking, and reporting of training 
events. 

[25] GAO, Department of Homeland Security-Strategic Management of 
Training Important for Successful Transformation, [hyperlink, 
http://www.gao.gov/products/GAO-05-888] (Washington, D.C.: Sept. 23, 
2005). 

[26] In addition to challenges tracking training data across 
components, a recent GAO report highlighted that a DHS component, the 
Federal Emergency Management Agency (FEMA) does not track how much it 
spends within its own component on the Disaster Assistance Employee 
(DAE) training, which hinders FEMA's ability to plan for future 
training. See GAO, Disaster Assistance Workforce-FEMA Could Enhance 
Human Capital Management and Training, [hyperlink, 
http://www.gao.gov/products/GAO-12-538] (Washington, D.C.: May 25, 
2012). 

[27] Data Stewards are those individuals ultimately responsible for 
the definition, management, control, integrity or maintenance of a 
departmental or Enterprise data resource. 

[28] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[29] The Kirkpatrick model of evaluation is a popular model of 
evaluation that includes measuring (1) participant reaction to the 
training program; (2) changes in employee skills, knowledge, or 
abilities; (3) changes in employee behavior or performance; and (4) 
the impact of the training on program or organizational results, which 
may be split into a fifth level that includes a return on investment 
assessment that compares training costs to derived benefits. For 
purposes of our review, we focused on three of these areas. 

[30] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[31] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[32] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[33] Section 303(e) of Exec. Order No.11348 and 5 C.F.R. § 410.201(c) 
requires agency heads to establish priorities for training and 
allocate resources according to those priorities. 

[34] A blanket purchase agreement is a simplified acquisition method 
that government agencies use to fill anticipated repetitive needs for 
supplies or services. Essentially, blanket purchase agreements are 
like "charge accounts" set up with trusted suppliers. According to the 
General Services Administration, BPAs offer an excellent option for 
federal agencies, providing convenience, efficiency, and reduced costs. 

[35] See [hyperlink, http://www.gao.gov/products/GAO-05-888]. 

[36] See [hyperlink, http://www.gao.gov/products/GAO-04-546G] and 
Corporate Leadership Council Learning and Development: LDR a Guide to 
Selecting the Most Suitable Development Method (Arlington, VA: 
Corporate Executive Board, 2011). 

[37] 5 U.S.C. § 4103(c) requires such evaluation on a regular basis 
whereas OPM's implementing regulations specify that such evaluations 
are to be conducted annually. 5 C.F.R. 410.202. 

[38] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[39] According to documentation on VA's Human Capital Investment Plan, 
the Plan is a human resource management transformational initiative to 
develop the department's human capital into a proactive, forward 
looking, well trained and professional workforce focused on delivery 
of high quality service to Veterans and their families. The plan 
includes a focus on improving the recruitment, hiring and retention of 
VA employees; investing in people development (e.g., leadership 
training); supporting and developing the capabilities of VA's Senior 
Executive Service (SES) employees; establishing a Veteran's Employment 
initiative to support veteran's recruitment, retention and 
reintegration; and, striving to build a broad set of human resources 
capabilities to support the delivery of quality care and benefit 
services to Veterans and their families. 

[40] Department of Veterans Affairs' Human Capital Investment Plan 
(HCIP) Impact Assessment /Year One: HCIP Monetization Summary, January 
2012. 

[41] We did not independently verify the reported cost savings or 
performance improvements reported in the VALU or VHA studies. 

[42] OPM's Director made these statements in a January 3, 2012 memo to 
agency CHCOs entitled "Government Performance and Results Act 
Modernization Act of 2010 Functional Competencies". 

[43] According to OPM documents, "Highly trainable" competencies 
consist primarily of knowledge and can be improved readily through 
training. "Moderately trainable" competencies have both a skills and 
ability component. The skills can be improved thru training, but 
improvement in ability may be limited by the degree of the employees' 
natural talent. These competencies are only partially responsive to 
training. "Less Trainable" competencies are identified as "trait" 
competencies that have personal characteristics that resist change. 
These competencies make a difference on the job, but are frequently 
not responsive to improvement through training. 

[44] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[45] The Standard Form-182 is the government-wide training form used 
to request, approve, and certify completion of training courses, 
conferences, seminars, symposia and academic courses. The primary 
purpose of this form is to document and track employee training, 
especially training paid with agency funds. The Standard Form-182 also 
requires that each training record should include all of the mandatory 
data. 

[46] OPM was also required by statute (5 U.S.C. § 4118) to prescribe 
regulations providing for the maintenance of necessary information 
concerning the conduct of training activities of agencies to enable 
the President and Congress to discharge effectively their respective 
duties and responsibilities for supervision, control, and review of 
these training programs. OPM implementing regulations require agencies 
to maintain records of training activities and expenditures for 
submission to OPM. 5 C.F.R. § 410.601(a). As previously noted, OPM's 
EHRI Program's Data Warehouse is the government's premier source for 
integrated Federal workforce information. The system is intended to 
collect, integrates, and publishes data for 2.0 million Executive 
Branch employees on a bi-weekly basis, supporting agency and 
government-wide analytics in various areas, including agency training 
data. 

[47] OPM has an EHRI technical office that collects the data from 
agencies and informs TED officials about which agencies are providing 
data each month and whether there are apparent problems with the data 
(e.g. missing or miscoded data). OPM requires agencies to provide a 
single source of data that represents training throughout their 
organization. 

[48] Data validity refers to whether the data actually represent what 
we think is being measured. Data reliability refers to the accuracy 
and completeness of computer-processed data, given the uses they are 
intended for. Computer-processed data may be data (1) entered into a 
computer system or (2) resulting from computer processing. GAO, 
Assessing the Reliability of Computer-Processed Data, [hyperlink, 
http://www.gao.gov/products/GAO-09-365G] (Washington, D.C.: February 
2009). 

[49] According to the 2009 analysis, the agency reports were intended 
to include data on the percent of managers receiving training, average 
cost spent per employee training, and percent of salary spent on 
training per agency, among other things. 

[50] OPM officials reported that they used this focus group to revise 
OPM's 2008 Guide for Collection and Management of Training Information. 

[51] Accessed August 7, 2012. [hyperlink, http://www.data.gov/]. 

[52] OPM's 2011Training Evaluation Field Guide: Demonstrating the 
Value of Training at Every Level defines return on expectations as the 
ultimate indicator of value. It is the conversion of stakeholders' 
expectations into observable, measurable outcomes of success for the 
training. 

[53] The guide discussed the Kirkpatrick Four-Level model (reaction, 
learning, performance and impact of training), the updated New World 
Kirkpatrick Four Levels and the Kirkpatrick Business Partnership Model 
as useful models for evaluating training. 

[54] Process indicators measure performance or key processes. They 
track such things as number of employees trained, number of training 
courses completed, or number of hours employees spent in training. 
Outcome indicators contain measures of service or performance results. 
These measure such things as an increase in employee and/or customer 
satisfaction, increase in employee job performance, and/or increase in 
organizational performance. Outcome indicators may require prolonged 
periods of measurement and multiple types and levels of measurements 
to include surveys, training impact measures, and return on investment 
analysis. 

[55] Officials from OPM's Agency Program Review office oversee OPM's 
Human Capital Assessment Accountability Framework program. According 
to OPM documents, the Human Capital Assessment Accountability 
Framework consists of five human capital systems that together provide 
a consistent, comprehensive representation of human capital management 
for the federal government: strategic alignment, leadership and 
knowledge management, results-oriented performance culture, talent 
management, and accountability. 

[56] The guide recommends that agencies identify their agency-wide 
competency gaps, assess whether training will address the gaps, 
identify and weigh the cost and benefits of training, and then build a 
business case to support their selected training strategy. 

[57] We are conducting a review of agency actions to address critical 
skill gaps, which include training and also have work underway 
assessing training in cross-cutting occupations, such as grant 
management and acquisition. 

[58] Under the Notification and Federal Employee Antidiscrimination 
and Retaliation Act of 2002 (No Fear Act) and OPM's implementing 
regulations, an agency must provide training to its employees 
regarding the rights and remedies available under the employment 
discrimination and whistleblower protection laws at least every two 
years. See, section 202(c) of Pub. L. No. 107-174, 116 Stat. 566, 569 
(May 15, 2002) and 5 C.F.R. § 724.203. 

[59] We did not verify the costs savings identified in the FHTP's 
report. 

[60] HR University was initially funded by the CHCO Council and 
operated in coordination with OPM. However, OPM officials reported 
that they are currently working on a MOU with the CHCO Council to 
transfer more ownership and operation of HR University to OPM, with 
the CHCO Council as an advisor. 

[61] According to the Executive Director of the CHCO Council, they 
developed this formula in consultation with the CLO Council. We did 
not independently assess the formula or verify the reported savings. 

[62] This definition of training includes mandatory training that 
governs how employees conduct themselves when carrying out their 
responsibilities, such as mandatory equal employment opportunity 
training or information technology security training. 

[63] All of these practices are described in [hyperlink, 
http://www.gao.gov/products/GAO-04-546G]. We also used several studies 
from the Corporate Leadership Council. 

[64] Appendix II includes a table that lists the laws, regulations, 
and executive order provisions that are related to the leading 
practices. 

[65] The CHCO Council was established with 25 members. However, the 
Director of OPM may designate other members of the Council. Such 
additional members may include, but are not limited to: (1) the CHCOs 
of other executive agencies and (2) members who are designated on an 
ex officio basis and who may be invited to contribute to projects, as 
particular skills and expertise are needed. The council agencies 
include the Departments of Agriculture, Commerce, Defense, Education, 
Energy, Health and Human Services, Homeland Security, Housing and 
Urban Development , Interior, Justice, Labor, State, Treasury, 
Transportation, and Veterans Affairs; Environmental Protection Agency: 
General Services Administration: National Aeronautics and Space 
Administration: National Science Foundation: Nuclear Regulatory 
Commission: Office of Management and Budget: OPM: Office of the 
Director of National Intelligence: Small Agency Council: Small 
Business Administration: Social Security Administration: and U.S. 
Agency for International Development. CHCOs from all 27 agencies 
responded to our questionnaire. 

[66] GPRAMA mandated the Director of OPM to: (1) within 1 year of 
enactment, identify, in consultation with the Performance Improvement 
Council, the key skills and competencies needed by federal government 
personnel for developing goals, evaluating programs, and analyzing and 
using performance information for the purpose of improving government 
efficiency and effectiveness; (2) within 2 years of enactment, 
incorporate, as appropriate, key skills and competencies into relevant 
position classifications; and (3) within 2 years of enactment, work 
with each agency to incorporate the key skills identified into 
training for relevant employees at each agency. Pub.L. No. 111-352, 
124 Stat. 3866 (Jan. 4, 2011). 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

[End of document] 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.