This is the accessible text file for GAO report number GAO-14-132 
entitled "Human Capital: Agencies Should More Fully Evaluate the Costs 
and Benefits of Executive Training" which was released on March 4, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to the Ranking Member, Committee on the Judiciary, U.S. Senate: 

January 2014: 

Human Capital: 

Agencies Should More Fully Evaluate the Costs and Benefits of 
Executive Training: 

GAO-14-132: 

GAO Highlights: 

Highlights of GAO-14-132, a report to the Ranking Member, Committee on 
the Judiciary, U.S. Senate. 

Why GAO Did This Study: 

The federal government annually spends millions of dollars on 
executive training programs administered by external training 
providers. GAO was asked to review the costs and value of this 
training. This review (1) describes what is known about how much CHCO 
Council agencies have spent on executive training offered by external 
providers and assesses the reliability of available cost data; (2) 
assesses how CHCO Council agencies evaluate the effectiveness of the 
training; (3) compares executive training efficiency opportunities 
identified by agency CHCOs to leading practices; and (4) provides 
views of career SES and SES candidates from selected agencies on the 
value of the training they received. GAO obtained information from 
CHCO Council agencies through a questionnaire, and selected three of 
26 CHCO Council agencies—-the Departments of Energy, Treasury, and 
Veterans Affairs—-to provide illustrative examples. GAO based its 
selection, in part, on workforce size, number of career SES, and total 
training costs. 

What GAO Found: 

Chief Human Capital Officers (CHCO) from 26 agencies reported that, 
from fiscal years 2008 through 2012, they spent almost $57 million (in 
constant 2012 dollars) on executive training provided by external 
providers. CHCOs reported using the Office of Personnel Management 
(OPM) Federal Executive Institute and the Harvard Kennedy School of 
Government most often. Agencies are required to maintain records of 
training plans, expenditures, activities, and (since 2006) to report 
training data to OPM. However, half of the CHCOs reported data to GAO 
that they deemed incomplete, or with limitations. For example, two 
agencies did not include travel related costs; one did not include 
costs for course materials; another did not include costs from all 
components. OPM officials agree that training cost data reported by 
agencies continues to be unreliable, and is probably lower than actual 
agency expenditures. OPM officials said they are meeting with agencies 
to address data deficiencies. However, OPM has not set interim 
milestones for meeting with agencies or established a timeframe to 
improve reporting. One leading practice is to establish such interim 
milestones and timeframes, in order to demonstrate progress towards 
achieving goals. By not establishing interim milestones and timeframes 
for improving the reliability of executive training cost data, OPM may 
be missing an opportunity to better position itself to hold agencies 
accountable for improving their data. 

Most CHCOs reported evaluating participant reaction and changing their 
training programs based on participant input, but fewer reported 
assessing training impact on agency mission. Agencies are required by 
statute and OPM implementing regulations to evaluate how well training 
programs contribute to mission accomplishment and performance goals. 
OPM is not sharing lessons learned from agencies that have experience 
assessing executive training impact on agency mission. However, OPM 
acknowledged that some agencies struggle with these evaluations; in 
response, OPM has issued guidance on the subject. CHCOs cited time, 
costs, and difficulty as reasons for not conducting the required 
evaluations and reported the need for additional OPM assistance. Eight 
agencies reported conducting these evaluations and may have lessons 
learned from which other agencies could benefit. For example, VA has a 
process for assessing the impact of executive training on its mission 
that it has used to make better training investment decisions.
CHCOs identified opportunities for agencies and OPM to achieve 
efficiencies in executive training. CHCOs said agencies could (1) 
increase interagency cooperation by sharing training facilities and 
expanding eligibility to Senior Executive Service (SES) candidates 
from other agencies, and (2) implement or expand computer-based 
training. CHCOs also said OPM could, among other things, (1) help 
centralize training offerings by creating a centrally funded SES 
candidate development program, and (2) assist agencies in identifying 
programs open to other agencies and departments. By not assessing 
lessons learned and other efficiencies identified by agencies, OPM may 
be missing an opportunity to better position agencies to achieve these 
efficiencies in executive training. 

Career SES and SES candidates generally said external executive 
training is useful and valuable, but suggested that it would be more 
cost-effective for the government to negotiate prices as a large-scale 
buyer, versus individual agencies purchasing training. 

What GAO Recommends: 

GAO recommends that OPM (1) establish interim milestones for meeting 
with agencies to address training data deficiencies and establish time 
frames for improving the reliability of agency data, (2) share 
information and examples of how agencies have evaluated the impact of 
executive training on agency mission and goals, and (3) assess 
potential efficiencies identified by agencies for possible government-
wide implementation. OPM concurred with the recommendations and has 
taken useful steps. GAO still believes that more effective activities 
can be taken. 

View [hyperlink, http://www.gao.gov/products/GAO-14-132]. For more 
information, contact Yvonne D. Jones at (202) 512-2717 or 
Jonesy@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Agencies Lack Complete and Reliable Data on the Cost of External 
Executive Training and Are Likely Underreporting Costs: 

Most Agencies Reported Evaluating Executive Training but Few Reported 
Assessing Its Impact on Agency Missions: 

CHCOs Identified Opportunities for Agencies and OPM to Achieve 
Efficiencies in Executive Training: 

Career SES and SES Candidates Said External Training is Valuable, but 
Training Programs Could be Improved: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Office of Personnel Management: 

Appendix III: GAO Contacts and Staff Acknowledgments: 

Tables: 

Table 1: OPM's Guidance Documents and Description: 

Table 2: External Providers Agencies Reported Using Most Often During 
Fiscal Years 2008 through 2012: 

Table 3: Illustrative Examples of Discrepancies in Selected Agencies' 
External Executive Training Costs (Fiscal Years 2008-12): 

Abbreviations: 

CDP: candidate development program: 

CHCO: chief human capital officer: 

CLO: chief learning officer: 

ECQ: executive core qualifications: 

EHRI: Enterprise Human Resources Integration: 

FEI: Federal Executive Institute: 

Energy: Department of Energy: 

ODNI: Office of the Director of National Intelligence: 

OPM: Office of Personnel Management: 

QRB: Qualifications Review Board: 

SES: Senior Executive Service: 

SESCDP: Senior Executive Service Candidate Development Program: 

TEI: Treasury Executive Institute: 

Treasury: Department of the Treasury: 

VA: Department of Veterans Affairs: 

[End of section] 

United States Government Accountability Office: 
GAO:
441 G St. N.W. 
Washington, DC 20548: 

January 31, 2014: 

The Honorable Charles E. Grassley: 
Ranking Member: 
Committee on the Judiciary: 
United States Senate: 

Dear Senator Grassley: 

Senior leaders of the federal workforce are responsible for directing 
a broad range of essential government activities, such as acquiring 
weapon systems to safeguard the country or providing accurate and 
timely benefits to the elderly. Effective leadership, especially at 
senior levels, is essential to providing accountable, committed, 
consistent, and sustained attention to these and other organizational 
issues facing the government. The Senior Executive Service (SES), the 
government's most senior-level career leaders, is relatively small--
about 7,000 members in 2012--and represents less than one percent of 
the over two million federal civilian employees. Underscoring the 
importance of high-value executive training, the Partnership for 
Public Service recently reported on weaknesses in talent development 
for the SES pipeline, and the need to ensure effective leadership in 
these positions.[Footnote 1] The federal government annually spends 
millions of dollars on executive training programs administered by 
training providers external to federal agencies, in order to provide 
its senior leaders with the necessary skills to fulfill their 
agencies' missions. These programs can range widely in both duration 
and costs, from a four-hour session at a cost of $200 per federal 
employee to a four-week program costing over $20,000 or more per 
federal employee. The cost of these training sessions has raised 
questions about the value they provide to the federal workforce. 

The Office of Personnel Management (OPM) and agency chief human 
capital officers (CHCOs) play important roles ensuring that federal 
training dollars are invested wisely. OPM provides leadership and 
guidance on the establishment and operation of efficient federal 
training and development programs government-wide; it also advises the 
President on the means for furthering federal training programs. The 
Chief Human Capital Officers Act of 2002 (CHCO Act) gave CHCOs the 
primary role in setting the workforce development strategies of their 
agencies, and in advising and assisting their agency heads (who are 
responsible for establishing priorities for needed training and 
development). The CHCO Act also established a Chief Human Capital 
Officers Council (CHCO Council) to advise and coordinate the 
activities of members' agencies on such matters as the modernization 
of human resources systems, improved quality of human resources 
information, and legislation affecting human resources operations and 
organizations. The decisions that these key players make in 
determining how to best use federal training dollars for executive 
education will ultimately have an important impact on how well the 
federal government is equipped to address its current and emerging 
performance and management challenges in an era of constrained 
resources. 

You asked us to review the costs of federal executive training 
programs provided by external providers and the value they bring to 
the federal workforce.[Footnote 2] This review (1) describes what is 
known about how much CHCO Council agencies have spent on executive 
training offered by external providers and assesses the reliability of 
available cost data; (2) assesses how CHCO Council agencies evaluate 
the effectiveness of the training; (3) compares executive training 
efficiency opportunities identified by agency CHCOs to leading 
training practices; and (4) provides views of career SES and SES 
candidates from selected agencies on the value of the training they 
received. 

To answer the first two questions, we developed and administered a set 
of standard questions via a structured questionnaire to the 27 
agencies represented on the CHCO Council to collect government-wide 
information on executive training programs, and if available, data on 
the amount agencies spent on executive training courses administered 
by external providers from fiscal years 2008 to 2012.[Footnote 3] We 
obtained agency executive training cost data from 26 of the 27 CHCO 
Council agencies and compared these data to training cost data from 
OPM's Enterprise Human Resources Integration (EHRI) training database. 
We assessed the reliability of related cost elements in OPM's training 
database based on reviews of our prior work on federal training 
investments,[Footnote 4] testing of agency data collected by OPM, and 
interviews conducted with OPM officials. We also assessed the 
reliability of executive training participant and cost data provided 
by agencies by examining the data for obvious errors and 
inconsistencies, and by querying 10 agencies to better understand the 
data systems each agency used to collect and report such data, as well 
as the quality of data entered into their system(s). We interviewed 
five external providers used by CHCO Council agencies to collect 
information on executive training offered by these providers. To 
answer the second question, we selected three agencies--Department of 
Energy (Energy), Department of the Treasury (Treasury), and Department 
of Veterans Affairs (VA)--to obtain illustrative examples of how they 
approached, implemented and evaluated the effectiveness of executive 
training. We selected these agencies based on workforce size, number 
of career SES, total training costs, types of evaluations conducted to 
measure effectiveness of executive training, and whether agencies 
identified potential opportunities for achieving efficiencies in 
executive training. 

To answer the third question, we analyzed responses to our 
questionnaire from agencies, analyzed OPM and agency executive 
training policies, procedures, and practices, and interviewed agency 
human capital officials to identify potential efficiencies. We 
examined how, if at all, OPM, the CHCO Council, and the Chief Learning 
Officers (CLO) Council (made up of chief learning officers from the 27 
agencies) share information on potential efficiencies by interviewing 
OPM officials and members of the CHCO Council and the CLO Council. To 
answer the fourth question, we conducted six focus groups consisting 
of career SES and SES candidates from our three selected agencies who 
had taken executive training to obtain their views on the usefulness 
and value of executive training. We then analyzed and summarized 
responses from focus group participants. 

We conducted this performance audit from December 2012 to December 
2013, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. Further 
details of our objectives, scope, and methodology are provided in 
appendix I. 

Background: 

The federal government has established a policy to develop employees, 
including its SES, through training and development programs in order 
to improve public service, increase efficiency and economy, and build 
and retain a force of skilled and efficient employees, among other 
things. Under 5 U.S.C. § 3396, OPM is responsible for establishing 
training and development programs for SES and SES candidates, or for 
requiring agencies to establish such programs that meet OPM criteria. 
If agencies establish these training programs, 5 U.S.C. § 3396(b) 
states that OPM is required to assist agencies in the establishment of 
such programs and monitor their implementation. In accordance with 
this, OPM has issued regulations on training and management 
development. For example, in 2006 OPM issued a regulation requiring 
agencies to report training expenditures to OPM through the government-
wide electronic reporting system.[Footnote 5] Executive Order No. 
11348 (issued in 1967 and later amended by Executive Order No. 12107 
in 1978) further requires OPM to assist agencies in developing sound 
programs and financial plans for training and to provide advice, 
information, and assistance to agencies on planning, programming, 
budgeting, operating, and evaluating training programs. OPM provides 
federal agencies with assistance through two primary mechanisms: 
guidance documents and technical assistance (see table 1 below). 

Table 1: OPM's Guidance Documents and Description: 

Guidance document: Executive Development Best Practices Guide (November 
2012); 
Description: A compilation of best practices used by the public and 
private sector to support the continued learning of executives. OPM 
developed this guide to help agencies meet the development needs of 
federal executives. 

Guidance document: Guide to Human Resources Reporting (March 2012); 
Description: Requirements for agencies to provide updates of their 
training information, including training costs, to OPM's Enterprise 
Human Resources Integration (EHRI) data warehouse. 

Guidance document: Draft Training Policy Handbook (December 2011); 
Description: Guidance on implementing training activities required by 
laws and regulations on training and development in the federal 
government. 

Guidance document: Training Evaluation Field Guide (January 2011); 
Description: Information on methodologies for conducting training 
evaluations with the goal of identifying cost effective training 
initiatives that maximize mission accomplishments. 

Guidance document: Hit the Ground Running: Establishing A Model 
Executive Onboarding Program (October 2011); 
Description: A "roadmap" for a one-year program developed to help new 
SES members acclimate to the roles and responsibilities associated 
with the SES corps. 

Guidance document: Guide to Senior Executive Service Qualifications 
(June 2010); 
Description: Information on the Executive Core Qualifications (ECQ) 
and how agency officials can ensure that their executive training 
programs align with these ECQs. 

Guidance document: Guide for Collection and Management of Training 
Information (August 2008); 
Description: Guidance to assist agencies in complying with the legal 
and regulatory instructions for collecting and managing federal 
training information. 

Source: GAO analysis of OPM guidance documents. 

[End of table] 

Executive training courses are one part of a foundation of executive 
education programs that include coaching and rotational assignments to 
help executives understand leadership roles and responsibilities in 
their respective agencies. OPM reviews and certifies agencies' SES 
candidate development programs (CDPs), which are designed to train 
potential SES members. OPM also convenes Qualifications Review Boards 
(QRBs), staffed by SES members across government, to provide an 
independent peer review of candidates proposed for initial career 
appointment to the SES. 

Agencies Lack Complete and Reliable Data on the Cost of External 
Executive Training and Are Likely Underreporting Costs: 

Agencies Reported Spending Millions of Dollars and Most Used Multiple 
External Providers from Fiscal Years 2008 through 2012: 

The 26 CHCO Council agencies responding to our questionnaire reported 
that, from fiscal years 2008 through 2012, they spent almost $57 
million (in constant 2012 dollars) for about 30,000 participants to 
attend executive training courses offered by external providers. On 
average, the agencies collectively reported spending between $9 and 
$15 million dollars a year (in constant dollars) on external executive 
training over the 5-year period. 

The 26 CHCO agencies reported using at least 410 external providers 
from fiscal years 2008 to 2012 for executive training. Among the 
external providers most often used, CHCO agencies reported using OPM's 
Federal Executive Institute (FEI) more than other external providers 
(see table 2). 

Table 2: External Providers Agencies Reported Using Most Often During 
Fiscal Years 2008 through 2012: 

External provider: Office of Personnel Management (OPM) - Federal 
Executive Institute (FEI); 
Number of agencies reporting using external provider, fiscal years 
2008-12: 22. 

External provider: Harvard University - Kennedy School of Government; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 17. 

External provider: OPM - courses not administered by FEI[A]; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 17. 

External provider: Brookings Institution; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 15. 

External provider: Center for Creative Leadership; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 11. 

External provider: Graduate School USA; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 10. 

External provider: American Management Association; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 8. 

External provider: Leading EDGE - Department of Veterans Affairs; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 7. 

External provider: Management Concepts, Inc.; 
Number of agencies reporting using external provider, fiscal years 
2008-12: 7. 

Source: GAO analysis of agency data. 

[A] Includes courses offered by OPM's Management Development Centers. 

[End of table] 

OPM requires agencies to maintain records of training plans, 
expenditures, and activities, and to submit the recorded data to OPM. 
Since 2006, OPM has required agencies to report training data to OPM 
monthly via OPM's EHRI system. OPM's Guide to Human Resources 
Reporting states that these data should include training materials 
cost, per diem cost, travel cost, and tuition and fees. Further, 
federal financial accounting standards state that reliable information 
on the costs of federal programs or activities is crucial for 
effective management of government operations; federal financial 
accounting standards also state that Congress needs cost information 
to evaluate program performance, to make program authorization 
decisions, and to compare alternative courses of action.[Footnote 6] 

External providers employed different approaches for executive 
training, which had different cost implications. Based on external 
providers' views, the cost of individual executive education training 
courses can vary greatly among external providers due to a variety of 
factors, including location of training, length of training, and 
whether courses are tailored to meet the mission and goals of a 
particular agency. For example, VA's Leading EDGE (Executives Driving 
Government Excellence) program offers half-day executive training 
workshops to all agencies where employees can participate either in 
person or online. The workshops can serve from about 300 to over 700 
participants. According to Leading EDGE's executive director, the 
annual cost for Leading EDGE is about $850 per participant without 
coaching and $2,100 per participant with coaching. Other external 
providers we contacted host residential training courses or programs 
which are off-site from participant's work locations and last over a 
period of several days or weeks, including the following: 

* Leadership for a Democratic Society: FEI hosts this 22-day course 
over 28 days (includes weekends), which addresses (among other 
subjects) constitutional principles, personal leadership, 
organizational theory, and the policy framework in which governmental 
leadership occurs. In FY 2012, attending this course cost about 
$20,000.[Footnote 7] 

* Senior Executive Fellows Program: the Harvard Kennedy School of 
Government hosts this 22-day program over 27 days (includes weekends) 
and addresses (among other subjects) political management, effective 
implementation, and performance management. In FY 2012, attending this 
course cost about $18,000.[Footnote 8] 

* Leadership at the Peak: The Center for Creative Leadership offers 
this 5-day course which focuses on key leadership competencies such as 
communication, influence, validating and articulating a vision, and 
leading outwardly. The course incorporates a five-month learning 
experience including five days of classroom training, and other 
activities such as assessments and executive coaching sessions. In FY 
2012, attending this course cost about $9,000.[Footnote 9] 

* Executive Leadership for America: The Brookings Institution conducts 
this 6-day course which addresses (among other subjects) the 
historical framework for constitutional government and how to create 
an organizational environment that is responsive to change. In FY 
2012, attending this course cost about $6,000.[Footnote 10] 

Agencies Experience Challenges in Collecting and Reporting Data: 

Agencies track some--but not all--costs of executive training 
administered by external providers. All 26 of the agency CHCOs that 
responded to our questionnaire provided data on executive training 
administered by external providers; however, half of the agencies 
reported that the data they provided were incomplete, and in some 
cases, had limitations. In addition, during our review, agency 
officials from 4 of the 26 agencies asked us to clarify the cost data 
we requested, noting that training cost data are not always readily 
available within their agencies. They explained that in order to 
respond to our data request, in some cases they would have to manually 
retrieve paper documents and tabulate the costs. Moreover, the 
agencies provided data that included different cost components. For 
example, one agency reported that it did not include costs for course 
materials. Two other agencies reported that they did not include costs 
for travel, lodging, or per diem. Other CHCO agencies cited additional 
problems with tracking and summarizing costs paid to external 
providers of executive education. For example, one agency reported 
experiencing challenges in tracking training costs agency-wide. 
According to agency officials, they were unable to completely (or 100 
percent) accurately track training costs across the department because 
of the multiple, incompatible learning management systems that the 
department uses to track training throughout the agency.[Footnote 11] 
In addition, they stated that the agency's practices for tracking 
training data are inconsistent within the agency because agency 
components do not always report training completed with external 
providers to the agency's shared training database system. Another 
agency said its agency-wide training data were incomplete and 
unreliable because its internal training and financial tracking 
systems were incompatible with each other. In light of all these 
factors, agencies are likely underreporting the amount they spent on 
executive training. 

In our interviews with Energy, Treasury, and VA, the CHCOs (or their 
representatives) reported that their agencies track and report 
training cost data to OPM in various formats, some automated and some 
manual. However, the three agencies reported facing challenges 
tracking cost data. Energy reported that executive training reporting 
is not consistent across the agency because many Energy components 
have direct funding streams from Congress and schedule their own 
training programs; thus, they do not always report training completed 
with external providers to Energy's Office of Human Capital. Moreover, 
because Energy's Office of Human Capital has no direct access to 
component organization records, the CHCO cannot ensure that all of 
Energy's funded and administered training programs are tracked and 
reported to EHRI and OPM; in addition, Energy's CHCO cannot identify 
whether executive training is reported to OPM by a component 
organization in a separate report or with other line item reporting of 
completed training. However, the cost and delivery of Energy's SES 
candidate development program and its executive training provided 
centrally by Energy's Office of Human Capital is normally tracked and 
reported on an annual basis although Energy has not had any recent 
centrally funded executive learning activities. Treasury and VA also 
reported difficulties in collecting and tracking reliable executive 
training data. Treasury reported that some of its individual bureaus 
were challenged by our request to provide executive training cost data 
because the costs had to be identified by a manual review of financial 
transaction listings. They stated that part of the reason the costs 
were difficult to pull together is that the training costs were not 
reported via OPM's Standard Form 182--which is used to collect and 
document training information for inclusion in participants' employee 
records and the EHRI database. Treasury officials explained that much 
of its reported executive training was done by contract and was not 
procured by either the individual or the individual offices that would 
normally complete the training form. As a result, Treasury enlisted 
multiple sources to pull the data together at each of the agency's 
bureaus. Treasury officials reported that the agency is working 
internally to determine how to track this information at the 
department level. VA reported two issues with its electronic reporting 
of data to EHRI: inconsistent recording of overall training cost data 
in its training management system, and incompatibility between VA's 
learning management system and EHRI. 

We found discrepancies between the data for external executive 
training costs that agencies submitted to us and the data from OPM's 
EHRI. In most instances where we identified discrepancies, the cost 
information that agencies provided to us was higher than the cost 
total from OPM's EHRI. Table 3 provides illustrative examples of 
discrepancies in cost information. 

Table 3: Illustrative Examples of Discrepancies in Selected Agencies' 
External Executive Training Costs (Fiscal Years 2008-12): 

Agency: Department of Energy; 
Executive external training costs reported to GAO by agency FYs 2008-
12[A]: $6,382,000; 
Executive external training costs from OPM's EHRI FYs 2008-12[A]: 
$331,000. 

Agency: Department of the Treasury; 
Executive external training costs reported to GAO by agency FYs 2008-
12[A]: $8,179,000b; 
Executive external training costs from OPM's EHRI FYs 2008-12[A]: 
$74,000. 

Agency: Department of Veterans Affairs; 
Executive external training costs reported to GAO by agency FYs 2008-
12[A]: $3,693,000; 
Executive external training costs from OPM's EHRI FYs 2008-12[A]: [C]. 

Source: Agency data obtained from GAO-administered questionnaire. OPM 
data obtained from OPM's Enterprise Human Resources Integration (EHRI) 
database. 

Note: FY = fiscal year. 

[A] In fiscal year 2012 constant dollars. 

[B] According to Treasury, these costs include all contract 
expenditures by Treasury's executive training facility, which also 
provides executive training to nine agencies and bureaus outside of 
Treasury. 

[C] OPM's EHRI database did not include any executive training data 
from VA for fiscal years 2008-12. 

[End of table] 

OPM officials agreed that agencies have challenges collecting and 
reporting costs for executive training through the EHRI database, and 
as a result, executive training cost data in EHRI is probably lower 
than actual agency expenditures. OPM officials explained that although 
agencies have taken steps recently to improve their reporting of 
training data, some agencies continue to struggle with the reporting 
requirements, in part because these agencies' learning management 
systems are not compatible with OPM's EHRI. For these agencies, 
reports on training costs require manual uploading because some 
agencies cannot sufficiently aggregate training information at the sub-
agency level. In September 2012, we reported on the weaknesses of 
training cost data that agencies report to OPM and found that several 
agencies were unable to collect reliable training cost data from 
components, due to multiple incompatible learning management systems 
and to inconsistent practices in tracking training cost data, such as 
the use of inconsistent coding schemes for tracking similar training 
activities. We recommended, among other things, that OPM (1) work with 
the CHCO Council to improve the reliability of agency training 
investment information, and (2) provide regular report summaries to 
agencies on EHRI training investment data and reliability. OPM agreed 
to work with agency CHCOs to continue to strengthen the use of the 
Standard Form 182 and to increase reliability of their training data; 
further, OPM stated that it would provide regular summaries to 
agencies on the training investment data submitted to OPM in order to 
improve transparency. However, OPM stated that these summaries will 
not directly lead to improved reliability of the data and explained 
that agencies must take action to improve the data to positively 
impact data reliability.[Footnote 12] 

OPM officials told us that they provide assistance and guidance for 
federal training efforts but because current training data is 
unreliable, OPM officials said they have not used it to inform their 
training guidance and assistance to agencies. OPM officials stated 
that they do not believe they have a viable way to enforce reporting 
requirements for training data. However, OPM has avenues, such as 
interagency councils, that can help them assist agencies with 
fulfilling their reporting obligations. For example, during 2013, OPM 
officials reported that they, in collaboration with the Chief Learning 
Officers Council, created an interagency group to develop quality 
standards and desired reporting data elements to help agencies collect 
the right information and to make informed training decisions, but as 
of September 2012, the interagency group has not established any 
timeframes for completing this effort. OPM reported that as of October 
2013, the working group is moving forward on a project to develop 
guidance on prioritizing and ranking federal training investments that 
will support a broader effort to develop a single standardized process 
for prioritizing training investments. In addition, OPM officials told 
us they have been meeting separately with CHCO Council agencies to 
discuss each agency's training data reported to OPM and any 
deficiencies noted by OPM. OPM believes this will help rectify 
situations where agencies are unable to report training data--among 
other types of data--accurately, completely, and in a timely fashion. 
OPM officials said that they also plan to meet with smaller agencies 
for these same purposes. 

However, OPM has not established interim milestones for the completion 
of these planned meetings with agencies to address training data 
deficiencies, nor has OPM set timeframes for agencies to have a plan 
in place for improving the completeness and reliability of agency data 
on executive training costs. One leading transformation practice is to 
establish milestones and timeframes to demonstrate progress towards 
achieving goals. Without interim milestones and timeframes for these 
efforts, OPM may be missing an opportunity to better position itself 
to hold agencies accountable for improving the data. 

Most Agencies Reported Evaluating Executive Training but Few Reported 
Assessing Its Impact on Agency Missions: 

Twenty-two of twenty-six agency CHCOs reported to us that they 
systematically evaluate the effectiveness of executive training by 
assessing participant feedback. Most agencies reported they do this by 
using end-of-course evaluations. Seventeen of twenty-six agency CHCOs 
reported measuring the impact of training on participant performance. 
Five agencies reported using post-course surveys given to supervisors 
or participants to measure if what was learned affected the 
participants' behavior. For example, one agency requested that 
managers and mentors of Senior Executive Service Candidate Development 
Program (SESCDP) candidates fill out a questionnaire about the impact 
and value of the program for their candidate; this questionnaire 
included to what degree the candidate applied lessons learned in the 
program toward their developmental assignments. 

Twenty of the twenty-six agency CHCOs said they changed their 
executive training program as the result of evaluations. Six agencies 
reported they revised their program's content and logistics as a 
result of feedback. For example, VA reported that it reorganized its 
training schedule to add subjects that its senior executives wanted 
included. Other agencies reported they expanded the program based on 
participant feedback. For example, one agency expanded the number of 
leadership development programs offered, while a second agency 
increased the number and types of training opportunities available for 
executives in order to increase participation in executive training. A 
third agency now offers more learning activities (e.g., virtual 
training events) that do not require travel, in order to better 
accommodate the executives' schedules, as well as the current budget 
constraints within the agency. 

A majority of agencies do not have a formal process for evaluating the 
impact of training on the agency's performance goals and mission. Ten 
of twenty-six agency CHCOs - or less than half - said they measured 
the impact of training on the agency's mission, while eight of twenty-
six agency CHCOs said they measured the impact on agency performance 
goals. 

Agencies are required by statute and OPM implementing regulations to 
evaluate how well training programs contribute to mission 
accomplishment and meet organizational performance goals.[Footnote 13] 
OPM's Training Evaluation Field Guide provides information on 
methodologies for conducting training evaluations with the goal of 
identifying cost effective training initiatives that maximize mission 
accomplishments. The guide is based on the Kirkpatrick model of 
evaluation, a popular model of evaluation that includes measuring (1) 
participant reaction to the training program; (2) changes in 
employees' skills, knowledge, or abilities; (3) changes in employees' 
behavior or performance; and (4) the impact of the training on program 
or organizational results, which may be split into a fifth level that 
includes a return on investment assessment that compares training 
costs to derived benefits. 

Similarly, our guidance on assessing training states that to measure 
the real impact of training, agencies need to move beyond data on 
inputs and outputs by developing additional indicators that help to 
determine how training contributes to the accomplishment of agency 
goals and objectives.[Footnote 14] The traditional approach of 
collecting and reporting data related to training and development 
often involves indicators that can be readily quantified, such as the 
number of employees trained, training hours per employee, and total 
training hours. While necessary, these kinds of measures do not fully 
provide agencies with the kind of information they need to determine 
how training efforts contributed to improved performance, to reduced 
costs, or to a greater capacity to meet new and emerging 
transformation challenges. 

Two of the agencies we interviewed, Treasury and VA, were among those 
who reported in their questionnaires that they measured the impact of 
executive training on the agency's performance goals and mission. VA 
uses various assessment measures to evaluate effectiveness against 
VA's overall mission and the VA Secretary's strategic objectives for 
accomplishing transformation: VA reported that these measures have led 
to better training investment decisions. For example, VA includes post-
course requirements for its Strategic Leadership Course II - Basic, 
which is focused on providing its senior executives with tools to 
support strategic decision-making, critical thinking, and leading and 
driving change. Approximately 30 days after the training, class 
participants are asked to provide their "ah ha" moment from the 
training, with regard to their leadership style or techniques; 
approximately 90 days after the training, participants are asked how 
they have been able to apply the concepts from the executive training 
to their agency's organization or mission. Sometimes specific 
questions of interest are substituted, such as recommendations on 
strategic challenge questions for future classes, or requests for 
programs or resources that they need to better accomplish their 
mission as strategic leaders. A second requirement may be assigned at 
the post 90-day training point. This requirement usually asks the 
participants to explain how they have applied lessons taken from the 
training to their organization or mission. Additionally, the agency 
provides assessment data to a team of program analysts at the VA 
Learning University who periodically provide the agency with feedback 
on the effectiveness of the program. According to VA, the evaluation 
tends to focus on the impact of training on agency mission and 
strategic goals, and also on return on investment; the evaluation also 
includes specific examples where senior executives applied what they 
gained at the training and applied it to their mission in the VA. 
Treasury officials explained that while it would be hard to isolate 
when training by itself contributes to an individual meeting an 
agency's goal, by aligning all of its programs to the department's 
strategic plan and to the executive core qualifications (ECQ), they 
are ensuring the training support the agency's goals. 

CHCO agencies recognized the importance of evaluating the impact of 
training on their mission but emphasized the difficulty in conducting 
these types of evaluations. For example, two agencies reported needing 
additional assistance from OPM for these types of activities, such as 
centralized evaluation information on the performance of external 
providers. In addition, we previously found that some agencies found 
it difficult to conduct these evaluations because their cost and time 
demands can be significant.[Footnote 15] OPM acknowledged that some 
agencies struggle with evaluating training programs. OPM reported that 
it has developed and published regulations related to training 
evaluation. According to OPM, these regulations were developed to 
enhance the impact of training on agency mission accomplishment. The 
regulations generally relate to training planning, execution, and 
demonstration of value. However, OPM is not sharing lessons learned 
from agencies that reported having assessed the impact of executive 
training on agency missions. Given that less than half of the CHCO 
agencies reported they are evaluating the impact of executive training 
on their missions, sharing lessons learned in its oversight role could 
help OPM better position the other CHCO agencies for conducting such 
evaluations. 

We recognize that higher levels of evaluation (such as evaluating the 
impact on organization performance or the return on investment) can be 
challenging to conduct because of the difficulty and costs associated 
with data collection and the complexity of directly linking training 
and development programs to improved individual and organizational 
performance. However, eight agencies that responded to our 
questionnaire (and that are also members of the CHCO Council) reported 
that they have conducted these evaluations and may have lessons 
learned from which other agencies could benefit. According to GAO 
guidance on evaluating training, factors to consider when deciding the 
appropriate level of evaluation include estimated costs of the 
training effort, size of the training audience, management interest, 
program visibility, and the anticipated life span of the effort. The 
guidance also states that each agency consider the feasibility and 
cost-effectiveness of conducting these in-depth evaluations, along 
with budgetary and staffing circumstances that may limit the agency's 
ability to complete such evaluations.[Footnote 16] Without conducting 
higher levels of evaluation, agencies are missing information that 
could help them make more effective training investment decisions; in 
addition, OPM is missing the opportunity to assist agencies in fully 
understanding the benefits of executive training on the continued 
development of senior executives and on the agency's performance. 

CHCOs Identified Opportunities for Agencies and OPM to Achieve 
Efficiencies in Executive Training: 

Opportunities Include Using Virtual Training and Centralizing Training: 

Twenty of twenty-six agency CHCOs responding to our questionnaire said 
they identified opportunities for achieving efficiencies in delivering 
executive training within their respective agencies. For instance, 
some agency CHCOs described steps agencies could take to increase 
participation in executive training programs within their respective 
organizations. For example, one CHCO said his agency could expand the 
base of eligible participants to include employees in other senior 
level positions outside the SES. Our guide for assessing strategic 
training and development efforts states that agencies should select 
employees for developmental programs, in part, based on the usefulness 
of training content given employees' current duties and existing 
skills and competencies.[Footnote 17] Employees in other senior level 
positions could build their skills and competencies through 
participation in executive training; at the same time, agencies could 
increase participation in executive training courses by expanding 
participation eligibility to employees in other senior level positions 
outside the SES, thus reducing unit costs of training delivery. 
Another CHCO said his agency is considering using executives and 
senior leaders to serve as facilitators for leadership events. As we 
previously concluded, efforts to identify cost-effective and robust 
options for designing training and development programs may include 
use of in-house staff, such as senior executives, based on their 
capability to implement training.[Footnote 18] 

Four CHCOs said they are considering implementing or expanding 
computer-based training, such as use of webinars, online books, 
virtual "executive summits" with developmental sessions and strategic 
meetings for executives, and other online training programs used for 
executive development. One CHCO also said his agency is considering 
use of blended learning activities--which incorporate classroom-based 
and computer-based training--to maximize impact and cost-
effectiveness. This is consistent with leading practices for training 
investments. Specifically, one such practice calls for comparing the 
merits of different methods of delivery--such as classroom or computer-
based training--and determining what mix of mechanisms to use to 
ensure efficient and cost-effective delivery.[Footnote 19] Another 
leading practice is for agencies to consider blended learning that 
combines different teaching methods (e.g., Web-based and instructor-
led) within the same training effort and provides trainees with the 
flexibilities to choose among different training delivery methods, 
while leveraging resources in the most efficient way possible. 
[Footnote 20] 

In our interviews with selected agencies, Treasury officials reported 
they are working to centralize training for all Treasury executives 
through the Treasury Executive Institute (TEI) in an attempt to 
enhance Treasury's return on investment for organizational performance 
and to provide more consistency for mission-critical leadership 
training. Treasury officials said they could use TEI to leverage 
training resources throughout the department, with a goal of providing 
"just-in-time" training to senior executives, rather than requiring 
them to set aside their normal job responsibilities for a number of 
months to attend training. Treasury officials told us the agency is in 
the beginning stages of obtaining OPM approval for an agency-wide 
SESCDP, which would allow Treasury to train executives from all of its 
components and would help Treasury fill vacant SES positions agency-
wide. Treasury officials said they are likely to operate the proposed 
agency-wide SESCDP alongside the Internal Revenue Service's SESCDP--
the only SESCDP currently in operation within Treasury. Treasury's 
efforts to centralize SES candidate training is consistent with a 
leading practice related to identifying the most appropriate mix of 
centralized and decentralized approaches for an agency's training and 
development programs.[Footnote 21] In particular, centralizing design 
can enhance consistency of training content and offer potential cost 
savings through standardized record keeping and simplified and more 
accurate reporting on courses, certifications, educational attainment, 
costs, or standards.[Footnote 22] 

VA officials said they performed after action reviews on and made 
changes to their Strategic Leadership Course II - Basic course for 
career SES members within VA. Based on the reviews and subsequent 
changes to its executive training program, VA officials negotiated a 
new price structure from the course vendor to deliver the training at 
a lower cost in 2013 than if they had used the course's 2012 price 
structure. We previously concluded that agencies can use evaluation 
feedback to identify problems and improve training and development 
programs as needed.[Footnote 23] Based on modifications to its 
executive training programs from after action reviews, VA was also 
able to reduce program costs, thus increasing the efficiency of 
training delivery. 

Further, 20 of 26 agency CHCOs said they found opportunities for 
achieving efficiencies government-wide. For instance, some CHCOs said 
agencies can increase interagency cooperation on executive training by 
sharing space in training seminars with other agencies, and allowing 
admission of SES candidates from other federal agencies into training 
programs such as SESCDPs. Treasury officials we interviewed said they 
actively market shared training services to the rest of the government 
through TEI. Energy told us the agency produces a catalog on a semi-
annual basis with real-time information on leadership development 
courses and programs located throughout the country; according to 
Energy officials, government officials could use the catalog to reduce 
travel and per diem costs for executive seminars conducted around the 
nation by identifying equivalent training that is available locally 
for senior executives. Additionally, several CHCOs said their agencies 
used VA's Leading EDGE program to deliver executive training. One CHCO 
offered to share training facilities with other agencies. However, one 
CHCO noted that courses sponsored by OPM and VA raised the possibility 
of redundancy in executive training delivery provided by the two 
agencies. According to our guide for assessing strategic training and 
development efforts, agencies can try to achieve economies of scale 
and avoid duplication of effort when assessing training delivery 
options by taking advantage of existing course content or training, 
such as sharable online courseware or multiagency training programs. 
Specifically, interagency training can help address common 
developmental needs government-wide and promote cost-efficiency while 
taking advantage of existing resources rather than creating similar 
programs in multiple agencies.[Footnote 24] 

CHCOs Reported OPM Can Help Agencies by Offering More Lower Cost or No 
Cost Training and by Centralizing Training Offerings: 

CHCOs from agencies responding to our questionnaire reported that OPM 
can take several steps to help both the federal government and 
individual agencies deliver executive training more efficiently. 
First, some agency CHCOs said that OPM should provide more training to 
senior executives government-wide that is free or lower cost to 
agencies. One CHCO noted that OPM could provide lower or no cost 
executive development opportunities to agencies by more effectively 
using technology to deliver virtual training. This would be consistent 
with leading training practices comparing the merits of different 
delivery vehicles, such as classroom or computer-based training, in 
order to assess what vehicles to use for ensuring efficient and cost-
effective delivery. OPM officials said that agencies cannot avoid 
training costs, but OPM is trying to reduce agency training costs by 
bringing courses closer to agency offices and by using free space for 
training rather than renting sites. OPM has also developed free and 
fee-based training solutions for use government-wide, and hosts 
speaker series and meta-leadership courses for senior executives. 
[Footnote 25] 

Second, agency CHCOs said that OPM could take steps to help centralize 
training offerings for senior executives, such as creating a centrally 
funded SESCDP, and could coordinate joint training efforts among 
agencies. For example, according to one CHCO, OPM could take steps to 
encourage agencies to fill paid-for, but unfilled seats for executive 
training programs across agency lines. CHCOs also said that OPM could 
assist agencies in locating executive training programs open to other 
agencies and departments, and provide a central source for listing 
executive development opportunities throughout the federal government. 
An example of this type of executive training program is the Leading 
EDGE program administered by VA. This program is operated by a senior 
executive within VA and offers executive training to agencies across 
the federal government. According to the program's deputy director, 17 
federal agencies are participating in Leading EDGE training. 
Treasury's TEI is another example of a shared service model that 
presents an opportunity to leverage resources across the government. 
According to Treasury, TEI serves 9 other bureaus and agencies in 
addition to Treasury. Our guide for assessing training practices 
previously found that centralizing design of training programs can, 
among other things, offer potential cost savings.[Footnote 26] 

OPM officials agree they could centralize training offerings for 
senior executives. For instance, OPM hosted the Office of the Director 
of National Intelligence's (ODNI) suite of 12 courses detailing the 
full performance management cycle; in turn, OPM's Human Resources 
University (HR University) made them available for use by other 
agencies, and OPM officials said they incurred savings of about $46 
million in 11/2 years.[Footnote 27] In terms of coordinating joint 
training efforts among agencies, OPM officials said they collaborate 
with the CLO Council, the CHCO Council, and federal agencies to 
provide joint training. For example, OPM collaborates with VA to 
implement the Leading EDGE program for use by many agencies. OPM 
officials also suggested that federal agencies use additional FEI 
courses--and to reimburse FEI for implementing such courses through 
user fees--so it could become the federal government's primary source 
for executive training. As mentioned earlier, sharing online 
courseware and implementing multiagency training programs can help 
agencies try to achieve economies of scale while avoiding duplication 
of effort in designing executive training courses that address similar 
content as existing courses. 

OPM Reports Taking Additional Actions to Improve Training Efficiency: 

OPM officials noted they are taking additional steps to increase 
training efficiencies beyond what agency CHCOs suggested. Attributes 
of effective training include planning training that considers 
government-wide reforms and other targeted initiatives to improve 
management and performance, as well as administration priorities, 
legislative reforms, and major management challenges.[Footnote 28] In 
response to our September 2012 recommendation on federal training 
investment practices, OPM officials said the agency is working with 
the CLO Council to draft a government-wide mandatory training 
curriculum and finish plans to implement a pilot course for mandatory 
training.[Footnote 29] OPM officials also said the agency is working 
with CLO Council representatives to standardize elements of federal 
executive training programs by developing a framework for supervisory 
and managerial leadership development. The supervisory framework has 
been distributed to agencies. According to OPM officials, senior 
executives and other managers can use the supervisory training 
framework to identify learning objectives they need to achieve, and 
then target the right job assignments and training courses to address 
the objectives. OPM officials said agencies are not required to use 
the framework to develop their executive training programs; however, 
some agencies are using the framework as a "roadmap" for aligning 
their training programs. 

In November 2012, OPM also released its Executive Development Best 
Practices guide, which includes key themes and recommendations OPM 
identified as hallmarks of successful executive development efforts 
across the public and private sectors. OPM has also developed a 
training and development webpage where agency officials can find and 
share products and courses for all areas of learning and development. 
However, the webpage does not describe examples of agency efforts to 
achieve efficiencies in delivering executive training. 

OPM's efforts so far to work with agencies to identify efficiencies 
for executive training have been relatively broad-based; while OPM has 
taken steps to increase efficiencies, it does not have a process for 
assessing lessons learned from agencies about potential efficiencies 
in executive training that other agencies could implement. Our 
questionnaire results suggest that OPM can use its oversight role to 
take additional steps to assess and promote ways agencies can achieve 
efficiencies in executive training that have government-wide 
application. By not taking such steps, OPM may be missing an 
opportunity to better position agencies to achieve such efficiencies. 

Career SES and SES Candidates Said External Training is Valuable, but 
Training Programs Could be Improved: 

Career SES and SES Candidates Found Executive Training Useful in 
Building Organizational Knowledge, Developing Leadership Skills, and 
Expanding Networks of Colleagues: 

Focus group participants said that executive training helped them 
develop a better understanding of their roles and responsibilities in 
their respective agencies. Following visits to field offices during 
executive training, one participant said there is a significant amount 
of "interconnectivity and intertwining between different operations" 
within the participant's agency. Participants also said they learned 
about key organizational issues and tools that they can use to address 
them. For instance, a group of SES candidates in one agency reviewed 
agency-wide telework implementation and developed a business case to 
overcome technical challenges related to phone systems. 

Focus group participants from the three agencies we reviewed also said 
they developed or strengthened key leadership skills during executive 
training that align with ECQs and related competencies, such as: 

* Leading and communicating with people. Participants we interviewed 
said they learned how to better motivate and communicate with staff. 
One participant noted that "how you interact with people is 
critical" - a lesson that was emphasized when the participant was 
implementing difficult changes in the participant's first SES 
position. The participant also learned to be both sensitive and 
emphatic when dealing with issues raised by staff members. The 
participant said that executive training enabled the participant to 
help transform an organization with "primarily a bad reputation" into 
one of the top performing organizations within the participant's 
agency. Leading People is one of five ECQs highlighted by OPM. 

* Problem-solving and decision-making. Executive training, according 
to participants, provided an environment where they could develop 
skills in problem solving. For instance, one participant described 
learning how to make more effective decisions by sharing best 
practices from different business units within an agency and noted 
that, as a result, the participant and agency colleagues "are making 
better decisions than if we were just kind of operating in our private 
little bubble." Another participant noted that executive learning 
helps agency officials reach out to other areas in their organization 
and leverage others' experiences in resolving problems, "instead of 
trying to reinvent the wheel" to identify solutions already considered 
within the agency. Problem-solving and decisiveness are two 
competencies identified under the Results Driven ECQ. 

* Strategic thinking. Executive training courses enabled some 
participants to step back from managing day-to-day operations and 
reconsider, from a strategic sense, what they wanted to accomplish in 
their position. One participant said the use of strategic thinking and 
leadership was necessary to create a vision for subordinates in 
conducting agency operations. A participant from another agency added 
that strategic thinking enables executives to consider the needs of 
stakeholders beyond one division and office, and to identify how any 
decisions the participant makes could affect an entire agency. 
Strategic thinking is a competency identified under the Leading Change 
ECQ. 

Focus group participants also said that networking with colleagues and 
senior agency officials in executive training was beneficial during 
training courses and after re-entry into their positions. In 
particular, participants noted the following networking opportunities 
in executive training: 

* Participation develops personal relationships that boost confidence 
and build networks across and within federal agencies. One participant 
said relationships built with others helped make the executive feel 
more comfortable in an SES position, and validated that the 
participant was "on track" when facing difficult situations on the 
job. Another participant emphasized that a significant amount of work 
in the participant's position involved relationships formed in small 
groups during executive training with people representing other 
agencies. 

* Participation builds accountability among executives within an 
organization. One SES candidate said that relationships formed during 
executive training extended into executive assignments following 
completion of training. The candidate said such relationships helped 
training participants hold each other accountable in their executive 
assignments, which can be a tremendous benefit to organizations. 

* Participants develop better responses to customer and stakeholder 
needs. Multiple participants said relationships they established in 
executive training helped them collaborate to resolve customer issues. 
For example, two participants from different components in one agency 
said they developed a relationship during executive training that 
enabled them to work together to provide financial relief to an agency 
customer without using additional agency resources. Another 
participant noted he completed a $400 million project with another 
agency before a scheduled deadline, based on trust and relationships 
he previously built with fellow participants in an SESCDP. 

* Participation promotes interaction and relationship building with 
senior leadership to address organizational issues. From interaction 
with the Postmaster General, one participant learned methods to 
motivate employees following "massive changes" within organizations, 
such as those experienced by the U.S. Postal Service. Another 
participant (who was addressing equal employment opportunity issues) 
established a connection with an agency undersecretary, who allowed 
the participant to send staff to the undersecretary's office for 
shadowing assignments to better resolve equal employment opportunity 
issues within the participant's office. 

* Participation facilitates brainstorming solutions to previously 
identified issues faced by career SES and SES candidates. Focus group 
participants in one agency discussed two cases where networking and 
brainstorming among executives helped solve problems and generated 
financial savings. One participant used a week-long training course to 
address implementation of a program involving overseas locations. 
Course members helped the participant develop ideas to adjust hiring 
practices that, according to the participant, saved the federal 
government money. Another participant noted that agency officials used 
executive training to determine steps needed to resolve backlogs of 
work within an agency; according to the participant, this work may 
result in cost savings for the federal government over the next 10 to 
20 years. 

Career SES and SES Candidates Said the Usefulness of Executive 
Training Depends, In Part, on Program Structures Used to Carry Out 
Training: 

Focus group participants found the use of cohorts--groups of career 
SES and SES candidates participating in the same training courses, and 
other types of groups--more conducive for problem-solving and 
knowledge sharing than use of large group settings. Participants from 
one agency, for example, supported the use of cohorts in executive 
training to foster connections and build a knowledge base outside 
their respective functions. Participants from Energy, Treasury, and VA 
said that participating in small groups and using learning tools--such 
as case studies and action learning projects--helped career SES and 
SES candidates enhance problem-solving skills and share ideas with 
each other. As an example, one participant said that members of small 
groups applied lessons learned from case studies towards real-life 
problems. Another participant noted that classmates from executive 
training regularly met in groups to discuss challenges and share best 
practices with each other. 

Participants also found executive training to be more useful when 
conducted off-site from the participants' offices and when 
participants could step back from day-to-day responsibilities. As 
described earlier, participants who were able to withdraw from 
managing day-to-day operations said they could incorporate more 
strategic thinking into considering organizational issues and 
accomplishments, and could better focus on learning needed skills to 
advance within their organization. Focus group members specifically 
praised the use of off-site training for allowing participants to 
develop ECQs, learn many of the skills needed to succeed as an 
executive, and refresh insights or skills they may not have used for a 
period of time, without distractions that may have hindered training 
at an on-site location. 

In contrast, some participants did not find large gatherings to be 
useful in encouraging participation or integrating lessons learned. 
One focus group member noted that colleagues took advantage of large 
gatherings to complete training requirements without the need to pay 
attention to lessons learned from the gathering. Another focus group 
member who participated in a large gathering noted that attendees 
listened to speakers for several hours without collaborating with the 
speakers or each other, despite being informed they would be able to 
interact with people from other agencies. 

Additionally, focus group participants said they found virtual 
training with computers, or with video or phone conferencing, to be 
less helpful than face-to-face training. For example, one participant 
said that virtual training tools, such as webinars, can provide 
effective training on technical or discrete subjects, yet are less 
effective for developing less tangible strategic, leadership, and 
executive skills. Participants said they found it difficult to build 
teams or individual relationships through virtual means, which limited 
networking and knowledge sharing. Other participants said they were 
multitasking and completing other tasks while taking virtual training; 
one participant noted that the training could be more interesting if 
delivered through other means and in a shorter time period. 

Career SES and SES Candidates Believe Agencies Can Take Steps to 
Enhance Executive Training Programs While Improving Efficiencies of 
Training Delivery: 

Focus group participants also provided suggestions to improve the 
usefulness of such programs to career SES and SES candidates, while 
also increasing efficiencies in delivering training courses that could 
potentially reduce training costs for federal agencies. 

As one example, focus group participants we interviewed suggested 
agencies could pool resources and consolidate executive training 
programs. One participant suggested the creation of a government-wide 
SESCDP through OPM or another agency; this step could standardize 
program financing and rules for program implementation government-
wide. Likewise, focus group participants said executive training 
courses could be consolidated through the use of economies of scale 
which would acquire such training for multiple government agencies. 
Participants noted that the federal government--instead of individual 
agencies--could use its influence as a large-scale buyer to leverage 
taxpayer dollars and negotiate lower prices for training from external 
providers. Another participant suggested that the federal government 
could achieve cost savings by requiring external providers to compete 
against each other to provide executive training courses. 

Focus group participants recommended the use of blended training 
courses--involving both face-to-face and virtual training--to increase 
efficiencies in training delivery. As mentioned earlier, participants 
described limitations in using virtual training such as limited 
interaction among participants and reduced attention paid to course 
material. However, participants said a blended learning model allows 
them to establish face-to-face relationships, with one participant 
noting that human interactions were a key part of learning. At the 
same time, use of computer-based technology and phone conferencing as 
part of a blended learning model enables participants to build upon 
such relationships and learn course material that can be effectively 
delivered through virtual means. 

Additionally, some focus group participants suggested that OPM and 
other federal agencies could increase use of workshops and brief 
training sessions to deliver training more efficiently to executives. 
Specifically, use of one-day workshops and training sessions could 
increase participation in training by bringing together executives 
from multiple agencies to encourage cross-pollination of ideas and to 
learn improved leadership techniques. Finally, participants said that 
agencies could establish programs to increase selection of SESCDP 
graduates for open SES positions government-wide. These steps, 
according to participants, would allow agencies to more efficiently 
use training acquired by SESCDP graduates, and to achieve greater cost 
efficiencies by filling vacant agency SES positions with SESCDP 
graduates from other agencies. 

Conclusions: 

Agencies are using their executive training programs to help develop 
the necessary leadership skills for executives to effectively manage 
the nation's most pressing priorities. Agencies recognize that 
evaluating executive training activities yields useful information for 
improving investment decisions. Yet, agencies can only make decisions 
that improve the value of executive training when they have complete 
and reliable training cost data and some measure of how the training 
benefits the agency beyond just a measure of participant feedback. 
Over 7 years have elapsed since agencies were required to report 
training cost data to OPM, and many agencies are not yet accurately 
and completely reporting this information. For corrective actions OPM 
has already initiated, establishing milestones could help ensure that 
efforts progress as intended. Moreover, by not establishing a 
timeframe for improving the reliability of executive training cost 
data, OPM may be missing an opportunity to better position itself to 
hold agencies accountable for improving the data. Improving the 
overall reliability of the data would better position OPM and the 
agencies to make sound decisions on training investments. 

Half of the CHCOs reported they do not have complete information on 
the cost of executive training provided by external training 
providers. Moreover, many agency CHCOs struggle to conduct evaluations 
that identify the impact of training on organizational performance and 
agency mission, although a few CHCO agencies reported conducting such 
evaluations. OPM has not used interagency councils to share lessons 
learned from agencies that have experience with assessing executive 
training's impact on agency missions. Although OPM has developed and 
published guidance on evaluating training's impact on agency mission 
and goals, over half of the CHCO agencies are not conducting these 
evaluations of executive training provided by external providers. 
Agency CHCOs reported that more assistance from OPM is needed. 
Additionally, many CHCOs believe there are opportunities for an 
increased role by OPM for improving efficiency in executive training 
and providing options for achieving efficiencies. By not assessing 
lessons learned and other potential efficiencies identified by 
agencies, OPM may be missing an opportunity to better position 
agencies to achieve these efficiencies in executive training. Given 
the fiscal challenges facing the nation, and the increasing complexity 
of policy issues managed by the SES, it is important for agency 
leaders and for OPM to take advantage of reasonable opportunities to 
share resources and improve the efficiency of training investments. 

Recommendations for Executive Action: 

To help ensure that agencies track and report comparable and reliable 
cost data and perform evaluations that assess the impact of executive 
training on agency performance or missions, we recommend that the 
Director of OPM, in coordination with the CHCO Council, take the 
following actions: 

* Establish interim milestones for meeting with agencies in order to 
address training data deficiencies and to establish well-defined 
timeframes for improving the reliability of the data in its EHRI 
database. 

* Improve assistance to agencies regarding evaluating the impact of 
executive training on mission and goals, for example by sharing 
information and examples of how agencies could better conduct such 
evaluations. 

To enhance the efficiency of executive training, we also recommend 
that the Director of OPM, in coordination with the CHCO Council: 

* Assess potential efficiencies identified by agencies for possible 
government-wide implementation, and then take the steps necessary to 
implement these, such as updating the guidance governing executive 
training programs. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the Director of OPM for review 
and comment. OPM provided written comments, which are reprinted in 
appendix II. OPM concurred with our recommendations and is engaged in 
a number of activities that would help to achieve the objectives of 
our recommendations. However, we believe that more steps can be taken. 

In response to our first recommendation, OPM discussed actions it has 
initiated to address agencies' training data deficiencies including 
launching the Federal Training Investments Work Group, a group of 
Chief Learning Officers, to help OPM define data elements that can and 
should be captured within agency automated learning management 
systems. OPM's plan is to provide technical assistance to individual 
agencies during FY 2014 to help build agency capacity and to allow 
agencies to set milestones for improvement after assessing their 
training data using forthcoming standards. We believe OPM could work 
with agencies to set timeframes for agencies to have a plan in place 
for improving completeness and reliability of agency data on executive 
training costs. Working with agencies to mutually set timeframes would 
help OPM hold agencies accountable for improving training data, thus 
moving agencies towards meeting reporting requirements set in 2006. 

In response to our second recommendation, OPM stated that it currently 
provides assistance to agencies on evaluating the impact of executive 
training through use of the OPM Training Evaluation Field Guide, which 
is included in our report, but did not indicate whether or how it 
measures the effectiveness of that guidance. We believe OPM could 
further strengthen its assistance by leveraging the experiences of 
agencies that have conducted these evaluations by sharing techniques 
and approaches with CHCO agencies that continue to struggle with 
evaluating executive training. OPM could also ask agencies to assess 
the effectiveness of its assistance. 

In response to our third recommendation, OPM highlighted its Executive 
Development Best Practices Guide which contains a compilation of 
executive development best practices used by the public and private 
sector to support the continued learning of executives, as well as 
additional actions it has taken to help educate agencies on achieving 
efficiencies in executive training. In its oversight role, OPM, 
together with agencies, can develop and use criteria to assess 
efficiencies put in place by agencies that have potential government-
wide application. For instance, OPM can assess agency initiatives--
such as creating a centrally-funded SESCDP, or coordinating joint 
training efforts among agencies--to determine whether these 
alternative approaches are appropriate for wider application. 

OPM also provided technical comments on the draft report, which we 
incorporated in the report as appropriate. 

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to the Director of OPM and other interested parties. In addition, the 
report will be available at no charge on the GAO website at 
[hyperlink, http://www.gao.gov]. If you have any questions about this 
report, please contact me at 202-512-2717 or jonesy@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. GAO staff who made key 
contributions to this report are listed in appendix III. 

Sincerely yours, 

Signed by: 

Yvonne D. Jones: 
Director Strategic Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of this engagement were to review the costs of federal 
executive training programs and the value they provide to the federal 
workforce. Specifically, this report (1) describes what is known about 
how much Chief Human Capital Officers Council (CHCO Council) agencies 
have spent on executive training offered by external providers and 
assesses the reliability of available cost data; (2) assesses how CHCO 
Council agencies evaluate the effectiveness of the training; (3) 
compares opportunities identified by agency CHCOs to achieve 
efficiencies in executive training to leading training practices, and 
(4) provides views of career Senior Executive Service (SES) and SES 
candidates from selected agencies on the value of the training they 
received. 

For the purposes of this review, we define the key terms "executive 
training," "external provider," "SES candidates," and "career SES" as 
follows: 

* Executive training is classroom or computer-based instruction in 
leadership or development programs with a focus on the Office of 
Personnel Management's (OPM) executive core qualifications (ECQ). Each 
individual training episode should be indicated by completion of 
Standard Form 182. 

* External provider is any organization (including educational 
institutions, professional societies, think tank organizations, or 
consulting firms) that provides training and is independent from the 
federal department or agency that employs the trainee. The external 
training provider is contracted to develop or provide training courses 
or programs. In this case, OPM's Federal Executive Institute (FEI) and 
Management Development Centers are considered external providers. 

* SES candidates are GS-15, GS-14, and equivalent employees who are 
enrolled in either (1) an OPM-approved SES Candidate Development 
Program or (2) any agency-funded program that will train and develop 
employees for the SES or an equivalent. 

* Career SES or equivalents are those members with civil service 
status who are appointed competitively to SES positions or equivalents 
in positions below the top political appointees in the executive 
branch of government. 

To obtain government-wide information on agency executive training 
programs, we developed a questionnaire on agency executive training 
practices and processes and administered it to 27 agencies represented 
on the CHCO Council.[Footnote 30] Through the questionnaire, we 
obtained high-level information on executive training programs, 
evaluation practices, and opportunities to achieve efficiencies in 
such programs, as identified by agencies. To the extent available, we 
also collected data on the amount agencies spent on executive training 
courses administered by external providers from fiscal years 2008 to 
2012.[Footnote 31] We provided a standard set of questions to each 
CHCO to ensure we consistently captured their responses to our 
questions on their executive training practices. We received 
responses, supporting documentation, or both from 26 of 27 agencies. 
[Footnote 32] To analyze the information we received, one analyst 
inputted responses to quantitative questions and analyzed the 
responses for univariate statistics using Microsoft Excel; a second 
analyst checked the input and analysis of responses for accuracy. For 
the responses to open-ended questions, we conducted a content analysis 
to identify common themes among the 26 respondent agencies. 

We assessed the reliability of executive training participant and cost 
data from OPM's Enterprise Human Resources Integration (EHRI) system 
based on reviews of our prior work on federal training investments, 
[Footnote 33] testing of agency data from an OPM executive training 
dataset, and interviews conducted with OPM officials. We determined 
that the data provided in EHRI was reliable for the purposes of 
selecting case examples and for comparing participant and cost data in 
EHRI to similar data provided directly by agencies. We also assessed 
the reliability of executive training participant and cost data 
provided by agencies in response to our questionnaire. Specifically, 
we examined the data provided for obvious errors and inconsistencies 
and we also queried 10 agencies to better understand the data systems 
each agency used to collect and report executive training participant 
and cost data, as well as the quality of data entered into their 
system(s).[Footnote 34] We determined that aggregated participant and 
cost data for fiscal years 2008 to 2012 are sufficiently reliable for 
the purposes of the report; however, we qualified these data 
throughout the report due to potential underreporting from agencies. 

We selected three agencies - the Department of Energy (Energy), 
Department of the Treasury (Treasury), and Department of Veterans 
Affairs (VA) - to obtain illustrative examples of how they approached 
executive training in their respective agencies and collected data 
related to costs; how they implemented and evaluated the results of 
their executive training programs identified in the questionnaire; and 
how they identified potential efficiencies and steps planned or 
already taken to achieve them. We selected these agencies based on 
workforce size, number of career SES, total training costs, and types 
of evaluations conducted to measure effectiveness of executive 
training, and whether agencies identified potential opportunities for 
achieving efficiencies in executive training. We interviewed agency 
officials from the three selected agencies with responsibility for 
administering their agency's executive training programs. To 
supplement and support the views provided by agency officials, we also 
obtained and examined available documentation (such as agency 
executive training plans, evaluation plans, and other documents 
associated with executive training) from each of our selected 
agencies. The findings are not generalizable to other agencies or to 
the federal government as a whole, because we designed our selection 
of agencies to examine the experiences, practices, and perspectives of 
a set of agencies that, in recent years, have utilized external 
providers to train executives and have reported that they (1) evaluate 
the benefits achieved through executive training, and (2) have 
identified opportunities to achieve efficiencies in executive training. 

Additionally, we conducted focus groups of current or prior executive 
training participants from the three selected agencies in order to 
obtain their views on the usefulness and value of such training and 
their ideas for increasing efficiencies in executive training delivery 
within agencies and government-wide. For two of these agencies, we 
conducted two focus groups for each: one representing current or 
recent participants in SES candidate development programs, and one 
representing career SES members. For the third agency, which did not 
have recent SES candidates at the time we selected our focus group, we 
also conducted two focus groups: one representing early career SES 
members who completed a basic agency leadership course, and one 
representing mid-to late-career SES members who completed an advanced 
agency leadership course. The insights gained from the discussions 
were not generalizable to all career SES and SES candidates within 
participants' agencies or to the federal government as a whole. 

We also collected documentation from and interviewed OPM officials 
involved in the agency's oversight function for federal training 
activities in order to obtain information on their roles and 
responsibilities in monitoring agency training programs, on their 
perspectives on executive training government-wide, and on steps they 
have taken to improve training efficiencies. We examined how, if at 
all, OPM, the CHCO Council, and the Chief Learning Officers Council 
(made up of chief learning officers from the 27 agencies) share 
information on potential efficiencies by interviewing OPM officials 
and members of the CHCO Council and the Chief Learning Officers 
Council. We interviewed five external training providers who were most 
frequently used by CHCO Council agencies, to collect information on 
executive training course offerings, costs, and evaluation strategies; 
however, the collected information is not generalizable to all 
external providers.[Footnote 35] We also interviewed officials from 
the Partnership for Public Service and Booz Allen Hamilton who had 
relevant expertise in federal leadership training activities in order 
to obtain information on prior studies of executive training, and on 
their perspectives about evaluating the effectiveness of executive 
training. 

We conducted this performance audit from December 2012 to December 
2013, in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Office of Personnel Management: 

United States Office of Personnel Management: 
Employee Services: 
Washington, DC 20415: 

January 17, 2014: 

Ms. Yvonne J. Jones: 
Director, Strategic Issues: 
U.S. Government Accountability Office of Personnel Management: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Jones: 

Thank you for providing the U.S. Office of Personnel Management (OPM) 
the opportunity to comment on the Government Accountability Office 
(GAO) draft report, "Agencies Should More Fully Evaluate the Costs and 
Benefits of Executive Training." We appreciate the opportunity to 
provide you with comments about this report. 

GAO Recommendations: 

To help ensure that agencies track and report comparable and reliable 
cost data and perform evaluations that assess the impact of executive 
training on agency performance or missions, we recommend that the 
Director of OPM, in coordination with the CHCO Council, take the 
following actions: 

* Establish interim milestones for meeting with agencies to address 
training data deficiencies and establish well-defined timeframes for 
improving the reliability of the data in its EHRI database. 

* Improve assistance to agencies on evaluating the impact of executive 
training on mission and goals, for example by sharing information and 
examples of how agencies could better conduct such evaluations. 

To enhance the efficiency of executive training, we also recommend 
that the Director of OPM, in coordination with the CHCO Council: 

* Assess potential efficiencies identified by agencies for possible 
government-wide implementation, and then take the steps necessary to 
accomplish these, such as updating the guidance governing executive 
training programs. 

We will address each of the recommended actions separately. 

OPM Management Response: 

Recommendation: Establish interim milestones for meeting with agencies 
to address training data deficiencies and establish timeframes for 
improving the reliability of the data in its EHRI database. 

OPM concurs with this recommendation and has initiated action to 
address agencies' training data deficiencies. Before any individual 
agency consultation should take place, cross-agency data standards 
must be defined. In order to help it establish this definition, OPM 
has sought the input of the Chief Learning Officers community. In 
2013, OPM launched the Federal Training Investments Work Group, a 
group of Chief Learning Officers who are helping OPM define data 
elements that can and should be captured within agency automated 
learning management systems. Once these standards are designed, OPM's 
Enterprise Human Resource Integration (EHRI) team will provide
individual agencies technical assistance to build agencies' capacity 
to deliver timely, accurate and complete training data. These agency 
consultations are planned for FYI4, once the data standards are 
designed. Agencies will then be able to assess their data quality and 
set milestones for improvement, using a scorecard that the Chief 
Learning Officers Council working group will help OPM to develop. OPM 
will monitor agency progress toward their data quality improvement 
milestones. 

Recommendation: Improve assistance to agencies on evaluating the 
impact of executive training on mission and goals, for example by 
sharing information and examples of how agencies could better conduct 
such evaluations. 

OPM concurs with this recommendation. OPM currently provides 
assistance to agencies on evaluating the impact of executive training 
through use of the "OPM Training Evaluation Field Guide." The field 
guide provides easy to understand, step-by-step instructions on 
training program evaluation. The field guide also contains agency best
practices in training evaluation. Agencies are invited, through our 
wiki and our various forums, to contact us if they require training on 
any aspects of planning, executing and demonstrating the value of 
training. OPM is actively involved in assisting agencies in evaluating 
the impact of executive training. For instance, OPM provided training 
and an evaluation model to Senior Executive Service Candidate 
Development Program (SES CDP) coordinators, provided several 
presentations to individual agency training staffs (virtual and 
classroom-based), and coordinated with the Chief Learning Officer 
Council to draft an evaluation model for agencies to use when 
evaluating the impact of their leadership development programs. 

Recommendation: Assess potential efficiencies identified by agencies 
for possible government-wide implementation, and then take the steps 
necessary to accomplish these, such as updating the guidance governing 
executive training programs. 

OPM concurs with this recommendation and is already engaged in this 
activity. OPM understands the importance of periodic assessments; and, 
in December 2012 published the Executive Development Best Practices 
Guide. This guide contains a compilation of executive development best 
practices used by the public and private sector to support the 
continued learning of executives. With an emphasis on impact, cost and 
innovation, the information was gathered through an extensive 
literature review, as well as through interviews with a number of 
Fortune 500 companies and Federal agencies. OPM developed this guide 
to help agencies more efficiently and effectively meet the development 
needs of Federal executives. 

OPM understands the potential efficiencies and has hosted several 
executive development programs. For example, in April 2013, OPM hosted 
two cost-free one-day leadership seminars, in Washington, DC, on the 
topic of Meta-Leadership for all current SES members. Meta-Leadership 
is a framework and practice method developed by faculty at the 
National Preparedness Leadership Initiative (NPLI), a joint venture of 
the Harvard School of Public Health's Division of Policy Translation 
and Leadership Development and the Harvard Kennedy School's Center for 
Public Leadership. The objective of the seminars was to support 
continual executive development in the Federal Government, preparing 
executive-level officials to meet complex challenges of the modern era 
through exceptional leadership across various sectors and 
organizations. In addition, OPM developed a free web-based course on 
Executive Excellence and Wellness. This course and other free courses 
for continuing executive development can be found in the Manager's 
Corner on HR University [hyperlink, http://www.hru.gov]. Nevertheless, 
by law, agencies are responsible for the training of their own 
employees, and training that is provided by OPM must ordinarily be 
provided on a reimbursable basis. 

Technical comments to the draft report are enclosed. Unless otherwise 
noted the suggested provisions are meant to provide technical accuracy. 

Sincerely, 

Signed by: 

Timothy F. Curry, for: 

Mark Reinhold: 
Acting Associate Director: 
Employee Services: 

Clarifying Recommendations to the GAO Report: 

p. 5: language is OPM "oversees the certification" of SES CDPs. Please 
note that OPM reviews and certifies Agency SES CDPs and does not 
"oversee" the certification process. 

P. 6: report refers to OPM courses not administered by FE!. The report 
should also list the courses for executives that are offered by the 
OPM Management Development Centers. 

On p. 20, the report mentions our collaboration with the CLO Council 
on a supervisory and management development framework and a mandatory 
pilot course. Please note the Supervisory Framework is no longer in 
draft and has been distributed to agencies. The framework can be
found here [hyperlink, 
http://www.chcoc.gov/transmittals/TransmittalDetails.aspx?TransmittalID=
5185]. 

[End of section] 

Appendix III: GAO Contacts and Staff Acknowledgments: 

GAO Contact: 

Yvonne D. Jones, Director, (202) 512-2717 or Jonesy@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Signora May (Assistant 
Director), Clifton G. Douglas Jr., Janelle House, Andrea Levine, Mark 
Ryan, Joseph Santiago, and Cynthia Saunders made key contributions to 
this report. Also contributing to this report were Sara Daleski, 
Deirdre Duffy, Rob Gebhart, and Rebecca Shea. 

[End of section] 

Footnotes: 

[1] Weaknesses in the SES pipeline identified by the Partnership for 
Public Service included (1) lack of a government-wide cohesive talent 
strategy, (2) insufficient attention to identifying, developing, 
recruiting, and selecting talent for the SES by senior agency leaders, 
and (3) a closed process that doesn't seek talent from outside the 
agency. Partnership for Public Service, Building the Leadership Bench, 
Developing a Talent Pipeline for the Senior Executive Service 
(Washington, D.C.: July 2013). 

[2] For purposes of this review, we define external providers as any 
organization (including educational institutions, professional 
societies, consulting firms, or other federal agencies) that provides 
training and is separate from the federal department or agency that 
employs the trainee. The external training providers are contracted to 
develop training courses or programs, provide the courses or programs, 
or do both. 

[3] We sent a questionnaire to all 27 members of the CHCO Council and 
received responses from 26 of the 27 members of the Council. The 
Department of Education declined to submit questionnaire responses 
because, according to an Education official, the agency had no formal 
executive training programs, continuing professional development or 
curriculum requirements for either SES candidates or career SES during 
our study time period. SES candidates are GS-15, GS-14 (and equivalent 
employees) who are enrolled in an OPM-approved SES Candidate 
Development Program or an agency-funded program that will train and 
develop employees for the SES or an equivalent. We selected FY 2008 as 
a starting point for a five-year analysis of cost data in part because 
of the lack of the availability of data prior to that year. For a 
complete list of CHCO Council members, see the full discussion of 
scope and methodology in appendix I. 

[4] See GAO, Federal Training Investments: Office of Personnel 
Management and Agencies Can Do More to Ensure Cost-Effective 
Decisions, [hyperlink, http://www.gao.gov/products/GAO-12-878] 
(Washington, D.C.: Sept. 17, 2012). 

[5] 5 C.F.R. § 410.701. 

[6] Statement of Federal Financial Accounting Standards 4, Managerial 
Cost Accounting Standards and Concepts establishes standards for 
managerial cost accounting at federal entities. The standards require 
federal entities to accumulate the costs of activities on a regular 
basis, using either costs accounting systems or cost finding 
techniques. 

[7] Includes costs for tuition, lodging, meals, and course materials. 

[8] Includes costs for tuition, lodging, most meals, books, supplies, 
and outside co-curricular events. 

[9] Includes costs for tuition, most meals and transportation during 
classroom training, classroom materials, use of Center for Creative 
Leadership campus facility and equipment, pre-program and post-program 
assessments, and pre-program and post-program executive coaching 
sessions. 

[10] Includes costs for tuition, lodging, all meals, and course 
materials. 

[11] A learning management system is a software application that 
automates the administration, tracking, and reporting of training 
events. 

[12] [hyperlink, http://www.gao.gov/products/GAO-12-878]. 

[13] 5 U.S.C. § 4103(c) requires such evaluation on a regular basis 
whereas OPM's implementing regulations specify that such evaluations 
are to be conducted annually. 5 C.F.R. § 410.202. 

[14] GAO, Human Capital: A Guide for Assessing Strategic Training and 
Development Efforts in the Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: Mar. 1, 
2004). 

[15] [hyperlink, http://www.gao.gov/products/GAO-12-878]. 

[16] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[17] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[18] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[19] [hyperlink, http://www.gao.gov/products/GAO-12-878]. 

[20] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[21] [hyperlink, http://www.gao.gov/products/GAO-12-878]. 

[22] [hyperlink, http://www.gao.gov/products/GAO-12-878]. 

[23] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[24] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[25] "Meta-leadership" can be defined as overarching leadership that 
intentionally connects the purposes and work of different 
organizations or organizational units. See Marcus, L.J, B.C. Dorn, and 
J.M. Henderson, "Meta-Leadership and National Emergency Preparedness: 
A Model to Build Government Connectivity." Biosecurity and 
Bioterrorism: Biodefense Strategy, Practice, and Science, vol. 4, no. 
2 (2006): 128-34. 

[26] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[27] We did not independently verify the reported costs savings 
reported by OPM. 

[28] [hyperlink, http://www.gao.gov/products/GAO-04-546G]. 

[29] See [hyperlink, http://www.gao.gov/products/GAO-12-878]. We 
previously recommended that the Director of OPM, in collaboration with 
the CHCO and CLO Councils, identify the best existing courses that 
fulfill government-wide training requirements, such as mandatory Equal 
Employment Opportunity training, or training in common federal 
occupations, such as basic training in financial management, and offer 
them to all agencies through HR University (or other appropriate 
platforms) in order to reduce costly and duplicative federal training 
investments. 

[30] The 27 CHCO Council agencies include the Departments of 
Agriculture, Commerce, Defense, Education, Energy, Health and Human 
Services, Homeland Security, Housing and Urban Development , Interior, 
Justice, Labor, State, Treasury, Transportation, and Veterans Affairs; 
the Environmental Protection Agency, General Services Administration, 
National Aeronautics and Space Administration (NASA), National Science 
Foundation, Nuclear Regulatory Commission (NRC), Office of Management 
and Budget, Office of Personnel Management, Office of the Director of 
National Intelligence, Small Agency Council, Small Business 
Administration, Social Security Administration, and U.S. Agency for 
International Development. The Federal Labor Relations Authority 
(FLRA) received a questionnaire instead of the Small Agency Council 
(SAC), as FLRA's CHCO represented SAC on the CHCO Council during the 
course of our work. 

[31] We selected fiscal year 2008 as a starting point for a five-year 
analysis because of the lack of availability of data prior to that 
year. 

[32] One agency - the Department of Education - declined to submit 
questionnaire responses because, according to an Education official, 
the agency had no formal executive training programs, continuing 
professional development, or curriculum requirements for either SES 
candidates or career SES during our study time period. 

[33] See GAO, Federal Training Investments: Office of Personnel 
Management and Agencies Can Do More to Ensure Cost-Effective 
Decisions, [hyperlink, http://www.gao.gov/products/GAO-12-878] 
(Washington, D.C.: Sept. 17, 2012), and Human Capital: A Guide for 
Assessing Strategic Training and Development Efforts in the Federal 
Government, [hyperlink, http://www.gao.gov/products/GAO-04-546G] 
(Washington, D.C.: Mar. 1, 2004). 

[34] We submitted questions to three agencies-Energy, Treasury and 
VA - because we selected them as illustrative examples for further 
review. We selected seven additional agencies based on agency size and 
total amount spent on executive training, including NASA, NRC, OPM, 
and the Departments of Commerce, Defense, Justice, and Transportation. 

[35] These five external providers include the Brookings Institution; 
the Center for Creative Leadership; the Harvard Kennedy School of 
Government; the Office of Personnel Management Federal Executive 
Institute; and the VA Leading EDGE program. We also interviewed 
officials from American University, another external provider of 
executive training. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO's commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO's actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO's 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]