GAO-04-291, Human Capital: Selected Agencies' Experiences and Lessons Learned in Designing Training and Development Programs


This is the accessible text file for GAO report number GAO-04-291 
entitled 'Human Capital: Selected Agencies' Experiences and Lessons 
Learned in Designing Training and Development Programs' which was 
released on January 30, 2004.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Chairman, Subcommittee on Oversight of Government 
Management, the Federal Workforce, and the District of Columbia, 
Committee on Governmental Affairs, U.S. Senate:

January 2004:

HUMAN CAPITAL:

Selected Agencies' Experiences and Lessons Learned in Designing 
Training and Development Programs:

GAO-04-291:

GAO Highlights:

Highlights of GAO-04-291, a report to the Chairman, Subcommittee on 
Oversight of Government Management, the Federal Workforce, and the 
District of Columbia, Committee on Governmental Affairs, U.S. Senate 

Why GAO Did This Study:

Effective training and development programs are an integral part of a 
learning environment, helping improve federal workforce performance in 
achieving agency results. Therefore, in this report GAO was asked to 
identify examples of selected federal agencies’ experiences and some 
of the key lessons they have learned in designing their training and 
development programs. This work focused on ways that these agencies 
(1) assessed agency skills gaps and identified training needs, (2) 
developed strategies and solutions for these training and development 
needs, and (3) determined methods to evaluate the effectiveness of 
training and development programs. 

GAO worked with five agencies to identify their experiences and 
lessons learned: the U.S. Army Corps of Engineers (USACE), Department 
of Defense; Fish and Wildlife Service (FWS), Department of the 
Interior (Interior); Internal Revenue Service (IRS), Department of the 
Treasury; the Office of Personnel Management (OPM); and Veterans 
Health Administration (VHA), Department of Veterans Affairs (VA). 
Agency officials provided information during interviews and furnished 
supporting documentation for analysis and review. 

What GAO Found:

GAO identified important lessons learned from five federal agencies’ 
experiences in designing training and development programs for their 
employees that could be useful to other agencies facing similar 
challenges. These lessons learned are related to the following three 
areas.

Assessing Agency Skill Requirements and Identifying Training Needs: 

The agencies used a variety of approaches to assess current and future 
skill requirements, such as implementing workforce planning models and 
conducting knowledge and skills inventories. Generally, the agencies
 are transitioning to more comprehensive approaches. One of the 
lessons learned was to involve stakeholders and benchmark with others 
to identify critical skills and competencies and related training 
needs. For example, IRS officials believed they needed a leadership 
competency model directly based on the work of their agency’s business 
units. To develop a comprehensive model, they interviewed top IRS 
leaders and benchmarked with leading practices in the public and 
private sector. 

Developing Strategies and Solutions for Training Needs: 

The agencies considered a mixture of delivery mechanisms, as well as 
potential sources for training and development opportunities. However, 
projecting costs and benefits of proposed training and development 
programs presented challenges for them. The agencies usually developed 
broad information on anticipated benefits and expected costs of 
potential investments, although often without tying benefits to 
specific performance improvements or considering all costs. One of the 
lessons learned was to establish mechanisms to avoid duplication or 
inconsistencies. Education Service Representatives in each regional 
VHA network, for example, coordinate training and development programs 
with headquarters—sharing information about successful practices and 
identifying areas where coordination is needed. 

Determining Methods for Evaluating Training Programs: 

Overall, the agencies relied primarily on participants’ end-of-course 
evaluations, but they are beginning to use more comprehensive 
evaluation approaches, including limited use of return-on-investment 
analysis techniques. One of the lessons learned is to plan for the use 
of multiple data types and sources in order to attain a balanced 
assessment once the course is implemented. For example, USACE’s 
training center incorporated pre- and post-tests on over 90 percent of 
its courses, as well as approaches to collect participants’ and course 
managers’ feedback, as part of the design.

Four of the five agencies provided comments on a draft of this report. 
Interior and VA said that they generally agreed with the report’s 
findings regarding their respective agencies. IRS and OPM said that 
they appreciated the opportunity to be included in the report and to 
share information on training activities. USACE provided no comments 
on the draft report.

What GAO Recommends:

www.gao.gov/cgi-bin/getrpt?GAO-04-291.

To view the full product, including the scope and methodology, click 
on the link above. For more information, contact George Stalcup at 
(202) 512-6806 or stalcupg@gao.gov.

[End of section]

Contents:

Letter: 

Results in Brief: 

Background: 

Agencies Used Varied Approaches in Assessing Skills and Competencies 
and Identifying Related Training Needs: 

Agencies Developed Strategies and Solutions for Their Training Needs: 

Agencies are Considering More Sophisticated Evaluation Approaches As 
Part of Designing their Training and Development Programs: 

Conclusions and Observations: 

Agency Comments and Our Evaluation: 

Appendixes:

Appendix I: Objective, Scope and Methodology: 

Appendix II: Background on Selected Agencies and Their Training and 
Development Functions: 

Appendix III: Core Characteristics of a Strategic Training and 
Development Process: 

Appendix IV: Comments from the Department of the Interior: 

GAO Comment: 

Appendix V: Comments from the Internal Revenue Service: 

Appendix VI: Comments from the Office of Personnel Management: 

Appendix VII: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Acknowledgments: 

Figures: 

Figure 1: Four Components of a Strategic Training and Development 
Process: 

Figure 2: Strategic Workforce Planning Process: 

Figure 3: OPM's Five-step Workforce Planning Process: 

Figure 4: IRS's Core Management Responsibilities and Leadership 
Competencies: 

Figure 5: Competencies in VHA's High Performance Development Model: 

Figure 6: Steps for Developing Strategies and Solutions for Training 
and Development Needs: 

Figure 7: Steps in Determining Methods for Evaluating Training 
Programs: 

Letter January 30, 2004:

The Honorable George V. Voinovich: Chairman: 
Subcommittee on Oversight of Government Management, the Federal 
Workforce, and the District of Columbia: 
Committee on Governmental Affairs: 
United States Senate:

Dear Mr. Chairman:

To effectively address the nation's most urgent priorities and take 
advantage of emerging opportunities, federal agencies need to continue 
to build their fundamental capabilities to carry our their work in an 
environment that is increasingly complex and rapidly changing. To build 
their capacity, agencies should invest resources wisely to ensure that 
their employees have the information, skills, and competencies they 
need to succeed. As you are well aware, these investments must include 
training and development efforts to continuously enhance the skills and 
competencies of the federal workforce and improve the quality of 
agencies' results.

As agreed with your office, this report provides information on 
selected federal agencies' experiences and lessons learned in key 
aspects of designing training and development programs for their 
employees. Specifically, we focused on the agencies' experiences and 
lessons learned related to:

* assessing current and future agency skill and competency requirements 
and identifying related training and development needs,

* developing strategies and solutions for training and development 
needs, and:

* determining methods to evaluate the effectiveness of training and 
development programs.

For this review, lessons learned were defined as knowledge that could 
be applied in the future that the agencies gained through either 
positive or negative experiences. The experiences and lessons learned 
from the five agencies we reviewed may well provide some valuable ideas 
and useful approaches that could be adopted by other federal agencies 
as they attempt to address ongoing training and development challenges-
-particularly those related to the elements within the training process 
that relate to design and development.

To address these issues and as agreed with your office, we focused our 
review on five federal agencies: U.S. Army Corps of Engineers (USACE), 
Department of Defense; Fish and Wildlife Service (FWS), Department of 
the Interior (Interior); Internal Revenue Service (IRS), Department of 
the Treasury; Office of Personnel Management (OPM); and Veterans Health 
Administration (VHA), Department of Veterans Affairs (VA). At the 
headquarters of the five agencies--and in some field locations--we 
collected and reviewed documents on the agencies' training and 
development efforts and interviewed officials from the agencies' human 
capital and training organizations, as well as assorted program 
offices. We selected the five agencies for various reasons, including 
your office's interests, the diversity of employee occupations within 
the agency, and reported innovative approaches for training and 
developing their employees. This agency selection process was not 
designed to identify examples that could be considered representative 
of all training and development efforts at the five agencies or the 
federal government as a whole. Furthermore, in citing examples that 
relate to the lessons learned on the design of training, we did not 
assess the effectiveness of these training programs and practices. 
Rather, we attempted to highlight some of the experiences and lessons 
that the agencies found helped them move forward in improving their 
training and development programs. Federal agencies' training and 
development strategies, and how they are designed to operate in 
conjunction with other strategies to improve individual and 
organizational performance, continue to change and evolve.

We conducted our review from August 2002 through November 2003 in 
accordance with generally accepted government auditing standards. See 
appendix I for additional information on our objective, scope, and 
methodology.

Results in Brief:

Officials from the five agencies in our review recognized the 
importance of assessing current and future agency skill and competency 
requirements to identify related workforce training needs. The agencies 
used several approaches to design training and development programs 
that focused on the skills and competencies their assessments indicated 
needed enhanced attention. The officials emphasized that agencies are 
transitioning to more formal and comprehensive planning approaches to 
assess skill and competency requirements and identify related training 
and development needs--primarily as part of broader efforts to 
incorporate workforce planning into ongoing strategic planning and 
budgeting processes focused on achieving results. The following are 
lessons learned identified during this review related to assessing 
skill and competency requirements and identifying training needs.

* Involve key stakeholders and benchmark with other organizations when 
identifying skills and competencies to help ensure that training and 
development programs are aligned with current and emerging needs and 
business practices.

* Analyze existing agency data on employees' skills and competencies 
and information from performance appraisals to help identify skills and 
competencies that need to be addressed throughout the agency as well as 
on an individual basis.

* Link the agency's workforce planning efforts with training needs 
assessments to ensure consistency and enhance strategic alignment.

* Consider the training needs of staff from other organizations that 
will likely use the agency's training programs or facilities to 
effectively leverage training investments and meet diverse needs.

These agencies developed a wide range of strategies and solutions to 
improve performance through designing training and development programs 
for their employees. Officials told us they considered a mixture of 
both on-the-job and other developmental programs and contemplated an 
assortment of mechanisms for delivering the training as well as 
potential sources for the learning need. Agency officials have found 
projecting costs and benefits of proposed training and development 
programs to be very challenging. Although they sometimes developed 
broad information on anticipated benefits and expected costs, this 
often did not involve tying anticipated benefits to specific 
performance improvements or considering all related costs. Our review 
identified the following lessons learned in developing strategies and 
solutions when designing agency training and development programs.

* Incorporate information on employees' various competency levels and 
job needs into the design of training and development programs to 
increase their relevancy and timeliness.

* Assess options for using other organizations' course content, staff, 
services, or facilities when designing a new training and development 
program in order to develop efficient and cost-effective strategies.

* Establish mechanisms and controls to avoid unnecessary duplication or 
inconsistency within and across agencies' training efforts.

* Develop and use criteria for determining the optimal mix of delivery 
mechanisms to use in order to select the most effective approaches 
given each learning situation.

* Ensure that employees have the needed equipment and technologies so 
that they can take maximum advantage of learning opportunities.

* Plan early when developing integrated solutions that complement other 
planned and ongoing strategies to improve performance so that when 
implemented the strategies work effectively and are aligned to help 
achieve agency goals.

* Plan for the direct participation of senior agency leaders and 
experienced staff in the delivery of training and development programs 
to increase buy-in and build support for organizational change.

Evaluating training programs is key to ensuring that training and 
development programs are effective. Overall, the five agencies relied 
primarily on standard end-of-course evaluations to obtain the 
participants' reaction to, and satisfaction with, a specific training 
course or learning opportunity. However, officials said that they have 
begun or are planning to use more comprehensive and sophisticated 
techniques for assessing the extent to which training and development 
programs increased employees' knowledge and skills and enhanced 
individual and organizational performance. These techniques included 
pre-and post-testing, tracking changes in individual and program 
performance, and some limited use of return-on-investment (ROI) 
analyses. The lessons agencies learned in designing methods to evaluate 
training and development programs included the following.

* Incorporate appropriate aspects of the evaluation approach when 
designing training and development programs by specifying what results 
are expected to better ensure the availability and use of quality 
performance data.

* Consider new approaches for collecting and analyzing performance data 
with the aim of increasing the quality and quantity of training 
evaluation feedback.

* Plan for the use of multiple data types and sources to provide a 
balanced approach in assessing the effectiveness of training and 
development programs.

* Take into account all relevant factors for determining the costs of a 
training and development program to better ascertain whether it is 
cost-effective in relation to benefits achieved.

Four of the five selected agencies provided comments on a draft of this 
report. Interior and VA said that they generally agreed with the 
report's findings relating to their respective agencies. IRS said that 
it was honored to share some of its lessons learned with us for 
governmentwide dissemination. OPM said that it appreciated the 
opportunity to be included in the report and to share information on 
its training and development activities and programs. USACE informed us 
that it had no comments on our draft report.

Background:

We recently issued an exposure draft of an assessment guide that 
introduces a framework for evaluating a federal agency's training and 
development efforts.[Footnote 1] This assessment guide consists of a 
set of principles and key questions that federal agencies can use to 
ensure that their training and development investments are targeted 
strategically and are not wasted on efforts that are irrelevant, 
duplicative, or ineffective. As detailed in our assessment guide, the 
training and development process can loosely be segmented into four 
broad, interrelated components: (1) planning/front-end analysis, (2) 
design/development, (3) implementation, and (4) evaluation. Figure 1 
depicts an overview of this process along with the general 
relationships between the four components that help to produce a 
strategic approach to federal agencies' training and development 
efforts. Although these components can be discussed separately, they 
are not mutually exclusive and encompass subcomponents that may blend 
with one another. For instance, evaluation is an integral part of the 
planning/front-end analysis as agencies strive to reach agreement up 
front on how the success of various strategies to improve performance, 
including training and development efforts, will be assessed. As noted 
in the assessment guide, agencies can build on lessons learned and 
performance data and feedback from previous experiences. This report 
can provide a starting point for agencies to use to build on the 
experiences and lessons learned by the five agencies we reviewed as 
part of their efforts to design and develop training and development 
programs. (See app. II for a description of the five agencies included 
in this study.):

Figure 1: Four Components of a Strategic Training and Development 
Process:

[See PDF for image]

[End of figure]

Our assessment guide also summarizes our observations on the core 
characteristics that make a training and development process effective 
and strategically focused on achieving results. These eight core 
characteristics are described in more detail in appendix III, and 
include:

* strategic alignment,

* leadership commitment and communication,

* stakeholder involvement,

* accountability and recognition,

* effective resource allocation,

* partnerships and learning from others,

* data quality assurance, and:

* continuous performance improvement.

A concerted effort to integrate these core characteristics can further 
an agency's efforts to continually improve its training and development 
process.

Agencies Used Varied Approaches in Assessing Skills and Competencies 
and Identifying Related Training Needs:

Federal agencies face diverse challenges in their efforts to identify 
and measure the skills and competencies that their employees must 
possess to support missions and goals. Officials from the five agencies 
in our review recognized the importance of assessing the need for 
specific skills and competencies now and in the future in order to 
identify related workforce training needs. These agencies generally 
focused on the desired performance of the agency and its employees, 
determined the difference between the desired and actual skill levels, 
and attempted to identify the key factors contributing to performance, 
including the need for enhanced workforce competencies. Officials used 
a variety of approaches and tools to assist in determining the human 
capital skills and competencies that are critical to achieving their 
long-term goals.

An agency's ultimate goal in undertaking training and development 
efforts is, of course, to optimize employee and organizational 
performance. To help ensure that each training program is linked to 
improving individual and agency performance, agencies first need to 
analyze their strategic and performance goals so that they can 
determine where training and development can most effectively enhance 
goal achievement.

Organizations can evaluate the extent to which human capital approaches 
support their accomplishment of current, emerging, and future strategic 
goals through the use of workforce planning. Workforce planning focuses 
on determining the skills and competencies needed now and in the future 
to meet the agency's goals; identifying the current and projected level 
of the skills and competencies of the workforce; and crafting 
strategies for acquiring, developing, and retaining people to address 
any identified needs. These needs include the knowledge, skills, and 
abilities needed for the agency to pursue its current and future 
mission as well as the size of the workforce and its deployment across 
the organization. After identifying the skills and competencies that 
employees need now or in the future, agencies must tackle the challenge 
of determining what combination of strategies to use, such as hiring 
new employees with needed skills and competencies, relying on 
outsourcing, and/or enhancing employees' skills and competencies 
through training and development. While agencies' approaches to 
workforce planning will vary, we have identified the need for a 
strategic workforce planning process to ensure that each agency's human 
capital program capitalizes on its workforce's strengths and addresses 
related challenges in a manner that is clearly linked to achieving the 
agency's missions and goals.[Footnote 2] Figure 2 presents a model of 
this strategic workforce planning process.

Figure 2: Strategic Workforce Planning Process:

[See PDF for image]

[End of figure]

The focus in this report is on the workforce strategies that involve 
the design of training and development programs.

Agencies' Experiences in Assessing Skill and Competency Requirements 
and Identifying Training Needs:

The five agencies used several approaches to design training and 
development programs that focused on skills and competencies their 
assessments indicated needed enhanced attention. One common approach 
that officials used to help identify training needs was interviewing or 
surveying managers, supervisors, and employees. Agencies also 
established councils and held conferences, made comparisons with 
leading organizations through benchmarking, and analyzed workforce data 
and trends.

OPM's 2001 skills assessment, for example, relied on a survey of agency 
managers and supervisors.[Footnote 3] Officials used the survey results 
to identify the most important occupational competencies needed to 
achieve OPM's mission, the level at which employees possessed those 
competencies, and the level at which they would be needed in the 
future. Their analysis identified minor gaps in the level of 
competencies needed for both current and future work in mission 
critical occupations. In addition, the analysis pointed out more 
serious developmental needs for OPM's retirement and insurance benefits 
specialists. These needs were related to the changing role of these 
specialists, who increasingly need to work more closely with clients in 
responding to complex issues.

Also in 2001, IRS established a workforce planning council consisting 
of senior management representatives from each of the agency's 
operating divisions. IRS officials told us that this council has become 
the primary vehicle for communicating workforce planning information 
among IRS's four operating divisions. At FWS, the human resources 
office hosted a 3-day workforce planning conference in 2002 to draw on 
the experience and expertise of agency personnel in identifying 
critical workforce issues for the agency for the next 3 to 5 years. 
Managers and program experts representing all eight major FWS program 
offices, seven field regions, and headquarters offices participated.

Agencies also compared their performance and needed skills and 
competencies with leading organizations through benchmarking. In August 
2000, VHA commissioned an internal task force charged with developing a 
well-defined, comprehensive succession plan for the agency. The ideas 
garnered from benchmarking led to VHA establishing an expectation for 
agency leaders to help identify and train their successors. The task 
force's December 2001 report presented a comprehensive succession plan 
for VHA, and implementing a comprehensive leadership development 
program was one of the six major components of this plan.

Agency officials also analyzed workforce data to assess skills and help 
identify training needs. Generally, they collected information on 
employee demographics and retirement eligibility and used these data to 
project attrition and retirement rates. OPM officials, for example, 
collected and analyzed attrition and turnover data on the agency's 
senior executives along with the distribution of current executives by 
unit and projected retirements through 2010.[Footnote 4] They also 
collected and analyzed data on hires, separations, and workforce 
diversity across the agency. As part of this analysis, officials 
assessed the agency's use of contractors and considered how sourcing 
alternatives could affect OPM's plans for hiring, training, and 
development.

Officials from the five agencies told us that they used a wide range of 
resources and tools to assess skills and competencies as part of 
identifying and designing needed training and development programs. 
They used workforce planning models; assessed the workforce in view of 
organizational, occupational, and unit-based competency standards; 
conducted knowledge and skills inventories; and evaluated job 
performance appraisals and information from individual development 
plans (IDP).[Footnote 5] To identify needed executive competencies, for 
example, OPM used a five-step workforce planning model that it had 
developed in its role of providing human capital tools for use by other 
federal agencies (see fig. 3). The resulting analysis called for 
enhancing leadership development within OPM.

Figure 3: OPM's Five-step Workforce Planning Process:

1. Set strategic direction to drive agency operations and define how 
the agency will know when and if it has succeeded; 

2. Analyze the workforce, identify skill gaps, and conduct workforce 
analysis; 

3. Develop a workforce action plan that lays out specific tasks 
and actions the agency needs to take in order to achieve the agency's 
human resources goals and objectives; 

4. Implement the workforce action plan by executing the schedule that 
includes measurable workforce goals and milestones; 

5. Monitor progress, evaluate success, and revise plan as needed.

Source: OPM.

[End of figure]

Officials said that their agencies have transitioned, or are in the 
process of transitioning, to more comprehensive, consistent planning 
approaches. This transition is coming about as agencies attempt to 
institutionalize their workforce planning efforts as part of their 
ongoing strategic planning and budgeting processes. In 2002, FWS 
conducted its first formal, agencywide workforce planning process. 
Although FWS initially employed a contractor to help develop a 
permanent workforce planning process, it plans to continue to manage it 
in-house. This workforce planning process is to be implemented on a 2-
year cycle that is integrated with the agency's strategic planning and 
budgeting processes.

Agencies' Lessons Learned in Assessing Skill and Competency 
Requirements and Identifying Training Needs:

By considering the viewpoints of a range of stakeholders and candidly 
and openly assessing progress toward meeting their goals, agencies can 
help ensure that their strategic and annual performance planning 
processes adequately reflect current ideas, policies, and practices in 
the field. Agencies continue to integrate workforce planning into these 
other planning processes. It is important to note that a wide variety 
of strategies other than training and development are also available to 
agency leaders as they attempt to transform their cultures and 
operations.[Footnote 6] Training and development is not always the best 
solution--reengineering processes or other actions may be needed to 
build an environment that effectively supports performance. In 
addition, training and development strategies frequently need to be 
implemented in conjunction with other initiatives, given that the day-
to-day environment and organizational culture may also need to change 
to enable employees to successfully use new skills or competencies on 
the job.

Our review identified four lessons learned by the agencies related to 
assessing skills and competencies to identify, focus, and prioritize 
training needs.

Lesson learned: Involve key stakeholders and benchmark with other 
organizations when identifying skills and competencies to help ensure 
that training and development programs are aligned with current and 
emerging needs and business practices.

Organizations in the private and public sectors have increasingly 
turned to developing competency models that outline behaviorally 
defined skills and competencies employees should possess and that can 
be tied directly to training and development plans and programs. We 
have found that an effective performance management system uses 
competencies to provide a fuller assessment of performance.[Footnote 7]

IRS involved key stakeholders and benchmarked with other organizations 
in developing its leadership competency model:

In working to build its leadership development program in the wake of 
the IRS Restructuring and Reform Act of 1998, IRS officials believed 
they needed a leadership competency model that was based directly on 
the work of IRS's business units. To identify the essential 
characteristics that enable IRS employees to function as effective 
leaders in the newly modernized agency, human capital specialists at 
IRS conducted behavioral interviews with 35 top IRS leaders in 1999, 
asking them to identify major successes and challenges during their 
careers. Using information gathered from these interviews, officials 
identified core management responsibilities and corresponding 
competencies required for leaders in IRS. With the assistance of a 
contractor, IRS validated the leadership competency model by comparing 
it against leading practices in the public and private sector and 
linking it to the mission and goals of the agency.[Footnote 8] The 
resulting competency model now forms the basis for IRS's leadership 
development efforts, as well as how IRS selects, evaluates, and 
recognizes its leaders. Figure 4 shows a listing of IRS's five core 
management responsibilities and the 21 corresponding leadership 
competencies.

Figure 4: IRS's Core Management Responsibilities and Leadership 
Competencies:

Leadership: 

* Adaptability; 
* Communication; 
* Decisiveness; 
* Integrity/honesty; 
* Service motivation; 
* Strategic thinking; 

Employee satisfaction: 

* Continual learning; 
* Developing others; 
* Diversity awareness; 
* Group leadership; 
* Teamwork; 

Customer satisfaction: 

* Customer focus; 
* Entrepreneurship; 
* External awareness; 
* Influencing/ negotiating; 
* Partnering; 

Business results: 
* Achievement orientation; 
* Business acumen; 
* Political savvy; 
* Problem solving; 
* Technical credibility; 

Equal employment opportunity (EEO) and diversity: 
* Supporting competencies[A].

Source: IRS.

[A] Italicized competencies support the "EEO and diversity" 
responsibility.

[End of figure]

VHA benchmarked with other organizations and used a pilot test in 
developing its high performance development model:

VHA conducted an extensive literature search and benchmarked with 
several leading private sector firms (including Bell South, Coca-Cola, 
and Motorola) to provide a foundation for its effort to create a new 
competency model for VHA employees. VHA used this information in 
developing its high performance development model, which was 
implemented throughout the department in 2002. This model consists of 
eight core competencies and related performance tools that represent 
the major skills and competencies that employees need to fulfill VHA's 
mission. The model was designed to serve as a framework for identifying 
and developing future leaders, as well as to enhance development of 
VHA's entire workforce. VHA said that using this model helped more 
effectively align training and development programs with agency 
priorities. According to VHA, the fact that the core competencies apply 
to all levels and functions within the agency helps ensure alignment 
within and between organizational units and is a key component in 
motivating sustained and improved performance. VHA also uses the model 
on an individual employee basis as a process for identifying specific 
developmental needs. Figure 5 lists the eight core competencies in 
VHA's high performance development model.

Figure 5: Competencies in VHA's High Performance Development Model:

* Personal mastery; 
* Interpersonal effectiveness; 
* Flexibility and adaptability; 
* Systems thinking; 
* Technical skills; 
* Customer service; 
* Creative thinking; 
* Organizational stewardship.

Source: VHA.

[End of figure]

Lesson learned: Analyze existing agency data on individual employee's 
skills and competencies and information from performance appraisals to 
help identify skills and competencies that need to be addressed 
throughout the agency as well as on an individual basis.

To obtain a unit-or agencywide perspective of skills and competencies, 
some agencies such as USACE and IRS have explored new ways of 
aggregating data from tools that are primarily focused on individual 
employees, such as IDPs, performance assessments, and 360-degree 
feedback instruments.[Footnote 9] Officials from these agencies told us 
that this information helped them discern a clearer picture of the 
overall strengths and weaknesses of their employees and offered 
direction in planning and designing training and development programs 
to help focus efforts to enhance skills and competencies throughout the 
agency.

USACE's automated training management program provides a Web-enabled 
integrated database:

According to USACE officials, using an automated training management 
program has allowed managers to identify divisionwide gaps in workforce 
skills and competencies. Using this system (currently in four of 
USACE's eight divisions) employees prepare an IDP assessing their 
knowledge, skills, and abilities in relation to a series of mission 
essential tasks. With supervisory guidance, each task is identified as 
critical, important, or beneficial and employees indicate whether they 
have received adequate, partial, or no training in that area. With this 
assessment as a guide, the supervisor and employee can consult the 
system's built-in course catalog to select internal or external 
training to enhance the employee's development. In addition, the system 
also has the capability of aggregating data. USACE officials said that 
this capability provides a simple method for division managers to 
obtain a picture of the level of skills and competencies in their 
workforce. This information informs decisions on training priorities 
and helps managers determine the most efficient use of available 
resources.

IRS aggregated data from 360-degree feedback instruments to help 
identify training needs:

To assess the progress and developmental needs of leaders within IRS, 
the agency's leadership development office recently aggregated and 
analyzed multiyear data from the 360-degree performance assessments of 
IRS managers. This analysis helped to show areas of strength and 
weakness in skills and competencies across the agency's managerial 
ranks. The director of leadership and organizational effectiveness at 
IRS said that, in the past, the agency did not sufficiently assist 
managers in effectively using 360-degree feedback they received. 
However, he said that IRS now emphasizes the importance of using 360-
degree feedback data, both on an organizational and individual basis, 
to focus on strengths in developing key leadership competencies. IRS's 
Extraordinary Leader Program involves designing unique developmental 
approaches to help managers become more effective leaders. Using 
results of the manager's 360-degree assessment, IRS creates a 
customized leadership development program focusing first on correcting 
any "fatal flaw" weaknesses and then building on the manager's 
demonstrated strengths in areas that IRS has identified as key to 
providing effective leadership within its organizational culture and 
operating environment.

Lesson learned: Link the agency's workforce planning efforts with 
training needs assessments to ensure consistency and enhance strategic 
alignment.

The agencies' training and development organizations had a range of 
responsibilities, including designing training and development 
programs based on strategic initiatives, soliciting input from 
stakeholders, and prioritizing and scheduling training based on 
strategic initiatives and stakeholder input. Generally, the training 
organization and agency stakeholders can work together more effectively 
when they better understand how each office or function within the 
agency contributes to achieving business goals. In some cases, this 
included efforts to link training needs assessments with the agency's 
overall workforce planning efforts. Officials told us that this linkage 
helped ensure that workforce plans developed by the agency's human 
capital office were consistent with training needs assessments done by 
the agency's training and development organization.

FWS involved key internal stakeholders in its planning processes:

When assessing workforce skills and competencies, FWS officials worked 
to ensure that the agency's workforce plan was linked with a training 
needs assessment done by its training center. The workforce planning 
effort identified broad competencies needed across the agency's 
workforce while the training needs assessment identified the types of 
training courses to develop skills and competencies within agency units 
and occupations. Officials said that they viewed the training needs 
assessment as a tool that was useful in refining the agency's workforce 
plan and in prioritizing and budgeting for the development and delivery 
of training. Both efforts involved key stakeholders from the human 
capital and training offices as well as other FWS units. In the future, 
officials said they will rely on the results of the agency's workforce 
planning efforts to directly serve as the agency's training needs 
assessment.

USACE relies on its Learning Advisory Board and automated training 
management program to effectively link planning efforts:

USACE relies chiefly on the coordination activities of its Learning 
Advisory Board to ensure its workforce planning efforts and training 
needs assessments are effectively linked. USACE in 2001 formed the 
Learning Advisory Board, comprised of senior managers from across the 
agency, to review the adequacy of USACE's training and development and 
ensure that training is properly aligned with the agency's missions, 
goals, and plans. In addition, the four divisions that use the 
automated training management program can also rely on data from that 
system to assess training needs. This system allows managers to compare 
information on individuals' skills and competencies with workforce 
planning results from within the division and across the agency. 
According to USACE officials, this systematic comparison more closely 
links workforce planning and training needs assessments to the 
essential mission-related operations.

Lesson learned: Consider the training needs of staff from other 
organizations that will likely use the agency's training programs or 
facilities to effectively leverage training investments and meet 
diverse needs.

When planning and designing training programs for its employees, FWS 
and USACE officials told us that they gained insight into the 
assessment of potential training solutions by considering the possible 
involvement of trainees from other organizations. In some instances, 
for example, it would not have been cost-effective to design, develop, 
and deliver a training effort for a small number of employees or 
occupations. However, officials' determinations that other agencies or 
organizations also needed similar training provided the critical mass 
needed to move forward. Officials said they found that partnering with 
other organizations helped make training efforts more cost effective to 
design, develop, and deliver.

FWS's training center assessed training needs of possible participants 
from other Interior components:

Officials from FWS's training center said that although they focus 
primarily on meeting the needs of FWS employees when planning and 
designing training, they also look at the training needs of other 
organizations, particularly other agencies within Interior. To aid in 
communication and coordination, agencies within the department 
designate employees to serve as liaisons between each of the Interior 
agencies and FWS's training center. These liaisons facilitate efforts 
to incorporate their agency's needs into the training center's plans 
and the design and delivery of training and development programs. 
According to agency officials, these liaisons help ensure that the 
center's courses remain current because they facilitate a dialogue 
between the various agencies. For example, a recent FWS course on 
wetland plant identification involved participants from FWS; U.S. 
Geological Survey, which is another Interior component; Natural 
Resources Conservation Service, an agency of the U.S. Department of 
Agriculture; USACE; and three private sector firms.

USACE's training center considers training needs of staff from other 
federal and state organizations:

Officials from USACE's training center told us that during the agency's 
annual training needs assessment, they assess the possible training 
demand from other organizations' employees in addition to identifying 
workforce development needs of units and offices within USACE. 
According to USACE, approximately 2,500 people each year--about 25 
percent of the participants in the agency's training center programs--
are from other federal and state organizations. Officials noted that 
the training center offers courses needed to obtain certifications for 
certain professional requirements. It offers courses accredited by 
several professional associations, including the National Society for 
Professional Engineers, the American Institute of Architects, and the 
International Association for Continuing Education and Training. USACE 
officials said that they would like to offer training to a greater 
number of employees from private sector firms; however, current law 
requires receipts for services provided to private individuals and 
organizations to be deposited into the general treasury as 
miscellaneous receipts.[Footnote 10] USACE officials said that this 
requirement hampers the ability of USACE's training center to keep 
funds it could generate to further invest in its training programs. 
However, when agencies are required to return receipts for services to 
the Treasury, the Congress is preserving its oversight and control over 
the programs generating the fees.

Agencies Developed Strategies and Solutions for Their Training Needs:

Agencies carry out their training and development efforts on the basis 
of estimated needs, priorities, and available resources and recognize 
that adequate planning allows them to establish priorities and 
determine the best ways to leverage investments to improve performance. 
The five agencies we reviewed set priorities for training and 
development on the basis of various factors, such as the results from 
skill and competency assessments, the availability of resources, and 
the interests of agency leaders. They usually relied on training 
officials, agency managers, and subject matter experts to assist in 
developing strategies and approaches for addressing training needs. 
Although the agencies designed and delivered training using both 
centralized and decentralized approaches, we found that leadership 
development programs were more highly centralized and managed at 
headquarters. Agency officials acknowledged that they found projecting 
costs and benefits of proposed training and development programs to be 
very challenging. Although they sometimes developed broad information 
on anticipated benefits and expected costs, this often did not involve 
tying anticipated benefits to specific performance improvements or 
considering all related costs.

As outlined in figure 6, agencies can plan and establish priorities by 
developing an annual training plan to target developmental areas of 
greatest need and outline the most cost-effective training approaches 
to address those areas. Considerations involved in assessing investment 
opportunities for the training plan include balancing the competing 
demands confronting the agency and the amount of resources available in 
order to determine how those demands can best be met with available 
resources. It is also important to consider how to effectively 
integrate all of the strategies the agency plans to use to improve 
performance and meet emerging demands. When training is identified as a 
solution to improve performance, agencies can compare various training 
strategies by weighing their estimated costs and anticipated benefits 
to build a convincing business case that supports the selected training 
strategy. Developing a business case that sets forth the expected costs 
and benefits of the performance improvement investment provides 
decision makers with essential information for allocating necessary 
resources.

Figure 6: Steps for Developing Strategies and Solutions for Training 
and Development Needs:

[See PDF for image]

[End of figure]

Agencies' Experiences in Developing Strategies and Solutions for 
Training Needs:

The agencies we reviewed used a wide range of strategies and solutions 
to improve performance through designing training and development 
programs for their employees. Officials told us they considered a 
mixture of both on-the-job and other developmental programs, 
contemplated an assortment of mechanisms for delivering the training, 
and assessed potential sources to meet their learning needs. For 
example, USACE's leadership development program for midlevel engineers 
and scientists involved formal classroom training, mentoring, and a 6-
month developmental assignment. VHA employees can access a wide variety 
of informational and educational content through the VA Knowledge 
Network, a satellite-based system of live and on-demand programming 
delivered directly to employees' desktops. IRS's training unit 
developed an automated ROI workbook tool that the agency's business 
units can use to assess whether proposed training programs should be 
delivered in a classroom or by an e-learning approach. OPM partnered 
with an employee union to offer a midcareer development program that 
provided an opportunity for current OPM employees to enhance existing 
skills, explore new career fields, and gain practical experience. 
Officials from FWS's training center cited courses in negotiation, 
communication, and interpersonal skills as examples of vendor-provided 
courses. They said the center decided not to invest in designing these 
courses since they do not require field experience and expertise, which 
are critical in designing other FWS courses.

Projecting costs and benefits of proposed training and development 
programs was a challenge for the five agencies. They usually developed 
broad information on anticipated benefits and expected costs, often 
without tying anticipated benefits to specific performance improvements 
or considering all costs related to the training program. For example, 
VHA officials told us that the agency's assessments of anticipated 
benefits and expected costs of proposed training are generally 
unsystematic. One VHA office or field location may not have analyzed 
the relative costs or benefits of proposed training while another 
office or location may have considered anticipated benefits and 
developed estimates of costs and savings using different training 
approaches. At FWS, some proposed training programs, such as its 
Advanced Leadership Development Program, involved detailed estimates of 
costs, both for the training center and for participating FWS field 
offices, as well as the identification of specific competencies to be 
developed in the program. Other proposed training programs at FWS did 
not have documented and detailed estimates of expected costs and 
benefits. FWS officials said that they assess the anticipated costs and 
benefits of all their proposed training and development programs but 
that the extent of these assessments and the amount of documentation 
supporting the assessments vary, depending on many factors, such as the 
content, delivery mechanism, and uniqueness of the proposed training.

According to agency officials we interviewed, limited funding sometimes 
affected agencies' abilities to design and deliver training and 
development programs that officials believed were needed. USACE 
officials said, for example, that in a recent survey, about two-thirds 
of agency supervisors and one-half of agency executives believed that 
the agency had less funding for training civilian employees than is 
needed. Funding and resource limitations sometimes forced the agencies 
to think of new and practical ways to ensure that their employees had 
the knowledge and skills needed to carry out their work. For example, 
because of an unexpected decrease in available travel funds, VHA 
officials canceled plans for a large national conference on the use on 
an automated managerial cost accounting system. This system was 
designed to provide VHA managers with data important in making clinical 
decisions, managing workload, and controlling medical care costs. 
Instead, to enable employees to obtain the information that would have 
been presented at the conference, VHA officials provided the content 
via a satellite broadcast along with a series of audio conferences.

At FWS, the number of people that the training center can train onsite, 
of course, is limited by the current capacity of the facility 
classrooms and residences. According to FWS officials, incorporating a 
blended learning approach into its curriculum has enabled trainees to 
perform part of the course work outside class, thus allowing 
instructors to focus on those topics that require special attention. 
Training center officials also said that the agency has increased its 
use of e-learning and other mechanisms to develop employees where they 
work instead of coming to the training center. Officials told us that 
FWS has reduced training costs by offering more training online and 
using CD ROMs to provide field offices with course material that 
previously had been offered only on location at the training center.

Agencies' Lessons Learned in Developing Strategies and Solutions for 
Their Training Needs:

Agency officials have encountered a variety of challenges in their 
efforts to design training programs to meet the developmental needs of 
their employees. How agencies respond to these challenges can greatly 
affect their success in aligning priorities with strategic direction, 
assessing strategies, identifying alternative sources and methods, 
weighing potential costs and anticipated benefits, and assessing how 
other performance improvement initiatives might complement training 
efforts. It is also important to consider evaluation feedback on an 
ongoing basis. We identified eight lessons learned related to the five 
agencies' efforts to develop strategies and solutions for their 
training and development needs.

Lesson learned: Incorporate information on employees' various 
competency levels and job needs into the design of training and 
development programs to increase their relevancy and timeliness.

When designing effective training and development programs, the way the 
work is actually to be done on the job and the developmental needs of 
the expected trainees are key considerations. Analyzing the tasks of 
specific jobs and occupations can help ensure that training accurately 
reflects the way employees are expected to perform on a day-to-day 
basis. To help ensure that the training effectively addresses 
employees' developmental needs, agencies can determine the workforce's 
level of proficiency in mission-critical skills and competencies by 
conducting skills assessments and using information obtained through 
interviews or surveys of employees and their supervisors. The increased 
information and insight provided by these approaches can allow agencies 
to incorporate information on employees' various competency levels and 
job needs into the design of training and development programs--
increasing both the relevancy and timeliness of the learning.

FWS targeted training to employees to leverage specific knowledge and 
experiences:

FWS's training center recognized employees' various competency levels 
and job needs when developing courses on the use of geographic 
information systems (GIS), which are becoming increasingly important 
mapping and information analysis tools for natural resources agencies, 
according to FWS officials.[Footnote 11] Rather than providing 
instruction focused solely on software features and functions of GIS, 
the training center designed its curriculum to teach the application of 
GIS to employees based on their roles in natural resources management. 
With a focus on these varied roles and related needs, the center 
developed separate GIS training courses for "explorers" (natural 
resource managers or others just wanting to know about GIS), "users" 
(biologists and other personnel using GIS in their daily job) and 
"developers" (those individuals designing and developing a natural 
resources GIS for use by others). Officials at FWS's training center 
said that with the training targeted to the specific background and 
needs of employees, FWS can minimize the time spent teaching 
participants information that they already know or do not need to know 
in carrying out their job responsibilities. The training center's 
curriculum consists of 15 separate GIS courses, with 3 additional 
courses under consideration or development. According to FWS officials, 
about 1,200 FWS employees at over 400 offices use GIS software in their 
jobs.

IRS used skills assessments to focus on developmental needs:

To assist in identifying employees' competency levels and incorporating 
job needs into training, IRS developed and used technical assessment 
batteries for the agency's field assistance personnel and customer 
service representatives. These multiple-choice instruments were 
designed to assess the key technical knowledge that the employee needs 
in order to carry out his or her job. On the basis of each employee's 
test results, the agency will recommend specific training and other 
appropriate interventions, such as mentoring, to improve performance. 
IRS officials told us that in some cases where the assessments showed 
that individuals were already knowledgeable in a particular area, 
employees still wanted to take the related training because they viewed 
training as a job benefit as well as a way to improve knowledge and 
skills for their jobs. This provides an example of how important 
agencies' considerations of the organizational culture and working 
environment when designing training and development programs are in 
preparing for and addressing issues that may arise during 
implementation. As we point out in our training guide, employees need 
to not only understand the goals of agencies' training and development 
efforts, but also to accept responsibility for developing their 
competencies and careers, as well as for improving their organization's 
performance.[Footnote 12]

Lesson learned: Assess options for using other organizations' course 
content, staff, services, or facilities when designing a new training 
and development program in order to develop efficient and cost-
effective strategies.

When thinking about strategies and sources for the design of a new 
training and development program, officials can potentially discover 
more efficient and cost-effective approaches through the use of other 
organizations' course content, staff, services, and/or facilities. 
Adequate planning can help an agency in meeting the developmental needs 
of trainees without overburdening the agency's training capacity or 
creating excess capacity. Obtaining reasonable estimates of likely 
costs and identifying potential obstacles of using others' training 
resources can help agencies develop more informed perspectives on ways 
to effectively leverage resources.

IRS considered several options in designing a course for senior 
managers:

In evaluating options for designing and delivering a new training 
course for its senior managers, IRS considered various sources for such 
training, including internal resources, contractor support, and 
partnerships. IRS officials said that they considered internal 
resources to design the training but quickly realized that the agency 
did not have sufficient expertise. The officials also considered using 
a contractor but concluded that the costs would be too high. Instead, 
IRS decided to partner with the Federal Executive Institute, an OPM-
sponsored training facility in Charlottesville, Virginia, that provides 
training to senior employees from across the federal government. IRS 
officials found that partnering IRS design and subject matter experts 
with institute and other renowned leaders in the field delivered the 
most cost-effective approach and yielded the best results. The design 
team produced a course called "Learning Through Others," delivered on 
the Charlottesville campus. According to IRS officials, this course 
surpassed agency needs and expectations and was less expensive than a 
direct contracting arrangement with an outside vendor. They added that 
participants in the training program could learn public service values 
through lessons and encounter competency-based experiential learning, 
business-related challenges, and a capstone simulation. IRS officials 
characterized the course as high quality and said the prestige 
associated with studying at the Federal Executive Institute provided an 
additional benefit for IRS participants.

Lesson learned: Establish mechanisms and controls to avoid unnecessary 
duplication or inconsistency within and across agencies' training 
efforts.

The agencies used both centralized and decentralized approaches by, for 
example, centrally managing reporting and record keeping while allowing 
some localized management of training content. Whatever mix of 
centralized and decentralized approaches is used, agencies recognize 
that it is important to limit overlap and duplication and ensure the 
delivery of an integrated message when appropriate. VHA and FWS 
officials found that establishing mechanisms and controls is important 
to limit duplication or inconsistencies within an agency, across 
component organizations within a department, or across the federal 
government as a whole.

VHA's Employee Education System helped limit duplication of effort:

VHA's Employee Education System, which serves as an internal training 
consulting team within VHA, assists the agency's 21 regional networks 
in designing and implementing programs to develop general and specific 
skills for VHA employees. Within each of the 21 regional networks, an 
Education Service Representative acts as a liaison in coordinating 
numerous developmental programs with VHA headquarters--sharing 
information with their counterparts about effective practices and 
identifying areas of possible duplication or inconsistency across VHA. 
According to VHA officials, the coordination and communication achieved 
through this organizational structure has helped ensure consistency in 
implementing the agency's national training priorities. For example, 
officials said that the consulting team assisted in implementing 
changes to VHA's processes for collecting third-party insurance 
reimbursements in the wake of legislation that required VA to make 
greater efforts to collect unpaid debts from veterans. They told us 
that these legislative changes enabled local VHA facilities to receive 
these reimbursements, but also overwhelmed the local billing and debt 
collection processes. To address the problem and help ensure 
consistency across the agency, the training consulting team 
participated in redesigning the processes for coding, billing, and debt 
collection; trained the employees responsible for billing and debt 
collection in the new processes; and created graphical representations 
of the new processes and posted them throughout the agency's facilities 
to aid employees in learning.

Interior's Training Directors Council facilitated communication across 
departmental components:

Interior used its Training Directors Council to facilitate 
communication across the department's different bureaus, thus helping 
to minimize duplicative training and development efforts. This council 
provides opportunities--through formal meetings and informal 
communications--for training managers from Interior's various bureaus 
to share curriculum and related training ideas with their colleagues. 
The director of the training center at FWS, who chairs the council, 
told us that on more than one occasion he has discovered through 
council business that other Interior components had developed 
strategies or solutions to address emerging or existing needs that 
FWS's training center had also identified. For instance, the training 
center at FWS was considering whether to add a new course on grants 
management to its curriculum. Through its participation in the council, 
FWS determined that another component agency, the Bureau of Land 
Management, already offered grants management courses through its 
National Training Center in Phoenix, Arizona. After reviewing the 
content of these Bureau of Land Management courses, officials at FWS's 
training center determined that they did not need to design and develop 
a separate grants management course. Instead, FWS and the Bureau of 
Land Management now jointly manage the delivery of this training 
course.

Lesson learned: Develop and use criteria for determining the optimal 
mix of delivery mechanisms to use in order to select the most effective 
approaches given each learning situation.

In response to emerging demands and the increasing availability of new 
technologies, agencies are faced with the challenge of choosing the 
optimal mix of training delivery mechanisms to design training that is 
as effective and efficient as possible. Agency officials consider a 
wide variety of instructional approaches to achieve learning--in the 
classroom, through distance learning, or through structured on-the-job 
experiences. Officials also took other factors into account, including 
whether to provide individualized instruction or team-based training 
and when to use blended learning that combines different teaching 
methods (e.g., Web-based and instructor-led) within the same training 
program. USACE found that identifying and systematically using criteria 
to help select effective delivery mechanisms assisted in building well-
supported justifications for the design of training and development 
programs.

USACE used criteria to select media and method of instruction:

To select the appropriate media and method of instruction for its 
training programs, USACE uses criteria contained in the Corps of 
Engineers Systems Approach to Training, the agency's documented process 
for developing training programs. The criteria include issues such as 
the expected frequency of changes to the training content, the size and 
diversity of the target population, and the degree of student 
interaction required. USACE officials told us that, using these 
criteria, course managers from USACE's training center coordinate with 
relevant agency program offices and subject matter experts to decide on 
the appropriate mode for training delivery. While most of the training 
center's courses occur in a conventional classroom setting, agency 
decision makers have focused on trying to identify courses (or modules 
of courses) to convert from classroom training to more economical modes 
of delivery, such as distance learning, computer-assisted instruction, 
computer-based instruction, or a combination of such approaches. USACE 
officials said that many of their courses now incorporate CD ROM and 
Internet-based materials as prework assignments before attending 
classroom training and for reference use during and after the training 
events.

VHA used a profiling tool to help in selecting delivery approaches:

VHA's internal training consulting team used a training delivery 
strategy tool that consists of a series of questions structured to 
guide users through the process of selecting an appropriate delivery 
approach for a proposed training effort. This team designed the 
training delivery strategy tool to help staff plan, analyze, develop, 
and deliver training and development activities. To aid in decision 
making, the tool includes factors such as audience composition, course 
goals and objectives, course modules, any prerequisites, participant 
preparation, and course follow-up and evaluation. VHA officials said 
that applying this tool and analyzing the resultant profile helps 
maximize learner understanding, retention, and application.

Lesson learned: Ensure that employees have the needed equipment and 
technologies so that they can take maximum advantage of learning 
opportunities.

Many organizations are taking advantage of more flexible design and 
delivery methods made possible by technology to, for example, deliver 
training to the user's desktop, thereby making training more accessible 
and cost effective. As agencies move forward in using new approaches, 
it is important to ensure that employees have the needed equipment and 
technology to take maximum advantage of learning opportunities.

IRS converted mandatory training courses to an online format:

IRS recently converted a series of mandatory training courses from 
face-to-face group briefings to an online format in order to more 
efficiently provide this training to its employees. Through these 
online mandatory training programs, which include computer security 
awareness, ethics issues, and prevention of sexual harassment, IRS 
wanted to (1) reduce the burden of managers who previously had to 
prepare for and deliver the training, (2) provide ready access of the 
information to line employees when and where they need it, and (3) 
lower the costs associated with the group briefings. IRS officials said 
that online delivery lessened employee time in taking the training from 
approximately 6 to 2 hours and in some cases eliminated the need for 
travel. To ensure the accessibility and usability of these online 
briefings, IRS worked to resolve various challenges in the conversion, 
such as designing the online product for the lowest computing 
capabilities of the bulk of the trainee population and providing 
alternative delivery mechanisms for individuals who were without a 
computer or Intranet access. The officials said that they also learned 
it is important to design the online briefings based on a common 
template and style guide to standardize their look and feel, provide 
online text-only versions of the training for persons who are visually 
impaired, and test the usability of the online briefings with end users 
on a range of equipment and allow sufficient time for needed revisions.

OPM initiated a pilot program for its employees to use online 
courseware:

To explore opportunities for increased use of e-learning approaches, 
OPM established a pilot program that allowed about 250 of its employees 
access to approximately 1,800 online courses through the Department of 
Transportation's Transportation Virtual University. OPM's training 
unit worked with offices throughout OPM to identify employees to 
participate in the pilot program. OPM officials said that all employees 
selected for the pilot program had access to the equipment and 
technology needed to make use of the University's online training, by 
using a computer either at the employee's desktop or at some central 
location. As a result of this pilot, OPM enhanced its offerings of 
online training for its employees by becoming a partner in the 
GoLearn.com initiative, a governmentwide online training center for 
federal employees. OPM officials said the pilot program also showed 
that not all employees function effectively in an online training 
environment and that some employees need a more structured format in 
order to learn.

Lesson learned: Plan early when developing integrated solutions that 
complement other planned and ongoing strategies to improve performance 
so that when implemented the strategies work effectively and are 
aligned to achieve agency goals.

When designing training and development programs, the agencies 
sometimes considered how they could integrate them with other 
strategies to improve performance and meet emerging demands. If the 
work environment is not conducive to providing opportunities to use new 
skills or work in different ways, no matter how good the training 
program is it may not be effective or successful in terms of changing 
on-the-job performance. In addition, training and development programs 
represent a significant investment of resources (including time and 
money) and may:

not always be part of an appropriate solution. The agencies developed 
integrated solutions that included developing and using job aids, 
performance support tools, and other approaches to enhance knowledge 
management[Footnote 13] and to aid employees on the job as a complement 
to training.

OPM planned for an electronic support tool to aid agency employees in 
using a new computer system to process retirement claims:

As part of its effort to reengineer and modernize its processing and 
support of federal employees' retirement claims, OPM is developing 
plans for an integrated Electronic Performance Support System to aid 
the agency's benefits specialists in using a new computer system. 
Procedural and information job aids are to be built directly into the 
software to provide documentation and guidance, "just in time" 
assistance, and error detection. This is intended to be an integrated 
system to permit coordination between different modes of training and 
enhance the learning and performance of the OPM employees working with 
reengineered business processes and the new computer system. OPM 
officials said, for example, that this system would assist employees in 
completing steps using actual data and circumstances of a particular 
case they were working on rather than consulting a manual or using data 
put together just for training. According to OPM officials, as the 
focus under modernization shifts from processing claims to providing 
customer service, this system will help employees working in OPM's 
retirement program to interact more directly with program participants 
to answer questions and solve problems about retirement issues.

USACE identified online solutions to help enhance and integrate 
training efforts:

As a complement to the training and development programs it offers to 
its employees, USACE recently entered into a joint project with the 
Department of Labor to use an online knowledge management system called 
Workforce Connections. This system, which resulted from a memorandum of 
understanding promoting cooperative efforts between the departments of 
Defense and Labor, will provide the USACE workforce with on-demand, 
online access to job aids, performance support materials, and:

course content 7 days a week, 24 hours daily. The system will feature 
development and maintenance of online communities of practice to 
support knowledge management of USACE's Learning Network, which is 
USACE's overall platform for delivering a wide variety of learning 
resources to agency employees.[Footnote 14] Another part of the 
learning network is USACE's Virtual Campus, a distance learning site 
that allows employees access to Web-based courses and training events. 
Another component of the learning network includes electronic 
performance support tools, such as job aids and other information 
resources. USACE officials said that they consider the systems in the 
learning network to comprise a powerful solution that effectively 
integrates the agency's training efforts.

Lesson learned: Plan for the direct participation of senior agency 
leaders and experienced staff in the delivery of training and 
development programs to increase buy-in and build support for 
organizational change.

Internal resources, such as subject matter experts and high performers, 
can often provide valuable insight into training design because of 
their familiarity with the agency's policies, programs, and corporate 
culture. To increase buy-in, help establish greater credibility, and 
build support for organizational change, the agencies have learned the 
value of planning for the direct involvement of senior managers in the 
training program.

IRS and OPM involved executives and managers:

IRS officials told us that a key feature of the agency's frontline 
managers course is that it was designed to use senior managers and 
experienced frontline managers drawn from the agency's business units 
to teach the course. In addition, executives participate in course 
modules that focus on emerging issues facing the agency. The deputy 
commissioner of IRS's Wage and Investment business unit served as the 
executive sponsor of the training program and participated in course 
modules featuring executives. The officials also told us that IRS 
executives partner with outside vendors to serve as an instructor team 
to deliver all courses designed for senior managers. Using business 
unit executives and managers as course instructors helped ensure that 
the course's content and emphasis related to the mission, goals, and 
guiding principles of the agency.

At OPM, agency managers have been the first to take special initiative 
training--such as courses on prohibited personnel practices, 
whistleblower procedures, and information technology security--before 
they are offered agencywide. OPM officials said that teaching the 
material to agency managers in advance of line employees enables the 
managers to model desired behaviors and learning for their employees 
and convincingly convey how they personally benefited from the 
training.

FWS and VHA relied on in-house experts:

FWS's training center brings in FWS field office personnel when 
building a cadre of senior, in-house instructors. Training center 
officials said that involving trainers from the field helps to build 
trust with trainees and provides an added level of credibility that 
neither academics nor other subject matter experts who lack field 
experience can easily replicate. According to these officials, many 
expert employees come from the field and stay to teach at the center 
for 3 or 4 years. Some, however, teach only one or two courses or get 
involved for a short duration before returning to their positions in 
the field. The director of the training center said that he views this 
passing-on of information from seasoned veterans to less experienced 
employees as crucial for maintaining the unique knowledge base of the 
agency.

VHA used "super users" to teach medical center personnel to use its 
computerized patient record system, a computer interface that allows 
hospital personnel to keep more comprehensive patient records and 
enables clinicians, managers, and other staff to review and analyze 
data gathered on any patient. The super users--VHA employees with other 
job-related duties and responsibilities--were trained to be thoroughly 
knowledgeable about the system so they could demonstrate its 
capabilities and directly relate the training to employees' work. VHA 
initiated this strategy when the agency began rolling out the 
application in 1997. VHA officials said that planning to build on the 
direct involvement of these super users was successful because they 
served as first-line resources for employees' questions about the new 
system and helped the agency to build organizational support for the 
system. According to VHA, the agency developed a cadre of more than 
2,500 super users, and about 180,000 VHA employees use the patient 
record system.

Agencies are Considering More Sophisticated Evaluation Approaches As 
Part of Designing their Training and Development Programs:

Without evaluation of training programs, participants may take 
ineffective courses that do not provide the necessary learning 
experience or that do not translate to improved performance on the job. 
Overall, the five agencies in our review relied primarily on standard 
end-of-course evaluations to obtain the participants' reaction to, and 
satisfaction with, a specific training course or learning opportunity. 
Although the agencies encountered challenges given some of the 
difficulties associated with measuring the impact of training on 
individual and organizational performance, they have begun or are 
planning to use more comprehensive and sophisticated evaluation 
techniques for assessing their training and development efforts. Such 
techniques include the use of pre-and post-testing to determine the 
extent of learning accomplished, tracking the performance or 
advancement of individuals and work units before and after training is 
completed to assess professional growth and improvements in 
organizational performance, and limited use of ROI analyses to compare 
the benefits (quantified in dollars) with the costs of a training and 
development program.

To help determine whether the objectives of training and development 
are achieved, agencies can begin by incorporating measures of 
effectiveness into the design of training and development programs. 
Defining objectives in a measurable way enables agency officials to 
offer a more convincing business case and contributes to improving the 
quality of feedback. Whenever possible, training goals should measure 
the individual and organizational results achieved rather than the 
training inputs or outputs (e.g., number of available courses or people 
trained). Figure 7 depicts some of the steps involved in determining 
the evaluation methods to use in designing training and development 
programs.

Figure 7: Steps in Determining Methods for Evaluating Training 
Programs:

[See PDF for image]

[End of figure]

Agencies' Experiences in Determining Methods for Evaluating Their 
Training and Development Programs:

Agency officials recognized the importance of determining during design 
how they planned to evaluate the effectiveness of their training and 
development programs. To collect information on participants' reaction 
to and satisfaction with the training program, for example, VHA uses a 
standard evaluation form with questions related to program design, 
delivery, outcomes, overall satisfaction, and logistics. In completing 
this survey, training participants evaluate their success in completing 
learning objectives and the performance of the faculty. Other agencies 
also obtained participant feedback though interviews or focus groups. 
OPM conducted exit interviews with individuals who participated in that 
agency's Presidential Management Intern program, a 2-year developmental 
program for individuals from a wide variety of academic disciplines who 
wish to enter the federal service. These exit interviews were designed 
to obtain feedback from the participants about their overall 
satisfaction and reaction to the Presidential Management Intern 
program, including their suggestions for enhancing the training 
provided.

The five agencies have begun to use, or are planning to use, more 
comprehensive and sophisticated evaluation techniques to assess the 
extent to which training and development programs increased employees' 
knowledge and skills or enhanced individual and organizational 
performance. One of these more sophisticated evaluation techniques is 
the use of pre-and post-testing to determine the extent of learning 
during the training program. USACE's training center conducts pre-and 
post-tests on over 90 percent of the courses it offers and is working 
toward the goal of using such tests for all courses. The agencies also 
tracked job performance and the advancement or movement of personnel to 
assess the potential effectiveness of training. FWS officials told us 
they track participants' career advancement to determine the extent to 
which participation in the leadership development program for midlevel 
employees contributed to increased mobility into more responsible 
leadership positions in the agency. According to training center 
officials, about 37 percent of the program graduates have taken either 
promotions or new lateral assignments since the program's inception in 
January 2002.

In addition, some agencies attempted to conduct ROI analyses to compare 
the benefits (quantified in dollars) to the costs of a particular 
training and development program. VHA officials pointed to concerted 
efforts to conduct ROI analyses on several training and development 
programs, including customer service, leadership development, and 
computer-based training. IRS officials, on the other hand, have decided 
that the challenges and difficulties in conducting such analyses are 
not worth the effort for the resultant information--given the challenge 
of isolating the performance improvements that might result from a 
specific training activity and the difficulty in monetizing identified 
benefits in order to calculate the ROI. IRS instead uses the concept of 
"time to capability" to determine whether and to what extent a training 
course, program, or other training intervention has improved the 
organization's ability to perform its mission successfully. IRS defines 
time to capability as the validated accumulation over time of employees 
who have been trained in specific competencies deemed critical to the 
success of an organizational unit. Under this approach, when IRS has 
trained a predetermined number of employees, officials consider that 
the agency has achieved the goal of training a critical mass within its 
workforce and conclude that the agency has an organizationwide 
capability in the specific competencies.

Agencies' Lessons Learned in Determining Methods for Evaluating Their 
Training and Development Programs:

Agencies' training and development efforts involve a continuous effort 
throughout planning, design, implementation, and evaluation. 
Therefore, it is important to recognize that evaluation is not a static 
requirement to be carried out after the fact. When undertaking design 
and development of training, agencies can rely on evaluations and 
benchmarking to determine what approaches work best given all the 
related elements, such as the proposed audience for the training 
program, the material to be covered, and possible delivery mechanisms 
that could be employed. Determining methods for evaluating training 
programs as part of their design can help identify and remove obstacles 
to successful implementation. For example, an agency officials said 
that catching potential problems early on saved valuable time and 
resources that a major redesign of training later on likely would have 
entailed. On the basis of our review at the five agencies, we 
identified four lessons learned regarding the agencies' efforts to 
determine methods for evaluating their training and development 
programs.

Lesson learned: Incorporate appropriate aspects of the evaluation 
approach when designing training and development programs by specifying 
what results are expected to better ensure the availability and use of 
quality performance data.

In assessing how and to what degree performance could be improved with 
a specific training program, agencies should try to establish a 
targeted level of improved performance as well as assess the possible 
consequences if the training were not to occur. Determining a target 
level for improved performance can aid agencies in assessing whether 
the expected costs associated with the proposed training are worth the 
anticipated benefits. Agreeing upon the planned evaluation approach in 
the design clearly sets forth the results the agency expects to achieve 
through the training. In addition, planning ahead helps ensure the 
availability and use of sound and relevant performance data.

VHA decided to evaluate a customer service training program after it 
was implemented:

In response to low scores on customer satisfaction surveys, a VHA 
regional network office pursued various initiatives to improve customer 
service, including the design, development, and implementation of a new 
training program focused on creating a more customer-service-oriented 
culture and improving employee morale and collaboration to better meet 
customer needs. After delivery of this new training program, called 
"The Customer," VHA selected a contractor to evaluate the effectiveness 
of the training effort. In its report, the contractor stated that 
because the opportunity to conduct the evaluation did not occur until 
after the training program had been delivered, the use of preferred 
evaluation methods was not possible. The contractor reported that this 
lack of preprogram planning had also been experienced in some other VHA 
network offices. Nevertheless, the contractor evaluated the customer 
service training program by surveying participating employees and their 
supervisors. From these survey results, the contractor concluded that 
the training program was viewed as successful for those who needed it, 
but that the training did not receive a sufficient level of support 
from supervisors. The contractor recommended that VHA obtain additional 
feedback from supervisors as well as from an individual hired to 
telephone or visit the VHA facilities in the network office to observe 
customer service activities.

Lesson learned: Consider new approaches for collecting and analyzing 
performance data with the aim of increasing the quality and quantity of 
training evaluation feedback.

The agencies faced various challenges in obtaining a high quality and 
quantity of feedback needed to evaluate their training and development 
programs. We previously reported that low participation on the part of 
employees and managers in surveys and focus groups may limit an 
agency's access to the data needed to complete valid and useful 
evaluations of training programs.[Footnote 15] With strong agency 
support and proper planning, stakeholders, including training 
participants, supervisors, managers, and trainers, are more likely to 
provide the information and feedback needed to successfully and 
effectively evaluate agency training and development programs. USACE, 
for example, recognized that it needed to ensure that it incorporated a 
wider variety of stakeholder perspectives in assessing the impact of 
training on employee and agency performance. Stakeholders' perspectives 
can be obtained through surveys and questionnaires, individual or group 
interviews, or communication with more formal multidisciplinary bodies 
such as advisory or education councils. Valuable sources of information 
include the training participants; training designers, developers, and 
facilitators; agency leaders, managers, supervisors, subordinates, and 
coworkers; employee organizations; internal and external customers; and 
functional and subject matter experts.

USACE's training center altered its approach to obtaining supervisory 
feedback on its training efforts:

USACE's training center has been attempting to obtain more 
sophisticated evaluation feedback to determine if its training courses 
affected employees' behavior on the job. Training center officials told 
us that they originally had planned to evaluate all of the center's 
training courses by obtaining feedback from employees and supervisors 6 
months after the training course on the extent to which employee on-
the-job behavior had changed. In a test of this approach, the training 
center sent out hard copies of two generic survey forms: one for the 
employee and one for the supervisor. Because the training center had 
locator information for trainees but not for their supervisors, both 
surveys were mailed to the employee, who was then asked to forward one 
to his or her supervisor. Training center officials said the response 
to this survey effort was disappointing. For one 35-person class, for 
example, the center received 3 employee surveys and 1 supervisor 
survey. To increase the level of feedback, officials told us that the 
center is transitioning to an electronic process whereby the survey 
forms are sent via e-mail to the employee, who is then requested to 
forward the survey via e-mail to his or her supervisor. Training center 
officials said that in a recent test of this new approach on one 
course, the center received a 67 percent response rate from employees 
and a 36 percent response rate from supervisors. The officials told us 
that they are working to develop unique evaluation forms for each 
course in order to obtain feedback on specific learning objectives 
rather than rely on a generic survey form for all courses.

Lesson learned: Plan for the use of multiple data types and sources to 
provide a balanced approach in assessing the effectiveness of training 
and development programs.

Successful organizations typically develop and implement human capital 
approaches based on a thorough assessment of the organizations' 
specific needs and capabilities. Valid and reliable data are the 
starting point for such assessments. To assess the results achieved 
through training and development, agencies can rely upon hard 
(quantitative) data, such as productivity/output, quality, costs, and 
time, or soft (qualitative) data, such as feedback on how well a 
training program satisfied employees' expectations. By taking steps to 
agree on measures of success up front, agency officials can decide on 
the objectives for each training and development program. Using a 
balanced approach that reflects feedback from customers and employees, 
as well as organizational results, is particularly important as 
agencies transform their cultures and operations. In addition, because 
the work of federal employees can be complex and often cannot be 
reduced to a single task, a balanced approach to both the types and 
sources of data helps to strengthen the linkages between training and 
development programs and improved performance.

USACE conducts evaluations of some training courses:

In addition to obtaining end-of-course participant feedback and 
administering pre-and post-tests for many of its courses, USACE's 
training center also conducts course evaluations of its offerings. To 
conduct this evaluation, the designated course manager from USACE's 
training center is responsible for observing the full course and 
assessing various aspects of its design and delivery, including the 
training content, materials, and instructors. Training center officials 
said that various factors can trigger the decision to conduct an 
overall course evaluation, including a significant decline in trainees' 
overall satisfaction with the course, the introduction of new 
instructors, the use of contractor assistance, or a specific 
recommendation from an agency office or unit. According to training 
center officials, the results of this course evaluation are assembled 
with the end-of-course participant survey feedback and pre-and post-
testing results to present a comprehensive and balanced view of the 
effectiveness of the training program.

Lesson learned: Take into account all relevant factors for determining 
the costs of a training and development program to better ascertain 
whether it is cost-effective in relation to benefits achieved.

Calculating the ROI for a training program involves identifying and 
monetizing the program's benefits and then dividing this by a full 
tabulation of the program's costs. These costs should usually include 
the cost of program materials provided to each participant; the cost of 
the facilities; the costs of the facilitator or instructor, including 
time for both preparation and delivery; any travel-related expenses for 
participants; salaries and benefits costs of the participants for the 
time they attend the program; and an allocation of relevant 
administrative and overhead costs.

VHA's evaluation of a leadership development program did not include 
all costs:

In cooperation with VHA's internal training consulting team, one of 
VHA's regional network offices designed, developed, and implemented a 
networkwide leadership development program called "Competency 
Development for Leaders in the 21st Century." According to information 
we gathered during our review, the costs incurred for the consulting 
team's efforts on this training program were not included in the ROI 
calculation even though it contributed substantially toward developing 
and implementing the program. VHA officials told us that these costs 
were not included in the analysis because the course designers only 
wanted to determine the return on the network's investment, not the 
agency's overall investment. Agency officials said that one of the main 
goals of involving the consulting team was to help the network develop 
the capability to use the ROI process to evaluate training. Although 
these costs were not included in the ROI cost tabulation, VHA did 
include the salaries (plus employee benefits) of the participants for 
the time they attended the developmental program. As we noted in our 
recently issued assessment guide, agencies might overlook the costs of 
participant attendance when calculating the total costs of a training 
program.[Footnote 16]

Conclusions and Observations:

Federal agencies are faced with the need to invest resources wisely to 
ensure that their employees possess the information, skills, and 
competencies required to carry out their work successfully. The 
examples provided in this report may help to address this need by 
describing some of the experiences and lessons learned that other 
agencies might find applicable or adaptable to their unique situations. 
This information is intended to both provide a realistic perspective on 
how agencies have approached designing their training and development 
programs to date as well as to take a more detailed look at some of the 
concepts explored in our recently issued assessment guide focused on 
strategic training and development efforts in the federal government.

Our work reviewing the selected agencies' efforts to design training 
and development programs reinforces the significance of good planning 
and design of these programs to ensure their successful implementation 
and evaluation. The experiences and lessons learned we identified also 
demonstrate how effective design efforts--as part of a strategic 
training and development process--rely on the eight core 
characteristics that we identified in our earlier work: (1) strategic 
alignment, (2) leadership commitment and communication, (3) stakeholder 
involvement, (4) accountability and recognition, (5) effective resource 
allocation, (6) partnerships and learning from others, (7) data quality 
assurance, and (8) continuous performance improvement. Indeed, by 
focusing on these eight core characteristics, agencies can improve not 
only the design of their training and development efforts but also the 
planning, implementation, and evaluation of their programs to better 
ensure that their employees have the information, skills, and 
competencies needed to carry out their work successfully.

Agency Comments and Our Evaluation:

We provided a draft of this report on December 12, 2003, to the 
Secretary of Defense, the Secretary of the Interior, the Commissioner 
of Internal Revenue, the Director of OPM, and the Secretary of VA. 
Interior, IRS, and OPM provided written comments on the draft report. 
In his written comments (see app. IV), Interior's Assistant Secretary 
for Fish and Wildlife and Parks generally agreed with the report's 
findings regarding the Department and the FWS. He said that the report 
provides important examples that can help the Department continue to 
move forward with additional confidence in its actions. In his written 
comments (see app. V), the Commissioner of Internal Revenue said that 
IRS was honored to share some of its lessons learned with us for 
governmentwide dissemination. He said that our review also provides IRS 
with practices from other agencies to assist IRS in its efforts to 
continually improve its programs. In her written comments (see app. 
VI), the Director of OPM said that she appreciated the opportunity for 
OPM to be included in the report and to share information on OPM's 
training and development activities and programs. Interior, IRS, and 
OPM also provided technical comments to clarify specific points 
regarding the information presented in the draft report, which we have 
incorporated as appropriate. In comments by E-mail through its GAO 
liaison, VA agreed with the information presented regarding the 
Department and had no additional comments on the draft report. USACE 
informed us that it had no comments on the draft report.

As agreed with your office we are sending copies of this report to the 
Ranking Minority Member, Subcommittee on Oversight of Government 
Management, the Federal Workforce and the District of Columbia, Senate 
Committee on Governmental Affairs; the Chairmen and Ranking Minority 
Members of the Senate Committee on Governmental Affairs and the House 
Committee on Government Reform; and other interested congressional 
parties. We are also providing copies to the Secretary of the Army, the 
Secretary of the Interior, the Secretary of the Treasury, the Director 
of OPM, and the Secretary of Veterans Affairs. This report is available 
to others upon request. In addition, the report is available at no 
charge on the GAO Web site at [Hyperlink, http://www.gao.gov] http://
www.gao.gov.

If you have any questions about this report, please contact me on (202) 
512-6806. Key contributors to this report are listed in appendix VII.

Sincerely yours,

Signed by:

George H. Stalcup: 
Director, Strategic Issues:

[End of section]

Appendixes: 

Appendix I: Objective, Scope and Methodology:

The objective of this review was to provide information on selected 
federal agencies' experiences and lessons learned in key aspects of 
designing training and development programs for their employees. 
Specifically, we focused on the agencies' experiences and lessons 
learned related to:

* assessing current and future agency skill and competency requirements 
and identifying related training and development needs,

* developing strategies and solutions for training and development 
needs, and:

* determining methods to evaluate the effectiveness of training and 
development programs.

For this review, lessons learned were defined as knowledge that could 
be applied in the future that the agencies gained through either 
positive or negative experiences.

To address this objective we focused on five agencies: the U.S. Army 
Corps of Engineers (USACE), Department of Defense; the U.S. Fish & 
Wildlife Service (FWS), Department of the Interior; the Internal 
Revenue Service (IRS), Department of the Treasury; the U.S. Office of 
Personnel Management (OPM); and the Veterans Health Administration 
(VHA), Department of Veterans Affairs. We chose the five agencies for a 
variety of reasons, including the diversity of employee occupations 
within the agencies, reported innovative approaches for training and 
developing their employees, and congressional requester interest. We 
selected USACE to obtain a Department of Defense perspective in light 
of that department's reputation as a leader in the area of training and 
developing military personnel. We included FWS and VHA to obtain 
information related to a broad mix of employee occupations. We selected 
IRS because of reported innovative approaches to training and 
development and included OPM because of its role as the federal 
government's human capital agency.

To obtain information and related documentation, we visited the 
following locations:

* USACE's headquarters human resources directorate in Washington, D.C., 
and its Professional Development Support Center in Huntsville, Alabama.

* FWS's headquarters human resources division in Arlington, Virginia, 
and its National Conservation Training Center in Shepherdstown, West 
Virginia.

* IRS's strategic human resources division in Arlington, Virginia, and 
the small business and self-employed business line.

* OPM's headquarters human resources office and the career development 
branch of the center for retirement and insurance service.

* VHA's Employee Education System headquarters in Washington, D.C., and 
the medical center and local network offices in Durham, North Carolina.

It is important to note that our methodology was not designed to 
identify examples that would be representative of all training and 
development efforts at the five agencies in our review or of the 
government as a whole. We did not verify the accuracy and reliability 
of the data provided to us or the systems used to produce the 
information. Further, in citing training and development programs as 
examples in connection with lessons learned, we did not assess the 
effectiveness of the training programs and practices. Rather, our 
intent was to highlight and briefly describe some experiences and 
lessons learned that agency officials believed helped each agency 
improve or enhance its training and development programs.

To obtain information about the five agencies' experiences and lessons 
learned related to designing training and development programs, we:

* Interviewed agency human capital and training officials and subject 
matter experts responsible for agency training, performance, and other 
initiatives; and:

* Reviewed and analyzed agency documents such as workforce plans, 
analyses, and reports; strategic, performance, and succession plans and 
reports; organizational, occupational, and unit-based competency 
standards; knowledge and skills inventories; skills gaps assessments; 
competency and skill assessments; surveys of agency employees; training 
plans and proposals; workforce demographic data; budget data; 
evaluation plans and reports; and performance measures.

The lessons learned we identified for inclusion in this report were 
based on (1) their linkages with one or more of the eight core 
characteristics of a strategic training and development process, which 
we had identified in our previous work (see app. III) and (2) 
sufficient evidence from the agency to support the experiences that 
they relayed to us. At the exit conferences for the five agencies, we 
presented agency officials with the list of lessons learned that we had 
identified and wished to attribute to their experiences. At that time, 
we also informed each agency of the specific examples from their 
experiences that we would likely attribute to these lessons learned. In 
these meetings, agency officials expressed no objections to the lessons 
learned we had identified and in some cases provided additional 
information to support specific examples from their experiences that we 
proposed to use for this report.

We conducted our audit work between August 2002 and November 2003 in 
accordance with generally accepted government auditing standards.

[End of section]

Appendix II: Background on Selected Agencies and Their Training and 
Development Functions:

The following summarizes key information on the five agencies included 
in this review. These summaries include information on the agencies' 
missions, organizational structures, and training and development 
functions.

U.S. Army Corps of Engineers:

The U.S. Army Corps of Engineers (USACE), part of the Department of the 
Army within the Department of Defense, is comprised of approximately 
34,600 civilian and 650 military men and women. USACE has a diverse 
workforce consisting of military and civilian engineers, biologists, 
geologists, hydrologists, natural resource managers, and other 
specialists who work in engineering and environmental matters. USACE's 
mission is to provide engineering services to the nation: (1) planning, 
designing, building and operating water resources and other civil works 
projects, (2) designing and managing the construction of military 
facilities for the Army and Air Force, and (3) providing design and 
construction management support for other Department of Defense and 
federal agencies. USACE headquarters office is located in Washington, 
D.C.

USACE's Professional Development Support Center, located in Huntsville, 
Alabama, serves as the center of learning and training for the agency. 
The training center manages and implements the Proponent-Sponsored 
Engineer Corps Training program, which provides job-related training 
through technical, professional, managerial, and leadership courses for 
USACE and other government agencies. USACE's training center offers 
more than 200 courses covering topics that support the agency's 
mission.

U.S. Fish and Wildlife Service:

The mission of the U.S. Fish and Wildlife Service (FWS), a component of 
the Department of the Interior, is working with others to conserve, 
protect, and enhance fish, wildlife, and plants and their habitats for 
the continuing benefit of the American people. FWS's headquarters is 
located in Washington D.C., while its field units are located 
throughout the United States. FWS employs more than 9,600 people and is 
supported by a volunteer force of 29,000. Nearly 90 percent of FWS 
employees work in field locations.

The National Conservation Training Center, located in Shepherdstown, 
West Virginia, is FWS's training center and is responsible for training 
a wide range of employees in the conservation community and serves as a 
gathering place where conservation professionals from government, 
nonprofit organizations, and corporations work toward common goals. 
Training for FWS's law enforcement personnel is primarily conducted 
through the Federal Law Enforcement Training Center, an interagency law 
enforcement training organization headquartered in Glynco, Georgia.

Internal Revenue Service:

The Internal Revenue Service (IRS) is a branch of the U.S. Department 
of the Treasury. IRS's mission is to provide America's taxpayers top 
quality service by helping them understand and meet their tax 
responsibilities and by applying the tax law with integrity and 
fairness to all. IRS's organizational structure includes the following 
business units: four operating divisions organized around four major 
customer segments (Wage and Investment, Small Business/Self-Employed, 
Large and Mid-Size Business, and Tax Exempt and Government Entities); 
four functional divisions (National Taxpayer Advocate, Appeals, 
Criminal Investigation, and Communications and Liaison); and two shared 
services/support divisions (Agency-Wide Shared Services and 
Modernization and Information Technology Services). As of March 2003, 
IRS had about 116,300 employees.

IRS takes a decentralized approach to training and developing its 
workforce. Each business unit has an embedded human resources component 
that provides advice and analysis on related policies and issues and 
formulates strategies, procedures, and practices to address the unit's 
human capital needs. Learning and Education, one of eight major 
divisions comprising IRS's Office of Strategic Human Resources, 
provides guidance and sets policy and standards on training and 
development for the agency's business units and headquarters offices.

U.S. Office of Personnel Management:

The U.S. Office of Personnel Management (OPM), the federal government's 
human capital agency, provides human resources policy leadership, 
technical advice and assistance, and products and services to federal 
agencies, employees, annuitants, and job seekers. It also oversees 
governmentwide compensation and performance management systems, and 
provides retirement, health benefit, and other insurance services to 
federal employees, annuitants, other beneficiaries, and agencies. In 
March 2003, OPM completed a major restructuring process through which 
it consolidated various agency functions. As of March 2003, OPM 
employed approximately 3,500 people, many of them stationed in agency 
headquarters in Washington, D.C. OPM has a field presence in 16 major 
U.S. cities as well as operating centers in Pennsylvania and Georgia.

OPM's training and development efforts are largely decentralized to the 
agency's various program and staff offices. The employee training and 
development unit within the agency's human capital management office is 
responsible for setting overall strategy and for planning and 
implementing agencywide training such as leadership development 
programs and various mandatory training programs. According to OPM, the 
agency's newly established Chief Human Capital Officer plays a 
significant role in advising the OPM Director on overall employee 
training and development initiatives and programs, as well as the 
establishment of the agency's training budget.

Veterans Health Administration:

The Veterans Health Administration (VHA), one of three major 
administrations within the Department of Veterans Affairs (VA), is 
responsible for providing primary care, specialized care, and related 
medical and social support services to veterans through an integrated 
health care system. VHA administers its functions through a group of 21 
regional network offices located around the United States. As of March 
2003, VHA employed about 203,500 people out of a total VA workforce of 
about 225,000 employees.

VA takes a decentralized approach to training and development 
operations. VA's human resources office provides advice and guidance on 
training to VHA and the other departmental components but delegates 
training and development operations to each component. VHA's 
organization includes the Employee Education System, which is an 
internal training consulting group that provides educational services 
that support the workforce development and continuing education needs 
for VHA employees. This internal consulting group of about 300 
individuals primarily helps to assess agency training needs at the 
national level, while VHA network offices and medical centers take lead 
responsibility for assessing their own local needs. These internal 
training consultants are available to assist VHA network offices and 
medical centers in designing, developing, implementing, and evaluating 
training and development programs to meet these local needs.

[End of section]

Appendix III: Core Characteristics of a Strategic Training and 
Development Process:

The following summarizes the eight core characteristics that make a 
training and development process effective and strategically focused on 
achieving results. We identified these core characteristics as part of 
our recent work in developing an assessment guide to assist federal 
agencies in evaluating their training and development efforts.[Footnote 
17]

* Strategic alignment. Clear linkages exist between the agency's 
mission, goals, and culture and its training and development efforts. 
The agency's mission and goals drive a strategic training and 
development approach and help ensure that the agency takes full 
advantage of an optimal mix of strategies to improve performance and 
enhance capacity to meet new and emerging challenges.

* Leadership commitment and communication. Agency leaders and managers 
consistently demonstrate that they support and value continuous 
learning, are receptive to and use feedback from employees on 
developmental needs and training results, and set the expectation that 
fair and effective training and development practices will improve 
individual and organizational performance.

* Stakeholder involvement. Agency stakeholders are involved throughout 
the training and development process to help ensure that different 
perspectives are taken into account and contribute to effective 
training and development programs. Stakeholders' views are incorporated 
in identifying needed performance enhancements, developing and 
effectively implementing well-thought-out strategies, and helping to 
conceptualize and use balanced measures that accurately reflect the 
extent to which training and development efforts contribute toward 
achieving results.

* Accountability and recognition. Appropriate accountability 
mechanisms, such as performance management systems, are in place to 
hold managers and employees responsible for learning and working in new 
ways. Appropriate rewards and incentives exist and are used fairly and 
equitably to encourage innovation, reinforce changed behaviors, and 
enhance performance.

* Effective resource allocation. The agency provides an appropriate 
level of funding and other tools and resources--along with external 
expertise and assistance when needed--to ensure that its training and 
development programs reflect the importance of its investment in human 
capital to achieving its mission and goals.

* Partnerships and learning from others. Coordination within and among 
agencies achieves economies of scale and limits duplication of efforts. 
In addition to benchmarking high-performing organizations, these 
efforts allow an agency to keep abreast of current practices, enhance 
efficiency, and increase the effectiveness of its training and 
development programs.

* Data quality assurance. The agency has established policies and 
procedures that recognize and support the importance of quality data 
and of evaluating the quality and effectiveness of training and 
development efforts. It establishes valid measures and validated 
systems to provide reliable and relevant information that is useful in 
improving the agency's training and development efforts.

* Continuous performance improvement. Agency practices and policies 
foster a culture of continuous improvement and optimal organizational 
performance regarding training and other activities. Stakeholders rely 
on and use program performance information and other data to assess and 
refine ongoing training and development efforts; target new initiatives 
to improve performance; and design, develop, and implement new 
approaches to train and develop employees.

[End of section]

Appendix IV: Comments from the Department of the Interior:

United States Department of the Interior:

OFFICE OF THE SECRETARY WASHINGTON, D.C. 20240:

JAN 23 2004:

Mr. George Stalcup 
Director, Strategic Issues 
U.S. General Accounting Office 
441 G Street, N.W. 
Washington, D.C. 20548:

Dear Mr. Stalcup:

Thank you for providing the Department of the Interior the opportunity 
to review and comment on the draft U.S. General Accounting Office 
report entitled, "Human Capital: Selected Agencies' Experiences and 
Lessons Learned in Designing Training and Development Programs" (GAO-
04-291) dated December 12, 2003. In general, we agree with the report's 
findings that pertain to the Department. The report provides important 
examples that can help us continue to move forward with additional 
confidence in our actions.

The Department has one specific comment. On page 37, the U.S. Fish and 
Wildlife Service's mission description should be changed to: "The 
mission of the U.S. Fish and Wildlife Service is, working with others, 
to conserve, protect and enhance fish, wildlife, and plants and their 
habitats for the continuing benefit of the American people.":

We look forward to receiving the final report.

Sincerely,

Signed by: 

Craig Manson: 

Assistant Secretary for Fish and Wildlife and Parks:

The following is GAO's comment on the Department of the Interior's 
letter dated January 23, 2004.

GAO Comment:

We have clarified the mission statement of the Fish and Wildlife 
Service to note its collaboration with others to accomplish its 
mission.

[End of section]

Appendix V: Comments from the Internal Revenue Service:

DEPARTMENT OF THE TREASURY: 
INTERNAL REVENUE SERVICE: 
WASHINGTON, D.C. 20224:

COMMISSIONER:

January 12, 2004:

Mr. George Stalcup: 
Director, Strategic Issues 
United States General Accounting Office 
Washington, D.C. 20548:

Dear Mr. Stalcup:

I have reviewed your draft report entitled, "Human Capital: Selected 
Agencies' Experiences and Lessons Learned in Designing Training and 
Development Programs" (GAO-04-291). Thank you for selecting the 
Internal Revenue Service to be a part of this important study. We are 
honored to share some of our lessons learned with you for government-
wide dissemination. I am particularly pleased that we could share our 
work in leadership development, strategic planning, skills assessments 
and use of technology as best practices. The study not only recognizes 
some of our accomplishments but also provides us with practices from 
other agencies to assist us in our efforts to continually improve our 
programs.

Strategic management of human capital is a vital part of moving the IRS 
forward. As a result, I have created a new Human Capital Office. Our 
Chief Human Capital Officer, Beverly Ortega Babers, will provide the 
leadership necessary to focus our resources on critical training and 
development programs. We will build on the good planning and design 
practices cited in your report to ensure that our employees have the 
skills and competencies they need to provide top quality service to 
America's taxpayers.

If you have any questions, please contact me or Beverly Ortega Babers, 
Chief Human Capital Officer, at (202) 622-7902.

Sincerely,

Mark W. Everson:

Signed for Mark W. Everson:

[End of section]

Appendix VI: Comments from the Office of Personnel Management:

OFFICE OF THE DIRECTOR:

UNITED STATES OFFICE OF PERSONNEL MANAGEMENT 
WASHINGTON, DC 20415-0001:

JAN 20 2004:

The Honorable David M. Walker 
Comptroller General:
General Accounting Office 
Washington, DC 20548:

Dear Mr. Walker:

Thank you for the opportunity to respond to GAO's draft report entitled 
Selected Agencies' Experiences and Lessons Learned in Designing 
Training and Development Programs. As you pointed out, it is critical 
that agencies invest training and development resources wisely to 
ensure that their employees have the necessary skills and competencies 
to carry out their agency's mission. Within the Office of Personnel 
Management (OPM), I am very committed to developing our employees and 
making the kinds of strategic investments that will enhance their 
ability to contribute to achieving OPM's important mission. This 
commitment is evident in OPM's current restructuring whereby OPM is now 
organized around mission goals and priorities. The restructuring better 
positions OPM to meet the President's plan to reform the Federal 
Government by recognizing the importance of talented employees to 
deliver a citizen-centered, results-oriented, and market-based 
Government.

Furthermore, OPM's Plan for the Strategic Management of OPM's Human 
Capital: FY 2004-2007, lays out an integrated and goal specific 
approach to ensuring OPM has the talent and capacity to meet our new 
and expanded responsibilities.

Since your staff conducted its review of our training and development 
programs, we have made enhancements to our programs. These enhancements 
are enclosed and are also included in our Plan for the Strategic 
Management of OPM's Human Capital: FY 2004-2007, which is also 
enclosed.

I appreciate the opportunity for OPM to be highlighted in your report, 
and to share the innovative training and development activities and 
programs we have created.

Sincerely,

Signed by: 

Kay Coles James: 
Director:

Enclosures:

Office of Personnel Management Comments on GAO draft report: Selected 
Agencies' Experiences and Lessons Learned in Designing Training and 
Development Programs:

Page 8 - regarding OPM's 2001 skills assessment...

In addition to the 2001 skills assessment described in this Section, in 
late FY 2003, OPM conducted another skills assessment to reflect our 
recent restructuring and new strategic priorities. We identified 
mission critical occupations and competencies needed in our current and 
future workforce, along with initiatives to address any gaps. This 
business forecasting was combined with techniques to recruit, select, 
place and train employees and ensure an organizational climate that 
fosters commitment. This workforce planning initiative was a major 
aspect of our recently published Plan for the Strategic Management of 
OPM's Human Capital, 2004-2007. The Plan identifies our mission 
critical occupations and the most critical competencies which enhance 
our ability to create strategic training and development plans to help 
us carry out our important mission.

Page 38 - regarding setting of overall training and development 
strategy...

OPM's training and development activities were accurately addressed in 
this Section, however, since the time that the draft report was 
completed, OPM has increased the role of its Chief Human Capital 
Officer (CHCO). The CHCO now plays a more significant role in advising 
the Director on overall employee training and development initiatives 
and programs, as well as the establishment of the agency's training 
budget. Under the direction of OPM's CHCO, program offices, with the 
assistance from the internal Human Capital Management Office, create 
specific training and development strategies and activities that 
address mission critical competencies identified in our FY 2003 skills 
assessment. This strategic approach, based upon recent workforce data, 
better positions OPM to prioritize its training needs, and forecast 
funds to support those needs.

[End of section]

Appendix VII: GAO Contacts and Staff Acknowledgments:

GAO Contacts:

George H. Stalcup or Susan Ragland, (202) 512-6806:

Acknowledgments:

In addition to the persons named above, K. Scott Derrick, Gerard Burke, 
T.J. Thomson, and Thomas Davies, Jr. made key contributions to this 
report.

(450147):

FOOTNOTES

[1] U.S. General Accounting Office, Human Capital: A Guide for 
Assessing Strategic Training and Development Efforts in the Federal 
Government - Exposure Draft, GAO-03-893G (Washington, D.C.: July 2003).

[2] U.S. General Accounting Office, Human Capital: Key Principles for 
Effective Strategic Workforce Planning, GAO-04-039 (Washington, D.C.: 
Dec. 11, 2003).

[3] In providing technical comments on a draft of this report, OPM 
noted that it conducted another skills assessment in late fiscal year 
2003 to reflect its recent restructuring and new strategic priorities.

[4] In previous work, we identified practices used by agencies in other 
countries to manage the succession of senior executives and other 
employees with critical skills. See Human Capital: Insights for U.S. 
Agencies from Other Countries' Succession Planning and Management 
Initiatives, GAO-03-914 (Washington, D.C.: Sept. 15, 2003).

[5] An IDP is a written plan, cooperatively prepared by the employee 
and his or her supervisor, that outlines the steps the employee will 
take to develop knowledge, skills, and abilities in building on 
strengths and addressing weaknesses as he or she seeks to improve job 
performance and pursue career goals. These individual development plans 
are also known as personal development plans, personal training plans, 
and individual training plans.

[6] For more information on key practices and implementation steps that 
can help agencies transform their cultures, see U.S. General Accounting 
Office, Results-Oriented Cultures: Implementation Steps to Assist 
Mergers and Organizational Transformations, GAO-03-669 (Washington, 
D.C.: July 2, 2003).

[7] U.S. General Accounting Office, Results-Oriented Cultures: Creating 
a Clear Linkage between Individual Performance and Organizational 
Success, GAO-03-488 (Washington, D.C.: Mar. 14, 2003).

[8] For more information on performance management of senior executives 
at IRS, see U.S. General Accounting Office, Results-Oriented Cultures: 
Using Balanced Expectations to Manage Senior Executive Performance, 
GAO-02-966 (Washington, D.C.: Sept. 27, 2002).

[9] The 360-degree feedback process is designed to provide a manager 
direct input from various sources--supervisor, peers, customers, and 
subordinates--and to compare those results to a self-evaluation. With 
this feedback, managers can identify action items and incorporate them 
into their individual performance plans. 

[10] 31 U.S.C. 3302.

[11] GIS is a system of computer software, hardware, and data used to 
manipulate, analyze, and graphically present a potentially wide array 
of information associated with geographic locations.

[12] GAO-03-893G, p. 7.

[13] Knowledge management is an approach to capturing, understanding, 
and using the collective body of information and intellect within an 
organization to accomplish its mission.

[14] Communities of practice provide an on-line resource for peers to 
ask and respond to questions and share knowledge.

[15] U.S. General Accounting Office, Human Capital: Design, 
Implementation, and Evaluation of Training at Selected Agencies, GAO/T-
GGD-00-131 (Washington, D.C: May 18, 2000).

[16] GAO-03-893G, p. 69.

[17] GAO-03-893G, p. 75.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 

 Voice: (202) 512-6000:

 TDD: (202) 512-2537:

 Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: