DHS Training: Improved Documentation, Resource Tracking, and Performance Measurement Could Strengthen Efforts
Highlights
What GAO Found
The Department of Homeland Security (DHS) has processes to evaluate training, track resources, and assess leader development. However, various actions could better position the department to maximize the impact of its training efforts.
Training evaluation: All five DHS components in GAO's review—U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center—have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities (see table). By updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations.
Summary of Training Evaluation Attributes DHS Could Better Document
|
|
Customs and Border Protection |
U.S. Coast Guard |
Immigration and Customs Enforcement |
Transportation Security Administration |
Federal Law Enforcement Training Center |
|
Methodology |
◗ |
● |
● |
◗ |
◗ |
|
Timeframes |
◗ |
◗ |
● |
◗ |
● |
|
Roles and responsibilities |
◗ |
● |
◗ |
◗ |
● |
●: Documented evaluation processes fully included information to meet the attribute.
◗: Documented evaluation processes partly included some information to address a given attribute.
Source: GAO analysis of documented evaluation processes. I GAO-14-688
Capturing training cost: DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures.
Leader development: DHS's Leader Development Program (LDP) Office is in the process of implementing a department-wide framework to build leadership skills. However, the LDP Office has not clearly identified program goals and the measures it uses to assess program effectiveness do not exhibit some attributes that GAO previously identified as key for successful performance measurement. These include linkage of performance measures to the program's goals, clarity, and establishment of measurable targets to assess the measures. By clearly identifying program goals and incorporating key attributes, the LDP could better ensure actionable information for identifying and making program improvements.
Why GAO Did This Study
DHS is the third-largest cabinet-level department in the federal government, with over 230,000 employees doing diverse jobs. To fulfill its complex mission, DHS's workforce must have the necessary skills and expertise. GAO previously reported on DHS's hiring and recruiting efforts. GAO was asked to assess DHS's training practices.
This report addresses (1) the extent to which DHS has documented processes to evaluate training and reliably capture costs and (2) the extent to which DHS measures the performance of its leader development programs. To conduct its work, GAO reviewed documented training evaluation processes, training cost data from fiscal year 2011 through fiscal year 2013, and leadership training programs. GAO also interviewed training officials at the department level and at the five DHS components selected for this review about their varieties of training and development programs. Information from these components cannot be generalized to all of DHS, but provides insights.
Recommendations
GAO recommends that DHS update its documentation to fully reflect key attributes of an effective evaluation, identify challenges to and corrective measures for capturing training costs department-wide, and clearly identify LDP goals and ensure that LDP performance measures reflect key attributes. DHS concurred and identified actions to address our recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should direct DHS components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised. | In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to evaluate training and development programs. We found that all five DHS components in GAO's review--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center--have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used.... However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities. We concluded that by updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations. We therefore recommended that DHS direct its components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised. In September 2016, DHS officials reported that a DHS-wide self-audit of training evaluation processes was completed on March 31, 2016 and found that DHS has current documentation addressing effective learning evaluation programs and that there are commonalities in evaluation procedures across the components. As a part of the self-audit, components provided policy and procedure documents related to their training evaluation processes and found that they adhere to sound instructional systems design models. Further, DHS updated its department-wide guidance on training evaluation in April 2016 to incorporate the attributes for effective training evaluation processes and as of August 2017, each of the components in our review have followed suit. Therefore, this recommendation is closed as implemented.
View More |
| Department of Homeland Security | To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges.. | In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to reliably capture costs. We found that DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has...
|
| Department of Homeland Security | In addition, to produce actionable information for DHS's LDP management to use in identifying the need for, and making, program improvements, the Secretary of Homeland Security should direct the Chief Human Capital Officer to clearly identify LDP goals and ensure LDP performance measures include key attributes, such as linkage, clarity, and measurable targets. | In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS measures the performance of its leader development programs. We found that DHS had established a Leader Development Program Office under the Office of the Chief Human Capital Officer to design, develop, and execute a department-wide leadership program. As of September 2014, the Leader Development Program Office was in the process of implementing a five-tier Leader Development Framework to build leadership skills across all staff levels, and had developed a program-wide assessment approach to analyze the impact of its efforts. However, we found that the Leader...
|