DHS Training: Improved Documentation, Resource Tracking, and Performance Measurement Could Strengthen Efforts
Highlights
What GAO Found
The Department of Homeland Security (DHS) has processes to evaluate training, track resources, and assess leader development. However, various actions could better position the department to maximize the impact of its training efforts.
Training evaluation: All five DHS components in GAO's review—U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center—have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities (see table). By updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations.
Summary of Training Evaluation Attributes DHS Could Better Document
|
|
Customs and Border Protection |
U.S. Coast Guard |
Immigration and Customs Enforcement |
Transportation Security Administration |
Federal Law Enforcement Training Center |
|
Methodology |
◗ |
● |
● |
◗ |
◗ |
|
Timeframes |
◗ |
◗ |
● |
◗ |
● |
|
Roles and responsibilities |
◗ |
● |
◗ |
◗ |
● |
●: Documented evaluation processes fully included information to meet the attribute.
◗: Documented evaluation processes partly included some information to address a given attribute.
Source: GAO analysis of documented evaluation processes. I GAO-14-688
Capturing training cost: DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures.
Leader development: DHS's Leader Development Program (LDP) Office is in the process of implementing a department-wide framework to build leadership skills. However, the LDP Office has not clearly identified program goals and the measures it uses to assess program effectiveness do not exhibit some attributes that GAO previously identified as key for successful performance measurement. These include linkage of performance measures to the program's goals, clarity, and establishment of measurable targets to assess the measures. By clearly identifying program goals and incorporating key attributes, the LDP could better ensure actionable information for identifying and making program improvements.
Why GAO Did This Study
DHS is the third-largest cabinet-level department in the federal government, with over 230,000 employees doing diverse jobs. To fulfill its complex mission, DHS's workforce must have the necessary skills and expertise. GAO previously reported on DHS's hiring and recruiting efforts. GAO was asked to assess DHS's training practices.
This report addresses (1) the extent to which DHS has documented processes to evaluate training and reliably capture costs and (2) the extent to which DHS measures the performance of its leader development programs. To conduct its work, GAO reviewed documented training evaluation processes, training cost data from fiscal year 2011 through fiscal year 2013, and leadership training programs. GAO also interviewed training officials at the department level and at the five DHS components selected for this review about their varieties of training and development programs. Information from these components cannot be generalized to all of DHS, but provides insights.
Recommendations
GAO recommends that DHS update its documentation to fully reflect key attributes of an effective evaluation, identify challenges to and corrective measures for capturing training costs department-wide, and clearly identify LDP goals and ensure that LDP performance measures reflect key attributes. DHS concurred and identified actions to address our recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should direct DHS components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised. |
In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to evaluate training and development programs. We found that all five DHS components in GAO's review--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center--have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities. We concluded that by updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations. We therefore recommended that DHS direct its components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised. In September 2016, DHS officials reported that a DHS-wide self-audit of training evaluation processes was completed on March 31, 2016 and found that DHS has current documentation addressing effective learning evaluation programs and that there are commonalities in evaluation procedures across the components. As a part of the self-audit, components provided policy and procedure documents related to their training evaluation processes and found that they adhere to sound instructional systems design models. Further, DHS updated its department-wide guidance on training evaluation in April 2016 to incorporate the attributes for effective training evaluation processes and as of August 2017, each of the components in our review have followed suit. Therefore, this recommendation is closed as implemented.
|
| Department of Homeland Security | To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges.. |
In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to reliably capture costs. We found that DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. We concluded that DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures. We therefore recommended that DHS identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges. As of September 2016, DHS reported that its Office of the Chief Human Capital Officer and Office of the Chief Financial Officer worked together to research the issue and determine the best course of action to standardize training cost reporting. However, this did not require a formal root cause analysis. In order to identify a way for the components to capture training costs in a standardized manner, DHS formed a Tiger Team that included officials from each component which identified 12 common functional areas for training to capture training costs. Components were directed to prepare implementation plans outlining how they will begin capturing cost data within the identified functional training areas and, according to DHS, began capturing the data on September 29, 2016. In January 2017, the DHS Chief Financial Officer provided guidance to the components as to how they are to report their training costs to DHS on a quarterly basis. Components provided their first quarterly submission to DHS in January 2017 and have since sustained that process over three quarters. Given that DHS has successfully demonstrated that the process can be sustained, we are closing this recommendation as implemented.
|
| Department of Homeland Security | In addition, to produce actionable information for DHS's LDP management to use in identifying the need for, and making, program improvements, the Secretary of Homeland Security should direct the Chief Human Capital Officer to clearly identify LDP goals and ensure LDP performance measures include key attributes, such as linkage, clarity, and measurable targets. |
In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS measures the performance of its leader development programs. We found that DHS had established a Leader Development Program Office under the Office of the Chief Human Capital Officer to design, develop, and execute a department-wide leadership program. As of September 2014, the Leader Development Program Office was in the process of implementing a five-tier Leader Development Framework to build leadership skills across all staff levels, and had developed a program-wide assessment approach to analyze the impact of its efforts. However, we found that the Leader Development Program Office could strengthen its performance measurement approach. In particular, we found that the office had not clearly identified program goals, and that the 12 measures that the office developed to assess its performance did not consistently exhibit attributes we have previously identified as key for successful measurement. We recommended that DHS clearly identify Leader Development Program goals and ensure program performance measures include key attributes, such as linkage, clarity, and measurable targets. In December 2014, Leader Development Program Office officials provided us with updated documentation on the program's assessment approach. This documentation established ten program goals. It also explained how the program's performance measures link to the ten program goals and to department-wide goals. Further, the documentation established targets for each performance measure and provided clarification for ambiguous measures. These enhancements to the Leader Development Program assessment approach should help produce actionable information for the program's management to use in identifying the need for, and making, program improvements. Therefore, this recommendation is closed as implemented.
|