Federal Telework: Program Measurement Continues to Confront Data Reliability Issues
Highlights
What GAO Found
To prepare for its reporting obligations under the Telework Enhancement Act of 2010, the Office of Personnel Management (OPM) assembled the Interagency Telework Measurement Group, consisting of officials from several federal agencies, to assist in revising the telework data call—the survey OPM has used since 2002 to collect telework data from executive agencies. This group standardized key terms such as telework, employee, and eligibility to promote a common reporting methodology among the agencies. The revised telework data call also included changes to the time period for which OPM requested agencies report telework data, and included more extensive training for respondents.
Because of changes made to the data call to allow OPM to meet requirements of the act and assist agencies in responding to the data call, OPM officials believe they will be able to provide to Congress an improved report on telework in June 2012. However, these changes also mean that OPM officials will not be able to use participation and frequency data from the 2011 data call to compare to data from previous years and across agencies. OPM officials have noted that this could limit OPM’s ability to report agency progress in its first report to Congress. The ability to compare with previous years is affected by:
agencies use of methods of varying reliability to collect telework data, and some agencies made changes to their data collection systems for the 2011 data call. Executive agencies provide telework participation and frequency data by relying on estimates, counting telework agreements, or using automated time and attendance records.
modifications to the data call instrument, including changes to terminology and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous conclusions.
Participants at the two data call training sessions may not have received the same reporting instructions, and uncertainty about whether all agency respondents attended training, created a risk that some respondents may be unaware of important terms and instructions. While some of the information provided at the two training sessions was similar, each session contained some new information, usually in response to questions raised at a previous session.
Future data call improvement efforts could result in a trade-off between the desire for maintenance of a consistent data series over time for comparison with previous data calls and a need to improve data collection. According to OPM, agencies will begin piloting automated telework data collection during 2012 and 2013. OPM expects that this method of data collection will provide it more reliable data than other methods. However, these efforts to standardize methods for tracking telework data may result in changes to agencies’ methods of data collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data that could occur during a transition to automated data collection.
Why GAO Did This Study
The Telework Enhancement Act of 2010 requires OPM to report to Congress on the degree of telework participation at executive agencies. To meet this requirement, OPM collects information on agency telework programs through an annual survey, which it refers to as the data call. However, concerns exist about the reliability of these data.
GAO was asked to assess OPM’s: (1) actions in response to the requirements of the act and (2) handling of identified data reliability issues in the 2011 telework data call. To address these objectives GAO reviewed its previous reports addressing telework data reliability, and used the Office of Management and Budget’s guidance for federal surveys to review OPM’s (1) plans to collect telework participation data from agencies and (2) development of a data collection instrument. GAO interviewed key OPM officials about its implementation of the 2011 data call.
Recommendations
GAO recommends that OPM (1) clearly report reliability limitations with the 2011 telework data call in its June 2012 report to Congress and (2) continue efforts to improve data collection and gather information to allow for the appropriate qualification of year-to-year comparisons and inform users about the effects of data collection changes going forward. OPM partially concurred with the first recommendation. However, GAO believes it should report limitations in its annual report. OPM fully concurred with the second. OPM provided a number of technical comments which GAO incorporated as appropriate.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Office of Personnel Management | To improve OPM's annual reporting of telework to Congress, the OPM Director should ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation. |
In its report, GAO noted that that comparisons to telework data from previous years are affected by several factors: (1) executive agencies use methods of varying reliability to collect telework data such as estimates, counting telework agreements, or using automated time and attendance records, and (2) modifications to the telework data call instrument. Since then, in its June 2012 annual report OPM stated that there are some limitations with respect to the participation and frequency findings that should be considered, and that agencies rely upon differing methodologies and data sources when gathering participation and frequency data, including time and attendance systems, counts of telework agreements, and surveys of employees. As a result, the final participation and frequency numbers may underreport telework with consequences for the reliability of reported results. Additionally, in the report's detailed methodology, OPM states that, while changes to the data call make some comparisons between previous years' data calls less appropriate, they were necessary in order to accurately gauge the changing nature of Federal telework programs.
|
| Office of Personnel Management | To improve OPM's annual reporting of telework to Congress, the OPM Director should continue efforts to improve data collection and gather information that allows for the appropriate qualification of year-to-year comparisons and informs users about the effects of data collection changes going forward. |
OPM revised its collection of telework participation data from agencies to include full FY participation data from FY12. This action would enable OPM to report year-to-year comparisons of telework participation in its 2014 Status of Telework in the Federal Government Report to Congress. This report was initially expected to be issued by OPM in late 2014/early 2015, but as of April 2015 this report had not been issued. Reporting an accurate year to year comparison of telework participation would complete the implementation of this recommendation. In December 2015, OPM publicly released the 2014 Telework Report. The report included data on year to year comparisons of telework participation and completes the implementation of this recommendation.
|