Office of Personnel Management: Agency Needs to Improve Outcome Measures to Demonstrate the Value of Its Innovation Lab
Highlights
What GAO Found
In March 2012, the Office of Personnel Management (OPM) opened its innovation lab, a distinct physical space with a set of policies for engaging people and using technology in problem solving. The goals of OPM's innovation lab are to provide federal workers with 21st century skills in design-led innovation, such as intelligent risk-taking to develop new services, products, and processes. OPM's lab was built at a reported cost of $1.1 million, including facility upgrades and construction, equipment and training, and other personnel costs. The lab employs approximately 6 full-time equivalents, including a director, and in fiscal year 2013, the lab's operating costs were approximately $476,000, including salaries.
View of OPM's Innovation Lab
OPM's innovation lab is similar in mission and design to other innovation labs GAO reviewed, and OPM has incorporated some of the prevalent practices that other labs use to sustain their operations. Specifically, OPM is using its lab for a variety of projects, including as a classroom for building the capacity to innovate in the federal government. Lab staff indicated that they plan to begin long-term immersion projects—complex projects with diverse users—within a few months. OPM plans to develop and implement evaluation plans specific to each immersion project that will help them track cost benefits or performance improvement benefits associated with the projects.
Starting in March 2013, OPM lab staff began work on a program evaluation framework to more systematically measure the lab's progress toward meeting its overarching goals. In addition, lab staff members are tracking lab activities, such as classes and workshops, and are surveying lab users about the quality of their experience in the lab. However, they have not developed performance targets or measures related to project outcomes, and without a rigorous evaluation framework that can help OPM track the lab's performance, it will be hard to demonstrate that the lab is operating as originally envisioned.
While labs provide a physical space where innovators can convene, federal agencies are not fully aware of their growing community. However, OPM is taking steps to ensure work done in the lab is shared across OPM and with other federal innovators—for example, by hosting weekly training sessions in the lab on best practices. Studies show that information sharing and interorganizational networks can be a powerful driver supporting innovation.
Why GAO Did This Study
Organizations from around the globe are emphasizing that strategies promoting innovation are vital to solving complex problems. To try to instill a culture of innovation in its agency, OPM followed the lead of a number of private sector companies, nonprofit organizations, and government bodies by creating an innovation lab. GAO was asked to examine the lab.
Specifically, GAO 1) described the lab's start-up costs, staffing and organization, activities, and policies governing the lab's use, and 2) assessed how OPM's innovation lab compares to other organizations' innovation labs, including how it uses benchmarks and metrics and how it addresses challenges to innovation. GAO reviewed cost, staffing, and performance information. GAO also reviewed relevant literature on innovation and interviewed officials from public, private, and nonprofit organizations with innovation facilities similar to OPM's lab.
Recommendations
Among other things, GAO recommends that the Director of OPM should direct lab staff to 1) develop a mix of performance targets and measures to help them monitor and report on progress toward lab goals, and 2) build on existing efforts to share information with other agencies that have innovation labs. OPM generally concurred with GAO's recommendations; in addition, they described the steps being taken and planned to refine their ongoing evaluation efforts and to further leverage other federal innovation labs.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Office of Personnel Management | To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to develop a mix of performance targets and measures to help them monitor and report on their progress toward lab goals. Output targets could include number and type of lab activities over the next year. Outcome targets and measures should correspond to the lab's overarching goals to build organizational capacity to innovate and achieve specific innovations in concrete operational challenges. |
In response to our recommendation, OPM met with leaders in innovation, including from Intuit, the University of Virginia's Darden School of Business, and Harvard University's Kennedy School of Government, to identify how to most appropriately establish performance goals for its innovation lab. OPM now has established output and outcome measures related to the lab's performance with targets and lab staff are developing benchmarks for additional outcome measures. Having clear and specific performance measures will help OPM track and evaluate the extent to which the lab is meeting its original intent and over time, to make any necessary adjustments.
|
| Office of Personnel Management | To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to review and refine the set of survey instruments to ensure that taken as a whole, they will yield data of sufficient credibility and relevance to indicate the nature and extent to which the lab is achieving what it intends to accomplish or is demonstrating its value to those who use the lab space. For example, lab staff should consider the following actions: (1) Developing a standard set of questions across all service offerings. (2) Revising the format and wording of existing questions related to skills development to diminish the likelihood of social desirability bias and use post-session questions that ask, in a straight-forward way, about whether, or the extent to which, new information was acquired. (3) Replacing words or phrases that are ambiguous or vague with defined or relevant terminology (e.g., terms actually used in the session) so that the respondent can easily recognize a link between what is being asked and the content of the session. |
As of March 2015, OPM had revised its survey instruments to include standard, understandable surveys for (1) those receiving coaching skills in human centered design, (2) Lab Fellows who will use human-centered design techniques in their home agencies, and (3) human-centered design workshops. The surveys aim to measure participant's satisfaction with sessions in the lab, as well as anticipated return on investment and other job-related improvements from work conducted in OPM's innovation lab.
|
| Office of Personnel Management | To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to build on existing efforts to share information and knowledge within the federal innovation community. For example, OPM lab staff could reach out to other agencies with labs such as Census, the Department of Housing and Urban Development, and the National Aeronautics and Space Administration's Kennedy Space Center to share best practices and develop a credible evaluation framework. |
In March 2015, the Office of Personnel Management (OPM) reported that it participates regularly in communities of practice with government innovation professionals and experts in human-centered design. OPM also reported that its lab staff members make presentations about OPM's lab and innovation practices to multiple audiences, including members of the federal innovation community. For example, in December 2014 the director of OPM's lab provided subject matter expertise to the Department of Health and Human Services to try to help ignite its innovation program curriculum. In addition, OPM noted that it engages regularly with its innovation lab workshop alumni to support their efforts to bring design-led innovation to their agencies.
|