Office of Personnel Management:

Agency Needs to Improve Outcome Measures to Demonstrate the Value of Its Innovation Lab

GAO-14-306: Published: Mar 31, 2014. Publicly Released: Apr 30, 2014.

Additional Materials:

Contact:

Seto J. Bagdoyan
(202) 512-4749
bagdoyans@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

In March 2012, the Office of Personnel Management (OPM) opened its innovation lab, a distinct physical space with a set of policies for engaging people and using technology in problem solving. The goals of OPM's innovation lab are to provide federal workers with 21st century skills in design-led innovation, such as intelligent risk-taking to develop new services, products, and processes. OPM's lab was built at a reported cost of $1.1 million, including facility upgrades and construction, equipment and training, and other personnel costs. The lab employs approximately 6 full-time equivalents, including a director, and in fiscal year 2013, the lab's operating costs were approximately $476,000, including salaries.

View of OPM's Innovation Lab

View of OPM's Innovation Lab

OPM's innovation lab is similar in mission and design to other innovation labs GAO reviewed, and OPM has incorporated some of the prevalent practices that other labs use to sustain their operations. Specifically, OPM is using its lab for a variety of projects, including as a classroom for building the capacity to innovate in the federal government. Lab staff indicated that they plan to begin long-term immersion projects—complex projects with diverse users—within a few months. OPM plans to develop and implement evaluation plans specific to each immersion project that will help them track cost benefits or performance improvement benefits associated with the projects.

Starting in March 2013, OPM lab staff began work on a program evaluation framework to more systematically measure the lab's progress toward meeting its overarching goals. In addition, lab staff members are tracking lab activities, such as classes and workshops, and are surveying lab users about the quality of their experience in the lab. However, they have not developed performance targets or measures related to project outcomes, and without a rigorous evaluation framework that can help OPM track the lab's performance, it will be hard to demonstrate that the lab is operating as originally envisioned.

While labs provide a physical space where innovators can convene, federal agencies are not fully aware of their growing community. However, OPM is taking steps to ensure work done in the lab is shared across OPM and with other federal innovators—for example, by hosting weekly training sessions in the lab on best practices. Studies show that information sharing and interorganizational networks can be a powerful driver supporting innovation.

Why GAO Did This Study

Organizations from around the globe are emphasizing that strategies promoting innovation are vital to solving complex problems. To try to instill a culture of innovation in its agency, OPM followed the lead of a number of private sector companies, nonprofit organizations, and government bodies by creating an innovation lab. GAO was asked to examine the lab.

Specifically, GAO 1) described the lab's start-up costs, staffing and organization, activities, and policies governing the lab's use, and 2) assessed how OPM's innovation lab compares to other organizations' innovation labs, including how it uses benchmarks and metrics and how it addresses challenges to innovation. GAO reviewed cost, staffing, and performance information. GAO also reviewed relevant literature on innovation and interviewed officials from public, private, and nonprofit organizations with innovation facilities similar to OPM's lab.

What GAO Recommends

Among other things, GAO recommends that the Director of OPM should direct lab staff to 1) develop a mix of performance targets and measures to help them monitor and report on progress toward lab goals, and 2) build on existing efforts to share information with other agencies that have innovation labs. OPM generally concurred with GAO's recommendations; in addition, they described the steps being taken and planned to refine their ongoing evaluation efforts and to further leverage other federal innovation labs.

For more information, contact Seto J. Bagdoyan at (202) 512-4749 or bagdoyans@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In response to this recommendation, OPM reported consulting with representatives from Harvard's Kennedy School of Government, Denmark's MindLab, and Intuit's Innovation Catalyst Group to improve its performance management planning. OPM developed a suite of output measures to include things like the number of meetings held and people who attended and the number of agencies partnering with the lab. This suite of measures also includes outcome related measures for the lab, including the amount of estimated tax dollars saved as a result of lab activities and the satisfaction levels of participants in lab activities. As of March 2015, OPM officials indicated that draft targets for these measures were in review by the agency. Lab officials also developed a retrospective document on the lab that highlights key projects and the results of those projects.

    Recommendation: To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to develop a mix of performance targets and measures to help them monitor and report on their progress toward lab goals. Output targets could include number and type of lab activities over the next year. Outcome targets and measures should correspond to the lab's overarching goals to build organizational capacity to innovate and achieve specific innovations in concrete operational challenges.

    Agency Affected: Office of Personnel Management

  2. Status: Closed - Implemented

    Comments: As of March 2015, OPM had revised its survey instruments to include standard, understandable surveys for (1) those receiving coaching skills in human centered design, (2) Lab Fellows who will use human-centered design techniques in their home agencies, and (3) human-centered design workshops. The surveys aim to measure participant's satisfaction with sessions in the lab, as well as anticipated return on investment and other job-related improvements from work conducted in OPM's innovation lab.

    Recommendation: To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to review and refine the set of survey instruments to ensure that taken as a whole, they will yield data of sufficient credibility and relevance to indicate the nature and extent to which the lab is achieving what it intends to accomplish or is demonstrating its value to those who use the lab space. For example, lab staff should consider the following actions: (1) Developing a standard set of questions across all service offerings. (2) Revising the format and wording of existing questions related to skills development to diminish the likelihood of social desirability bias and use post-session questions that ask, in a straight-forward way, about whether, or the extent to which, new information was acquired. (3) Replacing words or phrases that are ambiguous or vague with defined or relevant terminology (e.g., terms actually used in the session) so that the respondent can easily recognize a link between what is being asked and the content of the session.

    Agency Affected: Office of Personnel Management

  3. Status: Closed - Implemented

    Comments: In March 2015, the Office of Personnel Management (OPM) reported that it participates regularly in communities of practice with government innovation professionals and experts in human-centered design. OPM also reported that its lab staff members make presentations about OPM's lab and innovation practices to multiple audiences, including members of the federal innovation community. For example, in December 2014 the director of OPM's lab provided subject matter expertise to the Department of Health and Human Services to try to help ignite its innovation program curriculum. In addition, OPM noted that it engages regularly with its innovation lab workshop alumni to support their efforts to bring design-led innovation to their agencies.

    Recommendation: To help substantiate the lab's original goals of enhancing skills in innovation and supporting project-based problem solving, the Director of OPM should direct lab staff to build on existing efforts to share information and knowledge within the federal innovation community. For example, OPM lab staff could reach out to other agencies with labs such as Census, the Department of Housing and Urban Development, and the National Aeronautics and Space Administration's Kennedy Space Center to share best practices and develop a credible evaluation framework.

    Agency Affected: Office of Personnel Management

 

Explore the full database of GAO's Open Recommendations »

Jun 24, 2015

Jun 16, 2015

Jun 3, 2015

May 26, 2015

May 21, 2015

May 20, 2015

May 18, 2015

Apr 27, 2015

Apr 20, 2015

Looking for more? Browse all our products here