Technology Assessment Design Handbook

What GAO Found

The Technology Assessment (TA) Design Handbook identifies tools and approaches GAO staff and others can consider in the design of robust and rigorous technology assessments. The handbook underscores the importance of TA design (Chapter 1), outlines the process of designing TAs (Chapter 2), and describes approaches for mitigating select TA design and implementation challenges (Chapter 3). While the primary audience of this handbook is GAO staff, other organizations may also find portions of this handbook useful as they consider or conduct TAs. This is an update to the handbook published in December 2019, based on the experiences of GAO teams and a review of relevant literature and comments submitted by external experts and the public between December 2019 and December 2020.

The handbook identifies three general design stages, as shown in the figure below. The handbook also highlights seven cross-cutting considerations for designing TAs: the iterative nature of TA design, congressional and policymakers' interests, resources, independence, engaging internal and external stakeholders, potential challenges, and communication strategy. In addition, the handbook provides a high-level process for developing policy options, as a tool for analyzing and articulating a range of possible actions a policymaker could consider that may enhance the benefits or mitigate the challenges of a technology. Steps in developing policy options include, as applicable: determining the potential policy objective; gathering evidence; identifying possible policy options and the relevant dimensions along which to analyze them; analyzing policy options; and presenting the results of the analysis.

Summary of Key Stages of Technology Assessment Design

We found that GAO TAs can use a variety of design approaches and methods. The handbook includes TA design and methodology examples, along with example objectives commonly found in GAO TAs, such as: describe a technology, assess opportunities and challenges of a technology, and assess policy implications or options. For example, some GAO TAs include an objective related to describing the status and feasibility of a technology, which GAO teams have addressed by using methodologies such as expert panels, interviews, literature and document reviews, site visits, and determining the technology readiness level.

Also included in the handbook are examples of TA design and implementation challenges, along with possible mitigation strategies. We identified four general categories of challenges: (1) ensuring that the design and implementation of TAs result in useful products for Congress and other policymakers; (2) determining the policy objective and measuring potential effects; (3) researching and communicating complicated issues; and (4) engaging relevant stakeholders. For example, allowing sufficient time for writing, review, and any needed revisions is one potential mitigation strategy that could help teams write simply and clearly about technical subjects and ensure that the design and implementation of TAs result in useful products for Congress and other policymakers.

Why GAO Did This Study

In 2019, GAO created the Science, Technology Assessment, and Analytics team to expand its work on cutting-edge science and technology issues, and to provide oversight, insight, and foresight for science and technology. TAs can be used to strengthen decision-making, enhance knowledge and awareness, and provide early insights into the potential effects of technology. Systematically designing a TA can enhance its quality, credibility, and usefulness; ensure independence of the analysis; and ensure effective use of resources.

Under Comptroller General Authority, we developed this handbook by generally following the format of the 2012 GAO methodology transfer paper, Designing Evaluations. Below is a summary of the approach we used to affirm and document TA design steps and considerations for this handbook.

• Reviewed select GAO documents, including Designing Evaluations (GAO-12-208G), published GAO TAs, select GAO products using policy analysis approaches to present policy options, and other GAO reports
• Reviewed select Office of Technology Assessment reports
• Reviewed select Congressional Research Service reports
• Reviewed select English-language literature regarding TAs and related to development and analysis of policy options
• Consulted with external experts and performed outreach, including holding an expert meeting to gather input on TA design, soliciting comments from external experts who contributed to GAO TAs published since 2015, and soliciting comments from the public
• Reviewed experiences of GAO teams that have successfully assessed and incorporated policy options into GAO products and TA design, including challenges to TA design and implementation and possible solutions