This is the accessible text file for GAO report number GAO-11-698 entitled 'Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation' which was released on September 14, 2011. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: August 2011: Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation: GAO-11-698: GAO Highlights: Highlights of GAO-11-698, a report to congressional requesters. Why GAO Did This Study: Federal agencies with a budget of at least $100 million for research and development (R&D) conducted by others must participate in the Small Business Innovation Research (SBIR) program. SBIR has four purposes: meet federal R&D needs; stimulate technological innovation; increase commercialization (e.g., sales) of innovations based on federal R&D; and encourage participation in innovation by small businesses owned by disadvantaged individuals and women. The Small Business Administration (SBA) oversees the efforts of participating agencies, which make awards to small businesses using SBIR funds. Congress directed SBA to develop a database with commercialization data for government use in evaluating the program. GAO was asked to determine (1) how agencies have addressed SBIR’s purposes and (2) the extent of data available to evaluate progress in increasing commercialization. GAO analyzed program documents and interviewed officials at SBA and five agencies that accounted for about 96 percent of SBIR awards. What GAO Found: For fiscal years 2008 through 2011, the participating agencies GAO reviewed—-the Department of Defense (DOD), the Department of Energy (DOE), the National Aeronautics and Space Administration, the Department of Health and Human Services’ National Institutes of Health, and the National Science Foundation (NSF)-—addressed SBIR’s purposes through solicitations for award applications, technical assistance or matching funds programs, and outreach. In particular, the agencies addressed SBIR purposes related to meeting federal R&D needs and stimulating technological innovation through their solicitations, which included research topics that were designed to meet agencies’ respective R&D or mission needs. Agencies also addressed commercialization of innovations through solicitations, as well as through technical assistance for award recipients or through matching funds programs. To provide technical assistance, the agencies contracted with vendors and consultants for help in developing business plans and identifying potential customers for SBIR award recipients, among other things. Agency matching funds programs provided additional SBIR funds to award recipients that obtained commitments from outside investors. Agencies generally addressed the remaining SBIR purpose, encouraging participation by small businesses owned by disadvantaged individuals and women, through outreach activities aimed at a broader audience, such as sharing information on Web sites. However, the effectiveness of these efforts is difficult to evaluate, in part because SBA does not collect data on the number of SBIR applications submitted by such businesses, thus hindering analysis of trends in their submission of applications. Comparable data are not available across participating agencies to evaluate progress in increasing commercialization of SBIR technologies. SBA has not yet expanded an existing database to include commercialization data for program evaluation, but the agency has hired a contractor and allocated funds to develop the expanded database by August 2011. SBA has also worked with participating agencies to develop common metrics for commercialization. In the absence of the expanded database, agencies have independently gathered commercialization data for their own use that are not comparable. In collecting these data, agencies differed in the types of data collection instruments used, dates the instruments were administered, award recipient populations queried, and types of data requested. Furthermore, with the exception of DOD, agencies that GAO reviewed did not generally take steps to verify commercialization data they collected from award recipients, so the accuracy of the data is largely unknown. SBA has worked with SBIR agencies to identify best practices in other areas of program management but has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Implementing the expanded database should improve the comparability of commercialization data available, but a lack of consistent practices for verifying the accuracy of these data may limit their usefulness for programwide evaluation. What GAO Recommends: GAO recommends that SBA work with participating agencies to (1) collect data on applications from small businesses owned by disadvantaged individuals and women and (2) identify best practices for verification of commercialization data. SBA, DOE, and NSF generally agreed with these recommendations; the other agencies GAO reviewed neither agreed nor disagreed. [End of section] Contents: Letter: Background: Agencies Addressed SBIR's Purposes through Solicitations, Technical Assistance, Matching Funds Programs, and Outreach: Comparable Data Are Not Available across Participating Agencies to Evaluate Progress in Increasing Commercialization of SBIR Technologies: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Comments from the Small Business Administration: Appendix III: Comments from the Department of Energy: Appendix IV: Comments from the National Science Foundation: Appendix V: GAO Contact and Staff Acknowledgments: Table: Table 1: Summary of Five SBIR Agencies' Efforts to Gather Commercialization Data: Abbreviations: DOD: Department of Defense: DOE: Department of Energy: NASA: National Aeronautics and Space Administration: NIH: National Institutes of Health: NSF: National Science Foundation: R&D: Research and Development: Recovery Act: American Recovery and Reinvestment Act: SBA: Small Business Administration: SBIR: Small Business Innovation Research: STTR: Small Business Technology Transfer: [End of section] United States Government Accountability Office: Washington, DC 20548: August 15, 2011: The Honorable Ben Quayle: Chairman: Subcommittee on Technology and Innovation: Committee on Science, Space, and Technology: House of Representatives: The Honorable Adrian Smith: House of Representatives: Small businesses have been collectively described as an engine for economic growth and innovation. According to the Small Business Administration (SBA), such businesses have been responsible for creating 64 percent of net new jobs over the past 15 years and have produced 13 times more patents per employee than larger firms. To assist small businesses in undertaking and obtaining the benefits of research and development (R&D), Congress first passed legislation authorizing the Small Business Innovation Research (SBIR) program in 1982. SBA's Office of Technology administers the program, which presently has four overarching purposes: to use small businesses to meet federal R&D needs, stimulate technological innovation, increase commercialization of innovations derived from federal R&D efforts, [Footnote 1] and encourage participation in technological innovation by small businesses owned by disadvantaged individuals[Footnote 2] and women.[Footnote 3] Since the program was first authorized in 1982, there has been increased congressional interest in its results, particularly the commercialization aspect. Every federal agency with a budget of $100 million or more for extramural R&D--which is conducted by nonfederal employees outside federal facilities--is required to establish and operate an SBIR program funded by 2.5 percent of that budget. Eleven agencies currently participate in the program.[Footnote 4] In fiscal year 2009 (the most recent year for which SBA data are available), the 11 agencies reported more than $2.2 billion in SBIR awards. Although each of these agencies manages its own program, SBA oversees and coordinates agency efforts by setting policy, collecting program data, reviewing agency progress, and reporting annually to Congress. Since the creation of the SBIR program 29 years ago, it has been reauthorized, modified, and extended by Congress at various times. The Small Business Reauthorization Act of 2000 extended the program through September 30, 2008. This act was succeeded by a series of temporary extensions, the most recent of which extended the program through September 30, 2011. In this context, you asked us to examine the practices of participating agencies. Accordingly, we agreed to determine (1) how participating agencies have addressed the SBIR program's four overarching purposes when implementing their programs and (2) the extent of SBIR program data available to evaluate progress in increasing commercialization of SBIR technologies. Our review of the program focused on SBA and five agencies that accounted for about 96 percent of the total dollars awarded by the program in 2009: the Department of Defense (DOD), the Department of Energy (DOE), the National Aeronautics and Space Administration (NASA), the Department of Health and Human Services' National Institutes of Health (NIH),[Footnote 5] and the National Science Foundation (NSF).[Footnote 6] At two of these agencies--DOD and NIH-- we also reviewed program activities conducted by the three subcomponent agencies with the largest SBIR budgets because some key activities are carried out at that level. Specifically, for DOD, we examined the SBIR programs of the Army, Air Force, and Navy and for NIH, we examined the programs of the National Institute of Allergy and Infectious Diseases; the National Cancer Institute; and the National Heart, Lung, and Blood Institute. In conducting our review, we analyzed documentation from these agencies for fiscal years 2008 through 2010 and, when possible, for fiscal year 2011. The documentation we reviewed included, among other things, policy guidance, solicitations for applications for SBIR awards, and descriptions of commercialization assistance provided to SBIR awardees. We also reviewed applicable laws and regulations. In addition, we interviewed SBIR program officials at the agencies, as well as inspector general staff at NSF and knowledgeable stakeholders, including staff of the National Academy of Sciences' National Research Council and representatives of trade associations. We selected the trade associations based on their familiarity with the program, the technologies on which they focus, and whether their membership includes small businesses owned by disadvantaged individuals and women.[Footnote 7] To obtain further context for our review, we attended two national conferences on the SBIR program, as well as a workshop conducted by the National Research Council. We conducted this performance audit from June 2010 to August 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence we obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. A more detailed description of the scope of our review and the methods we used is contained in appendix I. Background: SBA coordinates and oversees the efforts of the 11 agencies currently participating in the SBIR program. SBA coordinates the agencies' schedules for issuing solicitations--announcements of opportunities for small businesses to apply for awards--and provides access to these solicitations through its Web site. As part of its oversight effort, SBA collects SBIR data from the participating agencies, aggregates the data, and uses the data to, among other things, monitor the program and report to Congress. SBA also provides guidance to participating agencies on the general conduct and operation of the program, which it periodically updates, for example, in response to changes in the program's authorizing legislation. Under the legislation and SBA's guidance, agencies have considerable flexibility to design their programs. For example, each agency determines, in consultation with SBA, such items as the number of solicitations to be issued during a fiscal year and the dates applications are due. Agencies also have discretion to determine what type of research to include in their solicitations, how to review applications for technical and scientific merit, which applications to fund, and the size of the award, among other things. The Small Business Innovation Development Act of 1982 provided for a competitive three-phased SBIR program. In phase I, participating agencies award up to $150,000 for a period of about 6 to 9 months to small businesses to conduct experimental or theoretical R&D. Small businesses whose phase I projects demonstrate scientific and technical merit, in addition to commercial potential, may compete for phase II awards of up to $1 million to continue the R&D for an additional period, normally not to exceed 2 years.[Footnote 8] Phase I and II award funds may be used for costs related to conducting the research, such as salaries, fringe benefits, equipment, and consulting services, as well as for profits and fees. To be eligible for a phase I or II SBIR award, a business must have 500 or fewer employees, be organized for profit with a place of business in the United States, and operate primarily in the United States or make a significant contribution to the U.S. economy. Generally, a business must also be at least 51 percent owned and controlled by one or more individuals who are U.S. citizens or permanent resident aliens.[Footnote 9] These eligibility requirements apply at the time that a phase I or II award is made. During phase III, businesses must secure non-SBIR funding to develop the commercial potential of the innovative technologies resulting from their SBIR projects;[Footnote 10] such funding may come from the private sector, federal agencies, or other sources. As the program has been reauthorized over the years, legislation has established a number of requirements related to the program's purposes. For example, the Small Business Research and Development Enhancement Act of 1992 directed SBA to make more information available about the SBIR program, particularly about participation by small businesses owned by disadvantaged individuals and women, and required that agencies increase their outreach to such businesses. In addition, the Small Business Reauthorization Act of 2000 directed that applicants for phase II SBIR awards be required to submit commercialization plans[Footnote 11], and it mandated that SBA develop, maintain, and make available to the public a database that contained SBIR award data. In addition, the act required SBA to develop and maintain, by June 2001, a restricted government-use database that would contain award-related data from the public database, as well as additional confidential data that would be accessible only to government agencies and other authorized users. The act stated that this database would be used exclusively for program evaluation--which, as we have noted in past work, involves the systematic collection and analysis of accurate, comparable, and complete data on program result[Footnote 12]s. The act required the government-use database to contain, among other things, data that applicants for phase II awards would be required to supply on the commercialization success of any prior phase II awards, such as data on sales of or additional investment in the technologies funded under the awards. The act further specified that the government-use database would contain annual updates to these data, which phase II award recipients would be requested to voluntarily provide for 5 years after the period covered by the award. To accomplish this mandate, SBA envisioned expanding an electronic database, known as Tech-Net, which it had developed in the late 1990s, into two sections: a public-use portion and a government-use portion containing commercialization data. The public-use portion of the database has been available since 2000, according to SBA, and it contains such award-related data as the phase of the award, amount of the award, name and location of the business receiving the award, an abstract of the work to be conducted under the award, and whether the business is categorized as owned by disadvantaged individuals or women. In October 2006, however, we reported that some SBIR agencies did not consistently provide or correctly format the awards-related data for several fields in the public-use portion of the database. [Footnote 13] For example, two of the eight agencies we reviewed had not consistently provided data on whether the businesses receiving the awards were categorized as owned by disadvantaged individuals or women.[Footnote 14] At that time, we also reported that SBA had not implemented the government-use portion of the database, primarily, according to SBA officials, because of increased security requirements for the database, agency management changes, and budgetary constraints. Additionally, we reported that while five of the agencies we reviewed had systematically collected commercialization data, their data collection efforts differed in ways that made it challenging to evaluate the program across agencies. In August 2009, we testified before Congress that SBA said the database would no longer accept incorrectly formatted awards-related data from participating agencies.[Footnote 15] A committee of the National Academy of Sciences' National Research Council has conducted a series of assessments of the SBIR program, both within and across agencies, as part of a legislatively mandated study. The results were summarized in a single report, in which the committee stated that SBIR is making significant progress in achieving congressional goals.[Footnote 16] The study concluded that the SBIR program is "sound in concept and effective in practice." The study also recommended changes that could make the program more effective. Among other things, the study recommended that SBA and participating agencies improve the collection of data that track participation in the SBIR program by businesses owned by disadvantaged individuals and women, develop targeted outreach to such businesses that is based on an analysis of factors that affect their participation, and improve documentation of commercialization success. The National Research Council is now undertaking another round of assessments to provide a second snapshot of the program's progress in achieving its legislative purposes. Agencies Addressed SBIR's Purposes through Solicitations, Technical Assistance, Matching Funds Programs, and Outreach: For fiscal years 2008 through 2011, the five participating agencies we reviewed addressed the SBIR purposes of using small business to meet federal R&D needs and stimulating technological innovation through their solicitations. Agencies also used solicitations, as well as technical assistance or matching funds programs, to address the SBIR purpose of increasing commercialization of innovations derived from federal R&D efforts. To address the remaining program purpose-- encouraging participation in technological innovation by small businesses owned by disadvantaged individuals and women--agencies relied mainly on outreach activities aimed at a broader audience. Agencies Addressed the SBIR Purposes Related to Meeting Federal Research Needs and Stimulating Innovation through Their Solicitations: All of the participating agencies that we reviewed designed the SBIR solicitations that they issued for fiscal years 2008 through 2011 to meet federal R&D or mission needs and stimulate technological innovation.[Footnote 17] All of these agencies selected research topics for their solicitations that were designed to meet their respective R&D or mission needs and specified that applications would be evaluated on the basis of responsiveness to those topics. The agencies that purchase SBIR-funded technologies for their own use-- DOD, DOE, and NASA--tended to select solicitation topics that met specific agency needs for R&D. For example, in fiscal year 2011, DOD solicited applications to develop a fuel cell system capable of converting ethanol into electricity in an efficient, small, lightweight, portable power system. According to the solicitation, such advanced fuel cell systems could provide soldiers power to complement batteries and to charge rechargeable batteries, reducing the number of batteries required for extended time in the field. In contrast, NIH and NSF, which generally do not purchase SBIR-funded technologies, tended to issue solicitations for a broader spectrum of R&D to support their missions of advancing biomedical and other scientific and engineering disciplines. Among the agencies we reviewed, NIH and its components gave applicants the most leeway in addressing agency needs: rather than limiting applications to specific research topics identified in solicitations, NIH and its components usually listed suggested topics and encouraged applicants to propose innovative projects that fit the agency's mission. Concerning innovation, each of the agencies included instructions in its SBIR solicitations about the type of information applicants had to provide about the innovativeness of the proposed work. For example, NASA informed phase I and II applicants that a competitive application would describe the proposed innovation relative to state-of-the-art knowledge in the field, among other things. In addition, these agencies explained to applicants how reviewers would consider evidence of the innovativeness of the applicants' proposed research approaches. For example, in its fiscal year 2010 solicitation, NSF stated that applications would be evaluated, in part, on the basis of whether they reflected state-of-the-art knowledge in the major research activities proposed and whether the work was likely to advance state-of-the-art knowledge. Agencies Worked to Increase Commercialization of Innovations through Solicitations, Technical Assistance, and Matching Funds Programs: The participating agencies we reviewed addressed the SBIR purpose of increasing commercialization of innovations through solicitations, as well as through technical assistance or matching funds programs. Solicitations. Of the five agencies we reviewed, all but NIH required in their solicitations for fiscal years 2008 through 2011 that applicants for phase I awards submit a commercialization strategy demonstrating that the applicants had taken steps such as identifying a market for their SBIR technologies, planning to secure financing, and estimating expected future sales. For phase II awards, all of the agencies we reviewed required that applicants submit a commercialization plan.[Footnote 18] In general, the solicitations we reviewed required that phase II commercialization plans discuss the potential market and competitors; the qualifications of key management and technical personnel; as well as financing, marketing, and manufacturing plans, among other things. The agencies we reviewed differed in their stated processes for evaluating the commercial potential of applications. For example, DOD guidance to applicants outlined a systematic process for how the agency would consider commercialization potential when evaluating applications submitted by small businesses that had received multiple prior awards. DOD indicated that, under this process, it would assign a commercialization achievement score to applicants that had completed the work for four or more phase II awards from any agency;[Footnote 19] this score would reflect how the applicants' commercialization experience compared with historical averages. Applicants whose scores fell within the lowest 20 percent would not be allowed to receive more than half the maximum number of points possible for commercialization potential, which was to be assessed on the basis of several factors, including the commercialization strategy or plan. DOD guidance stated that businesses with fewer than four completed phase II awards would not be affected by the absence of a commercialization achievement score. Although the other four agencies we reviewed did not outline as systematic a process for evaluating past commercialization success as a gauge of commercialization potential, they still indicated that commercialization potential would be taken into account in reviewing applications. For example, DOE's solicitation instructions encouraged phase I applicants to seek firm commitments for private-sector or non- SBIR federal funding prior to applying for a phase II award. The instructions further stated that phase II applicants that obtained such commitments were more likely to receive full credit for commercialization planning during the evaluation of their applications. In the case of NSF, solicitation instructions stated that proposals are usually reviewed by 3 to 10 outside experts in fields related to the proposal; according to NSF officials, these reviewers have business experience. NSF's solicitation instructions further stated that the agency would not review applications that lacked sufficient information on commercial potential. In 2010, two agencies we reviewed also issued SBIR solicitations under new programs that were explicitly oriented toward increasing commercialization. Specifically, in July 2010, DOE launched a program under which it solicited applications for phase III of SBIR, the commercialization phase. DOE documents indicated that the agency would make available approximately $30 million, including funding from the American Recovery and Reinvestment Act (Recovery Act),[Footnote 20] for phase III awards, which are intended to allow businesses to pursue commercial applications of work performed under phase I and II awards. In addition, NIH's National Cancer Institute began a program under which it solicited phase I applications to continue development of technologies that have originated in its laboratories, with the goal of advancing these technologies toward commercial products.[Footnote 21] SBA has designated the use of the SBIR program to encourage commercialization of agencies' internal research as a best practice on its SBIR Web site. Technical assistance. All five agencies included in our review provided technical assistance to help award recipients build their capacity to commercialize their technologies. To provide the assistance, the agencies contracted with vendors and consultants who have experience in bringing technologies to market. With the exception of NASA, the agencies supported the technical assistance at least in part through the use of SBIR funds.[Footnote 22] In fiscal years 2008 to 2010, DOD, DOE, NIH, and NSF spent SBIR funds on technical assistance for phase I award recipients. Some of the assistance was in the form of interactive training Webinars or online tools directed toward a broad spectrum of SBIR applicants and award recipients. For example, the Navy offered phase I award recipients the use of a software tool, known as WebTRIMS, that helps identify, quantify, and track risks associated with SBIR technology development and covers topics such as contracting strategies, business and transition planning, and manufacturing readiness. Other phase I assistance was more customized. For example, DOE offered phase I award recipients customized technical assistance designed to help them develop a commercialization plan complete with an implementation schedule and suggestions for product design. Similarly, on a first- come, first-served basis, NIH offered phase I award recipients assessments of their SBIR-funded technologies' likely niche in the existing commercial market, which could help recipients develop commercialization plans for phase II applications.[Footnote 23] Additionally, NSF offered phase I award recipients personalized mentoring and coaching sessions with an advisor. According to NSF officials, 92 percent of phase I recipients chose to participate in the technical assistance program in 2010. NASA did not provide technical assistance for phase I award recipients; NASA officials told us they believed technical assistance would have the most utility for phase II NASA award recipients. During at least a portion of the period we reviewed, all five agencies offered individualized technical assistance for phase II award recipients, although DOE curtailed such assistance in 2010, and NASA discontinued its assistance in 2008. Award recipients were selected for assistance on the basis of factors such as recommendations from SBIR program staff and the award recipients' potential for rapidly moving their technologies to phase III. The assistance consisted of in- depth training and one-on-one assistance from advisors and industry experts. For example, as part of its Commercialization Pilot Program, [Footnote 24] the Army assisted selected phase II SBIR award recipients in assessing commercialization potential, developing business plans, and matching their technologies with potential government and industry customers. At DOE, staff could nominate phase II award recipients for assistance in preparing to negotiate business deals, for example, including joint ventures and licensing agreements for use of their technologies. DOE officials told us that in 2010 the agency curtailed its use of SBIR funds for phase II technical assistance, spending such funds on assistance only for award recipients that had specifically budgeted for it in their applications. In 2007 and 2008, NASA partnered with the Navy to pilot a technical assistance program for NASA phase II recipients. The program was designed to help SBIR businesses develop a plan for transitioning to phase III, among other things. In 2007, 17 phase II companies with 19 SBIR projects participated in the program, and in the following year, 19 phase II companies with 20 SBIR projects participated. The program was not renewed for 2009; NASA officials told us that they believed the program was generally successful, but that they preferred to use SBIR funds to make larger awards.[Footnote 25] NIH offered selected current or past phase II award recipients the opportunity to work one-on-one with an advisor over a 9-month period to develop business plans to commercialize their technologies, as well as to prepare materials to help attract potential investors or partners. Since 2004, almost 700 award recipients have received the assistance, including the 80 award recipients currently participating, according to NIH officials. Finally, NSF offered customized assistance to phase II award recipients through its Innovation Accelerator Initiative. According to NSF officials, through this initiative, award recipients received help in connecting with potential investors and negotiating company acquisitions and mergers. NSF officials told us that, in 2010, approximately 33 percent of NSF's phase II recipients received this assistance. In some cases, agencies that we reviewed used non-SBIR funds to broaden the scope of the technical assistance they provided to help award recipients commercialize their technologies.[Footnote 26] For example, DOD used non-SBIR funds to host its annual Beyond Phase II Conference and Technology Showcase, a 3-day event that features matchmaking sessions with SBIR award recipients and prime contractors. Similarly, the Navy used non-SBIR funds to maintain databases with advanced searching capability to help award recipients identify potential business partners. The Navy also used non-SBIR funds for its Transition Assistance Program, which provides individualized help with commercialization planning, culminating in a conference designed to facilitate interaction with potential business partners. Moreover, in 2011, the National Cancer Institute launched its Regulatory Assistance Program using non-SBIR funds. According to agency officials and information from the agency's Web site, this program provides SBIR award recipients time with consultants experienced in various regulatory requirements--such as those for anticancer therapies, imaging technologies, and medical devices--to prepare strategies for obtaining regulatory approvals required before the technologies can be commercialized. The National Cancer Institute also used non-SBIR funds to support its Investor Forum, which provides competitively selected SBIR award recipients an opportunity to showcase their technologies and enter into discussions with the biotech investment community. In 2010, 14 award recipients that were selected on the basis of strength of research, impact on cancer, product development, and market potential participated in the forum along with more than 175 potential investors, according to the agency's Web site. Matching funds programs. Through matching funds programs, agencies provide additional SBIR funds to award recipients that obtain monetary commitments above certain thresholds from outside investors.[Footnote 27] SBA has designated matching funds programs as a best practice on its SBIR Web site, and all of the agencies we reviewed except DOE have established such programs.[Footnote 28] For example, for award recipients that obtain a minimum of $100,000 from an outside investor, NSF will match up to 50 percent of the outside investment for a maximum of $500,000 in NSF matching funds. NASA and NIH officials said that matching funds programs encourage outside investment during the early stages of R&D--a time when many investors are reluctant to invest. In particular, officials at the National Cancer Institute said that matching funds can help attract outside investment because they can be used as leverage to increase investors' potential returns. DOD and NSF offer matching funds to award recipients at the end of phase I and during phase II, while NASA and the National Cancer Institute offer matching funds during phase II. DOE has not established a matching funds program for its SBIR program. DOE officials told us, however, that they are exploring whether to do so and have held discussions with other SBIR participating agencies about their matching funds programs. Officials at DOD, NASA, and NIH said they have not collected data to compare the commercialization success of recipients that received matching funds with the success of those that did not.[Footnote 29] NSF conducted a study to assess the effect of the $18 million in matching funds it invested in fiscal year 2006 for 48 phase II award recipients that had raised a total of $58 million from outside investors[Footnote 30]. According to NSF officials, results of this study showed that, in the 5 years following the start of these phase II projects, 70 percent of recipients that had received matching funds achieved commercial success compared with a 30 percent success rate for recipients that had not received such funds. Agencies Generally Encouraged Participation by Disadvantaged Individuals and Women through Outreach Activities Directed toward a Broader Audience: SBA's guidance states that small businesses owned by disadvantaged individuals and women must compete for SBIR awards on the same basis as all other small businesses. However, to meet requirements for greater outreach to small businesses owned by disadvantaged individuals and women, SBA has encouraged participating SBIR agencies to reach out to such businesses and to develop methods that encourage their participation. SBA has also raised the topic of outreach during recent quarterly meetings of agency SBIR program managers.[Footnote 31] Officials at all of the agencies we reviewed told us they generally reach out to such businesses through activities directed toward a broader audience, such as by attending SBIR national conferences and industry-sponsored events and by sharing information via Web sites or e-mail lists. Agency officials also noted that they try to accommodate requests for speakers at events sponsored by, or likely to be attended by, small businesses owned by disadvantaged individuals and women--for example, events sponsored by trade organizations for minority-or women-owned businesses. However, officials from some trade organizations for businesses owned by disadvantaged individuals and women told us that the outreach of agencies we reviewed was often ineffective in educating the organizations' members about the SBIR program. Of the agencies we reviewed, NIH and NSF have made specific efforts, including the following, to improve their outreach: * For fiscal years 2010 and 2011, NIH developed a goal to increase awareness of its SBIR program among businesses owned by disadvantaged individuals and women, and it outlined specific activities aimed at reaching this goal. * Both NIH and NSF offered various fellowships for postdoctoral research conducted by disadvantaged individuals and women; these fellowships were available to support SBIR projects, as well as other research. * In 2010, NSF assigned a full-time staff member to help it develop a plan to increase participation in SBIR by businesses owned by disadvantaged individuals and women in response to a recommendation from its SBIR advisory committee. * Through a review of academic literature, as well as informal polling of NSF applicants and award recipients, NSF has identified barriers to SBIR participation by small businesses owned by disadvantaged individuals and women, NSF officials told us. These barriers include disparities in the owners' levels of education and access to capital compared with those of other entrepreneurs. To address identified barriers, NSF is, among other things, establishing partnerships with industry and academia to expose African American, Latino, and other college students to entrepreneurship in scientific and technical fields, according to NSF officials. Evaluation of the effectiveness of agencies' outreach efforts is hindered by a lack of accurate and complete data. Although SBA collects data on the number and dollar value of awards to small businesses owned by disadvantaged individuals and women, SBA officials told us that they cannot accurately tabulate data on such awards, particularly awards to women-owned businesses, because of inconsistencies in the data on business ownership. According to the officials, SBA has taken steps to correct the inconsistencies for data submitted after 2006 but has not done so for earlier years. Moreover, SBA does not collect data on the number of applications submitted by businesses owned by disadvantaged individuals and women. As a result, SBA's data do not allow for an examination of trends in the submission of applications from such businesses, analysis of the percentage of applications from these businesses that lead to awards, or correlation of these trends and percentages with outreach efforts. SBA officials told us in March 2011 that they were considering whether their database should include information on the numbers of applications submitted by these businesses. Comparable Data Are Not Available across Participating Agencies to Evaluate Progress in Increasing Commercialization of SBIR Technologies: SBA has not yet developed the government-use portion of its database for collecting comparable commercialization data on SBIR technologies, but it is taking steps to do so. In the interim, agencies have, for their own purposes, independently gathered commercialization data that are not comparable; the accuracy of these data is largely unknown. Implementing the government-use portion of the database should improve the comparability of the data. However, programwide evaluation of progress in increasing commercialization may continue to be impaired by long-standing challenges. SBA Has Not Implemented the Government-Use Portion of Its Database Intended to Collect Comparable Commercialization Data for Program Evaluation but Is Taking Steps to Do So: As of June 2011, SBA had not met the legislative mandate to develop and implement, by June 2001, a government-use database that can provide data on commercialization for evaluating the SBIR program. [Footnote 32] However, the agency's efforts to develop such a database recently gained additional prominence and resources. Specifically, SBA linked development of the government-use portion of its database to one of the agency's high-priority performance goals for fiscal years 2011 and 2012. Additionally, in September 2010, SBA allocated $1.4 million in Recovery Act funds to hire a new contractor to develop the government-use portion's capacity to accept commercialization data submitted by participating SBIR agencies and award recipients, as well as to make other improvements to the database. For example, SBA said that it has been working with the contractor to consolidate data on previous awards. SBA officials said that past award recipients have been assigned unique identifiers that will be used to track awards issued to those recipients over the lifetime of the SBIR program; unique identifiers are also to be assigned to small businesses newly entering the program. In the future, SBA intends for the unique identifiers to allow agencies to validate business information by comparing it against information in other federal databases such as the Central Contractor Registration database, which contains information on businesses that want to contract with the federal government.[Footnote 33] SBA officials told us that they expect to implement the government-use portion of SBA's database by August 2011 and to provide for its basic maintenance and support despite reductions in the agency's overall budget.[Footnote 34] The government-use portion is intended to allow both participating agencies and award recipients to enter commercialization data in a comparable format to assist in program evaluation. SBA officials told us that they have worked with participating agencies to develop common metrics for commercialization data, as well as a standardized data collection instrument that will accommodate the various types of SBIR technologies the agencies fund to meet their different missions. These metrics, which will correspond to fields in the database, include the following: * indication of whether an award resulted in a commercialized technology and whether other SBIR awards contributed to commercialization of the technology; * estimated sales; * estimated investment (other than SBIR funding); * any patents applied for or received related to the award; and: * any initial public offering, merger, or sale of the business that resulted, at least in part, from the award. SBA officials told us in May 2011 that they plan to implement the metrics and data collection instrument in August 2011.[Footnote 35] SBA is requesting that participating agencies voluntarily begin entering historical commercialization data into the government-use database before August 2011. To facilitate this process, SBA is working with its contractor to ensure that historical agency data can be matched to fields in the new database. Nevertheless, officials from SBA and participating SBIR agencies said that some agencies may not enter historical data or may be delayed in doing so because they either did not collect such data or do not have the data in electronic form. For example, NASA officials stated that much of their commercialization data are stored in paper format and expressed doubt that the agency would be able to convert the data into the required format for entering by SBA's August deadline. SBA officials also told us that, after the government-use portion of the database is available, some agencies may instruct applicants and award recipients to submit their commercialization data directly into the database. Other agencies, such as DOD, may continue to require applicants and recipients to submit commercialization data directly to the agencies, which would then upload the data into the database. As of May 2011, SBA officials were unsure which approach agencies would take, noting that agencies may wait to see how the database works before making a decision. Without the Government-Use Portion of the SBA Database, Agencies Have Independently Collected Commercialization Data That Are Not Comparable: In the absence of the government-use portion of SBA's database, the five participating SBIR agencies we reviewed have independently collected commercialization data that are not comparable. The agencies collected these data using various methods for their own purposes, as summarized in table 1. Table 1: Summary of Five SBIR Agencies' Efforts to Gather Commercialization Data: Agency: DOD; Data collection instrument: Commercialization report submitted as a component of applications for SBIR awards[A]; Dates administered: 2000 to present; Population queried: Applicants for phase I and phase II awards; Purpose of data collection: To assess an applicant's ability to commercialize SBIR technologies, which DOD considers when selecting projects for funding, and to assess the extent to which DOD's SBIR- funded technologies are commercialized. Agency: DOE; Data collection instrument: Annual survey; Dates administered: 1986-2007[B]; Population queried: Award recipients with active phase II awards[C]; Purpose of data collection: To assess the commercialization success of DOE's SBIR-funded projects. Agency: NASA; Data collection instrument: Annual survey; Dates administered: 1997-2002[D]; Population queried: Award recipients that received a phase II award from 1983 through 1996; Purpose of data collection: To assess the effect of changes that NASA made in 1995 to its SBIR program management and application review process--i.e., the effect of these changes on the success of award recipients in commercializing technologies and in integrating those technologies into the agency's mission programs. Agency: NIH; Data collection instrument: Periodic survey; Dates administered: 2002 (with follow-up surveys of 2002 respondents in 2004, 2005, and 2007); 2008; Population queried: 2002 and follow-up surveys: award recipients that received phase II awards from 1992 through 2001; 2008: award recipients that received phase II awards from 2002 through 2006; Purpose of data collection: To evaluate the extent to which NIH's program met the overall program goals, including the extent to which it increased commercialization of SBIR technologies that the agency funded. Agency: NSF; Data collection instrument: Periodic survey; Dates administered: 2005-present; Population queried: Phase II award recipients marking the 3rd, 5th, and 8th anniversary of the receipt of their awards; Purpose of data collection: To help assess the commercialization success of NSF's SBIR program. Source: GAO analysis of agency data. [A] Small businesses applying for DOD SBIR awards must report on the commercialization history of any prior phase II SBIR awards received. [B] According to DOE officials, they discontinued the annual survey for a combination of reasons, including contractor staff changes and resource constraints. However, DOE has indicated that it expects to resume surveying in fall 2011 using an improved survey instrument. [C] DOE award recipients were surveyed during the years in which their phase II awards were active and for up to 5 years after the awards ended. [D] According to NASA officials, the agency has not conducted a follow- up survey owing to a lack of resources, although the agency has continued to link its survey instrument to its SBIR solicitations for applicants to complete voluntarily. However, agency officials noted that response rates have been low. [End of table] In conducting their data collection efforts, agencies differed in the extent to which they asked award recipients to do the following, among other things: * Identify the type of customer and the amount of sales or further investment for SBIR-funded technologies. For example, most agencies asked award recipients to report federal and nonfederal sales separately, but NIH and NSF asked award recipients to report combined sales. * Account for indirect sales and nonfinancial indicators of commercialization. NASA, NIH, and NSF asked award recipients to indicate whether an SBIR-funded technology had resulted in licensing agreements with other businesses to sell the technology, while DOD and DOE did not ask that question. NASA further asked award recipients to estimate the financial value of such agreements, while the other agencies did not. Similarly NASA, NIH, and NSF asked award recipients to indicate whether specific SBIR-funded technologies had resulted in patents, while DOD and DOE asked award recipients to report the total number of patents resulting from all their SBIR awards. * Quantify the dollar values of cumulative sales. While most agencies asked award recipients to report a specific dollar amount in cumulative sales resulting from their SBIR-funded technologies over a period of time, NIH asked award recipients to report such sales by choosing among ranges, beginning with "$50,000 or less" and extending to "$50,000,000 or more." Because NIH has reported cumulative sales in ranges rather than specific dollar amounts, comparing its results with those reported by other agencies is difficult. While each agency's data collection efforts resulted in, among other information, estimates of total or average sales of SBIR technologies, differences in the agencies' data collection efforts make it difficult to compare results across agencies. The following are examples of commercialization data reported by agencies: * DOD estimated that commercialization of SBIR technologies that it funded generated federal and nonfederal sales and non-SBIR funding of $22 billion on a program investment of $11 billion from 2000 through March 2010.[Footnote 36] * DOE estimated that, from 1986 through 2007, SBIR technologies developed by recipients of phase II awards resulted in a total of $2.4 billion in federal and nonfederal sales and $1.6 billion in non-SBIR investment. On average, award recipients reported receiving more than $3 million in sales related to SBIR-funded technologies. During the same period, DOE reported that it had invested $1.6 billion in phase I and II SBIR awards. * NASA estimated that, as of 2002, SBIR technologies developed by award recipients that received a phase II award from 1983 through 1996 had generated approximately $2.8 billion in federal and nonfederal sales and non-SBIR funding compared with $1.1 billion in SBIR investment from NASA. * In NIH's 2002 survey, which covered 1992 through 2001, 27 percent of respondents reported an estimated total of $821 million in sales of SBIR technologies; the other respondents did not report any sales. [Footnote 37] NIH estimated that it invested $2.2 billion in phase I and phase II awards from 1992 through 2001. For the 2008 survey, which covered 2002 through 2006, 33 percent of respondents reported an estimated total of $396 million in federal and nonfederal sales of SBIR technologies.[Footnote 38] NIH estimated that it invested $2.7 billion in phase I and phase II awards from 2002 through 2006. NIH was the only agency we reviewed that reported sales lower than its SBIR investment for the periods it examined. According to NIH officials, many of the technologies that the agency supports through its SBIR program, such as drugs and medical devices, take longer to commercialize than those funded by other agencies because of the need for extensive clinical testing and regulatory approval. * NSF officials estimated that recipients marking the eighth anniversary of the receipt of their awards from July 2005 through May 2010 had realized a total of $1.05 billion in commercial revenue. [Footnote 39] NSF estimated that it invested $628 million in SBIR awards during roughly the same period.[Footnote 40] Further, with the exception of DOD, agencies we reviewed generally did not take steps to verify commercialization data that they received from award recipients, so the accuracy of the data is largely unknown. As officials from some of the agencies in our review noted, award recipients may have an incentive to overstate their commercialization success in the hope of improving their prospects of receiving future SBIR awards. While SBA has worked with SBIR agencies to identify best practices in other areas of SBIR program management, it has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Without consistent practices for verifying the accuracy of these data, the usefulness of the government- use portion of SBA's database as a tool for evaluating the SBIR program's success in increasing commercialization may be limited. To verify the accuracy of award recipients' commercialization data, DOD performs an annual review of all projects in its Company Commercialization Database, which contains the commercialization data it gathers from award recipients. This review includes checks to ensure that prior award recipients applying for new awards are not reporting the same project results more than once, substituting the results of one project for that of another, or incorrectly reporting sales to third parties. According to DOD officials, after its 2010 review, the agency sent approximately 300 e-mail queries to applicants whose reported commercialization data were identified as having potential problems. The officials said that applicants that do not respond to such queries are blocked from submitting further applications until concerns related to their commercialization reports are addressed. Even with these verification activities, however, Army officials expressed concern to us about the accuracy of the applicants' self-reported commercialization data; these officials stated their preference for using data from the Federal Procurement Data System, which contains government information on federal contracts, including sales.[Footnote 41] Moreover, a Navy official acknowledged the possibility that additional verification activities, such as selective spot visits to SBIR award recipients, could further deter recipients from misrepresenting their commercialization success, although he noted that such activities would compete with other administrative priorities. Similarly, officials from DOE and NIH stated that additional verification activities would be useful but also said that they needed to devote program administration resources to higher priority activities, such as preparing solicitations and supporting review panels for applications.[Footnote 42] Implementing the Government-Use Portion of the Database Should Improve the Comparability of Commercialization Data, but Long-standing Challenges May Still Impair Evaluation of Commercialization Progress: SBA's implementation of the government-use portion of its database should improve the comparability of commercialization data available for programwide evaluation. Nevertheless, long-standing challenges may continue to impair programwide evaluation of progress in increasing commercialization of SBIR-funded technologies. As we reported in October 2006, notable among these challenges is that prior award recipients that are no longer participating in the SBIR program are not required to provide updated commercialization data and may prefer not to do so.[Footnote 43] For example, DOD indicated in written comments to us that, from 2008 to 2010, 46 percent of nonparticipating prior phase II award recipients did not provide updates despite DOD's request that they update commercialization data annually after their awards ended.[Footnote 44] Similarly, in a report on its 2002 survey, NASA observed that many recipients of multiple awards elected not to respond to its survey despite "extensive telephone follow-up" and that many recipients that ultimately responded "would likely have preferred not to."[Footnote 45] Some award recipients may be reluctant to provide commercialization data because the data are business- sensitive. SBA officials told us that mechanisms to require or encourage nonparticipating recipients to report their data need to be explored. A NASA official told us that effective incentives to encourage wider voluntary reporting might include publicizing commercial success or giving monetary prizes for success. The difficulties agencies face in persuading prior award recipients to volunteer commercialization information can be compounded by challenges in maintaining contact with them. Specifically, prior award recipients can change names or personnel, go out of business, or be sold during the 10 or more years that it can take for an SBIR-funded technology to reach the marketplace. In NIH's 2002 survey of award recipients, for example, the portion of the sample that was "unusable"--a group that consisted primarily of recipients that no longer existed or could not be found--increased from 2 percent in the first year after the end of the award to 52 percent in the tenth year. Programwide evaluation--particularly efforts to compare commercialization success across agencies--can also be complicated by differences in the time required to commercialize various types of SBIR-funded technologies. Comparing agencies' commercialization results at a given point in time may not present a true picture of each agency's success because some agencies fund technologies that are relatively close to being market-ready while others fund technologies that need more extensive development or regulatory approval. Furthermore, as we have previously reported, the SBIR program's other goals remain important, and comparisons that focus on commercialization may not adequately take into account progress toward these goals.[Footnote 46] For example, one agency official told us that some SBIR-funded technologies, such as those related to national security, may never have great commercial potential but are important to the agency's mission. Conclusions: DOD, DOE, NASA, NIH, and NSF have designed their SBIR solicitations to address the program's purposes of using small businesses to meet federal R&D needs, stimulating technological innovation, and increasing commercialization of innovations derived from federal R&D efforts, and they have further addressed commercialization by providing technical assistance or matching funds to award recipients. These agencies have also conducted outreach and other activities to address the SBIR purpose of encouraging participation in technological innovation by small businesses owned by disadvantaged individuals and women. However, evaluation of progress in achieving the program's purposes is impeded by a lack of accurate, comparable, and complete data on program results. For example, it is difficult to evaluate the program's effectiveness in encouraging small businesses owned by disadvantaged individuals or women to participate in technological innovation because SBA does not collect data on the number of applications submitted by such businesses. It is also difficult to evaluate the program's effectiveness in increasing commercialization of SBIR-funded technologies because, although agencies participating in the program have gathered commercialization data for their own purposes, comparable data on commercialization are not available across agencies. SBA's planned implementation of a government-use portion of its database should go some way toward improving the comparability of the commercialization data as they are systematically collected using common metrics. However, the commercialization data that the database is intended to contain are largely self-reported by award recipients that may have an incentive to overstate their commercialization success. DOD has adopted practices for verifying the accuracy of commercialization data it collects from prior award recipients, but most of the participating agencies we reviewed did not verify the accuracy of commercialization data from their prior award recipients, and SBA has not identified best practices for participating agencies to use in doing so. As long as participating agencies do not consistently verify the accuracy of commercialization data, the usefulness of the government-use portion of SBA's database as a tool for evaluating the SBIR program's success in increasing commercialization may be limited. Recommendations for Executive Action: To build upon efforts to implement a government-use database for program evaluation, we recommend that the Administrator of the Small Business Administration work with participating SBIR agencies to take the following two actions: * collect data on the number of applications submitted by small businesses owned by disadvantaged individuals and women, and: * identify best practices for verifying the accuracy of data related to progress in increasing commercialization. Agency Comments and Our Evaluation: We provided a draft of this report to SBA, the Departments of Defense and Energy, the National Aeronautics and Space Administration, the National Institutes of Health, and the National Science Foundation for review and comment. SBA generally agreed with our findings as well as our recommendations, which it offered an action plan to address. Specifically, with respect to our first recommendation, SBA stated that, beginning in fiscal year 2012, it plans to use its database to collect information from agencies about applicants that did not receive awards--information that could include whether the applicants were small businesses owned by disadvantaged individuals or women. Further, SBA indicated that it plans to hold a workshop in fall 2011 for participating SBIR agencies to share best practices for reaching out to small businesses owned by disadvantaged individuals. According to SBA, the workshop should result in a commitment from agencies to develop baselines for numbers of applications from such businesses. Regarding our second recommendation, SBA indicated that it will seek to identify best practices and methods for verifying the accuracy of commercialization data and will work with agencies toward implementation of those practices and methods. SBA also noted that its effort to collect commercialization data is intended to establish a baseline, against which SBA can review progress in increasing commercialization. SBA's letter conveying its comments is contained in appendix II. Among the SBIR participating agencies that we reviewed, DOE and NSF concurred with our recommendations and provided general comments, which are included in appendixes III and IV, respectively. Both DOE and NSF also made technical comments, which we have incorporated into our report as appropriate. In its general comments, DOE stated that it collects information on the number of applications submitted by small businesses owned by disadvantaged individuals and women and is willing to report the data to SBA. DOE further stated that it does not verify commercialization data because of resource limitations--not a belief that verification is of limited value--and it expressed an interest in learning about best practices for verification of these data. In addition, DOE commented that, until universal metrics are identified for measuring the success of SBIR programs across agencies, the compatibility of available data among agencies will remain a secondary concern. NSF stated that it concurs with the underlying goals of our recommendations. Moreover, NSF affirmed its commitment to implementation of a government-use database for program evaluation, collection of data on participation in small business innovation, and identification of best practices for verification of commercialization data. The remaining agencies--DOD, HHS, and NASA--neither agreed nor disagreed with our recommendations but provided technical comments, which we have incorporated into our report as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, Administrators of SBA and NASA, Secretaries of Defense and Energy, Directors of NIH and NSF, and other interested parties. In addition, this report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V. Signed by: Frank Rusco: Director, Natural Resources and Environment: [End of section] Appendix I: Scope and Methodology: In conducting this study, we reviewed Small Business Innovation Research (SBIR) program-related activities of the Small Business Administration (SBA) and 5 of the 11 SBIR participating agencies--the Department of Defense (DOD), Department of Energy, National Aeronautics and Space Administration, Department of Health and Human Services' National Institutes of Health (NIH),[Footnote 47] and National Science Foundation (NSF). For the two agencies with the largest SBIR budgets--DOD and NIH--we reviewed program activities conducted by the three participating subcomponent agencies with the largest SBIR budgets because some key activities are carried out at that level. Specifically, for DOD, we examined the SBIR programs of the Army, Air Force, and Navy, and for NIH, we examined the programs of the National Institute of Allergy and Infectious Diseases; the National Cancer Institute; and the National Heart, Lung, and Blood Institute. The five participating agencies we reviewed accounted for about 96 percent of the total dollars awarded by the program in fiscal year 2009. We reviewed applicable laws and regulations and literature on the SBIR program, including our prior reports and assessments by a committee of the National Academy of Sciences' National Research Council. To obtain further context for our review, we attended two national conferences and a National Research Council workshop on the SBIR program, and we interviewed National Research Council staff with program expertise. More specifically, to determine how participating agencies have addressed the SBIR program's four overarching purposes when implementing their programs, we reviewed SBA documents and data, including SBA's policy directive on implementation of the SBIR program, minutes from selected meetings of SBA and SBIR program directors, SBA's SBIR annual report for fiscal year 2008 (the latest year for which an annual report was available), and data on the dollar value of SBIR awards by participating agencies in fiscal year 2009 (the latest year for which SBA could provide the data). We examined relevant documents from participating agencies for fiscal years 2008 through 2010, and for fiscal year 2011 when possible. Documents we reviewed included solicitations for applications issued by each of these agencies, instructions to applicants, minutes from meetings of SBA and SBIR program directors, performance plans and reports, descriptions of commercialization assistance provided to SBIR awardees, and minutes from meetings of agency advisory committees. In addition, we also identified and interviewed SBIR program officials at each agency and officials responsible for implementing programmatic goals. For these interviews, we asked a standard set of questions to help ensure that we obtained consistent information about the SBIR programs at each of the agencies. We also interviewed inspector general staff at NSF, which facilitated SBIR-related activities conducted by the Council of Inspectors General on Integrity and Efficiency. Finally, we interviewed representatives of trade associations about their views of the SBIR program. We selected the trade associations on the basis of their familiarity with the program, the technologies on which they focus, and whether their membership includes small businesses owned by disadvantaged groups and women. The views of the representatives of these associations cannot be generalized to other associations. To determine the extent of SBIR program data available to evaluate progress in increasing commercialization of SBIR technologies, we reviewed documents related to SBA's SBIR database, including terms of work, work schedules, and proposed guidance related to the development of the government-use portion of the database. For the five SBIR participating agencies whose programs we reviewed, we examined documents dating from 2002 through 2011; these documents reflected commercialization data for SBIR award recipients that had received awards from 1983 (the first year in which agencies issued SBIR awards) through 2010. The documents we reviewed included surveys and other data collection instruments that the agencies used to gather commercialization information from award recipients; reports on data collection results, including any information on SBIR award spending during the years corresponding to those covered in each of the commercialization data collection efforts; and anecdotal descriptions of commercialization success. We also reviewed agency solicitations from fiscal years 2008 through 2010--and for fiscal year 2011 when possible--that contained reporting requirements for award recipients. We interviewed officials at SBA and each of the five participating agencies included in our review to obtain information on the specific commercialization metrics they use to monitor the commercialization experience of award recipients, the history of each agency's data collection efforts, and the agencies' experience in obtaining such information from current and past award recipients. We conducted this performance audit from June 2010 to August 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence we obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Small Business Administration: U.S. Small Business Administration: Washington, DC 20416: July 28, 2011: Mr Frank Rusco: Director, Natural Resources and Environment: Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear MI Rusco: Thank you for your letter dated June 30, 2011, seeking the U.S. Small Business Administration's (SBA's) comment on the U S Government Accountability Office (GAO) draft report entitled: "Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation." We do not have any issue with the facts and findings presented in the draft report. In addition, we generally agree with the recommendations, and offer the following plan of action in response. With respect to the first recommendation: "Collect data on the number of applications submitted by small businesses owned by disadvantaged individuals and women," we plan the following actions: * The SBA currently collects the number of awards and aggregate amount of awards to women, minority-owned, and HUBZone small business concerns for the SBA-issued Annual Report to Congress. * In the new SBA database at SBIR goy, we plan on collecting non- awardee information at an applicant-level from agencies. This information could include designations of women-owned, minority-owned, and HUBZone small business concerns. Additionally, this information on non-awardees—-as stated in the report—-can be compared with information in the Central Contractor Registration database in order to determine whether the applicant is a women-owned, minority-owned or HUBZone small business concern We anticipate collecting this information in FY2012, to allow agencies to organize their databases in line with our new data collection schema. * This fall, the SBA plans to hold a workshop to share best practices in outreach to socially and economically-disadvantaged groups to apply to the SBIR program. The outcome of the workshop should be a commitment from agencies to baseline their current level of applicant and awardee activity from socially and economically-disadvantaged groups. If appropriate, agencies should make commitments to obtain applicant numbers from these groups. With respect to the second recommendation: "Identify best practices for verifying the accuracy of data related to progress in increasing commercialization," we plan the following actions: * We will look to identify Best Practices and verification methods and work with agencies toward their implementation. * While our plan is to collect commercialization metrics, the recommendation also addresses the tracking of such information to determine progress in "increasing commercialization" The current effort for data collection is to establish a baseline of such information for the program. Once that it is established, we can review the progress of increasing commercialization. SBA looks forward to working with you and the SBIR participating agencies. If you have any questions, please contact Shawn McKeehan at 202-205- 7729. Sincerely, Signed by: Sean J. Greene: Associate Administrator for Investment: [End of section] Appendix III: Comments from the Department of Energy: Department of Energy: Office of Science: Washington, DC 20585: July 27, 2011: Mr Franklin Rusco: Director, Natural Resources and Environment: Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Rusco, Thank you for the opportunity to comment on the draft Government Accountability Office (GAO) report entitled, "Small Business Innovation Research: SBA Should Work with Agencies to improve the Data Available for Program Evaluation" (GAO-11-698). We have reviewed the draft report and provide general comments below. The comments provided here have been coordinated with other relevant offices of the Department of Energy (DOE). The GAO was charged by the Energy and Environment and the Technology and Innovation Subcommittees of the House Committee on Science and Technology "to determine (1) how participating agencies have addressed the SBIR program's four overarching purposes when implementing their programs and (2) the extent of SBIR program data available to evaluate progress in increasing commercialization of SBIR technologies." We recognize that conducting a review of the five-noted SBIR agencies and associated activities is an enormous undertaking, and in particular, we appreciate the time and thoroughness the GAO took to review the Department's SBIR program. With regard to the two specific recommendations provided by GAO, we concur with these recommendations. DOE does collect data on the number of applications submitted by small businesses owned by disadvantaged individuals and women and is willing to include this data in its reporting to SBA. DOE is also interested in learning about the best practices for verifying the accuracy of data related to progress in increasing commercialization. It should be noted that the reason DOE does not currently verify commercialization data stems from limitations on resources and not from a belief that verification is of limited value. In addition, we would like to provide one comment regarding the second charge to GAO. The GAO report discusses the variations in commercialization data collected by the agencies and the challenges that result when trying to do program-wide assessment. Implicit in this discussion is the idea that there is one set of metrics that can be used to compare program success across all participating agencies. Agencies have different missions, and as noted in the report, also include non-financial factors, (e.g. national security) in assessing their SBIR programs. The GAO report does not address the underlying questions of whether (I) there exist a universal set of metrics for measuring the success of SBIR programs across multiple agencies and (2) what those universal metrics should be. Until these questions are addressed, the compatibility of the available data among agencies will remain a secondary concern. With regard to the draft report's content, please find attached specific comments and suggested edits for your consideration. Thank you, again, for the opportunity to provide comments on this draft report. If you have any questions or concerns, please call Manny Oliver at (301) 903-0309. We look forward to receiving your final report. Sincerely, Signed by: Patricia M. Dehmer: Deputy Director for Science Programs: DOE Office of Science: Attachment: [End of section] Appendix IV: Comments from the National Science Foundation: National Science Foundation: Office Of The Director: 4201 Wilson Boulevard, Room 1270: Arlington, Virginia 22230: Tel. 703-292-8040: Fax. 703-292-9040: July 29, 2011: Mr, Franklin Rusco: Director, Natural Resources and Environment: United States Government Accountability Office: Washington, DC 20548: Dear Mr. Rusco: The National Science Foundation (NSF) appreciates the opportunity to review and comment on the Government Accountability Office's (GAO's) draft report entitled Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation (GAO-11-698). NSF has a long history of participation in the Small Business Innovation Research (SBIR) program dating back to the 1980s and has strongly supported the program's overarching purposes (i.e., to meet federal R&D needs; to stimulate technological innovation; to increase commercialization of federal R&D-based innovations; and to encourage participation by businesses owned by women and disadvantaged individuals). NSF has been proactive in addressing the program's objectives through: * broad topical solicitations that emphasis commercial potential; * mentorship of its small business clients by program directors with extensive business and technical backgrounds; * technical assistance in commercialization planning; * matching funds to stimulate investment in innovative projects; * supplemental awards and outreach activities to encourage members of underrepresented groups to participate in the small business innovation process; and; * a proactive process to capture the commercial outcome of NSF Phase II awards over time. NSF concurs with the underlying goals of the GAO's recommendations—to enhance the collection and analysis of SBIR program data—and endorses the importance of sharing best practices for verifying data accuracy. NSF is committed to collaborate with the Small Business Administration and the other SBIR-participating agencies in the: * implementation of a government-use data base for program evaluation particular focused on commercial outcomes; * collection of data on the participation of underrepresented individuals in the small business innovation process; and; * identification of best practices for verification of data related to commercialization outcomes. NSF commends the GAO audit team for an outstanding job of capturing the characteristics and features of the SBIR process at NSF, as reflected in the GAO-developed statement of facts and in this draft report. In reviewing the draft report, a few inaccuracies were identified; therefore, the following amendments to the draft report are recommended. [Page and footnote numbers in the draft report may differ from those in this report] Page 7, Footnote 17. The report on the NRC Assessment of the SBIR Program at NSF, published in 2008, should be noted (consistent with similar reports for other agencies that are cited in the footnote). Propose the following, "...prepublication in November 2007, the DOE and NASA reports were released in prepublication in June 2008 and December 2008, respectively, and the NSF report was issued in 2008." Page 23, Line 8. The provided dollar figure of "$1.5 billion" that appears in the draft is incorrect. It should read..."evaluated from July 2005 through May 2010 had realized a total of $1.05 billion in commercial revenue." Page 23, Footnote 39. As written, the footnote introduces several inaccuracies. A revised version is offered. "NSF's broad estimate of commercial revenue includes sales revenue from products and/or services that are a consequence of the award, licensing revenue, revenue from mergers or sales of award recipients' businesses that resulted, at least in part from the awards. NSF's estimate does not include venture capital funding." Thank you for the opportunity to comment on this draft report. If you have any questions regarding this response, please contact Kathryn Sullivan at 703-292-7375. We look forward to receiving your final report. Sincerely, Signed by: Subra Suresh: Director: [End of section] Appendix V: GAO Contact and Staff Acknowledgments: GAO Contact: Frank Rusco (202) 512-3841 or ruscof@gao.gov: Staff Acknowledgments: In addition to the individual named above, key contributors to this report include Cheryl Williams, Assistant Director; Antoinette Capaccio; Stephen Carter; Nancy Crothers; Laurie Ellington; Cindy Gilbert; Cynthia Norris; Christine Senteno; and Kiki Theodoropoulos. [End of section] Footnotes: [1] For the SBIR program, SBA has defined commercialization as the "process of developing marketable products or services and producing and delivering products or services for sale (whether by the originating party or by others) to Government or commercial markets." Small Business Innovation Research Program Policy Directive, 67 Fed. Reg. 60,072, 60,083 (Sept. 24, 2002). [2] Generally, socially disadvantaged individuals are those who have been subjected to racial or ethnic prejudice or cultural bias within American society because of their identities as members of groups and without regard to their individual qualities. Economically disadvantaged individuals are socially disadvantaged individuals whose ability to compete in the free enterprise system has been impaired by diminished capital and credit opportunities as compared with that of others in the same or similar line of business who are not socially disadvantaged. Throughout this report, we refer to socially and economically disadvantaged individuals as "disadvantaged individuals." To be considered owned by disadvantaged individuals, businesses must be at least 51 percent owned by one or more disadvantaged individuals or, in the case of any publicly owned business, at least 51 percent of the stock must be owned by disadvantaged individuals, and those who are disadvantaged must control the management and daily business operations. [3] Encouraging participation by women-owned businesses was not included among the program's original purposes but was added by the Small Business Research and Development Enhancement Act of 1992. Pub. L. No. 102-564, § 102, 106 Stat. 4249, 4250. To be women-owned, a small business must be at least 51 percent owned by one or more women or, in the case of any publicly owned business, at least 51 percent of the stock must be owned by women, and women must control the management and daily business operations. [4] The 11 agencies are the U.S. Department of Agriculture, Department of Commerce, Department of Defense, Department of Education, Department of Energy, Department of Health and Human Services, Department of Homeland Security, Department of Transportation, Environmental Protection Agency, National Aeronautics and Space Administration, and National Science Foundation. [5] The National Institutes of Health accounted for more than 98 percent of the SBIR expenditures of the Department of Health and Human Services from fiscal year 2000 through fiscal year 2010. Four other agencies in the department--the Centers for Medicaid and Medicaid Services, the Centers for Disease Control and Prevention, the Administration for Children and Families, and the Food and Drug Administration--also issued SBIR solicitations during that period. [6] While the agencies that we reviewed account for the vast majority of SBIR program expenditures, the results of our review cannot be generalized to all participating agencies. [7] The views expressed by representatives of these trade associations cannot be generalized to other trade associations. [8] Prior to 2010, the award ceiling for phase I was $100,000, and the award ceiling for phase II was $750,000. According to SBA guidance, agencies may exceed these ceilings with appropriate justification. [9] In addition, small businesses that are majority-owned or - controlled by another business or joint venture that meets the 51 percent requirement are also eligible to compete for SBIR awards. 13 C.F.R. § 121.702(a). [10] In this report, we are using "technologies" to refer to products, systems, methods, services or other results of SBIR-funded R&D. [11] The requirement specifically applies to technologies that are moving toward commercialization. [12] See GAO, Small Business Innovation Research: Agencies Need to Strengthen Efforts to Improve the Completeness, Consistency, and Accuracy of Awards Data, [hyperlink, http://www.gao.gov/products/GAO- 07-38] (Washington, D.C.: Oct. 19, 2006) and GAO, Managing for Results: Measuring Program Results That Are under Limited Federal Control, [hyperlink, http://www.gao.gov/products/GGD-99-16] (Washington, D.C.: Dec. 11, 1998). [13] [hyperlink, http://www.gao.gov/products/GAO-07-38]. [14] For our 2006 report, we reviewed the SBIR programs of the Department of Agriculture, Environmental Protection Agency, and National Institute of Standards and Technology in addition to those programs of the agencies we reviewed for this report. [15] GAO, Small Business Innovation Research: Observations on Agencies' Data Collection and Eligibility Determination Efforts, [hyperlink, http://www.gao.gov/products/GAO-09-956T] (Washington, D.C.: Aug. 6, 2009). [16] National Research Council, An Assessment of the SBIR Program (Washington, D.C.: National Academies Press, 2008). This report is part of a series published by the National Academies in response to the mandate. [17] Our review did not include NASA's 2011 SBIR solicitation, which was released in July 2011. [18] NSF does not issue separate solicitations for phase II projects and considers phase II proposals only from recipients of NSF phase I awards. [19] DOD officials told us that for fiscal years 2008-2010, such scores were calculated for 42-45 percent of applicants. [20] American Recovery and Reinvestment Act of 2009, Pub. L. No. 111- 5, 123 Stat. 115. [21] According to NIH officials, the National Cancer Institute based its program on one originated by the National Institute of Standards and Technology. [22] Participating agencies have authority to spend or authorize expenditure of a portion of their SBIR funds for technical assistance to SBIR awardees. The value of the technical assistance provided with or authorized from SBIR funds can be up to $4,000 per award in phase I and up to $4,000 per award per year in phase II. [23] According to NIH officials, since the agency began offering such assistance in 2004, NIH has provided the assistance to nearly 700 of the 5,090 businesses that received phase I SBIR awards. [24] The National Defense Authorization Act for Fiscal Year 2006, Pub. L. No. 109-163, § 252, 119 Stat. 3136, 3178 (codified as amended at 15 U.S.C. § 638(y)) authorized DOD's Commercialization Pilot Program, which allows DOD components to use a portion of their SBIR funds for program administration activities that support accelerated transition of SBIR-funded technologies into phase III. DOD allows its components discretion to use these funds for their own specific needs. [25] NASA did not use SBIR funds to support the technical assistance pilot program; however, a NASA official indicated that, if the agency had continued the program, it likely would have done so using SBIR funds. NASA officials indicated that the agency has relied on other resources to help increase commercialization, such as national SBIR conferences hosted by states; industry-sponsored technology summits; the Space Alliance Technology Outreach Program, which is designed to speed the transfer of space technology to the private sector; non-SBIR- specific agency publications; and the agency's Web site. [26] In this report, we are using "non-SBIR funds" to refer to funds other than the 2.5 percent of the participating agencies' extramural R&D budgets that must be set aside for SBIR. [27] Outside investment can be in the form of capital, liquid assets, or convertible debt. SBA has identified outside (non-SBIR) investment as a metric for commercialization, as discussed later in this report. [28] NIH's National Cancer Institute refers to its matching funds awards as bridge awards. The National Cancer Institute is the only NIH component included in our review that offered matching funds. [29] NASA and the National Cancer Institute established their matching funds programs relatively recently--in 2010 and 2009, respectively. DOD began matching funds for phase I award recipients in 1995 and for phase II award recipients in 2000. [30] NSF established a matching funds program for its phase II award recipients in 1998 and later established a similar program for phase I award recipients in 2007. [31] In addition, in 2010 SBA made 20 awards to state organizations though the Federal and State Technology Partnership Program. Under the program, SBA, with the concurrence of NSF and DOD officials, makes awards to provide outreach, financial support, or technical assistance to small businesses participating in or interested in participating in the SBIR program. In making such awards, the agencies are required to consider whether the applications being funded address the needs of small businesses owned by minorities or women, among other things. [32] 15 U.S.C. § 638(k)(2) (2006). [33] Businesses are required to provide information for the Central Contractor Registration database such as their Employer Identification Number, legal business name, and the goods and services they provide. [34] SBA officials noted that budgetary uncertainties may affect related efforts. For example, the officials said that SBA has delayed plans to develop a commercialization survey for award recipients that do not apply for a new SBIR award within 1 year after their phase II award ends. [35] SBA officials said that the agency had previously obtained approval from the Office of Management and Budget, as required by the Paperwork Reduction Act, for data collection instruments that are substantially similar to the ones proposed, and they plan to rely on this approval. Paperwork Reduction Act, 44 U.S.C. §§ 3501-3520 (2006). [36] DOD's estimate of returns on investment also includes the commercialization results of awards made through DOD's Small Business Technology Transfer (STTR) program. The STTR program resembles the SBIR program in some respects but provides funding for research proposals that are developed and executed cooperatively between small businesses and research organizations. [37] For NIH's 2002 survey, 768 (73 percent) of the 1,052 recipients of phase II awards from 1992 through 2001 were located and responded to the survey. Follow-up surveys experienced declining responses from recipients. Of the 768 respondents to the 2002 survey, 275 (36 percent) were located and responded to the final follow-up survey in 2007. In reporting survey results, NIH calculated its response rate from the sample of award recipients that it deemed usable and eligible--that is, those award recipients that, among other criteria, could be located and were still operating in the United States. For the 2002 survey, NIH used the size of this sample--905 contacts out of the 1,052 award recipients--to determine that the 768 respondents represented a survey response rate of 85 percent. [38] For NIH's 2008 survey, 719 (69 percent) of the 1,037 recipients of phase II awards from 2002 through 2006 were located and responded to the survey. For the 2008 survey, NIH used the size of the sample deemed usable and eligible--918 out of the 1,037 award recipients--to determine that the 719 respondents represented a response rate of 78 percent. [39] NSF's estimate of commercial revenue includes sales revenue from SBIR-funded technologies; licensing revenue; and revenue from mergers or sales of the award recipients' businesses that resulted, at least in part, from the awards. NSF's estimate does not include venture capital funding. [40] NSF's estimated investment covers the period from October 2004 through October 2010. [41] As we reported in 2010, commercialization data are not always correctly entered into this database by DOD procurement officers. See GAO, Space Acquisitions: Challenges in Commercializing Technologies Developed under the Small Business Innovation Research Program, [hyperlink, http://www.gao.gov/products/GAO-11-21] (Washington, D.C.: Nov. 10, 2010). However, DOD officials said that they are improving data in this system by training officers to correctly account for SBIR contracts. [42] Agencies are prohibited from using SBIR funds for program administration with the exception of certain administrative activities authorized as part of DOD's Commercialization Pilot Program. [43] [hyperlink, http://www.gao.gov/products/GAO-07-38]. [44] According to DOD, this percentage does not necessarily represent historic averages. [45] NASA's survey, conducted from 1997 through 2002, achieved a 78 percent response rate. [46] [hyperlink, http://www.gao.gov/products/GAO/RCED-99-114]. [47] NIH accounted for more than 98 percent of the SBIR expenditures of the Department of Health and Human Services from fiscal year 2000 through fiscal year 2010. Four other agencies in the department--the Centers for Medicaid and Medicaid Services, the Centers for Disease Control and Prevention, the Administration for Children and Families, and the Food and Drug Administration--also issued SBIR solicitations during that period. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: