Small Business Innovation Research: SBA Should Work with Agencies to Improve the Data Available for Program Evaluation
Highlights
Federal agencies with a budget of at least $100 million for research and development (R&D) conducted by others must participate in the Small Business Innovation Research (SBIR) program. SBIR has four purposes: meet federal R&D needs; stimulate technological innovation; increase commercialization (e.g., sales) of innovations based on federal R&D; and encourage participation in innovation by small businesses owned by disadvantaged individuals and women. The Small Business Administration (SBA) oversees the efforts of participating agencies, which make awards to small businesses using SBIR funds. Congress directed SBA to develop a database with commercialization data for government use in evaluating the program. GAO was asked to determine (1) how agencies have addressed SBIR's purposes and (2) the extent of data available to evaluate progress in increasing commercialization. GAO analyzed program documents and interviewed officials at SBA and five agencies that accounted for about 96 percent of SBIR awards.
For fiscal years 2008 through 2011, the participating agencies GAO reviewed--the Department of Defense (DOD), the Department of Energy (DOE), the National Aeronautics and Space Administration, the Department of Health and Human Services' National Institutes of Health, and the National Science Foundation (NSF)--addressed SBIR's purposes through solicitations for award applications, technical assistance or matching funds programs, and outreach. In particular, the agencies addressed SBIR purposes related to meeting federal R&D needs and stimulating technological innovation through their solicitations, which included research topics that were designed to meet agencies' respective R&D or mission needs. Agencies also addressed commercialization of innovations through solicitations, as well as through technical assistance for award recipients or through matching funds programs. To provide technical assistance, the agencies contracted with vendors and consultants for help in developing business plans and identifying potential customers for SBIR award recipients, among other things. Agency matching funds programs provided additional SBIR funds to award recipients that obtained commitments from outside investors. Agencies generally addressed the remaining SBIR purpose, encouraging participation by small businesses owned by disadvantaged individuals and women, through outreach activities aimed at a broader audience, such as sharing information on Web sites. However, the effectiveness of these efforts is difficult to evaluate, in part because SBA does not collect data on the number of SBIR applications submitted by such businesses, thus hindering analysis of trends in their submission of applications. Comparable data are not available across participating agencies to evaluate progress in increasing commercialization of SBIR technologies. SBA has not yet expanded an existing database to include commercialization data for program evaluation, but the agency has hired a contractor and allocated funds to develop the expanded database by August 2011. SBA has also worked with participating agencies to develop common metrics for commercialization. In the absence of the expanded database, agencies have independently gathered commercialization data for their own use that are not comparable. In collecting these data, agencies differed in the types of data collection instruments used, dates the instruments were administered, award recipient populations queried, and types of data requested. Furthermore, with the exception of DOD, agencies that GAO reviewed did not generally take steps to verify commercialization data they collected from award recipients, so the accuracy of the data is largely unknown. SBA has worked with SBIR agencies to identify best practices in other areas of program management but has not identified best practices for agencies to use in verifying the accuracy of commercialization data. Implementing the expanded database should improve the comparability of commercialization data available, but a lack of consistent practices for verifying the accuracy of these data may limit their usefulness for programwide evaluation. GAO recommends that SBA work with participating agencies to (1) collect data on applications from small businesses owned by disadvantaged individuals and women and (2) identify best practices for verification of commercialization data. SBA, DOE, and NSF generally agreed with these recommendations; the other agencies GAO reviewed neither agreed nor disagreed.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Small Business Administration | To build upon efforts to implement a government-use database for program evaluation, the Administrator of the Small Business Administration should work with participating SBIR agencies to collect data on the number of applications submitted by small businesses owned by disadvantaged individuals and women. |
SBA updated its template that agencies use to report program information annually to include information on the number of proposals received from women-owned small businesses and socially and economically disadvantaged small businesses. Agencies used the revised template for their fiscal year 2013 data, which was submitted in June 2014.
|
| Small Business Administration | To build upon efforts to implement a government-use database for program evaluation, the Administrator of the Small Business Administration should work with participating SBIR agencies to identify best practices for verifying the accuracy of data related to progress in increasing commercialization. |
SBA surveyed the agencies participating in the SBIR program for best practices in verifying the accuracy of the commercialization data they independently collect. According to SBA, most of the agencies do not verify this data because they consider the costs to be prohibitive, but identified some best practices. Drawing from the experiences at the agencies, SBA officials said that, as of August 2015, SBA was implementing procedures to check the accuracy of commercialization information reported in the SBIR.gov database. These procedures involve identifying cases where the data appear inconsistent or incorrect and contacting the awardee directly for clarification.
|