This is the accessible text file for GAO report number GAO-14-525 entitled 'Small Business Administration: Office of Advocacy Needs to Improve Controls over Research, Regulatory, and Workforce Planning Activities' which was released on July 22, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Chairman, Subcommittee on Financial Services and General Government, Committee on Appropriations, United States Senate: July 2014: Small Business Administration: Office of Advocacy Needs to Improve Controls over Research, Regulatory, and Workforce Planning Activities: GAO-14-525: GAO Highlights: Highlights of GAO-14-525, a report to the Chairman, Subcommittee on Financial Services and General Government, Committee on Appropriations, United States Senate. Why GAO Did This Study: Congress created Advocacy within SBA in 1976 as an independent voice for small businesses. Questions have recently been raised about Advocacy's efforts to represent small businesses in regulatory activities and some of its research on small business issues. In light of these questions, GAO was asked to review Advocacy's activities. This report examines Advocacy's (1) research, (2) regulatory activities, and (3) workforce planning efforts. GAO analyzed Advocacy's research, comment letters, and other regulatory information for fiscal years 2008-2013; assessed Advocacy's policies and procedures against federal standards for internal control and information quality; and interviewed agency officials, and small business and industry representatives. GAO also analyzed the applicability of the Federal Advisory Committee Act to Advocacy roundtables. What GAO Found: The Office of Advocacy (Advocacy) within the Small Business Administration (SBA) fulfills its mission by researching small business issues and providing input into federal rulemaking and related regulatory activities. However, GAO identified key areas in Advocacy's system of internal control that could be improved. * Research. GAO found that Advocacy did not ensure that its staff monitored the quality of the information the office disseminated, as required. GAO reviewed 20 selected research products and found that in 16 cases a required quality review had not been documented. Advocacy recently established a review policy for its research, but it does not include procedures for selecting the reviewers or documenting that a review occurred and how reviewer comments are addressed. GAO also found that Advocacy staff had not followed federal information quality guidelines to retain data and could not substantiate the quality of information in two cost-estimation reports-—a research product it has contracted for every 5 years. Without better controls over its quality review process and efforts to substantiate the information it disseminates, Advocacy cannot ensure the validity of one of its core activities—-research in support of small businesses. * Regulatory activities. Advocacy recently updated procedures for its regulatory activities, but these could be strengthened. GAO found the extent to which individual staff maintained records varied, in part, because the procedures lacked policies for documentation. For instance, the procedures state that when staff decide to intervene in the rulemaking process, they must follow up as appropriate with the interested groups to ensure that Advocacy has sufficient information and data to support its case. However, there is no policy that these interactions be documented. Federal internal control standards state that documentation and records should be maintained. If key procedures are not being documented, managers do not have an institutional record that agency goals and objectives in this area are being met. GAO also found that the Federal Advisory Committee Act's transparency and other requirements do not apply to Advocacy's meetings with stakeholders to get input on regulations (roundtables). * Workforce planning. Advocacy's workforce efforts include training and mentoring for new staff, but do not include succession planning, which is recommended by the Office of Personnel Management. According to federal internal control standards, effective management of a workforce is essential to achieving program results. Officials told GAO that Advocacy was a small office and that additional staff were hired on an as-needed basis. However, some key staff have been with Advocacy for many years and their experience will be difficult to replace. If Advocacy does not incorporate succession planning strategies into its workforce planning efforts, it is at risk of not having the skills and expertise to meet its mission when key staff leave or retire. What GAO Recommends: GAO makes several recommendations to improve the Office of Advocacy's controls over the quality of its research, the documentation of its regulatory activities, and workforce planning. In commenting on a draft of this report, Advocacy agreed with our recommendations and noted some steps it will take to address them. View [hyperlink, http://www.gao.gov/products/GAO-14-525]. For more information, contact Cindy Brown Barnes at (202) 512-8678 or brownbarnesc@gao.gov. [End of section] Contents: Letter: Background: Advocacy's Internal Controls Do Not Ensure the Quality of Its Research and That Required Information Quality Practices Are Followed: Documentation of Key Regulatory Activities is Inconsistent and Roundtables Are Not Publicized According to Advocacy's Policies: Advocacy's Workforce Planning Efforts Lack Succession Planning: Conclusions: Recommendations for Executive Action: Agency Comments: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Office of Advocacy, Small Business Administration: Appendix III: Legal Analysis of the Applicability of the Federal Advisory Committee Act to the Office of Advocacy's Roundtables: Tables: Table 1: Advocacy Comment Letters by Policy Area, Fiscal Years 2009- 2013: Table 2: Number of Roundtables by Policy Area, Fiscal Years 2009-2013: Figures: Figure 1: Office of Advocacy Organizational Chart, Fiscal Year 2014: Figure 2: Office of Advocacy Organizational Chart, Fiscal Year 2014: Abbreviations: ADA: Americans with Disabilities Act: AAPS: Association of American Physicians and Surgeons: APA: Administrative Procedure Act: BATF: Bureau of Alcohol, Tobacco, and Firearms: CFPB: Consumer Financial Protection Bureau: EPA: Environmental Protection Agency: FACA: Federal Advisory Committee Act: FOIA: Freedom of Information Act: GSA: General Services Administration: HCAAF: Human Capital Assessment and Accountability Framework: HEW: Department of Health, Education, and Welfare: JAG: Judge Advocate General of the Army: OIRA: Office of Information and Regulatory Affairs: OMB: Office of Management and Budget: OPM: Office of Personnel Management: OSHA: Occupational Safety and Health Administration: OST: Executive Office of the President's Office of Science and Technology: RFA: Regulatory Flexibility Act: SBA: Small Business Administration: SBREFA: Small Business Regulatory Enforcement Fairness Act of 1996: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: July 22, 2014: The Honorable Tom Udall: Chairman: Subcommittee on Financial Services and General Government: Committee on Appropriations: United States Senate: Dear Senator Udall: Congress created the Office of Advocacy (Advocacy) within the Small Business Administration (SBA) in 1976 as an independent voice for small businesses within the federal government.[Footnote 1] The Chief Counsel for Advocacy, appointed by the President and confirmed by the U.S. Senate, directs the office and is tasked with representing the views and interests of small businesses to other federal agencies, including during rulemaking and related regulatory activities. Advocacy's mission is to encourage the development and growth of America's small businesses by: * producing research to keep policymakers and other stakeholders informed about the impact of federal regulatory burdens on small businesses, document the role of small businesses in the economy, and explore and explain issues of concern to the small business community; * intervening early in federal agencies' regulatory development process on proposals that affect small businesses and providing training in complying with the Regulatory Flexibility Act (RFA) to agency policymakers and regulatory development officials; and: * fostering two-way communication between federal agencies and the small business community. Questions have recently been raised about Advocacy's efforts to represent small businesses. For example, some stakeholders questioned how Advocacy staff gather and use information to develop comment letters on proposed regulations and whether that information actually conveys the views of small businesses. Also, some parties have expressed concerns that Advocacy's information-gathering events ("roundtables") are not fully transparent and open to the public, and others have criticized the methodology of some of the research conducted by contractors hired by Advocacy. You asked us to review the Office of Advocacy. This report examines Advocacy's (1) research activities; (2) regulatory activities, including the applicability of the Federal Advisory Committee Act's (FACA) transparency and other requirements to Advocacy's roundtables; and (3) workforce planning efforts. To understand Advocacy's mission, operations, and participation in the rulemaking process, we interviewed Advocacy officials, and reviewed Advocacy documents, including its policies and procedures, budget and strategic planning documents, annual reports to Congress on the RFA, and other publications. To evaluate Advocacy's research activities, we selected a nonprobability sample of 20 of its products and assessed them against Advocacy's review process for its research ("peer review"), the Office of Management and Budget's (OMB) Peer Review guidance, and OMB and SBA's Information Quality Guidelines.[Footnote 2] Our sample was designed to include a variety of topics and report types and focused on more recent publications in order to maximize the possibility that the research staff involved would be available for interviews. In addition, we evaluated three of Advocacy's most recent external studies on the economic costs of regulation and assessed the studies against OMB and SBA Information Quality Guidelines. We also reviewed peer reviewers' comments and other external reviews on the studies to understand questions that had been raised about the studies. Lastly, we interviewed the Advocacy officials responsible for this research to obtain information on the steps taken to substantiate the quality of these studies. To evaluate Advocacy's regulatory activities, we analyzed comment letters and information-gathering events (roundtables) from fiscal years 2009 through 2013 as well as relevant policies and procedures. We interviewed Advocacy staff who were responsible for these activities and reviewed supporting documentation to understand their origins and purposes. In addition, we attended three roundtables. To analyze whether FACA (5 U.S.C. App. II) applies to Advocacy's roundtables, we reviewed relevant statutes, case law, regulations, and guidance. We also considered Advocacy's views on the issue. Lastly, we interviewed officials from the entities that interact with Advocacy in rulemaking--the Environmental Protection Agency (EPA), Occupational Safety and Health Administration (OSHA), Bureau of Consumer Financial Protection, also known as the Consumer Financial Protection Bureau (CFPB), and OMB's Office of Information and Regulatory Affairs (OIRA). Also, we interviewed officials from six industry associations that represent small and large businesses; one large corporation; and two nonprofit organizations whose missions relate to policies or rules under consideration, to understand their perspectives on Advocacy's roundtables and other work. We made our selections to include representatives from a variety of sources, and while the results from our interviews cannot be projected to all entities that interact with Advocacy, the information we gathered provides insights into how the selected groups view Advocacy and its work in representing small businesses to federal policymakers. To assess Advocacy's workforce planning efforts, we interviewed senior management to discuss any workforce planning efforts that were in place. We evaluated those efforts and any related documents against applicable federal standards for internal control and the Office of Personnel Management's Human Capital Assessment and Accountability Framework (HCAAF).[Footnote 3] We conducted this performance audit from August 2013 to July 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Under legislation that established Advocacy in its current form in 1976, Advocacy' duties are to: * serve as a focal point for small businesses' concerns about the federal government's policies and activities; * advise small businesses on how to interact with the federal government; * develop proposals for federal agencies on behalf of small businesses; * represent the views and interests of small businesses before federal agencies; * enlist the cooperation and assistance of public and private agencies, businesses, and other organizations in disseminating information about the federal government's programs and services that benefit small businesses.[Footnote 4] Since its establishment, a series of laws and executive orders has increased Advocacy's roles and responsibilities. First, in 1980 the White House Conference on Small Business made recommendations that led directly to the passage of the RFA, which requires government agencies to consider the effects of their regulatory actions on small entities and, where possible, mitigate them.[Footnote 5] Under the RFA, agencies provide a small business impact analysis, known as an initial regulatory flexibility analysis, with every proposed rule published for notice and comment and a final regulatory flexibility analysis with every final rule.[Footnote 6] The Chief Counsel for Advocacy was charged with monitoring federal agencies' compliance with the act and with submitting an annual report to Congress. Second, in 1996 the Small Business Regulatory Enforcement Fairness Act (SBREFA) provided for the judicial review of agency compliance with key sections of the RFA. It also established a requirement that EPA and OSHA convene panels whenever these agencies are developing a rule for which an initial regulatory flexibility analysis would be required (SBREFA panels).[Footnote 7] These panels consist of the agency, OIRA, and Advocacy. The 2010 Dodd-Frank Act added the newly created CFPB to the agencies required to convene SBREFA panels.[Footnote 8] The SBREFA panels meet with representatives of the affected small businesses to review the agencies' draft proposed rules, identify alternative approaches to the rules, and provide insight on the anticipated impact of the rules on small entities. The panels issue a report, including any recommendations for minimizing the economic impact of the rule on small entities. Third, Advocacy's responsibilities were further expanded by Executive Order 13272, which was issued in 2002. The order required each agency to establish procedures and policies to promote compliance with the RFA and to publish a response in the Federal Register to any written comment received from Advocacy on rules published.[Footnote 9] This requirement was codified by the Small Business Jobs Act of 2010.[Footnote 10] Executive Order 13272 also directs Advocacy to provide training to federal agencies on how to comply with the RFA. Until 2010, Advocacy's budget was part of SBA's. As of fiscal year 2010, however, Advocacy was given statutory line-item funding in a Treasury account separate from other SBA funding, with Congress setting the amount available for Advocacy's direct costs.[Footnote 11] In fiscal year 2014, Advocacy's enacted budget was $8.75 million. Its fiscal year 2015 budget request was $8.46 million. Of that amount, $7.75 million (92 percent) is to be used to fund compensation and benefits for Advocacy's professional staff, with the balance of Advocacy's budget split almost equally between external research and all other direct expenses.[Footnote 12] Advocacy is organized in five offices, as shown in figure 1. Figure 1: Office of Advocacy Organizational Chart, Fiscal Year 2014: [Refer to PDF for image: Organizational Chart] Top level: Chief Counsel for Advocacy: * Confidential Assistant; * Deputy Chief Counsel for Advocacy. Second level, reporting up: Office of Interagency Affairs; Office of Economic Research; Office of Information; Office of Regional Affairs; Administrative Support Branch. Source: Office of Advocacy. GAO-14-525. [End of figure] Research Activities: Advocacy's Office of Economic Research produces both internal and external research, which is publicly disseminated, on a variety of small-business issues. More specialized research--requiring proprietary data or econometric analysis--typically is conducted by contractors (external). SBA handles the contracting process for Advocacy. The contracts generally last for 1 year. Advocacy economists act as the official contracting officers' representatives (contracting officer) overseeing and coordinating the work of the contractors. The contracting officers maintain a contract file for each external research product. Each year the Office of Economic Research solicits research topics from Advocacy staff and small business stakeholders, such as associations comprised of small businesses. In addition, Congress requests studies, either formally (by putting the requirement into a law), or informally (through discussions with Advocacy staff). A final list of potential research is presented to the Chief Counsel before the beginning of the fiscal year, and the counsel chooses the topics for the year. In fiscal year 2013, Advocacy produced 22 research products on topics that included access to capital, small business exporters, entrepreneurship, and minority-and women-owned businesses. Other Advocacy research addresses the concerns highlighted in Advocacy's authorizing statute, such as examining the role of small business in the American economy, assessing the effectiveness of existing federal subsidies and assistance programs for small businesses, and evaluating efforts to assist small veteran-owned small business concerns.[Footnote 13] Advocacy follows a peer review process that was revised and formalized in 2013 and applies to all internal and external research products. Advocacy staff, including the Director of the Office of Economic Research, conduct peer reviews for internal products. For external research products, the peer review is initiated when Advocacy receives the first draft of the product, typically 6 to 8 months after the contract has been awarded, and generally includes external reviewers. In addition to the peer review process, all of Advocacy's research products are required to pass Advocacy's internal clearance process, which involves review by the Director of Economic Research, editors in Advocacy's Office of Information, and individuals in Advocacy's Office of Chief Counsel (Senior Advisor, Deputy Chief Counsel, and Chief Counsel). Regulatory Activities: Advocacy's Office of Interagency Affairs oversees the office's regulatory activities, which aim to convey the views of small businesses on the impact of federal regulations and related costs. These activities generally fall into three categories--developing and issuing comment letters, convening information roundtables, and providing RFA training.[Footnote 14] Attorneys in this office ("regulatory attorneys") are expected to become experts in the policy areas they oversee and to establish and maintain broad and effective networks of small business experts (e.g., trade associations) in their policy area. The regulatory attorneys are encouraged to go to trade association and other industry meetings in order to maintain and expand those networks. In addition to maintaining working relationships with industry members and experts, the staff are to establish and maintain relationships with the regulatory staff within each agency who write the rules. One of the primary ways Advocacy provides input to agencies that are issuing rules and regulations of concern to small businesses is through public comment letters. Our review of comment letters from fiscal years 2009 through 2013 found that they covered a wide range of rulemakings on issues such as food labeling, designations for critical habitat, and emission standards. Advocacy made a number of recommendations in its comment letters that included creating an exemption for small businesses or strengthening economic analyses required by the RFA. Advocacy also issued "nonrule" letters that involved the agencies' other activities, such as agencies' scientific research. These letters constitute a small proportion of Advocacy's comment letters. Table 1 below shows the number of comment letters on rulemakings by fiscal year. Table 1: Advocacy Comment Letters by Policy Area, Fiscal Years 2009- 2013: Policy area: Environment; Fiscal year 2009: 11; Fiscal year 2010: 9; Fiscal year 2011: 10; Fiscal year 2012: 6; Fiscal year 2013: 7; Total: 43. Policy area: Natural resources and agriculture; Fiscal year 2009: 6; Fiscal year 2010: 4; Fiscal year 2011: 8; Fiscal year 2012: 5; Fiscal year 2013: 3; Total: 26. Policy area: Safety and transportation; Fiscal year 2009: 5; Fiscal year 2010: 9; Fiscal year 2011: 6; Fiscal year 2012: 2; Fiscal year 2013: 3; Total: 25. Policy area: Banking and economic regulation; Fiscal year 2009: 1; Fiscal year 2010: 2; Fiscal year 2011: 7; Fiscal year 2012: 3; Fiscal year 2013: 5; Total: 18. Policy area: Food, drug, and health; Fiscal year 2009: 3; Fiscal year 2010: 4; Fiscal year 2011: 6; Fiscal year 2012: 0; Fiscal year 2013: 3; Total: 16. Policy area: Labor and immigration; Fiscal year 2009: 1; Fiscal year 2010: 3; Fiscal year 2011: 6; Fiscal year 2012: 1; Fiscal year 2013: 1; Total: 12. Policy area: Procurement and small business; Fiscal year 2009: 1; Fiscal year 2010: 4; Fiscal year 2011: 1; Fiscal year 2012: 3; Fiscal year 2013: 1; Total: 10. Policy area: Rulemaking issues (no specific policy area); Fiscal year 2009: 2; Fiscal year 2010: 1; Fiscal year 2011: 3; Fiscal year 2012: 3; Fiscal year 2013: 0; Total: 9. Policy area: Telecommunications and energy; Fiscal year 2009: 3; Fiscal year 2010: 2; Fiscal year 2011: 2; Fiscal year 2012: 2; Fiscal year 2013: 0; Total: 9. Policy area: Taxes, pensions, and securities; Fiscal year 2009: 2; Fiscal year 2010: 0; Fiscal year 2011: 4; Fiscal year 2012: 1; Fiscal year 2013: 1; Total: 8. Policy area: Intellectual property; Fiscal year 2009: 0; Fiscal year 2010: 1; Fiscal year 2011: 0; Fiscal year 2012: 0; Fiscal year 2013: 2; Total: 3. Policy area: Not applicable; Fiscal year 2009: 0; Fiscal year 2010: 0; Fiscal year 2011: 0; Fiscal year 2012: 2; Fiscal year 2013: 0; Total: 2. Policy area: Total; Fiscal year 2009: 35; Fiscal year 2010: 39; Fiscal year 2011: 53; Fiscal year 2012: 28; Fiscal year 2013: 26; Total: 181. Source: GAO analysis of Advocacy data. GAO-14-525. [End of table] Regulatory attorneys also convene information-gathering roundtables to discuss the regulatory concerns of small businesses. Roundtables are convened on a regular basis in two policy areas--environment and labor safety--while events covering other areas are convened on an ad hoc basis, depending on which regulatory or rulemaking issues might be forthcoming. According to the regulatory attorneys, the most typical reason for convening a roundtable was an upcoming rule or legislation that would impact small businesses. The regulatory attorneys use the information gathered from the roundtables, together with other information, to inform Advocacy's positions on the issues involved, and to give Advocacy direction on proposed rules' economic impacts and possible regulatory alternatives. The attorneys also told us roundtable discussions help them set priorities, and broaden their knowledge base. The roundtables sometimes resulted in a comment letter, although not always. Table 2 shows the number of roundtables by policy area for fiscal years 2009 through 2013. Table 2: Number of Roundtables by Policy Area, Fiscal Years 2009-2013: Policy area[A]: Safety and transportation; Fiscal year 2009: 10; Fiscal year 2010: 10; Fiscal year 2011: 9; Fiscal year 2012: 9; Fiscal year 2013: 8; Total: 46. Policy area[A]: Environment; Fiscal year 2009: 7; Fiscal year 2010: 7; Fiscal year 2011: 11; Fiscal year 2012: 9; Fiscal year 2013: 7; Total: 41. Policy area[A]: Taxes, pensions, and securities; Fiscal year 2009: 2; Fiscal year 2010: 7; Fiscal year 2011: 6; Fiscal year 2012: 6; Fiscal year 2013: 2; Total: 23. Policy area[A]: Banking and economic regulation; Fiscal year 2009: 0; Fiscal year 2010: 1; Fiscal year 2011: 0; Fiscal year 2012: 4; Fiscal year 2013: 1; Total: 6. Policy area[A]: Labor and immigration; Fiscal year 2009: 1; Fiscal year 2010: 1; Fiscal year 2011: 3; Fiscal year 2012: 1; Fiscal year 2013: 0; Total: 6. Policy area[A]: Veteran issues; Fiscal year 2009: 1; Fiscal year 2010: 1; Fiscal year 2011: 3; Fiscal year 2012: 0; Fiscal year 2013: 0; Total: 5. Policy area[A]: Intellectual property; Fiscal year 2009: 0; Fiscal year 2010: 1; Fiscal year 2011: 1; Fiscal year 2012: 1; Fiscal year 2013: 0; Total: 3. Policy area[A]: Natural resources; Fiscal year 2009: 0; Fiscal year 2010: 1; Fiscal year 2011: 0; Fiscal year 2012: 1; Fiscal year 2013: 1; Total: 3. Policy area[A]: Labor safety/environment; Fiscal year 2009: 0; Fiscal year 2010: 0; Fiscal year 2011: 0; Fiscal year 2012: 2; Fiscal year 2013: 0; Total: 2. Policy area[A]: Rulemaking issues (no specific policy area); Fiscal year 2009: 1; Fiscal year 2010: 0; Fiscal year 2011: 0; Fiscal year 2012: 1; Fiscal year 2013: 0; Total: 2. Policy area[A]: Procurement and small business; Fiscal year 2009: 0; Fiscal year 2010: 1; Fiscal year 2011: 0; Fiscal year 2012: 0; Fiscal year 2013: 0; Total: 1. Policy area[A]: Telecommunications; Fiscal year 2009: 0; Fiscal year 2010: 0; Fiscal year 2011: 1; Fiscal year 2012: 0; Fiscal year 2013: 0; Total: 1. Policy area[A]: Total; Fiscal year 2009: 22; Fiscal year 2010: 30; Fiscal year 2011: 34; Fiscal year 2012: 34; Fiscal year 2013: 19; Total: 139. Source: GAO analysis of Advocacy data. GAO-14-525. [A] There are more policy areas for the roundtables since there are instances where Advocacy hosted roundtables in areas that did not have comment letters, such as veteran issues. [End of table] As discussed previously, Executive Order 13272 requires Advocacy to provide training to the agencies on how to comply with the RFA. According to data provided by Advocacy officials, in 2013, Advocacy staff provided training on the RFA to 159 officials at nine agencies and to 22 congressional staff. In addition to the formal training sessions, regulatory attorneys told us they were encouraged to interact regularly with the relevant rulemaking officials at the agencies as rules were developed in order to communicate the concerns of the small business advocates. Advocacy's Internal Controls Do Not Ensure the Quality of Its Research and That Required Information Quality Practices Are Followed: Producing research products, both internally and externally, on issues of importance to small businesses is one of Advocacy's primary responsibilities. However, we found that Advocacy's quality review process lacked some key controls to substantiate the quality of the research and did not take steps to ensure that staff were adhering to existing controls. Advocacy's Peer Review Process Lacks Guidance for Identifying Reviewers: According to Advocacy officials, peer review is the main quality control over the research it disseminates. Advocacy's current Chief Counsel recently directed the office to strengthen its peer review process with the intent to make it more rigorous and consistent. As a result, during the course of our review, Advocacy finalized a written peer review process. We found that the written guidance discussed the various levels of review for internal and external products as well as a process for initiating peer review. However, it did not specify how the economists who managed the research products were to identify peer reviewers. Instead, Advocacy officials told us that they relied on their own expertise and professional contacts to identify appropriate peer reviewers and provide recommendations through the Director of Research and to the Chief Counsel for Advocacy. The officials told us that within a specific subject matter there is often a small group of available peer reviewers, in part because their expertise is specialized and there are not many other alternatives. OMB's peer review guidance calls on agencies to select peer reviewers with the appropriate knowledge and expertise and to take into account their independence and lack of conflicts of interest.[Footnote 15] Advocacy managers told us that, in practice, the economists recommend peer reviewers based on knowledge and experience in both subject matter and databases--as discussed in the OMB guidelines. However, they did not provide specific written guidance to the economists on how to identify peer reviewers. Federal internal control standards state that internal control activities help ensure that management's directives are carried out and in implementing those standards management is responsible for developing the detailed policies, procedures and practices for their agency's operations.[Footnote 16] With additional guidance, Advocacy would be in a better position to help ensure that the economists fully understand how best to identify qualified peer reviewers and carry out the Chief Counsel's directive to improve the peer review process. Advocacy Has Not Consistently Documented Peer Reviews: Advocacy does not have consistent documentation showing whether a peer review occurred for all of its research products. Our review of 20 recent research products--10 internal and 10 external--revealed that 16 did not have documented peer reviews in the research files.[Footnote 17] According to interviews with Advocacy economists who managed the research, all of the 20 products underwent some form of peer review. The economists said that the type of review was commensurate with the methodological complexity of each product, among other factors. However, the Advocacy officials were unable to produce any documentation that peer reviews occurred for these 16 products. According to Advocacy's peer review guidance, the economists who manage the research should document all correspondence pertaining to the peer review and maintain this documentation in the research file. In addition, federal internal control standards require all transactions and significant events be documented and that the documentation be readily available. Advocacy officials do not have procedures to review the external research files to ensure that the peer reviews were documented. Furthermore, they noted that for some of the less in-depth internal research products--typically 2 to 5 pages--such documentation would be administratively burdensome.[Footnote 18] However, we note that the documentation could be likewise concise, such as a checklist, or a form that reviewers sign, similar to the one currently used by Advocacy for its internal clearance process. Otherwise, absent written documentation, Advocacy managers are limited in their ability to conduct oversight and ensure that this key quality control activity is happening. For example, for one of the internal research products we selected for review, the study author no longer worked at Advocacy, and therefore no one could tell us with any certainty whether the required peer review had occurred, or who had participated in it. Without adequate documentation--a key internal control--of its peer reviews, Advocacy does not have an institutional record of its activities and cannot demonstrate that it is following its own peer review process. In addition, Advocacy has not consistently documented how peer reviewers' comments were addressed by the authors of its external research products. Of the 10 external research files we reviewed, 4 had documentation that a peer review occurred, and 1 file included evidence that the peer reviewer comments were incorporated into the final report. Advocacy officials told us that the economists who managed external research consolidated the peer reviewers' comments and forwarded to the author those that needed to be addressed, including methodological and data issues and other comments, but not those that might change the scope of the contracted research.[Footnote 19] They also told us that they did not typically maintain documentation showing which comments had been addressed and why, but included the final report in the research file. However, Advocacy's peer review process states that the economist managing the research will analyze and incorporate, as needed, peer reviewers' suggestions and maintain all related documentation in the research file. However, since the economists are not keeping records and documenting that comments have been considered and addressed, management does not have an institutional record to provide reasonable assurance that its quality control process is being followed. Advocacy Has Not Followed Some Required Elements of Information Quality Guidelines: While Advocacy has quality review policies for its peer review process, it does not have policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies or taking other steps to substantiate the quality of information in such studies when they have not retained the data. Advocacy officials told us that they did not retain the original data or underlying computer codes for three external studies on the costs of regulation as required by the information quality guidelines.[Footnote 20] We focused on external studies on estimating the costs of regulation because it is a key research area for Advocacy, according to its originating statute and the mission statement of its Office of Economic Research. The OMB Information Quality Guidelines require that all agencies producing and disseminating "influential statistical information" help ensure a high degree of transparency about data and methods to facilitate the reproducibility of such information by qualified third parties. The SBA guidelines implement this standard for transparency by requiring that the underlying data be stored and made available for public review for as long as the agency-disseminated information based on the data are valid. The guidelines also state that all formulas, calculations, matrixes, and assumptions used in processing the data should be available. Because Advocacy classified two of the regulatory cost studies as "influential" according to the OMB guidelines, those data should have been maintained.[Footnote 21] Advocacy officials said that they did not maintain the data or models for influential external research because there might be a cost associated with obtaining such data, which would raise the costs of the studies, possibly making them prohibitively expensive. However, in the case of two of the studies, the original data were from publicly available sources and involved a relatively small dataset, suggesting the cost would not have been prohibitive.[Footnote 22] The OMB guidelines state that sufficient transparency--achieved in part by storing the relevant data--results in analyses that can be substantially reproduced. Not retaining the underlying information for these influential research papers makes it much more difficult to assess the quality of that work, including its objectivity--a key goal of the information quality guidelines. We also found that Advocacy staff had not taken additional steps, in the absence of retaining the underlying data, to substantiate the quality of the regulatory cost estimates in two of the studies that it sponsored and disseminated.[Footnote 23] The OMB guidelines state that when data and methods are not retained and made available to the public because of other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections, the agency shall apply rigorous checks to the analytical results and document what checks were undertaken. Because Advocacy had not retained data on the two cost estimation studies that had been criticized, we interviewed senior Advocacy officials, including the Director of its Office of Economic Research, about the information and methodologies used in the studies. We asked them a set of questions related to criticisms of the methodologies, data, and models used in the studies that were identified in our evaluation and the work of other researchers. Advocacy staff declined to answer many of our questions and instead directed us to the authors, stating that they, not Advocacy economists, were the experts on the issues covered in the studies.[Footnote 24] However, the authors would not speak with us, stating that they were no longer contractually obligated to respond to our requests for information. During our discussions with Advocacy officials, they stated that the purpose of the studies was not to estimate the overall costs of regulations, but rather to estimate the disproportionate share borne by small businesses. They also noted that, as with all contract research, the external research reports contain a disclaimer indicating that the views presented did not necessarily represent those of Advocacy. In addition, Advocacy added language to the 2010 report's cover page about the uncertainty of the authors' estimates of the costs of regulation.[Footnote 25] Advocacy officials noted that the majority of the research it conducts is not classified "influential" according to the OMB and SBA guidelines, and that they have no plans to engage in such work in the near future. However, given Advocacy's mission, it may do influential research in the future as it has in the past even if on a limited basis. We acknowledge that these reports may not necessarily be representative of all Advocacy's research efforts, but not substantiating the quality of the information in even one study could call into question the credibility of Advocacy's research program. Thus, establishing policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies or when such data are not retained because of certain compelling interests, and requiring additional steps to substantiate the quality of the information, would put Advocacy in a better position to provide reasonable assurance about the quality of its research program. OMB Information Quality Guidelines require that agencies develop policies to ensure that managers be able to substantiate the quality of information they disseminate. The guidelines also discuss narrow circumstances under which an agency does not have to substantiate the quality of the information that resulted from a research project of one of its contractors or grantees. In those circumstances, the researcher is to make clear with an appropriate disclaimer that the views expressed in the research are his or her own and do not necessarily reflect those of the agency. However, OMB cautions that "if an agency, as an institution, disseminates information prepared by an outside party in a manner that reasonably suggests that the agency agrees with the information, this appearance of having the information represent agency views makes agency dissemination of the information subject to these guidelines."[Footnote 26] As we noted previously, Advocacy placed a disclaimer on the two studies, but its actions indicate its agreement with the information in the studies. First, Advocacy made the two studies available on its website, where they continue to be available. Second, Advocacy's disclaimer that appeared on the 2010 study noted that it contained information and analysis that was reviewed and edited by Advocacy. Advocacy's description of its review role suggests that the agency contributed to the content, if not the conclusions of the study. Finally, several Advocacy comment letters have cited the 2005 study's regulatory cost estimates in support of their arguments.[Footnote 27] Because Advocacy's actions raise, at the least, an appearance of agreement with the information contained in the studies, we believe that the office was required to substantiate the quality of the estimates of the economic costs contained in the studies. Documentation of Key Regulatory Activities is Inconsistent and Roundtables Are Not Publicized According to Advocacy's Policies: Advocacy has practices and procedures ("policies") for its regulatory activities--comment letters and roundtables--however, documentation of these key regulatory activities is inconsistent. Furthermore, while we determined that transparency and other requirements in FACA do not apply to Advocacy's roundtables, Advocacy is not following its internal policies meant to ensure its roundtables are as open to the public as they could be. As a result Advocacy cannot demonstrate that it is always fully meeting its mission to foster two-way communication between small businesses and federal policymakers. Advocacy's Documentation for Key Regulatory Activities Is Inconsistent: Our review of Advocacy's key regulatory activities--developing and issuing comment letters and convening information roundtables--found that Advocacy staff are inconsistently maintaining documentation of key decisions and events. In April 2014, Advocacy provided us with a "practices and procedures guide" for the Office of Interagency Affairs, dated March 2014. The guide covers staff activities related to initiating comment letters and convening roundtables, among other things. Advocacy management told us that they update the guidance periodically, including amending it during the course of our review to make it clear that the decision to issue a public letter or hold a roundtable rests with the Chief Counsel for Advocacy. However, our review found that the updated guide continued to lack policies for documenting key decisions and activities. For example, the guide stated that when regulatory attorneys decide to intervene in the rulemaking process, they must follow up, as appropriate, with the interested associations to ensure that Advocacy has sufficient information and data to make its case for intervening. But there is no policy to document these interactions. The following are other instances where we found a lack of policies to document key activities. * Small Business Input into Comment Letters. Advocacy does not have any policies requiring that the regulatory attorneys retain documentation showing which entities provided input into comment letters, and we found that the attorneys do not consistently do so. Our analysis of Advocacy's comment letters found that about 57 percent of the letters referenced small business input or concerns. But when we interviewed seven regulatory attorneys about how they developed comment letters generally, some told us that they maintained a record of the entities providing input for specific letters, but others said that they did not. Further, when we asked Advocacy staff for documentation on the sources of the small business input referred to in a nonrepresentative sample of 11 comment letters, they were unable to provide specific emails or notes of conversations.[Footnote 28] * Reasons for Convening Roundtables. Advocacy's practices and procedures guide states that the Chief Counsel must approve the proposed agenda, speakers, and discussion topics for all proposed roundtables before participants are invited. But the guide contains no policy that this step be documented. We interviewed seven regulatory attorneys responsible for the roundtables and their management and were told that the attorneys set agendas and selected speakers based on their own assessment of the issues and, in some cases, with suggestions from small businesses or interested industry parties. Advocacy officials emphasized that they only held roundtables when there is sufficient interest or need on the part of small businesses. * Roundtable Discussions and Participants. Information gathered from the roundtables is used to inform Advocacy's positions on issues related to small businesses and in comment letters, but Advocacy's guidance contains no policies to document roundtable discussions. Our content analysis of Advocacy's comment letters showed 19 percent of them referred to roundtable discussions. However, staff did not routinely take and maintain notes of the discussions, according to the interviews we conducted with the regulatory attorneys. In addition, not all staff take attendance at the events. In most cases, Advocacy staff keep an "RSVP list" of those who have indicated that they will attend. However, some of the staff noted that the RSVP lists may not include all participants, including those on the phone. As a result it is difficult to determine the extent to which small businesses and related entities were represented at these events. Advocacy officials told us that they did not have guidance on maintaining documentation on the sources of input to comment letters or on roundtable discussions because they did not feel that setting such standards was required to fulfill their duties under Advocacy's statute. Furthermore, they noted that, in the case of roundtables, keeping records in a manner that identified specific speakers would inhibit discussion and limit their ability to gain valuable input. They also cited logistical challenges in taking accurate attendance at larger events. We acknowledge that specific parties might not want to be publicly named, but federal standards for internal control call for agencies to document significant events. Taking steps to balance the need for privacy so individuals can speak freely, with a commitment to maintain a basic level of documentation of these events--that is, documenting which entities provided input into its comment letters and roundtables--could help Advocacy demonstrate that it is meeting its mission to represent the interests of small businesses. Advocacy's Policy for Publicizing Information on Roundtable Events Is Not Followed: Key documents for Advocacy's roundtables--agendas and presentation materials--are not made available to the public at-large after the fact. The regulatory attorneys cultivate email lists of relevant stakeholders who are invited to roundtable events, and these lists are continually updated. It is Advocacy's policy to add any interested parties to the invitation list, if asked, and several of the attorneys we interviewed said they did so. They also told us they made agendas and presentations available to any interested parties who requested them after the roundtable. However, the agendas and presentations are not posted to the website or made publicly available in any systematic way and if small businesses and other interested parties do not know about the roundtables, they cannot request information from the events. Advocacy's policies and procedures state that agendas and presentations should be posted on Advocacy's website. Advocacy officials stated they had been unable to post roundtable materials to the website, which SBA maintains, because of difficulties in meeting certain readability and accessibility requirements in the Americans with Disabilities Act.[Footnote 29] However, a variety of other Advocacy reports and publications are posted on its website that also must meet these requirements, and Advocacy officials did not explain why information on the roundtables could not be likewise included. Making the roundtable materials available on its website would strengthen Advocacy's efforts to inform small businesses and other interested parties about its efforts to represent their interests to federal decision makers. Federal Advisory Committee Act Requirements Do Not Apply to Advocacy's Roundtables: In addition to evaluating whether Advocacy publicizes and conducts its roundtables in accordance with its internal policies and procedures, we also analyzed whether Advocacy's roundtable groups constitute "advisory committees" subject to the public notice and other transparency requirements of FACA.[Footnote 30] We found that the roundtables are not advisory committees, and thus Advocacy is not required by law to follow these rules. We first determined that Advocacy is an "agency" covered by FACA. It is an "authority of the Government of the United States, whether or not it is within or subject to review by another agency."[Footnote 31] In addition, Advocacy possesses the type of "substantial independent authority" required by the courts. While located within SBA, Advocacy is independent of SBA, and it has distinct statutory authorities and responsibilities, a separate statutory charter, and an appropriation account that is separate from the rest of SBA. Furthermore, its duties and authorities under statute and Executive Order are substantial and its recommendations made in furtherance of small entities' concerns must, by law, be given considerable weight by other agencies. While we found that Advocacy is a FACA "agency," we concluded that its roundtable groups are not "advisory committees" as defined by the statute and interpreted by the courts and implementing regulations. A covered advisory committee is a panel, task force, or similar group created or used by an agency for the purpose of providing advice or recommendations on particular matters. Participants' input must be sought as a group, not as a collection of individuals. The formality and fixed structure of the group also is an important factor in determining coverage under FACA. Advocacy does not seek roundtable participants' input as a group, however; rather, attendees provide their individual perspectives on the agency rule or policy under discussion, do not significantly interact with one another, and no attempt is made to reach a consensus. Advocacy's roundtables also do not have the requisite organized or fixed structure; rather, as noted, Advocacy's policy is to extend roundtable invitations to anyone who expresses a desire to attend. Finally, according to Advocacy, it is the agency itself, not the roundtable participants, that develops policy advice and recommendations, albeit based in part on the data and information provided by roundtable participants (as well as obtained elsewhere). Our detailed legal analysis of the applicability of FACA to Advocacy's roundtables is contained in appendix III. Advocacy's Workforce Planning Efforts Lack Succession Planning: Workforce planning presents some challenges for Advocacy, in part because the office is small and has a large number of positions, which, according to Advocacy officials, typically are replaced with each new administration. While the office currently does some workforce planning, its efforts do not include the long-term strategic plans that would help ensure that it maintains the expertise and skills needed to fulfill its mission. While Advocacy now has responsibility for developing its own budget, goals, and performance measures, our review of its strategic plan and goals did not find goals or objectives that discussed workforce issues such as staff development and succession planning. Advocacy officials told us that they used their organizational chart as the basis for their workforce planning efforts and discussed workforce issues with staff and managers at meetings as needed (see fig. 2). Advocacy officials said that because Advocacy was a small office with only 46 staff, the organizational chart and meetings to discuss staffing issues met their workforce planning needs. Advocacy officials told us that to develop staff they provided training to regulatory attorney and economist staff, had established a mentoring program for newer staff, and were implementing a knowledge-sharing database to help staff develop expertise in specialized policy areas. Figure 2: Office of Advocacy Organizational Chart, Fiscal Year 2014: [Refer to PDF for image: Organizational Chart] Top level: Chief Counsel for Advocacy (1): * Confidential Assistant (2); * Deputy Chief Counsel for Advocacy (1). Second level, reporting up: Office of Interagency Affairs (12); Office of Economic Research (7); Office of Information (6); Office of Regional Affairs (11); Administrative Support Branch (6). Source: Office of Advocacy. GAO-14-525. [End of figure] When we asked Advocacy officials about succession planning and their plans for addressing the departure of senior or experienced staff, the officials told us that most of the turnover in staff occurred during changes in administration. For example, according to Advocacy officials, the change of administration in 2009 resulted in the turnover of at least 15 of 20 senior officials. Among these were the 10 regional advocates, who are located in each of SBA's regional offices across the country, and make up the Office of Regional Affairs. Turnover tends to be low among the economists and regulatory attorneys, some of whom have been with the office for years. Advocacy officials said that because rulemaking often took so long, they generally had enough time to hire or realign any staff as needed. According to federal internal control standards, effective management of a workforce is essential to achieving program results. Our body of work on workforce planning has demonstrated the importance of such planning and the need to develop long-term strategies--such as training and succession planning--for acquiring, developing, and retaining staff to achieve programmatic goals.[Footnote 32] OPM's Human Capital Assessment and Accountability Framework states that agencies and offices with workforce planning are better able to manage their staff by, for example, ensuring that systematic processes are in place for identifying and addressing any gaps between current and future workforce needs. Further, OPM recommends that succession planning is needed to ensure continuity in leadership positions. In addition, workforce planning can help management determine the type of training and other strategies that are needed to address factors such as projected retirements and succession planning. In past work on succession planning, we have found that in addition to focusing on replacing individuals, succession planning strategies can also strengthen current and future organizational capacity.[Footnote 33] Given the length of time some attorneys and economists have been with Advocacy, the loss of their expertise through retirement, among other things, could leave significant gaps in needed skills and knowledge, according to Advocacy officials. Yet Advocacy has not incorporated succession planning into its workforce planning efforts, such as its training and mentoring initiatives. Further, Advocacy officials told us that they discussed future staffing needs and various options for addressing them, including training, but these efforts are not documented in a manner that would ensure consistent implementation. However, ensuring that activities such as staff training are consistently implemented is particularly important when senior management can change significantly every 4 or 8 years. Without incorporating long-term succession planning into its workforce planning efforts, Advocacy is not in the best position to ensure that it has qualified staff to fill leadership and other key positions and a skilled workforce able to meet the demands of its mission on behalf of small businesses. Conclusions: Effective internal controls are critical for the Office of Advocacy if it is to achieve program outcomes and minimize operational problems. Recognizing this, Advocacy recently has taken action to improve some of its guidance and controls. However, Advocacy's research, regulatory, and workforce planning functions could be improved by strengthening its internal controls as follows: * Research activities. In its research operations, Advocacy has taken some initial steps toward establishing stronger control policies by, for example, formalizing its peer review process for internal and external research products. However, the guidance provided did not include information on how to select appropriate peer reviewers--the experts whom Advocacy relies on to assess the quality of the research that it disseminates. The guidance also lacks policies for documenting that a peer review occurred and how reviewer comments were addressed for its external research products. Such guidance would help ensure that Advocacy research staff fully understand how best to identify the most qualified peer reviewers, and how to document and incorporate reviewer comments. In addition, Advocacy did not follow federal guidelines for information quality for influential studies that set out requirements for retaining data and taking additional steps to substantiate the quality of information it disseminates when it has certain compelling reasons to not retain the data. Improving policies and procedures for its research activities would help support Advocacy's mission to provide quality research on small business issues that decision makers and the public can rely on. * Regulatory activities. Advocacy's lack of documentation and transparency of the regulatory activities we reviewed made it difficult to validate Advocacy's efforts to represent small businesses. Specifically, weaknesses exist in Advocacy's documentation of both the sources of the small business input in comment letters and the views of small businesses discussed and conveyed at its roundtables. As a result, the extent to which the small businesses' views on regulations were being obtained and communicated is unknown. Improving its guidance to staff on its regulatory activities and emphasizing the importance of documentation would enable Advocacy to more effectively demonstrate to decision makers that it was obtaining and communicating the interests of small businesses. In addition, Advocacy is not following its internal policy to post materials from roundtables on its website. As a result, it is missing an opportunity to reach out more broadly to small businesses and other interested parties and increase the transparency of its activities. * Workforce planning activities. While Advocacy's workforce planning efforts help in managing its current staff, these efforts do not include any strategies to plan for succession, even though several staff have been with the office for many years and will eventually need to be replaced. Without this important element of workforce planning, Advocacy could be in a vulnerable position when critical staff leave the agency or staff face new demands. Although Advocacy is a relatively small office, having a skilled workforce is critical to meeting its mission. Succession planning would help ensure that Advocacy was better prepared to maintain qualified staff to conduct its research and regulatory activities in support of small businesses. Recommendations for Executive Action: To improve Advocacy's system of internal control, and help to provide reasonable assurance that the office is meeting its mission, we recommend that the Chief Counsel of Advocacy take the following five actions: * Strengthen the accountability of its research activities by taking the following two actions: - Enhance its peer review policies and procedures by including written guidance on selecting peer reviewers and documenting all steps of the peer review process--whether a peer review occurred and how and to what extent peer reviewer comments were addressed. - Develop policies and procedures that reflect the federal information quality guidelines on retaining data for influential studies, and when not retaining data, taking additional steps to substantiate the quality of information disseminated. * Strengthen the accountability of its regulatory activities by developing policies and procedures to ensure that key elements of that work--such as sources of input for comment letters and roundtable discussions--are consistently documented. * Coordinate with SBA officials who oversee website administration to comply with Advocacy's roundtable policy to make information on the events--agendas, presentation materials--publicly available on its website so that its regulatory activities are more transparent to the public. * Improve its workforce planning efforts to be better prepared to meet its future workforce needs by incorporating succession planning. Agency Comments: We provided a draft of this report to Office of Advocacy for its review and comment. In its written comments (reproduced in app. II), Advocacy agreed with our recommendations. Advocacy stated that for its research activities, its current effort to further formalize procedures for the peer review process will include steps for additional documentation and that it also plans to develop written guidelines for determining which research products are considered influential, which will clarify when Advocacy needs to take additional information quality steps. Advocacy also said that as it develops how it will disseminate information about its regulatory activities, it will incorporate approaches that are responsive to our recommendations. Finally, Advocacy agreed that workforce planning is important for ensuring that the office maintains the skills and resources needed to fulfill its mission and noted that the office is developing a Leadership Succession Plan in response to our recommendation. We are sending this report to the Office of Advocacy and interested congressional committees. The report also will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact Cindy Brown Barnes at (202) 512-8678 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. Sincerely yours, Signed by: Cindy Brown Barnes: Acting Director, Financial Markets and Community Investment: [End of section] Appendix I: Objectives, Scope, and Methodology: This report examines Advocacy's (1) research activities; (2) regulatory activities, including the applicability of the Federal Advisory Committee Act (FACA) to Advocacy's roundtables; and (3) workforce planning efforts. The scope of our work covered key activities conducted from fiscal year 2008 to fiscal year 2013, although in some cases, we broadened that scope to include fiscal years 2005 through 2008, as noted below. To understand Advocacy's mission, operations, and participation in the rulemaking process, we reviewed relevant laws and regulations, Advocacy documents, including its annual reports to Congress on the Regulatory Flexibility Act (RFA), budget and strategic planning documents, and other publications. To assess Advocacy's research activities, we determined: how Advocacy staff chose its research topics; how they conducted the research; and what controls they had in place to ensure quality products. We analyzed Advocacy's research products produced in the 5 most recent fiscal years--2008 through 2012--and interviewed relevant Advocacy officials on the processes by which they assess the quality of its research products.[Footnote 34] To evaluate Advocacy's research activities, we analyzed a nonprobability sample of research, which included 10 research products authored by Advocacy staff (internal), and 10 products produced by its contractors (external). To select a sample of internal research products that best represented the variety of internal research, we reviewed the 58 research products issued from fiscal years 2008 through 2012, and categorized each as either "routine" (meaning published either quarterly or semi-annually, and relying on the same underlying data set), or as "nonroutine." Within the routine category, we selected the mostly recently published product from each of the following routine publications: (1) Small Business Profiles of the States and Territories, (2) Small Business Quarterly Bulletin, (3) Quarterly Lending Bulletin, (4) Small Business Lending, and (5) The Small Business Economy. For the nonroutine products, we created five categories, such as "analysis" and "fact sheets." We selected, from each of those five categories, the most recently published report. Then, for each of the 10 selected internal research products, we used a data collection instrument and interviewed the Advocacy staff who wrote the products in order to see to what extent the products adhered to Information Quality Guidelines issued by the Office of Management and Budget (OMB) and the Small Business Administration (SBA) and Advocacy's internal quality review process ("peer review").[Footnote 35] To assess the quality of the peer review used for Advocacy's external research, we reviewed the contract files for the 10 most recently published external research products, as of year-end 2013, and compared how the work was produced, including Advocacy's peer review process, against the OMB Peer Review Guidance and OMB and SBA information quality guidelines, and applicable federal internal control standards.[Footnote 36] The results from our reviews cannot be projected to all Advocacy studies, but they provide an indication of how Advocacy staff conducted or oversaw the research and what controls were in place to ensure quality products. We also assessed Advocacy's contract studies that focused on the economic costs of regulation. We focused on external studies in these areas because they are key research areas for Advocacy, according to its originating statute and the mission statement of its Office of Economic Research. For the economic costs studies, we assessed them against OMB and SBA Information Quality Guidelines, and examined Advocacy's compliance with related data retention policies therein. [Footnote 37] We also reviewed peer reviewers' comments and other external reviews on the studies as part of our assessment. Lastly, we interviewed Advocacy officials to obtain information on steps they took to substantiate the quality of information for influential studies when data are not retained. To evaluate Advocacy's regulatory activities, we assessed: how and why Advocacy decides to issue comment letters and convene roundtables; the policies and practices in place that pertain to those activities; how Advocacy staff solicit input from small businesses and other parties. We interviewed relevant staff, as described below; reviewed relevant policies and procedures, and analyzed comment letters produced and roundtables convened from fiscal years 2009 through 2013.[Footnote 38] Specifically, we performed a content analysis on the 181 comment letters issued during that period, using NVivo software, to analyzed and categorize the content of the letters and the nature and source of the small business input provided.[Footnote 39] We also interviewed Advocacy staff who are responsible for rulemaking and related activities, weighting our choices of interviewees to reflect the distribution of comment letters by policy area, and reviewed supporting documentation for comment letters to understand how Advocacy staff develop and issue comment letters, and compared this information to federal internal control standards. Similarly, we reviewed available information on the 142 roundtables convened during the same 5-year period and interviewed the responsible Advocacy staff, weighting our choices of interviewees to reflect the distribution of roundtables by policy area. We reviewed Advocacy's policies for roundtables to understand their origin, purpose, and documentation requirements, and compared this information to federal internal control standards. In order to understand the perspective of those who attended the roundtables, we attended three of the events. In addition, we selected and interviewed a nonprobability sample of past participants from email lists provided by Advocacy's regulatory attorneys. Specifically, we talked to the following representatives: five from industry associations that represent small and large businesses; one from a large corporation; and one nonprofit organization whose mission relates to policies or rules under consideration.[Footnote 40] We made our selections to include representatives from a variety of sources, and while the results from our interviews cannot be projected to all entities that interact with Advocacy, the information we gathered does provide insights into how the selected groups view Advocacy and its work in representing small businesses to federal policymakers. We also reviewed Advocacy's training to agency staff in how to comply with the RFA and interactions with agency rulemaking officials. In addition, we interviewed officials from the entities that interact with Advocacy in rulemaking--the Environmental Protection Agency (EPA), Occupational Safety and Health Administration (OSHA), Bureau of Consumer Financial Protection, also known as the Consumer Financial Protection Bureau (CFPB), and Office of Management and Budget's (OMB) Office of Information and Regulatory Affairs (OIRA). To analyze whether FACA (5 U.S.C. App. II) applies to Advocacy's roundtables, we reviewed relevant statutes, case law, regulations, and guidance. In addition, we reviewed and considered Advocacy's written views on the issue. See appendix III for more information on the legal analysis we conducted. Finally, we reviewed Advocacy's workforce efforts. To understand Advocacy's workforce planning, we reviewed Advocacy's strategic goals and other planning and budget documents and interviewed senior management to determine what, if any, policies and procedures were in place related to workforce and succession planning. We assessed any such policies and procedures against applicable federal standards for internal control and the Office of Personnel Management's (OPM) Human Capital Assessment and Accountability Framework (HCAAF).[Footnote 41] We also reviewed GAO reports on workforce and succession planning to gain insights about key practices and how agencies have used them. We conducted this performance audit from August 2013 to July 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Office of Advocacy, Small Business Administration: SBA Office of Advocacy: 409 3rd Street, SW: Washington, DC 20416: [hyperlink, http://www.sba.gov/advocacy] Advocacy: the voice of small business in government July 11, 2014: Ms. Cindy Brown Barnes: Acting Director, Financial Markets and Community Investment: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Ms. Barnes: Thank you for providing the SBA Office of Advocacy with a copy of the Government Accountability Office (GAO) draft report entitled Office of Advocacy Needs to Improve Controls Over Research, Regulatory, and Workforce Planning Activities. We have reviewed GAO's recommendations and welcome the opportunity to comment. In addition, Advocacy appreciates the time and effort GAO took to understand how we fulfill the unique mission of our office. The following statements reference recommendations put forth by GAO: * Research Activities: Advocacy appreciates GAO's recommendations regarding our research activities. As GAO states, Advocacy is currently working on guidance to further formalize procedures for the peer review process, which will include additional documentation. Advocacy also plans to develop written guidelines for determining what Advocacy research products would be considered "influential," which will further clarify needed information quality checks. Advocacy agrees with GAO that such steps will strengthen our ability to provide robust and quality economic research for small businesses and policymakers. * Regulatory Activities: Advocacy's use of comment letters and roundtables is central to our mission to advocate for small businesses, and we appreciate GAO's recommendation to improve our documentation of these activities. Advocacy will continue to develop how we disseminate information about our regulatory activities in ways that are both responsive to the GAO recommendations and in furtherance of Advocacy's mission. * Advocacy's Workforce Planning Activities: GAO recommends that Advocacy improve our workforce planning efforts by incorporating succession planning. Advocacy agrees that workforce planning is an important part of ensuring that our office maintains the skills and resources needed to fulfill our important mission. As a result of this GAO recommendation, Advocacy is developing a Leadership Succession Plan that will formalize existing and future workforce planning efforts. Also, GAO agreed with SBA and Advocacy that Federal Advisory Committee Act requirements do not apply to Advocacy's roundtables. Advocacy values the thorough investigation that GAO performed. As mentioned before, our roundtables are an invaluable tool to hear from small businesses and these findings will help our office to continue this important part of our outreach. The SBA Office of Advocacy is committed to continuing to fulfill our mission as the independent voice for small business by researching small business issues and providing input into federal rulemaking and related regulatory activities. Sincerely, Signed by: Claudia Rayford Rodgers: Deputy Chief Counsel: SBA Office of Advocacy: [End of section] Appendix III: Legal Analysis of the Applicability of the Federal Advisory Committee Act to the Office of Advocacy's Roundtables: Introduction and Summary of Conclusions: As part of our review of the Small Business Administration Office of Advocacy's (Advocacy) activities in identifying and communicating the concerns of small businesses and other small entities as agencies develop regulations and policies, we analyzed whether the Federal Advisory Committee Act (FACA), 5 U.S.C. App. II, applies to "roundtable" groups that Advocacy convenes for the purpose of conveying and soliciting information and viewpoints about agency rules and policies. Based on our review of relevant statutes and legislative history, case law, regulations, and guidance, and consideration of Advocacy's views, and for the reasons detailed below, we conclude that these roundtable groups are not "advisory committees" within the meaning of FACA and thus Advocacy is not required to comply with the provisions of that statute.[Footnote 42] Background on Advocacy's Duties and Use of Roundtables: Advocacy is an office established by statute within the Small Business Administration (SBA). 15 U.S.C. § 634a. The office has a separate statutory charter[Footnote 43] and, as of 2010, the President must include a separate line-item appropriation request for Advocacy designated in a separate Treasury account.[Footnote 44] Advocacy is headed by a Chief Counsel who, among other things, has statutory authority to appear as an official "friend of the court" (amicus curiae) in all court challenges to agency regulations in order to "present his or her views with respect to compliance with [the Regulatory Flexibility Act, discussed below], the adequacy of the rulemaking record with respect to small entities and the effect of the rule on small entities." 5 U.S.C § 612(c). According to recent testimony, Advocacy is not required to obtain clearance of its comment letters, publications, and testimony as are other executive branch agencies.[Footnote 45] Among Advocacy's duties are serving as "a focal point for the receipt of complaints, criticisms, and suggestions" concerning federal policies affecting small businesses; developing proposals for changes in the policies and procedures of federal agencies that further small business interests and communicating those proposals to the appropriate federal agencies; and "representing the views and interests of small businesses before other Federal agencies whose policies and activities may affect small business." 15 U.S.C. § 634c. Advocacy also has key responsibilities and authorities under the Regulatory Flexibility Act (RFA), 5 U.S.C. §§ 601 et seq., and under Executive Order 13272 which helps implement the RFA. The RFA requires Advocacy to assist regulatory agencies during all stages of the rulemaking process to mitigate the potential adverse impact of the rules on small entities while achieving their regulatory objectives, and the Executive Order requires agencies to notify Advocacy of any proposed rules expected to have a significant economic impact on a substantial number of small entities and to consider any comments Advocacy submits on the rule. The RFA and Executive Order both require agencies to respond to Advocacy comments in their final rules and note any resulting changes. Advocacy plays a similar role with respect to certain designated agencies as a member of what are known as "SBREFA panels." The Small Business Regulatory Enforcement Fairness Act of 1996 (SBREFA)[Footnote 46] amended the RFA to require the Environmental Protection Agency, the Occupational Safety and Health Administration, and the Consumer Financial Protection Bureau to convene panels with officials from Advocacy, the Office of Management and Budget, and the rule-writing agency whenever the agencies develop rules for which an initial regulatory flexibility analysis is required. The panels must develop information solicited from small entity representatives and other sources concerning the potential impacts of the agency proposal; consider alternatives that minimize burdens on small entities; and prepare a report with recommendations to the agency head for consideration in the proposed rule. 5 U.S.C. § 609(b)-(e). One of the ways that Advocacy fulfills these statutory responsibilities is by convening roundtable meetings for representatives of small businesses and government agencies. Advocacy invites roundtable participants, typically by email, but it is Advocacy's policy to welcome participation by any interested party other than the press.[Footnote 47] According to Advocacy, it hosted 32 roundtables in Fiscal Year 2012 and 21 roundtables in Fiscal Year 2013.[Footnote 48] The roundtables typically focus on specific proposed agency regulations and stakeholders provide information about the potential economic impacts of the proposal and identify less burdensome alternatives. Advocacy often uses the results of these roundtables (and other information) as the basis of formal comment letters it submits to the proposing agency, for example disagreeing with the agency's impact analysis and urging less burdensome alternatives; it submitted 26 such letters in FY 2013[Footnote 49] to which agencies must, by statute, respond. 5 U.S.C. § 604(a)(3). In Advocacy's view, the concerns articulated by small business representatives at roundtables are influential; for example, according to Advocacy, after a roundtable regarding the Employee Benefits Security Administration's guidance on brokerage windows, the Administration issued revised guidance addressing the small business concerns. See Report on the Regulatory Flexibility Act, FY 2012, above, at 7. And after attending an Advocacy roundtable that addressed proposed revisions to Occupational Safety and Health Administration ("OSHA") noise standards, OSHA withdrew the rule and pledged to conduct additional stakeholder outreach before proceeding. See Report on the Regulatory Flexibility Act, FY 2011 at 4. Applicability of the Federal Advisory Committee Act: FACA establishes requirements for "advisory committees" to the federal government, defined as relevant here as "any committee, ... panel, task force, or other similar group ... which is established or utilized by one or more agencies, in the interest of obtaining advice or recommendations for ... one or more agencies or officers of the Federal Government ...." 5 U.S.C. App. II, § 3(2). Congress enacted FACA in 1972 "to enhance the public accountability of advisory committees and to reduce wasteful expenditures" that "result only in worthless committee meetings and biased proposals". Pub. Citizen v. U.S, Dep't of Justice, 491 U.S. 440, 459 (1989). Similarly, FACA's purpose was to "ensure that new advisory committees be established only when essential [;] ... that their creation, operation, and duration be subject to uniform standards and procedures; that ... the public remain apprised of their existence, activities, and cost; and that their work be exclusively advisory in nature." Id. at 446. For advisory committees covered by the act, FACA requires, among other things, that their meetings be open to the public; that timely notice of meetings be published in the Federal Register; that interested parties be allowed to attend, appear before, or file statements with the advisory committee; and that detailed minutes of advisory committee meetings be kept. In addition, for advisory committees established by agency heads, the agency must, as applicable, ensure that committee membership is fairly balanced in terms of points of view represented and functions to be performed and ensure that the recommendations of the committee are not inappropriately influenced by the agency or any special interest but rather reflect the committee's independent judgment. 5 U.S.C. App. II, § 10. Because Advocacy does not apply these FACA requirements to its roundtables, we examined whether the statute applies to these groups and their activities. 1. Is Advocacy a FACA "Agency"? As noted above, whether Advocacy roundtables are advisory committees subject to FACA depends in part on whether they have been "established or utilized by one or more agencies ...." (emphasis added). Thus as a threshold issue, we examine whether Advocacy is an "agency" for purposes of FACA. FACA defines "agency" by incorporating the term's general definition in the Administrative Procedure Act (APA). 5 U.S.C. App. II, § 3(3). With exclusions not relevant here, the APA defines "agency" as "each authority of the Government of the United States, whether or not it is within or subject to review by another agency ...." 5 U.S.C. § 551(1). Advocacy believes it is an "agency" for purposes of FACA. Citing the foregoing APA definition, Advocacy states that it is an "authority of the Government of the United States" and not covered by any of the exclusions. Advocacy Letter to GAO at 2. Yet as the courts have recognized, "the law on the simple question of what is an [APA] agency is quite complex." Lee Constr. Co. v. Federal Reserve Bank, 558 F. Supp. 165, 172 (D. Md. 1982). Likewise, the courts have acknowledged that "any general definition [of "agency" under the APA] can be of only limited utility ... [when] confronted with one of the myriad organizational arrangements for getting the business of the government done.... The unavoidable fact is that each new arrangement must be examined anew and in its own context." Washington Research Project, Inc. v. HEW, 504 F.2d 238, 245-46 (D.C. Cir. 1974), cert. denied, 421 U.S. 963 (1975)(citations omitted). Courts have taken somewhat different approaches to what constitutes an "agency" under the APA. The seminal case is Soucie v. David, 448 F.2d 1067 (D.C. Cir. 1971), which involved what was then called the Executive Office of the President's Office of Science and Technology (OST). The lawsuit sought the release of an OST document under the Freedom of Information Act (FOIA). Because the FOIA and APA definitions of "agency" were identical at the time of the lawsuit, the court looked to interpretations of the APA in determining whether OST was an "agency" under FOIA. The court found that while the meaning of "agency" under the APA was "not entirely clear," it "apparently confers agency status on any administrative unit with substantial independent authority in the exercise of specific functions." 448 F.2d at 1073 (emphasis added). Because OST evaluated federal science programs previously evaluated by the National Science Foundation, in addition to its non-"independent" responsibilities in advising and assisting the President, the court found OST had "substantial independent authority" and "must be regarded as an agency subject to the APA and [FOIA]." Id. at 1075. The court in McKinney v. Caldera, 141 F. Supp. 2d 25 (D.D.C. 2001), applied the Soucie test and found that the office of the Judge Advocate General of the Army (JAG) was not an APA "agency" because it lacked "substantial independent authority." Although JAG supervised the Army's military justice system, performed appellate review of court martial trials, and furnished legal services, the court focused on the fact that JAG's duties flowed from the authority of the Secretary of the Army. This rendered JAG insufficiently independent, the court ruled, notwithstanding that the APA contemplates the possibility of an "agency" within an "agency." Other courts have framed the APA "agency" definition slightly differently, looking to whether an entity can take "final and binding action." This standard is based in part on the Soucie "substantial independent authority" test but also on the APA's legislative history indicating Congress intended covered agencies to have "final and binding" legal authority. In Dong v. Smithsonian Institution, 125 F.3d 877 (D.C. Cir. 1997), cert. denied, 524 U.S. 922 (1998), for example, the court looked to whether the Smithsonian Institution was: "a part of government which is generally independent in the exercise of [its] functions and ... by law has authority to take final and binding action affecting the rights and obligations of individuals, particularly by the characteristic procedures of rule-making and adjudication." Id. at 881 (emphasis added). The court found that the Smithsonian was not an APA "agency" under this standard because while it exercised authority, the authority was not sufficiently "substantial." Although the Smithsonian is closely linked with the federal government, receives federal funding, has certain police powers to protect its physical plant, and publishes regulations in the Code of Federal Regulations, the court noted that it does not make binding rules of general application, determine rights and duties through adjudications, issue orders, or perform regulatory functions.[Footnote 50] Still other courts determining APA agency status have focused on whether an entity is one of the "centers of gravity" in the exercise of administrative power "where substantial 'powers to act' . . .are vested'" (see, e.g., Lee Constr. Co., above, 558 F. Supp. at 174, holding Federal Reserve Bank of Richmond was APA agency based on substantial delegated decision-making authority from Federal Reserve System Board of Governors), or whether the entity has "authority in law to make decisions" (see, e.g., Washington Research Project, Inc. v. HEW, above, 504 F.2d at 248, holding that the Department of Health, Education, and Welfare (HEW) peer review groups for grant applications were not APA agencies because they only recommended grants to a National Institutes of Health entity, which, in turn, made recommendations to the HEW Secretary). Applying these authorities and considering Advocacy's views, we conclude that Advocacy is an "agency" under the APA, and thus under FACA, because it possesses the requisite "substantial independent authority." First, it acts with independence. Although Advocacy is an "agency" within an "agency" (SBA), as noted, it has distinct statutory authorities and responsibilities and a separate statutory charter; a separate line-item budget request; its head can appear in court appeals to speak to the rulemaking agency's compliance with the RFA; and it does not have to obtain clearance of its comments and testimony. Unlike JAG in the McKinney case, above, which the court found was not an "agency" because its duties derived from the Army Secretary, the Chief Counsel's duties are related to but distinct from the SBA Administrator. Second, Advocacy's authorities and duties under its organic statute, the RFA, and Executive Order 13272 are "substantial." As noted, it is responsible among other things for identifying and advocating the views and concerns of small entities throughout the Executive Branch and for developing proposed changes in the policies and procedures of federal agencies in order to further the interests of small business. Although it does not issue regulations, conduct adjudications, or otherwise take traditional "final and binding action," its views and recommendations must, by law, be accorded considerable weight. Taken as a whole, we believe it has substantial authority under the case law. 2. Are Advocacy's Roundtable Groups FACA "Advisory Committees"? Having found that Advocacy is a FACA "agency," we examine whether its roundtables are FACA "advisory committees." As noted, FACA defines this term in part as "any committee, ... panel, task force, or other similar group ... which is established or utilized by one or more agencies, in the interest of obtaining advice or recommendations for ... one or more agencies or officers of the Federal Government ...." 5 U.S.C. App. II, § 3(2). Because of Advocacy's particular statutory responsibilities, notably to identify and communicate concerns of small businesses and entities, Congress clearly contemplated that Advocacy would have a close relationship with small business. Its mission virtually requires consultation with these entities to obtain data, opinions and possible advice, and one way it has chosen to do this is by holding roundtable meetings. In Advocacy's view, its roundtables are not FACA advisory committees. It believes the roundtables help it achieve its mission;[Footnote 51] indeed, Advocacy maintains that without the discussions that occur at roundtables, its "ability to fulfill its . . . statutory obligations would be severely curtailed." Advocacy Letter to GAO at 2. Explaining its position that these activities do not trigger FACA, it states: "although the definition [in] FACA is broad, FACA does not apply to all committees, meetings, or groups.... In enacting FACA, Congress was concerned with formally organized advisory committees which the President or an executive department or official directed to make recommendations on identified governmental policy for which specific advice was sought. It was not intended to apply to all amorphous, ad hoc group meetings; only groups having some sort of established structure and defined purpose constitute advisory committees within the act." Advocacy Letter to GAO at 3 (citations omitted). We agree with Advocacy that under the case law, not every group that provides information or advice to a federal agency is a FACA "advisory committee." Although no single aspect of a group's creation, structure or activities is controlling, most courts have focused on the formality, structure, and continuity of the group and whether the group is brought together to provide advice or recommendations as group rather than as a collection of individuals. As the Supreme Court explained in Public Citizen, above, Congress did not design FACA to cover every formal or informal consultation between a government agency and a group rendering advice; such a literalistic reading would cover "far more groups and arrangements than Congress could conceivably have intended." 491 U.S. at 464-65. Several early cases addressed whether an informal or "ad hoc" group such as Advocacy's roundtables was an advisory committee. The court in Food Chemical News v. Davis, 378 F. Supp. 1048 (D.D.C. 1974), considered whether two "informal" meetings of distilled spirits industry representatives, convened by the head of the Bureau of Alcohol, Tobacco and Firearms (BATF) to obtain advice on draft regulatory amendments, were subject to FACA. Noting that the act's purpose was to control the advisory committee process and to open to public scrutiny the manner in which government agencies obtain advice from private individuals, the court held it was "undisputed that [BATF] utilized an ad hoc committee of industry representatives in order to obtain advice. Such a relationship ... clearly comes within the terms of ... [FACA]." Id. at 1051. The following year, the same court considered whether bi-weekly meetings held at the White House and attended by selected groups invited by the President's Assistant for Public Liaison were subject to FACA. In Nader v. Baroody, 396 F.Supp. 1231 (D.D.C. 1975), the court concluded that the groups were not advisory committees, relying on two characteristics of the meetings: the absence of a specific request for advice on a matter of government policy and the lack of formal organization or continuity. The Second Circuit Court of Appeals considered Food Chemical News and Nader in National Nutritional Foods Association v. Califano, 603 F. 2d 327 (2d Cir. 1979). The Califano court determined that FACA applied to a single meeting between six officials from the Food and Drug Administration and six obesity experts "to assist the FDA in selecting the best course of action 'for regulating the production and promotion of [weight reduction products]...." Agreeing with the rulings of both previous cases, the court found the facts of Food Chemical News to be closer to the case before it because FDA had called on specific individuals to provide group advice on a particular proposed regulation and relied on the group advice. The element of "group advice" continued to play a key role in the seminal case of Association of Am. Physicians and Surgeons, Inc. v. Clinton, 997 F. 2d 898 (D.C. Cir. 1993) (AAPS), involving the Clinton Administration's Health Care Reform Task Force. The D.C. Circuit Court of Appeals held in AAPS that "a group is a FACA advisory committee when it is asked to render advice or recommendations, as a group, and not as a collection of individuals." Id. at 913 (emphasis in original). "[A]n important factor in determining the presence of an advisory committee [is] the formality and structure of the group," the court held, laying out three criteria needed to implicate FACA: "an organized structure, a fixed membership, and a specific purpose." Id. at 914. Characterizing variation in these factors as a continuum, the court explained that at one end of the continuum is a "formal group of a limited number of private citizens who are brought together to give publicized advice as a group"--this group is covered--while at the other end of the continuum is "an unstructured arrangement in which the government seeks advice from what is only a collection of individuals who do not significantly interact with one another." Id. at 915.[Footnote 52] See also Freedom Watch v. Obama, 807 F. Supp. 2d 28 (D.D.C. 2011) (applying AAPS criteria; finding colorable claim of FACA coverage based on fixed group membership and formation for specific policy purpose); Citizens for Responsibility and Ethics in Washington v. Leavitt, 577 F. Supp. 2d 427, 432 (D.D.C. 2008) (same; finding no FACA coverage where "[t]he groups had no formal organizational structure .... Attendees conveyed their own opinions regarding their individual areas of expertise" and "[n]o group report or other collaborative work product was created."). But see Aluminum Co. of America v. National Marine Fisheries Service, 92 F. 3d 902 (6th Cir. 1996) (cautioning against rigid application of AAPS criteria). Implementing regulations issued by the General Services Administration (GSA), the agency responsible for administering FACA government-wide, rely in part on this case law in defining what constitutes a FACA advisory committee. In issuing its final FACA management rule in 2001, GSA "recognize[d] that the broad definition in the Act of an 'advisory committee' might be interpreted to extend coverage by the Act to any gathering of two or more persons from whom ... Federal officers or agencies seek advice or recommendations. However, in the cases discussed above, the courts have rejected such a broad reading ...." 66 Fed. Reg. 37728, 37730 (July 19, 2001). GSA's regulations therefore exempt from FACA regulation "any group that meets with Federal official(s), including a public meeting, where advice is sought from the attendees on an individual basis and not from the group as a whole...." 41 C.F.R. § 102-3.40(e). GSA regulations further state that development of consensus among meeting participants does not in itself convert the group to a FACA committee, nor does the presence of trade association or other organization representatives affect whether advice is obtained "on an individual basis." 41 C.F.R. Parts 102-3, Subpart A, App. A, Key Points and Principles II, III. Applying these authorities and considering Advocacy's views, we conclude that Advocacy's roundtables are not FACA advisory committees. First, we recognize that roundtable participants are not random attendees but rather invited representatives of a particular "group"-- the small business community--whose concerns and perspective it is Advocacy's very mission to identify and advocate.[Footnote 53] As noted, however, Advocacy invites anyone (except the press) with an expressed desire to attend and thus there is no structured or fixed membership in a "roundtable group." Nor is group consensus sought from roundtable participants. Under the standards in AAPS, 603 F.2d at 913, the group's activities are not expected to, and do not appear to, benefit from the interaction among the participants. Rather, as contemplated by the GSA regulations, information is solicited from participants as individuals (including individuals representing trade associations), although a consensus may develop. See 41 C.F.R. Part 102-3, Subpart A, App. A, Key Point II ("The development of consensus among all or some of the attendees at a public or similar forum does not automatically invoke FACA."). As in Leavitt, above, no group report or other collaborative work product is created. And as in American Society of Dermatology v. Shalala, 962 F. Supp. 141 (D.D.C 1996), there is no attempt to create consensus among participants either overall or on any aspects of the issue under consideration. See id. at 148 (among the "main factors leading to the Court's determination that the panels fall into the non-advisory committee end of the [AAPS] continuum [is] . . . the absence of any attempt to achieve consensus."). Nor does Advocacy's submission of comment letters based on information obtained at roundtables establish that it convened the roundtable to obtain a group consensus. While Advocacy's position in a comment letter may represent a consensus of views expressed at a roundtable, that consensus appears to be a function of the fact that many of the participants, as small entities, share common concerns. Finally, according to Advocacy, it only obtains data and information from roundtable participants, not policy advice or recommendations. It is Advocacy itself that develops policy recommendations, it told us, based on information it obtains at roundtables and elsewhere and using its own judgment and expertise. Even if advice and recommendations were offered at roundtables, however (a small business representative recommending a regulatory policy change based on its impact on small business, for example), in our view, this would not outweigh the other factors indicating that the roundtables are not covered advisory committees.[Footnote 54] Conclusions: Advocacy is an "agency" under FACA first because it meets the statutory definition of this term in FACA and the Administrative Procedure Act incorporated by reference: it is an "authority of the Government of the United States, whether or not it is within or subject to review by another agency" and not subject to statutory exemption. In addition, Advocacy has the type of "substantial independent authority" and other indicia of "agency" status that courts interpreting the APA definition have required. However, Advocacy's roundtable groups are not FACA "advisory committees" because they do not have an organized or fixed structure; there is no attempt to reach consensus about an agency regulation or policy but instead viewpoints are sought from individual participants; and it is only data and information, according to Advocacy, rather than advice or policy recommendations, that participants provide. [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Cindy Brown Barnes (202-512-8678), brownbarnesc@gao.gov. Staff Acknowledgments: In addition to the contact named above, Kay Kuhlman and Triana McNeil (Assistant Directors), Kristeen McLain (Analyst-in-Charge), Emily Chalmers, William Chatlos, John McGrail, Lauren Nunnally, Roberto Pinero, Jena Sinkfield, Jack Wang, and Carrie Watkins made key contributions to this report. [End of section] Footnotes: [1] P. L. No. 94-305, §201, 90 Stat. 668 (1976). [2] OMB, Final Information Quality Bulletin for Peer Review, December 16, 2004. This Bulletin established government-wide guidance aimed at enhancing the practice of peer review of government science documents. OMB. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. 67 Fed. Reg. 8452 (Feb. 22, 2002). OMB's information quality guidelines implement section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (P. L. No. 106- 554). Section 515 directed OMB to issue government-wide guidelines that "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies." Agencies were required to issue their own implementing guidelines by October 1, 2002. For SBA's guidelines, accessed April 24, 2014, see [hyperlink, http://www.sba.gov/about- sba/sba_performance/open_government/information_quality/information_qual ity_guidelines]. Advocacy officials told us they follow SBA's information quality guidelines. [3] Human Resources Management in Agencies, HCAAF and HCAAF Systems, Standards, and Metrics, 73 Fed. Reg. 23012 (Apr. 28, 2008). [4] Prior to establishing a separate Office of Advocacy within SBA in 1976, Congress created the position of Chief Counsel for Advocacy within SBA. See Pub. L. No. 93-386, § 10, 88 Stat. 749 (1974). [5] Under the RFA, the term "small entity" includes small businesses, small governments, and small organizations. 5 U.S.C. § 601(6). For the purposes of this report, we use the term "small businesses" since that is our focus. [6] When an agency can determine that the rule would not have a "significant economic impact on a substantial number of small entities," the head of the agency may certify to that effect and forego the regulatory flexibility analysis requirements. (5 U.S.C. § 605). [7] The Small Business Regulatory Enforcement Fairness Act of 1996 (P. L. No. 104-121, title II). [8] The Dodd-Frank Wall Street Reform and Consumer Protection Act (P. L. No. 111-203). [9] Executive Order 13272, Proper Consideration of Small Entities in Agency Rulemaking, 2002. [10] Pub. L. No. 111-240, § 1601(4), 124 Stat. 2504, 2551 (codified at 5 U.S.C. § 604(a)(3)). [11] See Pub. L. No. 111-240, sec. 1602(b) [12] Under Public Law 111-240, SBA must provide support to Advocacy, including office space and equipment, communication and IT services, and equipment and facilities maintenance. Therefore any costs of those services, as well as centralized indirect expenses shared with other SBA offices, would appear in SBA's budget request, not Advocacy's. [13] P. L. No. 94-305, §202, 90 Stat. 668 (1976), as amended. [14] The regulatory attorneys also engage with rulemaking agencies through the interagency review process, which is run by OMB's Office of Information and Regulatory Affairs (OIRA). However, OIRA and its operations are outside the scope of this report. [15] OMB, Final Information Quality Bulletin for Peer Review, December 16, 2004. [16] GAO, Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). These internal control standards, first issued in 1983, present the internal control standards for federal agencies for both program and financial management. [17] Of the 10 external products we reviewed, 6 did not contain documentation on whether a peer review occurred. Of the 10 internal products we reviewed, none contained documentation on whether a peer review occurred. [18] According to our analysis, of the 70 internal research products produced from fiscal years 2008 through 2012, approximately 54 percent were shorter products, between 1 to 5 pages. [19] According to Advocacy officials, it is not permissible to expand the scope of the research or change the nature of the deliverables specified in the contract after an award has been made. Therefore, if the peer reviewer's comments discuss changes that would result in expanding the scope of the research (e.g., analyzing data from additional time periods), those comments could not be accommodated. [20] Andy Bollman, E. H. Pechan & Associates, Evaluation of Barrier Removal Costs Associated with the 2004 Americans with Disabilities Act (ADA) Accessibility Guidelines, prepared for SBA Office of Advocacy, under contract SBAHQ-03C0020 (Washington, D.C.: 2007); W. Marc Crain, The Impact of Regulatory Costs on Small Firms, prepared for SBA Office of Advocacy, under contract SBHQ-03-M-0522 (Washington, D.C.: 2005); and Nicole V. Crain and W. Mark Crain, The Impact of Regulatory Costs on Small Firms, prepared for SBA Office of Advocacy, under contract SBAHQ-08-M-0466 (Washington, D.C.: 2010). We refer to the latter two studies as "the Crain studies." [21] The study by Andy Bollman and E. H. Pechan & Associates was also classified as "influential." [22] In the case of the 2010 Crain study, the original underlying data for the economic regulation cost regression were publicly available from the World Bank and the Organisation for Economic Co-operation and Development, and the study's regressions contained only 118 records and 6 explanatory variables. Furthermore, since the regression models employed in the studies consisted of one equation each, we do not believe there would have been significant costs associated with requiring the Crains to provide Advocacy the data and computer coding written to produce the estimates. While Advocacy did not classify the 2005 Crain study as "influential" according to the OMB and SBA criteria, because the 2005 study used similar methodologies, we examined them together. [23] We asked Advocacy officials how they substantiated the two Crain studies. We did not ask the Advocacy officials to verify the quality of the third regulatory cost study (Evaluation of Barrier Removal Costs Associated with the 2004 Americans with Disabilities Act (ADA) Accessibility Guidelines) because our work did not reveal the same level of methodological weaknesses that we identified in the Crain studies. [24] Advocacy officials also provided us with rebuttals the Crains published in response to earlier criticisms by others of the studies. However, the rebuttals do not directly address important criticisms. [25] This one-page statement notes, among other things, that "The study is a top-down analysis of regulatory costs that uses certain assumptions to estimate totals, and is not a bottom-up precise accounting of the overall cost of regulations." Also, the statement says: "The overall figure of $1.75 trillion in costs is derived from a number of different assumptions and sources to create an estimate. As with almost any academic methodology, it was not intended to be considered a precise finding." For the full statement, see the report, available on Advocacy's website, accessed on April 24, 2014, [hyperlink, http://www.sba.gov/advocacy/7540/49291]. [26] OMB. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. 67 Fed. Reg. 8452 (Feb 22, 2002). [27] See Advocacy letter to Environmental Protection Agency, dated November 28, 2008, Re: Comments on EPA's Advance Notice of Proposed Rulemaking "Regulating Greenhouse Gas Emissions under the Clean Air Act," Docket ID No. EPA-HQ-OAR-2008-0318; Advocacy letter to Environmental Protection Agency, dated June 23, 2009, Re: Docket ID No. EPA-HQ-OAR-2009-0171, Comments on EPA's "Proposed Endangerment and Cause or Contribute Findings for Greenhouse Gases Under Section 202(a) of the Clean Air Act"; Advocacy letter to National Marine Fisheries Services, Department of Commerce, dated October 30, 2009, Re: RIN 0648- AW18 Magnuson-Stevens Fishery Conservation and Management Act: Regional Fisheries Management Councils; Operations. [28] We selected a nonrepresentative sample of recently published comment letters--published from 2011 through 2013--weighting our choices of letters to reflect the distribution of policy areas listed in table 2. [29] 42 U.S.C. § 12101 et seq. [30] 5 U.S.C. App. II. [31] 5 U.S.C. App. II, § 3(3), incorporating 5 U.S.C. § 551(1) (Administrative Procedure Act). [32] GAO, Human Capital: Key Principles for Effective Workforce Planning, [hyperlink, http://www.gao.gov/products/GAO-04-39] (Washington, D.C.: Dec. 11, 2003), and Securities and Exchange Commission: Improving Personnel Managements is Critical for Agency's Effectiveness, [hyperlink, http://www.gao.gov/products/GAO-13-621] (Washington, D.C.: July 18, 2013). [33] GAO, Human Capital: Selected Agencies Have Opportunities to Enhance Existing Succession Planning and Management Efforts, [hyperlink, http://www.gao.gov/products/GAO-05-585] (Washington, D.C.: June 30, 2005). [34] For the period we conducted our audit, not all the internal research publications for fiscal year 2013 were available in all cases, so we focused on internal research published from fiscal years 2008 through fiscal year 2012. [35] OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies (Washington, D.C.: February 2002). 67 Fed. Reg. 8452 (Feb. 22, 2002). Agencies were required to issue their own implementing guidelines by October 1, 2002. For SBA's guidelines, accessed on April 24, 2014, see [hyperlink, http://www.sba.gov/about- ba/sba_performance/open_government/information_quality/information_quali ty_guidelines]. [36] OMB, Final Information Quality Bulletin for Peer Review, December 16, 2004. This Bulletin established government-wide guidance aimed at enhancing the practice of peer review of government science documents. [37] Advocacy's studies examining total costs of regulation to the U.S. economy have been conducted approximately every 5 years, according to Advocacy officials. For this reason, we broadened the scope to fiscal year 2005, whereas in other research work, we focused on fiscal years 2008 through 2012. [38] Similar to our approach in assessing Advocacy's research work, we wanted to focus on the five most recent fiscal years. In the case of the regulatory activities--comment letters and roundtables-- information was available through the end of fiscal year 2013. [39] NVivo is a qualitative data analysis software system that allows organization and analysis of information from a variety of sources including complex nonnumeric or unstructured data. [40] Separate from interviewing the officials selected from the roundtable invitation lists, we interviewed representatives from an association that represents small businesses and a nonprofit whose mission relates to transparency in government. We discussed roundtables and other Advocacy activities with these parties. [41] Human Resources Management in Agencies, HCAAF and HCAAF Systems, Standards, and Metrics, 73 Fed. Reg. 23012 (Apr. 28, 2008). [42] Our practice when rendering opinions is to contact the relevant agency and obtain its legal views on the subject of the request. GAO, Procedures and Practices for Legal Decisions and Opinions, [hyperlink, http://www.gao.gov/products/GAO-06-1064SP] (Washington, D.C.: Sept 2006), available at [hyperlink, http://www.gao.gov/legal/resouces.html]. We contacted the Chief Counsel for Advocacy who, through Advocacy's Director of Interagency Affairs, provided us with the agency's views. Letter from Advocacy to Managing Associate General Counsel, GAO, April 24, 2014 (Advocacy Letter to GAO). [43] Pub. L. No. 94-305, Title II (1976), codified as amended at 15 U.S.C. §§ 634a et seq. [44] Pub. L. No. 111-240 (2010). [45] Testimony of Dr. Winslow Sargeant, Chief Counsel for Advocacy, U.S. Small Business Administration, before the Senate Committee on Small Business and Entrepreneurship (Apr. 17, 2013). [46] Pub. L. No.104-121, title II. [47] Advocacy roundtable invitations and agendas state in part: "Roundtable meetings are open to all interested persons, with the exception of the press, in order to facilitate an open and frank discussion about small business-related issues. Agendas and presentations are available to all, including the press. Anyone who would like to receive roundtables agendas or presentations, or be included in the regular distribution, should forward such requests to [Advocacy]." [48] SBA Office of Advocacy, Report on the Regulatory Flexibility Act, FY 2012 at 4; Report on the Regulatory Flexibility Act, FY 2013, at i. [49] Report on the Regulatory Flexibility Act, FY 2013, supra, at i. [50] Dong involved whether the Smithsonian was an "agency" under the Privacy Act, which in turn adopts the definition of "agency" under FOIA. See 5 U.S.C. § 552a(a)(1). FOIA incorporated the APA "agency" definition until 1974, at which point Congress broadened the core APA "agency" definition for FOIA purposes, but the portion of the Dong court's analysis cited here focuses on whether the Smithsonian was agency under the pre-1974 version of the APA. [51] According to Advocacy: "Advocacy holds numerous roundtable discussions with small business stakeholders to educate them about ongoing legal and regulatory developments across the federal government and discuss their effect on small entities. Roundtable attendance varies from meeting to meeting, and staff encourages wide public participation by encouraging invitees to forward roundtable invitations to interested parties to attend or to call in if in-person attendance is not practicable. On many occasions, officials from federal agencies and Congress attend Advocacy roundtables to provide information regarding agency activities and approaches and to pose questions to small entities and their representatives directly. Roundtables on specific topics also provide a platform for small businesses to provide their views on specific practices, policies and regulations promulgated by various government agencies. Roundtables also serve to inform Advocacy staff of a wide variety of practical and economic challenges small businesses might be facing as a result of federal policies. Advocacy's roundtables are particularly useful for the office because they allow staff to witness and foster dialogue among various stakeholders, including small businesses, their representatives, policymakers, and public interest groups. These conversations often shed light on what are usually complex questions that deserve thorough consideration, on often very short notice, by Advocacy staff in their own evaluation of the impact of regulations on small entities." Advocacy Letter to GAO at 1-2. [52] The AAPS court remanded the case to determine whether a particular working group of President Clinton's Health Care Reform Task Force was a FACA advisory committee under these criteria. [53] As summarized in recent testimony, Advocacy was "created ... to be a voice for small business within the federal government... [and] advances the views, concerns, and interests of small business...." Testimony of Dr. Winslow Sargeant, Chief Counsel for Advocacy, U.S. Small Business Administration, before the House Committee on Small Business, Subcommittee on Investigations, Oversight and Regulations (Mar. 14, 2013). [54] Because we conclude that Advocacy's roundtables are not FACA advisory committees for the reasons discussed here, we did not determine whether the additional FACA element was met of the roundtables being "established or utilized by" Advocacy. 5 U.S.C. App. II, § 3(2). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]