This is the accessible text file for GAO report number GAO-10-947 entitled 'Environmental Protection Agency: EPA Needs to Complete a Strategy for Its Library Network to Meet Users' Needs' which was released on November 1, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: September 2010: Environmental Protection Agency: EPA Needs to Complete a Strategy for Its Library Network to Meet Users' Needs: GAO-10-947: GAO Highlights: Highlights of GAO-10-947, a report to congressional requesters. Why GAO Did This Study: The Environmental Protection Agency’s (EPA) library network provides agency staff and the public with access to environmental information. A 2006 attempt by EPA to reorganize its network by consolidating libraries and making more materials and services available online caused concern among users, and in 2007, EPA put a moratorium on its reorganization plans. Congress requested that GAO report on the reorganization and has again requested a follow-up on these issues. Accordingly, GAO reviewed (1) the status of EPA’s overall strategy for its library network, (2) the status of EPA’s plan to reopen the libraries it closed and other actions planned or taken, (3) EPA’s efforts to digitize printed documents to make them electronically available, and (4) EPA’s efforts to communicate with staff and other stakeholders about its library network. GAO reviewed regulations and agency funding and inventory documents and interviewed EPA staff and contractors, as well as independent library professionals. GAO also assessed the reliability of EPA’s data on library holdings and from EPA’s staff survey on library use and needs. What GAO Found: Although EPA has taken a number of steps to meet the needs of library users, it has not completed a plan identifying an overall strategy for its library network, with implementation goals and a timeline of what it intends to accomplish. Scheduled for completion in 2008, the strategic plan was to provide EPA staff and the public a detailed view of EPA’s library operations and future direction. The draft outline of the strategic plan, however, is largely a placeholder list of current and planned EPA activities. For example, while it emphasizes the central role to be played by electronic library resources, the draft outline does not contain goals or a timeline for completing an inventory of holdings or digitizing those holdings. The draft outline also does not set out details of how funding decisions are to be made. Given the current economic environment, without a completed strategic plan, including a detailed strategy for acquiring, deploying, and managing funding, EPA may find itself hard-pressed to ensure that the network can meet its users’ needs. The agency has reopened libraries closed during reorganization, although about half the network’s 10 regional libraries are operating with reduced hours. EPA has also developed standards for the regional and headquarters libraries’ use of space, on-site collections, staffing, and services. The agency has also hired a national library program manager to carry out day-to-day activities and bring focus and cohesion to the network. Working closely with EPA management and library staff, the national library program manager, who is responsible for library network strategic planning, has set in motion a number of actions meant to improve library network operation and communication, including working closely with internal and external advisory boards and creating a library policy and related procedures. EPA has resumed digitizing some of its libraries’ documents, although it has not inventoried the network’s holdings. The agency is digitizing documents in three phases. Phase 1 was completed in January 2007, phase 2 is scheduled for completion in December 2010, and planning has begun for phase 3. Because EPA has not taken a complete inventory of its library holdings, however, it cannot determine which documents, or how many, will need to be digitized and, consequently, cannot accurately estimate the total cost of digitization or how long it will take. Since we reported on the library network reorganization in 2008, EPA has taken steps to communicate with staff and other stakeholders about its library network, including providing information about the libraries and soliciting information from library users. EPA has also made improvements to the main Internet gateway to the network, making more documents available electronically and providing better access to electronic documents and services. Nevertheless, because EPA’s 2009 survey of the information needs and library use of its staff had methodological flaws—similar to those GAO identified in 2008—the agency is unlikely to obtain accurate information that would enable it to make appropriate decisions on the corrective actions that would best address library users’ needs. What GAO Recommends: GAO recommends, among other actions, that EPA complete its strategic plan for the library network and ensure that survey methods provide reliable data on which to base decisions. With clarifications, EPA concurred with our recommendations. View [hyperlink, http://www.gao.gov/products/GAO-10-947] or key components. For more information, contact John Stephenson at (202) 512- 3841 or stephensonj@gao.gov. [End of section] Contents: Letter: Background: EPA Has Not Completed a Strategic Plan for Its Library Network Identifying an Overall Strategy for the Network: Since 2008, EPA Has Reopened Closed Libraries and Taken Other Actions: EPA Has Resumed Digitizing Unique EPA Documents but Has Not Inventoried Its Holdings: EPA Has Taken Steps to Communicate with Staff and Other Stakeholders about Its Network, but Its Staff Survey Was Flawed: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: Comments from the Environmental Protection Agency: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: EPA Library Operations Before, During, and After Reorganization: Table 2: Library Network Procedures for Implementing the May 2009 Policy: Figure: Figure 1: EPA's Library Network: Abbreviations: EPA: Environmental Protection Agency: NEPIS: National Environmental Publications Internet Site: NSCEP: National Service Center for Environmental Publications: [End of section] United States Government Accountability Office: Washington, DC 20548: September 30, 2010: Congressional Requesters: In 1971, soon after its creation, the Environmental Protection Agency (EPA) established a library network to provide agency staff and the public with access to environmental information. Comprising libraries and repositories at EPA headquarters and in regional EPA offices, research centers, and laboratories across the nation, this network houses information on a broad range of subjects, including science, environmental protection and management, and environmental laws. In 2006, partly in anticipation of fiscal year 2007 budget reductions, EPA began a reorganization effort meant to consolidate libraries in the network and make more materials and services available online. The reorganization plan proposed closing libraries; dispersing, disposing of, or digitizing some collections; cutting operating hours at certain libraries; and altering librarian services. In 2006, EPA closed 5 out of the 26 libraries it operated at the time. As EPA was implementing this reorganization, Congress, professional library associations, labor unions, and other library users raised several concerns. In response to congressional and other concerns, in January 2007 EPA placed a moratorium on its reorganization activities, directing its staff to make no further changes in library services until library policy and procedures were adopted. During the moratorium, 5 libraries remained closed; no additional closures were made; there were no additional reductions in hours of operation, services, or resources; and the disposal of library materials and document digitization stopped. At your request, we reported on the reorganization in 2008.[Footnote 1] In our report, we found that EPA had not followed its own recommended steps to prepare for reorganizing the libraries. Neither did the agency develop procedures to fully inform stakeholders, such as library users and library professionals, of the final configuration of the library network; rather, communication with stakeholders varied from library to library. We also found that EPA lacked an effective strategy or central leadership to ensure the continuity of library services after reorganization, and each library decided on its own whether to close its doors and how to disperse or dispose of its materials. Moreover, EPA did not specifically allot funds to help closing libraries manage their collections, instead leaving each program or regional office to use its annual funding for closure costs. In light of continuing congressional and public concern over EPA's plans for its library network, you asked us to follow up on these issues. Accordingly, this report reviews (1) the status of EPA's overall strategy for its library network, (2) the status of EPA's plan to reopen the libraries it closed and other actions planned or taken, (3) EPA's efforts to digitize printed documents to make them electronically available, and (4) EPA's efforts to communicate with staff and other stakeholders about its library network. To address these objectives, we reviewed relevant EPA funding and inventory documents, policies, plans, and guidance, as well as related regulations and requirements pertinent to the library network and efforts to improve its operations. We focused our review on EPA's Office of Environmental Information's headquarters library in Washington, D.C.; the 10 regional EPA libraries; and the two Office of Administration and Resources Management libraries, in Cincinnati, Ohio, and Research Triangle Park, North Carolina. We compared library operations before, during, and after the reorganization; obtained and reviewed the library network's policy and procedures; reviewed the agency's draft outline of a strategic plan for the library network; obtained and reviewed documents on EPA's digitization process; and reviewed EPA's efforts to communicate with and solicit input from users. We interviewed EPA librarians and library managers in selected EPA libraries, as well as EPA officials knowledgeable about EPA's library network and budget. We also interviewed management officials from the federal employees' union representing EPA staff and spoke with representatives from EPA's regional science councils, which consist of EPA scientists and technical specialists. We further sought information from library professionals, including representatives from the American Library Association and from contractors involved in digitizing EPA documents. After limited testing and discussions with EPA officials, we determined that EPA's data on library funding and on the number of digitized documents and those scheduled to be digitized were not sufficiently reliable for our purposes. Because these data were the only data available, however, we used them to discuss in broad terms EPA's funding history and the status of its digitization efforts, noting the data's limitations as appropriate. Appendix I describes our scope and methodology in greater detail. We conducted this performance audit from October 2009 through September 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: EPA was established in December 1970 to consolidate a variety of federal research, monitoring, standard-setting, and enforcement activities into one agency to ensure the protection, development, and enhancement of the total environment.[Footnote 2] To help accomplish its mission, EPA in 1971 established a library network that came to comprise 26 libraries located across the country. The library network functions as a collection of independent local libraries, catering primarily to the needs of local EPA staff and walk-in public patrons. The libraries are funded and managed by several different regional and program offices at EPA (see figure 1). EPA defines network libraries as those of its libraries with an official membership presence in the global Online Computer Library Center system. Figure 1: EPA's Library Network: [Refer to PDF for image: illustrated U.S.map] Region: 1; Regional library: Boston, Massachusetts; Research laboratory library: Narragansett, Rhode Island. Region: 2; Regional library: New York, New York; Regional library: Philadelphia, Pennsylvania. Region: 3; Library and repository: Washington, DC; Specialty library[A]: Washington, DC; Specialty library[A]: Fort Meade, Maryland. Region: 4; Regional library: Atlanta, Georgia; Library and repository: Research Triangle Park, North Carolina; Research laboratory library: Research Triangle Park, North Carolina; Research laboratory library: Athens, Georgia; Research laboratory library: Gulf Breeze, Florida. Region: 5; Regional library: Chicago, Illinois; Library and repository: Cincinnati, Ohio; Research laboratory library: Ann Arbor, Michigan; Research laboratory library: Duluth, Minnesota. Region: 6; Regional library: Dallas, Texas; Research laboratory library: Ada, Oklahoma. Region: 7; Regional library: Kansas City, Kansas. Region: 8; Regional library: Denver, Colorado; Specialty library[A]: Denver, Colorado. Region: 9; Regional library: San Francisco, California; Research laboratory library: Las Vegas, Nevada. Specialty library[A]: Region: 10; Regional library: Seattle, Washington; Research laboratory library: Corvallis, Oregon. Sources: EPA; Map Resources (map). [A] EPA identifies as specialty libraries its Legislative Reference Library, Office of General Counsel Law Library, and Headquarters and Chemical Library, all of which are in Washington, D.C. The other specialty libraries are its National Enforcement Investigations Center Environmental Forensics Library in Denver, Colorado, and the Environmental Science Center Library at Ft. Meade, Maryland. [End of figure] The combined EPA library network collection contains a wide range of general information on environmental protection and management and on basic and applied sciences, as well as extensive coverage of topics related to the statutory mandates that EPA must meet. Several of the libraries maintain collections focused on special topics to support specific regional or program office projects. The libraries thus differ in function, scope of collections, extent of services, and public access. In addition to its physical locations and holdings, the EPA network provides staff and public access to its collections through the following: * its online library system, a Web-based database of all of EPA library holdings, also known as EPA's online "card catalog"; * interlibrary loans to another library within the network or to other libraries; * a Web site combining two databases--EPA's National Service Center for Environmental Publications (NSCEP) and its National Environmental Publications Internet Site (NEPIS)--which provide an online gateway for access to available print and digital documents, respectively; and: * desktop access for staff to online journals, the Federal Register, news, databases of bibliographic information, and article citations. In addition, librarians are available in each library to catalog and maintain collections and to assist EPA staff and the public with research. In 2003, EPA began studying options for operating its library network in the future. In August 2006, the agency issued a reorganization plan, titled EPA FY 2007 Library Plan: National Framework for the Headquarters and Regional Libraries. The focus of this plan was a reorganization of the headquarters library and the 10 regional libraries, all of which received substantial funding from EPA's Office of Environmental Information. The 2007 library plan identified a new model for library services, which consisted of three components: a coordinated library network, instead of stand-alone operations; more electronic delivery of services; and maintenance of existing essential services. During implementation of this plan, EPA closed the Chicago, Dallas, and Kansas City regional libraries and closed its headquarters library to physical access, although the headquarters library remained as one of three repositories for storing print collections.[Footnote 3] Another library located at EPA headquarters within the Office of Prevention, Pesticides, and Toxic Substances (referred to as the "chemical library") was not subject to budget reductions and was not discussed in the reorganization plan, but, like the headquarters library, was also closed to physical access. EPA also reduced or eliminated the library staff at the closed libraries. Several other libraries reduced their operating hours, and some libraries disposed of their materials or dispersed them to other EPA libraries or to non- EPA libraries. EPA also began to digitize EPA documents not currently in NEPIS, beginning with documents in the libraries being closed. EPA's reorganization plan also discussed how the closed libraries were to handle their collections, directing them primarily to disperse the collections to other libraries. EPA's implementation of its reorganization plan caused widespread concern among its staff, the public, interested parties, and Congress. In response to these concerns, congressional committees directed $1 million in funding to restore the libraries recently closed or consolidated, asked us to review EPA's reorganization plan and its implementation, and directed EPA to prepare a report regarding actions to restore publicly available libraries. In addition, EPA in January 2007 imposed a moratorium on its reorganization efforts. Until April 2007, EPA's library network operations had been guided by EPA's Information Resources Management Policy Manual, which stated that the library network was to provide EPA staff with access to information to carry out the agency's mission and that the libraries were to provide state agencies and the public with access to the library collection. The Policy Manual also defined the role of a national library program manager, who was to have responsibility for coordinating major activities of the library network, although not budget authority for the libraries. EPA replaced this manual in April 2007 with an interim library policy and, in May 2009, with its final library policy. The final May 2009 policy also defined key roles and responsibilities, including those of the national library program manager and those of "federal library managers," who were to have first-line responsibility for operating the physical libraries and providing services. EPA lifted the moratorium in June 2009, following implementation of its May 2009 policy and 6 of 12 proposed procedures for the library network. After we issued our report in February 2008,[Footnote 4] Congress held hearings on EPA's library network reorganization efforts, which were followed by the release of EPA's March 2008 report addressed to Congress, in which EPA stated that it would reopen the closed libraries by September 30, 2008.[Footnote 5] In our 2008 report on EPA's library network reorganization, we assessed the reorganization effort against our past work on key practices and implementation steps to assist mergers and organizational transformations.[Footnote 6] These key practices include ensuring that top leadership drives the transformation, establishing a coherent mission and integrated strategic goals to guide the transformation, and setting implementation goals and a timeline to show progress from day one. EPA Has Not Completed a Strategic Plan for Its Library Network Identifying an Overall Strategy for the Network: Although it has been preparing a strategic plan for its library network for 3 years, EPA has not completed a plan identifying its overall network strategy, with implementation goals and a timeline for what it seeks to accomplish. In our 2008 report, we stated that EPA was developing a library strategic plan for 2008 and beyond, which was to detail library services for staff and the public and lay out a vision for the library network's future. EPA has had a draft outline of this strategic plan since July 2007. We reported that in October 2007, EPA's Office of Environmental Information asked local unions throughout the agency to comment on a draft of the plan. The draft outline of the strategic plan envisions the library network as "the premier environmental library network that provides timely access to information and library services to its employees and the public" and proposes to realize this vision by increasing emphasis on electronic resources and using new information technologies.[Footnote 7] The draft outline of the strategic plan also lists several principles as the foundation for present and future directions of the library network: setting overall goals and objectives and a direction for implementation; periodic review of the plan to evaluate progress and update strategies to respond to new opportunities or challenges; soliciting input from internal and external stakeholders; and developing the plan in a transparent manner by reporting progress and soliciting input from interested parties. According to EPA officials, since 2007, EPA has been in the process of assessing library users' needs, which is to be completed before they believe they can finish the strategic plan. EPA officials have stated that a working group led by the national library program manager is to resume work on the plan later in 2010. The draft outline of the strategic plan is largely a list of current and planned EPA activities--primarily placeholders to be completed. For example, under the heading, "Digitization," the text states that the digitization procedures will outline the methods to be used by EPA libraries to prepare and send EPA documents for digitization and inclusion in NEPIS; no goals or timeline for implementing these activities--which we previously reported are among the key components of successful organizational transformations--are given.[Footnote 8] We have found that an organization's transformation is strengthened when it publicizes implementation goals and a timeline to build momentum and show progress. Despite an emphasis on the central role to be played by electronic library resources, the draft outline of the strategic plan briefly describes procedures for packing and shipping documents to be digitized, without describing actions to be taken to digitize holdings or target dates for accomplishing those actions. EPA holds an enormous amount of environmental information, including publications generated by its program offices, as well as research publications generated under contracts, grants, and cooperative agreements. A large portion of this information exists only in print form. As part of its vision for the library network, according to the draft outline, EPA is seeking to convert this information into a digital format to make it more widely available and readily accessible to users. Yet the draft outline of the strategic plan does not describe criteria for deciding what documents should be digitized, for deciding whether or how to digitize copyrighted documents of value, or for scheduling the funding needed and a time frame for completing the digitization. Without such criteria, EPA cannot ensure that it is digitizing the most valuable or important documents or providing users with information most relevant to them. Furthermore, although the draft outline of the strategic plan includes a placeholder for a section describing a funding model for the network, it contains no detail. Under the heading, "Funding Model," the text states that this section in the plan will address how EPA will ensure that the network libraries have "adequate funding" and will discuss how funding decisions are made, along with the Office of Environmental Information's role in the funding process. But the draft outline of the strategic plan does not define what constitutes adequate funding, although inadequate funding has been a concern for the library network since fiscal year 2007. Until then, library spending had remained relatively stable, ranging from a high of $9.2 million in fiscal year 2002 to a low of $8.2 million in fiscal year 2006.[Footnote 9] In fiscal year 2007, when EPA's budget was reduced, library spending was $6.3 million. The draft outline of the strategic plan also does not set out the details of how funding decisions are made. Setting out details for how such decisions are to be made, to ensure that they are informed and transparent, is especially important because of the decentralized nature of the library network. The library network's funding remains subject to uncertainty in the future because the several different program and regional offices responsible for EPA's libraries generally decide how much to spend on their libraries out of funding available in larger accounts that support multiple activities. EPA's Office of Environmental Information, the primary source of funding for the regional libraries,[Footnote 10] typically provides funding through each region's support budget and generally gives regional management officials discretion on how to distribute this funding among the library and other support services, such as information technology, utilities, and mail room support. EPA officials told us that, starting in fiscal year 2010, they are increasing the amount of funding allocated to the libraries in the regions. The regions also obtain a much smaller portion of their library funding from other program offices. For example, the Superfund program office funds the storage and maintenance of information on the National Priorities List, EPA's list of the most seriously contaminated sites in the United States. The extent to which other program offices provide funding to the regional libraries varies. Thus, without a detailed strategy for how decisions are made to acquire, deploy, and manage funding resources across the library network, EPA may find it difficult, particularly in an era of declining budgets and competing national priorities, to achieve its vision for the library network and to fully meet the needs of library users. Moreover, although the draft outline of the strategic plan contains a section on communication among network libraries, it makes no mention of a strategy or a formal outreach plan to ensure that EPA communicates with and obtains feedback from users about improvements to its library network in a way that would allow it to measure whether such improvements are meeting users' needs. The section lists communication methods EPA is using, such as monthly network teleconferences among staff and federal managers. In addition, the section identifies comment cards, questionnaires, and a Web presence for how it solicits users' feedback, but there is no mention of how EPA plans to assess feedback on what is important to users or what improvements are working well or poorly. EPA has another draft document, titled "EPA Library Network Communication Strategies," whose purpose is to establish procedures by which libraries in the network are to communicate both internally and externally. Most of this document focuses on communication within the library network itself, explaining how the library network is coordinated and detailing mechanisms for internal communication, including annual meetings for library network staff and federal library managers. One of the final sections in this procedures document lists several means of communicating externally, including Web sites and various local options for libraries to reach out to local patrons, such as tours, signs, comment cards, and online feedback mechanisms. Beyond listing such mechanisms, however, this document, like the draft outline of the strategic plan, does not lay out a systematic communication strategy with communication and feedback performance goals that can be measured to determine progress. Without such a strategy, communication with library users is likely to remain piecemeal and reactive. Since 2008, EPA Has Reopened Closed Libraries and Taken Other Actions: EPA has reopened all of the libraries it closed in 2007 and has taken other actions to improve library operations. In its 2008 report addressed to Congress,[Footnote 11] the agency stated its commitment to have libraries in each region and at headquarters open to the public. EPA also committed to reestablishing on-site libraries for its staff and the public in the three regions where the libraries had been closed and at the combined headquarters and chemical library. EPA reopened all five closed libraries by September 2008, although the agency had to find new space for two of the three closed regional libraries and their collections, and all three of these regional libraries are operating on reduced schedules. Each of the reopened libraries was staffed with a professional librarian and required to maintain a collection of core reference materials and additional library resources to meet local needs and to ensure that staff had access to core library services and the public had access to the library and its collections. With the reopening of the closed libraries, one other regional library that had not been closed also began operating on reduced schedules, as compared with its hours before reorganization (see table 1). As of September 2010, about half of the 10 regional libraries were operating 4 days a week, rather than 5 days as they were before EPA's reorganization efforts--a reduction in hours due largely to funding constraints, according to library officials. All of the libraries are staffed with at least one full- time or part-time librarian, with several libraries having more than one librarian or additional library staff. Table 1: EPA Library Operations Before, During, and After Reorganization: Program office: Office of Environmental Information; Library, location: Headquarters library, Washington, D.C.[A]; Operating status of library: Before reorganization: Open 5 days, 40 hours; During reorganization: Closed to physical access; After reorganization: Open 5 days, 40 hours. Program office: Office of Prevention, Pesticides, and Toxic Substances; Library, location: Chemical library, Washington, D.C.; Operating status of library: Before reorganization: Open 5 days, 35 hours; During reorganization: Closed to physical access; After reorganization: Combined with headquarters library. Program office: Regional office; Library, location: Region 1 library, Boston, Massachusetts; Operating status of library: Before reorganization: Open 5 days, 35 hours; During reorganization: Reduced hours of operation 3 days, 18 hours; After reorganization: Open 4 days, 24 hours. Program office: Regional office; Library, location: Region 2 library, New York, New York; Operating status of library: Before reorganization: Open 4 days, 28 hours; During reorganization: Reduced hours of operation 3 days, 12 hours; After reorganization: Open 4 days, 28 hours. Program office: Regional office; Library, location: Region 3 library, Philadelphia, Pennsylvania; Operating status of library: Before reorganization: Open 5 days, 45 hours; During reorganization: Open 5 days, 45 hours; After reorganization: Open 5 days, 45 hours. Program office: Regional office; Library, location: Region 4 library, Atlanta, Georgia; Operating status of library: Before reorganization: Open 5 days, 42.5 hours; During reorganization: Open Some services provided by the library in Cincinnati, Ohio 5 days, 42.5 hours; After reorganization: Open 5 days, 42.5 hours. Program office: Regional office; Library, location: Region 5 library, Chicago, Illinois; Operating status of library: Before reorganization: Open 5 days, 25 hours; During reorganization: Closed to physical access; After reorganization: Open 4 days, 24 hours. Program office: Regional office; Library, location: Region 6 library, Dallas, Texas; Operating status of library: Before reorganization: Open 5 days, 40 hours; During reorganization: Closed to physical access; After reorganization: Open 5 days, 30 hours. Program office: Regional office; Library, location: Region 7 library, Kansas City, Kansas; Operating status of library: Before reorganization: Open 5 days, 40 hours; During reorganization: Closed to physical access; After reorganization: Open 4 days, 24 hours. Program office: Regional office; Library, location: Region 8 library, Denver, Colorado; Operating status of library: Before reorganization: Open 5 days, 40 hours; During reorganization: Open 5 days, 40 hours; After reorganization: Open 5 days, 40 hours. Program office: Regional office; Library, location: Region 9 library, San Francisco, California; Operating status of library: Before reorganization: Open 5 days, 24 hours; During reorganization: Reduced hours of operation 4 days, 24 hours; After reorganization: Open 4 days, 24 hours. Program office: Regional office; Library, location: Library, location: Region 10 library, Seattle, Washington; Operating status of library: Before reorganization: Open 5 days, 40 hours; During reorganization: Reduced hours of operation 5 days, 25.5 hours; After reorganization: Open 5 days, 40 hours. Source: GAO analysis of EPA data. Note: This table shows only those libraries whose operations were affected or potentially affected by reorganization. Days and hours of operation shown in the table are for both the public and EPA staff; the regional libraries in Atlanta, Denver, and Seattle are open 4 to 10 hours per week longer for EPA staff than for the public. While the libraries in Regions 5, 6, and 7 were closed, services for these libraries were provided by another network library. [A] Library also serves as a repository. [End of table] In addition to reopening the closed libraries, according to EPA officials we spoke with, EPA developed standards for the regional and headquarters libraries' use of space, on-site collections, staffing, and services. These standards specify, for example, that the libraries make adequate space available for in-person interactions between library staff and users, that on-site collections and materials should address local and regional needs, that staff and the public have certain minimum hours of access per week (at least 4 days per week on a walk-in or appointment basis in the regional libraries and at least 5 days per week on a walk-in or appointment basis in the headquarters library), and that the libraries provide interlibrary loans and reference or research assistance. One of the key actions taken by EPA in May 2007 was to hire a national library program manager, a position that had been vacant since 2005. [Footnote 12] Housed in EPA's Office of Information Analysis and Access, within the Office of Environmental Information, the national library program manager is charged with carrying out day-to-day activities of the library network and with bringing focus and cohesion to the network. Part of this charge involves agencywide responsibility for public information access, including strategic planning for the library network, and participating in policy formulation regarding access to EPA's public information. To fulfill this leadership role, EPA officials said, the national library program manager is to work closely with the management of EPA's Office of Environmental Information to set in motion a number of actions meant to improve library network operation and communication. To communicate with and gather feedback internally, the national library program manager initiated monthly teleconferences and annual meetings for all library managers and staff. Seeking to get the most out of the experience and knowledge of these library managers, librarians, and staff, the national library program manager established internal working groups to research improvement activities, address concerns, and recommend improvements. For example, the national library program manager established working groups on digitization, staff information needs, and development of the final May 2009 library policy and related procedures (see table 2). In addition, the national library program manager serves as the EPA-appointed representative in working with outside library professionals, specifically an external board of advisors created by the Federal Library and Information Center Committee, which advises EPA on future directions for the library network.[Footnote 13] Table 2: Library Network Procedures for Implementing the May 2009 Policy: Procedure: EPA library materials dispersal; Purpose: Establish agencywide procedures for libraries to retain, reduce, disperse, or dispose of their library contents when appropriate; Status, approval date: Approved May 15, 2009. Procedure: EPA library usage statistics; Purpose: Establish agencywide procedures by which the libraries collect statistics on the services they provide EPA staff and the public; Status, approval date: Approved May 15, 2009. Procedure: Digitization processes for EPA libraries; Purpose: Establish agencywide procedures to identify, prepare, and send EPA documents to NSCEP for digitization so they may be added to the NEPIS database (formalizes existing steps for digitizing documents); Status, approval date: Approved May 15, 2009. Procedure: EPA library reference and research services; Purpose: Establish agencywide procedures to provide reference and research services to EPA staff and the public; Status, approval date: Approved May 15, 2009. Procedure: EPA repository library management; Purpose: Establish agencywide procedures to ensure that official EPA documents and other materials that are hard to replace but infrequently used are available in perpetuity in their originally published format; Status, approval date: Approved May 15, 2009. Procedure: EPA library facility management; Purpose: Establish agencywide procedures to provide, make use of, and manage the facility allotted for library operations; Status, approval date: Approved May 15, 2009. Procedure: Interlibrary loan service; Purpose: Establish agencywide procedures to borrow or acquire materials not available in the local collections for EPA staff and the public; Status, approval date: In draft, due to be approved before the end of 2010. Procedure: Cataloging; Purpose: Establish agencywide procedures for libraries to catalog materials in the collections and provide access to remote electronic documents; Status, approval date: In draft, due to be approved before the end of 2010. Procedure: Public access; Purpose: Establish agencywide procedures to improve public access to EPA documents and environmental information; Status, approval date: In draft, due to be approved before the end of 2010. Procedure: Collection development; Purpose: Establish agencywide standard methods for libraries to acquire, organize, and manage materials in their local collections; Status, approval date: In draft, due to be approved before the end of 2010. Procedure: Disaster response and emergency; Purpose: Establish agencywide procedures by which the libraries prepare for and respond to disasters and provide continuing operations during and after a disaster; Status, approval date: In draft, due to be approved before the end of 2010. Procedure: Library network communications strategies; Purpose: Establish agencywide procedures by which the libraries in the network communicate, using a range of established mechanisms, with other EPA libraries, organizations, and the public; Status, approval date: In draft, due to be approved before the end of 2010. Source: EPA. [End of table] EPA Has Resumed Digitizing Unique EPA Documents but Has Not Inventoried Its Holdings: EPA has restarted its process of digitizing some of its libraries' holdings, but because the agency has not completed an inventory of its holdings, it does not know the total number of documents to be digitized. According to EPA data, which our limited testing found to be insufficiently reliable, EPA had digitized 16,175 documents from its libraries as of January 2010. Creating an electronic copy of a document by means of digitization is relatively simple--essentially the same scanning process for making photocopies--although it can be time-consuming and expensive if the document contains special features such as foldout pages, cannot be taken apart, or needs to be digitized at a high level of resolution or in color. After the resulting electronic files are uploaded to EPA's Web databases, the administrator of EPA's online library catalog is to ensure that links to the digitized documents are included in the bibliographic records for each document. According to EPA officials, the present digitization effort will expand NEPIS, EPA's sole electronic archive of published material, which, according to EPA officials, contains 40,000 publicly accessible digital documents as of June 2010, up from 4,000 in 1996. According to EPA documents, the digitization process is to take place in three phases: * Phase 1 covered unique EPA documents held by the libraries that were closed under the reorganization plan.[Footnote 14] EPA data show that 15,260 documents were digitized during this phase, which was completed in January 2007. * Phase 2 is to cover all remaining unique EPA documents except those larger than 11 by 17 inches. According to EPA officials, this phase is scheduled for completion in December 2010 and should produce 10,102 additional digitized documents, bringing the total number of digitized library documents available to the public to over 25,000. * Phase 3 is to include EPA documents of which more than one copy exists in the library network, plus unique EPA documents larger than 11 by 17 inches. As of July 2010, EPA was beginning to plan this phase. As of September 2010, the total estimated cost for digitizing EPA's library holdings remained unclear, in large part because EPA has not completed an inventory of its holdings and has therefore not determined the total number of documents that need to be digitized. When it began digitizing documents from the closed libraries in 2006 under phase 1, EPA estimated that the project would cost $80,000-- $78,000 for scanning and $2,000 for uploading the digital files to the Web databases--although, according to agency officials, the agency did not track the actual costs. For phase 2 digitization, which began in fiscal year 2009, EPA estimated the cost at about $327,000. EPA has not estimated the cost or a completion date for its final, phase 3 digitization effort, in part because the agency is still cataloging all its library holdings in a single database so it can inventory all the documents that need to be digitized. One regional librarian we spoke with, for example, told us that about 2,000 documents in the regional library's catalog were not in EPA's online library system, and it is still unknown which or how many of these documents will need to be digitized. Without a complete catalog or inventory of its holdings, EPA cannot determine which documents, or how many, will need to be digitized and, consequently, cannot accurately estimate the total cost of digitization or how long it will take. According to the national library program manager, an EPA workgroup is currently drafting a new cataloging procedure for the libraries and expects the procedure to be approved and implemented before the end of 2010. This procedure requires each network library to inventory its collection on a regular basis, either the entire collection every 3 years or one- third of the collection each year. In addition, EPA library officials observed that a significant number of the documents in EPA libraries are copyrighted, and to date EPA does not plan to digitize them. EPA, like other federal agencies, often contracts with entities in the private sector to do work. In addition, EPA provides financial assistance in the form of grants and cooperative agreements to various recipients, such as state, local, and tribal governments; educational institutions; hospitals; and nonprofit organizations. Such assistance is documented in an assistance agreement. Both contracts and assistance agreements may result in the production of copyrighted documents. In the case of contracts, federal regulations provide that when a contractor is permitted to assert a copyright in any document(s) produced, the government has a license to display the copyrighted work publicly, which includes posting it on a Web site accessible to the public. [Footnote 15] In the case of works produced under assistance agreements, on the other hand, the government has a right to reproduce, publish, or otherwise use a copyrighted work for federal purposes, but EPA's Office of General Counsel has determined that inclusion in EPA's online public library would not constitute a federal government purpose.[Footnote 16] On the advice of EPA's general counsel, EPA's digitization workgroup has recommended digitizing documents created under contract but not those created under EPA's assistance agreements.[Footnote 17] According to EPA's grant awards database, these agreements have resulted in more than 21,000 grants valued at over $40 billion in taxpayer dollars. Some of these grants led to publications, resulting in a substantial body of publicly funded written material. According to EPA's general counsel, EPA may digitize such documents so that staff and others may use them for federal government purposes but may not disseminate them for other purposes. EPA may also seek permission from copyright holders to digitize and disseminate online copyrighted documents produced under assistance agreements, although some costs may be associated with obtaining such permissions--tracking down copyright holders after years, or even decades, have passed, for example--further complicating any estimation of total digitization costs. An alternative practice has been in use by the Federal Library and Information Network,[Footnote 18] the business arm of the Federal Library and Information Center Committee: permission to use copyrighted material produced under assistance agreements is sought at the time an agreement is established. If the prospective copyright holder grants permission, then a statement to that effect is incorporated into the assistance agreement, incurring minimal, if any, additional costs. Without permission from copyright holders, however, documents prepared under EPA assistance agreements, using taxpayer dollars, will remain unavailable online to the public. EPA Has Taken Steps to Communicate with Staff and Other Stakeholders about Its Network, but Its Staff Survey Was Flawed: EPA has taken steps to communicate with staff and other stakeholders about its library network--including providing information about the libraries as well as soliciting information from library users--but a 2009 survey about its staff's information needs was flawed. In general, EPA staff and external stakeholders told us the agency is doing a better job of communicating with them and soliciting input on the operations and future direction of the library network, particularly at the local level. Representatives from EPA's employees' unions and regional science councils stated that communication about the library and its services--such as new resources, training, and open houses--is primarily done at the regional level, either through e- mail or the region's intranet page. Although staff have not been directly solicited for feedback, according to the representatives, no outstanding issues regarding the libraries have been raised, except that a few representatives said they would like to see the libraries open 5 days and 40 hours per week. To keep library managers and staff engaged in improving library operations, EPA has adopted a number of techniques to communicate with them and solicit their input. These techniques have allowed EPA to gather staff input for policies and procedures, operational issues, and Web page improvements. Examples include the following: * The national library program manager holds monthly network teleconferences with library managers and staff on matters of interest to the entire network or on operational topics, such as the library policy and procedures. * The national library program manager also holds ad hoc teleconferences with library managers elsewhere in the network to discuss their libraries' needs. * Managers and staff use mailing lists to communicate with one another about daily library operations or requests for assistance. * For the last 3 years, EPA has held an annual network meeting in different locations for library managers and staff to foster collaboration, provide training, and share information about the network. At the last meeting, in October 2009, participants discussed ways to address results of the 2009 staff survey, prepared for the next round of digitization, and discussed ways to improve library services. The next annual meeting is scheduled for March 2011. * EPA and the union representing EPA staff agreed to create a union- management advisory board with six members--three union representatives and three from EPA management.[Footnote 19] The board reviews and makes recommendations on library network policy and procedures and will review the library network strategic plan once it is completed. * In December 2009, EPA instituted a pilot program, an "ask a librarian" live chat. Ten libraries are participating in the program, which lets users contact a librarian through an electronic link to request services. As of July 2010, the "ask a librarian" pilot was available only to EPA staff. To begin to realize its vision of effectively implementing new information technologies and making documents readily available electronically, EPA in 2007 engaged a contractor to review the user- friendliness of the combined Web page, or gateway, to the NSCEP and NEPIS online databases.[Footnote 20] The review identified ways to improve the site's effectiveness and overall functionality for users. EPA implemented many of these improvements.[Footnote 21] For example, the gateway home page now describes the purpose of each database (NSCEP for print materials and NEPIS for electronic documents) and what types of publications they contain, noting that they contain only EPA publications. EPA made several changes to the document display page as well, such as placing a navigation bar at the top and bottom of the document with large icons and providing a button that allows users to obtain a copy in one of three formats. The display page allows users to put a copy of frequently used documents in a holding area for later retrieval. Work is also under way to integrate Google search capabilities into this gateway, as well as the ability to refine the precision of searches with user-friendly "clouds" of related keywords. In addition, EPA has added easy ways for users to offer feedback, which EPA may then incorporate to make improvements. For example, the navigation bar on the document display page now includes a "report an error" button, and every page has a "contact us" link. Furthermore, the left frame of the site contains a link to a customer satisfaction survey, and the site also has a separate page for user feedback. EPA has also made outreach efforts to library professionals outside EPA--primarily by presenting and exhibiting at professional library association trade shows and conferences, attending external training, visiting other federal national libraries, and interacting with its external board of advisors. Ties with the external board have been a particularly important part of EPA's response to concerns over the agency's library reorganization. From June 2007 through February 2010, the national library program manager met with the board approximately 20 times, working with it on the full range of key issues, from policy development to funding models to communication with stakeholders. The external board also advised EPA on the development of a survey to assess the information needs of EPA staff. One of the principles in the draft outline of the strategic plan is soliciting feedback from internal and external stakeholders about their information needs. To solicit such feedback from staff, EPA in early 2008, under the direction of the national library program manager, engaged a contractor to conduct interviews, hold focus groups, and conduct a Web-based survey. The survey was made available for approximately 1 month in 2009 via a secure Web site only to EPA staff (about 17,000 individuals), not all of whom were library users. After the survey was completed, the contractor conducted a series of focus groups and one-on-one interviews with EPA managers, focusing on significant issues identified in the Web-based survey; according to EPA officials, these in-person discussions were to help ensure that a comprehensive perspective of user needs was captured. EPA officials stated that the Web-based survey results corroborated what the agency learned in an earlier survey, done in 2004 to 2005. On the basis of the survey results, focus group discussions, and management interviews, the contractor developed recommendations for EPA's consideration. EPA received the results of the survey and discussions, along with the contractor's recommendations, in August 2009 and has assigned working groups of library staff to review the findings and suggest how EPA could address them. We found, however, that this survey had flaws, similar to those we identified in the 2004 to 2005 survey and discussed in our 2008 report, which greatly reduce its usefulness.[Footnote 22] First, in both the earlier and the 2009 surveys, the response rate was 14 percent, far lower than the 80 percent response rate that Office of Management and Budget guidance recommends for a survey to increase the likelihood that it adequately represents a universe of respondents. [Footnote 23] Neither EPA nor the contractor for the 2009 survey analyzed the results for the nonresponse bias that may occur at response rates lower than 80 percent, particularly if the group of respondents differs significantly in relevant ways from nonrespondents.[Footnote 24] Thus, EPA cannot be assured that either survey accurately described staff needs for information or their uses of the library. Second, respondents to both the 2009 and the earlier survey were self-selected, a methodology that often leads to biased samples, since the traits that affect a person's decision to participate in the survey--such as strong opinions or substantial knowledge--may be correlated with traits that affect survey responses; the result is an unrepresentative sample of possible respondents. The risk of potential bias through self-selection is increased by the fact that neither EPA nor the contractor for the 2009 survey instituted any safeguards to prevent respondents from submitting more than one survey each. Thus, there is no assurance that the survey results are unbiased and reflect a broad range of EPA staff perspectives and experiences. Third, in neither survey did EPA gather views from or determine the needs of other significant users of EPA libraries, such as state and local environmental organizations or the public at large.[Footnote 25] Although EPA officials told us that EPA is planning to assess the needs of public patrons, an assessment that does not correct the methodological weaknesses we found in EPA's two previous surveys of its staff is unlikely to produce results that accurately reflect the needs of public patrons. Conclusions: In the 4 years since EPA issued a reorganization plan for its library network, the agency has taken a number of steps to better communicate with, and meet the needs of, library users. EPA's lack of a completed strategic plan identifying its overall strategy for the network, however, leaves unclear how the agency will translate into reality its vision of a "premier environmental library" with an "emphasis on electronic resources." Steps the agency has taken, including hiring a national library program manager and establishing a uniform policy and some procedures for the libraries, have led to some improvements in library services and will undoubtedly enhance network cohesion. But without a completed strategic plan that contains implementation goals and timelines, neither EPA nor users of its libraries can have a clear view of what EPA plans to do, when EPA plans to do it, and whether EPA's actions will ultimately meet users' needs. In particular, without a strategy for acquiring, deploying, and efficiently allocating library funding, the library network could have difficulty maintaining high-quality service in the digital age. Moreover, EPA's approach to digitizing copyrighted works in the future--as well as the fact that the agency has not yet inventoried all library holdings-- could, if not revisited, detract significantly from the utility of the library network. Specifically, unless EPA revisits its decision not to digitize documents prepared with taxpayer dollars under assistance agreements, it may be missing opportunities to make these documents more readily available to users, including other federal users, who need them to better carry out their work. Finally, improvements to the library network's Internet gateway offer new means of seeking feedback from library patrons about their use of and need for library services. Nevertheless, EPA does not have a valid method for assessing those library users' needs. If, in future assessments of users' needs, EPA fails to correct the flawed methods of its previous staff surveys, the agency is unlikely to obtain accurate information that would enable it to make appropriate decisions on the corrective actions that would best address those needs. Recommendations for Executive Action: To ensure that EPA's library network continues to meet its users' needs, we recommend that the Administrator of EPA take the following six actions: * Complete EPA's strategic plan for the library network, including implementation goals and timelines. In so doing, EPA should outline details for how funding decisions are to be made, to ensure that they are informed and transparent. * Complete an inventory of the library network's holdings to identify what items remain to be digitized. * For assistance agreements already in place, EPA should digitize documents produced under the agreements and make them available to federal employees and other authorized users for federal government purposes. * In future assistance agreements, make explicit that EPA can include in the agency's public online database, without obtaining prior permission from the copyright holder, any documents produced under the agreements. * For future assistance agreements where EPA cannot make such an arrangement, EPA should digitize documents produced under the agreements and make them available to federal employees and other authorized users for federal government purposes. * Ensure that the data analysis protocols used for conducting surveys of users' needs--including sampling procedures and response rates--are sufficiently sound methodologically to provide reliable information on which to base decisions and allocate resources efficiently. Agency Comments and Our Evaluation: We provided EPA with a draft of this report for review and comment. With clarifications, the agency concurred with our recommendations. EPA acknowledged that the planning document available on the agency's Web site--which our report refers to as the draft outline of the strategic plan--has provided more of a working agenda than a strategic plan to guide the rebuilding of the library program. In responding to our recommendations, EPA wrote that it believes it now has enough information to develop an effective strategic plan for the library network and that it is time to complete and publish a formal plan identifying an overall network strategy, with implementation goals and a timeline for future accomplishments. The agency stated that it is moving forward with the strategic plan, which it aims to complete in fiscal year 2011. In addition, EPA said it will address the inventory of library holdings, completing a schedule for cataloging the inventory by November 1, 2010, and striving to complete the cataloging by September 30, 2011. The agency further agreed to take the necessary steps to ensure that any future assessments of users' needs employ methodologically sound data protocols and provide reliable information. Regarding our recommendations on the digitization of copyrighted documents produced under assistance agreements, EPA said it would address the feasibility and legality of digitizing products resulting from such agreements. For future assistance agreements, the agency said it will develop options for gaining advance permission to digitize products from these agreements and take these options to senior agency managers by mid-2011 for consideration and action. For existing assistance agreements, however, EPA wrote that, because of legal and technical constraints, it does not plan to digitize products produced under existing agreements. In further clarifying the agency's written comments, EPA officials told us that because the documents produced under existing assistance agreements are copyrighted, the agency cannot include them in its public online database. In the agency's view, EPA would therefore need to develop a forum for disseminating the documents to EPA staff and determine whether other federal employees needed access to the documents for federal government purposes. EPA officials also said that digitizing these documents was constrained by several factors, including agency priorities for which documents are to take precedence and efforts to identify which of the many types of assistance programs are likely to produce documents of most value to EPA staff. We have clarified the wording of our recommendations to eliminate any implied reference that EPA should make the copyrighted documents available to the general public in its online database, and we maintain that making copyrighted documents resulting from assistance agreements available solely for federal government purposes is permissible, feasible, and desirable. EPA's written comments appear in appendix II. EPA also provided technical corrections, which we incorporated. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, Administrator of EPA, and other interested parties. in addition, this report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff members have any questions on this report, please contact me at (202) 512-3841 or stephensonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Signed by: John B. Stephenson: Director, Natural Resources and Environment: List of Requesters: The Honorable Barbara Boxer: Chairman: Committee on Environment and Public Works: United States Senate: The Honorable Bart Gordon: Chairman: Committee on Science and Technology: House of Representatives: The Honorable Henry A. Waxman: Chairman: The Honorable John Dingell: Chairman Emeritus: Committee on Energy and Commerce: House of Representatives: [End of section] Appendix I: Scope and Methodology: To complete our work, we reviewed relevant Environmental Protection Agency (EPA) funding and inventory documents, policies, plans, guidance, and procedures, as well as related regulations and requirements pertinent to the library network and efforts to improve its operations. We limited our review to the 26 libraries belonging to EPA's library network, that is, libraries that are members of the Online Computer Library Center system. We focused on EPA's headquarters library, the 10 regional libraries funded in part by EPA's Office of Environmental Information, and the Office of Administration and Resources Management libraries in Cincinnati, Ohio (which is responsible for EPA's digitization and Web site maintenance), and Research Triangle Park, North Carolina. We compared library operations before, during, and after attempted reorganization in fiscal year 2007; obtained and reviewed library network policy and procedures; reviewed the agency's draft outline of a strategic plan for the library network; obtained and reviewed documents on EPA's digitization process; and reviewed EPA's efforts to communicate with and solicit input from users. We interviewed EPA librarians and library managers in selected EPA libraries, as well as Office of Environmental Information officials knowledgeable about EPA's library network and budget; when possible, we corroborated information provided to us during interviews with relevant documentation. We also interviewed management officials from the federal employees' union representing EPA staff and spoke with representatives from EPA's regional science councils, which consist of EPA scientists and technical specialists. We further sought information from library professionals, including representatives from the Library of Congress; the National Agriculture Library; and, through visits and interviews, from Lockheed Martin and Integrated Solutions and Services, contractors involved in digitizing EPA documents. In addition, we obtained information on library funding from each of the 26 libraries in the network from fiscal year 2002 to fiscal year 2010. Because EPA does not specifically track funding for the libraries, the information provided contained a mix of outlays for some fiscal years and budget authority for other fiscal years. In addition, the information provided by each of the libraries reflected only spending by the library and not funding sources. For example, a large portion of funding for regional office libraries comes from the Office of Environmental Information, but these libraries also receive funding from other EPA program offices, such as Superfund. Also, funding data from the libraries contained a mix of funding for contract support; library staff salaries; and acquisition costs for books, journals, and other materials. We interviewed EPA budget officials to assess data reliability and performed a limited test to verify the accuracy and completeness of the data provided by the libraries. On the basis of this test and discussions with EPA officials, we concluded that the data were not reliable enough to include in our report. We also obtained data on the number of EPA and other documents that have already been digitized and the number still to be digitized. After limited testing and discussions with EPA officials, we determined that EPA's data on library funding and on the number of digitized documents and those scheduled to be digitized were not sufficiently reliable for our purposes. Because these data were the only data available, however, we used them to some extent, noting their limitations in our report as appropriate. We also reviewed documents about EPA's digitization process, guidance on what documents should or should not be digitized, and digitization contracts, and we discussed the contents of these documents with EPA and digitization- contractor officials. We also discussed EPA's future digitization plans with Office of Environmental Information officials. In addition, we assessed EPA's survey of library users, examining the adequacy of the survey methodology, including response rate, sampling methodology, security measures, survey questions, and processes. To determine the adequacy of the response rate to EPA's survey, we followed an 80 percent response rate as a criterion, as Office of Management and Budget guidance recommends and we apply in our own surveys to increase the likelihood of sufficiently representing a universe of respondents. For surveys with response rates lower than 80 percent, we also perform an analysis to determine the existence of nonresponse bias. To generate its survey sample, however, EPA relied on self-selection, using a Web site to make the survey available to approximately 17,000 EPA staff; the response rate achieved was 14 percent. We performed a limited nonresponse analysis of EPA's survey data and determined that some staffing categories were represented in proportions different from those found in the population of EPA staff. Given the 14 percent response rate to EPA's survey, the nonrandom methodology that generated the sample, and the results of our limited analysis for nonresponse bias, we found EPA's survey results to be inadequate for EPA's purpose of obtaining a representative view of EPA library users. We also interviewed local union representatives from headquarters and some of EPA's regional offices. Furthermore, we interviewed regional science council representatives from some of the regional offices. The science councils are located in each regional office and consist of EPA scientists and technical specialists. To determine the extent to which EPA communicated with and solicited views from outside stakeholders, we interviewed representatives from several professional library associations and other external stakeholder groups, such as the American Library Association, the Library of Congress, the Federal Library and Information Center Committee, and the Union of Concerned Scientists. We also reviewed information EPA provided to the public via the EPA Web site or, when applicable, Federal Register notices. We conducted this performance audit from October 2009 through September 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Environmental Protection Agency: United States Environmental Protection Agency: Office Of Environmental Information: Washington, D.C. 20460 [hyperlink, http://www.epa.gov] September 10, 2010 Mr. John B. Stephenson: Director: Natural Resources and Environment: United States Government Accountability Office: Washington, DC 20548: Re: FPA Comments on the Government Accountability Office's (GAO) draft report to Congress entitled EPA Needs to Complete a Strategy for Its Library Network to Meet Users' Needs (GAO-l0-947): Dear Mr. Stephenson: This letter provides the U.S. Environmental Protection Agency's (EPA) comments on GAO's draft report entitled EPA Needs to Complete a Strategy for Its Library Network to Meet Users Needs (GA0-10-947). EPA appreciates the opportunity to review and comment on this report to Congress. EPA's focus over the past two years has been to restore confidence in our library network to deliver high quality library services to our staff and the public and to fulfill the commitments made in the Agency's February 14, 2008 letter to you. EPA believes that we have made great strides in all of the areas identified by the GAO in 2008, and we appreciate that our progress has been noted by GAO and others, including highly regarded members of the library profession. In considering your comments related to our strategic plan, we acknowledge that the current planning document available on our web site has provided more of a working agenda to guide the rebuilding of our library program. The outline and draft documentation has served as a living document to guide our library staff toward solid service delivery and enable them to explore innovative ways to benefit science and policy making at EPA. We believe that we have sufficient input and direction from our library users' survey, the exceptional collaboration and cooperation of the library community in meetings and conferences, the library support contractor and the Library Advisory Board to move forward. We agree, however, that having reached a new level of maturity in this library network, it is time to complete and publish a formal plan that identifies an overall network strategy, with implementation goals and timeline for future accomplishments. The Agency agrees to move forward with the strategic plan and to include the comments noted in the enclosure, which will be the major focus of our library team in FY 2011. In addition to the strategic plan effort, we will address the inventory of library holdings, the feasibility and legality of digitizing products received by the Agency as the result of assistance agreements, and the design of any future user surveys in our follow-up to this report. In closing, EPA staff have benefited from this review and the thoughtful dialogue around the development of this report. We have addressed the specific recommendations individually, as an enclosure with this letter, to be included in the final report. Thank you again for the opportunity to comment. If you would like to discuss these matters further, please contact me at 202-564-6665 or Mr. Robin Gonzalez, Director of the Office of Information Analysis and Access at 202-566-0600. Sincerely, Signed by: Malcolm D. Jackson: Assistant Administrator and Chief Information Officer: Enclosure: cc: Craig E. Hooks, Assistant Administrator, Office of Administration and Resources Management: David McIntosh, Associate Administrator, Office of Congressional and Intergovernmental Relations: Barbara J. Bennett, Chief Financial Officer: Scott Fulton, General Counsel: Assistant Regional Administrators: [End of letter] Enclosure: September 10, 2010: Mr. John B. Stephenson: Response to GAO Recommendations: EPA National Library Network: GAO Recommendation: Complete EPA's strategic plan for the library network, including implementation goals and timelines. In doing so, EPA should outline details for how funding decisions are to be made, to ensure that they are informed and transparent. EPA Response: EPA has made progress towards the goal of developing a strategic plan for the library network. The library network has engaged in many activities, both within and outside the agency. which has provided valuable information to inform the strategic planning process. These activities include developing and promulgating a library network policy and related procedures to establish standard operations across the network; conducting extensive and regular outreach with internal and external stakeholders, professional associations and federal partners about user information needs and library best practices; engaging with members of the library network through teleconferences and in-person meetings to discuss the business and goals of the network and related library services; enhancing library collections in response to user needs; exhibiting and participating in national library conferences as well as internal EPA conferences where users informally express their needs; implementing new services in response to user feedback, including live chat reference and critical electronic resources; planning for and implementing Phase II of the digitization project; and gathering user feedback through a centralized customer satisfaction feedback form. Additionally, EPA conducted a formal user needs assessment in FY2009. The findings, which correlate strongly with our ongoing, informal information gathering and communications, provide a valuable information framework upon which to base EPA's library strategic plan. EPA will develop a strategic plan for the library network using the needs assessment findings alongside the additional data gathered from users and other activities. The network has established a core work group of federal librarians who have committed to participate in the strategic planning process. This group will work together with the National Program Manager and other stakeholders to complete the strategic plan in FY11. GAO Recommendation: Complete an inventory of the library network's holdings to identify what items remain to be digitized. EPA Response: EPA agrees with this recommendation and has undertaken steps necessary to inventory library holdings. The library network has identified cataloging of library materials as a priority for individual libraries. The AWBERC Library in Cincinnati is providing cataloging services for network libraries at no cost. To date, 18 network libraries have taken advantage of this service, resulting in the cataloging of more than 8,000 additional items. This process will enable EPA to more accurately estimate the remaining items to be digitized. EPA will complete a schedule for cataloging the inventory by November I, 2010 and will strive to complete the cataloging by September 30, 2011. In addition, the library network has developed library procedures that outline requirements for verifying the completeness of collection inventories on a regular basis and is implementing the procedures. GAO Recommendation: For assistance agreements already in place, EPA should digitize documents produced under the agreements to make them more readily available to EPA employees and other users for federal government purposes. EPA Response: EPA recognizes that digitizing documents produced under assistance agreements would provide a useful resource to the Agency at large; however, the "federal purpose" license does not allow EPA to unilaterally make a financial assistance recipient's copyrighted material available to the public. As we have developed our digitization plans, we have focused on the unique documents held in network libraries, after which the Agency will be in a position to consider documents produced under assistance agreements. Due to legal and technical constraints, EPA does not plan to digitize products from ongoing assistance agreements. As noted below, we will develop options for gaining permission from future agreement recipients to digitize products from their agreements. GAO Recommendation: In future assistance agreements, make explicit that EPA can include in the agency's public online database, without obtaining prior permission from the copyright holder, any documents produced under the agreements. EPA Response: Over the next six months, the Office of Environmental Information (0EI), working with the Office of Administration and Resources Management (OARM) and the Office of General Counsel (OGC) will develop options for gaining advance permission to digitize products from future assistance agreements. These options will be taken to Senior Agency managers in mid-2011 for their consideration and action. GAO Recommendation: For future assistance agreements where EPA cannot make such an arrangement, EPA should digitize documents produced under the agreements to make them more readily available to EPA employees and other users for federal government purposes. EPA Response: As noted above, this issue will be among the options evaluated by OEI, OARM and OGC. GAO Recommendation: Ensure that the data analysis protocols used for conducting surveys of users' needs — including sampling procedures and response rates — are sufficiently sound methodologically to provide reliable information on which to base decisions and allocate resources efficiently. EPA Response: EPA recognizes the concerns of GAO related to the methodology used in the recent needs assessment. We believe that we have sufficient information from the survey and our continuing feedback from users, the Library Advisory Board, and the library network staff to enable us to develop an effective strategic plan for the library network. The needs assessment survey was supplemented by a series of six focus group sessions with key user group segments and seven one-on-one in- depth interviews with management level stakeholders. These additional venues focused on the deeper issues and learnings from the quantitative survey and were done to help ensure a comprehensive perspective of user needs. Additionally, the findings of the needs assessment correlate with other data gathered on an ongoing basis from library users, further suggesting we have valid information on which to base such a plan. EPA will take the steps necessary to ensure that any future assessments are done with data analysis protocols that are methodologically sound and provide reliable information for future decision making. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: John Stephenson, (202) 512-3841 or stephensonj@gao.gov: Staff Acknowledgments: In addition to the contact above, Ed Kratzer, Assistant Director; Ellen W. Chu; Pamela Davidson; Les Mahagan; John C. Martin; Ben Shouse; and Jeannette Soares made key contributions to this report. [End of section] Footnotes: [1] GAO, Environmental Protection: EPA Needs to Ensure That Best Practices and Procedures Are Followed When Making Further Changes to Its Library Network, [hyperlink, http://www.gao.gov/products/GAO-08-304] (Washington, D.C.: Feb. 29, 2008). [2] Reorganization Plan No. 3 of 1970, 35 Fed. Reg. 15623 (Dec. 2, 1970) (5 U.S.C. Appendix 1). [3] A repository library is a central place that collects and preserves EPA documents and other materials deemed of value to the library network. EPA has designated three libraries as repository libraries, each with a subject specialty: the Headquarters Library for waste, pesticides, regional or local environmental topics deemed important to staff, and other topics not included elsewhere; the Research Triangle Park Library in Research Triangle Park, North Carolina, for air and health effects; and the Andrew W. Breidenbach Environmental Research Center in Cincinnati, Ohio, for water and risk assessment materials. [4] [hyperlink, http://www.gao.gov/products/GAO-08-304]. [5] Environmental Protection Agency, EPA National Library Network Report to Congress (Washington, D.C., 2008). EPA prepared and issued this report in response to a directive in the explanatory statement accompanying the fiscal year 2008 Consolidated Appropriations Act. [6] GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations, [hyperlink, http://www.gao.gov/products/GAO-03-669] (Washington, D.C.: July 2, 2003). [7] Environmental Protection Agency, "Draft Annotated Outline for the EPA Library Strategic Plan," Washington, D.C., July 25, 2007. [8] [hyperlink, http://www.gao.gov/products/GAO-03-669]. [9] These spending estimates, expressed in constant 2010 dollars, came from individual libraries in EPA's network and were provided to us by EPA. Aside from limited testing of the data from one region, we did not independently verify the reliability of the estimates. [10] Over the past decade, the Office of Environmental Information has provided from 41 to 47 percent of the library network's funding, except for fiscal years 2007 and 2008, when, because of reductions in its budget, the office provided 34 percent. In 2008, library spending was $7.3 million, reflecting $1 million added by Congress, and in fiscal year 2009, EPA increased library funding to $8.3 million. [11] Environmental Protection Agency, EPA National Library Network Report. [12] In our 2008 report (GAO-03-804, 36), we observed that it was essential that top leadership drive transformation of the library network and that without a national manager for the library network, EPA had no official providing essential oversight and guidance. [13] This board of advisors was established in response to a request from EPA for assistance in improving current and future library operations. Created in 1965, the Federal Library and Information Center Committee comprises the directors of the four national libraries--Library of Congress, National Library of Medicine, National Library of Education, and National Agriculture Library--and representatives of cabinet-level executive departments, as well as legislative, judicial, and independent federal agencies with major library programs; it is chaired by the Librarian of Congress. The committee's mission is to foster excellence in federal library and information services through interagency cooperation and to provide guidance and direction for the Federal Library and Information Network. [14] EPA defines a unique document as a document published by EPA or on behalf of EPA that exists in only one copy within EPA's National Library Network and that is not already electronically available in NEPIS. [15] Copyright in works arising under contracts is governed by Subpart 27.4 of the Federal Acquisition Regulation (FAR) (48 C.F.R. §§ 27.400- 27.409). FAR Data Rights Clause No. 52.227-14, Rights in Data--General (48 C.F.R. § 52.227-14), is found in most EPA research and development contracts. [16] 40 C.F.R. § 31.34 (copyrights in works produced under grants and cooperative agreements to state and local governments); 40 C.F.R. § 30.36 (copyrights in works produced under grants and cooperative agreements to institutions of higher education, hospitals, and other nonprofits). [17] Environmental Protection Agency, Digitization of EPA Library Materials (Washington, D.C., 2009). [18] The Federal Library and Information Network is the business subsidiary of the Federal Library and Information Center Committee and serves federal libraries and information centers as their purchasing, training, and resource-sharing consortium. [19] In July 2008, after a February 2008 arbitration ruling, EPA management and representatives of its employees' union entered into a memorandum of agreement to resolve concerns surfaced during the reorganization. [20] Environmental Protection Agency, Target User Accessibility Review: NSCEP/NEPIS: EPA's Gateway to Free Digital and Paper Publications (Washington, D.C., 2008). [21] These efforts were aided by discussions with staff from other federal libraries, including the National Agricultural Library and the Library of Congress. [22] [hyperlink, http://www.gao.gov/products/GAO-08-304]. [23] Office of Management and Budget, Standards and Guidelines for Statistical Surveys, Washington, D.C., 2005, and Office of Management and Budget, Guidance on Agency Survey and Statistical Information Collections, Washington, D.C., 2006. [24] According to the Office of Management and Budget's guidance, a low response rate may indicate an error known as nonresponse bias, which can significantly lower the accuracy of survey results. Such bias may occur if survey participants differ substantially and systematically from nonparticipants in ways that might influence their responses to survey questions. Similarly, surveys with few respondents may also be biased because they may fail to capture the true range of variability in the surveyed population. [25] According to EPA estimates, 20 to 40 percent of the reference requests received by regional libraries come from the public. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: