This is the accessible text file for GAO report number GAO-13-99 
entitled 'Information Management: National Technical Information 
Service's Dissemination of Technical Reports Needs Congressional 
Attention' which was released on November 19, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office:
GAO: 

Report to Congressional Committees: 

November 2012: 

Information Management: 

National Technical Information Service's Dissemination of Technical 
Reports Needs Congressional Attention: 

GAO-13-99: 

GAO Highlights: 

Highlights of GAO-13-99, a report to congressional committees. 

Why GAO Did This Study: 

NTIS was established by statute in 1950 to collect scientific and 
technical research reports, maintain a bibliographic record and 
repository of these reports, and disseminate them to the public. NTIS 
charges fees for its products and services and is required by law to 
be financially self-sustaining to the fullest extent possible. 

GAO was mandated by Congress to update its 2001 report on aspects of 
NTIS’s operations and the reports in its collection. Specifically, GAO’
s objectives were to determine (1) how NTIS is currently organized and 
operates, including its functions, current staffing level, reported 
cost of operations, and revenue sources; (2) the age of and demand 
trends for reports added to NTIS’s repository; and (3) the extent to 
which these reports are readily available from other public sources. 
To do this, GAO reviewed agency documentation, analyzed a sample of 
reports added to NTIS’s collection from fiscal years 1990 through 2011 
(reports from the period since GAO’s last study and other older 
reports), and interviewed relevant agency officials. 

What GAO Found: 

As a component of the Department of Commerce, the National Technical 
Information Service (NTIS) is organized into five primary offices that 
offer the public and federal agencies a variety of products and 
services. As of late October 2012, NTIS was supported by 181 staff, 
all except 6 of which held full-time positions. NTIS reports its 
progress toward agency goals to the Deputy Secretary of Commerce, and 
the Director of NTIS reports to the Director of Commerce’s National 
Institute of Standards and Technology. In addition, NTIS receives 
oversight of its functions and strategic direction from an advisory 
board with members appointed by the Secretary of Commerce. NTIS’s 
product and service offerings include, among other things, 
subscription access to reports contained in its repository in both 
print and electronic formats, distribution of print-based 
informational materials to federal agencies’ constituents, and 
digitization and scanning services. 

NTIS revenues are generated exclusively from direct sales or 
subscriptions for its products and services. NTIS reported that net 
revenues from all its functions (products and services) totaled about 
$1.5 million in fiscal year 2011. However, over most of the last 11 
years, its costs have exceeded revenues by an average of about $1.3 
million for its products. While NTIS has not recovered all of its 
costs for products through subscriptions and other fees, it has been 
able to remain financially self-sustaining because of revenues 
generated from its services such as distribution and order 
fulfillment, web hosting, and e-training. The NTIS strategic plan 
states that the electronic dissemination of government technical 
information by other federal agencies has contributed to reduced 
demand for NTIS’s products. As a result, the agency is taking steps to 
reduce its net costs, such as improving business processes and 
increasing the breadth and depth of its collection. 

NTIS’s repository has been growing with mostly older reports, but the 
demand for more recent reports is greater. Specifically, NTIS added 
approximately 841,500 reports to its repository during fiscal years 
1990 through 2011, and approximately 62 percent of these had 
publication dates of 2000 or earlier. However, the agency was more 
likely to distribute (by direct sale or through a subscription) 
reports published more recently. For example, GAO estimated that 100 
percent of the reports published from 2009 through 2011 had been 
distributed at least once, while only about 21 percent of reports 
published more than 20 years ago had been. 

Of the reports added to NTIS’s repository during fiscal years 1990 
through 2011, GAO estimates that approximately 74 percent were readily 
available from other public sources. These reports were often 
available either from the issuing organization’s website, the federal 
Internet portal [hyperlink, http://www.USA.gov], or from another 
source located through a web search. Reports published from 1990 to 
2011 were more likely to be readily available elsewhere than those 
published in 1989 or earlier. Further, GAO estimated that 95 percent 
of the reports available from sources other than NTIS were available 
free of charge. NTIS’s declining revenue associated with its basic 
statutory function and the charging for information that is often 
freely available elsewhere suggests that the fee-based model under 
which NTIS currently operates for disseminating technical information 
may no longer be viable and appropriate. 

What GAO Recommends: 

GAO is suggesting that Congress reassess the appropriateness and 
viability of the fee-based model under which NTIS currently operates 
for disseminating technical information to determine whether the use 
of this model should be continued. In comments on a draft of this 
report, the Department of Commerce stated that NTIS believes GAO’s 
conclusions do not fully reflect the value that the agency provides. 
However, GAO maintains that its conclusions and suggestion to Congress 
are warranted. 

View [hyperlink, http://www.gao.gov/products/GAO-13-99]. For more 
information, contact Valerie C. Melvin at (202) 512-6304 or 
melvinv@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

NTIS Carries out a Variety of Functions Related to Its Mission, but 
Currently Is Financially Sustained by Its Service Offerings: 

Additions to NTIS's Repository Have Mostly Included Older Reports, but 
Demand for More Recent Reports Is Greater: 

Most Reports Added Since 1990 Are Freely Available from Other Websites: 

Conclusions: 

Matter for Congressional Consideration: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Commerce: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: NTIS Staff as of October 31, 2012: 

Table 2: NTIS Office Responsibilities and Functions: 

Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS Products 
and Services, Fiscal Years 2001-2011: 

Figures: 

Figure 1: NTIS Organization Chart: 

Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net Costs for 
NTIS Products and Services: 

Figure 3: Year of Publication for Documents Added to NTIS Repository, 
1990 through 2011: 

Figure 4: Estimated Percentage of Reports Distributed and Not 
Distributed by Publication Date during Fiscal Years 2000 through 2011: 

Figure 5: Estimated Availability of Reports by Year of Publication: 

Figure 6: Estimated Availability of Reports Distributed and Not 
Distributed During Fiscal Years 2000 through 2011: 

Figure 7: Of Reports Found Elsewhere, Estimated Percent Available for 
Free and Not Free, by Year of Publication: 

Abbreviations: 

Commerce: Department of Commerce: 

NASA: National Aeronautics and Space Administration: 

NIST: National Institute of Standards and Technology: 

NTIS: National Technical Information Service: 

SSA: Social Security Administration: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

November 19, 2012: 

The Honorable Barbara Mikulski: 
Chairwoman: 
The Honorable Kay Bailey Hutchison: 
Ranking Member: 
Subcommittee on Commerce, Justice, Science, and Related Agencies: 
Committee on Appropriations: 
United States Senate: 

The Honorable Frank Wolf: 
Chairman: 
The Honorable Chaka Fattah: 
Ranking Member: 
Subcommittee on Commerce, Justice, Science, and Related Agencies: 
Committee on Appropriations: 
House of Representatives: 

The Department of Commerce's National Technical Information Service 
(NTIS) serves as a permanent repository and disseminator of 
scientific, technical, engineering, and business-related information. 
[Footnote 1] NTIS acquires such information--largely in the form of 
research reports--primarily from federal agencies and their 
contractors and grantees, as well as from other domestic and foreign 
sources. NTIS estimates that it maintains in its central repository 
more than 2.5 million records[Footnote 2] covering 378 technical and 
business-related subject areas. In addition, NTIS performs various fee-
based information services for other federal agencies. The legislation 
that established NTIS[Footnote 3] requires all of the services and 
functions it provides to be financially self-sustaining, to the extent 
feasible. 

In May 2001, we reported on NTIS's operations,[Footnote 4] noting 
among other things, the availability of many of the reports maintained 
in its repository from other sources, such as the originating 
agencies' websites. House Report 112-169, accompanying the 
Consolidated and Further Continuing Appropriations Act, 2012,[Footnote 
5] directed GAO to update aspects of its prior study. Our specific 
objectives were to determine (1) how NTIS is currently organized and 
operates, including its various functions, current staffing level, 
reported costs of operations, and revenue sources; (2) the age of and 
demand trends for reports added to NTIS's repository; and (3) the 
extent to which these reports are readily available from other public 
sources. 

To determine how NTIS is organized and operates, we reviewed the 
agency's strategic plan and documentation on its organizational and 
reporting structure, office staffing level and assigned 
responsibilities, and types of products and services offered. We also 
reviewed cost data contained in the agency's financial reports. 
[Footnote 6] In addition, we analyzed relevant laws and regulations 
describing NTIS's authority and responsibilities,[Footnote 7] and 
reviewed our previous reports that discussed its mission and 
operations.[Footnote 8] We supplemented our analyses with interviews 
of the Director of NTIS and other relevant agency officials; we also 
interviewed officials of the Department of Commerce and its National 
Institute of Standards and Technology (NIST), which have specific 
reporting relationships with NTIS. 

To determine the age of and demand trends for reports added to NTIS's 
repository, we first analyzed the distribution of the publication 
dates for all reports added to the repository since 1990. We chose to 
include reports added since 1990 to determine the current availability 
of older documents in light of NTIS's role as a permanent repository, 
and to focus on the reports added since our previous review in 
2001.[Footnote 9] We subsequently selected a stratified random sample 
from this body of reports[Footnote 10] and then used cumulative data 
provided by NTIS on the sales of any of these reports to analyze the 
demand for reports in the repository. 

To determine the extent to which reports NTIS collects are readily 
available from other public sources, we searched the Internet to 
determine if each of the reports included in our sample could be found 
elsewhere and at no cost. Using a tiered approach, we searched the 
following four sources in the order shown: (1) the issuing 
organization's website; (2) the U.S. Government Printing Office's 
Federal Digital System website--[hyperlink, http://www.gpo.gov/fdsys]; 
(3) the federal government Internet portal USA.gov--[hyperlink, 
http://www.USA.gov]; and (4) a web search conducted using the 
commercial search engine [hyperlink, http://www.Google.com]. 
Specifically, we determined whether each report was first available at 
no cost on the issuing organization's website and, if so, concluded 
the Internet search at this point. However, if the report was not 
available, then the search continued to the second source, and so on, 
until either the report was found to be available at one of the 
remaining sources, or all sources were exhausted. We then used our 
results to estimate the percentage of the total population of NTIS 
reports added to the repository during fiscal years 1990 through 2011 
that was available from other public sources. 

All of the results derived from the sample analyses constituted 
estimates that are subject to sampling errors. These sampling errors 
measure the extent to which the sample size and structure are likely 
to differ from the population they represent. Because we followed a 
probability procedure based on random selections, our sample is only 
one of a large number of samples that we might have drawn. Since each 
sample could have provided different estimates, we express our 
confidence in the precision of our particular sample's results as a 95 
percent confidence interval. This is the interval that would contain 
the actual population value for 95 percent of the samples we could 
have drawn. 

A more detailed discussion of our objectives, scope, and methodology 
is provided in appendix I. 

We conducted this performance audit from February 2012 to November 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

NTIS's basic statutory function is to collect research reports, 
maintain a bibliographic record and permanent repository of these 
reports, and disseminate them to the public. In addition, NTIS has 
developed a variety of information-related services. NTIS charges user 
fees for the sale of its products to the public and services to 
federal agencies. 

Under statutory authority enacted in 1950,[Footnote 11] NTIS collects 
reports containing scientific, technical, and engineering information 
from both domestic and foreign sources in a repository and makes the 
information available to (1) business and industry, (2) state and 
local governments, (3) other federal agencies, and (4) the general 
public to increase U.S. competitiveness in the global economy. The 
statute does not define scientific, technical, and engineering 
information. However, the Secretary of Commerce has interpreted this 
to include "all types of information which have a more or less direct 
bearing on business and industry generally." [Footnote 12] The 
Secretary of Commerce described such information as including 
"economic information, market information, and related information so 
long as it is reasonably specific and bears some direct relationship 
to the organization and operation of industrial or business 
enterprise." NTIS's enabling legislation authorized it to charge fees 
for its products and established a policy to recover all costs, as 
feasible, through the fees.[Footnote 13] 

NTIS's authority was revised in the National Technical Information Act 
of 1988.[Footnote 14] This act gave the agency authority to also enter 
into contracts, cooperative agreements, joint ventures, and other 
transactions as necessary in the conduct of the business of NTIS, and 
declared the NTIS repository to be a permanent federal function that 
could not be transferred to the private sector without congressional 
approval. The act was subsequently amended by the American Technology 
Preeminence Act of 1991,[Footnote 15] which: 

* required all costs associated with acquisition, processing, storage, 
bibliographic control, and archiving to be recovered primarily by fees; 

* required agencies to transfer unclassified scientific, technical, 
and engineering information which results from federally funded 
research and development to NTIS; and: 

* provided that NTIS's use of new methods or media for information 
dissemination should include producing and disseminating information 
in electronic format. 

Further, the Commerce, Justice, and State, the Judiciary, and Related 
Agencies Appropriations Act of 1993[Footnote 16] established a 
revolving fund for the payment of all expenses incurred by NTIS and 
gave it the authority to use that fund without further appropriations 
action by Congress. 

Our prior work found that, from fiscal years 1988 through 1994, NTIS 
showed excess revenue from the sale of its products and services in 
all years except one and generated cumulative net earned revenues of 
$5.8 million. However, over the next 5-year period (from fiscal year 
1995 through fiscal year 1999), NTIS had a cumulative net cost 
[Footnote 17] of about $4.2 million. Accordingly, in August 1999, the 
Secretary of Commerce proposed closing NTIS by September 30, 2000, 
because he believed that declining sales revenues would not continue 
to be sufficient to recover all of the agency's operating costs. The 
Secretary attributed this decline partly to other agencies' practice 
of making their research results available to the public for free 
through the Web. He also proposed transferring NTIS's research report 
archives to the Library of Congress and requiring federal agencies to 
give the public free online access to new research reports. 

In June 2000,[Footnote 18] we reported that NTIS could possibly remain 
solvent for a few years because Commerce and NTIS officials had acted 
in fiscal year 1999 to reduce costs when it appeared there would not 
be sufficient funds to operate. However, we noted that the agency's 
solvency beyond fiscal year 2003 was in doubt based on our forecast of 
its revenues and costs, which suggested that costs would exceed 
revenues beginning in fiscal year 2001 and continue. We pointed out 
that, if these net costs were realized, cumulative net earned revenues 
from previous years' operations could be absorbed and the agency could 
run out of operating funds after fiscal year 2003. Based on these 
findings, we suggested that Congress consider whether NTIS (or another 
agency) was needed to serve as a permanent repository and disseminator 
for research reports; we also suggested that Congress could be aided 
in this assessment by reviewing NTIS annual financial and operational 
reports that Commerce was required to submit, but which it had not 
done since fiscal year 1995. 

We again reported on NTIS's operations in May 2001.[Footnote 19] We 
noted that, according to its Deputy Director, the agency had 
significantly reduced its costs after our June 2000 report and had 
shown net earned revenues of $2.2 million for fiscal year 2000. 
Moreover, the Deputy Director had projected a net earned revenue of 
$1.5 million for fiscal year 2001. We also noted that the agency was 
providing a variety of other fee-based services for agencies and that, 
while demand for electronic products was on the rise, research reports 
and other scientific, technical, and engineering information 
maintained by NTIS were also becoming increasingly available on agency 
websites and through other public sources--often at no cost. We 
included as a matter for congressional consideration that Congress 
look at how this information was defined; whether there was a need for 
a central repository of this information; and if a central repository 
was maintained, whether all information should be retained 
permanently, and what business model should be used to manage it. 

The Secretary of Commerce agreed with our assessment and raised as a 
primary question whether there was a need for a central repository in 
view of the increasing availability of newer publications from sources 
other than NTIS. The Secretary also noted that the need for a central 
repository depended on whether the information would be permanently 
maintained by agencies and whether the information would be easy to 
locate without the kind of bibliographic control that NTIS provides. 

Subsequent to the issuance of our reports, in December 2003, Congress 
passed the 21st Century Nanotechnology Research and Development Act, 
which provided a coordinated federal approach to stimulating 
nanotechnology research and development. The act directed the 
Secretary of Commerce to establish a clearinghouse for information 
related to the commercialization of nanotechnology research using the 
resources of NTIS to the extent possible. As of September 2012, NTIS 
noted that it held over 700 publications in its nanotechnology 
collection. 

NTIS Carries out a Variety of Functions Related to Its Mission, but 
Currently Is Financially Sustained by Its Service Offerings: 

NTIS currently operates as one of 12 independent bureaus within 
Commerce, with the mission to help promote the nation's economic 
growth by providing access to information that stimulates innovation 
and discovery. Headquartered in Alexandria, Virginia (with warehouses 
in Alexandria, Virginia and in Brandywine, Maryland), the agency is 
organized into five primary offices. To carry out its statutory 
functions of collecting and maintaining a permanent repository and 
bibliographic record of research reports, and to disseminate them, the 
agency offers a variety of products, such as fee-based access to the 
reports in its repository. In addition, NTIS offers information-
related services to federal agencies, such as distribution and order 
fulfillment, web hosting, and e-training, that are less directly 
related to its basic statutory function. While NTIS's service 
offerings have resulted in increased revenues, allowing the agency to 
remain financially self-sustaining, it has experienced a net cost 
relative to its products, calling into question whether the agency's 
function of acting as a self-financing repository of technical 
information is still viable. 

NTIS Receives Guidance and Oversight from Commerce Department Entities 
and Carries out Its Functions through Five Primary Offices: 

NTIS is led by a director, who is aided by two executives--a chief 
information officer and a chief financial officer. In addition, three 
operational offices have a variety of responsibilities for providing 
products and services that include collecting and disseminating 
technical reports, offering access to other information sources, and 
providing information-related services to federal agencies. Figure 1 
displays the NTIS organization. 

Figure 1: NTIS Organization Chart: 

[REfer to PDF for image: Organization Chart] 

Top level: 
Director; 
* Deputy Director. 

Second level: 
Office of the Chief Financial Officer; 
Office of the Chief Information Officer. 

Third level: 
Office of Federal Services; 
Office of Product Management and Acquisitions; 
Office of Production Services. 

Source: NTIS. 

[End of figure] 

NTIS operates as a unit within Commerce and receives oversight from 
the Deputy Secretary of Commerce, the Director of NIST,[Footnote 20] 
and an advisory board. In this regard, the NTIS Director communicates 
progress toward agency goals to the Deputy Secretary of Commerce. For 
example, the Director participates in biweekly executive management 
meetings that are held with the Deputy Secretary. At these meetings, 
executive leads from each of Commerce's components report on the 
status of performance and strategic goals within their offices, among 
other things. 

In addition, under a Commerce departmental order,[Footnote 21] the 
NTIS Director also reports to the Director of NIST. As outlined in the 
Commerce organization chart and stated in the Commerce departmental 
organization order, the NTIS Director is to report and be responsible 
to the Director of NIST. In turn, the Director of NIST prepares the 
NTIS Director's annual performance appraisal. Beyond this, the 
Director of NIST stated that the agency does not provide any other 
operational or financial oversight functions for NTIS. For example, 
NIST does not approve NTIS's budget (although it does coordinate 
NTIS's budget for final inclusion in the department's overall budget, 
which is approved by Commerce). 

Further, NTIS receives guidance on its operations from the NTIS 
Advisory Board, which was established by law[Footnote 22] in 1988 to 
review the general policies and operations of NTIS, including policies 
related to fees and charges for its products and services. The 
board,[Footnote 23] comprised of a chairperson and four members 
appointed by the Secretary of Commerce, is required to meet at least 
every 6 months to discuss NTIS's activities. The board's last meeting 
took place in late October 2012, and, according to the notice of open 
meetings from the Federal Register, the intended focus was on the 
agency's strategic business plan. The board submits an annual report 
to the Secretary of Commerce, which includes strategic and tactical 
recommendations regarding NTIS's future operations. 

Table 1: NTIS Staff as of October 31, 2012: 

NTIS office: Director; 
Full-time employees: 2; 
Part-time employees: 0; 
Full-time contractors: 0; 
Part-time contractors: 0; 
Total: 2. 

NTIS office: Chief Financial Officer; 
Full-time employees: 13; 
Part-time employees: 0; 
Full-time contractors: 0; 
Part-time contractors: 0; 
Total: 13. 

NTIS office: Chief Information Officer; 
Full-time employees: 26; 
Part-time employees: 0; 
Full-time contractors: 26; 
Part-time contractors: 5; 
Total: 57. 

NTIS office: Federal Services; 
Full-time employees: 15; 
Part-time employees: 0; 
Full-time contractors: 3; 
Part-time contractors: 0; 
Total: 18. 

NTIS office: Product Management and Acquisition; 
Full-time employees: 29; 
Part-time employees: 1; 
Full-time contractors: 5; 
Part-time contractors: 0; 
Total: 35. 

NTIS office: Production Services; 
Full-time employees: 17; 
Part-time employees: 0; 
Full-time contractors: 39; 
Part-time contractors: 0; 
Total: 56. 

NTIS Total staff: 
Full-time employees: 102; 
Part-time employees: 1; 
Full-time contractors: 73; 
Part-time contractors: 5; 
Total: 181. 

Source: NTIS data. 

[End of table] 

As of late October 2012, NTIS was supported by 181 staff, all except 6 
of which held full-time positions. These included 103 NTIS employees 
and 78 contractors. Table 1 shows the number of staff dedicated to the 
Office of the Director and each of the agency's other primary offices 
as of October 2012. 

Table 2: NTIS Office Responsibilities and Functions: 

Office: 
Key responsibilities: 

Office of the Director; 
* Manages and is responsible for all activities of NTIS. In addition 
to the Director, officials in this office include a contracting 
officer, legal personnel, and program analysts. 

Office of the Chief Financial Officer; 
* Develops NTIS budgets, financial plans, financial reports, and 
analyses;
* Develops, maintains, and operates NTIS financial systems;
* Provides financial advice to the Director of NTIS;
* Provides management analysis, administrative management, 
procurement, facilities management, property management, and office 
support services to NTIS components and other agencies;
* Provides financial management services to NTIS components, 
customers, and source clients; 

Office of the Chief Information Officer; 
* Develops and administers plans, policies, and standards for NTIS’s 
acquisition and use of information technology;
* Represents NTIS on information technology matters, including 
computer security issues;
* Plans, develops, operates, and services centralized and 
decentralized network computer-based capabilities and associated 
telecommunications networks;
* Provides training, consultation, technical documentation, and 
operation of data processing and telecommunications. 

Office of Federal Services; 
* Develops strategies for information dissemination services and 
Internet-based service business opportunities for federal agencies;
* Manages service projects using in-house capabilities or through 
joint venture partners;
* Assists agencies in meeting their multimedia product requirements;
* Coordinates marketing and exhibit activities for information 
dissemination and Internet-based services. 

Office of Product Management and Acquisition; 
* Builds the NTIS bibliographic database by cataloging, indexing, and 
abstracting information products from either hard copy or electronic 
media;
* Identifies and acquires scientific, technical, and business-related 
information from domestic and international sources;
* Coordinates the production and sale of clearinghouse products;
* Manages customer relationships involving order receipt/entry, 
inquiry and complaint resolution, and product identification functions;
* Gathers data about customers’ or potential customers’ needs and 
expectations;
* Promotes available federal government information, services, and 
programs to customers through demand and subscription sales. 

Office of Production Services; 
* Provides services relative to order fulfillment, information product 
dissemination and service delivery, order processing, product 
manufacturing, warehousing, specialized storage, inventory management, 
and distribution;
* Operates, maintains, and manages specialized equipment and computer-
based systems for the imaging, publication, and reproduction of 
information products;
* Provides project- and product-management support for individual 
products and services. 

Source: GAO analysis of Commerce data. 

[End of table] 

NTIS Offers Technical Reports and Other Products to a Variety of 
Customers: 

As part of its basic statutory function to collect and disseminate 
technical reports, NTIS offers a variety of fee-based products. 
According to the agency, its customer base for these products include 
scientists, engineers, the business community, librarians, information 
specialists in government, academia, and the general public. The 
agency has organized its products into three lines of business--
"Technical Reports," "Clearinghouse," and "Publishing"--as described 
below. 

Technical Reports: 

To carry out its statutory function of collecting and disseminating 
such information, NTIS maintains a searchable repository that contains 
bibliographic records for the over 2.5 million scientific, technical, 
engineering, and business research reports it has acquired from 
federal government agencies, state and local governments, 
international organizations, and private-sector organizations. NTIS's 
bibliographic database provides for the classification and cataloging 
of the records in this database. For example, according to the NTIS 
Database Search Guide, NTIS classifies its records into 39 categories. 
These categories can be used for searching the contents of NTIS's 
database. The database covers a host of scientific and technical 
subjects, such as biology, chemistry, physics, transportation, health 
care, and the environment. Of the 2.5 million reports, NTIS noted that 
approximately 700,000 reports have been digitized, with the remainder 
in physical form, such as paper or microfilm. 

Access to the reports is provided both through the direct sale of 
individual reports and by subscription. Individual reports can be 
purchased via postal mail; phone; e-mail; and NTIS's online ordering 
system, accessed through its website, [hyperlink, 
http://www.ntis.gov]. Subscription-based access to the reports is 
obtained through, among other things, NTIS's National Technical 
Reports Library, which provides subscribers with the ability to search 
the bibliographic records repository and to access the approximately 
700,000 digitized reports in portable document format (PDF).[Footnote 
24] In addition, NTIS's Selected Research Service allows a subscriber 
to select from more than 378 subcategories and automatically receive 
reports tailored to that area of interest. According to NTIS, reports 
that have not been digitized can be provided in a digital format when 
a customer purchases a copy. 

Prices for individual reports and subscriptions vary. For example, an 
electronic copy of a report from the Economics and Statistics 
Administration, Benefits of Manufacturing Jobs: Executive Summary, can 
be purchased for $15; a "customized CD" for this report can be 
purchased for $30. Further, an electronic copy of a report from the 
National Aeronautics and Space Administration (NASA) Marshall Space 
Flight Center, NASA Robotics for Space Exploration, can be purchased 
for $15; a "customized CD" for this report can be purchased for $30. 
With regard to subscriptions, access to the National Technical Reports 
Library is sold as an annual subscription to institutions based on the 
number of individuals accessing the library. For example, as of 
October 2010, an annual subscription providing access for up to 3,000 
individuals costs $2,100, while an annual subscription providing 
access for 18,001 to 28,000 individuals costs $11,200. In addition, 
subscriptions can be purchased for a specific number of issues of a 
particular document type. For example, six issues of the Livestock, 
Dairy, and Poultry Outlook can be purchased for $91. 

Clearinghouse: 

In addition to the technical reports that it collects and 
disseminates, NTIS disseminates publications covering a wide array of 
topics on behalf of other federal agencies. According to NTIS, these 
agencies request that NTIS distribute the publications in print or 
electronically. The following are examples of the federal products 
distributed: 

* Standard Occupational Classification Manual. A manual containing 
information on all occupations in the national economy classified 
according to the system used by federal statistical agencies for the 
purpose of collecting, calculating, analyzing, or disseminating 
occupational data. 

* Food and Drug Administration Food Code Manual. A code and reference 
document that provides technical and legal information about the 
regulations of the retail and food service industry. 

* North American Industry Classification System. A publication that 
details a system for the collection, analysis, and dissemination of 
industrial statistics used by government policy analysts, academics, 
researchers, the business community, and the public. 

* Export Administration Regulations. A compilation of regulations 
issued by Commerce's Bureau of Industry and Security relating to the 
control of certain exports, re-exports, and activities. 

* National Correct Coding Policy Manual. A manual developed by the 
Centers for Medicare and Medicaid Services to control improper coding 
leading to inappropriate payment in Medicare Part B claims. The manual 
provides guidance for providers on the correct coding of claims being 
submitted for reimbursement. 

Publishing: 

Through a memorandum of understanding or interagency agreement, NTIS 
also provides access to information collected from federal agencies, 
which it refers to as its "Publishing" line of business. In some 
instances, NTIS repackages the information with additional features. 
According to NTIS, agencies initiate the request for these services. 
These offerings include the following: 

* Drug Enforcement Administration database. The Drug Enforcement 
Administration database identifies persons and organizations 
authorized to handle certain controlled drug substances and chemicals 
under the Controlled Substance Act. NTIS is the authorized official 
distributor of the database. NTIS provides online subscription access 
to the database on a weekly, monthly, or quarterly basis. NTIS also 
provides this information via a searchable CD-ROM. 

* Death Master File. This file, maintained by the Social Security 
Administration (SSA), contains approximately 85 million records of 
deaths that have been reported since 1936. The file is used by 
government; credit reporting organizations; and financial, 
investigative, medical research, and other industries to verify 
deaths. Through an agreement with SSA, NTIS is the only authorized 
official distributor of the Death Master File. In this regard, NTIS 
provides access to this information, including on DVD, and provides 
the means to search and download the Death Master File online. 

* World News Connection. NTIS offers access to this online news 
service, which provides translated and English-language news and 
information from non-U.S. media sources. The information is obtained 
from full-text, and summaries of newspaper articles, conference 
proceedings, television and radio broadcasts, periodicals, and 
nonclassified technical reports. The material in World News Connection 
is provided to NTIS by the Open Source Center, a U.S. government 
agency that provides analysis of foreign open source intelligence. 

NTIS Provides Fee-Based Services to Federal Agencies: 

In addition to its product offerings, NTIS offers a variety of fee-
based services to federal agencies that are less directly related to 
its basic statutory function of collecting and disseminating 
scientific and technical information. These include services that 
leverage capabilities NTIS has developed in the course of carrying out 
its mission. Its five service offerings are distribution and order 
fulfillment, web-based services and federal cloud computing, brokerage 
services, e-training and knowledge management services, and 
digitization and scanning services. To provide its services, NTIS 
enters into memorandums of understanding or interagency agreements 
with agencies. Further, NTIS offers some of these services through 
public-private partnerships with private industry, which it refers to 
as "joint venture partners." The five types of services are described 
below. 

Distribution and Order Fulfillment Services: 

Through a memorandum of understanding or interagency agreement, NTIS 
distributes large amounts of information products for federal 
agencies. According to NTIS officials, these services differ from that 
provided by its Clearinghouse in that they are used for distributing 
large quantities of agencies' products rather than selling individual 
copies of publications. NTIS identified five primary clients for these 
services: the Internal Revenue Service, the Department of Agriculture, 
the Department of Education, the Pension Benefit Guaranty Corporation, 
and the Administrative Office of the U.S. Courts. For example, NTIS 
currently has an agreement with the Department of Agriculture to 
distribute on its behalf health and nutrition educational materials in 
the form of brochures, posters, and similar nutritional products. It 
also has an agreement with the Department of Education to perform 
similar services. 

Web-Based Services and Federal Cloud Computing: 

In 1988, Congress required NTIS to implement new methods or media for 
disseminating technical information; a 1992 amendment[Footnote 25] 
specified that this should include producing and disseminating 
products in electronic formats. According to NTIS, this has been a 
primary basis for NTIS's transformation from a static paper-based 
distribution operation to a modern, computer-based model and also a 
basis for NTIS to provide information-dissemination services to other 
agencies. For example, NTIS's information systems infrastructure 
enables it to host federal agencies' applications and websites. The 
agency currently has an agreement with the Federal Insurance and 
Mitigation Administration to provide hosting for two of its websites 
and associated systems and to host its web-based tool for reviewing 
underwriting and claims operations. NTIS has also expanded its 
infrastructure to provide cloud computing[Footnote 26] services and, 
according to the agency, is currently offering infrastructure-as-a-
service and software-as-a-service.[Footnote 27] For example, along 
with joint venture partner Carney, Inc., NTIS had an agreement with 
the National Archives and Records Administration and currently has an 
agreement with the Social Security Administration to configure and 
host the "Jive" platform.[Footnote 28] According to NTIS, the agency 
has eight primary clients for its web-based services and federal cloud 
computing offerings, including the Department of Homeland Security, 
the Internal Revenue Service, and other federal agency initiatives. 
[Footnote 29] 

Brokerage Services: 

NTIS provides billing and collection services on a reimbursable basis 
to other agencies that, like itself, charge for products and services 
but which lack the necessary financial infrastructure to do their own 
billing and collecting. The agency refers to these as its "Brokerage 
Services." For example, NTIS had an agreement with the National 
Agricultural Library to develop, implement, and operate account 
maintenance, invoicing, and collection procedures for the fees charged 
by the National Agricultural Library to users of its photocopy and 
loan services. In addition, the agency had an agreement with the 
National Library of Medicine to perform invoicing, accounting, and 
collection services for its Interlibrary Loan services. NTIS officials 
stated, however, that the agency plans to stop marketing its brokerage 
services due to the decrease in demand for this service. 

E-Training and Knowledge Management Services: 

NTIS's service offerings have also been expanded to e-training and 
knowledge management. Specifically, in conjunction with joint venture 
partners, the agency provides collaborative software solutions, 
learning management systems and support services, training evaluation 
software, and talent management applications. For example, NTIS 
entered into an agreement with Booz Allen Hamilton to provide, among 
other things, program management; secure Internet hosting; and 
operations, maintenance, and support services for the Defense Manpower 
Data Center's enterprise training program. According to NTIS, as of 
May 2012, it had 28 primary government clients for this service 
offering, including the Departments of Justice, the Interior, and the 
Treasury, and the U.S. Patent and Trademark Office. 

Digitization and Scanning Services: 

As another service, NTIS digitizes various document types, such as 
microfilm or microfiche and paper forms, to assist agencies in 
complying with section 508 standards.[Footnote 30] Further, it offers 
storage and distribution for the documents that it digitizes. For 
example, NTIS has an agreement with SSA to provide alternative modes 
of receiving SSA notices and other communications. NTIS provides this 
service with the assistance of its joint venture partners, Vastec, 
Inc. and Braille Works, Inc. NTIS offers this information in data 
compact disc, large print, and audio compact disc. As of May 2012, the 
agency had six primary clients for its digitization and scanning 
services, including SSA and the Department of Justice. 

NTIS’s Reported Revenues and Costs over the Past Decade: 

As a fee-for-service entity, NTIS’s revenues are generated exclusively 
from its products and services, and all its revenues, expenses, and 
capital expenditures are deposited into and paid out of its revolving 
fund. Overall, NTIS had net earned revenues for 8 of the last 11 
fiscal years. For example, for fiscal year 2011, the agency reported 
that net earned revenues from all its functions (products and 
services) totaled about $1.5 million. According to NTIS’s Financial 
Report for fiscal year 2011, the revolving fund ending unobligated 
balance[Footnote 31] was approximately $7.4 million. 

However, over most of the last decade, the agency has incurred net 
costs for its products. Specifically, NTIS product expenditures 
exceeded revenues for 10 out of the past 11 fiscal years. The agency 
lost, on average, about $1.3 million over the last 11 years on its 
products. 

In contrast, NTIS’s overall financial performance has been supported 
by revenues from its service offerings. The agency’s service revenues 
increased, on average, about $1.8 million over the last 11 years. In 
particular, for fiscal year 2011, revenues were about $53.5 million, 
costs incurred were about $52 million, and the overall net earned 
revenue was approximately $1.5 million from its service offerings. 
NTIS officials attribute most of the net earned revenue to the agency’
s agreements with the Departments of Agriculture and Education and the 
Social Security Administration for various service offerings. Table 3 
identifies the net earned revenue or net cost for NTIS products and 
services over the last 11 fiscal years, as reported by NTIS. In those 
years in which NTIS had net costs, the agency was sustained by 
cumulative net earned revenues from previous years’ operations. 

Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS Products 
and Services, Fiscal Years 2001-2011: 

Fiscal year: 2001; 
Products: $398,233; 
Services: $1,772,011; 
Net earned revenues (net cost): $2,170,244. 

Fiscal year: 2002; 
Products: ($1,197,257); 
Services: $2,276,535; 
Net earned revenues (net cost): $1,079,278. 

Fiscal year: 2003; 
Products: ($1,409,026); 
Services: $1,685,945; 
Net earned revenues (net cost): $276,919. 

Fiscal year: 2004; 
Products: ($1,315,773); 
Services: $1,824,157; 
Net earned revenues (net cost): $508,384. 

Fiscal year: 2005; 
Products: ($1,973,646); 
Services: $2,465,149; 
Net earned revenues (net cost): $491,503. 

Fiscal year: 2006; 
Products: ($1,889,437); 
Services: $2,179,183; 
Net earned revenues (net cost): $289,746. 

Fiscal year: 2007; 
Products: ($1,683,794); 
Services: $834,741; 
Net earned revenues (net cost): ($849,053). 

Fiscal year: 2008; 
Products: ($2,072,519); 
Services: $768,982; 
Net earned revenues (net cost): ($1,303,537). 

Fiscal year: 2009; 
Products: ($2,217,263); 
Services: $1,099,377; 
Net earned revenues (net cost): ($1,117,886). 

Fiscal year: 2010; 
Products: ($730,347); 
Services: $3,041,803; 
Net earned revenues (net cost): $2,311,456. 

Fiscal year: 2011; 
Products: ($690,736); 
Services: $2,182,535; 
Net earned revenues (net cost): $1,491,799. 

Total; 
Products: ($14,781,565); 
Services: $20,130,418; 
Net earned revenues (net cost): $5,348,853. 

Source: GAO analysis of NTIS-reported data. 

[End of table] 

Figure 2 illustrates the net earned revenues and net costs associated 
with NTIS's products and services from fiscal year 2001 through fiscal 
year 2011. 

Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net Costs for 
NTIS Products and Services: 

[Refer to PDF for image: multiple line graph] 

Fiscal year: 2001; 
Products: $398,000; 
Services :$1,772,000. 

Fiscal year: 2002; 
Products: -$1,197,000; 
Services :$2,277,000. 

Fiscal year: 2003; 
Products: -$1,409,000; 
Services :$1,686,000. 

Fiscal year: 2004; 
Products: -$1,316,000; 
Services :$1,824,000. 

Fiscal year: 2005; 
Products: -$1,974,000; 
Services :$2,465,000. 

Fiscal year: 2006; 
Products: -$1,889,000; 
Services :$2,179,000. 

Fiscal year: 2007; 
Products: -$1,684,000; 
Services :$835,000. 

Fiscal year: 2008; 
Products: -$2,072,000; 
Services :$769,000. 

Fiscal year: 2009; 
Products: -$2,217,000; 
Services :$1,099,000. 

Fiscal year: 2010; 
Products: -$730,000; 
Services :$3,042,000. 

Fiscal year: 2011; 
Products: -$690,000; 
Services :$2,183,000. 

Source: GAO analysis based on NTIS provided data. 

[End of figure] 

The decline in revenue for its products continues to call into 
question whether NTIS's basic statutory function of acting as a self-
financing repository and disseminator of scientific and technical 
information is still viable. This is further highlighted by the fact 
that the services which are financially sustaining the agency are less 
directly related to this function. 

Recognizing its financial stance, NTIS has conducted analyses, and 
identified in its 2011-2016 Strategic Plan actions to help address net 
costs from its products, including its technical reports. The plan 
emphasizes that the agency's collection, culture, and information 
technology infrastructure are its main strengths; that continued use 
of less robust business systems and an aging work force are its 
primary weaknesses; that growth opportunities still exist in the 
various sectors served by NTIS, whether through products or services; 
and that NTIS is threatened by and will have to overcome a shrinking 
customer base for its products. The plan identifies three strategic 
initiatives to guide NTIS during this period: 

1. Increase revenue by enhancing the number of acquisitions, creating 
new products, reaching more customers, and adding value to what NTIS 
collects, and to reduce costs by reviewing and improving key business 
processes. 

2. Improve NTIS's utilization by other agencies by increasing the 
breadth and depth of its own collection and enhancing the suite of 
information management services that it can provide. 

3. Achieve workforce excellence by focusing on identifying and 
acquiring the critical workforce skills required to accomplish the 
agency's mission in a rapidly changing world. 

Beyond these initiatives identified in the strategic plan, the 
Director of NTIS also provided information on several other 
initiatives under way to address the budget shortfalls from products. 
These initiatives include the following product and organizational 
improvements: 

* Enhancing the accessibility of federal science content by shifting 
from a pricing model for stand-alone products (e.g., paper/print, 
microfiche, and compact disk media) to one that is subscription-based. 

* Repositioning NTIS to support open government initiatives in 
science--meeting with agencies such as NIST, the National Oceanic and 
Atmospheric Administration, and the Government Printing Office to 
address how NTIS can reposition its programs to support current 
science information needs. 

* Building collections of reports based on themes and categories that 
will be supported through subscriptions. 

* Adjusting the NTIS business model to support the increased demand 
for subscriptions to customers. 

* Reducing staff--for example, the agency has received authorization 
to provide early retirement for eligible employees and has stopped 
hiring additional staff. The agency anticipates employee attrition to 
further reduce current staffing levels. 

Notwithstanding these efforts, NTIS could likely continue to face 
challenges in recouping the costs of its products given the increasing 
availability of technical information from other sources. 

Additions to NTIS's Repository Have Mostly Included Older Reports, but 
Demand for More Recent Reports Is Greater: 

From fiscal year 1990 through 2011,[Footnote 31] most of the additions 
to NTIS's repository were older reports published in the year 2000 or 
before; however, the greater demand was for more recently published 
reports. In this regard, the agency added 841,502 reports to its 
repository from 1990 through 2011. Of the reports added, approximately 
62 percent, or 524,256, had publication dates of 2000 or earlier, 
while approximately 38 percent, or 317,246, of the reports were 
published from 2001 to 2011. Specifically, reports that were added to 
the repository during this period were as follows: 

* 79,943 reports published in 1989 and prior years, 

* 444,313 reports published from 1990 through 2000, 

* 129,591 reports published from 2001 through 2004, 

* 126,225 reports published from 2005 through 2008, and: 

* 61,430 reports published from 2009 through 2011. 

Figure 3 shows the distribution of reports that were added by each 
publication date age group from 1990 through 2011. 

Figure 3: Year of Publication for Documents Added to NTIS Repository, 
1990 through 2011: 

[Refer to PDF for image: pie-chart] 

Oldest publication date-1989: 9.5%; 
1990-2000 publication date: 52.8%; 
2001-2004 publication date: 15.4%; 
2005-2008 publication date: 15.0%; 
2009-2011 publication date: 7.3%. 

Source: GAO estimated based on NTIS provided data. 

[End of figure] 

With regard to demand for the reports, we estimate that, during fiscal 
years 2000 through 2011, NTIS distributed (sold) one or more copies of 
about 419,657, or almost 50 percent, of the 841,502 reports added to 
its repository during fiscal years 1990 through 2011.[Footnote 32] Of 
these 419,657 reports, approximately 78 percent were distributed 
through a subscription only.[Footnote 33] NTIS officials attributed 
this to the fact that subscriptions are a cost-effective way for 
libraries to meet their collection development requirements within a 
specific or broad area of interest. The officials noted that direct 
sales are generally to customers interested in specific topics. 

However, as shown in figure 4, the agency was more likely to 
distribute a higher percentage of more recently published reports than 
older ones. For example, we estimate that between 96 and 100 percent 
of the reports published from 2001 through 2011 had been distributed, 
while only about 21 percent of reports published in 1989 or earlier 
were distributed during this time period. That is, the demand for 
older holdings in the NTIS repository is lower than for newer 
publications. 

Figure 4: Estimated Percentage of Reports Distributed and Not 
Distributed by Publication Date during Fiscal Years 2000 through 2011: 

[Refer to PDF for image: vertical bar graph] 

Publication date range: 1989 or Earlier 
Distributed: 21%; 
Not distributed: 79%. 

Publication date range: 1990 - 2000; 
Distributed: 30%; 
Not distributed: 70.1%. 

Publication date range: 2001 - 2004; 
Distributed: 97.1%; 
Not distributed: 2.9%. 

Publication date range: 2005 - 2008; 
Distributed: 96.4%; 
Not distributed: 3.6%. 

Publication date range: 2009 - 2011; 
Distributed: 100%; 
Not distributed: 0%. 

Also depicted: 95% confidence level, upper and lower bound estimate 
for each date range. 

Source: GAO estimated based on NTIS provided data. 

Note: This figure presents estimates based upon our random sample of 
384 reports. The heights of the bars show the percentage point 
estimate for each category on the horizontal axis relative to the 
others. The line on each bar represents the 95 percent confidence 
interval associated with each estimate. Estimates are rounded to the 
nearest tenth of a percent, and may sum to greater than 100 percent 
due to rounding. See appendix I for an explanation of confidence 
intervals. 

[End of figure] 

Most Reports Added Since 1990 Are Freely Available from Other Websites: 

Based on our sample, we estimate that most (about 74 percent) of the 
reports added to NTIS's repository during fiscal years 1990 through 
2011 were readily available from other public websites, and nearly all 
of these (95 percent) could be obtained for free.[Footnote 34] 
Specifically, we estimate that approximately 621,917, or about 74 
percent, of the 841,502 reports added to NTIS's repository from fiscal 
years 1990 through 2011 are readily available from one of the other 
four publicly available sources we searched (i.e., the issuing 
organization's website; the Government Printing Office's Federal 
Digital System website; the U.S. government's official web portal, 
USA.gov; or from another website located through a Google search). 
[Footnote 35] The source that most often had the report we were 
searching for was another website located through [hyperlink, 
http://www.Google.com]. 

In addition, of the reports added since fiscal year 1990, those with 
more recent publication dates were more likely to be available from 
other public sources than older ones. For example, approximately 87 
percent of the reports with publication dates from 2009 to 2011 were 
available elsewhere, while 55 percent of those published in 1989 or 
earlier were.[Footnote 36] Figure 5 shows the estimated availability 
of reports added to NTIS's repository since fiscal year 1990 by date 
of publication. 

Figure 5: Estimated Availability of Reports by Year of Publication: 

[Refer to PDF for image: vertical bar graph] 

Year of publication: 1989 or earlier; 
Available at other sources: 55%; 
Not available at other sources: 45%. 

Year of publication: 1990-2000; 
Available at other sources: 70%; 
Not available at other sources: 30%. 

Year of publication: 2001-2004; 
Available at other sources: 87%; 
Not available at other sources: 13%. 

Year of publication: 2005-2008; 
Available at other sources: 91%; 
Not available at other sources: 9%. 

Year of publication: 2009-2011; 
Available at other sources: 87%; 
Not available at other sources: 13%. 

Also depicted: 95% confidence level, upper and lower bound estimate 
for each date range. 

Source: GAO estimated based on NTIS provided data. 

[End of figure] 

As shown in figure 6, of the reports that were found to be readily 
available from one of the other four sources that we searched, about 
61 percent of those reports had been distributed (sold) by NTIS. 
Conversely, of the reports that were not found to be readily available 
from one of the other four sources, most, or about 82 percent, had not 
been distributed by NTIS.[Footnote 37] 

Figure 6: Estimated Availability of Reports Distributed and Not 
Distributed During Fiscal Years 2000 through 2011: 

[Refer to PDF for image: vertical bar graph] 

Distributed: 
Available at other sources: 60.9%; 
Not available at other sources: 18.4%. 

Not distributed: 
Available at other sources: 39.1%; 
Not available at other sources: 81.6%. 

Also depicted: 95% confidence level, upper and lower bound estimate 
for each date range. 

Source: GAO estimated based on NTIS provided data. 

[End of figure] 

Not only were most of the reports in our sample available from sources 
other than NTIS, but about 95 percent of the reports available 
elsewhere could be obtained free of charge from one of the four other 
sources we searched.[Footnote 38] The remaining 5 percent were 
available from the public sources for a fee.[Footnote 39] Moreover, 
the year of publication did not appear to have an effect on whether a 
report was available free of charge. For example, the following 
reports available for a fee from NTIS were available free of charge 
from the issuing organization's website: 

Hazardous Waste Characteristics Scoping Study, November 1996, 
Environmental Protection Agency, 278 pages. (At NTIS, print on demand 
costs $73, electronic $25): 

"Homeland Security: Intelligence Indications and Warning," December 
2002, Naval Postgraduate School, 5 pages. (At NTIS, print on demand 
costs $17, electronic $15): 

Export Controls: System for Controlling Exports of High Performance 
Computing Is Ineffective, 2000, GAO, 60 pages. (At NTIS, print on 
demand costs $48, electronic $15): 

FDA Enforcement Report: July 20, 2011, July 2011, Food and Drug 
Administration, 28 pages. (At NTIS, print on demand costs $33, 
electronic $15): 

Principal Rare Earth Elements Deposits of the United States: A Summary 
of Domestic Deposits and a Global Perspective, 2010, Geological 
Survey, 104 pages. (At NTIS, print on demand costs $60, electronic 
$25): 

Of those reports we found available elsewhere, figure 7 shows the 
estimated percent that were available elsewhere for free by year that 
the document was published. 

Figure 7: Of Reports Found Elsewhere, Estimated Percent Available for 
Free and Not Free, by Year of Publication: 

[Refer to PDF for image: vertical bar graph] 

Year of publication: 1989 or earlier; 
Free: 98%; 
Not free: 2%. 

Year of publication: 1990-2000; 
Free: 94%; 
Not free: 6%. 

Year of publication: 2001-2004; 
Free: 93%; 
Not free: 7%. 

Year of publication: 2005-2008; 
Free: 96%; 
Not free: 4%. 

Year of publication: 2009-2011; 
Free: 98%; 
Not free: 2%. 

Also depicted: 95% confidence level, upper and lower bound estimate 
for each date range. 

Source: GAO estimated based on NTIS provided data. 

Note: These estimates are for reports readily available from one of 
the other four publicly available sources that we searched. 

[End of figure] 

The Director of NTIS acknowledged two factors that contributed to the 
free and public availability of reports from other sources: 

* Federal agencies are providing information, including their federal 
scientific, technical, and engineering information products, on their 
websites in electronic format and on central federal information 
websites such as [hyperlink, http://www.data.gov] and [hyperlink, 
http://www.science.gov]. The information is available for immediate 
download at no cost. 

* Federal agencies are participating in programs with Internet search 
engines that permit the public to locate their products for free or 
for less than they are when purchased from NTIS. In addition, 
commercial vendors are also able to obtain these information products 
from agency websites or through Internet search and are able to make 
these products available for free or at a price lower than that 
offered by NTIS. 

Further, NTIS acknowledged in its strategic plan that because the 
Internet continues to change the way people acquire and use 
information and permits federal agencies to make their information 
products available for free, NTIS is challenged to meet its statutory 
mandate as a self-financing repository and disseminator of technical 
information. Notwithstanding these acknowledgments, NTIS continues to 
charge for reports that are freely available from other public sources. 

Conclusions: 

NTIS serves as a permanent repository and disseminator of technical 
information, and by statute, is required to be financially self-
sustaining, to the fullest extent feasible, by charging fees for its 
products and services. While the agency had cumulative net earned 
revenues as of September 30, 2011, its costs exceeded revenue by an 
average of about $1.3 million over the last 11 years from the sale of 
technical information. The agency's net revenue now comes primarily 
from services that are less directly related to its basic statutory 
function. The decline in sales of technical information is due in part 
to the increasing availability of this information from other sources, 
including websites and Internet search tools, and often at no charge. 
Charging for information that is freely available elsewhere is a 
disservice to the public and may also be wasteful insofar as some of 
NTIS's customers are other federal agencies. Taken together, these 
considerations suggest that the fee-based model under which NTIS 
currently operates for disseminating technical information may no 
longer be viable or appropriate. 

Matter for Congressional Consideration: 

In light of the agency's declining revenue associated with its basic 
statutory function and the charging for information that is often 
freely available elsewhere, Congress should consider examining the 
appropriateness and viability of the fee-based model under which NTIS 
currently operates for disseminating technical information to 
determine whether the use of this model should be continued. 

Agency Comments and Our Evaluation: 

The Acting Secretary of Commerce provided written comments on a draft 
of this report, which are reprinted in appendix II. In its comments, 
Commerce expressed appreciation for our study and for our focus on the 
initiatives that NTIS has undertaken. However, the department said 
NTIS did not believe our conclusions (that the fee-based model under 
which it operates for disseminating technical information may no 
longer be viable or appropriate) fully reflect the additional value 
that NTIS provides with the work that it performs. Commerce stated 
that, through its federal clearinghouse and repository, the agency 
provides federally funded reports that are not otherwise readily 
available, such as most of those issued prior to l989. Additionally, 
Commerce stated that NTIS recognizes that it cannot remain financially 
solvent solely through sales and subscriptions of technical reports 
with expectations that these products will be widely available for 
free. It added that NTIS is moving in the direction of "open and 
improved access" to information, but recognizes that it needs to 
maintain a sustainable financial model and continue providing enhanced 
value to the information generated by other federal agencies. In this 
regard, Commerce described features that it believes have added value 
to the technical reports that NTIS maintains, how these features 
improve access to the documents and related data, and specific 
information services that the agency provides to science information 
professionals. 

Our report highlighted various initiatives that NTIS has undertaken to 
provide older reports that might not otherwise be readily available, 
and to increase the value of its technical reports, information 
management services, and technology transfer capabilities. However, as 
discussed in the report, we found that the demand for older holdings 
in the agency's repository is lower than for new publications. For 
example, we estimate that between 96 and 100 percent of the reports 
published from 2001 through 2011 had been distributed, while only 
about 21 percent of reports published in l989 or earlier were 
distributed during this time period. Further, as the agency 
acknowledged, its financial health is based on both its information 
product and service missions. Also, as we state in our report, the 
agency's net revenue now comes primarily from services that are less 
directly related to its basic statutory function, while sales of its 
technical information products have resulted in net losses. This 
decline in sales of NTIS's technical reports is due in part to the 
increasing availability of this information from other sources, 
including websites and Internet search tools, and often at no charge. 
With these factors in mind, we stand by our conclusion that the fee-
based model under which NTIS currently operates for disseminating 
technical information may no longer be viable or appropriate. 

NTIS also provided technical comments on the report via e-mail, which 
we have incorporated as appropriate. 

We are sending copies of this report to interested congressional 
committees. We are also sending copies to the Secretary of Commerce; 
the Director, NTIS; the Director, NIST; and other interested parties. 
Copies of this report will also be available at no charge on GAO's 
website at [hyperlink, http://www.gao.gov]. 

Should you or your staffs have any questions on information discussed 
in this report, please contact me at (202) 512-6304 or 
melvinv@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. GAO staff who made major contributions to this report are 
listed in appendix III. 

Signed by: 

Valerie C. Melvin: 
Director: 
Information Management and Technology Resources Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine (1) how the National Technical 
Information Service (NTIS) is currently organized and operates, 
including its various functions, current staffing level, reported cost 
of operations, and revenue sources; (2) the age of and demand trends 
for reports added to NTIS's repository; and (3) the extent to which 
these reports are readily available from other public sources. 

To determine how NTIS is organized and operates, we reviewed the 
agency's strategic plan, and documentation on its organizational and 
reporting structure, office staffing level and assigned 
responsibilities, and types of products and services offered. We also 
reviewed cost data contained in the agency's annual financial reports. 
[Footnote 40] In addition, we reviewed relevant laws and regulations 
on NTIS's authority and responsibilities[Footnote 41] and our previous 
reports that discussed its mission and operations.[Footnote 42] We 
supplemented our analyses with interviews of the Director of NTIS and 
other relevant agency officials; we also interviewed officials of the 
Department of Commerce and its National Institute of Standards and 
Technology (NIST), which have specific reporting relationships with 
NTIS. 

For our second and third objectives, we obtained from NTIS the full 
list of document accession numbers for reports added to its repository 
since fiscal year 1990. We chose to include reports added since fiscal 
year 1990 to determine the current availability of older documents in 
light of NTIS's role as a permanent repository, and to focus on the 
reports added since our previous review in 2001.[Footnote 43] We then 
stratified the list of document accession numbers into four strata by 
the year in which a document was added to the repository. We 
subsequently selected a stratified random sample with 96 document 
accession numbers per strata, for a total sample size of 384 reports. 
All of the estimates made with this sample are weighted to reflect the 
stratified design. NTIS provided us with the full bibliographic data 
for each document in our sample. 

To determine the age of reports added to NTIS's repository since 
fiscal year 1990, we used the year of publication for the reports in 
our sample to estimate the age range (prior to 1990; 1990-2000; 2001-
2004; 2005-2008; and 2009-2011) for all documents added from fiscal 
year 1990 through fiscal year 2011. To determine the demand trends for 
reports added to NTIS's repository during fiscal years 1990 through 
2011, we requested the sales data from fiscal year 2000 through fiscal 
year 2011 for the 384 reports in our stratified sample. NTIS provided 
the distribution data for both direct sales and subscriptions for all 
of these documents. We then used our sample and the sales data to 
estimate the extent and prevalence of the sales among all reports 
added to the NTIS repository. Specifically, we used these data to 
estimate the (1) total number of reports distributed by direct sale 
and subscription, (2) total number of reports distributed one or more 
times, (3) percentage of reports distributed relative to the year the 
reports were published, and (4) percentage of reports distributed 
relative to their availability elsewhere. The observations were 
statistically weighted in the estimation process to reflect the 
stratified sample design that we used. 

To determine the extent to which reports in the repository are readily 
available from other public sources, we first developed a methodology 
for conducting systematic Internet searches to determine availability 
elsewhere. More specifically, as part of this methodology, we searched 
the Internet to determine if each of the reports included in our 
sample of 384 reports could be found elsewhere and at no cost. Using a 
tiered approach, we searched the following four sources in the order 
shown: (1) the issuing organization's website; (2) the U.S. Government 
Printing Office's Federal Digital System website--[hyperlink, 
http://www.gpo.gov/fdsys]; (3) a web search conducted using the 
federal government Internet portal USA.gov--[hyperlink, 
http://www.USA.gov]; and (4) a web search conducted using the 
commercial search engine [hyperlink, http://www.Google.com].[Footnote 
44] Specifically, with this methodology, we determined whether each 
report was first available at no cost on the issuing organization's 
website and, if so, concluded the Internet search at this point. 
However, if the report was not available, then the search continued to 
the second source, and so on, until either the report was found to be 
available at one of the remaining sources, or all sources were 
exhausted.[Footnote 45] We then used our results to estimate the 
percentage of the total population of NTIS reports added to the 
repository during fiscal years 1990 through 2011 that was available 
from other public sources. 

All of the results derived from the sample analyses constituted 
estimates that are subject to sampling errors. These sampling errors 
measure the extent to which samples of this size and structure are 
likely to differ from the population they represent. Because we 
followed a probability procedure based on random selections, our 
sample is only one of a large number of samples that we might have 
drawn. Since each sample could have provided different estimates, we 
express our confidence in the precision of our particular sample's 
results as a 95 percent confidence interval. This is the interval that 
would contain the actual population value for 95 percent of the 
samples we could have drawn. 

To determine the reliability of the data provided from NTIS's 
repository of reports, we performed basic steps to ensure the data 
provided were valid, and reviewed relevant information describing the 
database supporting the repository. We tested for duplicate records, 
missing values, and out-of-range values in the data received from 
NTIS. We did not assess the reliability of the system used to maintain 
these data or the processes used in extracting the data for our 
engagement purposes. To determine the reliability of the sales data 
provided by NTIS, we conducted interviews with agency officials to 
gain an understanding of the process by which accounts receivable 
records are added and managed within NTIS's system of accounts 
receivable-"CIS.PUB." Further, we asked cognizant agency officials 
specific questions to understand the controls in place for ensuring 
the integrity and reliability of the data contained in CIS.PUB. In 
addition, we met with NTIS officials to discuss data collected from 
NTIS and obtained their assertions regarding the data it provided. 
Based on the results of these efforts, we found the data sources to be 
sufficiently reliable, given the way they are reported herein. 

We conducted this performance audit from February 2012 to November 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the Department of Commerce: 

United States Department of Commerce: 
The Secretary of Commerce: 
Washington, D.C. 20230: 

November 8, 2012: 

Valerie C. Melvin: 
Director, Information Management and Technology Resources Issues: 
U.S. Government Accountability Office: 
441 G. Street, NW: 
Washington, DC 20548:  

Dear Ms. Melvin:  

Thank you for the opportunity to review and comment on the U.S. 
Government Accountability Office's (GAO) draft report entitled 
"Information Management: National Technical Information Service's 
Dissemination of Technical Reports Needs Congressional Attention" (GAO-
13-99).  

We appreciate the work that the GAO has done to update the prior 2001 
GAO Report on the National Technical Information Service (NTIS). We 
are pleased that your team recognizes that NTIS successfully performs 
their statutory missions and responsibilities while remaining 
financially self-sustaining, The GAO team focused on current 
initiatives that NTIS has undertaken to increase the value of its 
technical reports, information management services, and technology 
transfer capabilities. NTIS executed these enhancements while 
containing costs and remaining financially self-sustaining.  

NTIS recognizes that it cannot remain financially solvent solely 
through sales and subscriptions of technical reports with expectations 
that these products be widely available for free. NTIS is moving in 
the direction of "open and improved access" but we also recognize that 
NTIS needs to maintain a sustainable financial model and to continue 
providing enhanced value to the information generated by other federal 
agencies. While "open and improved access" is one goal, the challenge 
for science information management is more than free and open access to 
documents. NTIS has been an active agent for the transition and 
expansive access to technical reports and positioned for data linking. 
NTIS has invested in information technologies through NTIS service 
offerings to federal clients and with the Federal Science Repository 
Service.  

If you have any questions regarding our response, please contact Bruce 
Borzino, Director of the National Technical Information Service, at 
(703) 605-6405.  

Sincerely,  

Signed by: 

Rebecca M. Blank: 
Acting Secretary of Commerce:  

Enclosure: 

Department of Commerce: 
National Technical information Service: 
Comments to the Draft GAO Report Entitled: "Information Management: 
National Technical Information Service's Dissemination of Technical 
Reports Needs Congressional Attention" (GAO-13-99, November 2012): 

General Comments: 

Thank you for the opportunity to review and comment on your draft 
report regarding the National Technical Information Service (NTIS). We 
appreciate the work of your staff and thank them for their diligence. 
Your team recognized that NTIS successfully performs its statutory 
missions and responsibilities while remaining financially self-
sustaining. NTIS was pleased that during this review, the GAO team 
focused on current initiatives that NTIS has undertaken to increase 
the value of its technical reports, information management services, 
and technology transfer capabilities. We appreciate that GAO 
recognizes NTIS has executed enhancements while containing costs and 
remaining financially self-sustaining. NTIS acknowledges that its 
financial health is based on both the information product mission that 
GAO reviewed as well the information service mission that GAO did not 
review during this engagement. When viewed in combination, NTIS 
continues to remain financially healthy as a fee based organization. 
NTIS acknowledges that recommendations to provide value-added 
information products, such as subscriptions to technical reports, 
cannot be considered or addressed in isolation from the overall 
mission and mandates. 

NTIS Response to GAO Conclusion: 

The draft GAO report states, "Charging for information that is freely 
available elsewhere is a disservice to the public and may be wasteful 
insofar as some NTIS customers are other federal agencies. Taken 
together, these considerations suggest that the fee-based model under 
which NTIS currently operates for disseminating technical information 
may no longer be viable or appropriate." We do not believe the 
conclusion fully reflects the additional value that NTIS provides with 
the additional work NTIS performs. 

NTIS provides federally funded reports that are not otherwise readily 
available: 
NTIS research has determined that most federally originated technical 
reports prior to 1989 do not have assured availability from the source 
agency, other federal public sources, or from generally available 
internet sources. However, these reports are currently available to 
the public, industry, and academia because NTIS operates a permanent 
federal clearinghouse and repository containing authenticated agency 
technical reports and associated value-added bibliographic meta-data 
for over 2.5 million federal publications. 

In addition to making the reports available, NTIS has added value to 
the reports: 
NTIS is moving in the direction of "open and improved access", but we 
are also mindful of the need to be responsible stewards of science 
information particularly as such efforts impact national efforts for 
economic growth. NTIS needs to maintain a sustainable financial model 
in information management in an era of reduced federal spending. NTIS 
sees open information access as a foundation for new business inspired 
growth using open source technologies, agile development of 
information services valued by federal clients, and continuous 
improvement of information services to the federal community. 

NTIS is surviving in the "open" information environment by providing a 
value-added service which consists of the following process: 

* Acquisition of new documents in two formats: (1) the document file, 
and (2) the meta-data for each document. 

* Conversion to searchable text and that text is added to the meta-
data, resulting in a user friendly aggregate scientific, technical, 
and engineering information (STEI) database of federally funded 
research. 

* Quality control for content integrity that is reviewed, abstracted, 
and indexed by a team of information specialists. 

* The addition of a new set of data points is added to the meta-data 
of each document, e.g. data points include keywords, subject 
categories, related subject categories and more. 

The improved access to the documents and related data results in: 

* Facilitating greater discoverability of federally funded research 
meta-data that is indexed and organized. 

* Ensuring availability to the academia, corporate and the public via 
the NTIS website or through one of NTIS's subscription-based access 
sites. 

* The document, meta-data, and the index each becomes a part of the 
NTIS Collection which greatly increases the qualitative 
discoverability for research and technology transfer endeavors that 
benefits the public. 

The National Technical Reports Library...a valued service to science 
information professionals: 

To increase the value and availability of those reports, NTIS launched 
the National Technical Reports Library (NTRL) in 2009. NTRL is a 
collection of all NTIS electronic technical reports available on a 
subscription basis at the institutional level, such as a federal 
agency or university. NTRL currently has 70 institutional subscribers 
providing access to 1.6 million users. Subscribers have repeatedly 
reported the value of the NTRL, is in delivering specific information 
to the subscriber. This value added is created by NTIS staff and is in 
addition to the content of the underlying technical reports. NTIS 
continues to work closely with federal and public scientific 
communities to seek innovative solutions to meet general and 
scientific information needs. 

Representative Sample of NTRL Subscribers: 

* U.S. Department of Interior: 69,000 I.P. User License. 

* Emory Riddle Aeronautical University: 8,800 I.P. User License. 

* U.S. EPA National Library Network: 5,000 I.P. User License. 

* Princeton University: 7,600 I.P. User License. 

* Stanford University: 8,775 I.P. User License. 

* University of California (multiple sites): 125,000 I.P. User License. 

The Federal Science Repository Service...a valued service to federal 
science agencies: 

As noted in the GAO Report, NTIS is effectively serving a wide variety 
of federal clients in innovative solutions to general and scientific 
information needs. These information needs have been transformational 
in positioning NTIS organization skill sets on basic needs such as 
digitization, repository development, and website management to very 
advanced information technologies. These technologies utilize open 
source solutions for enhancing science information management options. 
Additionally, Federal Science Repository Service (FSRS) establishes a 
gateway for potential links to collaborative productivity, innovative 
knowledge management solutions, and interoperable data and records 
management. 

In 2011, NTIS launched the FSRS to federal clients. The FSRS is based 
on the same open source repository technology used in the NTRL and 
gives federal data and records managers a path forward in distributed 
information technology solutions. NTIS is currently working with the 
U.S. State Department and the National Oceanic and Atmospheric 
Administration (NOAA) in FSRS projects. 

NTIS sees that the services provided by FSRS are needed to continue 
growing information services in a federal culture of efficiency 
innovation. The GAO notes that the end result of NTIS information 
services is to assist financial support to the core STEI mission. It 
also needs to be recognized that the technical skill sets acquired for 
these services benefit the federal customers as well providing a 
future investment in technical skills sets needed for a variety of 
information challenges going forward. 

Moving forward in the Science Information Service: 
NTIS provides innovative value and solutions to the evolution of 
information management: 

* Value of aggregation of science content utilizing core indexing 
across source platforms. 

* Value of permanent and persistent access with a solid link to a 
National Archives and
Records Administration (NARA) archival solution. 

* Discovery of science information that goes beyond PDFs and Technical 
Reports, linking to data and records management. 

* Embracing the emerging "economy of data" in all science. 

* Science information that is not easily available elsewhere via 
"Digital-on-Demand." 

* Private-public cooperation in advancing the science information. 

* Metrics focused on the effectiveness of federal research dollars 
invested. 

* Science information in secure environments. 

NTIS...Working in a Science Networked Environment" 
A core function of NTIS is to act as a clearinghouse and repository 
for technical information, such as reports and databases, from across 
the Federal Government. In furtherance of this core function, NTIS 
continues to work closely with federal and public scientific 
communities to seek innovative solutions to meet general and 
scientific information needs. While science information management has 
changed dramatically in the past decade, NTIS has been an active agent 
for the transition and expansive access to technical reports and 
positioned for data linking. NTIS has invested in information 
technologies through NTIS service offerings to federal clients and 
with the Federal Science Repository Service. NTIS needs to continue 
with the central index to science content while utilizing the advanced 
features of networked science. 

Statement Summary: 
The challenge for science information management is more than free and 
open access to documents. The primary challenge is to further the 
discovery of science content that results in broader use of reports, 
data, and related linked materials resulting in more efficient 
technology transfer, economic growth, and societal benefits. 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Cynthia Scott (Assistant 
Director), Carl Barden, Virginia Chanley, Elena Epps, Nancy Glover, 
Alina J. Johnson, Lee McCracken, Constantine Papanastasiou, David 
Plocher, Bradley Roach, and Tina Torabi made key contributions to this 
report. 

[End of section] 

Footnotes: 

[1] In the remainder of this report, we use the term "technical 
information" to refer collectively to these types of information. 

[2] These 2.5 million records contain, among other things, the 
metadata that identify all of the reports held by NTIS, such as the 
report's author, publication date, and the agency where the report 
originated. These records link to the actual reports, which NTIS 
maintains elsewhere in hardcopy or as electronic documents or 
microfiche. According to the agency, these records represent all 
reports that have been catalogued in the electronic repository since 
1964. As part of our further discussion in this report, we use the 
term "reports" in referring to the repository holdings. 

[3] 15 U.S.C. 1153. 

[4] GAO, Information Management: Dissemination of Technical Reports, 
[hyperlink, http://www.gao.gov/products/GAO-01-490] (Washington, D.C.: 
May 18, 2001). 

[5] Pub. L. No. 112-55 (Nov. 18, 2011). 

[6] The cost data reflected in our report were taken from 
documentation provided by NTIS. However, we did not test the 
reliability of NTIS's cost accounting system, from which the agency 
derives its cost data. 

[7] For example, 15 U.S.C.1151-57, 15 U.S.C. 3704b-1 & 2, and 15 
C.F.R. Part 1180. 

[8] GAO, Information Policy: NTIS' Financial Position Provides an 
Opportunity to Reassess Its Mission, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-00-147] (Washington, D.C.: June 
30, 2000) and [hyperlink, http://www.gao.gov/products/GAO-01-490]. 

[9] We did not include reports added for fiscal year 2012 because our 
study only focused on those fiscal years that had been completed at 
the time that our study was initiated. 

[10] Our selected sample consisted of 384 reports. 

[11] 15 U.S.C. 1152. 

[12] In 1954, the Comptroller General found this to be a reasonable 
interpretation of the law's use of the word "technical." 34 Comp. Gen. 
58 (1954). 

[13] 15 U.S.C. 1153. 

[14] Pub. L. No. 100-519 (Oct. 24, 1988); 15 U.S.C. 3704b. 

[15] Pub. L. No. 102-245 (Feb. 14, 1992); 15 U.S.C. 3704b-1. 

[16] Pub. L. No. 102-395 (Oct. 6, 1992). 

[17] In the remainder of this report, we use the term "net costs" to 
refer to financial losses or gross costs that exceeded revenues. 

[18] [hyperlink, http://www.gao.gov/products/GAO/GGD-00-147]. 

[19] [hyperlink, http://www.gao.gov/products/GAO-01-490]. 

[20] NIST is a Commerce operating unit that, among other things, 
promotes U.S. innovation and industrial competitiveness by advancing 
measurements for science, standards, and technology. 

[21] Department of Commerce Departmental Organization Order 30-7A 
(Washington, D.C.: November 2008). 

[22] 15 U.S.C. 3704b(c). 

[23] NTIS Advisory Board members can include individuals who are both 
internal and external to the Department of Commerce. 

[24] The Merriam-Webster Dictionary defines PDF as "a computer file 
format for the transmission of a multimedia document that is not 
intended to be edited further and appears unaltered in most computer 
environments." Merriam-Webster Online, accessed October 5, 2012, 
[hyperlink, http://www.merriam-webster.com/dictionary/pdf]. 

[25] 15 U.S.C. 3704b(e)(5). 

[26] Cloud computing is an emerging form of computing that relies on 
Internet-based services and resources to provide computing services to 
customers. 

[27] According to NIST, infrastructure-as-a-service is a model where 
the service provider delivers and manages the basic computing 
infrastructure of servers, software, storage, and network equipment on 
which a platform to develop and execute applications can be developed 
by the consumer. Software-as-a-service is a model where the service 
provider delivers one or more applications and the computational 
resources and underlying infrastructure to run them for use on demand 
as a turnkey service. 

[28] "Jive" is a social networking product for enterprise 
implementation. See its website: [hyperlink, 
http://www.jivesoftware.com]. More specifically, it intends to help 
employees, customers, and partners to connect, communicate, and 
collaborate. 

[29] The other six primary clients are the Defense Manpower Data 
Center, Department of Commerce, Wage Determination Online, U.S. 
Commission on Civil Rights, Joint Task Force National Capital Region, 
and National Telecommunications and Information Administration. 

[30] Section 508 of the Rehabilitation Act of l973 requires federal 
agencies to ensure that their development, procurement, maintenance, 
or use of electronic and IT equipment takes into account the need for 
people with disabilities to have access to and use of information that 
is comparable to that of individuals without disabilities. 

[31] We did not include reports added for fiscal year 2012 because our 
study only focused on those fiscal years that had been completed at 
the time that our study was initiated. 

[32] The 95 percent confidence interval for the estimated percent of 
reports distributed one or more times is (44.9, 54.8) percentage 
points. For purposes of this report, demand is defined as the 
distribution (sale) of a report by direct sale or subscription. 

[33] The 95 percent confidence interval for the estimated percent of 
reports distributed through a subscription only is (69.7, 84.4) 
percentage points. 

[34] The 95 percent confidence interval for the estimated percentage 
of reports available elsewhere which could be obtained for free is 
(90.7, 97.5) percentage points. 

[35] The 95 percent confidence interval for the estimated percentage 
of reports available through one or more of the four publicly 
available sources we searched is (67.9, 80.0) percentage points. 

[36] For reports published between 2009 and 2011, we estimate that 
86.5 percent are available elsewhere. The 95 percent confidence 
interval for this estimate is (74.2, 93.5) percentage points. For 
reports published in 1989 or earlier, we estimate that 55 percent are 
available elsewhere. The 95 percent confidence interval for this 
estimate is (34.1, 74.3) percentage points. 

[37] The 95 percent confidence interval for the estimated percent 
available reports that were distributed is (54.0, 67.4) percentage 
points. The 95 percent confidence interval for the estimated percent 
of reports not available and not distributed is (73.2, 87.8) 
percentage points. 

[38] The 95 percent confidence interval for this estimate is (90.7, 
97.5) percentage points. 

[39] The 95 percent confidence interval for this estimate is (2.5, 
9.3) percentage points. 

[40] The cost data reflected in our report were taken from 
documentation provided by NTIS. However, we did not test the 
reliability of NTIS's cost accounting system, from which the agency 
derives its cost data. 

[41] For example 15 U.S.C. 1151-57, 15 U.S.C. 3704b-1 & 2, and 15 
C.F.R. Part 1180. 

[42] GAO, Information Policy: NTIS' Financial Position Provides an 
Opportunity to Reassess Its Mission, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-00-147] (Washington, D.C.: June 
30, 2000) and Information Management: Dissemination of Technical 
Reports, [hyperlink, http://www.gao.gov/products/GAO-01-490] 
(Washington, D.C.: May 18, 2001). 

[43] We did not include reports added for fiscal year 2012 because our 
study only focused on those fiscal years that had been completed at 
the time that our study was initiated. 

[44] These sources and the order in which they were searched were 
based on an effort to duplicate the methodology used in our previous 
2001 study of NTIS. 

[45] In identifying the reports' availability elsewhere, we did not 
assess if the source at which the report was found was the original 
issuer of the report or whether the report's content was unaltered 
from its original issuance. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]