This is the accessible text file for GAO report number GAO-02-195 
entitled 'NASA: Better Mechanisms Needed for Sharing Lessons Learned' 
which was released on January 30, 2002. 

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the 
printed version. The portable document format (PDF) file is an exact 
electronic replica of the printed version. We welcome your feedback. 
Please E-mail your comments regarding the contents or accessibility 
features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States General Accounting Office: 
GAO: 

Report to the Subcommittee on Space and Aeronautics, Committee on 
Science, House of Representatives: 

January 2002: 

NASA: 

Better Mechanisms Needed for Sharing Lessons Learned: 

GAO-02-195: 

Contents: 

Letter: 

Executive Summary: 

Purpose: 

Background: 

Results in Brief: 

Principal Findings: 

Recommendations: 

Agency Comments and Our Evaluation: 

Chapter: 

Introduction: 

Faster, Better, Cheaper: A Management Philosophy: 

Mars Surveyor Program: 

Mishap Investigations Raised Concern That NASA Did Not Heed Lessons 
Learned: 

NASA Efforts to Apply Lessons Learned from Mars Mishaps: 

Lessons Learning: A Mechanism to Learn from Successes As Well As 
Mistakes: 

Objectives, Scope, and Methodology: 

Chapter 2: 

NASA Policies, Processes, and Systems for Lessons Learning and 
Knowledge Sharing: 

NASA’s Policies and Procedures Require Lessons Learning: 

NASA Uses Numerous Methods for Lessons Learning: 

Additional Steps Taken by NASA to Facilitate Lessons Learning: 

Chapter 3: 

Lessons Learned Are Not Being Consistently Shared or Applied Toward 
Future Program Success: 

Fundamental Weaknesses Exist in NASA’s Processes, Procedures, and 
Systems for Lesson Learning: 

Chapter 4: 

Creating an Environment for Lessons Learning through Knowledge 
Management: 

Principles of Knowledge Management Could Help Mitigate Fundamental 
Weaknesses in Lessons Learning: 

Chapter 5: 

Conclusions and Recommendations: 

Recommendations for Executive Action 

Agency Comments and Our Evaluation: 

Appendix I: Comments from the National Aeronautics and Space 
Administration: 

Appendix II: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Acknowledgments: 

Tables: 

Table 1: Reasons for Spacecraft Failures: 

Table 2: NASA’s Framework for Knowledge Management: 

Table 3: Program and Project Manager Observations and Suggestions to 
Improve LLIS: 

Figures: 

Figure 1: Generic Lessons Learned Process: 

Figure 2: Program and Project Managers Knowledge of Lessons Learned: 

Figure 3: Many Program and Project Managers Have Not Submitted Lessons 
to the Lessons Learned Information System: 

Figure 4: Most Program and Project Managers Could Not Identify Helpful 
Lessons for Their Program or Project: 

[End of section] 

United States General Accounting Office: 
Washington, DC 20548: 

January 30, 2002: 

The Honorable Dana Rohrabacher: 
Chairman: 
Subcommittee on Space and Aeronautics: 
Committee on Science: 
House of Representatives: 

The Honorable Bart Gordon: 
Ranking Minority Member: 
Subcommittee on Space and Aeronautics: 
Committee on Science: 
House of Representatives: 

In response to your request, this report discusses the mechanisms that 
the National Aeronautics and Space Administration (NASA) has in place 
to capture, disseminate, and apply past lessons learned towards future 
mission success. We found that NASA’s processes, procedures, and 
systems do not effectively capture and share lessons learned and 
therefore, NASA has no assurance that lessons are being applied 
towards future missions. We include recommendations to the NASA 
administrator on ways to strengthen the agency’s lessons learning 
processes and systems. 

As agreed with your office, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 3 days 
from its issue date. At that time we will send copies of this report 
to the NASA administrator and interested congressional committees. We 
will also provide copies to others on request. If you or your staffs 
have any questions regarding this report, please contact me at (202) 
512-4841, or John Oppenheim, assistant director, at (202) 512-3111. 
Key contributors are listed in appendix II. 

Signed by: 

Allen Li:
Director:
Acquisition and Sourcing Management: 

[End of section] 

Executive Summary: 

Purpose: 

In the early 1990s, the National Aeronautics and Space Administration 
(NASA) administrator challenged the agency to complete projects 
faster, better, and cheaper. The intent was to reduce costs, become 
more efficient, and increase scientific results by conducting more and 
smaller missions in less time. Although NASA maintained a high success 
rate under the faster, better, and cheaper strategy, a few significant 
mission failures also occurred—particularly the loss of the Mars Polar 
Lander and Climate Orbiter spacecraft. NASA investigations of these 
failures, as well as its review of other programs, raised concern that 
lessons from past experiences were not being applied to current 
programs and projects. 

At the request of the Chairman and Ranking Minority Member, 
Subcommittee on Space and Aeronautics, House Committee on Science, GAO 
assessed whether NASA has adequate mechanisms in place to ensure that 
past lessons learned from mission failures are being applied. 
Specifically, GAO (1) identified the policies, procedures, and systems 
NASA has in place for lessons learning, (2) assessed how effectively 
these policies, procedures, and systems facilitate lessons learning, 
and (3) determined whether further efforts are needed to improve 
lessons learning. 

Background: 

NASA’s procedures and guidelines require that program and project 
managers review and apply lessons learned from the past throughout a 
program’s or project’s life cycle and to document and submit any 
significant lessons to the agency’s Lessons Learned Information System 
(LLIS) in a timely manner. NASA defines a lesson learned as knowledge 
or understanding gained by experience. The experience may be positive, 
such as a successful test or mission, or negative, such as a mishap or 
failure. A lesson must be significant in that it has a real or assumed 
impact on operations; valid in that it is factually correct; and 
applicable in that it identifies a specific design, process, or 
decision that reduces or eliminates the potential for failures and 
mishaps, or reinforces a positive result. 

Results in Brief: 

NASA recognizes the importance of learning from the past to ensure 
future mission success and uses several mechanisms to capture and 
disseminate lessons learned. The principal source NASA has established 
for the agency-wide collection and sharing of lessons is the LLIS, a 
Web-based lessons database that managers are required to review on an 
ongoing basis. In addition, NASA uses training, program reviews, and 
periodic revisions to agency policies and guidelines to communicate 
lessons. Several NASA centers and key programs also maintain lessons 
learned systems that are geared toward their own staff. Recently, NASA 
has taken steps to improve the way it captures and shares information 
by developing a business strategy called knowledge management. 
Knowledge management can be defined as the way that organizations 
create, capture, and reuse knowledge to achieve their objectives. 
According to NASA officials, knowledge management has the potential to 
link agency staff with the knowledge and resources they need to 
complete tasks faster, better, and cheaper. In pursuit of knowledge 
management, NASA has developed a strategic plan, established a 
management team to coordinate knowledge management activities at 
NASA’s centers, and initiated several information technology pilot 
projects. 

Despite the processes and procedures in place to capture and share 
lessons learned, there is no assurance that lessons are being applied 
toward future missions success. A GAO survey of NASA program and 
project managers revealed weaknesses in the collection and sharing of 
lessons learned agency-wide. While some lessons learning does take 
place, our survey found that lessons are not routinely identified, 
collected, or shared by programs and project managers. Respondents 
reported that they are unfamiliar with lessons generated by other 
centers and programs. In addition, many respondents indicated that 
they are dissatisfied with NASA’s lessons learned processes and 
systems. Managers also identified challenges or cultural barriers to 
the sharing of lessons learned, such as the lack of time to capture or 
submit lessons and a perception of intolerance for mistakes. They 
further offered suggestions for areas of improvement, including 
enhancements to LLIS and implementing mentoring and “storytelling,” or 
after-action reviews, as additional mechanisms for lessons learning. 

While NASA’s current knowledge management efforts should lead to some 
improvement in the sharing of agency lessons and knowledge, they lack 
ingredients that have been shown to be critical to the success of 
knowledge management at leading organizations. Cultural resistance to 
sharing knowledge and the lack of strong support from agency leaders 
often make it difficult to implement an effective lessons learning and 
knowledge sharing environment. We found that successful industry and 
government organizations have overcome barriers by making a strong 
management commitment to knowledge sharing, developing a well-defined 
business plan for implementing knowledge management, providing 
incentives to encourage knowledge sharing, and building technology 
systems to facilitate easier access to information. The application of 
these principles could increase opportunities for NASA to perform its 
basic mission of exploring space faster, better, and cheaper more 
successfully. 

Principal Findings: 

NASA’s Policies and Procedures for Lessons Learning: 

NASA uses various mechanisms to communicate lessons garnered from past 
programs and projects. Policies and guidelines, programmatic and 
technical reviews, mentoring and training programs, the Academy of 
Program and Project Leadership, and LLIS are the mechanisms employed 
by NASA for capturing and sharing lessons learned. LLIS is the 
“official” agency-wide repository for such lessons. Lessons entered in 
the LLIS database are screened for relevance and to ensure that they 
do not contain sensitive or proprietary information. Initial reviews 
of lessons are usually conducted by the centers,[Footnote 1] with a 
final review by the Office of Safety and Mission Assurance. After a 
lesson is entered into the system, it remains in the database 
indefinitely and is not reviewed for currency or relevance. Currently, 
the system contains over 900 lessons on topics ranging from program 
management to technical cause of failure. 

In response to the Mars Program failures and the recommendations of 
agency reviews of program and project execution, NASA has recently 
taken action to improve its policies and practices for capturing and 
sharing knowledge by developing a business strategy referred to as 
knowledge management. Implementation of knowledge management can lead 
to increased productivity, collaboration, and innovation in the 
workplace. To coordinate and guide its efforts, NASA recently formed a 
knowledge management team, which developed a strategic plan that laid 
out broad goals and objectives for knowledge management. In addition, 
several pilot projects are underway at various NASA centers to enhance 
knowledge sharing. 

Fundamental Weaknesses Exist in the Collection and Sharing of Lessons 
Learned: 

A survey we conducted of all NASA program and project managers 
revealed fundamental weaknesses in the collection and sharing of 
lessons learned agency-wide. Although NASA’s processes and procedures 
require that program and project managers review and apply lessons 
learned throughout a program’s or project’s life cycle, our survey 
found that managers do not routinely identify, collect, or share 
lessons. Respondents indicated that LLIS, NASA’s primary method for 
disseminating lessons learned agency-wide, is not the primary source 
for lessons learning. Instead, managers identified program reviews and 
informal discussions with colleagues as their principal sources for 
lessons learned. One reason LLIS is not widely used, according to one 
center official, is because its lessons cover so many topics that it 
is difficult to search for an applicable lesson. Another respondent 
indicated that it is difficult to weed through all the irrelevant 
lessons to get to the few “jewels” that you need to find. 

Respondents also identified challenges or cultural barriers to the 
sharing of lessons learned as well as areas of improvement. Managers 
noted that there is a reluctance to share negative lessons for fear 
that they might not be viewed as good project managers, and there is a 
lack of time for lessons learning to take place. One manager stated, “
Until we can adopt a culture that admits frankly to what really worked 
and didn’t work, I find many of these tools to be suspect.” Managers 
suggested that NASA could improve lessons learning by implementing 
mentoring and “storytelling” activities, and it could enhance LLIS by 
increasing its search functions, including more positive lessons, and 
developing a mechanism to disseminate key lessons to users. 

In discussions with NASA officials, we found there was general 
agreement with the results of our survey as well as suggested 
improvements for lessons learning. Officials indicated that lessons 
learning has taken on greater importance in recent years due to the 
implementation of more programs and projects under the faster, better, 
cheaper strategy and the continuing loss of agency expertise due to 
attrition. They acknowledge that LLIS has not been an effective 
mechanism for agency-wide sharing of lessons. Although the system is 
viewed as providing a useful repository for storing lessons, officials 
agreed with managers’ concerns about the difficulties involved in 
searching the system and finding relevant lessons, the inconsistent 
quality of information contained in the system, and the lack of 
lessons about positive project experiences. However, while program and 
project managers’ suggested improvements would help increase the 
usability of LLIS, they have not targeted some of the more fundamental 
problems hampering NASA’s ability to share lessons, such as persistent 
cultural barriers. 

Creating an Environment for Lessons Learning through Knowledge 
Management: 

Leading organizations are discovering that actively managing knowledge 
creates value by increasing productivity and fostering innovation. 
Likewise, NASA’s paramount concern should be about capturing and 
sharing organizational knowledge and using it to perform its basic 
mission of exploring space faster, better, and cheaper. Although NASA 
has recently taken action to improve the way in which the agency 
captures, organizes, and shares knowledge, these efforts do not fully 
address the fundamental weaknesses in lessons learning identified by 
our survey: namely, cultural resistance to sharing knowledge and the 
lack of an effective strategic framework and management attention for 
overcoming such resistance. 

NASA has made a reasonable start by developing a strategic plan for 
knowledge management, but the agency has not made a good business case 
for how it will implement and use knowledge management within the 
organization. In addition, while successful industry and government 
organizations have made a firm commitment to making knowledge 
management practices work, NASA has not provided the leadership, 
support, and resources needed for effective knowledge management to 
take place. Furthermore, knowledge management organizations have 
employed incentives, processes, and systems designed to address 
cultural barriers to continuous lessons learning and knowledge 
sharing. For example, organizations that value knowledge sharing have 
encouraged employees to spend time sharing knowledge, helped 
facilitate communities of practice based around common interests, and 
provided rewards when knowledge has been shared and applied. NASA has 
not done so on an agency-wide basis. 

Recommendations: 

NASA needs to strengthen its lessons learning in the context of its 
overall efforts to develop and implement an effective knowledge 
management program. Improvement of NASA’s lessons learning processes 
and systems can help to ensure that knowledge is gained from past 
experiences and applied to future missions. 

We recommend that the NASA administrator strengthen the agency’s 
lessons learning processes and systems by 

* articulating the relationship between lessons learning and knowledge 
management through an implementation plan for knowledge management; 

* designating a lessons learned manager to lead and coordinate all 
agency lessons learning efforts; 

* establishing functional and technical linkages among the various 
center-level and program-level lessons learning systems; 

* developing ways to broaden and implement mentoring and “storytelling”
as additional mechanisms for lessons learning; 

* identifying incentives to encourage more collection and sharing of 
lessons among employees and teams, such as links to performance 
evaluations and awards; 

* enhancing LLIS by coding information and developing an easier search 
capability to allow users to identify relevant lessons, including more 
positive lessons, providing a means to disseminate key lessons to 
users; and soliciting user input on an ongoing basis; and; 

* tracking and reporting on the effectiveness of the agency’s lessons 
learning efforts using objective performance metrics. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, NASA generally 
concurred with our recommendations for improving the agency’s lessons 
learned processes and systems. NASA stated that it must do a better 
job of communicating the various lessons learned sources to employees, 
improving mechanisms to link these sources, and ensuring appropriate 
training for employees in order to maximize lessons learning. 

Executive Summary Footnote: 

1. NASA consists of NASA headquarters, nine centers, the Jet 
Propulsion Laboratory (operated under contract to NASA by the 
California Institute of Technology), and several ancillary 
installations and offices in the United States and abroad. The 
implementation of NASA programs and aeronautical and space/earth 
science research occurs primarily at the centers. 

[End of section] 

Chapter 1: Introduction: 

The 1990’s was a decade of significant challenge for the National 
Aeronautics and Space Administration (NASA) brought about, in part, by 
decreased budgets, a reduced work force, and technological innovation. 
To adapt to this changing environment, the NASA administrator 
challenged agency personnel to do projects faster, better, and cheaper 
(FBC) by streamlining practices and becoming more efficient. The goal 
was to shorten program development times, reduce cost, and increase 
scientific return by flying more and smaller missions in less time. 
However, the failure of the Mars Polar Lander and Climate Orbiter 
spacecraft raised concerns that lessons learned from past experiences 
were not being passed along and applied toward future mission success. 
To ensure that individuals learn from past experiences, effective 
processes and systems must be in place to collect, store, and 
disseminate lessons learned. 

Faster, Better, Cheaper: A Management Philosophy: 

In 1992 NASA adopted the FBC philosophy as a way of managing programs 
and projects. An important element of this approach was a reduction in 
NASA headquarters management and moving more program responsibility to 
NASA’s centers. This philosophy also increased the demand for program 
and project managers at a time when NASA was experiencing a 
significant reduction in staff due to retirements, downsizing, and 
departures to industry. Prior to FBC, there were fewer missions, and 
program and project managers accumulated significant first-hand 
experience before managing a program. Under FBC, with a threefold 
increase in projects and fewer staff, this was not always the case. 
Relatively unseasoned managers who were challenged to be more 
efficient and innovative and to take greater risks in designing and 
implementing missions led many projects. 

NASA’s record in designing, developing, and operating smaller 
spacecraft under the FBC concept, while mostly successful, has 
experienced a few notable failures. A number of projects including 
Clementine, Near Asteroid Rendezvous, Mars Pathfinders, and Mars 
Global Surveyor proved successful within the schedule and fiscal 
constraints imposed under the FBC concept. However, for other 
programs, including the Mars Polar Lander and Climate Orbiter, the 
challenge was too great, resulting in undue risk-taking and failure. 
After-action reports commissioned by NASA found that the Mars Program 
failures resulted from cost and schedule constraints and a lack of 
rigorous attention to sound process and practices. 

Mars Surveyor Program: 

In 1993 NASA initiated the Mars Surveyor Program with the objective of 
conducting an on-going series of missions to explore Mars. NASA’s Jet 
Propulsion Laboratory (JPL) was selected as the lead center for this 
program and was responsible for mission design, spacecraft and payload 
development, as well as integration, testing, and launch of the 
proposed spacecraft. 

The first Mars Surveyor Program spacecraft, the Mars Global Surveyor 
and the Mars Pathfinder, were launched in 1996 and both were highly 
successful. After completing its primary mission of acquiring 
knowledge of the climate, subsurface resources, and topography of 
Mars, the Mars Global Surveyor is conducting reconnaissance of future 
Mars landing sites. The Mars Pathfinder and Sojourner Rover were a 
demonstration of a way to use technology to deliver an instrumented 
lander and a free-ranging robotic rover to the surface of Mars. Two 
additional Mars Surveyor Program spacecraft were launched in late 1998 
and early 1999, respectively, to explore and collect additional 
scientific data on Mars. 

The Mars Climate Orbiter, which cost $75 million to develop, was 
launched December 11, 1998, and was intended to observe Mars’ seasonal 
climate and daily weather from a low orbit around the planet. Nine and 
a half months after launch, in September 1999, the spacecraft was 
supposed to fire its main engine to achieve an elliptical orbit around 
Mars and then pass through its upper atmosphere for several weeks 
before moving into a low circular orbit around the planet. On 
September 23, 1999, the Orbiter spacecraft was destroyed when it 
entered the Martian atmosphere on a lower than expected trajectory, 
causing the spacecraft to burn up. A NASA mishap investigation team 
determined that the loss was due to the mistaken use of English rather 
than metric units in the navigation software. As a result, an 
incorrect trajectory was computed and the spacecraft was lost. 

The Mars Polar Lander was launched January 3, 1999, and cost $113 
million to develop. The Lander was a robotic spacecraft designed to 
land softly near the South Pole of Mars to study the planet’s layered 
polar terrain. Attached to the Lander was the Deep Space 2 mission—two 
small probes designed to separate from the Lander about 5 minutes 
prior to touchdown and penetrate at high speed into the Martian soil 
to search for traces of vaporized water ice. The Deep Space 2 probes 
cost $29.6 million to develop. Unfortunately, the Lander was lost 
December 3, 1999, while attempting to land. The two small probes also 
were lost. NASA’s mishap investigation team concluded that the 
premature shutdown of the Lander’s descent engines was the most 
probable cause for the loss of the spacecraft. 

Mishap Investigations Raised Concern That NASA Did Not Heed Lessons 
Learned: 

The failure of the Mars Polar Lander and Climate Orbiter spacecraft 
raised concern that lessons learned from past mishaps and programs 
were not being applied effectively toward future mission success. 
Support for this comes from reports commissioned by NASA that reviewed 
spacecraft failures in the Mars Program, Shuttle wiring problems, and 
an assessment of the agency’s approach in executing “Faster, Better, 
Cheaper” projects.[Footnote 1] These reports identified the root and 
contributing causes for failure as well as making broader 
recommendations on ways NASA might improve its general approach to 
executing programs and projects. 

NASA’s decision to leave critical information-gathering mechanisms off 
the Mars Polar Lander is a case in point. The importance of downlink 
telemetry[Footnote 2] during critical mission events was a lesson 
learned from the failure of NASA’s 1992 Mars Observer mission that was 
not heeded during design of the Mars Polar Lander seven years later. 
The design of Lander precluded transmission of critical communications 
or telemetry data during entry, descent, and landing on Mars. A 
Special Review Board determined that the probable cause of the loss of 
the spacecraft was due to a premature shutdown of the descent engines 
due to spurious signals generated when the Lander’s legs deployed. The 
spurious signals gave a false indication that the spacecraft had 
landed. The board also determined that the spurious signals resulted 
from inadequate software design and systems testing. 

Concern that NASA had not heeded past lessons learned was also 
confirmed in a recent study[Footnote 3] conducted by RAND’s National 
Security Research Division. The purpose of the study was to provide 
guidance on practices that reduce risk and improve the performance of 
next-generation spacecraft by including an examination of NASA’s 
successes and failures in building spacecraft both before and after 
implementing the faster, better, cheaper approach. The RAND study 
identified the top ten sources of failures in NASA programs and 
reported that a significant source of error has been the failure of 
NASA to incorporate lessons previously learned and consistently apply 
them. As shown in the following table, the reasons for failure of the 
Mars Polar Lander and Climate Orbiter are not new; they have been 
identified as areas of concern by major program reviews and have 
occurred during the development of several other space systems. 

Table 1: Reasons for Spacecraft Failures: 

Reasons for Failure: 

Major Program Reviews: 

Broad Area Review: 
Cost and Schedule Constraints: [Check]; 
Insufficient Risk Assessment and Planning: [Empty]; 
Underestimation of Complexity and Technology Maturity: [Check]; 
Insufficient Testing: [Empty]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: [Check]; 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

Lockheed Martin Independent Assessment Team: 
Cost and Schedule Constraints: [Empty]; 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Empty]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: [Check]; 
Inadequate Review Process: [Check]; 
Design Errors: [Empty]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

Faster, Better, Cheaper Task Force: 
Cost and Schedule Constraints: [Empty]; 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Check]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: [Check]; 
Inadequate Review Process: [Check]; 
Design Errors: [Empty]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Empty]. 

Major Mishap Reviews: 

WIRE[A]: 
Cost and Schedule Constraints: [Check]; 
Insufficient Risk Assessment and Planning: [Empty]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Empty]. 

Mars Climate Orbiter: 
Cost and Schedule Constraints: 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: [Check]; 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

Mars Polar Lander: 
Cost and Schedule Constraints: 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Empty]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

Lewis: 
Cost and Schedule Constraints: [Check]; 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Check]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: 
Inadequate Review Process: [Check]; 
Design Errors: [Empty]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Empty]. 

SOHO[B]: 
Cost and Schedule Constraints: 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Check]; 
Insufficient Testing: [Empty]; 
Poor Team Communication: [Empty]; 
Inattention to Quality and Safety: 
Inadequate Review Process: [Check]; 
Design Errors: [Empty]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

Mars Observer: 
Cost and Schedule Constraints: [Check]; 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Check]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Empty]; 
Inattention to Quality and Safety: [Empty]; 
Inadequate Review Process: [Empty]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Check]; 
Inadequate or Under Trained Staff: [Check]. 

DC-X[C]:
Cost and Schedule Constraints: 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Empty]; 
Inattention to Quality and Safety: [Empty]; 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; 
Inadequate System Engineering: [Empty]; 
Inadequate or Under Trained Staff: [Check]. 

Challenger: 
Cost and Schedule Constraints: [Check]; 
Insufficient Risk Assessment and Planning: [Check]; 
Underestimation of Complexity and Technology Maturity: [Empty]; 
Insufficient Testing: [Check]; 
Poor Team Communication: [Check]; 
Inattention to Quality and Safety: [Check]; 
Inadequate Review Process: [Check]; 
Design Errors: [Check]; [Empty]; 
Inadequate System Engineering: [Empty]; 
Inadequate or Under Trained Staff: [Check]. 

[A] WIRE: Wide-Field Infrared Explorer. 

[B] SOHO: Solar and Heliospheric Observatory. 

[C] DC-X: Delta Clipper-Experimental. 

Source: RAND, used with permission. 

[End of table] 

NASA Efforts to Apply Lessons Learned from Mars Mishaps: 

In December 2000, NASA issued a report, Enhancing Mission Success, 
which identified specific actions the agency planned to take to apply 
prior mishap findings and recommendations broadly in order to improve 
its approach to executing programs and projects.[Footnote 4] The 
development of this report was driven principally by the multiple 
mission failures associated with the Mars Program. As part of its 
assessment, NASA reviewed a total of 165 recommendations originating 
from the Mars mishaps reports, the Shuttle Independent Assessment, and 
the agency-wide assessment of faster, better, cheaper projects. As a 
result, NASA recommended 17 actions that are integrated into five 
broad themes: people, technology, risk management, program and project 
management, and communications. 

One of the 17 recommended actions, which deals with improving 
communications, concerns knowledge management[Footnote 5] and the 
recognition that NASA needs to do better in capturing, disseminating, 
and utilizing knowledge. This includes improving the capture and 
application of lessons learned from programs, projects, and missions, 
with the goal of ensuring that NASA does not have to keep “relearning” 
the lessons of the past-relearning evidenced by the reoccurrence of 
similar causes to mission failures or difficulties. The report also 
indicated a lack of access to and process for lessons learned. The 
recommended action further stated that the continuous capture and 
application of project knowledge and lessons learned must become a 
core business process within the agency’s program and project 
management environment. 

Lessons Learning: A Mechanism to Learn from Successes As Well As 
Mistakes: 

Use of lessons learned is a principal component of an organizational 
culture committed to continuous improvement. Lessons learned 
mechanisms serve to communicate acquired knowledge more effectively 
and to ensure that beneficial information is factored into planning, 
work processes, and activities. Lessons learned provide a powerful 
method of sharing good ideas for improving work processes, facility or 
equipment design and operation, quality, safety, and cost-
effectiveness. 

Chapter 1: Introduction 

Elements of the Lessons Learned Process: 

The mechanisms or processes used to collect, share, and disseminate 
lessons learned may vary, but in general such a process is comprised 
of four main elements: collection, verification, storage, and 
dissemination. Figure 1 is a generic representation of the lessons 
learned process. 

Figure 1: Generic Lessons Learned Process: 

[Refer to PDF for image: illustration] 

Observe Lessons or Best Practices: 
Collect: 
Collection of Lessons and Practices: 
Verify: 
Review for Applicability: 
Store: 
Lessons Learned Repository: 
Disseminate: 
Observe Lessons or Best Practices: 
Reuse: 
Applying Knowledge. 

Source: Based on Weber, R., Aha, D., and Becerra-Fernanadez, I. 
Categorizing Intelligent Lessons Learned Systems. Intelligent Lessons 
Learned Systems: Papers from the AAAI Workshop (Technical Report AIC-
00-005). Aha, D.W. and Weber, R. (Eds.) pp. 63-67. Washington, DC: 
Naval Research Laboratory, Navy Center for Applied Research in 
Artificial Intelligence, 2000. 

[End of figure] 

* The collection process involves the capture of information through 
structured and unstructured processes such as mishap or accident 
reporting, project critiques, written forms, and meetings. The 
collection of lessons may come from as many sources as an organization 
is willing to solicit. Lessons learned can be based upon positive 
experiences that prevent accidents or save money or on negative 
experiences that result in undesirable outcomes. However, if an 
organization focuses only on failures, its overall program’s 
effectiveness will be reduced and it will miss opportunities to 
improve all its processes. 

* The verification process serves to verify the correctness and 
applicability of lessons submitted. Domain or subject matter experts 
may be involved in coordinating and conducting reviews to determine 
whether or not a lesson is relevant across many other projects, is 
unique to a particular department or project, or applies globally to 
the organization as a whole. 

* The storage aspect of lessons learned usually involves incorporating 
lessons into an electronic database for the dissemination and sharing 
of information. Information should be stored in a manner that allows 
users to identify applicable information searches. In addition, each 
program should include a keyword and functional category search 
capability to facilitate information retrieval. 

* The final element, and the most important, is the dissemination of 
lessons learned, since lessons are of little benefit unless they are 
distributed and used by people who will benefit from them. 
Dissemination can include the revision of a work process, training, 
and routine distribution via a variety of communication media. Lessons 
can be “pushed,” or automatically delivered to a user, or “pulled” in 
situations where a user must manually search for them. Lessons can 
also be disseminated with an assigned priority descriptor, which 
denotes the risk, immediacy, and urgency of the lessons learned 
content. 

Objectives, Scope, and Methodology: 

The Chairman and the Ranking Minority Member, Subcommittee on Space 
and Aeronautics, House Committee on Science, requested that we examine 
whether NASA has sufficient mechanisms in place to ensure that past 
lessons learned are applied effectively towards future mission 
success. Specifically, we (1) identified what policies, procedures, 
and systems NASA has in place for lessons learning, (2) assessed how 
effectively these policies, procedures, and systems facilitate lessons 
learning, and (3) determined whether further efforts are needed to 
improve lessons learning. 

We reviewed NASA policies and procedures and interviewed agency staff 
to determine how they captured, processed, and used lessons learned in 
their programs and projects. We also obtained briefings and documents 
on the different mechanisms used by NASA to capture and disseminate 
lessons learned. In addition, we conducted site visits at NASA 
headquarters, Washington, D.C.; Goddard Space Flight Center, 
Greenbelt, Maryland; Kennedy Space Center, Florida; Langley Research 
Center, Hampton, Virginia; and the Jet Propulsion Laboratory, Pasadena, 
California.[Footnote 6] 

To assess how effectively NASA’s policies and procedures facilitate 
lessons learning, we conducted a survey of NASA program and project 
managers. We issued an interim report in September 2001 on the results 
of our survey.[Footnote 7] NASA officials generally agreed with the 
survey results and indicated that the agency must do a better job of 
communicating various lessons learning resources to employees, 
improving linkages among the sources, and encouraging training to 
increase lessons learning. We build on the findings and results of our 
survey in this report. The methodology we employed in conducting this 
survey included the following steps: 

* To obtain the views of NASA program and project managers on lessons 
learned processes and systems at NASA, we conducted a survey in June 
and July 2001 using a self-administered electronic questionnaire 
posted on the World Wide Web. The survey contained four groups of 
questions on (1) how NASA personnel collect, access, and use lessons 
learned; (2) the strengths and weaknesses of current NASA lessons 
learned processes, procedures and systems, including the Lessons 
Learned Information System; (3) potential challenges or barriers to 
sharing lessons learned within NASA; and (4) ways to improve the 
lessons learned processes and systems at NASA. 

* In designing the questionnaire, we interviewed NASA officials and 
program and project managers, as well as other government and industry 
officials who had insight into lessons learned processes and systems. 
We also obtained and reviewed NASA documents and guidance pertinent to 
lessons learned. To further guide the development of appropriate 
questions, we reviewed current literature on lessons learned and 
knowledge management. To validate the content and structure of the 
questionnaire, we submitted it to officials at NASA headquarters and 
others for review and comment. To verify the clarity, length of time 
of administration, and suitability of the questions, we also pre-
tested the questionnaire with selected program and project managers at 
Goddard Space Flight Center. 

* NASA officials provided us with a list of the e-mail addresses of 
all NASA program and project managers as of April 2001. On June 5, 
2001, we sent e-mail messages to each of the 199 NASA managers in this 
survey population, notifying them of the survey and asking them to 
complete the questionnaire. Subsequently, we discovered one additional 
NASA manager who was eligible for our survey, and removed seven 
managers who were ineligible for the survey because they were not 
project or program managers at the time of the survey. We also 
eliminated one duplicate listing, resulting in a final survey 
population of 192. 

* Over the following several weeks, until closing the survey on July 
13, 2001, we received a total of 115 usable responses, for an overall 
response rate of 60 percent. All of the responses had been submitted 
using the Web questionnaire, although one participant had asked to 
submit a paper version of the survey, due to problems in accessing the 
Web survey. Of the 77 non-respondents, 9 provided partial 
questionnaire responses but had not indicated that they were finished 
with the questionnaire. Partial responses were not included in our 
survey results. 

While we believe that our survey results are generalizable to the 
population of NASA program and project managers as described above, 
the practical difficulties of conducting any survey may introduce 
errors into estimates made from surveys. Although we administered 
questionnaires to all known members of the population, and thus our 
results are not subject to sampling error, non-response to the entire 
survey or individual questions can introduce a similar type of 
variability or bias into our results to the extent that those not 
responding differ from those who do respond in how they would have 
answered our survey questions. In addition, population coverage errors 
can occur if some members of the population are excluded from the 
survey. Measurement errors can arise from how questions are 
interpreted by respondents and mistakes made by respondents. Data 
processing errors can arise during the handling or analysis of 
responses. We took steps in the design, data collection, and analysis 
phases of our survey to minimize such errors, such as pre-testing 
questionnaires before the survey, following up with those not 
reachable at original e-mail addresses or otherwise not immediately 
responding, and checking for errors in computer programming used to 
analyze survey results. In addition, the distribution of respondents 
across NASA enterprise areas generally reflected the actual 
distribution of the entire population, which was consistent with our 
belief that non-response error was not significant. 

To identify practices for improving NASA’s lessons learning processes 
and systems, we reviewed existing literature on lessons learning and 
knowledge management practices. In addition, we identified a selected 
number of commercial and government organizations that are 
practitioners of knowledge management and obtained information related 
to their experience. In making our selections, we reviewed literature 
and spoke with industry and academic experts to find organizations 
recognized for their ability to share lessons learned or effectively 
manage knowledge. We identified the following organizations: World 
Bank, United States Department of Energy, Ford Motor Company, TRW, 
Boeing Space and Communications, Northrop Grumman, and Lockheed Martin. 

We discussed with representatives from each organization the processes 
they used for sharing information, emphasizing those practices that 
were critical for successful lessons learning. During our discussions 
with the organizations, we compared and contrasted their practices 
with those of NASA. This allowed us to identify practices that NASA 
could use to improve its processes and systems. For each organization, 
we interviewed key managers and obtained documentation to determine 
(1) the processes and tools used to enhance information sharing and 
(2) the extent to which these processes and tools affected the 
organizations’ ability to share information. 

Our work was performed from October 2000 through September 2001 in 
accordance with generally accepted government auditing standards. 

Chapter 1 Footnotes: 

1. Mars Program Independent Assessment, chaired by Mr. A. Thomas Young 
(retired), Lockheed Martin; Mars Climate Orbiter (MCO) Mishap 
Investigation, chaired by Mr. Arthur Stephenson, director, Marshall 
Space Flight Center; NASA Faster, Better, Cheaper Task Force, chaired 
by Mr. Anthony Spear (retired), Jet Propulsion Laboratory; and Shuttle 
Independent Assessment, chaired by Dr. Henry McDonald, director, Ames 
Research Center. 

2. Telemetry is information on the condition of the spacecraft and its 
subsystems, such as the temperature and voltage of spacecraft 
batteries, that is transmitted to spacecraft operators on the ground. 

3. Sarsfield, Liam. The Application of Best Practices to Unmanned 
Spacecraft Development: An Exploration of Success and Failure in 
Recent Missions. (Santa Monica, CA: RAND, 2000). 

4. National Aeronautics and Space Administration. Enhancing Mission 
Success – A Framework for the Future. A Report by the NASA Chief 
Engineer and the NASA Integrated Action Team, December 21, 2000. 

5. Knowledge management can be defined as the way that organizations 
create, capture, and re-use knowledge to achieve organizational 
objectives. 

6. The Jet Propulsion Laboratory is a federally funded research and 
development center. In our report, we treat the Jet Propulsion 
Laboratory as a NASA center in our discussions. 

7. U.S. General Accounting Office, Survey of NASA’s Lessons Learned 
Process, [hyperlink, http://www.gao.gov/products/GAO-01-1015R] 
(Washington, D.C.: 2001). 

[End of section] 

Chapter 2: NASA Policies, Processes, and Systems for Lessons Learning 
and Knowledge Sharing: 

NASA uses various mechanisms to communicate lessons garnered from past 
programs and projects. Policies and guidelines, programmatic and 
technical reviews, mentoring and training programs, the Academy of 
Program and Project Leadership, and the Lessons Learned Information 
System (LLIS) are all mechanisms employed by NASA for capturing and 
sharing lessons learned. While not the only means available for 
capturing and disseminating lessons learned, LLIS is the “official” 
agency-wide repository for such lessons. In addition to agency-wide 
mechanisms for collecting and sharing lessons learned, several NASA 
centers also operated lessons learned systems configured to support 
specific programs, such as the International Space Station. 

To improve the way the agency captures and shares information and 
lessons learned, NASA has recently formed a knowledge management team. 
Knowledge management is a business strategy used by many organizations 
that strive to make more effective use of the experience and expertise 
of employees within an organization. Organizations believe that by 
developing new ways to capture and share knowledge, they can increase 
productivity, collaboration, and innovation. 

NASA’s Policies and Procedures Require Lessons Learning: 

NASA’s policies and procedures require the continuous capture, 
dissemination, and utilization of lessons learned. NASA defines a 
lesson learned as knowledge or understanding gained by experience. The 
experience may be positive, as in a successful test or mission, or 
negative, as in a mishap or failure. A lesson must be significant in 
that it has a real or assumed impact on operations; valid in that it 
is factually correct; and applicable in that it identifies a specific 
design, process, or decision that reduces or eliminates the potential 
for failures and mishaps, or reinforces a positive result. 

Program and Project Managers Are Directed to Review Lessons Learned: 

To ensure that lessons learned from previous experiences are used as a 
resource, NASA Procedures and Guidelines for Program and Project 
Management Processes and Requirements (NPG 7120.5A) directs each 
program and project manager to review and apply significant lessons 
learned from the past throughout the program or project life cycle. 
Managers are also directed to consult LLIS prior to major milestones 
to gain lessons from past programs and projects that are documented 
and collected as a benefit to future programs and projects. Program 
and project managers are also directed to document and submit to the 
LLIS in a timely manner any significant lessons learned throughout the 
life of a project or program. One of the objectives of the agency’s 
guidance is to protect against the recurrence of past mistakes and 
provide a mechanism for sharing best practices. 

NASA Mishap Reporting and Investigating Policy (NPD 8621.1G) and NASA 
Procedures and Guidelines for Mishap Reporting, Investigating, and 
Recordkeeping (NPG 8621.1) also require the development of lessons 
learned from the mishaps for possible application to existing or 
future programs.[Footnote 1] In addition, NASA is supposed to exchange 
lessons learned and other pertinent safety information of common 
interest with other federal agencies, international partners, and NASA 
contractors. Since NASA uses contractors to produce many of its 
systems, NASA’s Procedures and Guidelines for Management of Government 
Safety and Mission Assurance Surveillance Functions for NASA Contracts 
(NPG 8735.2) requires that lessons learned during the contract 
surveillance activities be captured and submitted to LLIS. 

NASA Uses Numerous Methods for Lessons Learning: 

Lessons learned mechanisms are used by many commercial and government 
organizations to share and use knowledge derived from experience to 
(1) promote the recurrence of desirable outcomes, or (2) preclude the 
recurrence of undesirable outcomes. Policies and guidelines, 
programmatic and technical reviews, training programs, and the LLIS 
are all mechanisms employed by NASA for capturing and sharing lessons 
learned. While not the only means available for capturing and 
disseminating lessons learned, LLIS is NASA’s only agency-wide 
repository for such lessons. 

LLIS is NASA’s Principal Mechanism for Sharing Lessons Learned: 

The purpose of LLIS is to capture and share lessons learned from 
programs, projects, and missions and to help ensure that NASA does not 
have to keep “relearning” the lessons of the past. LLIS was created in 
1995 and subsequently updated to include a Web interface. Program and 
project managers are directed to use LLIS to identify and share 
lessons learned, thus facilitating the early incorporation of safety, 
reliability, maintainability, and quality into the design of flight 
and ground support hardware, software, facilities, and procedures. 
Currently, the system contains over 900 lessons learned on the 
development and design of aeronautics and space systems. 

Lessons entered in the LLIS database are screened for relevance and to 
ensure that they do not contain sensitive or proprietary information. 
Initial reviews of lessons are usually conducted by the centers with a 
final review conducted by the Office of Safety and Mission Assurance. 
After a lesson is entered into the system, it is not reviewed for 
currency or relevance. In other words, once lessons are entered, they 
remain in the database indefinitely. The Office of Safety and Mission 
Assurance monitors the system to determine the number of times it is 
accessed, a search is conducted, and/or search results are obtained, 
but does not collect information on the number of times lessons are 
applied to a program or project. 

Administrator for Safety and Mission Assurance is Responsible for LLIS: 

NASA’s Quality Management System Policy (NPD 8730.3) establishes the 
Quality Management Systems used by the agency to identify and control 
processes to assure the quality of hardware, software, and services 
provided.[Footnote 2] According to this policy, the Associate 
Administrator for Safety and Mission Assurance is responsible for 
ensuring that the LLIS database is maintained and accessible. 
Furthermore, the policy states that the associate deputy administrator 
and the center directors are responsible for ensuring that LLIS is 
used to ensure quality and document, investigate, and apply lessons 
learned for all programs and projects. 

Center and Program Based Systems Used to Collect and Disseminate 
Lessons Learned: 

While NASA’s LLIS is supposed to be the NASA-wide system, several NASA 
centers maintain their own electronic lessons learned systems. For 
example, the Goddard Space Flight Center uses the Flights Programs and 
Projects Directorate Lessons Learned Database to collect lessons 
learned for each flight project. Once a lesson is approved, it is 
passed along to the center data manager for review and eventual 
submission to LLIS, thereby ensuring that what has been learned at the 
Goddard Space Flight Center is accessible to other facilities. One 
lesson that was passed along from the Goddard Space Flight Center 
system is the benefit of testing deployable devices, such as the 
Landsat 7 solar array, early in satellite development. Testing devices 
early is a cost-effective tactic for developing complex and critical 
mechanism subsystems. By testing the Landsat 7 solar array early, 
program officials were able to discover problems that could have 
resulted in a program delay and increased costs. Another lesson in the 
Flights Programs and Projects Directorate Lessons Learned Database 
makes a recommendation on minimizing the effects associated with 
ongoing construction at mission-critical facilities. Specifically, 
NASA officials learned the importance of developing a strategy to 
minimize conflicts between the construction schedule and project 
requirements. One of the recommendations for future projects is that 
construction activities in mission-specific support areas of a mission-
critical facility should be suspended 30 days prior to launch to 
ensure that the facility is ready to support mission operations. 

In addition, program-based lessons learned databases have been 
established for key NASA programs. For example, the International 
Space Station and Phase 1 Lessons Learned Database documents 
historical lessons learned from the Phase 1 Program, U.S./Russian 
Program Office of the International Space Station Program; the top ten 
lessons learned during the Skylab program; and current lessons learned 
from the International Space Station Program.[Footnote 3] The 
International Space Station and Phase 1 Lessons Learned Database is 
the official system for submitting new lessons learned in the 
International Space Station program. The lessons learned collected in 
that Lessons Learned Database are intended to be used as tools for the 
improvement of International Space Station operations. NASA officials 
are deliberating the feasibility of transferring the lessons learned 
from the International Space Station database to the NASA-wide LLIS. 

NASA Uses Training for Capturing and Sharing Lessons: 

Another mechanism used by NASA to share lessons is training. NASA’s 
Academy of Program and Project Leadership (APPL) provides numerous 
courses throughout the year.[Footnote 4] Many of the classes employ 
case studies to learn from successes and failures. According to an 
academy official, lessons learned are incorporated into the different 
courses. Lessons incorporated into the curriculum are generic and can 
be applied to all programs (i.e., better communications) rather than 
technical issues unique to a particular program. About 4,000 program 
management employees have attended academy courses, and thus they 
contribute to sharing information within the organization. 

In addition to formal training, APPL engages in development 
activities, which play a role in diffusing knowledge throughout NASA. 
For example, it hosts a series of forums on “hot topics” to gather and 
disseminate information. Another activity involves capturing and 
sharing the experiences of program managers. Once a year, APPL 
sponsors two sessions in which project managers considered the “best” 
by their centers are sent to discuss project management. At these 
sessions, two or three project managers from each center share what 
works and the types of tools that they need to accomplish their jobs. 

Another activity used by APPL is encouraging senior program managers 
to share their knowledge through a series of short stories made 
available through its Web site. Each story discusses a topic that will 
help program managers succeed. One recently completed story dealt with 
the importance of saying no. According to a NASA official, many 
developing project and program managers are driven by NASA’s “can do” 
attitude and have a hard time knowing when to say “no” during a 
project. NASA is also identifying capable individuals to mentor new 
project and program managers. In addition, managers are provided 
several online tools and training resources, such as project 
management tools that can be used to effectively manage a project. 

Changes to Policy Also Reflect Lessons Learned: 

According to NASA officials, periodic revisions to agency policies and 
procedures are another way that the agency shares lessons learned. For 
example, NASA is currently revising its Program and Project Management 
Processes and Requirements (NPG 7120.5A) and will incorporate some of 
the issues raised by the Mars Climate Orbiter and Polar Lander 
mishaps. This document is intended to provide the basic processes, 
requirements, and responsibilities for managing NASA programs and 
projects. 

Additional Steps Taken by NASA to Facilitate Lessons Learning: 

In response to the Mars Program failures and the recommendations of 
the NASA Integrated Action Team, NASA has taken steps to promote the 
capture and sharing of knowledge within the agency by developing new 
policies, processes, and practices for program and project management. 
Foremost among these is the development of a business strategy 
commonly referred to as knowledge management. According to NASA 
officials, knowledge management has the potential to link people with 
the information and resources they need to complete tasks faster, 
better, and cheaper. Knowledge management can be defined as the way 
that organizations create, capture, and re-use knowledge to achieve 
organizational objectives. Organizations that manage knowledge 
effectively claim higher performance, innovation, and collaboration 
among employees. Lesson learning in the context of knowledge 
management is defined as “knowledge, both positive and negative, 
gained through experience, which if shared, would benefit the work of 
others.” 

NASA Created a Knowledge Management Team to Coordinate Knowledge 
Sharing Activities: 

NASA has recently created a knowledge management team, composed of 
headquarters and center officials, to advise the agency’s chief 
information officer (CIO). The team does not act as a central 
authority for knowledge management, but rather as a focal point for 
coordinating knowledge management activities conducted throughout the 
agency. Its goal is to connect the various knowledge management 
activities ongoing in the agency and avoid duplication. The team is 
currently working on efforts to baseline knowledge management in NASA, 
gather requirements, and define opportunities for applying knowledge 
management. 

Knowledge Management Projects Conducted by Selected Centers: 

Several centers have initiated knowledge management projects in the 
past few years. For example, the Ames Research Center is designing and 
developing a comprehensive information technology program called 
Design for Safety/Engineering for Complex Systems to help programs and 
projects agency-wide manage risk and capture and disseminate knowledge 
and lessons learned. The Johnson Space Center is piloting a Quality 
Assurance database, and the Jet Propulsion Laboratory is developing an “
experts” directory as well as a Technical Questions database. In 
addition, the Kennedy Space Center has created a category of knowledge 
managers in each of its directorates who are responsible for 
identifying and recognizing core knowledge management competencies. 
Furthermore, the Langley Research Center has initiated a process to 
reward scientists if knowledge they have shared is used by others. The 
re-use of the knowledge must be documented in order for the scientist 
to receive the reward. 

CIO-Sponsored Knowledge Management Pilot Projects: 

NASA’s CIO supports agency knowledge management efforts by investing 
in information technology projects to facilitate knowledge sharing. 
The NASA Knowledge Management Team advises the CIO on projects 
submitted by the centers for funding. In fiscal year 2000, the CIO 
funded three knowledge management pilot projects totaling $567,000. 
For fiscal year 2002, the CIO has a budget of $3 million that can be 
used to fund selected knowledge management initiatives. Because 
knowledge management projects are not centrally managed or funded, 
there is no visibility over how much the various NASA centers spend on 
them. 

One of the projects being funded is to develop an agency-wide Web 
portal to bring together NASA’s online information resources,[Footnote 
5] which are currently dispersed across different agency functional 
areas, centers, and programs. The portal will integrate existing 
resources and provide users with more direct access to information 
from across the agency. A second project is intended to develop a 
directory of agency expertise, which will allow NASA personnel to 
quickly locate others working in a related field or on a particular 
project and thus enhance collaboration among distributed groups. When 
complete, the directory will be integrated into the NASA Web portal. 

The third CIO-funded project will improve the capture and re-use of 
lessons learned by augmenting the agency’s current LLIS. The pilot 
project was a collaborative effort involving the Jet Propulsion 
Laboratory, Langley Research Center, and the Air Force Research 
Laboratory. The Jet Propulsion Laboratory team was responsible for 
defining the processes used to create and capture lessons learned, as 
well as the functional and system requirements for the system’s 
redesign. The Langley Center was responsible for developing the 
digital library technology for the system. The Air Force Research 
Laboratory role was to evaluate the system design. 

After spending about $135,000 to develop a system requirements 
document and a new system software design, NASA terminated the pilot 
project. According to a NASA official, the Office of Safety and 
Mission Assurance considered the functional and system requirements 
document to be inadequate, and their office lacked the resources to 
implement the proposed prototype. However, since the termination of 
the pilot, the Goddard Space Flight Center has been tasked with 
determining ways to improve the LLIS. Officials there have proposed a 
subscription service that would enable users to receive lessons 
learned tailored to their interests. Other follow-on LLIS improvement 
efforts under consideration include an improved searching and storage 
capability and the ability to append to, update, and control lessons 
configuration over extended periods of time. 

Strategic Plan for Knowledge Management: 

In March 2001, the Knowledge Management Team issued a Strategic Plan 
for Knowledge Management that provides a framework for addressing 
knowledge management in NASA in terms of people, process, and 
technology (see table 2). According to the plan, “people” are the most 
important component of knowledge management. It calls for recognizing 
and rewarding people for sharing knowledge as well as encouraging 
story telling and establishing communities of practice to promote 
knowledge sharing. The plan also advocates the development of policies 
and procedures for capturing and managing knowledge that must be 
published and understood by users. In addition, the plan emphasizes 
the need for better information technology to facilitate knowledge 
management. In the context of lessons learning, the plan suggests 
providing a subscription service that tailors specific lessons to user-
defined interests and pushing positive lessons to users’ desktops 
based upon project tasks or organizational characteristics. 

Table 2: NASA’s Framework for Knowledge Management 

People: 
Enable remote collaboration; 
Support communities of practice; 
Reward and recognize knowledge sharing; 
Encourage storytelling. 

Process: 
Enhance knowledge capture; 
Manage information. 

Technology: 
Enhance system integration and data mining; 
Utilize intelligent agents; 
Exploit expert systems. 

Source: Strategic Plan for Knowledge Management, March 2001. 

[End of table] 

Chapter 2 Footnotes: 

1. NASA defines a mishap as “an unplanned event that results in injury 
to non-NASA personnel caused by NASA operations; damage to public or 
private property (including foreign property) caused by NASA 
operations; occupational injury or occupational illness to NASA 
personnel; damage to NASA property caused by NASA operations; or 
mission failure.” 

2. NASA’s Quality Management System is implemented in accordance with 
the International Organization for Standardization’s ISO 9000 Quality 
Management System Standards. 

3. Skylab was the United States’ first experimental space station. The 
Skylab program’s objectives were to prove that humans could live and 
work in space for extended periods, and to expand our knowledge of 
solar astronomy. 

4. The Academy of Program and Project Leadership is the NASA education 
organization responsible for training program and project managers. 

5. A portal is a Web site that is a major starting site for users when 
they connect to the Internet or Intranet. A portal can include 
features such as a topical index, search engine, e-mail, news, and 
links to other online tools. 

[End of section] 

Chapter 3: Lessons Learned Are Not Being Consistently Shared or 
Applied Toward Future Program Success: 

Although NASA recognizes it is more critical than ever to share 
knowledge and collect, store, and distribute lessons learned, the 
agency’s current array of processes, systems, and culture do not 
facilitate such an environment. Importantly, our survey revealed that 
lessons are not routinely identified and shared by program and project 
managers, LLIS has not proven useful to these managers, and there is 
little incentive for managers to share. As a result, program and 
project managers are not maximizing opportunities to apply lessons 
learned toward its faster, better, and cheaper efforts. 

Fundamental Weaknesses Exist in NASA’s Processes, Procedures, and 
Systems for Lesson Learning: 

NASA’s procedures and guidelines require that program and project 
managers review and apply lessons learned from the past throughout a 
program’s or project’s life cycle and document and submit any 
significant lessons learned in a timely manner. However, a survey we 
conducted of all NASA program and project managers revealed 
fundamental weaknesses agency-wide in the collection and sharing of 
lessons learned. While some lessons learning does take place, our 
survey identified that lessons are not routinely identified, 
collected, or shared by program and project managers. In addition, 
many respondents indicated that they are dissatisfied with NASA’s 
lessons learned processes and systems. Respondents also identified 
challenges or cultural barriers to the sharing of lessons learned as 
well as areas of improvement. As a consequence, there is no assurance 
that lessons are being learned and applied to future mission success. 
We provided the basic results of our survey in a September 2001 
report.[Footnote 1] The following analysis builds on those results. 

Limited Sharing of Lessons Learned Agency-Wide: 

To better understand the extent that lessons are being collected and 
shared within NASA, we asked program and project managers what they 
knew about lessons generated by their own programs and centers as well 
as by other centers. As shown in figure 2, program and project 
managers responded that while they are very or somewhat knowledgeable 
about lessons generated by their own programs and centers, they know 
less about lessons generated elsewhere. This fact was further 
substantiated by survey results which show that managers primarily 
identify lessons through program-or center-based activities such as 
project reviews or informal discussions with colleagues. For example, 
one project manager stated that monthly reviews with center management 
provided a way to capture and share lessons learned. The LLIS, NASA’s 
primary method for disseminating lessons learned agency-wide, was not 
identified as a primary source for lessons learning. 

Figure 2: Program and Project Managers Knowledge of Lessons Learned: 

[Refer to PDF for image: vertical bar graph] 

Own program: 
Very knowledgeable: 89.6%; 
Somewhat knowledgeable: 7.8%; 
Not very knowledgeable: 0%; 
No answer: 1.7%. 

Programs and Projects in Own Center: 
Very knowledgeable: 14.6%; 
Somewhat knowledgeable: 74.8%; 
Not very knowledgeable: 7.8%; 
No answer: 2.6%. 

Programs and Projects in Other Centers: 
Very knowledgeable: 0%; 
Somewhat knowledgeable: 46.1%; 
Not very knowledgeable: 51.3%; 
No answer: 1.7%. 

[End of figure] 

Respondents to our survey also indicated that their principal sources 
for identifying lessons learned were (1) system and engineering 
reviews, (2) program and project briefings, and (3) informal 
discussions with colleagues. According to NASA program officials, 
regular program and project meetings, reviews, and briefings provide 
an opportunity to share information, including lessons learned. For 
example, officials at Kennedy Space Center explained that after each 
space shuttle launch they gather to discuss what went well and what 
could be done better. Program and project managers also indicated that 
they maintain informal networks with colleagues where lessons learning 
take place. 

Project reviews and informal discussions with colleagues are important 
mechanisms for lessons learning; however, they do not facilitate the 
systematic, agency-wide sharing of lessons. LLIS, which was 
established for such agency-wide sharing, is not widely used, 
according to survey respondents, even though NASA guidelines state 
that program and project managers should consult it prior to major 
milestones and submit any significant lessons learned in a timely 
manner. One reason the system is not widely used, according to one 
center official, is that its lessons cover so many topics that it is 
difficult to search for an applicable lesson. One of our survey 
respondents indicated that “It is difficult to weed through all the 
irrelevant lessons to get to the few ‘jewels’ that you need to find. 
There should be better categories to find relevant lessons.” Also, 
contractors, who perform much of NASA’s work, are unable to query the 
system easily. As shown in figure 3, in the past 2 years, 43 percent 
of program and project managers have not submitted a lesson to the 
LLIS compared to 23 percent of managers who have. The survey revealed 
there is also a low level of awareness of the system among managers; 
27 percent of program and project managers were not aware of LLIS 
before our survey. 

According to some project officials, accessing LLIS is time consuming 
and uses significant project resources. One official said it took 
about two weeks to review the lessons in the database. Another said 
that developing the process of assessing the applicability of lessons 
learned from the database for that project was very labor intensive 
and costly. For example, one project reviewed all the lessons learned 
in LLIS and attached the appropriate lessons learned as an appendix to 
the project’s Mission Assurance Plan. In contrast, another project 
developed a matrix containing all the lessons learned from LLIS 
categorized by functional area and technical discipline. Project teams 
organized by discipline then reviewed the lessons learned for 
applicability to the project. Further, while some projects formally 
reviewed lessons learned from the Mars Program failures, others did 
not. However, other projects reviewed the lessons learned from both 
LLIS and from the Mars Program failures. 

Figure 3: Many Program and Project Managers Have Not Submitted Lessons 
to the Lessons Learned Information System: 

[Refer to PDF for image: pie-chart] 

None: 43%; 
1-2 times: 12%; 
3-5 times: 6%; 
6-9 times: 2%; 
10 or more times: 3%; 
No answer: 4%. 

[End of figure] 

Dissatisfaction with Lessons Learned Processes and Systems: 

Our survey also determined that more managers are very or generally 
dissatisfied than are satisfied with NASA’s lessons learned processes 
and systems, including LLIS. This level of dissatisfaction may stem 
from the fact that 58 percent of managers stated that current 
processes and systems do not allow them to retrieve the right lessons 
at the right time. As shown in figure 4, the level of dissatisfaction 
may also stem from the fact that 53 percent managers stated that they 
found lessons useful less than 25 percent of the time. One senior 
center official stated LLIS serves more as a repository of information 
that favors technical lessons, and is not effective for passing along 
such information. 

Also, managers stated that it would be beneficial if more emphasis 
were placed on the reporting of positive lessons that are derived from 
successful program outcomes. For example, a positive lesson noted on 
the Jet Propulsion Laboratory’s Deep Space 1 program was having the 
same program management team from the beginning through the end of the 
primary mission. One manager noted that positive lessons can be even 
more helpful to a project than negative ones because even if a project 
avoids negatives, without positives managers may not follow the most 
effective and efficient path to mission success. 

Figure 4: Most Program and Project Managers Could Not Identify Helpful 
Lessons for Their Program or Project: 

[Refer to PDF for image: pie-chart] 

Always or almost always: 4%; 
Very Often: 14%; 
Often: 18%; 
Sometimes: 39%; 
Seldom if ever: 15%; 
Do not know: 5%; 
No answer: 5%. 

[End of figure] 

Barriers Exist to Lessons Learning: 

If lessons learning is to succeed, it is important that the 
organizational culture fosters the value of sharing knowledge based on 
others’ experiences. Our survey respondents identified several notable 
cultural barriers that affect lessons learning within NASA. These 
included: 

* perception that there is no benefit from lessons learned; 

* lack of time to share knowledge; 

* lack of trust; 

* intolerance for mistakes. 

Our survey found that many program and project managers (39 percent) 
believe that implementing lessons learned contributes only some or 
little to improving mission effectiveness. This perception was 
highlighted by one manager who stated that “Until we can adopt a 
culture that admits frankly to what really worked and didn’t work, I 
find many of these tools to be suspect.” 

Another critical barrier highlighted by the survey is the lack of time 
available for lessons learned. Managers reported that they and their 
project staff are very busy conducting assigned project tasks, and 
little, if any, time is allotted for lessons learning. One manager 
noted that “It is time consuming to agree on correct lessons learned 
for a project and accurately describe the issue…in a way that is 
helpful to a project.” Managers also noted that there is reluctance to 
share negative lessons for fear that they might not be viewed as good 
project managers. According to some respondents, this appears to stem 
from a culture that sees lessons learned as negative, i.e., an 
admission of failure. For example, one manager noted that “People are 
never rewarded for telling about how they screwed-up and caused a 
problem/mistake…. This will continue to be a problem until a way is 
found to allow and encourage people to talk about their mistakes 
without feeling that they are risking their careers.” The cultural 
barriers to lessons learning identified above present a serious 
challenge for NASA and the agency may well be missing fundamental 
opportunities to share and apply knowledge toward future mission 
success. 

Suggested Improvements to Lessons Learning: 

While program and project managers identified a number of ways for 
improving lessons learning within NASA, mentoring was their first 
choice. One manager noted that the most effective lessons learned 
process results from the passing down of lessons from experienced 
people to those less experienced. We found that mentoring programs 
have been reinvigorated at a few centers in response to lessons 
learned from the faster, better, and cheaper changes as well as 
concerns that management expertise is being lost because of increasing 
retirements. Other suggestions for improvement included “storytelling” 
or “after-action reviews.” For example, NASA’s Academy of Program and 
Project Leadership asks senior program managers to share their 
knowledge through a series of short stories. Each story is 1 to 3 
pages and discusses a topic that will help program mangers to succeed. 
A recently completed story deals with the importance of knowing when 
to say “no” in a project. However, most managers in our survey did not 
identify the academy as a principal source for lessons learning. 
Another notable suggestion was that senior management should be 
accountable for the infusion or engineering of lessons into a project 
with some kind of a matrix/metric to validate that the lessons were 
actually incorporated. 

Managers also suggested a number of improvements regarding LLIS. As 
shown in table 3, suggested improvements were related to (1) 
developing a means to disseminate lessons from users, (2) improving 
the search capability, and (3) including more positive lessons. 
Currently with the LLIS, there is no automated dissemination of 
lessons, it is difficult to find relevant lessons, and there are few 
positive lessons included. 

Table 3: Program and Project Manager Observations and Suggestions to 
Improve LLIS: 

Category: Dissemination; 
Specific Suggestion: 
* For non-NASA employees, it is extremely difficult to be granted 
access on a project basis. Team members on my project must apply for 
and be granted access on an individual basis. The process for access 
is tedious to say the least and this discourages routine use of the 
LLIS.
* Send out lessons of the month.
* Include a bulletin each week in the center newsletters about a 
lesson learned.
* Send out alerts.
* Tailor the dissemination or notification to target opportunities for 
benefit rather than blanket notification. Too much data inhibits 
information. 

Category: Searchability; 
Specific Suggestion: 
* Have Mission Assurance Manager or Systems Engineer review LLIS on 
periodic basis & disseminate info where appropriate; this same 
individual could review failure/discrepancy reports for submission to  
LLIS.
* Separate safety and mishap from program, technical, etc., lessons 
learned. Provide possible separate systems for each category: safety, 
technical, theoretical, Commercial product usage, contracting, project 
management, etc.
* Better indexing - LLIS search usually does not return a high ratio 
of relevant information
* Improved search engine.
* Categorize the lessons into categories like the various technical 
disciplines and management disciplines, so that it would be easy to 
scan the ones you think are relevant to you. 

Category: More Positive Lessons; 
Specific Suggestion: 
* Need more "positive" lessons learned. They can be even more helpful 
to a project than the "negative" ones. Even if a project avoids 
"negatives", without "positives" they may not follow the most 
effective and efficient path to mission success.
* When NASA has a failure, there are all sorts of reports on what went 
wrong from independent review panels (e.g. Mars Reports). It seems 
there would definitely be some benefit from similar type reports on 
projects that were successful, pointing out what these projects have 
done right.
* Instead of focusing on the negative, focus on the positive things. 
There are some really well run projects. If people can look to great 
successes, instead of worrying about failures, I believe that they 
will try to emulate those successes. NASA has always been known as a 
risk taker because of what we do. We have no one else to look to, but 
within ourselves. I'm beginning to see something new at NASA - fear of 
taking a risk. As long as risks are calculated, and the 
project/program buys-in to taking that risk, it's not a bad thing. 

[End of table] 

In discussions with NASA officials, we found there was general 
agreement with the problems and suggested improvements for lessons 
learning that were identified in our survey. Officials indicated that 
lessons learning has taken on greater importance in recent years due 
to the implementation of more programs and projects under the FBC 
strategy and the continuing loss of agency expertise due to attrition. 
They acknowledge that LLIS has not been an effective mechanism for 
agency-wide sharing of lessons. Although the system is viewed as 
providing a useful repository for storing lessons, officials agreed 
with managers’ concerns about the difficulties involved in searching 
the system and finding relevant lessons, the inconsistent quality of 
information contained in the system, and the lack of lessons about 
positive project experiences. However, while program and project 
managers’ suggested improvements would help increase the usability of 
LLIS, they do not target some of the more fundamental problems 
hampering NASA’s ability to share lessons, such as the cultural 
barriers that persist. 

Chapter 3 Footnote: 

1. U.S. General Accounting Office, Survey of NASA’s Lessons Learned 
Process, [hyperlink, http://www.gao.gov/products/GAO-01-1015R] 
(Washington, D.C.: 2001). 

[End of section] 

Chapter 4: Creating an Environment for Lessons Learning through 
Knowledge Management: 

The limitations in NASA’s ability to share lessons learned point 
toward two underlying problems: cultural resistance to sharing 
knowledge and the lack of an effective strategic framework and 
management attention for overcoming such resistance. The same problems 
often hamper organizations not only in sharing lessons learned, but 
also in broader efforts to use knowledge as a tool for increasing 
productivity and fostering innovation. We found that successful 
industry and government organizations have learned to overcome such 
barriers by developing strategic plans that provide a framework for 
how knowledge management will be implemented and by securing the 
management support and commitment needed to bring these plans to 
fruition. Moreover, they employ incentives, processes, and systems 
designed to overcome resistance and other cultural impediments to 
knowledge sharing. Just as employing these practices enables leading 
organizations to maintain a competitive edge, doing so can better 
position NASA to perform its basic mission of exploring space, faster, 
better, and cheaper. 

Principles of Knowledge Management Could Help Mitigate Fundamental 
Weaknesses in Lessons Learning: 

We contacted practitioners of knowledge management in both government 
and industry to gain insight relative to how the application of 
knowledge management principles could mitigate the fundamental 
weakness in lessons learning identified by our survey. This effort 
included site visits and interviews of government and industry 
knowledge management practitioners, analysis of published case 
studies, and review of available literature. The outcome was the 
identification of common characteristics inherent in the design and 
implementation of knowledge management programs operated and 
maintained by government and commercial organizations. Literature 
searches and reviews of case studies also identified general 
principles deemed critical to the success of knowledge management. 

Knowledge Management Must Be Linked to a Business Plan: 

According to practitioners of knowledge management, the first and 
perhaps most important element of knowledge management is to put in 
place a strategic plan that makes a business case for knowledge 
management. This involves translating the abstract concept of 
knowledge management into a vision, with goals and a roadmap for 
sharing and using knowledge within the organization. It also involves 
developing a collective vision that is long term and a commitment by 
senior management that they will see the plan through. 

NASA has developed a strategic plan that identifies knowledge 
management goals and objectives the agency hopes to achieve. NASA, 
however, has not developed a business plan for achieving 
implementation of its vision and goals. Such a plan at a minimum 
should address the following: 

* roles and responsibilities; 

* knowledge needs and how they it relate to business processes; 

* the role of information technology; 

* a timetable for implementing for knowledge management; 

* resources needed for implementing a knowledge strategy; 

* cultural barriers to learning; 

* metrics needed for tracking and measuring results; and; 

* training. 

Senior Management Support is Key to Knowledge Management’s Success: 

While NASA senior management has shown its support for knowledge 
sharing and lessons learning, our survey nonetheless indicates that 
many program and project managers believe senior management support is 
lacking. Over one-third of survey respondents indicated that the 
effective utilization of lessons learning at NASA is inhibited because 
senior management lacks commitment to its use. One respondent stated 
that if management is serious about capturing and using lessons 
learned, then senior management must be seen and heard implementing 
the policy. 

Of the organizations we observed, most had senior management who saw 
the strategic value in using knowledge and communicated to their 
employees that sharing knowledge is critical to their success. 
[Footnote 1] At Ford Motor Company, the chief executive officer serves 
as a role model for knowledge sharing, personally writing weekly e-
mails to employees with comments on his thoughts about and experiences 
of the past week.[Footnote 2] Moreover, Ford executives provide 
sponsorship and support for knowledge management. They encourage 
employee participation and recognize employees who share information. 

An aerospace company believes that one of the best ways to disseminate 
lessons learned is to have management use them. This means that for 
the dissemination of lessons learned to be considered important by all 
the employees in the company, the executive team must understand and 
value them. This is accomplished by including as part of its executive 
training course items such as teaching managers how to file after 
action reports. 

Management of Knowledge Sharing Activities: 

Although NASA’s policies and procedures require continuous collection 
and sharing of lessons learned, NASA has not established a central 
management function to coordinate and direct its lessons learning 
processes and systems. Lessons learning activities are dispersed 
across several agency components. The Office of Safety and Mission 
Assurance has designated an administrator to maintain the LLIS; 
however, responsibility for LLIS is one of several assigned duties for 
the administrator. As a result, little effort is devoted to managing 
LLIS beyond keeping it operating. NASA has also established a team to 
coordinate recent knowledge management initiatives, but the team lacks 
authority to shape the direction or provide resources and tools to 
improve knowledge sharing. 

Our research indicates that organizations that have implemented 
knowledge sharing programs often designate a knowledge management 
steward. For example, a recent article on knowledge management notes 
that “... a job title at Philip Morris is knowledge champion, Monsanto 
has a director of knowledge management, and Dow Chemical has a 
director of intellectual assets management.”[Footnote 3] Once such a 
position has been created, the steward is then responsible for 
identifying needs and opportunities for improvement, obtaining and 
distributing resources, applying and enforcing policies, and 
conducting daily operations. At Ford, a best practices coordinator 
manages the knowledge-sharing program. Similarly, at the World Bank, a 
Knowledge Management Office leads an organization-wide effort at 
knowledge sharing. 

Investing in Knowledge Sharing Is Viewed as Important by Successful 
Organizations: 

NASA currently invests little in lessons learning. LLIS costs $50,000 
to $100,000 annually to operate. And, according to NASA officials, 
funding is insufficient to upgrade LLIS in any significant way. 
Although there is no set answer as to the appropriate dollar amount 
that should be devoted to knowledge management, our review shows that 
those organizations that dedicate resources tend to be more successful 
at sharing knowledge effectively. Many of the organizations we 
contacted reported significant investments in knowledge management 
activities. For example, the World Bank invests about 3 percent of its 
total administrative budget in knowledge management. These resources 
are used for information technology systems, training, and dedicated 
staff to facilitate knowledge sharing. 

A Corporate Culture That Encourages Knowledge Sharing Is a Key Element 
for Success: 

Although NASA acknowledges the importance of having an organizational 
culture that promotes teamwork and knowledge sharing, our survey 
indicated that significant cultural barriers within the agency inhibit 
an environment and culture that supports continuous learning. Our 
research indicates that practitioners of knowledge management foster 
an environment and culture that support continuous learning. Many 
knowledge management leaders agree that knowledge management is 90 
percent culture and 10 percent technology.[Footnote 4] A culture of 
knowledge sharing can be encouraged in many ways. 

Effective Knowledge Sharing Requires Adequate Time: 

A common complaint among NASA survey respondents is the lack of time 
for sharing lessons learned. Consequently, knowledge-sharing 
activities at NASA are seen as simply an additional burden on an 
already tight schedule. Yet, if an organization values knowledge 
sharing it will allow time for knowledge sharing. Knowledge can only 
be shared when employees are given adequate time, as well as 
established places where they can actually transfer knowledge. 

Formal And Informal Knowledge Sharing: 

Our survey indicated that much of the knowledge sharing at NASA is 
done on an informal basis. While talking informally with colleagues is 
one method of sharing information, it is not necessarily the most 
efficient way. Such informal exchanges do not allow the information to 
be further disseminated and do not necessarily allow the information 
to be validated. 

Information exchanges, however, are important because they allow 
people to discuss issues spontaneously. Many of the organizations that 
we examined instituted more formal arrangements for sharing 
information. Ford and the World Bank have established “communities of 
practice” as a method for sharing knowledge. They formed groups of 
employees who are bound together by shared expertise and a passion for 
joint enterprise.[Footnote 5] These communities serve as knowledge 
transfer programs because employees can share information regarding a 
given issue. At the World Bank there are approximately 120 communities 
of practice—-known as “thematic groups” arranged according to common 
interests, such as “urban poverty” or “urban services.” The urban 
services group, for example, focuses on improving the living 
conditions in slums around the world by capturing and sharing past 
experiences and adapting them to today’s experiences. 

Incentives for Knowledge Sharing: 

Recently, some activities at NASA centers have sought to find ways to 
encourage knowledge sharing. For example, scientists at NASA’s Langley 
Research Center are monetarily rewarded if knowledge they capture and 
share is re-used. Nevertheless, NASA has not developed a plan that 
details the principles, requirements, and architecture of how to 
implement its knowledge sharing strategy. While NASA realizes that 
rewards and recognition should be part of its initial priority area 
for knowledge management, the agency has not yet initiated any agency 
wide incentives for knowledge sharing. 

Most of the organizations that we examined have developed incentives 
for knowledge sharing. Ford rewards employees who submit lessons 
learned that are adopted by the company. Managers are also encouraged 
to share because they are evaluated annually on the basis of knowledge 
sharing. Similarly, the World Bank has made learning and knowledge 
sharing part of core behaviors covered in each employee’s performance 
evaluation. 

Information Is Deemed Valuable: 

NASA is less successful at infusing lessons learned because the 
sharing of lessons learned is not highly valued by some program and 
project managers. Successful knowledge sharing organizations share 
knowledge because they view it as critical to their success. At Ford, 
for example, every plant is responsible for making a 5 percent 
productivity increase each year. Employees refer to this as “task.” 
Ford’s knowledge-sharing database (Best Practice Replication system) 
provides managers with suggestions for improving efficiency, which in 
turn will enable managers to meet “task.” Knowledge is shared because 
it meets a critical need. Moreover, managers at Ford use the lessons 
provided by the Best Practice Replication system because they trust 
the information, and experts within the organization have checked the 
best practice in the database for accuracy.[Footnote 6] This is key to 
the Ford program. 

Information Technology is Important, but Should Not Be the Only 
Mechanism for Knowledge Sharing: 

A pitfall of many knowledge management programs is the assumption that 
a database will automatically lead to knowledge sharing. Indeed, 
NASA’s agency-wide effort at lessons learning is in practice limited 
to LLIS. At Ford, its electronic database is supplemented by other 
interactions. Not only do employees meet at community of practice 
meetings, but production engineers from the various plants also meet 
quarterly to discuss how each plant has implemented best practices. 
The World Bank not only uses a variety of databases to share 
information; employees also meet face-to-face at thematic group 
meetings and at knowledge fairs. 

Well-designed information systems and databases, however, are 
important to facilitate knowledge sharing, especially for 
organizations such as NASA that have employees located at multiple 
centers across the country. Organizations with effective knowledge 
management have designed their database systems with heavy end-user 
involvement. The individuals who are intended to use the databases 
should be involved in the development and implementation of the system 
if it is to be successful. In addition, organizations often use 
dissemination mechanisms to “push” important and relevant information 
to the right people rather than have users to search for the 
information. NASA’s LLIS requires users to search for possible useful 
lessons. In contrast, at the Department of Energy users receive 
automatic e-mail alerts regarding important lessons learned. 
Similarly, individuals at Ford receive new best practices every time 
they open the database system. Also, we found that organizations 
frequently dedicate “gatekeepers” to manage and monitor knowledge-
sharing databases in order to keep the information up to date and 
relevant. At Ford, a “focal point” is assigned to manage the best 
practices system at each plant. The focal point is usually a 
production engineer appointed by the plant manager. They receive the 
pushed messages and also enter best practices derived from their own 
plant. Similarly, at the Department of Energy a lessons learned 
coordinator maintains the lessons learned system at each site. 

Chapter 4 Footnotes: 

1. Thomas Davenport and Laurence Prusak, Working Knowledge: How 
Organizations Manage What They Know (Boston: Harvard Business School 
Press,1998). 

2. Ford Motor Company 1999 Annual Report. 

3. Verna Allee, Linking People, Learning and Performance - 12 
Principles of Knowledge Management, American Society for Training and 
Development, (2001). 

4. Directorate of eBusiness & Knowledge Management, Office of the 
Chief Information Officer, DOD Knowledge Management Primer, Department 
of Defense. 

5. Wenger, Etienne and William M. Snyder. “Communities of Practice: 
The Organizational Frontier.” Harvard Business Review. Jan/Feb. 
2000:139-145. 

6. Nancy M. Dixon, Common Knowledge: How Companies Thrive by Sharing 
What They Know, (Boston: Harvard Business School Press, 2000). 

[End of section] 

Chapter 5: Conclusions and Recommendations: 

The failures of the Mars Polar Lander and Climate Orbiter spacecraft 
raised concern that NASA was not learning from its past mistakes and 
applying lessons learned toward future mission success. Our survey of 
NASA program and project managers and subsequent analysis found that 
there are weaknesses in NASA’s processes, procedures, and systems for 
lessons learning and knowledge sharing. Program and project managers 
indicate that current processes and systems do not allow them to 
retrieve the right lesson at the right time. Managers also report that 
they are not very knowledgeable about lessons generated from other 
centers and programs. This is partly due to the fact that NASA’s LLIS 
is not widely used. As a result, there is no assurance that lessons 
learned from past mishaps or program successes are being applied to 
current programs and projects. 

NASA has recognized that it must improve its lessons learning and 
knowledge sharing capabilities. The mandate to deliver leaner, more 
demanding, and increasingly more complex missions has resulted in a 
greater dependency on creating multi-disciplinary teams, building 
alliances with contractors, and being able to quickly link to and 
learn from other agency activities. Following a series of mission 
failures and mishaps, the NASA Integrated Action Team reported that 
the continuous capture and application of project knowledge and 
lessons learned must become a core business process within the 
agency’s program and project management environment. More importantly, 
the team stated that program and project managers must regularly use 
knowledge management tools to apply previous lessons learned to their 
projects. 

NASA has taken steps to promote the capture and sharing of lessons and 
other knowledge within the agency through knowledge management 
practices. According to NASA officials, knowledge management has the 
potential to rectify many of the weaknesses identified by our survey 
through more effective management of the agency’s knowledge resources. 
Current knowledge management initiatives by NASA should lead to 
improvements in knowledge sharing, but the lack of a coordinated and 
well-supported effort will limit knowledge sharing agency-wide. 
Stronger commitment and efforts are needed to create an environment 
that encourages knowledge sharing. To achieve this environment, NASA 
must overcome communication barriers and create opportunities where 
open and candid communications are the norm and knowledge sharing is 
valued. In addition, time and resources must be dedicated in order for 
lessons learning and knowledge sharing to occur. Furthermore, 
improvements to information technology systems such as LLIS are needed 
to facilitate the collection, dissemination, and application of 
knowledge. 

Recommendations for Executive Action: 

NASA needs to strengthen its lessons learning in the context of its 
overall efforts to develop and implement an effective knowledge 
management program. Improvement of NASA’s lessons learning processes 
and systems can help to ensure that knowledge is gained from past 
experiences and applied to future missions. 

We recommend that the NASA administrator strengthen the agency’s 
lessons learning processes and systems by taking actions in the 
following areas: 

Strategic Planning: 

* Articulate the relationship between lessons learning and knowledge 
management through development of an implementation plan for knowledge 
management. 

Coordination: 

* Designate a lesson learned manager to lead and coordinate all agency 
lessons learning efforts. 

* Establish functional and technical linkages among the various center-
level and program-level lessons learning systems. 

Culture: 

* Develop ways to broaden and implement mentoring and “storytelling” 
as additional mechanisms for lessons learning. 

* Identify incentives to encourage more collection and sharing of 
lessons among employees and teams, such as links to performance 
evaluations and awards. 

LLIS Enhancement: 

Enhance LLIS by: 

* coding information and developing an easier search capability to 
allow users to identify relevant lessons; 

* including more positive lessons; 

* providing a means to disseminate key lessons to users; and; 

* soliciting user input on an ongoing basis. 

Performance Measurement: 

* Track and report on the effectiveness of the agency’s lessons 
learning efforts using objective performance metrics. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, NASA generally 
concurred with our recommendations for improving the agency’s lessons 
learned processes and systems. NASA stated that it must do a better 
job of communicating the various lessons learned sources to employees, 
improving mechanisms to link these sources, and ensuring appropriate 
training for employees in order to maximize lessons learning. NASA 
further indicated that it will develop plans to implement our report 
recommendations. NASA’s comments are reprinted in appendix I. 

[End of section] 

Appendix I: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration: 
Office of the Administrator: 
Washington. DC 20548-0001: 

January 14, 2002: 

Mr. Allen Li: 
Director: 
Acquisition and Sourcing Management: 
U.S. General Accounting Office: 
Washington, DC 20548: 

Dear Mr. Li: 

We appreciate the opportunity to review and provide comments to the 
draft report, entitled "NASA: Better Mechanisms Needed for Sharing 
Lessons Learned". We have reviewed the report, and although may not 
agree with the conclusions that correlate Mars Mission failures with 
deficiencies in the application of lessons learned, we are essentially 
in agreement with the recommendations for improving our Lessons 
Learned Process. These recommendations are generally on the mark, and 
their intent will be implemented. There are several initiatives 
currently underway, and implementation plans are being developed. 

We continue to evaluate NASA processes and will do the same for 
lessons learned. Major revisions to NASA Procedures and Guidelines 
(NPG) 7120.5, NASA Program and Project Management Processes and 
Requirements, are in the final stages and include more clarity in the 
use and scope of the Lessons Learned Information System (WS). Since 
NASA uses numerous sources for lessons learned, WS will no longer be 
referred to as the primary source, but one of many. 

Programs and projects benefit from lessons learned day to day in many 
different forms. Some of these include status reporting, programmatic 
and technical reviews, and functional staff meetings. The more formal 
and widespread forms include technical standards and updating NPG's 
and other documentation. The various NASA training programs also 
incorporate lessons learned into their curricula, including 
"storytelling" by current and retired NASA program and project 
managers. One forum Is designed specifically for sharing lessons 
learned and is titled Project Management Shared Experiences Program. 
The overall objective of this program is to provide a forum to 
understand key initiatives influencing NASA project management and for 
project people to share knowledge, experiences, and creative 
approaches to project management. 

NASA must do a better job of communicating the various lessons learned 
sources, improving mechanisms by which these various sources are 
linked, and encouraging the proper training for employees in order to 
maximize the ability of current activities to draw upon the lessons of 
others. We plan continued improvement in these areas. 

We appreciate the GAO assessment effort and subsequent recommendations 
on improving NASA's lessons learned system. This is a good example of 
agencies teaming for improvement. 

At this time, we provide the following response, if the 
recommendations contained in the draft report are retained in the 
final report. 

GAO Recommendation 1: Articulate the relationship between lessons 
learning and knowledge management through an implementation plan for 
knowledge management. 

NASA Response: Concur. NASA has developed a "Strategic Plan for
Knowledge Management" and "Near-Term Implementation Opportunities for
Knowledge Management" which articulates the relationship between 
lessons learning and knowledge management. The Near-Term Implementation
Opportunities plan will be revised to strengthen the linkage between 
knowledge management activities and processes for lessons learning. 
Some of the implementation activities are underway and others will be 
initiated as the overall implementation progresses. 

GAO Recommendation 2: Designate a lessons learned manager to lead and
coordinate all agency lessons learning efforts. 

NASA Response: Concur. NASA is planning on designating a lessons learned
manager. 

GAO Recommendation 3: Establish functional and technical linkages 
among the various center-level and program-level lessons learning 
systems. 

NASA Response: Concur. As NASA progresses in the implementation of
knowledge management, methods to link the multiple lessons learning 
systems will be pursued and incorporated. 

GAO Recommendation 4: Develop ways to broaden and implement mentoring
and "storytelling" as additional mechanisms for lessons learning. 

NASA Response: Concur. NASA will continue to hold several project
management knowledge sharing forums, the annual publication of several 
issues of the online Academy Sharing Knowledge Journal, and the 
development of additional case studies and online knowledge sharing 
resources. Efforts are also underway to improve the systematic 
identification and utilization of NASA leaders as mentors
and teachers, and processes and procedures will be completed in 2002. 

GAO Recommendation 5: Identify incentives to encourage more collection 
and sharing of lessons among employees and teams, such as links to 
performance evaluations and awards. 

NASA Response: Concur. In October 2001, the Agency began a new study--
"Aligning Recognition and Awards with the Agency's Values and 
Expectations." A significant issue of this study scope is "What 
recognition practices in a technology-based organizational setting 
would drive and support particular initiatives in teaming, knowledge 
sharing, and organizational learning." Recommendations resulting from 
this study will be proposed by the end of this fiscal year. 

GAO Recommendation 6: Enhance LLIS by coding information and 
developing an easier search capability to allow users to identify 
relevant lessons, including more positive lessons, providing a means 
to disseminate key lessons to users; and soliciting user input on an 
ongoing basis. 

NASA Response: Concur. Since the inception of the LLIS in 1995 we have
offered users an extremely easy-to-use "keyword" search capability for 
all data in the LLIS that functions in the same way as an Internet 
search browser such as "Yahoo" or "Lycos". Since 1999, we have also 
offered our users an advanced search capability. Currently, this 
advanced search capability allows users to search on several 
combinations of criteria (NASA Center/Facility, Enterprise, and
Program/Project life-cycle stage), including keyword search. This 
collection of search combinations gives our users unparalleled 
searching flexibility. 

Recent improvements now provide the capacity to disseminate key 
lessons to users with our new "push" capability. NASA will continue to 
study improvement possibilities for the LLIS to ease the system's use. 

The LLIS web site offers several means by which users can presently 
contact us to provide comments or feedback on the system. We have a 
"submit a suggestion about the LLIS" link and a "comment on a lesson" 
link. We also have a hotline/help desk telephone number: 301-286-LLIS. 

GAO Recommendation 7: Track and report on the effectiveness of the 
agency's lessons learning efforts using objective performance metrics. 

NASA Response: Concur. Although challenging, NASA is pursuing the
appropriate metrics to measure the effectiveness of our lessons 
learning efforts. We expect to have this in place later this year. 

Thank you again for the opportunity to review this draft report. 
Should you have any questions, please call Keith Hudkins, Deputy Chief 
Engineer, at (202) 358-1823. The point of contact for this audit in 
the Office of the Chief Engineer is Gregory Robinson; he can be 
reached at (202) 358-2541.  

Sincerely, 

Signed by: 

Michael D. Christensen: 
Associate Deputy Administrator for Institutions: 

cc: 
AE/Mr. Keegan: 
AE/Mr. Hudkins: 
AE/Mr. Robinson: 
AO/Mr. Holcomb: 
AO/Ms. Hibbert: 
F/Ms. Novak: 
FT/Ms. Acoveno: 
FP/Ms. Peterson: 
G/Mr. Stephens (Acting): 
JMr. Sutton: 
JM/Ms. Flickinger: 
JM/Mr. Werner: 
L/Mr. Bingham (Acting): 
M/Mr. Gregory (Acting): 
Q/Dr. Greenfield (Acting): 
R/Mr. Venneri: 
U/Dr. Olsen (Acting): 
Y/Dr. Asrar: 
ARC/Dr. McDonald: 
DFRC/Mr. Petersen: 
GRC/Mr. Campbell: 
GSFC/Mr. Diaz: 
JSC/Mr. Estess (Acting): 
KSC/Mr. Bridges: 
LaRC/Dr. Creedon: 
MSFC/Mr. Stephenson: 
SSC/Mr. Craig (Acting): 
JPUDr. Elachi: 

[End of section] 

Appendix II: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Allen Li, (202) 512-4841: 
John Oppenheim, (202) 512-3111: 

Acknowledgments: 

In addition to those named above, Christina Chaplain, Diana 
Dinkelacker, James Elgas, Jose Ramos, Carl Ramirez, and Lorene Sarne 
made key contributions to this report. 

[End of section] 

GAO’s Mission: 

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analysis, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO’s commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO’s Web site [hyperlink, 
http://www.gao.gov] contains abstracts and fulltext files of current 
reports and testimony and an expanding archive of older products. The 
Web site features a search engine to help you locate documents using 
key words and phrases. You can print these documents in their entirety, 
including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
[hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail 
alert for newly released products” under the GAO Reports heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. General Accounting Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, NelliganJ@gao.gov: 
(202) 512-4800: 
U.S. General Accounting Office: 
441 G Street NW, Room 7149:
Washington, D.C. 20548: