This is the accessible text file for GAO report number GAO-02-631 
entitled 'Missile Defense: Knowledge-Based Decision Making Needed to 
Reduce Risks in Developing Airborne Laser' which was released on July 
16, 2002.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products’ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.



Report to the Chairman, Subcommittee on National Security, Veterans’ 

Affairs, and International Relations, Committee on Government Reform, 

House of Representatives:



United States General Accounting Office:



GAO:



July 2002:



Missile Defense:



Knowledge-Based Decision Making Needed to Reduce Risks in Developing 

Airborne Laser:



Missile Defense:



GAO-02-631:



Contents:



Letter:



Results in Brief:



Background:



Original Cost and Schedule Goals Are Based on Inadequate Knowledge:



New Strategy Incorporates Some Knowledge-Based Practices, but 

Additional Practice Would Reduce Program Risk:



Conclusion:



Recommendations for Executive Action:



Agency Comments and Our Evaluation:



Scope and Methodology:



Appendix I: Technology Readiness Level Assessment Matrix:



Appendix II: Comments from the Department of Defense:



Appendix III: GAO Contact and Staff Acknowledgments:



GAO Contact:



Acknowledgments:



Figures:



Figure 1: Airborne Laser Aboard Boeing 747 Aircraft:



Figure 2: Current Airborne Laser Technology Readiness Levels:



Figure 3: The Knowledge-Based Process:



Letter:



July 12, 2002:



The Honorable Christopher Shays

Chairman, Subcommittee on National Security,

 Veterans’ Affairs, and International Relations

Committee on Government Reform

House of Representatives:



Dear Mr. Chairman:



In 1996, the Department of Defense, through the Air Force, launched an 

acquisition program to develop and produce a revolutionary laser weapon 

system. The system, known as the Airborne Laser because it is being 

developed for installation in a modified Boeing 747 aircraft, is 

intended to destroy enemy ballistic missiles almost immediately after 

their launch (in the so-called “boost phase”) before they pose a threat 

to civilian populations and military assets. The Air Force originally 

estimated development costs at $2.5 billion and projected fielding of 

the system to begin in 2006. However, by August 2001, the Air Force 

determined that maturing the technologies and developing the system 

would cost about 

50 percent more and take another 4 years. The development cost estimate 

increased to $3.7 billion, and the fielding date slipped to 2010.



Against this backdrop of cost increases and schedule delays, the 

Department of Defense transferred responsibility for the Airborne Laser 

in October 2001 from the Air Force to the Ballistic Missile Defense 

Organization, which shared responsibility with the armed services for 

developing ballistic missile defense systems. Subsequently, in January 

2002, the Secretary of Defense announced broad changes to the 

department’s strategy for developing and acquiring missile defense 

systems. Specifically, the Defense Secretary designated the Ballistic 

Missile Defense Organization as the Missile Defense Agency and granted 

the agency expanded responsibility and authority. The Secretary 

directed the agency to develop an integrated system with various 

elements that have the capability to attack enemy missiles in all 

phases of their flight, transition responsibility for the production 

and fielding of systems to the individual services, and encourage 

incremental improvements by inserting new technologies through a series 

of block upgrades.[Footnote 1] The Airborne Laser is one of many 

systems affected by the new strategy.



Concerned about significant cost and schedule problems associated with 

developing the Airborne Laser, you asked us to determine (1) why the 

systemís development is costing more and taking longer than the Air 

Force originally estimated and (2) whether the Missile Defense Agencyís 

new strategy for developing the Airborne Laser incorporates the 

practices that characterize successful programs.



Results in Brief:



The Air Force was unable to meet the Airborne Laser’s original cost and 

schedule goals because it established those goals before it fully 

understood the level of effort that would be required to develop the 

critical system technology needed to meet the user’s requirements. When 

the Air Force launched the Airborne Laser acquisition program, 

Department of Defense policy required that program cost and schedule 

goals be established[Footnote 2]. In 1996, at program launch, the Air 

Force did not have enough knowledge about the technology challenges 

facing the program. As a result, the Air Force underestimated the 

complexity of the engineering task at hand and misjudged the amount of 

time and money that the program would need. Some critical technologies 

that the system’s design depends upon remain immature, making it very 

difficult, even today, for analysts to establish realistic cost and 

schedule goals.



The Missile Defense Agency’s new strategy for developing the Airborne 

Laser incorporates some knowledge-based practices that characterize 

successful programs. For example, one practice that the agency 

implemented is a requirements process that gives the agency the 

flexibility to develop a system that has some capability without being 

held to requirements that cannot be met with currently available 

technology. A second knowledge-based practice is the provision of 

additional time and facilities for testing. Increased testing allows 

agency officials to reach a better understanding of the capabilities of 

the technology so that they can establish more realistic requirements 

and ultimately more accurate estimates of the time and money needed to 

meet those requirements. A third practice is the collection of the 

types of information needed to determine whether the technology is “in-

hand” to give war fighters an Airborne Laser with some, if not all, 

desired capabilities. For example, the agency intends to compare 

developed capabilities with data derived from intelligence sources on 

the likely launch points and types of missiles that the system could 

encounter.



However, the agency has not established knowledge-based decision points 

and associated criteria for moving forward from technology development 

to product development and on to production. Separating technology 

development from product development has been a critical determinant 

for successful program outcomes. Without decision points and criteria, 

the agency risks beginning new and more costly activities before it has 

the knowledge to determine the money and time required to complete the 

activities and whether additional investment in those activities is 

warranted. Also, the agency risks beginning the activities before it 

has the knowledge to complete them without the need for expensive 

rework.



We are making recommendations that are intended to make the Missile 

Defense Agency’s acquisition process more disciplined and provide 

better information for decision makers as additional investments in the 

Airborne Laser are considered.



In commenting on a draft of this report, the Department of Defense 

partially concurred with our recommendations. The department stated 

that Secretary of Defense direction is not needed to implement our 

recommendations, the Missile Defense Agency’s acquisition process for 

ballistic missile defense already uses tailored versions of the 

knowledge-based practices recommended by us, and the agency intends to 

expand the use of knowledge-based criteria in the future. The Missile 

Defense Agency’s acquisition process separates acquisition into three 

phases--development, transition, and production. While the process 

definitely separates product development from production, it clearly 

does not separate technology development from product development. 

Also, it does not establish the knowledge-based criteria characteristic 

of successful programs at any of these decision points. Because we have 

not seen evidence in the Airborne Laser’s strategy that such decision 

points and criteria are in place, we have retained our recommendations.



Background:



The effort to develop the Airborne Laser is based on over 25 years of 

scientific development in the Departments of Defense and Energy. It 

evolved primarily from Airborne Laser laboratory research to develop 

applications for high-energy lasers. This research culminated in a 

demonstration that showed that a low-power, short-range laser was 

capable of destroying a short-range air-to-air missile. Although this 

demonstration was considered militarily insignificant because of the 

laser’s low power and short range, it did succeed in identifying 

technologies that were necessary for the development of an operational 

Airborne Laser system. The research showed that an operational system 

would need optics that could compensate for the atmospheric turbulence 

that weakens and scatters a laser beam, optical devices that could 

withstand the heat produced by a high-energy laser without the added 

weight of water-cooling devices, and a new chemical laser with higher 

energy levels that would produce a stronger laser beam.



In 1996, the Air Force launched the Airborne Laser program to develop a 

defensive system that could destroy enemy missiles from a distance of 

several hundred kilometers. Engineers determined that if they were to 

meet this requirement, the system would need a 14-module[Footnote 3] 

oxygen iodine laser. They also determined that the system would need a 

beam control/fire control assembly that could (1) safely move the laser 

beam through the aircraft, (2) shape the beam so that it was not 

scattered or weakened by the atmosphere, and (3) hold the beam on 

target, despite the movement of the aircraft. In addition, engineers 

determined that the system would need a battle management and control 

system capable of planning and executing an engagement.



The Air Force planned to have the science and technology community 

develop extensive knowledge about the laser and beam control/fire 

control technologies before it launched an Airborne Laser acquisition 

program. However, according to the retired manager of the science and 

technology project, the budgets for technology efforts were limited, 

and the science and technology community could not fund the technology 

maturation effort. The Air Force knew that a program office was more 

likely to command the large budget needed to fully mature technologies, 

so it launched an acquisition program and assigned the program manager 

responsibility for both technology and product development. The program 

manager planned to demonstrate critical Airborne Laser technologies by 

first building a six-module version of the oxygen iodine laser, 

installing it along with other system components aboard a Boeing 747 

aircraft (see 

fig. 1), and testing the capability of this scaled system in system-

level flight tests. The tests would conclude in 2003 with an attempt to 

shoot down a short-range ballistic missile target at a distance of 100 

kilometers. If this final test were successful, the Airborne Laser 

would have moved into product development.



Figure 1: Airborne Laser Aboard Boeing 747 Aircraft:



[See PDF for image]



Source: Airborne Laser Program Office.



[End of figure]



Original Cost and Schedule Goals Are Based on Inadequate Knowledge:



The Air Force launched the Airborne Laser acquisition program and 

identified cost and schedule goals before officials had the knowledge 

to make realistic projections. In 1996, when the program was launched, 

Department of Defense regulation 5000.2 required, and still requires 

today, that when a military service initiates a major acquisition 

program, it must establish cost and schedule goals. However, the Air 

Force could not make realistic estimates when it began the program 

because it had no way of knowing how much engineering effort would be 

needed to complete the development of technology critical to the 

system. Even today, some critical technologies that the system’s design 

depends upon remain immature, making it very difficult for analysts to 

determine how long it will take and how much it will cost to develop 

and produce the system.



Technologies Were Immature at Program Launch:



At the time the Airborne Laser program was launched, the laser and beam 

control/fire control technologies needed to develop the Airborne Laser 

system was immature. The Department of Defense’s science and technology 

community was actively researching and developing the laser and had 

produced a weak beam in a laboratory setting--but this major component 

had not reached the level of maturity needed to proceed into product 

development. The technology necessary to develop the beam control/fire 

control was even less advanced. Most of the scientists’ work was 

limited to analytical studies, wherein a few tests of laboratory 

hardware were linked together to work somewhat like the intended 

component.



Because technology development is a process of discovery, the Air Force 

soon learned that there were too many unknowns regarding the 

development of Airborne Laser technology to make good cost and schedule 

estimates. As the technology development progressed, unanticipated 

technical challenges affected the program’s cost and schedule. 

Department of Defense analysts reported that the Airborne Laser program 

experienced cost and schedule growth because the program and its 

contractors underestimated the complexity of 

(1) designing laser components, (2) the system’s engineering analysis 

and design effort, and (3) engineering the system to fit on board the 

aircraft. As system development progressed and the Air Force gained a 

better understanding of the technical complexity of the system, the Air 

Force increased its cost and schedule estimates.



Some Critical Technologies Remain Immature:



The Air Force has made some progress in developing the Airborne Laser’s 

critical technologies, but many remain immature. We asked the Airborne 

Laser program office to determine the technologies most critical to the 

Airborne Laser system and to use technology readiness levels[Footnote 

4] to assess the maturity of each. The officials determined that if the 

Airborne Laser is to meet the requirements established by the war 

fighters, then engineers must mature technologies in six areas, all of 

which are needed to successfully design the system. These technologies 

are:



* devices that stabilize the laser system aboard the aircraft so that 

the beam can be maintained firmly on the target,



* optics--mirrors and windows--that focus and control the laser beam 

and allow it to pass safely through the aircraft,



* optical coatings that enhance the optics’ ability to pass laser 

energy through the system and to reflect the laser energy,



* hardware that works in tandem with computer software to actively 

track the target missile,



* devices that measure atmospheric turbulence and compensate for it so 

that it does not scatter or weaken the laser beam, and:



* safety systems that automatically shut down the high energy laser in 

the event of an emergency.



At our request, the program office also assessed the maturity of the 

oxygen iodine laser.



As figure 2 shows, program officials assessed the optical coatings at 

level five and the safety systems, atmospheric compensation, and 

target-tracking components at level six. At technology readiness level 

five, the technology being tested is incorporated into hardware whose 

form and fit are coming closer to that needed for an operational 

component and integrated with reasonable realistic supporting elements 

so that the technology can be tested in a simulated environment. At 

level six, the technology is incorporated into a prototype and tested 

in a high-fidelity laboratory environment or in a simulated operational 

environment. The program officials identified the optics and 

stabilizing devices as the least mature--at level four. At this level, 

engineers have shown that a technology is technically feasible but have 

not shown whether the technology will have the form, fit, or function 

required in the operational system. We agreed with all but one of the 

program officials’ assessments for these technologies.



Figure 2: Current Airborne Laser Technology Readiness Levels:



Note: Appendix I contains information describing technology readiness 

levels.



[See PDF for image]



Source: GAO’s analysis.



[End of figure]



Our one disagreement centered on the maturity of the laser component of 

the system. While the program office assessed it at a technology 

readiness level of six, we consider the laser technology to be at level 

four because tests have been conducted only for a one-module laser in a 

controlled laboratory environment using surrogate components. For 

example, the tests used a stable laser resonator, rather than the 

unstable resonator that will be used in system-level flight 

tests.[Footnote 5] We also found that during tests of the one-module 

laser, the resonator was operating in multimode rather than single-

mode.[Footnote 6] The resonator in the operational system will operate 

in single mode. Furthermore, the chemical storage and delivery 

subcomponents used in these tests were not representative of those that 

will be incorporated into the system’s design. According to program 

office officials, conducting a more realistic test would have cost time 

and money that were not available.



Documents summarizing the tests of the one-module laser stated that the 

tests were successful in reducing the technical risks associated with 

the one-module system but that a new set of technical risks linked with 

developing a multimodule system must still be addressed during testing 

of the six-module system. In our opinion, the program office will 

demonstrate the laser technology in a relative environment (technology 

readiness level six) when the six-module system is integrated and 

successfully tested at full power within the high-fidelity laboratory 

environment of the Airborne Laser Systems Integration Laboratory, 

currently under construction at Edwards Air Force Base, California. 

According to the program office, this type of demonstration will not 

occur until February 2003.



New Strategy Incorporates Some Knowledge-Based Practices, but 

Additional Practice Would Reduce Program Risk:



The Missile Defense Agency’s new strategy for developing the Airborne 

Laser incorporates some of the knowledge-based practices that 

characterize successful programs, but the agency would benefit from 

adopting another that would add greater discipline to its acquisition 

process. The new strategy allows more flexibility in setting 

requirements, makes time and facilities available to mature and test 

the critical technologies, and collects information needed to match the 

war fighters’ requirements to demonstrated technology. However, the 

agency has not established decision points with associated knowledge-

based criteria for moving forward from (1) technology development to 

system integration, (2) system integration to system demonstration, and 

(3) system demonstration to production. At each of these points, the 

agency would stop to assess its knowledge and decide whether investment 

in the program’s next phase is warranted.



New Strategy Introduces Knowledge-Based Practices:



The first new practice allows the Missile Defense Agency to refine 

requirements on the basis of the results of system engineering. The 

Department of Defense ordinarily faces significant hurdles in matching 

requirements to resources. The fundamental problem is twofold. First, 

under the department’s traditional process, requirements must be set 

before a program can be approved and a program must be approved before 

the product developer conducts systems engineering. Second, the 

competition for funding encourages requirements that will make the 

desired weapon system stand out from others. Consequently, many of the 

department’s product development programs include unrealistic 

requirements set by the user before the product developer has conducted 

the system engineering necessary to identify the time, technology, and 

money necessary to develop a product capable of meeting requirements.



A second practice that is likely to improve the Airborne Laser’s 

development is making the time and facilities available to mature and 

test critical technologies. To implement this practice, the agency 

increased the time available to test the six-module laser system and is 

building a new test facility. Instead of following the Air Force’s plan 

to complete system-level flight tests of the six-module system in the 

last quarter of fiscal year 2003, the agency has delayed the 

demonstration to the first quarter of fiscal year 2005. This delay will 

allow additional time to learn from and correct problems discovered 

during system-level tests that are scheduled to begin in the last 

quarter of fiscal year 2003 and end with the fiscal year 2005 

demonstration. In addition, the agency plans to increase the Airborne 

Laser’s ground-testing capability by awarding a contract in 2003 for 

what the agency is calling an “iron bird,” which is essentially an 

aircraft hull with installed laser equipment. The “iron bird” is 

expected to allow testing of a fully integrated Airborne Laser system 

on the ground so that technologies for future blocks can be evaluated 

before being installed in an aircraft.



The information gained from testing informs the requirements process. 

Because testing allows developers to gauge the progress being made in 

translating an idea into a weapon system, it enables the developer to 

make a more informed decision as to whether a technology is ready to be 

incorporated into a system’s design. With this knowledge, the developer 

can determine whether the technology is so important to the system’s 

design that additional time and money should be spent to mature the 

technology or whether the system’s initial performance requirements 

should be reduced.



A third practice that the agency plans to adopt is matching 

requirements to available technology. According to the Missile Defense 

Agency’s Technical Director, the agency defines the war fighters’ 

requirement as a system that has the capability to destroy some threat 

ballistic missiles during their boost phase at a range representative 

of an operational scenario. The Technical Director told us that the 

agency will attain the knowledge to determine if it has the technology 

in-hand to meet this requirement by examining each block’s capabilities 

during simulated and system-level flight test and comparing those 

capabilities with data derived from intelligence sources on the likely 

launch points and types of missiles that the system could encounter. 

Our previous work with successful development programs shows that once 

the technology is in-hand to meet the customer’s requirements, the 

developer can make more accurate initial estimates of the cost and time 

needed to develop and produce an operational system.



Successful Developers Recognize Need for Knowledge-Based Decision 

Points:



Successful developers have instilled discipline in their acquisition 

processes by requiring that certain criteria for attaining knowledge 

are met as an acquisition program moves forward. (See fig. 3.) They 

recognize that the focus and cost of activities change over time and 

that less rework is required if all activities with the same focus are 

completed before beginning other activities.



Figure 3: The Knowledge-Based Process:



[See PDF for image]



Source: GAO’s analysis.



[End of figure]



In successful development programs, decisions are made when the 

knowledge is available to support those decisions. The first decision 

point, or knowledge point, occurs when the focus of a developer’s 

activities changes from technology development to system integration--

the first phase of product development. The criterion for deciding to 

move forward is having the knowledge to match requirements and 

available resources (time, technology, and funds). The second knowledge 

point occurs between system integration and system demonstration when 

the developer has successfully integrated subsystems and components 

into a stable design that not only meets the customer’s performance 

requirements but also is optimized for reproducibility, 

maintainability, and reliability. The decision criterion used here is 

usually having completed about 90 percent of the engineering drawings. 

The third knowledge point separates system demonstration from 

production. The decision to invest in production is generally based on 

a determination that the product performs as required during testing 

and that the manufacturing processes will produce a product within 

cost, schedule, and quality targets.



The cost of a program’s activities increases as it moves closer to 

production. In commercial acquisitions, product development is 

typically much more costly than technology development. During 

technology development, small teams of technologists work to perfect 

the application of scientific knowledge to a practical problem. As 

product development begins, developers begin to make larger investments 

in human capital, bringing on a large engineering force to design and 

manufacture the product. In addition, product development requires 

significant investments in facilities and materials. These investments 

increase continuously as the product approaches the point of 

manufacture. In fact, industry experts estimate that identifying and 

resolving a problem during product development can cost 10 times more 

than correcting that problem during technology development and that 

correcting the problem during manufacturing is even more costly.



Knowledge-Based Decisions Missing from Airborne Laser Strategy:



We examined the Airborne Laser’s acquisition strategy and determined 

that it does not include decision points at which officials would use 

knowledge-based criteria to determine if the program is ready to move 

from technology development to system integration, system integration 

to system demonstration, and system demonstration to production. We 

found that the agency’s process has three phases: development, 

transition, and production.



* Development includes all developmental activities and system-level 

demonstrations of military utility.



* Transition will involve preparation of the operational requirements 

document by the appropriate armed service and conducting operational 

testing.



* Production will involve producing and fielding the final weapon 

system.



The agency’s strategy also calls for developing the Airborne Laser 

incrementally, rather than trying to initially develop a system with 

all desired capabilities. In the near term, the agency plans to 

complete the six-module laser system aircraft, now known as block 

2004,[Footnote 7] and use it to demonstrate critical Airborne Laser 

technologies. Beginning in March 2003, the agency intends to begin 

developing another demonstration aircraft, known as block 2008, which 

will incorporate new capabilities and technologies. The Airborne Laser 

program manager told us that blocks 2004 and 2008 are primarily test 

assets for the purpose of technology demonstration. While some of the 

block 2008 activities are focused on improving subsystems and 

components, such as reducing the weight of laser components and 

improving optics, other activities are focused on the integration of 

these pieces into a block 2008 design.[Footnote 8]



The agency expects to develop subsequent blocks, or system 

configurations to introduce additional capabilities. If system-level 

tests show that any one of these configurations performs at a level 

that merits fielding, the Air Force will prepare a requirements 

document based on the configuration’s demonstrated capabilities and 

make plans for operational testing and production. This “baseline” 

capability would be improved in subsequent blocks as more advanced 

technology becomes available and as the threat warrants.



We did not find that the agency’s strategy includes a disciplined 

process that separates technology development, system integration, 

system demonstration, and production with decision points supported by 

knowledge-based criteria. Instead, the agency has put in place a 

decision point for moving from the development to the transition phase. 

According to the agency’s strategy, when the agency determines that it 

has the technology in-hand to produce a system that merits fielding, it 

will begin to transition the system over to the appropriate military 

service. Also, at the end of the transition phase, a system would enter 

the formal Department of Defense acquisition process at Milestone C--

the point at which the decision is made to enter low rate initial 

production. We did not find, however, an established set of decision 

points with associated criteria that would enable the agency to make a 

knowledge-based decision on whether to invest in system integration 

and, subsequently, system demonstration and production. That is, even 

though the agency might know that it has the technology in-hand to 

develop a useful military capability, it has not established a first 

decision point where it would determine the cost and time needed to 

move the program forward and whether the program should proceed into a 

system integration phase during which the design would be matured and 

optimized for reproducibility, maintainability, and reliability. 

Neither does the agency’s strategy include a second decision point that 

would allow agency officials to use the knowledge they have attained 

regarding the design’s maturity to determine whether to invest further 

to demonstrate that the system meets requirements and that 

manufacturing processes are in place to repeatedly produce a quality 

product. Only after the agency successfully moves the program through 

all of these decision points and successfully demonstrates the system’s 

capabilities and manufacturing processes would the agency’s production 

decision be fully knowledge based. Without this disciplined process, 

the agency would be accepting greater cost and schedule risks and is 

much less likely to realize the full potential benefits of its new 

approach to developing missile defense systems.



Conclusion:



The revolutionary nature of missile defense weapon systems demands 

cutting-edge technology. Although there is no one approach that ensures 

that a developer can deal successfully with the unknowns inherent in 

developing a product from such technology, the knowledge-based process 

has proven to yield good results within cost and schedule estimates. 

The Missile Defense Agency has implemented practices that are part of 

the knowledge-based approach, and these practices are likely to improve 

the agency’s ability to gather the knowledge it needs to develop an 

Airborne Laser capability acceptable to the war fighter. However, the 

agency has the opportunity to make its acquisition process more 

disciplined. By establishing knowledge-based decision points at key 

junctures, the agency would be in a better position to decide whether 

to move from one development phase to the next. Also, the agency would 

be better able to hold system developers accountable for planning all 

of the activities required to develop a quality product, approaching 

those activities in a systematic manner so that no important steps are 

skipped and problems are resolved sooner rather than later, and making 

cost and schedule projections when they have the knowledge to make 

realistic estimates. With this disciplined process in place, the agency 

is much more likely to achieve a needed capability for the war fighter 

within established cost and schedule goals.



Recommendations for Executive Action:



To make its acquisition process more disciplined and provide better 

information for decision makers as additional investments in the 

Airborne Laser are considered, we recommend that the Secretary of 

Defense direct the Director of the Missile Defense Agency to establish 

decision points separating technology development from system 

integration, system integration from system demonstration, and system 

demonstration from production. For each decision point, we recommend 

that the Secretary instruct the Director to establish knowledge-based 

criteria and use those criteria to determine where additional 

investments should be made in the program.



Agency Comments and Our Evaluation:



In commenting on a draft of this report, the Department of Defense 

partially concurred with our recommendations (see appendix II). The 

department stated that Secretary of Defense direction is not needed to 

implement our recommendations, the Missile Defense Agency’s acquisition 

process for ballistic missile defense already uses tailored versions of 

the knowledge-based practices recommended by us, and the agency intends 

to expand the use of knowledge-based criteria in the future.



The Department of Defense has not fully implemented the knowledge-based 

process recommended in our reports. Effective product development 

depends on gaining sufficient knowledge about technology, design, and 

manufacturing processes at key points in a system’s development. At 

those points, using metrics--such as technology readiness levels to 

measure the maturity of technology--that are commonly understood allow 

informed trade-offs to be made between resources, including cost and 

time, and performance. We have found that product development 

activities, such as building engineering prototypes of an integrated 

system and then demonstrating that the system can be manufactured to 

acceptable cost and quality standards, are ineffective unless the 

technologies needed to meet the product’s intended capabilities are 

fully matured and ready for system integration. Virtually every world-

class product developer we have spoken with agrees with this.



The Airborne Laser program does not appear to have established this 

type of decision-making process. The Missile Defense Agency appears to 

have set up a development phase that combines maturing technologies 

with establishing a stable design. It does not include any visible 

decision points or standards to clearly indicate when technology 

development is concluded and system integration work to establish a 

design begins. Thus, it appears to us that this acquisition process 

forces the agency to manage significant risk from immature technologies 

simultaneously with trying to build a stable product design during this 

phase. Further, separating system integration from system demonstration 

and system demonstration from production and using common metrics in 

deciding to move forward will enhance the future likelihood that 

decisions on the Airborne Laser will be cost-effective. Such a process 

will also enhance decision-makers’ ability across the range of missile 

defense elements by facilitating comparisons across elements. 

Therefore, we have retained our recommendations.



Scope and Methodology:



To address our objectives, we reviewed the contractor’s monthly cost 

performance reports, Defense Contract Management Agency analyses of 

those reports, and Defense Acquisition Executive Summaries and Selected 

Acquisition Reports prepared by the Airborne Laser program office. We 

also discussed cost and schedule problems with Airborne Laser program 

officials, Kirtland Air Force Base, New Mexico; and contractor 

officials at the Boeing Company, Seattle, Washington; Lockheed Martin, 

Sunnyvale, California; and TRW, Los Angeles, California. In addition, 

we obtained a technology readiness level analysis of the system’s 

critical technologies from the Airborne Laser program office. We 

compared this analysis with information obtained during our prior 

review to determine if progress had been made in maturing the critical 

technologies to higher technology readiness levels. We obtained 

detailed briefings from program office personnel and Missile Defense 

Agency officials, Arlington, Virginia; and from the contractors about 

the status of critical technologies and the problems associated with 

maturing the technologies required for the laser, the beam control/fire 

control system, and the required aircraft modifications. We also 

obtained detailed briefings from program office and Missile Defense 

Agency officials regarding the new Missile Defense Agency acquisition 

process and the implementation of this process within the Airborne 

Laser program. We conducted our review from July 2001 through May 2002 

in accordance with generally accepted government auditing standards.



As agreed with your office, unless you publicly announce the contents 

of this report earlier, we plan no further distribution until 30 days 

from the report date. At that time, we will send copies of this report 

to the congressional defense committees; the Secretary of Defense; the 

Director, Missile Defense Agency, the Secretary of the Air Force; and 

the Director, Office of Management and Budget. We will also make copies 

available to other interested parties upon request. In addition, the 

report will be available at no charge on the GAO Web site at http://

www.gao.gov.



Please contact me at (202) 512-4841 if you or your staff have any 

questions concerning this report. Key contributors to this report are 

identified in appendix III.



Sincerely yours,



R. E. Levin

Director, Acquisition and

 Sourcing Management:



Signed by R. E. Levin:



[End of section]



Appendix I: Technology Readiness Level Assessment Matrix:



Technology readiness level: 1. Basic principles observed and reported.; 

Description: Lowest level of technology readiness. Scientific research 

begins to be translated into applied research and development. Examples 

might include paper studies of a technology’s basic properties.



Technology readiness level: 2. Technology concept and/or application 

formulated.; Description: Invention begins. Once basic principles are 

observed, practical applications can be invented. The application is 

speculative, and there is no proof or detailed analysis to support the 

assumption. Examples are still limited to paper studies.



Technology readiness level: 3. Analytical and experimental critical 

function and/or characteristic proof of concept.; Description: Active 

research and development is initiated. This includes analytical studies 

and laboratory studies to physically validate analytical predictions of 

separate elements of the technology. Examples include components that 

are not yet integrated or representative.



Technology readiness level: 4. Component and/or breadboard. Validation 

in laboratory environment.; Description: Basic technological 

components are integrated to establish that the pieces will work 

together. This is relatively “low fidelity” compared with the eventual 

system. Examples include integration of “ad hoc” hardware in a 

laboratory.



Technology readiness level: 5. Component and/or breadboard validation 

in relevant environment.; Description: Fidelity of breadboard 

technology increases significantly. The basic technological components 

are integrated with reasonably realistic supporting elements so that 

the technology can be tested in a simulated environment. Examples 

include “high fidelity” laboratory integration of components..



Technology readiness level: 6. System/subsystem model or prototype 

demonstration in a relevant environment.; Description: Representative 

model or prototype system, which is well beyond the breadboard tested 

for technology readiness level five, is tested in a relevant 

environment. Represents a major step up in a technology’s demonstrated 

readiness. Examples include testing a prototype in a high-fidelity 

laboratory environment or in simulated operational environment.



Technology readiness level: 7. System prototype demonstration in an 

operational environment.; Description: Prototype near or at planned 

operational system. Represents a major step up from technology 

readiness level six, requiring the demonstration of an actual system 

prototype in an operational environment, such as in an aircraft, 

vehicle, or space. Examples include testing the prototype in a test bed 

aircraft.



Technology readiness level: 8. Actual system completed and “flight 

qualified” through test and demonstration.; Description: Technology has 

been proven to work in its final form and under expected conditions. In 

almost all cases, this technology readiness level represents the end of 

true system development. Examples include developmental test and 

evaluation of the system in its intended weapon system to determine if 

it meets design specifications..



Technology readiness level: 9. Actual system “flight proven” through 

successful mission operations.; Description: Actual application of the 

technology in its final form and under mission conditions, such as 

those encountered in operational test and evaluation. In almost all 

cases, this is the end of the last “bug fixing” aspects of true system 

development. Examples include using the system under operational 

mission conditions.



[End of table]



[End of section]



Appendix II: Comments from the Department of Defense:



OFFICE OF THE UNDER SECRETARY OF DEFENSE:



3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:



ACQUISITION, TECHNOLOGY AND LOGISTICS:



0 8 JUL2002:



Mr. Robert E. Levin:



Director, Acquisition and Sourcing Management U. S. General Accounting 

Office:



441 G. Street, N.W. Washington, DC 20548:



Dear Mr Levin:



This is the Department of Defense (DoD) response to the GAO Draft 

Report “MISSILE DEFENSE: Knowledge-based Decision Making Needed to 

Reduce Risks in Developing Airborne Laser,” dated June 25, 2002 (GAO 

Code 120079/GAO-02-631).



The DoD has reviewed the draft report and partially concurs with the 

recommendations. Specific comments for each recommendation are 

enclosed. We have recommended some factual corrections. We also 

provided some administrative comments under separate cover. My action 

officer for this effort is Major Mark Arbogast, (703) 695-7328, mark. 

arbogast@osd.mil.



We appreciate the opportunity to comment on the draft report.



Glenn F. Lamartin Director:



Strategic and Tactical Systems:



Signed by an official for Glenn F. Lamartin:



Attachment:



GAO DRAFT REPORT - DATED JUNE 25, 2002 GAO CODE 120079/GAO-02-631:



“MISSILE DEFENSE: KNOWLEDGE-BASED DECISION MAKING NEEDED TO REDUCE 

RISKS IN DEVELOPING AIRBORNE LASER”:



DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:



RECOMMENDATION 1: The GAO recommended that the Secretary of Defense 

direct the Director of the Missile Defense Agency to:



Establish decision points separating technology development from system 

integration; system integration from system demonstration; and system 

demonstration from production. (p. 16/GAO Draft Report):



DOD RESPONSE: Partially concur. Secretary of Defense direction is not 

needed to implement the recommendation. The Department of Defense’s 

acquisition process for ballistic missile defense consists of three 

phases: development, transition, and procurement and operations. That 

process already uses the knowledge-based practices recommended by the 

GAO, tailoring them as appropriate to meet the objectives of the 

Ballistic Missile Defense System (BMDS) program. The Department plans 

to demonstrate system performance in a realistic test environment 

before the transition phase and the subsequent procurement and 

operations phase.



RECOMMENDATION 2: The GAO recommended that the Secretary of Defense 

direct the Director of the Missile Defense Agency (MDA) to:



Establish knowledge-based criteria, for each of the decision points 

mentioned in the initial recommendation, and use that criteria to 

determine where additional investments should be made in the program. 

(p. 16/GAO Draft Report):



DOD RESPONSE: Partially concur. Secretary of Defense direction is not 

needed to implement the recommendation. MDA uses an event-based 

approach to development, where a program moves forward based on its 

progress as demonstrated by events, not by calendar date. MDA intends 

to expand the use of knowledge-based criteria as the basis

for these key events, including technology development, system 

integration, system demonstration, and production; tailoring the 

criteria, consistent with a capability-based spiral development 

approach directed by the Secretary of Defense for the BMDS.



[End of section]



Appendix III GAO Contact and Staff Acknowledgments:



GAO Contact:



R. E. Levin (202) 512-4841:



Acknowledgments:



In addition to the contact named above, Christina Chaplain, Marcus 

Ferguson, Tom Gordon, Subrata Ghoshroy, Barbara Haynes, Matt Lea, Hai 

Tran, Adam Vodraska, and John Warren made key contributions to this 

report.



[End of section]



FOOTNOTES



[1] The Missile Defense Agency plans to develop a series of Airborne 

Laser configurations, which are referred to as “blocks.” It is expected 

that each block will include improved technology that was not available 

in the prior block.



[2] This policy implements statutory planning and reporting 

requirements for major defense acquisition programs. 



[3] The chemical reaction that generates the laser energy occurs in the 

laser modules. The amount of laser energy produced increases as the 

number of laser modules increases.



[4] Technology readiness levels were developed by the National 

Aeronautics and Space Agency and are recommended for use by the 

Department of Defense and the military services. (See appendix I for 

their definition.)



[5] A resonator consists of two mirrors placed at opposite ends of a 

laser cavity. As the reaction of chemicals within the laser cavity 

produces photons of light, the photons are reflected back and forth 

between the two mirrors, which generates additional photons and creates 

a state of high energy within the cavity. In a stable resonator, one 

mirror is fully reflective while the other is partially reflective and 

partially transmissive. Energy that escapes from the laser cavity 

through the transmissive portion of the mirror in a stable resonator 

forms a high-energy beam. In an unstable resonator, both mirrors are 

fully reflective, and one is much smaller in diameter. As the photons 

are reflected from the larger mirror in the direction of the smaller 

mirror, energy escapes from the laser cavity around the edges of the 

smaller mirror and forms a doughnut-shaped beam.



[6] As photons are generated in a laser resonator, the photons 

oscillate or move in different ways. A resonator operating in single-

mode suppresses all photons except those oscillating at a certain 

frequency so that the beam produced can be directed at one spot on the 

target. A resonator operating in multimode does not suppress any 

photons, regardless of their frequency. While a multimode resonator 

directs more energy toward the target, all of that energy will not be 

focused on one area of the target. 



[7] The six-module system is referred to as block 2004 because testing 

will conclude in December 2004. Testing of a second configuration, 

known as block 2008, will be completed in December 2008.



[8] One of the major technical challenges is accommodating the laser’s 

weight. Engineers determined that the six-module system would weigh 

180,000 pounds, but the original system requirement was that the system 

must weigh no more than 175,000 pounds with 

14 laser modules. Because each additional module weighs about 6,000 

pounds, the agency intends to redesign some components to reduce their 

mass and redistribute the weight using a passenger version of the 

Boeing 747 as the block 2008 aircraft. The passenger version of the 747 

can accommodate the crew on an upper deck, thereby allowing the laser’s 

weight to be moved forward where it places less stress on the aircraft 

frame. 



GAO’s Mission:



The General Accounting Office, the investigative arm of Congress, 

exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability 

of the federal government for the American people. GAO examines the use 

of public funds; evaluates federal programs and policies; and provides 

analyses, recommendations, and other assistance to help Congress make 

informed oversight, policy, and funding decisions. GAO’s commitment to 

good government is reflected in its core values of accountability, 

integrity, and reliability.



Obtaining Copies of GAO Reports and Testimony:



The fastest and easiest way to obtain copies of GAO documents at no 

cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains 

abstracts and full-text files of current reports and testimony and an 

expanding archive of older products. The Web site features a search 

engine to help you locate documents using key words and phrases. You 

can print these documents in their entirety, including charts and other 

graphics.



Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its 

Web site daily. The list contains links to the full-text document 

files. To have GAO e-mail this list to you every afternoon, go to 

www.gao.gov and select “Subscribe to daily E-mail alert for newly 

released products” under the GAO Reports heading.



Order by Mail or Phone:



The first copy of each printed report is free. Additional copies are $2 

each. A check or money order should be made out to the Superintendent 

of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to:



U.S. General Accounting Office



441 G Street NW,



Room LM Washington,



D.C. 20548:



To order by Phone: 	



	Voice: (202) 512-6000:



	TDD: (202) 512-2537:



	Fax: (202) 512-6061:



To Report Fraud, Waste, and Abuse in Federal Programs:



Contact:



Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov



Automated answering system: (800) 424-5454 or (202) 512-7470:



Public Affairs:



Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.



General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.



20548: