GAO-02-949T, Missile Defense: Knowledge-Based Process Would Benefit Airborne Laser Decision-Making


This is the accessible text file for GAO report number GAO-02-949T 
entitled 'Missile Defense: Knowledge-Based Process Would Benefit 
Airborne Laser Decision-Making' which was released on July 16, 2002.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products’ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.



Testimony:



Before the Subcommittee on National Security, Veterans’ Affairs, and 

International Relations, Committee on Government Reform, House of 

Representatives:



United States General Accounting Office:



GAO:



For Release on Delivery Expected at 10:00 a.m., EDT, Tuesday, July 16, 

2002:



MISSILE DEFENSE:



Knowledge-Based Process Would Benefit Airborne Laser Decision-Making:



Statement of Robert E. Levin, Director, Acquisition and Sourcing 

Management:



Technology Development:



GAO-02-949T:



Mr. Chairman and Members of the Subcommittee:



Thank you for inviting me. I am pleased to be here today to discuss the 

Missile Defense Agency’s strategy for acquiring ballistic missile 

defense systems. Pursuant to your request, Mr. Chairman, we issued a 

report on July 12, 2002, that examined whether the Missile Defense 

Agency’s new strategy for developing the Airborne Laser includes 

practices that are characteristic of successful developmental 

programs.[Footnote 1] Our observations today will reflect the knowledge 

that we gained from that work.



If the Department of Defense is successful in its efforts to develop 

and field the Airborne Laser, it could be an important system because 

it would give the United States some capability to destroy enemy 

missiles over the enemy’s own territory. Enemy states that could launch 

ballistic missiles with chemical or biological warheads toward the 

United States, its deployed troops, or allies could be deterred from 

doing so if that enemy knew that the chemical and biological agents 

might fall back to earth over its own territory.



When the Air Force launched the Airborne Laser program in 1996, it 

estimated that developing the system would cost $2.5 billion and that 

it would be fielded by 2006. However, by August 2001, the Air Force 

determined that maturing the technologies and developing the system 

would cost $3.7 billion, or about 50 percent more, and the system would 

not be fielded until 2010, 4 years later than originally planned. 

Against this backdrop of cost increases and schedule delays, the 

Department of Defense, in October 2001, transferred responsibility for 

the Airborne Laser from the Air Force to the Ballistic Missile Defense 

Organization, now known as the Missile Defense Agency.



Because of your interest in the agencyís recently expanded 

responsibility and authority for acquiring a capability to protect 

against enemy ballistic missiles, you requested that today we:



* describe the Secretary of Defenseís specific direction to the agency;



* explain the agencyís requirements-setting process and plans for 

testing;



* provide details on the agencyís investment decision-making process;



* explain how the maturity of technology critical to the systemís 

design will be measured;



* comment on the role of the Director, Operational Test and Evaluation; 

and:



* remark on changes in the agencyís test plans since Antiballistic 

Missile Treaty restrictions are no longer in place.



To address our objectives, we reviewed documents and held discussions 

with officials at the Boeing Company, Seattle, Washington; Lockheed 

Martin, Sunnyvale, California; TRW, Los Angeles, California; Airborne 

Laser Program Office, Kirtland Air Force Base, New Mexico; Missile 

Defense Agency, Arlington, Virginia; and the Office of the Director, 

Operational Test and Evaluation, Arlington, Virginia. We conducted our 

review from August 2001 through July 2002 in accordance with generally 

accepted government auditing standards.



In summary Mr. Chairman, we found the following:



* In January 2002, the Secretary of Defense directed the Missile 

Defense Agency to quickly develop a ballistic missile defense system 

that included various elements, such as the Airborne Laser. These 

elements are to work together to defend the United States, its deployed 

troops, allies, and friends by engaging enemy ballistic missiles at 

various points during their flight. The Secretary also directed the 

agency to cancel existing requirements documents and, instead, develop 

elements with currently available technology that would be capable of 

defeating some, if not all, of the threat. This capability could then 

be improved over time.



* In response to this direction, the agency adopted changes that are 

characteristic of successful development programs. First, the agency 

adopted a flexible requirements-setting process that allows it to 

refine system requirements based on the results of system engineering. 

This process can result in less risk of cost and schedule growth 

because it does not establish requirements until systems engineering 

shows what is achievable. However, the agency must set the Airborne 

Laser’s requirements when it determines that it has a match between the 

technology, money, and time needed to design and demonstrate an 

operational system so that the agency can use those requirements to 

measure the progress being made during product development. The agency 

is also implementing other changes to improve Airborne Laser’s 

development. For example, the agency is allowing more time to mature 

and test technologies critical to the system’s design, and it is 

improving ground test facilities so that in the future improved 

components can be tested on the ground before being installed in an 

aircraft.



* The agency has not yet implemented another practice that has proven 

beneficial in successful development programs and that we recommended 

to the agency in our July 12 report. Successful developers make 

decisions on whether to proceed with a program when the focus and cost 

of their activities are about to change. The criteria for beginning new 

activities is a determination by the developer that the program has the 

knowledge to begin the activities and that investing in them is the 

best use of the developer’s resources. Our work over the years has 

found that the focus and cost of activities will change when the 

program is ready to move from technology development to system 

integration, then again into system demonstration, and finally on into 

production. Although, the Missile Defense Agency does not currently 

make decisions at each of these points or use the same criteria proven 

by successful developers, the agency is considering how to implement 

such a process.



* The Airborne Laser Program Office used technology readiness levels to 

assess the maturity of the technology critical to the design of the 

Airborne Laser. Our work shows that developers greatly enhanced their 

likelihood of success by beginning development only when they had 

demonstrated a system prototype in an operational environment. The 

program office’s assessment showed that some Airborne Laser technology 

is almost to this point, but technology such as the mirrors and windows 

that focus and control the laser beam and allow it to pass safely 

through the aircraft (collectively referred to as optics) require 

additional engineering work before reaching this stage of maturity. 

Except for its evaluation of the laser, we agreed with the program 

office’s technology assessment. We believe further testing is needed to 

demonstrate the maturity of the laser technology because the program 

has only tested a one-module laser (rather than the six-module laser 

planned for the first Airborne Laser configuration) in a controlled 

laboratory environment using surrogate components.



* By law, the Director of Operational Test and Evaluation (OT&E) is 

responsible for operational rather than developmental testing. However, 

OT&E officials said that being involved in developmental tests provides 

them insight and understanding to prepare for live-fire testing and 

later operational testing and evaluation. The officials told us that 

they have primarily been working on issues related to future live-fire 

lethality tests of the Airborne Laser, but they expect their 

involvement in Airborne Laser’s developmental tests will increase when 

system-level flight tests begin. The OT&E Director has not been given 

authority to approve Airborne Laser’s developmental test and evaluation 

master plan, but OT&E officials have been asked to assist agency 

officials in preparing the plan and will provide comments before the 

plan is finalized.



* The Airborne Laser Program Office said that the United States’ 

withdrawal from the Antiballistic Missile Treaty has not changed its 

plan to initially test the first Airborne Laser configuration against a 

short-range ballistic missile. However, the agency is considering 

future tests with longer-range missile targets. Such tests, which would 

have been restricted by the treaty, could occur in fiscal year 2005 or 

2006.



Defense Secretary Directs New Approach for Acquiring and Deploying 

Missile Defenses:



In 2001, the Department of Defense conducted missile defense reviews to 

determine how to best fulfill the nation’s need to defend against enemy 

ballistic missile attacks. As a result of these studies, the Secretary 

of Defense in January 2002 directed that the Ballistic Missile Defense 

Organization be elevated to the status of an agency and renamed as the 

Missile Defense Agency. The Secretary’s key priorities were to field 

quickly an integrated missile defense system that could defend the 

United States, deployed forces, allies, and friends by engaging enemy 

ballistic missiles at various points during their flight. Another of 

the Secretary’s priorities was to provide an early capability by using 

test assets or prototypes in the event of an emergency. To enable the 

agency to achieve his priorities, the Secretary directed the agency to 

abandon its traditional requirements-setting process that required a 

military service to establish technical requirements when a weapon 

system acquisition program was launched and adopt a more flexible, 

capability-based process that would allow the agency to use available 

technology to develop a weapon system that could engage some, if not 

all, of the current threat. This “base-line” capability would then be 

improved over time. The Airborne Laser system was one of many systems 

affected by these changes.



Agency Adopts New Requirements-Setting Process and Other Practices:



In response to the Secretary’s direction, the Missile Defense Agency 

adopted a new Airborne Laser development strategy that incorporates 

some of the practices characteristic of successful programs. These 

practices include a more flexible requirements-setting process, 

allowing more time to mature and test the Airborne Laser’s critical 

technologies, and improving test facilities.



Our work shows that the flexible requirements-setting process can 

result in less risk of cost and schedule growth because requirements 

are not set until systems engineering shows what is achievable. 

However, the agency must set the Airborne Laser’s requirements once it 

determines that it has a match between the technology, money, and time 

needed to design and demonstrate an operational system so that the 

agency can use those requirements to measure the progress being made 

during product development.



The Department of Defense ordinarily faces significant hurdles in 

matching requirements to available resources (time, technology, and 

money). The fundamental problem is two-fold. First, under the 

Department’s traditional process, requirements must be set before a 

program can be approved and a program must be approved before the 

product developer conducts systems engineering. Second, the competition 

for funding encourages requirements that will make the desired weapon 

system stand out from others. Consequently, many of the Department’s 

product development programs include unrealistic requirements set by 

the user before the product developer has conducted the system 

engineering necessary to identify the time, technology, and money 

necessary to develop a product capable of meeting requirements.



The agency is also adopting other practices that are likely to improve 

Airborne Laser’s development. These practices include allowing more 

time for testing and developing facilities to mature and test critical 

technologies. The agency is initially developing and testing a six-

module laser system to demonstrate technologies critical to the 

Airborne Laser’s design. When the Air Force was responsible for the 

Airborne Laser program, it planned to complete system-level flight 

tests of the six-module Airborne Laser system in the last quarter of 

fiscal year 2003, but the agency has delayed completion of the test to 

the first quarter of fiscal year 2005. This delay allows additional 

time to learn from and correct problems discovered during the tests. In 

addition, the agency plans to increase the Airborne Laser’s ground-

testing capability by awarding a contract in 2003 for what they are 

calling an “iron bird,” which is essentially an aircraft hull with 

laser equipment installed. The “iron bird” is expected to allow testing 

of a fully integrated Airborne Laser system on the ground so that 

technologies for future configurations can be evaluated before being 

installed in an aircraft.



Allowing more time for testing is important because testing informs the 

requirements process. Because testing allows developers to gauge the 

progress being made in translating an idea into a weapon system, it 

enables the developer to make a more informed decision as to whether a 

technology is ready to be incorporated into a system’s design. With 

this knowledge, the developer can determine whether the technology is 

so important to the system’s design that additional time and money 

should be spent to mature the technology or whether the system’s 

initial performance requirements should be reduced.



The “iron bird” is expected to reduce the cost of testing technologies 

planned for future Airborne Laser configurations. With it, the agency 

can mature new component-level technologies to higher levels in the 

less expensive ground-testing environment before installing them on an 

aircraft.



Changes in Agency’s Decision-Making Process Are Needed:



We reported in July 2002, that the Missile Defense Agency’s new 

Airborne Laser acquisition strategy does not include decision points 

with appropriate knowledge-based criteria for moving the Airborne Laser 

program forward. However, the agency is now considering how it can 

implement such a process.



In successful developments, developers make decisions when the focus 

and cost of program activities is about to change. At these points, the 

developers decide whether they have the knowledge to begin new 

activities and whether investing further time and money in their 

product is the best use of their resources. The first decision point 

occurs when the focus of the developer’s activities change from 

technology development to system integration. This point is reached 

when the developer has incorporated technology into subcomponents with 

the form, fit, and function needed in an operational system and the 

developer is ready to design a system that integrates those 

subcomponents. The criterion for deciding to move the program forward 

is having the knowledge to match requirements and available resources 

(time, technology, and funds). The second decision point occurs between 

system integration and system demonstration when the developer has 

successfully integrated subsystems and components into a design that 

not only meets the customer’s performance requirements but also has 

optimized the design for reproducibility, maintainability, and 

reliability. A developer moves the program forward at this point only 

if the design is stable, which is generally considered to be the point 

at which about 90 percent of the design’s engineering drawings have 

been released. The third decision point separates system demonstration 

from production. The decision to invest in production is based on 

having the knowledge to determine that the product performs as required 

during testing and that the manufacturing processes will produce a 

product within cost, schedule, and quality targets.



Product Development:



Production:



Knowledge Point 1: Needs match resources:



Knowledge Point 2: Design is stable:



Knowledge Point 3: Production can meet cost, schedule, and quality 

targets:



System Integration:



System Demonstration n:



Program Launch:



Figure 1: The Knowledge-Based Process:



Decisions are made at these points not only because the focus of 

activities is changing, but also because the cost of the activities are 

increasing. Our work shows that product development is typically much 

more costly than technology development. This is because during 

technology development, small teams of technologists work to perfect 

the application of scientific knowledge to a practical problem. As 

product development begins, developers begin to make larger investments 

in human capital, bringing on a large engineering force to design and 

manufacture the product. In addition, product development requires 

significant investments in facilities and materials. These investments 

increase continuously as the product approaches the point of 

manufacture.



The Airborne Laser acquisition process has three phases separated by 

two decision points. The phases are development, transition, and 

procurement. The development phase includes all developmental 

activities and system level demonstrations of military utility. 

Transition includes preparation of the operational requirements 

document by the appropriate armed service and operational testing; and 

production includes producing and fielding the weapon system. The first 

decision point occurs between development and transition. A decision 

will be made to begin the transition phase if the agency determines 

that it has the technology in-hand to produce a system that merits 

fielding. At the end of the transition phase, the Airborne Laser would 

enter the formal Department of Defense acquisition process at Milestone 

C--the point at which the Department decides whether a system should 

begin low-rate initial production.



The Missile Defense Agency’s current decision-making process puts at 

risk the agency’s ability to develop a useful military capability on 

time and within budget. This is because the agency’s process does not 

include an established set of decision points with appropriate 

knowledge-based criteria for deciding whether to invest in system 

integration and, subsequently, system demonstration and production. For 

example, the agency does not separate technology development from 

system integration with a decision point or use the knowledge the 

program has attained to determine if the technology can be incorporated 

into a mature system design within available time and funding 

constraints. Agency officials are considering how to fit such decision 

points into Airborne Laser’s acquisition process, as well as the 

acquisition process of other elements.



Technology Readiness Levels Are Used to Measure Technology Maturity:



In 2002, the Airborne Laser Program Office assessed the maturity of 

technologies critical to the development of the Airborne Laser system. 

To make this assessment, the program office used a tool known as 

technology readiness levels that was developed by the National 

Aeronautical and Space Administration. A comparison of a 1999 

assessment and the 2002 assessment shows that the Airborne Laser 

program has made progress in maturing critical technologies, but much 

remains to be done.



In 1996, the Air Force launched the Airborne Laser program to develop a 

defensive system that could destroy enemy missiles from a distance of 

several hundred kilometers. Engineers concluded that if they were to 

meet this requirement, the system would need a fourteen-module oxygen 

iodine laser. They also determined that the system would need a beam 

control/fire control assembly that could (1) safely move the laser beam 

through the aircraft, (2) shape the beam so that it would not be 

scattered or weakened by the atmosphere, and (3) hold the beam on 

target despite the movement of the aircraft. In addition, engineers 

determined that the system would need a battle management and control 

system capable of planning and executing an engagement. To determine if 

the technology was “in hand” to meet this requirement, the Air Force 

planned to build a six-module Airborne Laser configuration and test it 

against a short-range ballistic missile. Under the Missile Defense 

Agency’s new development strategy, this six-module configuration is now 

known as Block 2004.[Footnote 2]



Officials began their 2002 assessment by determining the technologies 

critical to designing Block 2004 and future configurations. These 

technologies are: (1) devices that stabilize the laser system aboard 

the aircraft so that the beam can be maintained firmly on the target; 

(2) optics--mirrors and windows--that focus and control the laser beam 

and allow it to pass safely through the aircraft; (3) optical coatings 

that enhance the optics’ ability to pass laser energy through the 

system and to reflect the laser energy; (4) hardware that works in 

tandem with computer software to actively track the target missile; (5) 

devices that measure atmospheric turbulence and compensate for it so 

that it does not scatter or weaken the laser beam; and (6) safety 

systems that automatically shut down the high energy laser in the event 

of an emergency. At our request, the Airborne Laser Program Office also 

assessed the maturity of the oxygen iodine laser.



The program office assessed the optics and stabilizing devices at 

technology readiness level four, the optical coatings at level five, 

and the safety systems, atmospheric compensation, and target tracking 

devices at level six. At level four, scientists have shown that a 

technology is technically feasible, but have not shown whether the 

technology will have the form, fit, or function required in the 

operational system. When a technology progresses to level five, the 

technology being tested is incorporated into hardware whose form and 

fit are coming closer to that needed for an operational component and 

that hardware is integrated with reasonably realistic supporting 

elements so that the technology can be tested in a simulated 

environment. Finally, at technology readiness level six, integrated 

testing of a prototype system has occurred in a relevant environment. 

Our prior work has shown that demonstrating a technology readiness 

level of seven, that is, demonstrating that components can work 

together as a system in a realistic setting, prior to establishing cost 

and schedule estimates and beginning system integration, is an 

important determinant of program success.



We disagreed with the agency’s assessment in only one instance--the 

assessment of the maturity of the system’s laser component. The agency 

assessed the maturity of the oxygen iodine laser at a readiness level 

of six while we consider it to be at a level four. The tests of a one-

module laser that the program office relied upon to prove the maturity 

of the laser were conducted in a controlled laboratory environment with 

surrogate components. In our opinion, the program office will 

demonstrate the laser technology in a relative environment (technology 

readiness level six) when the six-module system is integrated and 

successfully tested at full power within the high fidelity laboratory 

environment of the Airborne Laser Systems Integration Laboratory, 

currently under construction at Edwards Air Force Base, California.



Operational Test Community Expects to be Involved in Developmental and 

Operational Testing:



By law, the Director of the Office of Operational Test and Evaluation 

(OT&E) is the principal advisor to the Secretary of Defense and the 

Under Secretary of Defense for Acquisition, Technology, and Logistics 

on operational test and evaluation of Department of Defense weapon 

systems and is also responsible for monitoring and reviewing live fire 

testing activities.[Footnote 3] The Director may not be assigned any 

responsibility for developmental test and evaluation, other than to 

provide advice to officials responsible for such testing. However, OT&E 

officials said that being involved in developmental tests aids them in 

preparing for live fire testing and later operational testing and 

evaluation, for which they are responsible.



The officials told us that they have primarily been working on issues 

related to future live-fire lethality tests of the Airborne Laser, but 

they expect their involvement in Airborne Laser’s developmental tests 

will increase when system-level flight tests begin. This expectation is 

based on the agency’s request that the OT&E officials work with Missile 

Defense Agency officials to construct developmental test and evaluation 

master plans for the integrated missile defense system and each 

element. Although OT&E has not been given authority to approve the 

Airborne Laser’s developmental test and evaluation master plan, OT&E 

officials said that they will assist in preparing the plan and will 

review and offer comments before it is finalized.



In addition to participating in the creation of the developmental test 

and evaluation master plans, the Director, OT&E, will participate in 

approving each element’s operational test plan and will evaluate their 

operational tests. Further, under the agency’s new strategy, OT&E 

officials are members of the Missile Defense Support Group and its 

working group. These groups are expected to provide insight and advice 

to the Missile Defense Agency Director and to the Senior Executive 

Council that is responsible for all major missile defense decisions. 

The agency has also assured the operational test community that it will 

be included in all meetings and reviews regarding testing so that test 

officials can annually review and report on the adequacy and 

sufficiency of the Missile Defense Agency’s testing program.



Targets in Future Tests Could Change:



Airborne Laser program officials told us that the Airborne Laser was 

originally planned as defense against short-range ballistic missiles. 

However, with the United States withdrawal from the Antiballistic 

Missile Treaty, the Missile Defense Agency is considering tests against 

longer-range missiles. While such a change would not affect the program 

office’s plan to test the Airborne Laser against a short-range missile 

in December 2004, tests conducted in fiscal year 2005 or 2006 could 

include targets representative of longer-range missiles. The 

Antiballistic Missile Treaty would have restricted tests that include 

longer-range missiles as Airborne Laser targets.



Conclusion:



In an effort to field a missile defense capability quickly, the 

Department of Defense has directed the Missile Defense Agency to adopt 

a new acquisition strategy. Some of the practices that we observed are 

being implemented in the Airborne Laser program are practices that have 

been proven in successful development programs. For example, similar to 

successful developers, the agency has deferred the establishment of the 

Airborne Laser’s requirements until the knowledge is available to set 

realistic requirements. Successful developers also improve their 

product incrementally, just as the agency plans to improve the Airborne 

Laser through a series of upgrades. However, the agency has the 

opportunity to make its acquisition process more knowledge-based. By 

establishing knowledge-based decision points at key junctures, the 

agency would be in a better position to decide whether to invest in the 

next phase of the Airborne Laser’s development. Also, the agency would 

be better able to hold the Airborne Laser Program Office accountable 

for planning all of the activities required to develop a quality 

product, approaching those activities in a systematic manner so that no 

important steps are skipped and problems are resolved sooner rather 

than later, and making cost and schedule projections when they have the 

knowledge to make realistic estimates. With this disciplined process in 

place, the agency will be much better positioned to decide whether to 

invest further in the Airborne Laser or use available time and funds 

for some other element of the missile defense system.



Our July 12 report recommended that the Director of the Missile Defense 

Agency not only establish decision points to separate technology 

development from system integration; system integration from system 

demonstration; and system demonstration from production, but also 

establish knowledge-based criteria that would be used to determine 

whether additional investments should be made in the Airborne Laser 

program. We are encouraged that the agency is now considering actions 

that could prove to be significant steps toward implementing these 

recommendations.



Mr. Chairman, this concludes my statement. I would be pleased to 

respond to any questions you or members of the committee may have.



FOOTNOTES



[1] U.S. General Accounting Office, Missile Defense: Knowledge-Based 

Decision Making Needed To Reduce Risks in Developing Airborne Laser, 

GAO-02-631 (Washington, D.C.: July 12, 2002)..



[2] This configuration is known as Block 2004 because it is to be 

completed in December 2004.



[3] 10 USC 139