This is the accessible text file for GAO report number GAO-07-388 
entitled 'Best Practices: An Integrated Portfolio Management Approach 
to Weapon System Investments Could Improve DOD's Acquisition Outcomes' 
which was released on March 30, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Committee on Armed Services, U.S. Senate: 

United States Government Accountability Office: 

GAO: 

March 2007: 

Best Practices: 

An Integrated Portfolio Management Approach to Weapon System 
Investments Could Improve DOD's Acquisition Outcomes: 

GAO-07-388: 

GAO Highlights: 

Highlights of GAO-07-388, a report to the Committee on Armed Services, 
U.S. Senate 

Why GAO Did This Study: 

Over the next several years, the Department of Defense (DOD) plans to 
invest $1.4 trillion in major weapons programs. While DOD produces 
superior weapons, GAO has found that the department has failed to 
deliver weapon systems on time, within budget, and with desired 
capabilities. While recent changes to DOD’s acquisition policy held the 
potential to improve outcomes, programs continue to experience 
significant cost and schedule overruns. 

GAO was asked to examine how DOD’s processes for determining needs and 
allocating resources can better support weapon system program 
stability. Specifically, GAO compared DOD’s processes for investing in 
weapon systems to the best practices that successful commercial 
companies use to achieve a balanced mix of new products, and identified 
areas where DOD can do better. In conducting its work, GAO identified 
the best practices of: Caterpillar, Eli Lilly, IBM, Motorola, and 
Procter and Gamble. 

What GAO Found: 

To achieve a balanced mix of executable development programs and ensure 
a good return on their investments, the successful commercial companies 
GAO reviewed take an integrated, portfolio management approach to 
product development. Through this approach, companies assess product 
investments collectively from an enterprise level, rather than as 
independent and unrelated initiatives. They weigh the relative costs, 
benefits, and risks of proposed products using established criteria and 
methods, and select those products that can exploit promising market 
opportunities within resource constraints and move the company toward 
meeting its strategic goals and objectives. Investment decisions are 
frequently revisited, and if a product falls short of expectations, 
companies make tough go/no-go decisions. The companies GAO reviewed 
have found that effective portfolio management requires a governance 
structure with committed leadership, clearly aligned roles and 
responsibilities, portfolio managers who are empowered to make 
investment decisions, and accountability at all levels of the 
organization. 

In contrast, DOD approves proposed programs with much less 
consideration of its overall portfolio and commits to them earlier and 
with less knowledge of cost and feasibility. Although the military 
services fight together on the battlefield as a joint force, they 
identify needs and allocate resources separately, using fragmented 
decision-making processes that do not allow for an integrated, 
portfolio management approach like that used by successful commercial 
companies. Consequently, DOD has less assurance that its investment 
decisions address the right mix of warfighting needs, and, as seen in 
the figure below, it starts more programs than current and likely 
future resources can support, a practice that has created a fiscal bow 
wave. If this trend goes unchecked, Congress will be faced with a 
difficult choice: pull dollars from other high-priority federal 
programs to fund DOD’s acquisitions or accept gaps in warfighting 
capabilities. 

Figure: Costs Remaining Versus Annual Appropriations for Major Defense 
Acquisitions: 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

What GAO Recommends: 

GAO is making several recommendations for DOD to implement an 
integrated portfolio management approach to weapon system investments. 
DOD stated that it is undertaking several pilot efforts to improve the 
department’s approach and that implementation of any new business rules 
will be contingent upon the outcomes of these efforts. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Michael J. Sullivan at 
(202) 512-4841 or sullivanm@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Successful Companies Take a Disciplined, Integrated Approach to 
Prioritize Market Needs and Initiate a Balanced Mix of Executable 
Development Programs: 

Lacking an Integrated, Portfolio-Based Approach, DOD Has Too Many 
Programs Competing for Limited Resources: 

Conclusions: 

Recommendations: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Related GAO Products: 

Table: 

Table 1: Cost and Cycle Time Growth for 27 Weapon Systems: 

Figures: 

Figure 1: DOD's Weapon System Investment Process: 

Figure 2: Portfolio Management Approach to Product Investments: 

Figure 3: IBM Market Segmentation: 

Figure 4: Risk Versus Rewards Matrix: 

Figure 5: Service Allocations of DOD's Investment Budget: 

Figure 6: Governance of DOD's Investment Process: 

Figure 7: Costs Remaining Versus Annual Appropriations for Major 
Defense Acquisitions: 

Abbreviations: 

ACAT: Acquisition Category: 
AOA: Analysis of Alternatives: 
AT&L: Acquisition, Technology, and Logistics: 
DAS: Defense Acquisition System: 
DOD: Department of Defense: 
FY: fiscal year: 
GAO: U.S. Government Accountability Office: 
IBM: International Business Machines: 
ICD: Initial Capabilities Document: 
IPD: Integrated Product Development: 
JCIDS: Joint Capabilities Integration and Development System: 
JCS: Joint Chiefs of Staff: 
JROC: Joint Requirements Oversight Council: 
JTRS: Joint Tactical Radio System: 
MS: Milestone: 
NPV: Net Present Value: 
OSD: Office of the Secretary of Defense: 
PA&E: Program Analysis and Evaluation: 
PPBE: Planning, Programming, Budgeting, and Execution: 
RDT&E: research, development, test, and evaluation: 
USD: Under Secretary of Defense: 

United States Government Accountability Office: 
Washington, DC 20548: 

March 30, 2007: 

The Honorable Carl Levin: 
Chairman: 
The Honorable John McCain: 
Ranking Member: 
Committee on Armed Services: 
United States Senate: 

Although the Department of Defense (DOD) produces the best weapons in 
the world, it has not been able to deliver planned systems on time and 
within budget. It is not unusual to see cost increases that add up to 
tens or hundreds of millions of dollars, schedule delays that add up to 
years, and large and expensive programs being scrapped after years of 
failing to achieve promised capabilities.[Footnote 1] While recent 
changes to DOD's acquisition policy held the potential to improve such 
outcomes, programs have continued to experience significant cost and 
schedule overruns and performance shortfalls.[Footnote 2] Over the next 
several years, DOD plans to invest $1.4 trillion in major weapon system 
programs--doubling what it planned to spend on such programs 5 years 
ago. Continued failure to deliver weapon systems on time and within 
budget not only delays providing critical capabilities to the 
warfighter, but results in less funding being available for other DOD 
and federal priorities. 

In the commercial market, effectively developing and marketing new 
products is fundamental to the continued growth and success of 
companies. Without a steady stream of product innovations to meet 
evolving market needs, companies are likely to see their sales and 
profits fall. At the same time, if the products in development outstrip 
their resources or do not meet customer needs, companies can face 
financial ruin. Several recent studies issued by leading thinkers in 
the area of product innovation and development have reported that 
leading commercial companies achieve success in product development by 
following a disciplined process for ensuring they have the right mix of 
new products that meet customer needs within available resources. 

In fiscal year 2006, the Senate Armed Services Committee raised 
concerns that DOD's poor track record with acquisition programs was 
linked not only to the department's Defense Acquisition System (DAS) 
for managing product development, but also to the department's Joint 
Capabilities Integration and Development System (JCIDS) for identifying 
the warfighters' needs and the Planning, Programming, Budgeting and 
Execution (PPBE) process for allocating resources. Consequently, the 
Committee directed GAO to examine how DOD's needs identification and 
resource allocation processes can better support program stability in 
major weapon systems acquisition.[Footnote 3] This report (1) 
identifies best practices of successful commercial companies for 
ensuring that they pursue the right mix of programs to meet the needs 
of their customers within resource constraints and (2) compares DOD's 
enterprise-level processes for investing in weapon systems to these 
practices.[Footnote 4] 

To identify best practices of successful companies, we reviewed related 
professional and academic publications, and interviewed knowledgeable 
officials from five successful commercial companies: Caterpillar, Eli 
Lilly, IBM, Motorola, and Procter & Gamble. While the products 
developed by these companies range from heavy construction equipment 
and high-end electronics to pharmaceuticals and household items, each 
of the companies manages a large diversified portfolio of products, 
spends billions of dollars annually on research and development, and 
has thousands of employees worldwide. To examine DOD's processes for 
making investment decisions, we reviewed related legislation and DOD 
directives, instructions, and guidance; conducted interviews with and 
received briefings from relevant Joint Staff, Office of the Secretary 
of Defense (Comptroller, Program Analysis and Evaluation, and 
Acquisition, Technology and Logistics), and other government officials; 
reviewed current literature assessing DOD's decision-making processes; 
and analyzed DOD requirements documents. We compared DOD's enterprise- 
level practices to commercial best practices to identify potential 
areas for improvement. For additional details on how we performed our 
review, see appendix I. Our work was conducted between March 2006 and 
February 2007 in accordance with generally accepted government auditing 
standards. 

Results in Brief: 

Successful commercial companies use an integrated portfolio management 
approach to prioritize market needs and allocate resources; thus, they 
avoid pursuing more products than their resources can support and 
optimize the return on their investment. Through portfolio management, 
all of a company's product investments are addressed collectively from 
an enterprise level, rather than as independent and unrelated 
initiatives. Potential product developments are identified and assessed 
through a systematic and disciplined screening process. Companies weigh 
the relative costs, benefits, and risks of each proposed product using 
established criteria and methods to select the best mix of products to 
develop. They not only select those products that have a sound business 
case to warrant further investment, but also those that help the 
company balance near-and future-term market opportunities, different 
product lines, and available resources against the demand for product 
investments. Once initial investment decisions are made, they are 
revisited at multiple stages throughout product development in a gated 
review and assessment process to ensure products are still of high 
value. If not, companies make tough decisions to defer or terminate 
investments and rebalance their product portfolios. To be effective, 
portfolio management is enabled by strong governance with committed 
leadership, clearly aligned organizational roles and responsibilities, 
empowered portfolio managers who determine the best way to invest 
resources, and accountability at all levels of the organization. 

Although the military services fight together on the battlefield as a 
joint force, they do not identify warfighting needs and make weapon 
system investment decisions together in an integrated manner. DOD has 
taken steps to identify warfighting needs through a joint requirements 
process, but its service-centric structure and fragmented decision- 
making processes do not allow for the portfolio management approach 
used by successful commercial companies to make investment decisions 
that benefit the organization as a whole. DOD largely continues to 
define warfighting needs and make investment decisions on a service-by- 
service basis, an approach that has contributed to duplication in 
programs and equipment that does not operate effectively together. 
Also, DOD assesses warfighting needs and their funding implications 
under separate decision-making processes, impeding its ability to 
prioritize warfighting needs so that it pursues not only the ones that 
are most important but also the ones it can afford. While DOD's JCIDS 
process provides a framework for reviewing and validating the initial 
need for proposed capabilities, it does not focus on the cost and 
feasibility of acquiring the capability to be developed and fielded. 
Instead, these considerations are addressed through separate budgeting 
and acquisition processes. Moreover, although DOD policy provides for a 
series of early reviews--focused on the concept refinement and 
technology development phases of proposed weapon system programs--in 
prior work we found that the reviews are often skipped or are not fully 
implemented. Consequently, proposed programs build momentum and move 
toward starting product development with little if any early department-
level assessment of the costs and feasibility. Committing to programs 
before they have this knowledge contributes to poor cost, schedule, and 
performance outcomes and destabilizes acquisition programs as the 
department attempts to pay for poorly performing programs by taking 
funds from others. 

The department has begun to pilot-test several interrelated initiatives 
intended to address shortfalls in its existing approach to investment 
decisions. These initiatives include a new approach to an early 
decision gate for reviewing proposed programs at the concept stage, 
testing portfolio management approaches in selected capability areas, 
and setting up capital budgeting accounts for programs in development. 
However, as currently structured, the initiatives are intended to 
operate within DOD's existing organizational and process framework and 
may not allow for sufficient authority and control over resources to 
effectively influence weapon system investments. 

To improve DOD's ability to deliver a balanced mix of weapon system 
programs at the right time and right cost, we are recommending the 
department establish an integrated, portfolio-based approach to 
investments that incorporates best practices of successful commercial 
companies. To ensure the success of such an approach, we are also 
recommending that DOD establish a single point of accountability at the 
department level with the responsibility, authority, and accountability 
for ensuring that portfolio management for weapon system investments is 
effectively implemented across the department. DOD concurred with the 
majority of our recommendations and partially concurred with two. 
Generally, in responding to these recommendations, DOD stated that it 
is undertaking several initiatives and pilot efforts to improve the 
department's approach to investment and program decision making, and 
that implementation of any new business rules will be contingent upon 
the outcome of these initiatives. However, we believe the department's 
current initiatives do not fundamentally change DOD's service-centric 
framework or sufficiently integrate its decision-making processes. DOD 
did not provide comments regarding our recommendation that the 
Secretary establish a single point of accountability. 

Background: 

DOD's programs for acquiring major weapon systems have taken longer, 
cost more, and delivered fewer quantities and capabilities than 
planned. We have documented these problems for decades. Most recently, 
we reported that 27 major weapon programs we have assessed since they 
began product development have experienced cost increases of nearly 34 
percent over their original research, development, test, and evaluation 
(RDT&E) estimates, and increases of almost 24 percent in acquisition 
cycle time (see table 1).[Footnote 5] 

Table 1: Cost and Cycle Time Growth for 27 Weapon Systems: 

Billions of constant 2007 dollars. 

Total cost; 
First full estimate: $506.4; 
Latest full estimate: $603.1; 
Percentage change: 19.1. 

RDT&E cost; 
First full estimate: $104.7; 
Latest full estimate: $139.7; 
Percentage change: 33.5. 

Weighted average acquisition cycle time[A]; 
First full estimate: 137.9 months; 
Latest full estimate: 170.2 months; 
Percentage change: 23.5. 

Source: GAO analysis of DOD data. 

[A] This is a weighted estimate of average acquisition cycle time for 
the 27 programs based on total program costs at the first full and 
latest estimates. The simple average for these two estimates was 98.9 
months for the first full estimate and 124.6 months for the latest 
estimate resulting in a 26.1 percent change. 

[End of table] 

When cost and schedule problems occur in one program, DOD often 
attempts to pay for the poorly performing program by taking funds from 
others. Doing so has destabilized other programs and reduced the 
overall buying power of the defense dollar as DOD and the military 
services are forced to cut back on planned quantities or capabilities 
to stay within budget limitations. The F-22A Raptor program is a case 
in point: As costs escalated in the program, the number of aircraft the 
Air Force planned to buy was drastically reduced from 648 to 183. 
Similarly, as the Joint Tactical Radio System (JTRS) encountered 
development problems, the number of requirements was reduced or 
deferred by about one-third. As a result, several programs that were 
dependent on JTRS also had to make adjustments and go forward with 
alternative, less capable solutions. DOD's approach to managing weapon 
system investments ultimately results in less funding being available 
for other competing needs in DOD as well as other federal priorities, 
as the expenditure of tax dollars within DOD reduces the amount of 
funding available for those priorities. 

Taking into account the differences between commercial product 
development and weapons acquisitions, we have recommended that DOD 
adopt a knowledge-based, incremental approach to developing and 
producing weapon systems. This type of an approach requires program 
officials to demonstrate that critical technologies are mature, product 
designs are stable, and production processes are in control at key 
junctures in the acquisition process.[Footnote 6] 

DOD has three major processes involved in making weapon system 
investment decisions. These processes, depicted in figure 1, are the 
Joint Capabilities Integration and Development System (JCIDS), for 
identifying warfighting needs; the Planning, Programming, Budgeting and 
Execution (PPBE) system, for allocating resources; and the Defense 
Acquisition System (DAS), for managing product development and 
procurement. Much of our prior work has focused on identifying 
commercial best practices that could be used to improve the Defense 
Acquisition System--from the point just before product development 
starts onward. In this report, however, we look at earlier stages in 
DOD's investment process--from the point where gaps in warfighting 
capability are assessed in JCIDS through the point where alternative 
solutions to resolve those gaps are analyzed under the DAS (see fig. 
1). 

Figure 1: DOD's Weapon System Investment Process: 

[See PDF for image] 

Source: DOD (data); GAO ( analysis and presentation). 

[End of figure] 

Successful Companies Take a Disciplined, Integrated Approach to 
Prioritize Market Needs and Initiate a Balanced Mix of Executable 
Development Programs: 

To ensure they achieve a balanced mix of executable development 
programs, the successful commercial companies we reviewed use a 
disciplined and integrated approach to prioritize market needs and 
allocate resources. This approach, known as portfolio management, 
requires companies to view each of their investments from an enterprise 
level as contributing to the collective whole, rather than as 
independent and unrelated. With this enterprise viewpoint, companies 
can effectively (1) identify and prioritize market opportunities and 
(2) apply available resources to potential products to select the best 
mix of products to exploit the highest priority--or most promising-- 
opportunities. Ultimately, each of the companies we reviewed seeks to 
achieve a balanced portfolio that maximizes the return on investments 
and moves the company toward achieving its strategic goals and 
objectives. This type of approach depends on strong governance with 
committed leadership, clearly aligned responsibility, and effective 
accountability at all levels of the organization. 

As depicted in figure 2, a portfolio management approach begins with an 
enterprise-level identification and definition of market opportunities 
and then the prioritization of those opportunities within resource 
constraints. Once opportunities have been prioritized, companies draft 
initial business cases for alternative product ideas that could be 
developed to exploit each of the highest priority opportunities. Each 
alternative product proposal--represented by a black dot--enters a 
gated review process. At each review gate, product proposals are 
assessed against corporate resources, established criteria, competing 
products, and the goals and objectives of the company as a whole. As 
alternatives pass through each review gate, the number is expected to 
decrease, until only those alternatives with the greatest potential to 
succeed make it into the product portfolio. 

Figure 2: Portfolio Management Approach to Product Investments: 

[See PDF for image] 

Source: GAO analysis and presentation of commercial best practices. 

[End of figure] 

Identifying and Prioritizing Market Opportunities Lays the Foundation 
for Achieving the Right Mix of Products: 

To make informed decisions about what market opportunities to target, 
the companies we reviewed first establish a strategy that lays out the 
overall goals, objectives, and direction for the company. As part of 
their strategy, companies identify enterprise-level sales and profit 
targets, strategic business areas they want to focus on, the extent to 
which current products and new development efforts will support their 
growth objectives, and how they will allocate resources across business 
units and functional areas. This strategy provides a framework for the 
companies' investment decisions. Within this framework, companies 
conduct a series of market analyses to develop a comprehensive 
understanding of the market environment, including product trends, 
technology trends, and customer needs. 

IBM for example, follows a structured market planning process to 
identify, prioritize, and target attractive market segments. The first 
phase of this process, called Market Definition, focuses on 
understanding the marketplace, including identifying potential 
customers and their needs.[Footnote 7] During this phase, IBM examines 
the marketplace and technology environments and identifies attractive 
market segments that contain potential market opportunities--where 
customer wants or needs exist. Each segment is categorized into one of 
four areas based on needs of the customers and the company's product 
offerings (see fig. 3): "strike zone," "traditional," "pushing the 
envelope," and "white space." The strike zone represents IBM's core 
business--market segments where IBM has an established customer base 
that it is successfully serving with existing product offerings. In 
contrast, white space represents market segments of new customers with 
wants and needs that are new and different for IBM. White space 
opportunities often require discovery and innovation. The traditional 
and pushing the envelope areas fall between the strike zone and white 
space. Traditional opportunities exist when new customers could be 
attracted to an existing market--one IBM is already active in--by 
modifying or enhancing existing products or services. Pushing the 
envelope opportunities exist where the needs of current customer groups 
move them into a new market segment. These attractive market segments 
are prioritized during the next phase of IBM's process, known as the 
Capability Assessment phase. During this phase each segment's overall 
attractiveness and potential profitability are assessed, along with 
IBM's available resources--like capital, cash, and current products-- 
and its competitive position within each segment. This analysis leads 
to the selection of targeted market segments. 

Figure 3: IBM Market Segmentation: 

[See PDF for image] 

Source: IBM's IPD process (data); GAO (analysis and presentation). 

[End of figure] 

Motorola emphasizes the importance of targeting the right market 
segments at the enterprise level to ensure that a balanced mix of 
project and resource investments is maintained. Officials noted that 
excessively focusing on segments that require new and innovative 
products can result in long cycle times, wasted money, and lost 
opportunities elsewhere. Likewise, critical opportunities can be lost 
when too much emphasis is placed on simply continuing to invest in old 
markets with old products. According to the officials we spoke with, 
the current investment mix for Motorola's Government and Enterprise 
Mobility Solutions business unit is roughly 70-20-10, where 70 percent 
of its projects and resources are dedicated to maintaining its core 
business, while 20 percent are invested in pursuing new markets with 
existing products or introducing new or enhanced products into existing 
markets, and the remaining 10 percent are dedicated to discovering new 
markets and new products. 

As part of their market analyses, companies increasingly refine their 
understanding of who their customers are and what they need. For 
several of the companies we met with, determining the needs of their 
customers is complex because they have multiple groups of customers to 
consider. For example, Eli Lilly has four customer groups with diverse 
needs: patients, doctors, insurance companies, and government 
regulators. This complexity is compounded when considering that success 
in a worldwide market is critically dependent on a company's ability to 
operate within different governmental systems, laws, and regulations; 
and regional markets. Several of the companies we reviewed use a 
variety of methods--including interviews, surveys, focus groups, and 
concept tests--to actively engage their customers and help determine 
what they need. Some companies also observe customer behaviors to 
identify unstated wants and needs that if met--assuming corporate 
knowledge and resources allow--could actually exceed customer 
expectations. While companies actively seek customer input to identify 
products that show the most promise and satisfy customer needs, 
customers generally do not identify specific products to be developed. 

Companies Follow a Disciplined Process to Identify New Products and 
Achieve a Balanced Portfolio: 

Once companies have identified and prioritized their market 
opportunities, they follow a disciplined process to assess the costs, 
benefits, and risks of potential product alternatives and allocate 
resources to achieve a balanced portfolio that spreads risk across 
products, aligns with the company's strategic goals and objectives, and 
maximizes the company's return on investment. At an early stage, each 
alternative product is expected to be accompanied by an initial 
business case that contains knowledge-based information on strategic 
relevance and estimates of cost, technology maturity, and the cycle 
time for getting the product from concept to market. To ensure 
comparability across alternatives, companies require initial business 
case information to be developed in a transparent manner, to use 
specific standards, and to report estimates within certain levels of 
confidence or allowable deviations. Each of the companies we reviewed 
also stressed the importance of having multiple management review 
points, or gates, at early phases to assess and prioritize alternative 
products. As products move through review gates, from ideas, to more 
concrete concepts, to the start of development where a final business 
case is made, companies expect uncertainties--which are typically 
inherent in the early phases--to be addressed and estimates to become 
more precise. Consequently, the number of viable alternatives tends to 
decrease at each review gate as those with the lowest potential for 
success and least value are terminated or deferred, while those that 
are poised to succeed and providing the best value are approved to 
proceed (illustrated in fig. 2). Companies emphasized that making tough 
go/no-go decisions is critical to keeping a balanced portfolio. Over 
time, as potential new products are identified, companies review them 
against other product investments (proposed and existing) and rebalance 
their portfolios based on those that add the most value. 

The companies we visited each follow a disciplined, gated review 
process to ensure that they commit to development programs that help 
balance the portfolio and that are executable given available corporate 
resources. This allows companies to avoid committing to more programs 
than their resources can support and ensure stability in the programs 
they invest in. Although the number of review gates prior to the start 
of full-scale product development varied between companies--ranging 
from four at Procter & Gamble, to eight at Motorola--they all required 
potential products to follow an established, disciplined process and 
meet specified criteria at each review point. For example, Caterpillar 
assesses product alternatives at four review gates prior to the start 
of development--three of which were recently added to enhance the rigor 
of its investment decision making. Each alternative must be supported 
by a draft business case that includes quantifiable data that can be 
compared with specific standards and used to determine if the related 
product can move past that gate. At each gate, alternatives are 
reviewed to ensure that knowledge about critical technologies, life- 
cycle costs, product reliability, and product affordability is being 
acquired and that the product contributes to achieving the company's 
strategic goals and objectives. 

Because developing a new drug is costly and time consuming, Eli Lilly 
requires that the data supporting potential new drugs must meet high 
standards to ensure that managers are informed to make sound investment 
decisions.[Footnote 8] Each potential new drug must be supported by an 
initial business case that contains information about safety and 
efficacy; forecasted revenue; expected unit demand; capital, medical, 
supply and material, development, and selling and marketing expenses; 
and general administrative costs. The initial business case must also 
identify critical success factors, state the probability of technical 
success, and provide a timeline that details when major milestone 
events are expected and how long it will take to get the associated 
drug to market. Eli Lilly assesses, approves, and funds proposed new 
drugs incrementally. At each milestone review a contract is established 
between the project team and a gatekeeper committee, which contains 
deliverables, time frames, and the costs to get to the next milestone. 
Once this contractual agreement is reached the budget is allocated for 
the entire phase. The gatekeeper committees expect each new drug 
proposal to achieve an 80-percent confidence level in their cost and 
schedule estimates for the next phase. This high level of confidence is 
achievable in large part because final budget estimates are not 
developed by project teams until 2 months prior to the milestone 
review. Projects are terminated at early points in the review process 
when it is determined that their critical success factors cannot be 
achieved. Because Eli Lilly's projects typically have a high degree of 
technical risk, only about 1 percent of those that start early 
development actually make it to the marketplace. 

Motorola officials also emphasized the importance of having sound 
information when assessing potential new products. They noted that a 
process without sound information will not produce good outcomes. 
Therefore, Motorola's Government and Enterprise Mobility Solutions 
business unit expects potential products to be supported by initial 
business cases containing data that meet specific standards and levels 
of confidence at each review gate. For example, cost estimates for 
potential products are developed in several phases and are expected to 
increase in confidence with each successive phase. Early in the 
investment planning, when an initial business case is first drafted, 
the confidence parameters are generous, ranging from as much as 75 
percent higher to as much as 25 percent lower than what the project 
will likely end up costing. By the time a product alternative reaches 
the beginning of product development, when a final business case is 
made, Motorola expects the cost estimates to be at confidence levels of 
10 percent higher and 5 percent lower. Proposed products that fail to 
meet the specified criteria at early review gates are either terminated 
or sent back to further mature and reenter the review process from the 
beginning. 

The companies we reviewed use a variety of portfolio management tools 
and methods to inform the investment and resource allocation decisions 
they make at each review gate. Some companies employ scoring methods, 
using experts to rate products based on a number of factors--such as 
strategic fit, risk, and economic value--and use that information to 
prioritize alternative products. Another common tool plots alternative 
products on a decision matrix that compares factors such as costs and 
benefits, or risks and rewards of competing alternatives. Using this 
type of matrix, alternative products are often represented by circles, 
where the size of the circles provides information about key 
constraints such as available annual resources or the estimated annual 
costs for each alternative. For example, figure 4 compares risk and 
expected rewards[Footnote 9] by plotting competing alternatives on a 
matrix. Alternatives that fall into the upper left quadrant are high 
risk and low reward, while alternatives that fall into the lower right 
quadrant are low risk and high reward. By weighing risk against rewards 
and considering constraints such as annual resources or annual cost, 
this tool provides critical information and a structured means to help 
managers make informed decisions. Company officials at Procter & Gamble 
emphasized the importance of selecting a balanced mix of products to 
pursue. They noted that pursuing only low-risk and high-reward products 
at the expense of more innovative, higher-risk products could cause the 
company to miss out on opportunities to improve their competitive 
standing in the marketplace. Likewise, excessive pursuit of higher-risk 
products with the potential for high returns could also result in lost 
opportunities to elsewhere. 

Figure 4: Risk Versus Rewards Matrix: 

[See PDF for image] 

Source: GAO. 

[End of figure] 

Recognizing the inherent risks in pursuing a new development program-- 
that overruns or underruns in one business case result in lost 
opportunity to invest resources in another worthwhile project--IBM 
permits products to deviate from their original business case estimates 
as long as the deviation is within established limits. These limits are 
specified in a contractual document resulting from negotiations between 
senior management and project managers and signed at the beginning of 
product development. Product development teams are expected to execute 
according to the contract; if established thresholds are breached, 
action is taken immediately to reassess the product within the context 
of the portfolio and determine whether it is still a relevant and 
affordable investment to pursue. 

Successful Portfolio Management Requires Strong Governance with 
Committed Leadership, Empowered Decision Makers, and Effective 
Accountability: 

Successful portfolio management requires strong governance with 
committed leadership that empowers portfolio managers to make decisions 
about the best way to invest resources and holds those managers 
accountable for the outcomes they achieve. The companies we reviewed 
indicated that it is critical to have commitment from the top leaders 
of the organization and recognition at all levels that what is best for 
the company must be a priority, and not simply what is best for a 
particular business unit or product line. In addition, the companies 
emphasized that roles and responsibilities for implementing portfolio 
management, including the designation of who is responsible for product 
investment decisions and oversight, must be clearly defined. Because 
portfolio managers are on the front line, the companies we reviewed 
empower these managers to make product investment decisions and hold 
them accountable for outcomes, not just for individual products but 
also for the overall performance of their portfolios. To support their 
portfolio managers, the companies encourage collaboration and 
communication, including sharing bad news early. Several companies also 
emphasized the importance of supporting their portfolio managers with 
cross-functional teams, composed of representatives from the key 
functional areas within the company--such as science and technology, 
marketing, engineering, and finance--to ensure that they are adequately 
informed when making investment decisions. To ensure accountability, 
companies often use incentives and disincentives, including promotion 
and termination. We have previously reported that high-performing 
organizations have monetary and other rewards that clearly link 
employee knowledge, skills, and contributions to achieving the 
organization's goals and objectives.[Footnote 10] These organizations 
underscore the importance of holding individuals accountable and 
aligning performance expectations with organizational goals and cascade 
those expectations down to lower levels. Companies stressed that the 
transformation to portfolio management takes time and requires not only 
process changes but cultural changes throughout the company. 

Eli Lilly emphasized that a key to making its portfolio management 
process work is having a single committee with a high-level official in 
charge responsible for making product investment decisions. Previously, 
the company had a multi-layered committee structure in place, and 
decisions were made based on reaching a consensus--an approach that was 
viewed as cumbersome and lengthy. Eli Lilly also ensures accountability 
by directly linking management and employee bonuses to the overall 
success of the company. Individual employee performance objectives are 
aligned with specific company objectives, such as meeting budgetary 
goals, time frames, and data quality levels for a given project. 
Achievement of individual employee objectives is measured periodically 
to provide feedback to the employee. Eli Lilly officials stressed that 
having the right performance metrics in place is important because 
ultimately you get what you measure; therefore, be sure to measure the 
right things. 

Motorola considers accountability to be the critical factor in making 
its portfolio management process successful. In addition, Motorola's 
culture is not averse to reporting bad news to management. Project 
managers are encouraged to report problems early so that they can be 
addressed before they get out of control. Senior managers, however, are 
not intimately involved in the day-to-day decision making for 
individual products. That responsibility, in nearly every case, is 
delegated to the business unit general manager. The general manager of 
a business unit is held accountable for ensuring that the products 
within his unit succeed at all levels. The general manager is 
responsible for holding product managers accountable for the attainment 
of critical knowledge at key points and the performance of their 
individual products overall. General managers and product managers can 
be fired for not meeting objectives. Motorola believes that if managers 
are held accountable for results, then they have more desire to get it 
right. 

Lacking an Integrated, Portfolio-Based Approach, DOD Has Too Many 
Programs Competing for Limited Resources: 

Although the military services fight together on the battlefield as a 
joint force, they do not identify warfighting needs and make weapon 
system investment decisions together. DOD has taken steps to identify 
warfighting needs through a more joint requirements process, but the 
department's service-centric structure and fragmented decision-making 
processes are at odds with the integrated, portfolio management 
approach used by successful commercial companies to make enterprise- 
level investment decisions. Consequently, DOD has less assurance that 
its weapon system investment decisions address its most important 
warfighting needs and are affordable in the context of its overall 
fiscal resources. In addition, DOD commits to products earlier than the 
companies we reviewed and with far less knowledge about their cost and 
feasibility. This leads to poor program outcomes and funding 
instability, as the department attempts to fix troubled programs by 
taking funds from others. 

Service-centric Structure and Fragmented Decision-making Processes 
Impede DOD's Ability to Prioritize Warfighting Needs: 

Although recent DOD policy emphasizes a more joint approach to 
identifying and prioritizing warfighting needs,[Footnote 11] DOD's 
service-centric structure and fragmented decision-making processes 
hinder the policy's successful implementation. This policy, which 
introduced the JCIDS process, calls for a wider range of stakeholders 
than before, including more customer (i.e., combatant command) 
involvement; introduces new methodologies intended to foster jointness; 
and groups warfighting needs into eight functional areas based on 
warfighting capabilities--such as netcentric, force application, and 
battlespace awareness[Footnote 12]--that cut across the military 
services and defense agencies. The JCIDS process emphasizes early 
attention to the fiscal implications of newly identified needs, 
including identifying ways to pay for new capabilities by divesting the 
department of lower priority or redundant capabilities. Despite these 
provisions, assessments of warfighting needs continue to be driven by 
the services and to be based on investment decision-making processes 
that do not function together to ensure that DOD pursues needs that its 
resources can support. 

The military services identify warfighting needs individually, and 
department-level organizations are not optimized to integrate the 
services' results or evaluate their fiscal implications early on. 
Historically, this approach has contributed to duplication in weapon 
systems and equipment that does not interoperate. At the department 
level, Functional Capability Boards oversee each of the eight 
functional areas, reviewing the services' assessments, and providing 
recommendations to the Joint Requirements Oversight Council (JROC), 
which leads the JCIDS process. However, defense experts and DOD 
officials report that the Functional Capability Boards do not have the 
staff or analytical resources required to effectively evaluate service 
assessments within the context of the broader capability portfolio and 
assess whether the department can afford to address a particular 
capability gap. Several recent studies have recommended that DOD 
increase joint analytical resources for a less stovepiped understanding 
of warfighting needs.[Footnote 13] In addition, the boards lack the 
authority to allocate resources and to make or enforce decisions to 
divest their capability area of existing programs to pay for new ones-
-authority successful companies provide to their portfolio managers. 
Finally, some defense experts contend that the service ties of JROC's 
members--that is, the services' Vice Chiefs and the Assistant 
Commandant of the Marine Corps--reinforce service stovepipes. To better 
ensure a more joint perspective, they recommend a more diverse JROC, 
with representatives from other department-level organizations and the 
combatant commands.[Footnote 14] 

Resource allocation decisions are made through a separate process--the 
Planning, Programming, Budgeting, and Execution system (PPBE)--which 
hinders the department's ability to weigh the relative costs, benefits, 
and risks of investing in new weapon systems early on. Within the PPBE 
system, the individual military services are responsible for budgeting 
and allocating resources under authority that is commonly understood to 
be based on Title 10 of the United States Code.[Footnote 15] PPBE is 
structured by military service and defense program, although the 
department integrates data on the services' current and projected 
budget requests under 11 crosscutting mission areas called Major Force 
Programs. The cross-cutting view provided by the Major Force Program 
structure is intended to facilitate a strategic basis for resource 
allocation, allowing the Secretary of Defense to more easily see where 
the greatest mission needs are and to re-allocate funds to meet those 
needs regardless of which service stands to gain or lose. However, we 
have reported in the past that the Major Force Program structure has 
not provided sufficient visibility in certain mission areas.[Footnote 
16] Moreover, although they cut across the services, the program 
mission areas are not consistent with the more recently established 
capability areas used in the JCIDS process,[Footnote 17] and as a 
result, it is difficult to relate resources to capabilities. For 
example, in prior work, we observed that the Major Force Programs 
contain large numbers of programs with varied capabilities, 
complicating comparisons needed to understand defense capabilities and 
associated trade-off decisions.[Footnote 18] We have recommended that 
DOD report funding levels for defense capabilities in its Future Years 
Defense Program report to the Congress, which is currently organized by 
the Major Force Programs. 

In addition, our analysis of DOD's investment accounts--which pay for 
developing, testing, and buying weapon systems and other equipment-- 
indicates that DOD generally does not allocate resources on a strategic 
basis. Figure 5 illustrates that the service allocations as a 
percentage of the department's overall investment budget have remained 
relatively static for the 25-year period we examined, even though DOD's 
strategic environment and warfighting needs have changed dramatically 
during that time, with the demise of the cold war and the emergence of 
the global war on terror.[Footnote 19] In contrast, successful 
commercial companies using portfolio management would expect to see 
their resource allocations across business areas to reflect changes in 
the marketplace and the competitive environment. 

Figure 5: Service Allocations of DOD's Investment Budget (FY1986 
through FY2011): 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

PPBE and JCIDS are led by different organizations (see fig. 6), as is 
the third of the three processes involved in DOD's weapon system 
investment decisions, the Defense Acquisition System (DAS), making it 
difficult to hold any one person or organization accountable for 
investment outcomes. The 2006 Quadrennial Defense Review highlighted 
the need for governance reforms,[Footnote 20] and a 2006 study 
commissioned by DOD observed that the budget, acquisition, and 
requirements processes are not connected organizationally at any level 
below the Deputy Secretary of Defense, concluding that this structure 
induces instability and erodes accountability.[Footnote 21] The Under 
Secretary of Defense/Acquisitions, Technology, and Logistics (USD/ 
AT&L) has stated that weapon system investment decisions are a shared 
responsibility, and, therefore, no one individual is accountable for 
these decisions. At a broader, strategic level, we have stated in prior 
work that DOD has lacked sustained leadership and accountability for 
various department-wide management reform efforts,[Footnote 22] 
including the establishment of an effective risk management approach as 
a framework for decision making.[Footnote 23] This approach would link 
strategic goals to plans and budgets, assess the value and risks of 
various courses of action as a tool for setting investment priorities 
and allocating resources at the department level, and use performance 
measures to assess outcomes. To address the lack of sustained 
leadership, we have supported legislation to create a chief management 
official at DOD.[Footnote 24] 

Figure 6: Governance of DOD's Investment Process: 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

The Office of the Secretary of Defense (OSD) does not assess the 
funding implications of a proposed program at the front end of the 
investment process, when it is initially validated by JROC. JCIDS is a 
continuous, need-driven process that unfolds in response to warfighting 
needs as they are identified. However, PPBE is a calendar-driven 
process comprised of phases that occur over a 2-year cycle, thus OSD's 
formal review of a proposed program is not often synchronized with 
JROC's, and can occur several years later.[Footnote 25] Nevertheless, 
according to Joint Staff and AT&L officials we met with, proposed 
programs begin to gain momentum when they are validated by JROC, and 
they become very difficult to stop. These officials indicated that 
momentum begins to gather because the services start programming and 
budgeting for the proposed capability right away to secure funding, 
generally several years before actual product development begins and 
before OSD formally reviews the services' programming and budgeting 
proposals. In the interim, the services have not only budgeted for 
their proposed programs, but established a program office, conducted 
their Analysis of Alternatives, and identified specific user 
requirements. OSD's programming and budgeting review occurs at the back 
end of the investment process, when it is difficult and disruptive to 
make changes, such as terminating existing programs to pay for new, 
higher priority programs. 

These practices have contributed to the department starting more 
programs than its resources can support. DOD defers much of the 
additional cost of its programs into the future, resulting in what some 
have characterized as a fiscal bow wave (illustrated in fig. 7). This 
bow wave has grown at a pace that greatly exceeds DOD's annual funding 
increases. The cost remaining for DOD's major weapons programs 
increased almost 135 percent between 1992 and 2006, while the 
department's annual funding level only increased 57 percent over that 
same time period. If this trend goes unchecked, Congress will likely be 
faced with a difficult choice: pull funds from other high-priority 
federal programs to support DOD's acquisitions or accept less 
warfighting capability than originally promised. 

Figure 7: Costs Remaining Versus Annual Appropriations for Major 
Defense Acquisitions: 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

DOD Commits to a Solution Earlier and with Less Knowledge: 

DOD commits to a solution to address a warfighting need earlier in the 
investment process than commercial companies do and before it has 
adequate knowledge about cost and technical feasibility. Proposed 
options for resolving a gap in military capability are submitted in an 
Initial Capabilities Document (ICD). DOD guidance states that this 
document should contain a range of approaches based in part on the cost 
and technological feasibility posed by the approaches, laying the 
foundation for a more detailed Analysis of Alternatives to be conducted 
under the Defense Acquisition System. In addition, JROC is to receive a 
briefing on the ICD that follows a standard format and addresses such 
issues as: 

* linkage of the proposal to strategic guidance; 

* the time frame within which the capability is needed; 

* the threat/operational environment; 

* risks and assumptions (including the risk associated with proceeding 
and not proceeding with solutions to each); and: 

* a description of the best materiel and non-materiel approaches based 
upon cost, efficacy, performance, technology maturity, and risk. 

Although DOD guidance calls for the analysis of a solution's cost and 
feasibility, we found that ICDs contained little of this type of 
information. Several DOD officials we met with, who are directly 
involved in the JCIDS process, did not believe cost and feasibility 
information was mandated at this point. In our review of 14 
unclassified ICDs approved by JROC from 2003-06, we found that 11 did 
not contain acquisition cost estimates and 12 did not contain estimates 
of the technical feasibility of proposed solutions.[Footnote 26] We 
also found that JCIDS guidance does not specify the level of accuracy 
sought in cost and feasibility estimates, and a white paper that does 
provide recommendations in this regard is advisory.[Footnote 27] 

We found that ICDs generally focused on the strategic, or operational, 
relevance of proposed solutions, but a lack of guidance and an evolving 
methodology have raised questions about the accuracy of data supporting 
those assessments. JCIDS uses new joint warfighting concepts[Footnote 
28] to translate top-level military strategy into the capabilities a 
commander might need on the battlefield. The joint concepts underpin a 
capabilities-based approach[Footnote 29] to identifying requirements, 
in which analyses are expected to focus on broad military capabilities 
rather than service-specific platforms. However, the joint concepts and 
capabilities-based assessments are works in progress. The concepts are 
being updated due to concerns about their scope, and guidance on 
conducting a capabilities-based analysis has been lacking. Several DOD 
officials we met with stated that assessments vary in their rigor, and 
a senior Joint Staff official said that training on requirements 
development is one of three central challenges at present. In January 
2007, we reported that DOD officials described concerns about the 
analytical framework for a capabilities-based assessment on joint 
seabasing, which could lead to inaccurately identifying gaps in 
implementing the concept.[Footnote 30] Enhancing a seabasing capability 
is expected to be costly and could be the source of billions of dollars 
of investment if DOD chooses an option involving the development of new 
ships. 

DOD does not consistently follow a disciplined review process to ensure 
that proposed solutions are making progress toward an executable 
development program, although DOD policy emphasizes that such reviews 
are necessary.[Footnote 31] DOD's policy identifies several key 
decision points prior to starting a new weapon system development 
program: 

* an initial decision point, where the Initial Capabilities Document is 
reviewed, validated, and approved by the JROC; 

* a Concept Decision review, where entry into the concept refinement 
phase of the Defense Acquisition System should be authorized; and: 

* a Milestone A decision point, where a preferred solution and a 
technology development strategy should be reviewed and 
approved.[Footnote 32] 

Since Initial Capabilities Documents generally do not contain 
information on cost and technical feasibility, the JROC does not have a 
sufficient basis for making go/no-go decisions at the initial decision 
point. In the 4 years since JCIDS was implemented, nearly all of the 
warfighting needs identified by the services and submitted for review 
in an ICD have been validated and sent into the acquisition pipeline 
for further analysis as potential programs, which calls into question 
whether go/no-go decisions are the point of this first key gate. 
Information on cost and feasibility is generally developed after the 
ICD is approved and proposed solutions undergo further refinement 
through an Analysis of Alternatives (AOA). An AOA should compare 
alternative solutions in terms of life-cycle cost, schedule, and 
operational effectiveness, leading up to the identification of a 
preferred alternative. However, officials from PA&E and the Joint Staff 
indicate that AOAs often make a case for a single preferred solution. 
Several of them indicated other concerns about AOAs, such as not 
setting up trade-off discussions, lack of analytical rigor, length, and 
timeliness. 

In any case, the next review points--the Concept Decision and Milestone 
A--are often skipped; thus, the opportunity to review an evolving 
business case and to make go/no-go decisions is bypassed. In prior 
work, we found that 80 percent of the programs we reviewed entered the 
Defense Acquisition System at Milestone B without holding any prior 
major reviews,such as a Milestone A review.[Footnote 33] Such reviews 
are intended to provide acquisition officials with an opportunity to 
assess whether program officials had the knowledge needed to develop an 
executable business case. Senior officials with OSD confirmed that this 
is a common practice among defense acquisition programs. We concluded 
that this practice eliminates a key opportunity for decision makers to 
assess the early product knowledge needed to establish a business case 
that is based on realistic cost, schedule, and performance 
expectations. In addition, we found that programs are regularly 
approved to begin development even though officials reported levels of 
knowledge below the criteria suggested in DOD's acquisition policy. 

There is, then, generally little department-level oversight between the 
point at which an ICD is approved and when system-level requirements 
are validated and product development is initiated. At this point, as 
we indicated earlier, there is generally no turning back, because the 
services have invested considerable time and money, established a 
budget, and formed a constituency for a proposed program, and decision 
makers become reluctant to terminate a program or send it back for 
further study. 

DOD Is Piloting Several Initiatives to Address Disconnects in 
Investment Decision-making: 

In response to the 2006 Quadrennial Defense Review and other recent 
acquisition reform studies,[Footnote 34] DOD has undertaken several 
key, interrelated initiatives intended to strengthen the department's 
approach to investment decisions. The initiatives include (1) taking a 
new approach to reviewing proposed concepts that will provide decision 
makers with an early opportunity to evaluate trade-offs among 
alternative approaches to meeting a capability need, (2) testing 
portfolio management approaches in selected capability areas to 
facilitate more strategic choices about how to allocate resources 
across programs, and (3) using capital budgeting as a potential means 
to stabilize program funding. While promising, these initiatives do not 
fundamentally change DOD's existing service-centric framework for 
making weapon system investment decisions. 

To address a perceived gap between DOD's major decision-making 
processes and provide a department-level means to assess potential 
solutions (materiel and non-materiel) to fill a validated capability 
need, DOD is testing a new approach to a Concept Decision review, which 
will take place after a warfighting need is validated by the JROC. This 
new approach is intended to focus attention on the affordability and 
feasibility of potential solutions and generate early cost, schedule, 
and performance trade-offs prior to the point of a significant 
investment commitment. As currently proposed, the Concept Decision will 
be informed by a newly required Evaluation of Alternatives that will 
integrate the Functional Solutions Analysis conducted under JCIDS with 
the Analysis of Alternatives conducted under the acquisition system and 
lay out the relative merits and limitations of potential solutions. 
Furthermore, concept decision reviews will be implemented by a tri- 
chair board consisting of lead decision makers from the JCIDS, PPBE, 
and DAS processes. While promising, the Concept Decision review largely 
reinstitutes a review point that already existed but was only 
intermittingly used. For Concept Decision reviews to be effective, DOD 
will have to establish enforcement and accountability mechanisms to 
ensure the reviews are actually implemented. In addition, the extent to 
which the concept reviews can achieve desired effects will depend on 
what authority Concept Decisions carry and who will be held 
accountable, particularly in light of the service-dominated investment 
structure that currently exists. 

The department has also begun to pilot-test capability-based portfolio 
management, selecting four joint capability areas to focus on--joint 
command and control, joint net-centric operations, battlespace 
awareness, and joint logistics. The intent is to enable the department 
to develop and manage capabilities, as opposed to simply individual 
programs, and enhance the integration and interoperability within and 
across sets of capabilities. Each portfolio is being structured 
somewhat differently to help the department determine how best to 
proceed with portfolio management. All, however, are intended to focus 
initially on existing programs and to operate within DOD's existing 
decision-making framework. The portfolios are largely advisory and 
will, as a first step, provide input to decisions made through the 
JCIDS, PPBE, and DAS processes. At this point, the capability portfolio 
managers have not been given direct authority to manage fiscal 
resources and make investment decisions. Without portfolios in which 
managers have authority and control over resources, DOD is at risk of 
continuing to develop and acquire systems in a stovepiped manner and of 
not knowing whether its systems are being developed within available 
resources. 

DOD is also examining the use of capital accounts as a potential means 
of stabilizing program funding, which has long been cited as a 
significant issue in program management. This capital budgeting pilot 
initiative is in the early stages of planning, and the specifics of how 
such accounts will be implemented are being developed, but the intent 
is for DOD to commit a set amount of funding for the development 
portion of a project and hold to that commitment by not adjusting 
funding up or down until the product is delivered. In addition to 
resource constraints, programs would be given a fixed amount of time to 
get from one milestone to the next. If successful, this initiative 
could represent a step toward stabilizing long-term costs within major 
defense acquisition programs, as well as a strengthening of the ability 
of program managers to conduct long-term planning and control costs. 
However, for this initiative to be effective, DOD will need to overcome 
long-standing problems it has had in starting programs without 
sufficient knowledge of the costs, requirements, and technologies 
needed to develop proposed weapon systems. Unless this changes, it is 
unlikely that capital accounts will lead to increased program 
stability. 

Conclusions: 

While DOD has increasingly strengthened its ability to operate as a 
joint force on the battlefield, the department's organizational 
structures, processes, and practices for planning and acquiring weapon 
systems are not similarly joint. Put simply, DOD largely continues to 
base its investment decisions on service-driven analyses that do not 
provide an enterprise-level understanding of overall warfighting needs 
and on individual platforms rather than broader sets of capabilities. 
In contrast, successful commercial companies use an integrated 
portfolio management approach to focus early investment decisions on 
products collectively at an enterprise level and to ensure there is a 
sound basis to justify the commitment of resources. By following a 
disciplined, integrated process--where the relative pros and cons of 
market opportunities and competing product proposals are assessed based 
on available resources and customer needs, and where tough decisions 
about which investments to pursue are made--companies are able to 
reduce duplication between business units, move away from 
organizational stovepipes, and effectively support each new development 
program they commit to. Until DOD takes a joint, portfolio management 
approach to weapon system acquisition--with functionally aligned 
entities that have the requisite responsibility, authority, and control 
over resources--it will continue to struggle to effectively prioritize 
warfighting needs, make informed trade-offs, and achieve a balanced mix 
of weapon systems that are affordable, feasible, and provide the best 
military value to the warfighter. Committing to more programs than the 
budget can support and approving programs based on insufficient 
knowledge to effectively manage risks will further delay providing 
critical capabilities to the warfighter and lead to lost opportunities 
to address other current and emerging needs. 

Recommendations: 

We recommend that the Secretary of Defense implement an enterprise-wide 
portfolio management approach to making weapon system investments that 
integrates the assessment and determination of warfighting needs with 
available resources and cuts across the services by functional or 
capability area. To ensure the success of such an approach, the 
Secretary should establish a single point of accountability at the 
department level with the authority, responsibility, and tools to 
ensure that portfolio management for weapon system investments is 
effectively implemented across the department. 

In addition, the Secretary should ensure that the following commercial 
best practices, identified in this report, are incorporated: 

* implement a review process in which needs and resources are 
integrated early and in which resources are committed incrementally 
based on the achievement of specific levels of knowledge at established 
decision points; 

* prioritize programs based on the relative costs, benefits, and risks 
of each investment to ensure a balanced portfolio; 

* require increasingly precise cost, schedule, and performance 
information for each alternative that meets specified levels of 
confidence and allowable deviations at each decision point leading up 
to the start of product development; 

* establish portfolio managers who are empowered to prioritize needs, 
make early go/no-go decisions about alternative solutions, and allocate 
resources within fiscal constraints; and: 

* hold officials at all levels accountable for achieving and 
maintaining a balanced, joint portfolio of weapon system investments 
that meet the needs of the warfighter within resource constraints. 

We also recommend that the Secretary take steps to support department- 
level decision makers and portfolio managers by developing a stronger 
joint analytical capability to assess and prioritize warfighting needs. 

Agency Comments and Our Evaluation: 

DOD provided us with written comments on a draft of this report. The 
comments appear in appendix II. 

DOD concurred with the majority of our recommendations and partially 
concurred with two. Generally, in responding to these recommendations, 
DOD stated that it is undertaking several initiatives and pilot efforts 
to improve the department's approach to investment and program decision 
making, and that implementation of any new business rules will be 
contingent upon the outcome of these initiatives. The department also 
stated that it is experimenting with portfolio management, related 
authorities and organizational constructs, and integrated decision- 
making processes. 

We believe that these initiatives and pilot efforts may be steps in the 
right direction, but we are concerned that they do not go far enough to 
address the systemic cultural and structural problems identified in 
this report. DOD has attempted many similar acquisition reform efforts 
over the past 3 decades, including significant revisions to both 
defense requirements and acquisition policy. However, despite these 
efforts, weapon system acquisition programs continued to experience 
cost overruns, schedule slips, and performance shortfalls. The 
department's current initiatives are likely to face the same fate 
because they do not fundamentally change DOD's service-centric 
framework or sufficiently integrate its decision-making processes for 
making weapon system investments. 

DOD did not provide comments regarding our recommendation that the 
Secretary establish a single point of accountability at the department 
level with the authority, responsibility, and tools to ensure that 
portfolio management for weapon system investments is effectively 
implemented across the department. We believe that a single point of 
accountability is necessary to successfully implement a portfolio 
management approach and integrate DOD's fragmented decision-making 
processes under one senior official who is accountable for weapon 
system investment outcomes. We further believe that our recommendations 
would better position DOD to make tough, knowledge-based choices among 
potential weapon system investments. 

We are sending copies of this report to the Secretary of Defense; the 
Secretaries of the Air Force, Army, and Navy; and the Director of the 
Office of Management and Budget. We will provide copies to others on 
request. This report will also be available at no charge on GAO's Web 
site at http://www.gao.gov. 

If you have any questions about this report or need additional 
information, please call me at (202) 512-4841 (sullivanm@gao.gov). Key 
contributors to this report were John Oppenheim, Assistant Director; 
Lily Chin; John Krump; Matthew Lea; Travis Masters; Sean Seales; Karen 
Sloan; Susan Woodward; and Rebecca Yurman. 

Signed by: 

Michael J. Sullivan: 
Director, Acquisition and Sourcing Management: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

This report examines the Department of Defense's (DOD) requirements 
identification and resource allocation processes for major weapons 
systems. The primary focus is on identifying successful private-sector 
principles and practices that could be adopted by DOD to help improve 
stability in weapon system acquisition programs. Specifically, our 
objectives were to (1) identify best practices of successful commercial 
companies for ensuring that they pursue the right mix of programs to 
meet the needs of their customers within resource constraints and (2) 
compare DOD's enterprise-level processes for investing in weapon 
systems to those practices. Our work was conducted between March 2006 
and February 2007, in accordance with generally accepted government 
auditing standards. 

We analyzed the outputs of DOD's investment decision-making support 
processes--the requirements determination process known as JCIDS and 
the resource allocation process known as PPBE--using criteria 
established in DOD policy and in previous GAO reports. We identified 
impacts of the existing processes by analyzing quantitative and 
qualitative data on DOD spending trends, conducting interviews with DOD 
officials, and reviewing previous reports by GAO and by other 
knowledgeable audit and research organizations. In addition, we met 
with officials representing the Office of the Secretary of Defense, 
Joint Staff, and military services. At each of these locations, we 
conducted interviews that helped us describe the current condition of 
DOD's requirements identification and resource allocation processes. We 
also reviewed DOD and military service policies and funding documents 
pertaining to the DOD requirements identification process and resource 
allocation decisions for major weapons systems. Specifically, we 
reviewed the contents of 14 unclassified Initial Capability Documents 
that were finalized after June 24, 2003--the publication date for the 
JCIDS instruction--to assess the extent to which they contained cost 
and technical feasibility information. Those 14 ICDs were unclassified, 
weapon system-related ACAT I, II, or III ICDs that were contained in 
the Joint Staff requirements database. We relied on previous GAO 
reports that highlight both the symptoms and causes of unstable 
requirements and funding in DOD weapons acquisition programs. A list of 
these reports can be found at the end of this report. In addition, we 
reviewed recent key studies and reports addressing acquisition reform 
issues by the Center for Strategic International Studies, Institute for 
Defense Analysis, the U.S. Naval War College, the Defense Acquisition 
Performance Assessment Project, the Joint Defense Capabilities Study 
Team, the Joint C4ISR Decision Support Center, the Defense Science 
Board, and the 2001 and 2005 Quadrennial Defense Reviews. 

We also reviewed pertinent literature from authoritative corporate, 
academic, and professional organizations, to identify commercial best 
practices and processes that could be used by DOD to improve its weapon 
system investment decision-making processes. In addition we conducted 
case studies of five leading commercial companies. In selecting them, 
we sought to identify companies that were recognized in the literature 
for best practices, had large and diversified portfolios of products, 
and make significant investments in the development and production of 
new products. For each of the companies, we interviewed management 
officials knowledgeable about their requirements identification and 
resource allocation activities, to gather consistent information about 
processes, practices, and metrics the companies use to help achieve 
successful product development outcomes. Below are descriptions of the 
five companies featured in this report: 

Motorola: 

Motorola is a Fortune 100 global communications leader that provides 
seamless mobility products and solutions across broadband, embedded 
systems, and wireless networks. According to Motorola's 2005 Corporate 
Profile, the company is the market leader in mission critical wireless 
communication systems, two-way radios, embedded telematics systems, 
digital set-top shipments, cable modem shipments, digital head-ends, 
embedded computer systems for communication applications, CDMA 
infrastructure sales (excluding the United States), and second in world 
wide wireless handsets. Motorola achieved net sales of $31.323 billion 
and spent $3.060 billion on research and development in 2004. The 
corporation has approximately 68,000 employees, in 320 facilities, 
spanning 73 countries. We met with the management of Motorola's 
Government & Enterprise Mobility Solutions and Global Telecom Solutions 
sectors in Schaumburg, Illinois. 

International Business Machines (IBM): 

IBM is one of the world's largest technological companies, spending 
about $3 billion annually on research and development activities. It is 
the largest supplier of hardware, software, and information technology 
services. With 3,248 U.S. patents, IBM earned more patents than any 
other company for the 12th consecutive year in 2004. In the past 4 
years, IBM inventors received more than 13,000 patents--approximately 
5,400 more than any other patent recipient. IBM has over 329,000 
employees worldwide. We met with managers from IBM Integrated Product 
Development (IPD) in Somers, New York. 

Procter & Gamble (P&G): 

Procter & Gamble Corp. (P&G) is a leading producer of consumer goods. 
It currently leads in global sales and marketshare among all fabric 
care, baby care, feminine care, and hair care products. It currently 
has over 130,000 employees in 80+ countries. Twenty-two of its brands 
have annual gross sales exceeding $1 billion each. In fiscal year 2005/ 
2006, P&G invested $2.075 billion or 3 percent of net sales in research 
and development (R&D). This ranks them as one of the top 20 largest 
research & development investors among U.S.-based companies. P&G has 
more Ph.D.s working in labs around the world than the combined science 
and engineering faculties of Harvard, MIT, and Berkeley. We met with 
the management of P&G's New Initiative Delivery team in Cincinnati, 
Ohio. 

Eli Lilly Corporation: 

Eli Lilly is a global pharmaceutical company and one of the world's 
largest corporations. It was founded over 130 years ago and currently 
employs approximately 42,000 people worldwide, including 13,991 
employed at its headquarters in Indianapolis, Ind. Approximately 8,336 
employees (19 percent of the total work force) are engaged in research 
and development (R&D); clinical research is conducted in over 50 
countries; there are R&D facilities in 9 countries; and manufacturing 
plants in 13 countries. Its products are marketed in 143 countries. 
Lilly's net sales in 2005 were $14.6 billion. Eli Lilly strives to grow 
sales by 6 percent to 7 percent each year. In 2005, $3 billion was 
spent on R&D, a $334.4 million increase from the previous year. 
Currently, R&D represents 20.7 percent of sales. Lilly's total R&D 
investment in the last 5 years from continuing operations was $12.5 
billion. We met with managers from Eli Lilly's Corporate Headquarters 
in Indianapolis, Ind. 

Caterpillar Corporation: 

Caterpillar is a technology leader and the world's leading manufacturer 
of construction and mining equipment, diesel and natural gas engines, 
and industrial gas turbines. In 2005, its total sales and revenues were 
$36.3 billion, and its total R&D expenditures exceeded $1 billion, 
compared with $20.5 billion sales and $696 million R&D in 2001. Between 
2001 and 2005, the average return on equity of its stockholders' shares 
more than doubled. Caterpillar has over 85,000 employees, and over 
105,000 people are employed by Caterpillar's dealers worldwide. We met 
with managers responsible for Caterpillar's New Product Introduction 
(NPI) process in Peoria, Illinois. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Under Secretary Of Defense: 
3000 Defense Pentagon: 
Washington, DC 20301-3000: 
Acquisition, Technology And Logistics: 

Mr. Michael J. Sullivan: 
Director, Acquisition and Sourcing Management: 
U. S. Government Accountability Office: 
441 G. Street, N. W. 
Washington, DC 20548: 

Mar 21 2007: 

Dear Mr. Sullivan: 

This is the Department of Defense (DoD) response to the GAO Draft 
Report, GAO-07-388 "Best Practices: An Integrated Portfolio Management 
Approach to Weapon System Investments Could Improve DoD's Acquisition 
Outcomes," dated February 14, 2007 (GAO Code 120516). 

The DoD concurs with five recommendations and partially concurs with 
two of the draft report recommendations. The rationale for our position 
is included in the enclosure. 

We appreciate the opportunity to comment on the draft report. My point 
of contact for this effort is Mr. David Crim, (703) 697-5385, 
david.crim@osd.mil: 

Sincerely, 

Signed by: 

Dave G. Ahem: 
Director: 
Portfolio Systems Acquisition: 

Enclosure: 
As stated: 

GAO Draft Report Dated February 14, 2007 GAO-07-388 (GAO Code 120516): 

"Best Practices: An Integrated Portfolio Management Approach To Weapon 
System Investments Could Improve Dod's Acquisition Outcomes" 

Department Of Defense Comments To The GAO Recommendations: 

Recommendation 1: The GAO recommended that the Secretary of Defense 
implement an enterprise-wide portfolio management approach to making 
weapon system investments that integrates the assessment and 
determination of warfighting needs with available resources and cuts 
across the services by functional or capability area. (p. 25/GAO Draft 
Report): 

DOD Response: Concur. As noted in the report, in response to the 2006 
Quadrennial Defense Review (QDR) and other recent acquisition reform 
studies, the DoD is undertaking several initiatives and pilot 
activities to improve the department's approach to investment and 
program decision making. These initiatives take into account many of 
the best practices employed by commercial industry as described in the 
report, including corporate-level priorities, a framework linking 
strategic goals to plans and budgets, and performance metrics. 
Implementation of any new business rules will be contingent upon the 
outcome of these pilot initiatives, a thorough understanding of the 
best overall approach, and the overall benefits gained. 

Recommendation 2: The GAO recommended that the Secretary of Defense 
implement a review process in which needs and resources are integrated 
early and in which resources are.committed incrementally based on the 
achievement of specific levels of knowledge at decision points. (p. 26/ 
GAO Draft Report): 

DOD Response: Concur. As noted in the report, in response to the 2006 
QDR and other recent acquisition reform studies, the DoD is undertaking 
several initiatives and pilot activities to integrate the department's 
major decision processes. Concept Decision and Capital Accounts are two 
examples of such initiatives that are consistent with this report 
recommendation. Foundational to these and other pilot activities is 
available knowledge to support senior leader decision making at key 
points in the life cycle. The Department's efforts to promote 
transparent data, along with its major decision processes support the 
incremental commitment of resources. Implementation of any new business 
rules will be contingent upon the outcome of these initiatives, a 
thorough understanding of the best overall approach and the overall 
benefits gained. 

Recommendation 3: The GAO recommended that the Secretary of Defense 
prioritize programs based on the relative costs, benefits, and risks of 
each investment to ensure a balanced portfolio. (p. 26/GAO Draft 
Report): 

DOD Response: Partially Concur. As noted in the report, in response to 
the 2006 QDR and other recent acquisition reform studies, the DoD is 
undertaking several initiatives to enable strategic choice. This 
involves balancing resources at three levels: strategic, portfolio and 
program levels. Implementation of any new business rules will be 
contingent upon the outcome of initiatives focused in this area, a 
thorough understanding of the best overall approach and the overall 
benefits gained. 

Recommendation 4: The GAO recommended that the Secretary of Defense 
require increasingly precise cost, schedule, and performance 
information for each alternative that meet specified levels of 
confidence and allowable deviations at each decision point leading up 
to the start of product development. (p. 26/GAO Draft Report): 

DOD Response: Concur. As noted in the report, in response to the 2006 
QDR and other recent acquisition reform studies, the DoD is undertaking 
several initiatives and pilot activities to improve the department's 
approach to investment and program decision making. Implementation of 
any new business rules will be contingent upon a thorough understanding 
of the best overall approach and the overall benefits gained. 

Recommendation 5: The GAO recommended that the Secretary of Defense 
establish portfolio managers who are empowered to prioritize needs, 
make early go/no-go decisions about alternative solutions, and allocate 
resources within fiscal constraints. (p. 26/GAO Draft Report): 

DOD Response: Partially Concur. As noted in the report, in response to 
the 2006 QDR and other recent acquisition reform studies, the DoD is 
undertaking several initiatives and pilot activities to experiment with 
Capability Portfolio Management. We have selected four highly joint 
areas with which to initiate a portfolio construct, and are 
experimenting with their participation and integration in the 
Department's major processes, as well as with their authorities and 
organizational constructs. Implementation of any new business rules 
will be informed by the experience gained and information gathered from 
the Capability Portfolio management effort and based on a thorough 
understanding of the best overall approach and the overall benefits 
gained. 

Recommendation 6: The GAO recommended that the Secretary of Defense 
hold officials at all levels accountable for achieving and maintaining 
a balanced, joint portfolio of weapon system investments that meet the 
needs of the warfighter within resource constraints. (p. 26/GAO Draft 
Report): 

DOD Response: Concur. In response to the 2006 QDR, the Department 
developed an execution roadmap for Institutional Reform and Governance. 
This roadmap is aimed at improving the defense enterprise by creating 
more integrated and responsive decision making processes, organizations 
and business practices. The roadmap seeks to develop a decision 
management approach that enables a clear and transparent link from 
strategy to outcomes. The approach will clearly delineate decision 
making responsibilities of the governance, management and execution 
levels of the Department. It will also enable senior leadership to 
focus on strategic choice and empower management to carry out their 
responsibilities in a manner that ensures transparency, accountability, 
and sound performance management. An example acquisition transformation 
initiative is the utilization of a senior-level Tri-Chaired Committee 
that integrates Department-wide acquisition processes to conduct 
Concept Decision Reviews. The Concept Decision Review goal is to 
ensure, as early as possible, that DoD is making the right corporate 
investment choices, balancing operational and programmatic risks, to 
ensure that those choices are affordable, and that any resulting non-
materiel solutions and/or materiel acquisitions are designed for 
lifecycle success. 

Recommendation 7: The GAO recommended that the DoD take steps to 
support department level decisions-makers and portfolio managers by 
developing a stronger joint analytical capability to assess and 
prioritize warfighting needs. (p. 26/GAO Draft Report): 

DOD Response: Concur. As part of the overall Institutional Reform and 
Governance effort described above, we will work to improve the 
Department's analytic framework, build more transparent business 
information across the Department, integrate decision processes to 
enable strategic choice and align roles and responsibilities in a way 
that maximizes decision making effectiveness across the enterprise. As 
an example, the Department is continuing to develop its weapon system 
readiness and sustainment modeling capabilities. Such models are aimed 
at enhancing the ability of acquisition and sustainment professionals 
to assess, for example, trade-offs between alternative investments in 
elements such as sustainment capabilities, support structures, cycle 
times, reliability, and alternative policies. The goal of these 
modeling capabilities is to ensure that DoD can make the intelligent, 
informed resource application decisions that will optimize acquisition 
and sustainment operations and maximize achieved materiel readiness at 
optimum total weapon systems ownership cost. Such modeling capabilities 
support the ongoing integration of acquisition and sustainment 
activities into an end-to-end continuum. 

[End of section] 

Related GAO Products: 

Best Practices: Stronger Practices Needed to Improve DOD Technology 
Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006. 

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost 
and Schedule Problems under DOD's Revised Policy. GAO-06-368. 
Washington, D.C.: April 13, 2006. 

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, 
D.C.: November 15, 2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000. 

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Programs 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 
16, 1999. 

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999. 

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 17, 1998. 

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996. 

FOOTNOTES 

[1] GAO, Defense Acquisitions: Actions Needed to Get Better Results on 
Weapons Systems Investments, GAO-06-585T (Washington, D.C.: Apr. 5, 
2006). 

[2] GAO, Defense Acquisitions: Major Weapon Systems Continue to 
Experience Cost and Schedule Problems under DOD's Revised Policy, GAO-
06-368 (Washington, D.C.: Apr. 13, 2006). 

[3] S.Rep.No. 109-69 at 343-346 (2005). 

[4] The Senate mandate specifically asked GAO to review the 
requirements and budgeting portions of DOD weapon system acquisition. 
Therefore, we do not focus on the DAS in this report. GAO has, however, 
produced a body of best practices work focusing on the DAS; many of 
those reports are listed in the related GAO products section of this 
report. 

[5] GAO, Defense Acquisitions: Assessments of Selected Major Weapon 
Programs, GAO-07-406SP (Washington, D.C.: Mar. 30, 2007). 

[6] GAO, Best Practices: Capturing Design and Manufacturing Knowledge 
Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 
15, 2002); Best Practices: Better Matching of Needs and Resources Will 
Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.: 
Mar. 8, 2001); and Best Practices: Better Management of Technology 
Development Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 
(Washington, D.C.: July 30, 1999). 

[7] IBM focuses on providing its customers with what it calls "industry 
integrated solutions." As a result, portfolios within IBM are 
structured around the customers' and buyers' behaviors and needs and 
not on specific product offerings. In other words, the company focuses 
on providing functional solutions to the customers, which could be done 
with a number of product offerings and services, and not simply 
limiting themselves to narrowly focused, single product solutions that 
the customer can take or leave, an approach that IBM believes caused 
problems for the company in the past. 

[8] According to Eli Lilly representatives, the average cost to 
discover and develop a new drug ranges from $800 million to $1 billion, 
and the average length of time to get a new drug from discovery to the 
market is 10 to 15 years. 

[9] Companies commonly measure rewards and benefits using cash flow 
analysis known as net present value (NPV) analysis. NPV techniques can 
show, in today's dollars, the relative net cash flow of various 
alternatives over a long period of time. Simply stated, net cash flow 
is the amount of dollars that is left after sales and revenues have 
offset expenses. In general, the greater the net cash flow for a 
particular investment, the greater the return on the investment. 

[10] GAO, Results-Oriented Cultures: Creating a Clear Linkage between 
Individual Performance and Organizational Success, GAO-03-488 
(Washington, D.C.: Mar. 14, 2003). 

[11] Chairman of the Joint Chiefs of Staff Instruction, Joint 
Capabilities Integration and Development System, CJCSI 3170.01E (May 
11, 2005). The original instruction was CJCSI 3170.01C (June 24, 2003). 

[12] The other capability areas are command and control, focused 
logistics, force management, force protection, and joint training. 

[13] Institute for Defense Analyses, Improving Integration of 
Department of Defense Processes for Capabilities Development and 
Planning (Sept. 2006); Center for Strategic and International Studies, 
Beyond Goldwater-Nichols: Defense Reform for a New Strategic Era, Phase 
1 Report (Mar. 2004); and Joint Defense Capabilities Study Team (DOD), 
Joint Defense Capabilities Study: Improving DoD Strategic Planning, 
Resourcing and Execution to Satisfy Joint Capabilities. Final Report 
(Jan. 2004). 

[14] Assessment Panel of the Defense Acquisition Performance Assessment 
Project for the Deputy Secretary of Defense, Defense Acquisition 
Performance Assessment Report (Jan. 2006); Center for Strategic and 
International Studies, Beyond Goldwater-Nichols: U.S. Government and 
Defense Reform for a New Strategic Era, Phase 2 Report (July 2005); and 
M. Thomas Davis, "The JROC: Doing What? Going Where?" National Security 
Studies Quarterly (Summer 1998). 

[15] Sections 3013, 5013, and 8013 of Title 10 grant authority to the 
Secretaries of the Army, the Navy, and the Air Force, respectively, to 
conduct all affairs of their departments, including recruiting, 
organizing, supplying, equipping, training, servicing, mobilizing, 
demobilizing, administering, maintaining, and military construction and 
maintenance. 

[16] GAO, Military Transformation: Actions Needed by DOD to More 
Clearly Identify Triad Spending and Develop a Long-term Investment 
Approach, GAO-05-540 (Washington, D.C.: June 30, 2005). 

[17] The Major Force Programs are as follows: Strategic Forces; General 
Purpose Forces; Command, Control, Communications, and Intelligence; 
Mobility Forces; Guard and Reserve Forces; Research and Development; 
Central Supply and Maintenance; Training, Medical, and other General 
Personnel Activities; Administration and Associated Activities; Support 
of Other Nations; and Special Operations Forces. 

[18] GAO, Future Years Defense Program: Actions Needed to Improve 
Transparency of DOD's Projected Resource Needs, GAO-04-514 (Washington, 
D.C.: May 7, 2004). 

[19] The fiscal framework of the federal government constrains DOD's 
ability to totally control its investment allocations. Ultimately 
resource allocation decisions are made by the Congress in the annual 
authorization and appropriations process. 

[20] The 2006 Quadrennial Defense Review led to the Institutional 
Reform and Governance project, which is focusing on (1) integrating 
core decision processes, (2) aligning and focusing the department's 
governance and management functions under an integrated enterprise 
model, as well as (3) establishing a common and authoritative 
analytical framework to link strategic decisions to execution. 

[21] Assessment Panel of the Defense Acquisition Performance Assessment 
Project for the Deputy Secretary of Defense, Defense Acquisition 
Performance Assessment Report (January 2006). 

[22] GAO, Department of Defense: Sustained Leadership Is Critical to 
Effective Financial and Business Transformation, GAO-06-1000T 
(Washington, D.C.: Aug. 3, 2006); and Department of Defense: Further 
Actions Are Needed to Effectively Address Business Management Problems 
and Overcome Key Business Transformation Challenges, GAO-05-140T 
(Washington, D.C.: Nov. 18, 2004). 

[23] GAO, Defense Management: Additional Actions Needed to Enhance 
DOD's Risk-Based Approach for Making Resource Decisions, GAO-06-13 
(Washington, D.C.: Nov. 15, 2005). 

[24] S. 780, 109th Cong. §1 (2005). 

[25] DAS is an event-driven process structured into discrete phases 
separated by major decision points called milestones that can be 
tailored to individual programs. 

[26] The 14 ICDs we reviewed are related to weapon systems and were 
finalized after June 24, 2003 (the publication date for the initial 
JCIDS instruction). They are unclassified ACAT I, II, or III ICDs 
contained in the Joint Staff's Knowledge Management Decision Support 
Tool database, which serves in part as a repository for JCIDS 
documents. 

[27] The white paper suggests roughly characterizing the 20-year- 
lifecycle costs of proposed solutions in terms of developmental costs, 
facility or infrastructure costs, per-unit and rough force-level 
acquisition costs, and recurring operating costs. It states that rough 
estimates of the technical feasibility of proposed solutions should be 
developed, not at the engineering level because of the broad range of 
possibilities, but at the least to characterize as no risk, very low 
risk, low risk, medium risk, and high risk using legitimate technology 
experts. "Whitepaper on Conducting a Capabilities-Based Assessment 
Under the Joint Capabilities Integration and Development System," Joint 
Chiefs of Staff J-8/Force Application Assessment Division (Jan. 2006). 

[28] Joint future concepts are visualizations of future operations that 
describe how a commander might employ warfighting capabilities to 
achieve effects and objectives. 

[29] The 2001 Quadrennial Defense Review directed DOD to implement a 
capabilities-based planning approach. Capabilities-based planning has 
been described as a framework for defense planning and decision making 
in a strategic environment characterized by uncertainty and a fiscal 
environment characterized by limited resources. See Paul K. Davis, 
Analytic Architectures for Capabilities-Based Planning, Mission System 
Analysis, and Transformation (Santa Monica, Calif.: RAND, 2002). 

[30] GAO, Force Structure: Joint Seabasing Would Benefit from a 
Comprehensive Management Approach and Rigorous Experimentation before 
Services Spend Billions on New Capabilities, GAO-07-211 (Washington, 
D.C.: Jan. 26, 2007). Joint seabasing is one of several evolving 
concepts describing how commanders in the future will project and 
sustain forces for conducting joint military operations without relying 
on immediate access to nearby land bases. 

[31] CJCSI 3170.01E , Joint Capabilities Integration and Development 
System; DODI 5000.2, Operation of the Defense Acquisition System. 

[32] The JROC has another decision point just prior to program 
initiation, when it reviews, validates, and approves a Capabilities 
Development Document. Approval and validation of a Capabilities 
Development Document is a key entrance criteria for initiating a new 
development program at Milestone B. We did not include the Capabilities 
Development Document decision point in our current review because it is 
closely associated with Milestone B, the focus of many of our former 
reviews. See the list of related GAO products on the last pages of this 
report. 

[33] GAO, Defense Acquisitions: Major Weapon Systems Continue to 
Experience Cost and Schedule Problems under DOD's Revised Policy, GAO-
06-368 (Washington, D.C.: Apr. 13, 2006). Our review focused on the 
Concept Decision and Milestone A reviews points and did not assess the 
extent to which JROC reviews were held. 

[34] Defense Acquisition Performance Assessment (January 2006); Defense 
Science Board Summer Study on Transformation: A Progress Assessment 
(February 2006); Beyond Goldwater-Nichols: U.S. Government and Defense 
Reform for a New Strategic Era, Phase 2 Report (July 2005); and Joint 
Defense Capabilities Study (January 2004). 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts 
newly released reports, testimony, and correspondence on its Web site. 
To have GAO e-mail you a list of newly posted products every afternoon, 
go to www.gao.gov and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 441 G Street NW, Room LM 
Washington, D.C. 20548: 

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 
512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S. 
Government Accountability Office, 441 G Street NW, Room 7125 
Washington, D.C. 20548: 

Public Affairs: 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, D.C. 20548: