This is the accessible text file for GAO report number GAO-06-780 
entitled 'Particulate Matter: EPA Has Started to Address the National 
Academies' Recommendations on Estimating Health Benefits, but More 
Progress Is Needed' which was released on July 19, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

July 2006: 

Particulate Matter: 

EPA Has Started to Address the National Academies' Recommendations on 
Estimating Health Benefits, but More Progress Is Needed: 

GAO-06-780: 

GAO Highlights: 

Highlights of GAO-06-780, a report to congressional requesters 

Why GAO Did This Study: 

A large body of scientific evidence links exposure to particulate 
matter—a widespread form of air pollution—to serious health problems, 
including asthma and premature death. Under the Clean Air Act, the 
Environmental Protection Agency (EPA) periodically reviews the 
appropriate air quality level at which to set national standards to 
protect the public against the health effects of particulate matter. 
EPA proposed revisions to these standards in January 2006 and issued a 
draft regulatory impact analysis of the revisions’ expected costs and 
benefits. 

The estimated benefits of air pollution regulations have been 
controversial in the past. A 2002 National Academies report generally 
supported EPA’s approach but made 34 recommendations to improve how EPA 
implements its approach. GAO was asked to determine whether and how EPA 
applied the Academies’ recommendations in its estimates of the health 
benefits expected from the January 2006 proposed revisions to the 
particulate matter standards. GAO examined the draft analysis, met with 
EPA officials, and interviewed members of the National Academies’ 
committee. In providing technical comments on the report, EPA officials 
said it was fair and balanced and noted the agency’s progress in 
addressing recommendations via research and development and other 
analyses. 

What GAO Found: 

EPA has begun to change the way it conducts and presents its analyses 
of health benefits in response to recommendations from the National 
Academies. Specifically, EPA applied, at least in part, 22—or about two-
thirds—of the Academies’ recommendations to its health benefit analysis 
of proposed revisions to particulate matter standards. For example, in 
response to some of the recommendations, EPA took steps toward 
conducting a more rigorous assessment of uncertainty by, for instance, 
evaluating how benefits could change under different assumptions and 
discussing sources of uncertainty not included in the benefit 
estimates. In one case, EPA applied an alternative technique, called 
expert elicitation, for evaluating uncertainty by systematically 
gathering expert opinion about the uncertainty underlying the causal 
link between exposure to particulate matter and premature death. 
Consistent with the National Academies’ recommendation to assess 
uncertainty by developing ranges of estimates and specifying the 
likelihood of attaining them, EPA used expert elicitation to develop 
ranges of reductions in premature death expected from the proposed 
revisions. EPA officials said that ongoing research and development 
efforts will allow the agency to gradually achieve more progress in 
applying the recommendations. We note that robust uncertainty analysis 
is important because estimates of health benefits can be highly 
uncertain, as the draft regulatory impact analysis for particulate 
matter illustrates. EPA viewed the estimates in this analysis as so 
uncertain that it chose not to present them in the executive summary. 

For various reasons, EPA has not applied the remaining 12 
recommendations to the analysis, such as the recommendation to evaluate 
the impact of using the simplifying assumption that each component of 
particulate matter is equally toxic. EPA officials viewed most of these 
recommendations as relevant to its health benefit analyses and, citing 
the need for additional research and development, emphasized the 
agency’s commitment to continue to respond to the recommendations. For 
example, EPA did not believe that the state of scientific knowledge on 
the relative toxicity of particulate matter components was sufficiently 
developed to include in the January 2006 regulatory impact analysis, 
and the agency is currently sponsoring research on this issue. In 
addition, a senior EPA official said that insufficient resources 
impeded the agency’s progress in applying the recommendations, citing, 
in particular, the limited availability of skilled staff, time, and 
other resources to conduct the required analyses and research and 
development. EPA officials also said that some of the recommendations 
the agency did not apply to the draft analysis, such as one calling for 
a summary table describing key analytical information to enhance 
transparency, will be applied to the analysis supporting the final 
rule. To the extent that EPA continues to make progress addressing the 
Academies’ recommendations, decision makers and the public will be 
better able to evaluate the basis for EPA’s air regulations. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-780]. 

To view the full product, including the scope and methodology, click on 
the link above.
For more information, contact John B. Stephenson at (202) 512-3841 or 
stephensonj@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

EPA Is in the Process of Addressing Many of the Academies' 
Recommendations: 

Concluding Observations: 

Agency Comments: 

Appendixes: 

Appendix I: Scope and Methodology: 

Appendix II: Recommendations Applied or Partially Applied to the Draft 
Particulate Matter Regulatory Impact Analysis: 

Appendix III: Recommendations Not Applied to the Draft Particulate 
Matter Regulatory Impact Analysis: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables Tables: 

Table 1: Recommendations Applied or Partially Applied to the Draft 
Particulate Matter Regulatory Impact Analysis: 

Table 2: Recommendations Not Applied to the Draft Particulate Matter 
Regulatory Impact Analysis: 

Abbreviations: 

Academies: National Academies: 

ACS: American Cancer Society: 

Council : Advisory Council on Clean Air Compliance: 

EPA: Environmental Protection Agency: 

NAAQS: National Ambient Air Quality Standards: 

OMB: Office of Management and Budget: 

PM: particulate matter: 

RIA: regulatory impact analysis: 

SAB: Science Advisory Board: 

July 14, 2006: 

The Honorable James M. Inhofe: 
Chairman: 
Committee on Environment and Public Works: 
United States Senate: 

The Honorable George Voinovich: 
Chairman: 
Subcommittee on Clean Air, Climate Change, and Nuclear Safety: 
Committee on Environment and Public Works: 
United States Senate: 

A large body of scientific evidence links exposure to particulate 
matter--a ubiquitous form of air pollution commonly referred to as 
soot--to serious health problems, including asthma, chronic bronchitis, 
heart attack, and premature death. The many sources releasing 
particulate matter into the air include cars, trucks, power plants, 
industrial processes, forest fires, and waste incinerators. In 1971, 
the Environmental Protection Agency (EPA) first established national 
air quality standards to protect the public against the health effects 
of particulate matter, one of the six widespread criteria pollutants 
considered harmful to public health. Under the Clean Air Act, EPA 
determines the appropriate level at which to set national air quality 
standards, and the states must develop programs to achieve and maintain 
compliance with them. Further, EPA must review the standards every 5 
years to determine whether they adequately protect human health and 
welfare, given the latest scientific information available, and revise 
them if they do not. 

In January 2006, after its most recent review of the national air 
quality particulate matter standards, EPA proposed revisions to the 
standards and issued a draft regulatory impact analysis. Overall, the 
draft regulatory impact analysis discussed the scope and magnitude of 
the particulate matter problem, the likely benefits of the proposed 
revisions for public health and the environment, and the expected costs 
of implementing the standards. Regarding public health, the analysis 
presented estimates of expected health benefits for the particulate 
matter revisions in five major urban areas, including reductions in the 
number of premature deaths and emergency room visits for asthma. Among 
the changes EPA said it plans to make in its final regulatory impact 
analysis is providing national estimates of expected health benefits. 
EPA is required under a court order to issue the final rule on 
particulate matter standards by September 2006.[Footnote 1] 

In 2000, at the direction of the Senate Appropriations Committee, EPA 
asked the National Academies (Academies) to evaluate EPA's methodology 
for estimating the health benefits of proposed air pollution 
regulations.[Footnote 2] According to the National Academies, these 
estimates have often been controversial, and the methods EPA has used 
to prepare them have been questioned. For example, some observers, such 
as researchers and industry groups, have expressed concerns that EPA 
does not adequately factor uncertainty into its estimates of health 
benefits. Some level of uncertainty is unavoidable, in part because the 
scientific information used to develop estimates, such as the inventory 
of particulate matter emissions, will never be perfect or complete. 
However, according to some observers, EPA's estimates of benefits 
appear more definitive than they really are because the agency does not 
adequately account for uncertainty in its analyses or in its reporting 
of health benefit estimates. 

The National Academies' 2002 report on this subject generally supported 
EPA's approach to estimating health benefits but, nevertheless, made 34 
detailed recommendations to improve how EPA implements its 
approach.[Footnote 3] Overall, these recommendations focus on 
conducting more rigorous assessments of uncertainty, increasing the 
transparency of how EPA estimates benefits, conducting more detailed 
analyses of exposure, and estimating the benefits of each regulatory 
option under consideration. Many of the recommendations include 
qualifying language indicating that it is reasonable to expect that 
they can be applied in stages, over time; and a number of the 
recommendations are interrelated and, in some cases, overlapping. 

You asked us to determine whether and how EPA applied the National 
Academies' recommendations in its estimates of the health benefits 
expected from the January 2006 proposed revisions to the particulate 
matter standards. To respond to this objective, we reviewed EPA's draft 
regulatory impact analysis presenting the costs and benefits of the 
proposed rule and met with senior officials from EPA's Office of Air 
and Radiation, which was responsible for developing the proposed rule 
and analyzing its costs and benefits, and with officials from EPA's 
Office of Policy, Economics, and Innovation. We also reviewed EPA's and 
the Office of Management and Budget's (OMB) guidance on conducting 
economic analyses, prior GAO reports on EPA's regulatory impact 
analyses, and other relevant reports. As requested, our work addressed 
the application of the National Academies' recommendations to EPA's 
draft regulatory impact analysis supporting the 2006 proposed 
particulate matter rule; thus, we did not examine how EPA applied the 
recommendations to other recent air rules. Our work focused on broadly 
characterizing EPA's progress toward applying the recommendations; we 
did not evaluate the effectiveness and quality of the scientific and 
technical actions the agency has taken to apply the recommendations. 
See appendix I for a more detailed description of the scope and 
methodology of our review. We performed our work from January 2006 to 
July 2006 in accordance with generally accepted government auditing 
standards. 

Results in Brief: 

EPA has begun to change the way it conducts and presents its analyses 
of health benefits in response to the National Academies' 
recommendations, which focused on conducting more rigorous assessments 
of uncertainty, increasing the transparency of how EPA estimates 
benefits, conducting more detailed analyses of exposure, and estimating 
the benefits of each regulatory option under consideration. EPA 
applied, at least in part, about two-thirds of the recommendations to 
its health benefit analysis. Specifically, of the 34 recommendations, 
EPA applied 8 and partially applied 14. For example, EPA responded to 
some of the recommendations by taking steps to conduct a more rigorous 
assessment of uncertainty by, for instance, evaluating how benefits 
might change given alternative assumptions and discussing sources of 
uncertainty not included in the benefit estimates. More specifically, 
EPA applied an alternative technique for evaluating one important 
source of uncertainty in its analysis--the uncertainty underlying the 
causal link between exposure to particulate matter and premature death. 
Consistent with the National Academies' recommendation to assess 
uncertainty by developing ranges of estimates of benefits and 
specifying the likelihood of attaining that level of benefits, EPA 
systematically gathered expert opinions about this link--through a 
process called expert elicitation--and developed ranges reflecting the 
experts' confidence in attaining reductions in premature death expected 
from the proposed revisions. However, the health benefit analysis does 
not similarly assess how the benefit estimates would vary in light of 
other key uncertainties as the Academies had recommended. Consequently, 
this represents a partial application of the recommendation. Agency 
officials told us that ongoing research and development efforts will 
allow EPA to gradually achieve more progress in applying this and other 
recommendations to future analyses. 

For various reasons, EPA did not apply the remaining 12 recommendations 
to the analysis, such as the recommendation to evaluate the impact of 
using the assumption that the components of particulate matter are 
equally toxic. EPA officials viewed most of these recommendations as 
relevant to its health benefit analyses, but noted that the agency was 
not ready to apply specific recommendations because of, among other 
things, the need to overcome technical challenges stemming from 
limitations in the state of available science. These officials 
emphasized the agency's commitment to continue to respond to the 
recommendations. For example, EPA did not believe that the state of 
scientific knowledge on the relative toxicity of particulate matter 
components was sufficiently developed to include it in the January 2006 
regulatory impact analysis, and the agency is sponsoring research on 
this issue. In addition, according to a senior EPA official, 
insufficient resources have impeded the agency's progress in applying 
the recommendations, including the limited availability of skilled 
staff, time, and other resources to conduct and oversee the required 
analyses and research and development. Finally, EPA officials stated 
that some of the recommendations the agency did not apply to the draft 
analysis, such as one calling for a summary table describing key 
analytical information to enhance transparency, will be applied to the 
regulatory impact analysis supporting the final rule. To the extent 
that EPA continues to make progress addressing the Academies' 
recommendations, decision makers and the public will be able to better 
evaluate the basis for EPA's air regulations. 

We provided a draft of this report to EPA for review. EPA provided 
technical comments that we incorporated, as appropriate. Officials from 
EPA's Office of Air and Radiation noted in their technical comments 
that the report provides a fair and balanced representation of the 
agency's application of the recommendations to the particulate matter 
regulatory impact analysis and cited EPA's progress in meeting the 
National Academies' recommendations through other analyses of air 
programs and through research and development efforts. 

Background: 

EPA is required by the Clean Air Act to conduct reviews of the National 
Ambient Air Quality Standards (NAAQS) for the six criteria pollutants, 
including particulate matter, every 5 years. The overarching purpose of 
such reviews is to determine whether the current standards are 
sufficient to protect public health and welfare at large, with an 
adequate margin of safety, given the latest scientific information 
available at the time of the review. Major steps in the NAAQS process 
include the following: 

* developing a criteria document that synthesizes new research on 
health and environmental effects; 

* preparing a staff paper that assesses the policy implications of the 
scientific information in the criteria document, which also discusses 
possible ranges for air quality standards; and: 

* determining whether and how EPA should revise the NAAQS. 

If EPA decides to revise the NAAQS, the agency proposes the changes in 
the Federal Register. As part of the federal rule-making process, EPA 
is to comply with Executive Order 12866, which directs federal agencies 
to analyze the costs and benefits of proposed and final rules expected 
to affect the economy by $100 million or more per year.[Footnote 4] In 
September 2003, the Office of Management and Budget (OMB) issued its 
Circular A-4, which presents guidance and best practices and states 
that agencies should analyze the costs and benefits in accordance with 
the principles of full disclosure and transparency. Further, in cases 
such as the particulate matter rule, where expected economic impacts 
exceed $1 billion annually, Circular A-4 also states that agencies 
should conduct a comprehensive assessment of key uncertainties in their 
analyses of costs and benefits, which EPA also refers to as regulatory 
impact analyses.[Footnote 5] EPA's January 2006 regulatory impact 
analysis presents estimates of the costs and benefits for the proposed 
particulate matter rule. 

The focus of the National Academies' 2002 report was on how EPA 
estimates the health benefits of its proposed air regulations. To 
develop such estimates, EPA conducts analyses to quantify the expected 
changes in the number of deaths and illnesses that are likely to result 
from proposed regulations. The regulatory impact analyses also estimate 
the costs associated with implementing proposed air regulations, 
although, under the Clean Air Act, EPA is not permitted to consider 
costs in setting health-based standards for the criteria air 
pollutants, such as particulate matter. 

Soon after the National Academies issued its report in 2002, EPA staff 
identified key recommendations and developed a strategy, in 
consultation with OMB, to apply some of the recommendations to benefit 
analyses for air pollution regulations under consideration at the time. 
EPA roughly approximated the time and resource requirements to respond 
to the recommendations, identifying those the agency could address 
within 2 or 3 years and those that would take longer. According to EPA 
officials, the agency focused primarily on the numerous recommendations 
related to analyzing uncertainty. 

Both the National Academies' report and the OMB guidance emphasize the 
need for agencies to account for uncertainties and to maintain 
transparency in the course of conducting benefit analyses. Identifying 
and accounting for uncertainties in these analyses can help decision 
makers evaluate the likelihood that certain regulatory decisions will 
achieve the estimated benefits. Transparency is important because it 
enables the public and relevant decision makers to see clearly how EPA 
arrived at its estimates and conclusions. In prior work on regulatory 
impact analyses, we have found shortcomings in EPA's analyses of 
uncertainty and the information the agency provides with its estimates 
of costs and benefits.[Footnote 6] 

EPA Is in the Process of Addressing Many of the Academies' 
Recommendations: 

EPA applied--either wholly or in part--approximately two-thirds of the 
Academies' recommendations to its January 2006 regulatory impact 
analysis and continues to address the recommendations through ongoing 
research and development. The January 2006 regulatory impact analysis 
demonstrated progress toward an expanded analysis of uncertainty and 
consideration of different assumptions. EPA officials cited time and 
resource constraints, as well as the need to mitigate complex technical 
challenges, as the basis for not applying other recommendations. 
According to EPA officials, the agency did not apply some of the more 
complex recommendations because it had not achieved sufficient progress 
in the research and development projects under way. 

EPA Applied, at Least in Part, about Two-thirds of the Recommendations 
to Its Particulate Matter Health Benefit Analysis in the Proposed Rule: 

The January 2006 regulatory impact analysis on particulate matter 
represents a snapshot of an ongoing EPA effort to respond to the 
National Academies' recommendations on developing estimates of health 
benefits for air pollution regulations. Specifically, the agency 
applied, at least in part, approximately two-thirds of the 
recommendations--8 were applied and 14 were partially applied--by 
taking steps toward conducting a more rigorous assessment of 
uncertainty for proposed air pollution regulations by, for example, 
evaluating the different assumptions about the link between human 
exposure to particulate matter and health effects and discussing 
sources of uncertainty not included in the benefit estimates. According 
to EPA officials, the agency focused much of its time and resources on 
the recommendations related to uncertainty. In particular, one 
overarching recommendation suggests that EPA take steps toward 
conducting a formal, comprehensive uncertainty analysis--the systematic 
application of mathematical techniques, such as Monte Carlo simulation--
and include the uncertainty analysis in the regulatory impact analysis 
to provide a "more realistic depiction of the overall uncertainty" in 
EPA's estimates of the benefits.[Footnote 7] A number of the other 
recommendations regarding uncertainty are aimed at EPA's developing the 
information and methodologies needed to carry out a comprehensive 
uncertainty analysis. 

Overall, the uncertainty recommendations suggest that EPA should 
determine (1) which sources of uncertainties have the greatest effect 
on benefit estimates and (2) the degree to which the uncertainties 
affect the estimates by specifying a range of estimates and the 
likelihood of attaining them. In response, EPA devoted significant 
resources to applying an alternative technique called expert 
elicitation in a multiphased pilot project. The pilot project was 
designed to systematically obtain expert advice to begin to better 
incorporate in its health benefit analysis the uncertainty underlying 
the causal link between exposure to particulate matter and premature 
death. EPA used the expert elicitation process to help it more 
definitively evaluate the uncertainty associated with estimated 
reductions in premature death--estimates that composed 85 percent to 95 
percent of EPA's total health benefit estimates for air pollution 
regulations in the past 5 years, according to the agency.[Footnote 8] 
EPA developed a range of expected reductions in death rates based on 
expert opinion systematically gathered in its pilot expert elicitation 
project and provided the results of this supplemental analysis in an 
appendix to the regulatory impact analysis.[Footnote 9] However, the 
National Academies had recommended that EPA merge such supplemental 
analyses into the main benefit analysis. 

Moreover, the Academies recommended that EPA's main benefit analysis 
reflect how the benefit estimates would vary in light of uncertainties. 
In addition to the uncertainty underlying the causal link between 
exposure and premature death that EPA analyzed, other key uncertainties 
can influence the estimates. For example, there is uncertainty about 
the effects of the age and health status of people exposed to 
particulate matter, the varying composition of particulate matter, and 
the measurements of actual exposure to particulate matter. EPA's health 
benefit analysis, however, does not account for these key uncertainties 
by specifying a range of estimates and the likelihood of attaining 
them, similar to estimates derived from the expert elicitation 
addressing causal uncertainty. For these reasons, EPA's responses 
reflect a partial application of the Academies' recommendation. 

In addition, the Academies recommended that EPA both continue to 
conduct sensitivity analyses on sources of uncertainty and expand these 
analyses. In the particulate matter regulatory impact analysis, EPA 
included a new sensitivity analysis regarding assumptions about 
thresholds, or levels below which those exposed to particulate matter 
are not at risk of experiencing harmful effects. EPA has assumed no 
threshold level exists--that is, any exposure poses potential health 
risks.[Footnote 10] Some experts have suggested that different 
thresholds may exist and the National Academies recommended that EPA 
determine how changing its assumption--that no threshold exists--would 
influence the estimates. The sensitivity analysis EPA provided in the 
regulatory impact analysis examined how its estimates of expected 
health benefits would change assuming varying thresholds. 

Another recommendation that EPA is researching and partially applied to 
the draft regulatory impact analysis concerns alternative assumptions 
about cessation lags--the time between reductions in exposure to 
particulate matter and the health response. The National Academies made 
several recommendations on this topic, including one that EPA 
incorporate alternative assumptions about lags into a formal 
uncertainty analysis to estimate benefits that account for the 
likelihood of different lag durations. In response, EPA has sought 
advice from its Advisory Council on Clean Air Compliance Analysis on 
how to address this recommendation and has conducted a series of 
sensitivity analyses related to cessation lags. EPA is also funding 
research to explore ways to address lag effects in its uncertainty 
analysis. According to an EPA official, specifying the probability of 
different lag effects is computationally complex, and the agency is 
working to resolve this challenge. 

In response to another recommendation by the National Academies, EPA 
identified some of the sources of uncertainty that are not reflected in 
its benefit estimates. For example, EPA's regulatory impact analysis 
disclosed that its benefit estimates do not reflect the uncertainty 
associated with future year projections of particulate matter 
emissions. EPA presented a qualitative description about emissions 
uncertainty, elaborating on technical reasons--such as the limited 
information about the effectiveness of particulate matter control 
programs--why the analysis likely underestimates future emissions 
levels. EPA also applied the Academies' recommendation on the 
presentation of uncertainty, which encouraged the agency to present the 
results of its health benefit analyses in ways that convey the 
estimated benefits more realistically by, for example, placing less 
emphasis on single estimates and rounding the numbers. EPA's regulatory 
impact analysis presented ranges for some of the benefit estimates. 
Also, EPA sought to convey the overall uncertainty of its benefit 
estimates in a qualitative manner by clearly stating that decision 
makers and the public should not place significant weight on the 
quantified benefit estimates in the regulatory impact analysis because 
of data limitations and uncertainties. 

Another example of EPA's response to the National Academies' 
recommendations involves exploring the various regulatory choices 
available to decision makers. The Academies recommended that EPA 
estimate the health benefits representing the full range of regulatory 
choices available to decision makers. In the particulate matter 
analysis, EPA presented health benefits expected under several 
regulatory options targeting fine particulate matter. Citing a lack of 
data and tools needed to conduct an accurate analysis, EPA did not 
estimate the benefits expected under the proposed regulatory options 
for coarse particulate matter but, consistent with the National 
Academies' recommendation, presented its rationale for not doing so. 
Overall, we considered this a partial application of the 
recommendation. (See app. II for more detail on the recommendations 
that EPA has applied or partially applied to the draft particulate 
matter regulatory impact analysis.) 

EPA Plans to Address Some of the Remaining Recommendations in the Final 
Rule and Has Research and Development Under Way to Address Others: 

EPA did not apply the remaining 12 recommendations to the analysis for 
various reasons. While EPA applied some recommendations--either wholly 
or in part--that require additional studies, methodologies, or data to 
its particulate matter analysis, the agency had not made sufficient 
progress in addressing others and therefore did not apply them to the 
analysis. EPA officials viewed most of these recommendations as 
relevant to its health benefit analyses and, citing the need for 
additional research and development, emphasized the agency's commitment 
to continue to respond to the recommendations. According to a senior 
EPA official, insufficient resources impeded the agency's progress in 
applying the recommendations. This official cited limited availability 
of skilled staff, time, and other resources to conduct the required 
analyses and research and development. According to EPA, some of the 
more complex, long-term recommendations include the following: relying 
less on simplifying assumptions, such as the assumption that the 
various components of particulate matter have equal toxicity;[Footnote 
11] conducting a formal assessment of the uncertainty of particulate 
matter emissions; and assessing the expected reduction of any harmful 
effects other than air pollution or human health problems. 

For example, EPA is in the process of responding to a recommendation 
involving the relative toxicity of components of particulate matter, an 
emerging area of research that has the potential to influence EPA's 
regulatory decisions in the future.[Footnote 12] Specifically, the 
agency could, hypothetically, refine national air quality standards to 
address the potentially varying health consequences associated with 
different components of particulate matter. The National Academies 
recommended that EPA strengthen its benefit analyses by evaluating a 
range of alternative assumptions regarding relative toxicity and 
incorporate these assumptions into sensitivity or uncertainty analyses 
as more data become available.[Footnote 13] EPA did not believe the 
state of scientific knowledge on relative toxicity was sufficiently 
developed at the time it prepared the draft regulatory impact analysis 
to include this kind of analysis.[Footnote 14] However, EPA is 
sponsoring research on this issue. For example, EPA is supporting long-
term research on the relative toxicity of particulate matter components 
being conducted by EPA's intramural research program, its five 
Particulate Matter Research Centers, and the Health Effects Institute, 
an organization funded in part by EPA. In addition, an EPA contractor 
has begun to investigate methods for conducting a formal analysis that 
would consider sources of uncertainty, including relative toxicity and 
lag effects. To date, the contractor has created a model to assess 
whether and how much these sources of uncertainty may affect benefit 
estimates in one urban area. 

The National Academies also recommended that EPA incorporate an 
assessment of uncertainty into the early stages of its benefit analyses 
by characterizing the uncertainty of its emissions estimates on which 
the agency is going to base its benefit estimates.[Footnote 15] While 
the agency is investigating ways to assess or characterize this 
uncertainty, EPA did not conduct a formal uncertainty analysis for 
particulate matter emissions for the draft regulatory impact analysis 
because of data limitations. These limitations stem largely from the 
source of emissions data, the National Emissions Inventory,[Footnote 
16] an amalgamation of data from a variety of entities, including state 
and local air agencies, tribes, and industry. According to EPA, these 
entities use different methods to collect data, which have different 
implications for how to characterize the uncertainty. Furthermore, the 
uncertainty associated with emissions varies by the source of 
emissions. For example, the analytical methods for evaluating the 
uncertainty of estimates of emissions from utilities would differ from 
those for car and truck emissions because the nature of these emissions 
and the data collection methods differ. In sum, to apply this 
recommendation, EPA must determine how to characterize the uncertainty 
of the estimates for each source of emissions before aggregating the 
uncertainty to a national level and then factoring that aggregation 
into its benefit estimates. According to EPA officials, the agency 
needs much more time to resolve the complex technical challenges of 
such an analysis. EPA officials also noted that the final particulate 
matter analysis will demonstrate steps toward this recommendation by 
presenting emissions data according to the level emitted by the 
different kinds of sources, such as utilities, cars, and trucks. 

Another recommendation that EPA is researching but did not apply to the 
draft regulatory impact analysis concerns whether the proposed 
revisions to the particulate matter standards would have important 
indirect impacts on human health and the environment. According to an 
EPA official, the agency could not rule out the possibility that the 
revisions could have indirect impacts on the environment, such as 
whether reductions to particulate matter emissions would reduce the 
amount of particulate matter deposited in water bodies, thereby 
decreasing water pollution. EPA has considered indirect impacts of air 
pollution regulations on sensitive water bodies in the past and plans 
to include a similar analysis in the final particulate matter rule. An 
agency official further noted that ongoing research about environmental 
impacts could reveal additional indirect impacts for future analyses. 

Other recommendations that EPA did not apply to its benefit estimates 
in the regulatory impact analysis concern issues such as transparency 
and external review of EPA's benefit estimation process. For example, 
the National Academies recommended that EPA clearly summarize the key 
elements of the benefit analysis in an executive summary that includes 
a table that lists and briefly describes the regulatory options for 
which EPA estimated the benefits, the assumptions that had a 
substantial impact on the benefit estimates, and the health benefits 
evaluated. EPA did not, however, present a summary table as called for 
by the recommendation or summarize the benefits in the executive 
summary. As EPA stated in the particulate matter analysis, the agency 
decided not to present the benefit estimates in the executive summary 
because they were too uncertain. Specifically, officials said the 
agency was not able to resolve some significant data limitations before 
issuing the draft regulatory impact analysis in January 2006--a 
deadline driven by the need to meet the court-ordered issue date for 
the final rule in September 2006. According to EPA officials, EPA has 
resolved some of these data challenges by, for example, obtaining more 
robust data on anticipated strategies for reducing emissions, which 
will affect the estimates of benefits. The officials also said that EPA 
intends to include in the executive summary of the regulatory impact 
analysis supporting the final rule a summary table that describes key 
analytical information. EPA officials also acknowledged other 
presentation shortcomings, including references to key analytical 
elements that were insufficiently specific, that officials attributed 
to tight time frames and the demands of working on other regulatory 
analyses concurrently. They said they plan to address these 
shortcomings in the final regulatory impact analysis. 

Regarding external review, the National Academies recommended that EPA 
establish an independent review panel, supported by permanent technical 
staff, to bolster EPA's quality control measures for its regulatory 
impact analyses, such as the one for particulate matter. The National 
Academies noted that peer review of EPA's regulatory impact analyses 
would be advantageous when the agency designs and conducts its economic 
analysis.[Footnote 17] EPA has not directly addressed this 
recommendation. According to the Director of the Office of Policy 
Analysis and Review in EPA's Office of Air and Radiation, establishing 
and supporting independent committees is costly, making it important 
for EPA to take advantage of existing panels rather than set up new 
ones. Further, an official in the Office of Air and Radiation who 
oversees the development of regulatory impact analyses said that the 
cost of reviewing all regulatory impact analyses would be substantial. 
In this regard, EPA officials identified peer reviews the agency 
received from its existing independent committees, such as the Clean 
Air Scientific Advisory Committee and the Advisory Council on Clean Air 
Compliance.[Footnote 18] For example, to respond to the Academies' 
recommendations about lag effects, EPA sought independent advice on the 
assumptions it was developing regarding the time between reduced 
exposure to particulate matter and reductions in incidences of health 
effects.[Footnote 19] Finally, EPA officials noted that although the 
agency does not have each regulatory impact analysis peer reviewed, EPA 
typically does have the methodologies that will be applied to 
regulatory impact analyses peer reviewed. (See app. III for more detail 
on these recommendations and others that EPA did not apply to the draft 
particulate matter regulatory impact analysis.) 

Concluding Observations: 

While EPA has taken a number of steps to respond to the Academies' 
recommendations on estimating health benefits, continued commitment and 
dedication of resources will be needed if EPA is to fully implement the 
improvements endorsed by the National Academies. In particular, the 
agency will need to ensure that it allocates resources to needed 
research on emerging issues, such as the relative toxicity of 
particulate matter components; assessing which sources of uncertainty 
have the greatest influence on benefit estimates; and estimating other 
benefits, such as environmental improvements. In addition, it is 
important for EPA to continue to improve its uncertainty analysis in 
accordance with the Academies' recommendations. The agency's draft 
regulatory impact analysis illustrates that estimates of health 
benefits can be highly uncertain. In fact, EPA officials viewed these 
estimates as so uncertain that they chose to not present them in the 
executive summary of the regulatory impact analysis. While EPA 
officials said they expect to reduce the uncertainties associated with 
the health benefit estimates in the final particulate matter analysis, 
robust uncertainty analysis will nonetheless be important for decision 
makers and the public to understand the likelihood of attaining the 
estimated health benefits. According to EPA officials, the final 
regulatory impact analysis on particulate matter will reflect further 
responsiveness to the Academies' recommendations by, for example, 
providing additional sensitivity analysis and improving the 
transparency of the regulatory impact analysis by highlighting key data 
and assumptions in the executive summary. Moreover, these officials 
emphasized the agency's commitment to further enhancing the 
transparency of the analysis by presenting clear and accurate 
references to the supporting technical documents, which detail the 
analytical assumptions and describe the data supporting the estimates. 
To the extent EPA continues to make progress addressing the Academies' 
recommendations, decision makers and the public will be able to better 
evaluate the basis for EPA's air regulations. 

Agency Comments: 

We provided a draft of this report to EPA for review. EPA provided 
technical comments that we incorporated, as appropriate. Officials from 
the Office of Policy Analysis and Review within EPA's Office of Air and 
Radiation noted in their technical comments that the report provides a 
fair and balanced representation of EPA's efforts to apply the National 
Academies' recommendations to the draft particulate matter regulatory 
impact analysis. However, these officials also cited progress made in 
applying the National Academies' recommendations through analyses of 
other air programs and through research and development efforts. We 
note that this report does identify, as appropriate, EPA's research and 
development efforts for recommendations EPA did not apply to the draft 
particulate matter analysis, its plans to apply some additional 
recommendations to the final particulate matter regulatory impact 
analysis, and the agency's responses to recommendations in prior rule- 
making analyses of air programs. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the EPA 
Administrator and other interested parties. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at [Hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-3841 or s [Hyperlink, stephensonj@gao.gov] 
tephensonj@gao.gov. Contact points for our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. Key contributors to this report are listed in appendix IV. 

Signed by: 

John B. Stephenson: 
Director, Natural Resources and Environment: 

[End of section] 

Appendix I: Scope and Methodology: 

We were asked to determine whether and how the Environmental Protection 
Agency (EPA) applied the National Academies (Academies) recommendations 
in its estimates of the health benefits expected from the January 2006 
proposed revisions to the particulate matter national ambient air 
quality standards. In response to this objective, we assessed EPA's 
response to the Academies' recommendations and present an overview of 
the agency's completed, ongoing, and planned actions addressing the 
recommendations. To develop this overview, we reviewed EPA's 
particulate matter regulatory impact analysis, EPA's economic analysis 
guidelines, and Office of Management and Budget (OMB) guidance on 
regulatory impact analysis. We also analyzed documentation addressing 
current and future agency efforts to address the recommendations, such 
as project planning memorandums and technical support documents 
discussing the application of economic techniques. In addition, we met 
with senior officials from EPA's Office of Air and Radiation, which was 
responsible for developing the proposed rule and analyzing its economic 
effects, and with officials from EPA's Office of Policy, Economics, and 
Innovation to discuss the agency's responses to the recommendations. We 
interviewed several experts outside EPA, including (1) the Chair and 
other members of the National Academies' Committee on Estimating the 
Health-Risk-Reduction Benefits of Proposed Air Pollution Regulations, 
to clarify the basis for their recommendations; and (2) economists at 
Resources for the Future, to discuss the technical issues underlying 
the recommendations on uncertainty analysis. 

While the 2002 National Academies' report is generally applicable to 
EPA air pollution regulations, our review focused on the application of 
the recommendations to the proposed revisions to the particulate matter 
standards, as requested. Our work focused on broadly characterizing 
EPA's progress toward applying the recommendations; we did not evaluate 
the effectiveness or quality of the scientific and technical actions 
the agency has taken to apply them. To assess whether and how EPA has 
made progress in responding to the recommendations, we developed the 
following recommendation classification continuum: applied, partially 
applied, and not applied. The applied and partially applied categories 
refer to completed and initiated actions in EPA's health benefit 
analysis of particulate matter that corresponds to components of the 
National Academies' recommendations. The not applied category includes 
recommendations that EPA did not apply when conducting the analysis for 
the January 2006 particulate matter regulatory impact analysis and 
identifies those for which ongoing research and development efforts 
were not far enough along to apply to the particulate matter analysis. 
We performed our work from January 2006 to July 2006 in accordance with 
generally accepted government auditing standards. 

[End of section] 

Appendix II: Recommendations Applied or Partially Applied to the Draft 
Particulate Matter Regulatory Impact Analysis: 

Table 1 provides a summary of the National Academies' recommendations 
that EPA has applied or partially applied to its draft regulatory 
impact analysis (RIA) for particulate matter (PM). This table also 
provides GAO's assessment of EPA's progress in applying each 
recommendation, in terms of steps EPA has taken thus far to address 
issues highlighted in the National Academies' report. The final column 
characterizes EPA's comments regarding each recommendation, including, 
as pertinent, contextual information, potential impediments to 
application, and intended next steps. 

Table 1: Recommendations Applied or Partially Applied to the Draft 
Particulate Matter Regulatory Impact Analysis: 

1; 
National Academies' recommendation: To the extent possible, EPA should 
estimate the benefits for several regulatory options that represent the 
full range of choices available to the decision maker. The regulatory 
options should include graded levels of stringency requirements and the 
time schedule for achieving reductions in emissions or exposures. If 
options are eliminated at an earlier stage, the rationale for doing so 
should be provided; 
Status: Partially applied; 
GAO assessment: EPA estimated the health benefits for the base case and 
some of the proposed standards for fine particulate matter, but did not 
estimate benefits that represented the full range of choices available 
to the decision maker. In addition, EPA did not estimate the benefits 
for coarse particulate matter. While EPA provided an explanation as to 
why it did not estimate benefits for the regulatory options for coarse 
particulate matter, it did not present its rationale for not estimating 
the benefits of the full range of options for fine particulate matter; 
EPA's response: EPA did not estimate benefits for coarse particulate 
matter primarily because of a lack of data and tools needed to conduct 
an accurate analysis. In addition, EPA said that time and resources 
limited the number of regulatory options to be modeled. EPA said that 
given these constraints, the agency selected regulatory options that 
represent reasonable bounds on alternatives. 

2; 
National Academies' recommendation: EPA should modify the air quality 
models used in translating predicted emissions into predicted levels of 
ambient air quality to reduce resources required for air quality 
modeling. This change is necessary if EPA is to evaluate multiple 
regulatory alternatives and to evaluate each alternative at reasonable 
time intervals, such as every 5 years. Evaluation of the ambient air 
quality associated with more emissions scenarios is also essential if 
the uncertainty inherent in emissions estimates is to be carried 
through to the estimation of avoided cases of mortality and morbidity; 
Status: Partially applied; 
GAO assessment: EPA has modified air quality models but has not yet 
demonstrated that these changes reduce resources. EPA reports faster 
modeling runs once it has designated a model but has also needed to 
conduct more complex model runs, which require time; 
EPA's response: EPA characterized its response to this recommendation 
as a work in progress. EPA said that the agency has achieved greater 
efficiency in the models, but that the overall time and resources 
devoted to modeling have not decreased because of the increased 
complexity and increased volume of air quality model runs. Citing 
emissions data limitations as an ongoing challenge, EPA reported 
notable improvements to the emissions inventory that contributed to 
efficiency gains in air quality modeling runs. Overall, EPA is working 
to balance efforts to streamline models with the demands for more 
sophisticated analyses. In addition, EPA cited additional analytical 
requirements, such as new peer review policies and OMB's expanded data 
quality guidelines, that add to the complexity of, and therefore time 
and resources needed for, its models. 

3; 
National Academies' recommendation: EPA should incorporate estimates of 
future trends in background mortality and morbidity for the major 
health outcomes, such as those that make up two-thirds of total deaths 
or lost life years that are under consideration; 
Status: Partially applied; 
GAO assessment: EPA incorporated future trends in background mortality 
rates but did not do so for future morbidity trends; 
EPA's response: EPA did not incorporate future morbidity trends because 
of time constraints. As time permits, EPA plans to incorporate such 
projections in the final analysis. 

4; 
National Academies' recommendation: As in all other stages of the 
benefits analysis, EPA should justify and clearly describe the 
assumptions and methods used to assess exposure, choose health 
outcomes, and select studies and concentration-response functions, 
paying careful attention to assessing and communicating key sources of 
uncertainty; 
Status: Partially applied; 
GAO assessment: The draft PM RIA presented EPA's justifications and 
included clear descriptions of a number of assumptions and methods used 
in the benefit analysis. As discussed elsewhere, EPA's assessment of 
key sources of uncertainty generally relied on qualitative discussion 
and sensitivity analysis (see recommendation 14). However, the extent 
to which EPA provided justifications and clear descriptions varied. For 
example, EPA included detailed information about why it chose studies 
related to the concentration-response function, but the agency did not 
present its justification for an assumption used to assess exposure; 
EPA's response: EPA intends to include references in the final RIA, 
detailing the assumptions and methods underlying its health benefits 
analyses. EPA stated that rather than restating information from prior 
RIAs, it provides references to these discussions in order to manage 
the length of the RIA and save time. Furthermore, EPA stated that it is 
not necessary to document assumptions for tools that have already been 
peer reviewed, such as BenMAP, the model used to estimate health 
benefits. 

5; 
National Academies' recommendation: Because pollution modeling rarely 
addresses the smaller-scale issue of how local concentrations from 
specific source categories interact with human time-activity patterns, 
EPA should examine how different major source categories--for example, 
mobile versus large stationary sources--affect total exposures per unit 
emissions; 
Status: Partially applied; 
GAO assessment: EPA assessed local concentrations in terms of how 
different source categories, such as stationary and mobile sources of 
PM, affect total exposures. However, EPA has not yet assessed how human-
time activity patterns, such as lifestyles, affect exposure to PM; 
EPA's response: EPA focused on assessing the local concentrations from 
specific source activities but assumed the same time and activity 
patterns for each scenario. EPA stated that it does not intend to 
conduct a detailed analysis of micro-environmental issues or human-time 
activity patterns. 

6; 
National Academies' recommendation: EPA should consider data from U.S. 
and non-U.S. studies to extrapolate beyond the age groups evaluated and 
incorporate other relevant outcomes not evaluated in its current 
benefit analyses; 
Status: Applied; 
GAO assessment: EPA considered data from additional U.S. studies as 
part of its effort to expand the age groups in its estimates of the 
health outcomes (premature mortality and illness). EPA extrapolated 
beyond the age groups in some cases, but not all; 
EPA's response: EPA determined, on the basis of advice from the Science 
Advisory Board, that it would only extrapolate data to other ages when 
it found a reasonable physiological basis for doing so. For example, 
EPA used data from a study on asthma in children ages 7 to 11 to 
estimate the reductions in asthma for the entire child age group--ages 
6 to 18. 

7; 
National Academies' recommendation: EPA provided little information in 
the benefit analyses reviewed by the committee on causal association 
between particular types of air pollution and adverse health outcomes. 
EPA should summarize the evidence for causality to justify the 
inclusion or exclusion of the health outcomes and to assess the 
uncertainty associated with the assumption of causality; 
Status: Applied; 
GAO assessment: EPA referred readers to a prior RIA for information on 
the causal association between particulate matter and adverse health 
outcomes; 
EPA's response: [Empty]. 

8; 
National Academies' recommendation: EPA should investigate and, if 
necessary, develop methods of evaluating causal uncertainty relating to 
key outcomes so that this uncertainty can be represented in the final 
benefit estimates; 
Status: Applied; 
GAO assessment: EPA investigated one method--expert elicitation--to 
evaluate how causal uncertainty affects final benefit estimates. See 
recommendation 14 for more details; 
EPA's response: EPA intends to continue its effort to better 
characterize the uncertainty in key health outcomes. 

9; 
National Academies' recommendation: Although the committee believes the 
use of the American Cancer Society (ACS) study to derive premature 
mortality estimates was reasonable, EPA should thoroughly review its 
selection of the best estimate for long-term effects of air pollution 
on mortality. Several new studies have been published since the ACS 
study, including an extended analysis of the ACS study, a new U.S. 
cohort study, and other non-U.S. studies. EPA should also consider 
whether the derivation of a weighted mean estimate from the cohort 
studies is appropriate following review of the database; 
Status: Applied; 
GAO assessment: EPA reviewed its selection of the best estimate for 
long-term effects of air pollution on mortality and concluded that data 
from the extended analysis of the ACS study provided the best estimates 
of premature mortality. EPA consulted the Science Advisory Board to 
reach its decision to emphasize the ACS data. EPA also determined that 
it was not appropriate to derive a weighted mean estimate from cohort 
studies; 
EPA's response: EPA incorporated new studies regarding estimates of 
premature mortality and justified the concentration-response functions 
derived from these studies. These justifications were included in 
technical appendices to the draft PM RIA. EPA also consulted with the 
Science Advisory Board regarding the best estimates for premature 
mortality. EPA decided not to derive a weighted mean estimate because 
the use of this estimate would risk losing the variability across 
cohort studies. 

10; 
National Academies' recommendation: To evaluate short-term effects of 
air pollution, EPA should use concentration-response functions from 
studies that integrate over several days or weeks the exposure period 
and the time period to the event (cumulative or distributed lag 
models), rather than those that restrict these time periods to 1 or 2 
days; 
Status: Applied; 
GAO assessment: EPA used studies that integrate distributed lag models 
that account for the onset of health effects occurring more than 
several days after exposure; 
EPA's response: [Empty]. 

11; 
National Academies' recommendation: Although the assumption of no 
thresholds in the most recent EPA benefit analyses was appropriate, EPA 
should evaluate threshold assumptions in a consistent and transparent 
framework using several alternative assumptions in the formal 
uncertainty analysis; 
Status: Partially applied; 
GAO assessment: EPA evaluated several different threshold assumptions 
in a sensitivity analysis but has not yet considered these assumptions 
in a formal uncertainty analysis. The sensitivity analysis was 
transparent--EPA clearly explained the basis for the different 
threshold assumptions; 
EPA's response: EPA reported some progress toward improving its 
approach to characterize uncertainties and to conduct a formal 
uncertainty analysis. (See recommendation 14 for more information.) 

12; 
National Academies' recommendation: The committee found little 
justification for the 5-year time course of exposure and outcome 
assumed in more recent EPA analyses and recommends that EPA more fully 
account for the uncertainty regarding lags in health effects by 
incorporating a range of assumptions and probabilities about the 
temporal relationship; 
Status: Partially applied; 
GAO assessment: EPA sought advice from the Advisory Council on Clean 
Air Compliance (Council) and incorporated the advice by adjusting the 
agency's assumptions regarding lag times. As for the uncertainty 
surrounding lags in health effects, EPA conducted a series of 
sensitivity analyses on the temporal relationship but did not assess 
underlying probabilities; 
EPA's response: Consistent with the recommendation made by the Council, 
EPA now uses a 20-year lag model. The revised lag model involves a 
higher percentage of mortality reductions occurring in the first year 
(30 percent) than the previous EPA lag model (20 percent). EPA told us 
that it will not include probability-based distributions in the final 
RIA because of computational complexities. 

13; 
National Academies' recommendation: EPA is encouraged to estimate and 
report benefits by age, sex, and other demographic factors. The 
committee recognizes, however, that evaluating the differences for 
various subgroups adds complexity and uncertainty to the analysis and 
that caution must be exercised in the interpretation of such results; 
Status: Partially applied; 
GAO assessment: EPA included some health benefit estimates for specific 
age groups in an appendix of the draft PM RIA. However, the agency did 
not report the primary benefit estimates according to age groups or 
other demographic factors; 
EPA's response: EPA intends to present in the final RIA a detailed 
summary table with the age breakdown of its sample population. EPA will 
not, however, estimate and report benefits by other demographic factors 
because of time and resource constraints. EPA also cited data 
limitations, noting that it lacks data accounting for local variations 
in demographic factors. The agency intends to overcome this limitation 
using tract-level data, which captures details about local level 
conditions, from the U.S. Census data in future analyses. 

14; 
National Academies' recommendation: EPA should begin to move the 
assessment of uncertainties from its ancillary analyses to its primary 
analyses. This shift will require the specification of a probability 
distribution for each uncertainty source that is added to the primary 
analysis and, as necessary, the specification of joint distributions 
for the uncertainty sources that are not independent of each other. 
Expert judgment, as well as data, will be required to specify these 
distributions. Although the effect on the mean of the resulting 
probability distribution might increase, decrease, or remain the same, 
the effect on the spread of the distribution will be a predictable 
widening and, therefore, a more realistic depiction of the overall 
uncertainty in the analysis; 
Status: Partially applied; 
GAO assessment: EPA has taken some steps toward the formal uncertainty 
analysis called for by this recommendation, but the primary analysis in 
the draft PM RIA generally addresses uncertainty in a qualitative 
manner. Overall, the numerical benefit estimates do not capture the key 
sources of uncertainty. The agency generally relied on sensitivity 
analysis to assess some uncertain factors one at a time rather than 
using more comprehensive techniques for assessing probability 
distributions of multiple variables. In addition, EPA used expert 
elicitation to help assess uncertainty relating to the concentrations 
of PM linked to premature death and the dollar value of risk reductions 
associated with reductions in PM. However, these results were presented 
in an appendix and not in the primary analysis in the draft RIA. In 
addition, the health benefit analysis did not present a quantitative 
assessment of how the benefit estimates would vary in light of other 
key uncertainties; 
EPA's response: EPA reported some progress toward improving its 
approach to characterize uncertainties with particular emphasis on one 
source of uncertainty--premature death linked to PM exposure. 
Specifically, EPA used expert elicitation to begin to specify a 
distribution for the uncertainty in concentration levels of PM linked 
to premature death. The agency cited technical challenges, such as lack 
of data or reliable methods, and resource constraints, including 
limitations to its progress to fully characterize uncertainty. 
Moreover, EPA stated that its focus on the expert elicitation technique 
limited the time and resources necessary to address other aspects of 
uncertainty in premature death as well as illnesses linked to PM 
exposure. 

EPA applied another formal method for assessing uncertainty--Monte 
Carlo analysis--to a previous regulatory impact analysis but, according 
to EPA, did not have time to incorporate this work in the PM RIA. Time 
permitting, the agency plans to present the results of a Monte Carlo 
analysis in the final PM RIA. 

15; 
National Academies' recommendation: Because the incorporation of expert 
judgment when data are unavailable will influence the estimates of 
health benefits as well as the uncertainty analyses, the committee also 
recommends that EPA clearly distinguish between data-derived estimates 
of some components, such as the concentration-response function, and 
expert opinions about other components that are lacking in scientific 
data, such as the degree of compliance with a particular regulation 30 
years into the future. In this way, policymakers will better understand 
how existing data and expert judgment combine to produce estimates and 
where new data would be most valuable; 
Status: Applied; 
GAO assessment: EPA distinguished between data-derived estimates and 
those from expert judgment. For example, in the appendix, EPA clearly 
distinguished between data derived from experts and the data based on 
an empirical study. In addition, EPA discusses the basis for 
assumptions, which require analytical judgment, either directly in the 
draft PM RIA or by reference to supporting documents; 
EPA's response: [Empty]. 

16; 
National Academies' recommendation: As EPA begins the transition to 
incorporate additional sources of uncertainty into its primary health 
benefits analyses, it should continue the sensitivity analyses it has 
traditionally conducted. These analyses should be expanded, however, to 
consider sources of uncertainty jointly rather than singly; 
Status: Partially applied; 
GAO assessment: EPA expanded the sensitivity analysis by, for example, 
considering how benefit estimates change according to different 
threshold assumptions, such as cut points-- concentrations of 
particulate matter below which there would be no benefit to further 
reductions. This sensitivity analysis examined how different assumed 
cut points would change the estimates of avoided cases of death. EPA 
did not, however, conduct a sensitivity analysis to consider the 
sources of uncertainty jointly rather than singly; 
EPA's response: EPA has begun to explore how to expand sensitivity 
analysis and consider sources of uncertainty jointly. EPA noted that in 
recent air pollution analyses, the agency considered sources of 
uncertainty jointly for the last stage of the benefit estimation 
process--valuing the benefits in dollar terms--by doing a probabilistic 
assessment of the value of a statistical life. 

In addition, EPA's policy and economics division has begun a long-term 
project to expand uncertainty analysis. EPA reports that one technique 
that considered how much different sources of uncertainty affect the 
estimates is not ready to be applied to a rule-making analysis. 

17; 
National Academies' recommendation: In presenting the probability 
distribution for each health benefit produced by a primary analysis, 
EPA should emphasize even more than it has in the past the sources of 
uncertainty that remain unaccounted for in the primary analysis. These 
uncertainties should continue to be described as completely and 
realistically as possible; 
Status: Partially applied; 
GAO assessment: EPA did not present a probability distribution for the 
primary benefit analysis but included some discussion of sources of 
uncertainty not incorporated into the benefit estimates. EPA also 
referred the reader to a previous analysis involving particulate matter 
for detailed discussions on sources of uncertainty. The citations are 
incomplete, however, leaving readers to search within the voluminous 
document for the relevant information; 
EPA's response: EPA attributed incomplete or lacking references in the 
draft PM RIA to time constraints, noting that it would present clear 
references in the final version. EPA also clarified that it relies on 
references to other documents, rather than repeating information that 
has not changed, in order to keep the presentation to a manageable 
size. 

(See also recommendation 14 for EPA's basis for not including 
probability distributions.) 

18; 
National Academies' recommendation: EPA should consider providing a 
preliminary analysis that estimates, in current populations, the health 
benefits resulting from hypothetical changes to current levels of 
emissions. These preliminary analyses would help EPA develop an idea of 
the lower bound on the uncertainty of future consequences and would 
have fewer uncertainties than analyses of the impacts of proposed 
regulatory actions on future exposures and health outcomes; 
Status: Applied; 
GAO assessment: According to EPA, the agency considered providing the 
preliminary analysis described in this recommendation; 
EPA's response: EPA determined that this information would not provide 
meaningful information to the draft PM benefit analysis. Aside from its 
questions about the technical feasibility of responding to this 
recommendation, EPA expressed doubts that doing this analysis for 
current populations would establish a lower bound on the uncertainty of 
future consequences. One EPA official concluded that the analysis could 
introduce more uncertainty into its benefit estimates because benefit 
projections will depend on controls implemented in the future. Finally, 
EPA clarified that, as provided by Circular A-4, it estimates benefits 
for a future year when the regulatory revisions become effective, not 
the current year, which would not account for changes in variables 
other than emissions. 

19; 
National Academies' recommendation: EPA should continue to strive to 
present the results of its health benefit analyses in ways that avoid 
conveying an unwarranted degree of certainty. Such ways include 
rounding to fewer significant digits, increasing the use of graphs, 
presenting projected baselines along with projected health benefits, 
and placing less emphasis on single numbers (for example, the mean of 
the probability distribution for a health benefit) and greater emphasis 
on ranges (for example, the range between 5th and 95th percentiles of 
the distribution); 
Status: Applied; 
GAO assessment: EPA followed some of the National Academies' 
suggestions to present the data in a way that avoids conveying an 
unwarranted level of certainty. EPA rounded the estimates to fewer 
significant digits. EPA increased the emphasis on ranges by presenting 
some data in ranges of benefit and cost estimates. EPA summarized the 
primary benefit estimates in a table; 
EPA's response: [Empty]. 

20; National Academies' recommendation: EPA should perform similar 
detailed analyses of uncertainty in the valuation of health benefits 
and in the regulatory cost analyses that the committee recommends for 
the health benefit analyses; 
Status: Partially applied; 
GAO assessment: The draft PM RIA demonstrated some steps toward 
analysis of cost uncertainty but did not present an uncertainty 
analysis of the benefit valuation--the stage when it assigns a dollar 
value to the benefit estimates. In terms of cost uncertainty, EPA 
presented a qualitative discussion of the uncertainties about costs and 
the expected impact on the cost estimates but did not perform a formal 
uncertainty analysis for the costs; 
EPA's response: Overall, EPA cited technical challenges, including data 
limitations, as the primary reason for not applying a formal 
uncertainty analysis to the cost estimates in the PM RIA. According to 
EPA, the number and breadth of PM sources considered under the National 
Ambient Air Quality Standards rule poses a challenge to development of 
a characterization of all costs and their uncertainties. Moreover, EPA 
noted that the collection process for cost information is not 
systematic, and there is a limited amount of information about the cost 
of implementing some types of controls. Even when costs are known, it 
may be difficult to specify the underlying probability distributions. 

EPA told us that it will incorporate more refined information, such as 
emissions control analysis, affecting cost estimates in the final RIA. 
EPA expects the final RIA to include more reasonable cutoffs such that 
cost estimates will not be based on ineffective controls and excessive 
costs (i.e., controls that are not likely to be used). 

In terms of health valuation uncertainty, EPA has made progress toward 
a formal uncertainty analysis by doing a probabilistic assessment of 
the value of a statistical life. EPA conducted this analysis in prior 
RIAs but did not have enough time to complete model runs in time for 
the draft RIA. EPA plans to incorporate an uncertainty analysis related 
to the mortality valuation in the final PM RIA. 

EPA has focused its expansion of uncertainty analysis more on benefits 
than on costs, in part because the benefit estimates for air rules tend 
to be much larger. EPA identified efforts to analyze cost uncertainty 
factors, including sensitivity analyses in previous RIAs and plans for 
accountability analysis comparing estimated costs to actual costs. 
Finally, EPA pointed out that the work it conducts to understand the 
benefits also carries over to the cost estimates. 

21; 
National Academies' recommendation: EPA should provide health benefit 
estimates in ways that will support multiple kinds of analysis, 
including various approaches to mortality valuation and aggregation of 
benefits using quality-adjusted life years; 
Status: Partially applied; 
GAO assessment: EPA's detailed breakdown of the benefit estimates and 
the reference to its benefit model allows others to apply various 
approaches to mortality valuation, such as alternative estimates of the 
value of a statistical life. EPA did not, however, aggregate benefits 
using quality-adjusted life years; 
EPA's response: Although EPA used the quality-adjusted life years 
approach in a prior RIA to aggregate benefits, EPA did not use this 
approach in the draft PM RIA because of time constraints. EPA told us 
that it plans to aggregate the benefits using quality-adjusted life 
years in the final PM RIA. 

22; 
National Academies' recommendation: Each analysis should provide 
results according to demographic or other subgroups when the expected 
changes in pollution and, thus, the health benefits are not distributed 
uniformly across the population. This information would aid decision 
makers in situations in which equity issues might be involved; 
Status: Partially applied; 
GAO assessment: EPA included some health benefit estimates for specific 
age groups in an appendix, but the agency did not present the primary 
benefit estimates according to age groups or other demographic factors. 
(See also recommendation 13.); 
EPA's response: EPA intends to present in the final RIA a detailed 
summary table with the age breakdown of its sample population. The 
agency noted that it will not present the benefits according to other 
demographic factors, such as race and income, because of political 
sensitivities and data limitations. EPA clarified that it lacks the 
data on local conditions but that it intends to overcome this 
limitation using tract- level data--which captures details about local 
conditions--from the U.S. Census data in future analyses.  

Source: GAO analysis of National Academies and EPA information. 

[End of table]

[End of section] 

Appendix III: Recommendations Not Applied to the Draft Particulate 
Matter Regulatory Impact Analysis: 

Table 2 provides a summary of the National Academies' recommendations 
that EPA has not applied to its draft regulatory impact analysis (RIA) 
for particulate matter (PM). This table provides GAO's assessment of 
EPA's progress to date regarding recommendations that required 
additional research and development, were deemed as not relevant to the 
PM National Ambient Air Quality Standards (NAAQS) by the agency, or 
were not included in the draft PM RIA due to time and resource 
constraints. The final column characterizes EPA's comments regarding 
each recommendation, including contextual information, potential 
impediments to application, justification for not addressing the 
recommendation, and intended next steps, if applicable. 

Table 2: Recommendations Not Applied to the Draft Particulate Matter 
Regulatory Impact Analysis: 

1; 
National Academies' recommendation: The uncertainty in emissions 
estimates should be quantified and carried through the health benefit 
analysis to the calculation of avoided cases of mortality and 
morbidity; 
Status: Not applied--research and development under way; 
GAO assessment: EPA has not quantified the uncertainty related to 
emissions because of limited data and computational complexities and 
has therefore not yet carried such uncertainty through in the health 
benefit analysis; 
EPA's response: EPA stated that the application of this recommendation--
conducting a formal analysis of emissions uncertainty--requires long-
term research and development. EPA reports that it discussed the 
possibility of conducting a quantitative uncertainty analysis for 
emissions in its particulate matter analysis, but the fixed timeline 
prevented EPA from doing this work. The primary challenge stems from 
the nature of the emissions inventory--data are collected from a 
plethora of entities, complicating the agency's ability to evaluate 
uncertainty. 

EPA told us that, currently, the only way to assess emissions 
uncertainty is through qualitative means. EPA also stated that its 
final particulate matter analysis will demonstrate steps toward this 
recommendation because it will present a sensitivity analysis of the 
emissions data and will present emissions data according to the level 
emitted by the different kinds of sources, such as utilities, cars, and 
trucks. 

2; 
National Academies' recommendation: Because a regulation to improve air 
quality may affect pathways other than air, EPA should determine 
whether there are likely to be any important indirect impacts of a 
regulation on human health and the environment. If any such impacts are 
identified, EPA should include in the analysis a plan to assess them 
more completely; 
Status: Not applied--research and development under way; 
GAO assessment: While EPA did not provide information in the draft PM 
RIA to show that it considered indirect impacts involving pathways 
other than air, EPA's ongoing research on the environmental impacts may 
identify important indirect impacts; 
EPA's response: EPA stated that in past rules, the agency has looked at 
indirect impacts in terms of deposition of nitrogen and sulfates to 
sensitive water bodies. EPA plans to incorporate this analysis in the 
final PM RIA. EPA said it has not yet identified any other indirect 
impacts. While an EPA official suggested that this recommendation did 
not seem relevant to the NAAQS analysis in terms of human health 
impacts--EPA could not determine how human health would be affected by 
exposure to PM from pathways other than air--the agency is conducting 
research to identify important indirect impacts on the environment. EPA 
characterized environmental impacts as an area of research. 

3; 
National Academies' recommendation: EPA has typically made the 
assumption of equivalent potency across particle types because of 
insufficient scientific information. As more data become available, EPA 
should strengthen its benefit analyses by evaluating a range of 
alternative assumptions regarding relative particle toxicity and 
incorporate these assumptions in sensitivity or uncertainty analyses; 
Status: Not applied--research and development under way; 
GAO assessment: Although EPA assumed equivalent toxicity in the PM RIA 
and did not include related sensitivity or uncertainty analyses, EPA is 
sponsoring research directed at incorporating its findings on relative 
toxicity into future analyses; 
EPA's response: EPA stated that it does not have sufficient information 
to distinguish between particle components for the final rule. EPA is 
funding long-term research on relative toxicity, including technical 
studies to understand any differential toxicities as well as economic 
analyses to explore ways to characterize the uncertainty in benefit 
estimates. For example, an EPA contractor conducted a sensitivity 
analysis of relative toxicity of particle components, including 
carbons, nitrates, crustal material, and sulfates. To date, the 
contractor has created a model to assess whether and how much these 
sources of uncertainty may affect benefit estimates in one urban area. 
EPA is also supporting research to explore relative toxicity through 
its intramural research program, its five Particulate Matter Research 
Centers, and the Health Effects Institute, an organization funded in 
part by EPA. EPA's science grant program recently awarded $40 million 
to the five Particulate Matter Research Centers. 

4; 
National Academies' recommendation: As it incorporates additional 
sources of uncertainty into its primary health benefit analyses, EPA 
should consider conducting analyses to determine which uncertainty 
sources have the greatest influence on the mean and spread of the 
probability distribution. The need for these sensitivity analyses will 
be particularly great for distributions that are based on expert 
judgment. The uncertainty sources that have the greatest consequences 
for decision making, including those that have the greatest impact on 
the spread of the distribution, should be given high priority for 
additional research; 
Status: Not applied--research and development under way; 
GAO assessment: In the draft PM RIA, although EPA presented a 
qualitative discussion about the importance of its assumptions that 
impact uncertainty, it did not consider which sources of uncertainty 
have the greatest influence on the mean and spread of the probability 
distribution. EPA is sponsoring research to incorporate influence 
analysis in future analyses. (See recommendation 14; EPA did not 
specify probability distributions for uncertainty in its primary 
analysis); 
EPA's response: An EPA contractor is researching techniques for 
influence analysis, an expanded form of uncertainty analysis that would 
determine which uncertainty sources have the greatest influence on the 
benefit estimates. The influence analysis work targeted three sources 
of uncertainty, including the concentration-response function (which 
involves threshold and slope); lag effects; and relative toxicity. The 
contractor compiled a draft report discussing techniques to conduct 
this analysis and incorporate the uncertainty analysis in benefit 
estimates. 

EPA and the contractor stated that because these techniques are in the 
exploratory stage, it is premature to apply this work to a specific 
rule-making analysis. For example, EPA cited the need to determine how 
much uncertainty is explained by differential toxicity or by different 
thresholds. Finally, EPA stated that because it focused its resources 
on the expert elicitation work (see recommendation 14), the influence 
analysis received fewer resources and has not advanced as quickly. 

5; 
National Academies' recommendation: EPA should estimate the benefits 
over the regulatory time period, including both the implementation 
period and the expression period of all important health effects. 
Because calculating benefits for every future year is resource 
intensive and unlikely to show true increases in precision, 
calculations can be made, for example, every fifth year with simple 
interpolation techniques applied to estimate benefits for intervening 
years; 
Status: Not applied; 
GAO assessment: EPA did not estimate benefits covering the 
implementation period and expression period of all important health 
effects. EPA estimated the benefits expected in one year only--the base 
case year, 2015. The year 2015 is the first attainment date when states 
should be in compliance with the new standards. The Clean Air Act 
allows for up to a 5-year extension for states that cannot meet the 
standards by the attainment year. Therefore, the implementation period 
for some states may extend to 2020; 
EPA's response: EPA cited limited time and resources to estimate 
benefits for years other than 2015, but plans to also include estimates 
for 2020, in the final RIA. In addition, EPA did not think that this 
recommendation is particularly meaningful to the NAAQS analysis because 
the only variable change over the course of the PM implementation 
period is population. EPA concluded that there would not be much 
difference between the benefit estimates given in 5-year increments 
(i.e., 2010 and 2015). 

6; 
National Academies' recommendation: The components of emissions 
estimates (such as number of vehicles in a class, average miles 
traveled per vehicle, and emissions per mile) should be presented with 
and without implementation of the regulation at the national level. 
This will help readers judge how reasonable these predictions are and 
will suggest which components of emissions estimates drive the 
emissions reductions associated with the regulation. Historical trends 
in these components should also be presented; 
Status: Not applied; 
GAO assessment: EPA presented a qualitative discussion about emissions, 
but did not address components of emissions estimates or provide 
information to allow readers to understand which components of 
emissions estimates drive the reductions associated with regulation, 
such as activity level or emissions intensity. EPA did not present 
historical trends; 
EPA's response: EPA stated that the kind of information called for by 
this recommendation--key input data, assumptions, and intermediate 
modeling outcomes--would be useful in future rule-making analyses, but 
that the agency needs to review a large amount of data to determine 
which elements would be most helpful. EPA questioned the value of 
incorporating all of the recommended information in regulatory impact 
analyses, noting that it works to present enough information to readers 
while maintaining a document of manageable length. 

EPA stated that it needs time to design data reporting strategies that 
would be; appropriate for the different scales and scopes of the 
regulatory impact analyses. EPA plans to explore this recommendation 
more thoroughly as part of its comprehensive economic analysis of the 
Clean Air Act; Finally, EPA officials suggested that the final PM RIA 
will respond to this recommendation in part by providing information 
about how emissions reductions might vary across PM sources in order to 
show the primary drivers of emissions reductions in the final RIA. EPA 
also plans to compare current data to the historical trends predicted 
in the past and show that EPA's current predictions are for the future. 

7; 
National Academies' recommendation: EPA should quantify uncertainties 
with regard to future population distributions and background disease 
rates. EPA should also summarize what is known about the potential 
importance of disease interactions and competing risks affecting the 
health outcomes of primary interest and discuss the possible biases 
that might be introduced in the final analysis by changes in those 
factors; 
Status: Not applied; 
GAO assessment: EPA did not quantify the uncertainties related to 
future population distributions and background disease rates. EPA did 
not summarize what is known about the potential importance of disease 
interactions and competing risks affecting health outcomes of primary 
interest; 
EPA's response: EPA said its models are not configured to quantify 
these sources of uncertainty. EPA would need to modify the model in 
order to do this analysis. Agency officials noted that the agency tries 
to be selective when determining which sources of uncertainty to assess 
because the cost of doing this work might outweigh the value added from 
the information. 

In addition, EPA officials said the agency does not have all of the 
data necessary to reconfigure the models to quantify key sources of 
uncertainty. EPA disagrees with the National Academies' comment that 
lack of information should not preclude the quantification of 
uncertainty. EPA believes that using the techniques to quantify 
uncertainty without empirical data would generate results that could be 
more misleading than the results that do not account for uncertainty. 
EPA plans to add relevant uncertainty characterizations as it obtains 
data. 

8; 
National Academies' recommendation: The lack of clear categorization of 
severity of certain health outcomes in benefits analyses has 
implications for the quantification and the valuation of these 
outcomes. Although EPA has made some attempt to recognize this issue, 
it should continue to develop and improve methods used to reconcile 
differences between the severity of disease described in air pollution 
epidemiology and that commonly used to develop estimates of background 
disease prevalence and incidence; 
Status: Not applied; 
GAO assessment: The model EPA used to estimate benefits, BenMAP, did 
not account for the potential variations in severity of illnesses and 
prevalence and expected incidence of health effects; 
EPA's response: EPA has begun work on categorizing severity of some 
health outcomes, including chronic bronchitis and asthma incidence. EPA 
is continuing to refine its BenMAP model to better quantify and 
monetize health outcomes. 

9; 
National Academies' recommendation: EPA should give more emphasis to 
the assessment, presentation, and communication of changes in 
morbidity. Although often difficult to quantify, these factors may 
begin to play a more dominant role in benefit analysis if the value 
assigned to mortality decreases; 
Status: Not applied; 
GAO assessment: EPA did not place more emphasis on the assessment, 
presentation, and communication of changes in morbidity. Prior RIAs 
have quantified expected changes in morbidity, such as reductions in 
asthma and chronic bronchitis. The draft PM RIA did not include 
additional morbidity information. Moreover, the main benefit estimates 
in the draft PM RIA did not include morbidity estimates; 
EPA's response: EPA acknowledged the importance of morbidity--i.e., 
illness--benefits and is working to expand that analysis. For example, 
EPA told us that it plans to include sensitivity analysis of changes in 
illnesses in the final PM RIA. EPA also stated that the final PM RIA 
will reflect updates to its model-- EPA is working to include 
projections that will allow the agency to evaluate expected changes in 
illnesses such as asthma and chronic bronchitis. 

10; 
National Academies' recommendation: There is a common misperception 
that a high degree of certainty is required for regulatory actions to 
take place to protect public health. As a result, primary health 
benefit analyses that more fully and accurately portray the 
uncertainties might not be considered useful. It is unrealistic for EPA 
to defer decisions until it can make them on the basis of perfect 
science. A careful and deliberate balancing of the benefits and costs 
is required, and this balancing must be informed by a fair assessment 
of the current levels of uncertainty and a realistic evaluation of the 
likely reductions in uncertainty attainable through further research; 
Status: Not applied; 
GAO assessment: EPA did not balance the costs and benefits--the Clean 
Air Act prohibits EPA from basing revisions to the NAAQS on costs. We 
note that in response to other recommendations, EPA has taken steps 
toward assessing current levels of uncertainty--see appendix II, 
recommendations 14 and 20; 
EPA's response: EPA characterized this recommendation as not germane to 
the particulate matter regulatory impact analysis because the Clean Air 
Act prohibits the agency from considering the costs when revising 
NAAQS. 

11; 
National Academies' recommendation: EPA should provide a summary of the 
analysis containing information as outlined in the National Academies' 
report (table 6-1). This information would allow the reader to evaluate 
the study design and verify estimates obtained in the analysis; 
Status: Not applied; 
GAO assessment: EPA did not provide the summary table in the executive 
summary as outlined in the National Academies' report, including a 
description of regulatory options, boundaries of analysis, regulatory 
baseline, and assumptions that have a significant impact on results of 
analysis; 
EPA's response: Due to data and time constraints, EPA did not summarize 
its conclusions in the executive summary of the PM RIA. 

12; 
National Academies' recommendation: To enhance the quality of future 
regulatory benefit analyses, a standing, independent technical review 
panel should advise EPA in the initial stages of its benefit analysis. 
This panel should have expertise in regulatory options analysis, 
emissions and exposure assessment, toxicology, epidemiology, risk 
analysis, biostatistics, and economics and should be appointed with 
strict attention to avoiding conflict of interest, balancing biases and 
ensuring broad representation. This panel should be supported by 
permanent technical staff to ensure consistency of reviews over time. 
EPA should follow the panel's guidance on the need for peer review; 
Status: Not applied; 
GAO assessment: EPA did not convene a standing group of experts to 
guide the agency's initial work on the draft PM RIA. In the course of 
developing economic methodologies, EPA sought and considered 
information from existing advisory committees on the soundness of 
certain assumptions, such as cessation lags; 
EPA's response: EPA stated that in addition to the advice it sought 
from the National Academies, the agency continues to seek input from 
independent committees, including the Clean Air Scientific Advisory 
Committee, the Advisory Council on Clean Air Compliance Analysis, and 
subcommittees chartered by the Science Advisory Board's Environmental 
Economics Advisory Committee. The committees advised EPA on data, 
methods, and modeling choices applicable to various economic analyses, 
including the draft PM RIA. 

EPA stated that the costs involved in convening an entirely new panel 
could be prohibitively expensive, due to organizing costs and travel 
expenses and the scope of the Academies' recommendation. As a result, 
EPA said that it has attempted to take advantage of existing groups, 
such as the Advisory Council on Clean Air Compliance Analysis, rather 
than arrange for a new panel; Furthermore, EPA noted that new 
methodologies and assumptions used in the PM RIA were peer-reviewed and 
that previously used methods used in the RIA had already been reviewed 
and validated in a prior context. 

Source: GAO analysis of National Academies and EPA information. 

[End of table] 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

John B. Stephenson, (202) 512-3841 or s [Hyperlink, 
stephensonj@gao.gov] tephensonj@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Christine Fishkin, Assistant 
Director; Kate Cardamone; Nancy Crothers; Cindy Gilbert; Tim Guinane; 
Jessica Lemke; and Meaghan K. Marshall made key contributions to this 
report. Timothy Bober, Marcia Crosse, and Karen Keegan also made 
important contributions. 

(360660): 

FOOTNOTES 

[1] American Lung Ass'n v. Whitman, No. 1:03CV00778 (D. D.C. 2003). 

[2] The National Academies comprises four organizations: the National 
Academy of Sciences, National Academy of Engineering, Institute of 
Medicine, and National Research Council. 

[3] National Research Council, Estimating the Public Health Benefits of 
Proposed Air Pollution Regulations (Washington, D.C., 2002). 

[4] The Clean Air Act prohibits EPA from basing revisions to the 
national air quality standards on costs. Because most national air 
quality revisions qualify as significant actions under Executive Order 
12866, EPA usually assesses the expected benefits and costs of the 
standards under the Executive Order. 

[5] Specifically, Circular A-4 states that agencies should conduct a 
formal probabilistic assessment of key uncertainties underlying its 
cost-and-benefit estimates. OMB Circular No. A-4, Regulatory Analysis 
(Sept. 17, 2003). 

[6] GAO, Air Pollution: Information Contained in EPA's Regulatory 
Impact Analyses Can Be Made Clearer, GAO/RCED-97-38 (Washington, D.C.: 
Apr. 14, 1997); and Cost-Benefit Analysis Can Be Useful in Assessing 
Environmental Regulations, Despite Limitations, GAO/RCED-84-62 
(Washington, D.C.: Apr. 6, 1984). 

[7] Monte Carlo simulation refers to a computer-based analysis that 
uses probability distributions for key variables, selects random values 
from each of the distributions simultaneously, and repeats the random 
selection over and over. Rather than presenting a single outcome--such 
as the mostly likely or average scenario--Monte Carlo simulations 
produce a distribution of outcomes that reflect the probability 
distributions of modeled uncertain variables. 

[8] It is important to note, according to EPA, that quantified benefit 
estimates do not include other potential benefits, such as reduction in 
certain illnesses or environmental impacts, because of limited data. 
The fraction of total benefits attributable to reductions in mortality 
may therefore vary as other benefits are incorporated in the numerical 
estimates. 

[9] The pilot expert elicitation, based on methods that were peer- 
reviewed, involved structured, daylong interviews with five experts 
about particulate matter exposure and death. EPA then analyzed the 
experts' responses to develop ranges reflecting the experts' confidence 
in estimated reductions in premature death associated with the proposed 
revisions. Pending completion of a peer review, EPA plans to include 
the analysis of a full-scale expert elicitation panel in the final 
regulatory impact analysis. 

[10] Recent EPA analyses used the natural background concentrations of 
particulate matter, rather than zero, for its assumption of no 
threshold level. The National Academies supported the assumption of no 
threshold level, but it recommended that EPA conduct a consistent and 
transparent sensitivity analysis to consider various threshold levels. 

[11] Particulate matter is a highly complex mixture comprising 
particles emitted directly from sources and particles formed through 
atmospheric chemical reactions. Particles span many sizes and shapes 
and consist of hundreds of different chemicals. EPA identifies the 
major components of fine particulate matter as carbon, sulfate and 
nitrate compounds, and crustal/metallic materials such as soil and ash. 

[12] Relative toxicity refers to the premise that different components 
of particulate matter have different levels of potency affecting 
premature mortality and illness. In the draft particulate matter 
regulatory impact analysis, EPA assumed equivalent toxicity, stating 
that "while it is reasonable to expect that the potency of components 
may vary across the numerous effect categories associated with 
particulate matter, EPA's interpretation of scientific information 
considered to date is that such information does not yet provide a 
basis for quantification beyond using fine particle mass." EPA, Draft 
Regulatory Impact Analysis for the PM-2.5 National Ambient Air Quality 
Standards (Washington, D.C., 2006), 3-21. 

[13] In the context of the National Academies recommendations, a 
sensitivity analysis would assess how changes in one or more variables 
affect the outcome, whereas a comprehensive or formal uncertainty 
analysis would evaluate the probability distributions of multiple 
variables. 

[14] In a separate report issued in 2004, the National Academies 
identified relative toxicity as a priority research topic, noting that 
technical challenges have impeded research progress. The Clean Air 
Scientific Advisory Committee also noted the need for more research and 
concluded in 2005 that not enough data are available to base the 
particulate matter standards on composition. OMB, however, encouraged 
EPA in 2006 to conduct a sensitivity analysis on relative toxicity and 
referred the agency to a sensitivity analysis on relative toxicity 
funded by the European Commission. 

[15] Because the precise levels of total emissions are not knowable but 
rather approximations based on a sample of measurements, there is 
uncertainty about the true quantity of emissions. 

[16] EPA compiles the National Emissions Inventory, a national database 
of air emissions data that includes estimates of annual emissions, by 
source, of air pollutants in each area of the country on an annual 
basis. 

[17] In prior work on regulatory economic analyses, we recommended that 
OMB direct agencies to obtain peer review of these analyses. See GAO, 
Regulatory Reform: Agencies Could Improve Development, Documentation, 
and Clarity of Regulatory Economic Analyses, GAO/RCED-98-142 
(Washington, D.C.: May 26, 1998). 

[18] The Advisory Council on Clean Air Compliance, composed of rotating 
membership, serves to advise the agency on choices relating to data, 
modeling, and methodology associated with air programs and does not 
review regulatory impact analyses. Established under the Clean Air Act 
and operated through the Science Advisory Board, the advisory council 
advises EPA, under Section 812 of the act, on developing the 
"statutorily mandated comprehensive analyses of the total costs and 
total benefits of programs implemented pursuant to the Clean Air Act." 
These analyses are commonly referred to as the 812 studies. 

[19] In 2004, EPA asked for and received guidance from the Advisory 
Council on Clean Air Compliance on the lag estimates the agency should 
use in the particulate matter analyses. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: