This is the accessible text file for GAO report number GAO-13-552 
entitled 'Grants Performance: Justice and FEMA Collect Performance 
Data for Selected Grants, but Action Needed to Validate FEMA 
Performance Data' which was released on June 24, 2013. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

June 2013: 

Grants Performance: 

Justice and FEMA Collect Performance Data for Selected Grants, but 
Action Needed to Validate FEMA Performance Data: 

GAO-13-552: 

GAO Highlights: 

Highlights of GAO-13-552, a report to congressional requesters. 

Why GAO Did This Study: 

The Government Performance and Results Act (GPRA) of 1993 began moving 
the federal government toward a more results-oriented culture, and in 
2010, GPRAMA established a new framework for taking a more integrated 
approach to focusing on results and improving government performance 
by establishing requirements for government-wide and agency-wide 
goals. Reporting progress on these and other goals may include grant 
performance information. GAO was asked to review grant performance 
reporting. This report examines (1) the extent to which OMB provides 
guidance to agencies on collecting, validating, and reporting 
performance data that demonstrate progress toward agency-wide and 
federal goals; and (2) how the selected agencies collect and validate 
data on the performance of selected grant programs and use those data 
to report progress toward agency-wide performance goals. For this 
review, GAO selected two grant programs each at Justice and FEMA. GAO 
reviewed OMB guidance on grant performance and relevant legislation, 
interviewed officials at OMB, Justice, and FEMA, and conducted a 
literature review to identify performance measurement issues. 

What GAO Found: 

The Office of Management and Budget (OMB) provides detailed guidance 
in Circular No. A-11 for the agency goal development and the 
performance data verification and validation processes. OMB Circular 
No. A-11 directs agencies to have a data validation plan for 
performance reporting and cost-effective validation and verification 
techniques in place to ensure the completeness and reliability of all 
performance measurement data used in annual performance plans. In 
2012, OMB revised Circular No. A-11 to incorporate the requirements of 
the Government Performance and Results Act Modernization Act of 2010 
(GPRAMA), which requires agencies to establish 2-year agency priority 
goals for areas where they seek to achieve near-term performance 
results but provides agencies with discretion to select their goals 
and the performance data used to show progress towards achieving these 
goals. Agencies must ensure that the data they use to report progress 
toward performance goals is accurate and reliable. 

The Justice Assistance Grant (JAG) program at the Department of 
Justice (Justice) collects performance data quarterly and qualitative 
summary narrative information annually. Justice’s Juvenile 
Accountability Block Grant (JABG) program collects quantitative and 
qualitative performance data annually and financial performance data 
quarterly. For both JAG and JABG, performance data are passed up from 
the grant program offices to the Office of the Chief Financial Officer 
within the Office of Justice Programs and are ultimately made 
available to the Justice Management Division. JAG’s data collection 
tool has an edit-check function to improve the reliability of data 
entered into the system, and Justice has established a policy to 
implement a new data validation tool for future use with JABG 
grantees. Justice established four agency priority goals for 2012 to 
2013, but it does not use JAG and JABG performance data as part of its 
evaluation of progress towards agency priority goals, as is allowed 
under GPRAMA and incorporated into Circular No. A-11. 

The Federal Emergency Management Administration’s (FEMA) Emergency 
Management Performance Grants (EMPG) and Assistance to Firefighters 
Grants (AFG) programs collect performance information through a 
variety of reporting mechanisms. These mechanisms collect performance 
data used by FEMA regional offices and headquarters for different 
purposes. For example, headquarters focuses on the development of 
future program priorities and reporting progress toward the National 
Preparedness Goal, while regions use program information to monitor 
primary grant recipients. The Department of Homeland Security (DHS), 
of which FEMA is a part, developed agency priority goals that reflect 
agency-wide, near-term priorities. According to FEMA officials, the 
EMPG and AFG programs have an indirect link to a DHS agency priority 
goal, as well as the National Preparedness Goal, because they support 
states’ level of preparedness for disasters. According to FEMA 
officials, neither program has a standardized tool with which to 
validate the performance data that are self-reported by recipients; 
additionally, the regions are inconsistent in their approaches to 
verifying program performance data. The absence of a formal 
established validation and verification procedure, as directed by 
Circular No. A-11, could lead to the collection of erroneous 
performance data. 

What GAO Recommends: 

GAO recommends that FEMA ensure, in accordance with OMB Circular No. 
A-11, that there are consistent procedures in place at the program 
office and regional level to validate and verify grant performance 
data that allow FEMA to attest to the reliability of EMPG and AFG 
grant data used to report progress toward goals. DHS, of which FEMA is 
a part, concurred with this recommendation. 

View [hyperlink, http://www.gao.gov/products/GAO-13-552]. For more 
information, contact Stanley J. Czerwinski at (202) 512-6806 or 
czerwinskis@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

OMB Provides Agencies with Guidance on Reporting Progress toward Goals 
through Its Circular and Resource Management Officers: 

Selected Justice and FEMA Grant Programs Have Performance Data 
Collection Systems that Link Data to Goals, but FEMA Programs Lack 
Verification and Measurement Capabilities: 

Conclusions: 

Recommendation for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: The National Preparedness Goal Mission Areas and Core 
Capabilities: 

Core Capabilities: 

Appendix III: Comments from the Department of Homeland Security: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Core Capabilities by Mission Area: 

Figures: 

Figure 1: Upward Flow of Programmatic and Financial Performance Data 
for the Justice Assistance Grants and the Juvenile Accountability 
Block Grants: 

Figure 2: Upward Flow of Programmatic and Financial Performance Data 
for the Emergency Management Performance Grants (EMPG): 

Figure 3: Upward Flow of Programmatic and Financial Performance Data 
for the Assistance to Firefighters Grants (AFG): 

Abbreviations: 

AFG: Assistance to Firefighters Grants: 

BJA: Bureau of Justice Assistance: 

BSIR: Biannual Strategy Implementation Report: 

DCTAT: Data Collection and Technical Assistance Tool: 

DHS: Department of Homeland Security: 

DOJ: Department of Justice: 

EMPG: Emergency Management Performance Grants: 

FEMA: Federal Emergency Management Agency: 

FFR: Federal Financial Report: 

GPD: Grant Programs Directorate: 

GPRA: Government Performance and Results Act: 

GPRAMA: Government Performance and Results Act Modernization Act: 

JAG: Justice Assistance Grant: 

JABG: Juvenile Accountability Block Grants: 

JMD: Justice Management Division: 

NAPA: National Academy of Public Administration: 

NFIRS: National Fire Incident Reporting System: 

NPAD: National Preparedness Assessment Division: 

NPG: National Preparedness Goal: 

OJJDP: Office of Juvenile Justice and Delinquency Prevention: 

OMB: Office of Management and Budget: 

PIO: Performance Improvement Officer: 

PMT: Performance Measurement Tool: 

PPR: Semi-Annual Programmatic Performance Report: 

QPPR: Quarterly Performance Progress Reports: 

SPR: State Preparedness Report: 

THIRA: Threat and Hazard Identification and Risk Assessment: 

[End of section] 

GAO:
United States Government Accountability Office: 
441 G St. N.W. 
Washington, DC 20548: 

June 24, 2013: 

The Honorable Thomas R. Carper: 
Chairman: 
Committee on Homeland Security and Governmental Affairs: 
United States Senate: 

The Honorable Tom Coburn, M.D.
Ranking Member: 
Committee on Homeland Security and Governmental Affairs: 
United States Senate: 

The Honorable Claire McCaskill: 
Chairman: 
Subcommittee on Financial and Contracting Oversight: 
Committee on Homeland Security and Governmental Affairs: 
United States Senate: 

Grants are a tool agencies use at the direction of Congress to address 
national priorities, carry out their missions, and achieve program 
goals. While federal grants to state and local governments decreased 
in federal fiscal year 2012 to about $545 billion from a high of $608 
billion in fiscal year 2010 as funds from the American Recovery and 
Reinvestment Act of 2009 were spent, grant spending is and will 
continue to be of critical importance.[Footnote 1] As with any federal 
spending, tracking the performance of grant recipients could be 
helpful in ensuring transparency and accountability of federal agency 
operations, programs, and achievements to Congress as well as to 
taxpayers. Our past work has identified that opportunities exist to 
improve the design and implementation of federal grants, and that two 
issues with regard to grant performance reporting--lack of appropriate 
performance measures, and data collection and validation--present 
challenges.[Footnote 2] The level of federal spending on grant 
programs combined with the federal government's continued long-term 
fiscal challenges and constrained budgets only increases the 
importance of accurately measuring the performance of grant programs 
to ensure they contribute to federal and agency goals. 

You asked us to review grant performance reporting. This report 
examines (1) the extent to which the Office of Management and Budget 
(OMB) provides guidance to agencies on how to collect, validate, and 
report performance information that demonstrates progress toward 
agency-wide and federal goals; and (2) how selected agencies collect 
and validate information on the performance of selected grant programs 
and use that information to report progress toward their agency-wide 
performance goals. For this review, we selected two grant programs at 
the Department of Justice (Justice), the Edward Byrne Memorial Justice 
Assistance Grant (JAG) program and the Juvenile Accountability Block 
Grants (JABG) program, and two at the Federal Emergency Management 
Agency (FEMA), which is part of the Department of Homeland Security 
(DHS), the Emergency Management Performance Grants (EMPG) program and 
the Assistance to Firefighters Grants (AFG). The two Justice programs 
allowed us to review formula grant programs with diverse obligation 
amounts directed to states.[Footnote 3] The FEMA programs allowed us 
to review one formula grant program directed to states and one project 
grant program directed to local fire departments and emergency medical 
services organizations.[Footnote 4] These programs were selected to 
provide a variety of grants in terms of grant size (obligation 
amounts), grant type (project or formula), grantee type (state or 
local government), and links to agency goals. We selected agencies and 
programs that leveraged and built upon our prior knowledge of selected 
grant programs and related performance reporting.[Footnote 5] 

To determine the extent to which OMB provides guidance to agencies on 
collecting, validating, and reporting performance information, we 
reviewed OMB guidance on grant performance for the executive agency 
level. We also reviewed the Government Performance and Results Act 
(GPRA) of 1993,[Footnote 6] as updated by the GPRA Modernization Act 
(GPRAMA) of 2010,[Footnote 7] and hearings related to GPRAMA, along 
with agency goals established through GPRAMA to determine any 
relationship between selected grant programs and relevant goals. We 
interviewed officials at OMB responsible for developing and 
disseminating grant performance reporting guidance to executive 
agencies to determine the type and extent of guidance offered to 
agencies on grant performance data collection, validation, and 
reporting. 

To determine how the selected agencies collect and use data on the 
performance of their grant recipients, we reviewed key agency 
documents on the selected grant programs, such as grant solicitations 
and performance requirements, examples of related performance 
measures, and internal guidance. We interviewed officials at FEMA and 
Justice to determine the type and usefulness of guidance received for 
measuring grant performance and how these officials incorporated this 
guidance into measuring progress toward government-wide and agency 
goals. To identify performance management issues raised, we conducted 
a literature review. This included our previous reports on the 
selected grant programs at Justice and FEMA and reports by their 
Offices of Inspectors General, reports by the Congressional Research 
Service, and the Congressional Budget Office.[Footnote 8] For a more 
detailed version of our objectives, scope, and methodology, see 
appendix I. 

We conducted this performance audit from May 2012 to June 2013 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Background: 

Since the enactment of GPRA in 1993, the federal government has been 
moving toward a more results-oriented performance culture to improve 
its effectiveness and service delivery, as well as internal 
management. We have long reported on the challenges agencies face in 
creating a focus on results, including the challenge of developing 
meaningful performance measures and collecting reliable performance 
information. In 2010, GPRAMA established a new framework aimed at 
taking a more crosscutting and integrated approach to focusing on 
results and improving government performance. GPRAMA requires that 
executive branch agencies designate agency-level priority goals 
beginning with the fiscal year 2013 annual performance plans and 
report progress toward those goals, including goals measured using 
grant performance data. Accurate and reliable measurement of grant 
performance forms the basis for managing programs and decision making, 
evaluating the effectiveness of grant programs, gauging progress 
toward the achievement of broader agency-wide goals, and reporting on 
progress toward agency priority goals. We have also reported on the 
importance of measuring performance, communicating performance 
information frequently and effectively, and using performance 
information as a basis for decision making.[Footnote 9] 

GPRA required agencies to develop performance plans that establish 
performance goals and to compare actual performance achieved with 
those goals. GPRAMA emphasizes the need for credible performance 
information by requiring that agencies provide additional information 
about how they will ensure the accuracy and reliability of the data 
they use to report progress towards performance goals. Specifically, 
agencies are required to explain in their performance plans and 
reports the means used to verify and validate measured values; the 
sources of the data; the level of accuracy required for the intended 
use of the data; any limitations to the data at the required level of 
accuracy; and how the agency has compensated for such limitations if 
needed to reach the required level of accuracy. OMB's Circular No. A-
11 provides guidance to executive branch agencies on establishing 
standards for performance data validation and verification to ensure 
the completeness and reliability of all performance measurement data 
contained in reports.[Footnote 10] 

Grant performance may be defined and measured in a variety of ways, 
which include financial performance, programmatic performance, and 
administrative performance. 

* Financial performance pertains to disclosing grant expenditures 
related to federally-sponsored projects in accordance with reporting 
requirements. 

* Programmatic performance includes both grant outputs (that is, 
direct results of the grant, such as individuals served or the number 
of programs implemented) and outcomes (that is, long-term results such 
as behavioral changes). 

* Administrative performance includes measurement of how well program 
offices manage their operations and interactions with customers. 
Performance management information can inform key management decisions 
such as setting program priorities, allocating resources, identifying 
program problems, taking corrective action to solve those problems, 
and helping determine progress in meeting the goals of programs or 
operations. 

The two Justice programs we reviewed were the Edward Byrne Memorial 
Justice Assistance Grant (JAG) program and the Juvenile Accountability 
Block Grants (JABG) program.[Footnote 11] Both are formula grant 
programs directed to states. In fiscal year 2012, according to 
Justice, the amount obligated for the JAG program was $288 million and 
for the JABG program it was $22 million.[Footnote 12] JAG is the 
primary provider of federal criminal justice funding to state and 
local jurisdictions. The JAG program provides states and units of 
local government with funding necessary to support a range of program 
areas including law enforcement, prosecution and court programs, and 
prevention and education programs. JABG assists states and units of 
local government in their efforts to strengthen their juvenile justice 
systems. Both grant programs require states to select an agency to 
administer these grant funds and may pass funds from the state 
administering agency to units of local government. 

The two FEMA grant programs we selected were the Emergency Management 
Performance Grants (EMPG) program and the Assistance to Firefighters 
Grant (AFG) program.[Footnote 13] The EMPG program makes grants to 
states to assist state, local, and tribal governments prepare for all 
hazards. The program plays an important role in supporting the 
development and sustainment of core capabilities to fulfill the five 
mission areas (prevention, protection, mitigation, response, and 
recovery) of the National Preparedness Goal (NPG).[Footnote 14] The 
AFG program provides federal grants directly to local fire departments 
and emergency medical services organizations that are unaffiliated 
with fire departments to help pay for equipment, training, and other 
firefighter needs.[Footnote 15] In fiscal year 2012, according to 
FEMA, the amount obligated for EMPG was $340 million and for AFG it 
was $321 million. 

Some of our previous reports have raised concerns about grant 
performance reporting at Justice and FEMA. For example, we reported in 
2010 that Justice's JAG program revised its performance measures and 
we recommended that Justice continue to revise JAG performance 
measures and validate the integrity of self-reported performance 
data.[Footnote 16] Justice concurred with this recommendation and 
since then, according to Justice officials, it has revised the JAG 
performance measures in consultation with a group of grant recipients. 
With regard to FEMA, in March 2011, we reported that it was able to 
gather data on the progress, capabilities, and accomplishments of a 
state's preparedness program through State Preparedness Reports (SPR). 
These reports include self-reported data that may be subject to 
interpretation by the reporting organizations in each state and not be 
readily comparable to other states' data. We also reported that, in 
general, FEMA officials said that evaluation efforts they used to 
collect data on national preparedness capabilities were useful for 
their respective purposes, but that the data collected were limited by 
data reliability and measurement issues related to the lack of 
standardization in the collection of data.[Footnote 17] 

FEMA stated it had taken steps to address these reliability 
challenges, including creating a web-based survey tool to standardize 
how it collects state preparedness information. We also reported in 
February 2012 that FEMA was considering how to make better use of 
grant information from four grant programs--the State Homeland 
Security Program, Urban Area Security Initiative, Port Security Grant 
Program, and the Transit Security Grant Program--and we recommended 
that FEMA collect project information with the level of detail needed 
to better position the agency to identify any potential unnecessary 
duplication within and across the four grant programs, weighing any 
additional costs of collecting these data[Footnote 18].: 

DHS concurred with these recommendations and has actions under way to 
address them. 

OMB Provides Agencies with Guidance on Reporting Progress toward Goals 
through Its Circular and Resource Management Officers: 

The Office of Management and Budget plays a central role in guiding 
agencies on grant making and grant performance measurement by 
providing direction for the agency priority goal process.[Footnote 19] 
OMB revised Circular No. A-11 in 2012 to incorporate the requirements 
of GPRAMA to provide detailed guidance to executive branch agencies to 
establish government-wide performance reporting requirements and 
standards for performance data validation and verification to ensure 
the completeness and reliability of all performance measurement data 
contained in annual reports. OMB's Circular No. A-11 states that 
agencies should have cost-effective verification and validation 
techniques in place to ensure the completeness and reliability of all 
performance measurement data contained in their annual performance 
plans and reports as appropriate to the intended use of the data. 
[Footnote 20] OMB Circular No. A-11 directs agencies to have a data 
validation plan for performance reporting and to include an assessment 
by the agency head of the reliability and completeness of the 
performance data they selected to include in the plan and report. The 
description must include how the agency ensures the accuracy and 
reliability of the data used to measure progress toward its 
performance goals.[Footnote 21] 

OMB provides agencies additional guidance through its resource 
management officers (RMO). RMOs are charged with working with a 
portfolio of agencies organized by subject matter. As part of this, 
RMOs offer analysis and evaluation to support government-wide 
initiatives. According to OMB officials, RMOs are central to the 
conversation on performance with their assigned agencies and focus on 
targeting resources to improve program performance. 

Selected Justice and FEMA Grant Programs Have Performance Data 
Collection Systems that Link Data to Goals, but FEMA Programs Lack 
Verification and Measurement Capabilities: 

Justice Collects Grant Performance Information by Purpose Area for 
Selected Grant Programs, which Allows It to Assess Program Output: 

For 2012 to 2013, Justice has established four agency priority goals, 
which are aligned with its strategic goals and objectives as outlined 
in the Justice strategic plan. Justice's four agency priority goals 
are to: (1) better inform the intelligence community; (2) reduce gang 
violence; (3) protect the American people from financial and health 
care fraud; and (4) protect those most in need of help, with a special 
emphasis on child exploitation and civil rights. The Justice 
Management Division (JMD) serves as the focal point for performance 
information across Justice, with respect to both receiving guidance 
from OMB and ultimately sharing Justice's performance with OMB. 
Justice's performance improvement officer (PIO) works within this 
component and represents the agency on the Performance Improvement 
Council.[Footnote 22] According to Justice officials, JMD offers 
performance measure development feedback to all Justice components 
when requested, and the PIO schedules monthly agency-wide conference 
calls related to performance management for all Justice component 
performance representatives. 

As shown in figure 1, for both the Justice Assistance Grant (JAG) and 
Juvenile Accountability Block Grants (JABG), programmatic and 
financial performance data are passed up from the grant program 
offices to the Office of the Chief Financial Officer within the Office 
of Justice Programs and ultimately made available to JMD. 

Figure 1: Upward Flow of Programmatic and Financial Performance Data 
for the Justice Assistance Grants and the Juvenile Accountability 
Block Grants: 

[Refer to PDF for image: illustration] 

Data flow: 

Receipt of Performance Data: 
Primary recipients (states) of the Justice Assistance Grant (JAG) and 
the Juvenile Accountability Block Grant (JABG) enter grant performance 
data directly into electronic data collection systems. 

Flows to: 

Data Validation: 
Grant program offices verify performance data. 

Flows to: 

Review and Analysis: 
Grant program offices use grant performance data for: 
1. Program management decisions; 
2. Providing technical assistance to grantees; 
3. Formulating budget requests. 

Flows to: 

Review and Analysis, DOJ: 
Data may be used by DOJ as needed to satisfy performance data requests 
or report progress toward goals. JAG and JABG do not contribute to 
reporting progress toward agency priority goals. 

Source: GAO analysis of Department of Justice information. 

[End of figure] 

Performance information from the JAG and the JABG programs is not used 
to report progress toward agency priority goals. Based on GPRAMA, 
agencies have discretion to select the data sources to use to report 
progress toward their goals. According to Justice officials, these two 
grant programs were excluded from reporting progress toward Justice's 
agency priority goal of reducing gang violence because JAG does not 
collect data on specific targeted behaviors to reduce gang violence 
and only a small fraction of JABG subgrantees, if any, use grant funds 
for direct service programming related to gang prevention. 

Justice Assistance Grant: Performance Data Collection, Validation, and 
Reporting: 

JAG performance measures are organized by seven overall program 
purpose areas.[Footnote 23] According to Justice officials, the JAG 
performance measures were redesigned beginning in 2011, and were 
developed in cooperation with grant recipients and shared with 
professional criminal justice organizations before being adopted in 
2012. The JAG program collects quantitative performance data on a 
quarterly basis and qualitative narrative data on an annual basis. JAG 
grantees enter quantitative performance data directly into JAG's 
electronic data collection system, the Performance Measurement Tool 
(PMT). For example, for the law enforcement purpose area, the PMT 
collects information on the number and type of programs implemented, 
expanded, or sustained during the reporting period. To help improve 
data reliability, JAG program staff review grantee reports on a 
quarterly basis to evaluate performance information and look for data 
anomalies. The PMT has edit checks built into the system which, along 
with manual review of quarterly reports, help prevent data entry 
errors.[Footnote 24] The edit check prompts grantees if they make 
obvious keystroke errors. For example, grantees cannot enter a grant 
amount for more than the grant award given. The system also notifies 
grantees if they enter a number that is inconsistent with a previous 
response. 

JAG collects qualitative performance data in reports from primary 
grant recipients about their grant-related activities during the past 
year. Grantees create a report from the PMT system and eventually 
upload it to the Grants Management System.[Footnote 25] For example, 
in the law enforcement purpose area, a qualitative measure asks grant 
recipients to describe any accomplishments during the reporting period 
along with any benefits or changes observed as a result of JAG-funded 
activities. The data allow the Bureau of Justice Assistance (BJA) to 
report progress toward its bureau-level goals as identified in its 
strategic plan as well as for the annual JAG report.[Footnote 26] For 
example, Justice officials pointed to the direct link between JAG 
performance measures on drug courts and BJA's second major focus area 
of reduction of recidivism and prevention of unnecessary confinement. 

Juvenile Accountability Block Grants: Performance Data Collection, 
Validation, and Reporting: 

JABG primary grant recipients report data in support of core 
performance measures annually and financial performance quarterly. 
These core measures include the number and percent of programs and 
initiatives employing evidence-based programs and the number of youth 
served by JABG-funded programs. Grant recipients are also required to 
report at least one output or outcome measure for each of the 17 
purpose areas under which they apply for funding.[Footnote 27] JABG 
grant recipients enter grant performance information into the Office 
of Juvenile Justice and Delinquency Prevention's (OJJDP) Data 
Collection and Technical Assistance Tool (DCTAT), an electronic data 
collection system designed to help grantees compile performance data 
and make sure they are in the correct format for submission to OJJDP. 
Justice officials explained that there are some DCTAT system-level 
data validation and verification checks in place to prevent data entry 
errors. In addition, OJJDP reviews the aggregate data findings and 
grantee-level data reports for any obvious errors or inconsistencies. 
The contractor who manages the DCTAT system and the grant managers 
themselves verify and validate the data. Justice officials explained 
that a data validation tool has been piloted for OJJDP's Title II 
Formula Grants Program, but is not yet available OJJDP-wide.[Footnote 
28] However, Justice officials noted that as of February 2013, OJJDP 
established a policy which states the department's intent to implement 
the use of this new data validation and verification tool for future 
use with JABG grantees. Along with quantitative performance data 
entered into the DCTAT, grantees provide narrative descriptions of 
their projects. According to Justice officials, these descriptions 
help JABG grant managers compare descriptions to performance data and 
understand the nuances among different programs. 

FEMA Collects Grant Performance Data through Several Broad 
Preparedness Reporting Mechanisms, but Faces Challenges Identifying 
Outcomes at the Agency Level: 

To support the National Preparedness Goal, FEMA uses the Emergency 
Management Performance Grants (EMPG) and the Assistance to 
Firefighters Grants (AFG) to enhance core capabilities of state 
governments (through EMPG) and fire departments and local emergency 
medical services organizations (through AFG), its primary grant 
recipients (see appendix II).[Footnote 29] Like Justice, the 
Department of Homeland Security developed agency priority goals that 
reflect agency-wide, near-term priorities.[Footnote 30] 

According to FEMA officials, the EMPG and AFG programs have an 
indirect link to an agency priority goal, as well as to the NPG, 
because both programs support states' overall level of disaster 
preparedness. For example, EMPG-supported activities, such as building 
or sustaining capabilities through personnel training, plan 
development, and participation in exercises, may be specifically 
reported in the State Preparedness Reports (SPR). The SPR is one of 
FEMA's evaluation tools used to gather data on the progress, 
capabilities, and accomplishments of states' preparedness programs, 
and these state-reported assessments are linked to the agency priority 
goal for ensuring resilience to disasters. AFG performance data are 
self-reported by fire houses and local emergency medical services 
organizations but may have an impact on NPG core capabilities and 
therefore may be included in each state's SPR. However, identifying 
the specific outputs or impacts of EMPG and AFG from within this broad 
reporting mechanism is not always possible because states do not 
always report on the performance of these grants individually in their 
SPRs.[Footnote 31] A FEMA official noted that FEMA has its own 
performance group and it has access to the DHS PIO and OMB performance 
guidance as needed. 

Emergency Management Performance Grants: Performance Data Collection, 
Validation, and Reporting: 

As shown in figure 2, FEMA collects EMPG grant performance data 
through multiple reporting mechanisms. All EMPG primary grant 
recipients are required to report on progress toward the EMPG program 
priorities. FEMA establishes its administrative and federal financial 
reporting requirements in its grant solicitation. The grant 
solicitation requires applicants to submit a work plan that includes a 
program and budget narrative, personnel data table, training data 
table, exercise data table, and grant activities outline. FEMA has 
developed the Threat and Hazard Identification and Risk Assessment 
(THIRA) and related guidance as a tool to help states identify and 
assess their risks or greatest hazards.[Footnote 32] States, 
territories, and urban areas submit THIRAs to FEMA. States and 
territories submit the SPR to FEMA. The SPR assesses a state's current 
preparedness level against its THIRA targets.[Footnote 33] The SPR and 
the THIRA data assist FEMA in annually summarizing progress across all 
31 preparedness core capabilities. A range of reports--Quarterly 
Performance Progress Reports, Biannual Strategy Implementation 
Reports, and quarterly Federal Financial Reports are submitted by the 
primary recipients to the FEMA regions; however, review of these 
reports varied from region to region.[Footnote 34],[Footnote 35] One 
FEMA regional official said that the FEMA analysts in their region 
review the quarterly financial and program reports for timeliness, 
accuracy, grantee match dollars, and level of funds drawn 
down.[Footnote 36] Another FEMA regional official said that in their 
region, they review the quarterly financial and program reports to 
compare the recipient's application work plan with actual progress 
reported. One FEMA regional official commented that even though it is 
labor intensive to read the programmatic narrative reports, these 
reports help regional officials manage grantees by identifying 
successes and challenges with the grant. 

Figure 2: Upward Flow of Programmatic and Financial Performance Data 
for the Emergency Management Performance Grants (EMPG): 

[Refer to PDF for image: illustration] 

Data flow: 

Collection of Performance Data: 
Primary recipients (states) submit performance data to FEMA regions 
and headquarters via several reporting mechanisms: Federal Financial 
Reports (FFR), Performance Progress Reports (PPR), Biannual Strategy 
Implementation Reports (BSIR), Threat and Hazard Identification and 
Risk Assessments (THIRA), and State Preparedness Reports (SPR). 

Flows to: 

Receipt and Review of Performance Data: 
Grant Programs Directorate (GPD) receives summaries of the cash 
analyses and monitoring reports from the Regions. GPD uses these data 
to provide technical assistance and inform the grant solicitation. 

Receipt and Review of Performance Data: 
National Preparedness Assessment Division (NPAD) reviews the SPRs. 
NPAD compiles and analyzes data to track progress toward goals. 

Receipt and Review of Performance Data: 
FEMA regional offices review the FFR, PPR, and the BSIR, and have 
access to the THIRA and SPR. FEMA headquarters and regional offices 
also review performance data to provide technical assistance and 
program improvements for grantees. 

Flows to: 

Review and Analysis: 
DHS Program Analysis & Evaluation uses performance data from the SPRs 
and THIRAs to report progress toward an agency priority goal 
(preparedness and emergency response). 

National Preparedness Goal: 
EMPG performance data is used to report progress toward the National 
Preparedness Goal in the annual National Preparedness Report. 

Source: GAO analysis of FEMA information. 

[End of figure] 

FEMA officials said that not all states identify the impact of EMPG 
explicitly within their SPR. Officials explained that the SPR provides 
the opportunity to include grant results, and states may use data on 
programs, such as training and exercises, funded through EMPG grants 
in this report. The 2012 EMPG grant solicitation stated that the 
program supports all core capabilities and lists as one of the EMPG 
objectives the development and maintenance of multi-year Training and 
Exercise Plans.[Footnote 37] These trainings and exercises funded by 
EMPG may be reported on and identified in the SPR. For example, one 
state reported in its 2012 SPR that exercises and trainings are 
planned and plans are in place to mobilize hazardous incident teams 
and hazmat mutual aid. The various EMPG reporting mechanisms, 
including the quarterly programmatic and financial reports, are used 
by FEMA units for different purposes such as checking on project cash-
on-hand or for compliance with the work plan. The result is that 
performance data are used to report progress toward an agency priority 
goal as well as the NPG. 

Assistance to Firefighters Grants: Performance Data Collection, 
Validation, and Reporting: 

AFG establishes its performance reporting requirements in the grant 
solicitation. The FEMA regions collect grant performance data 
including obligation and expenditure data through the Semi-Annual 
Programmatic Performance Reports. Additionally, primary recipients 
must report obligations and expenditures semi-annually using the 
Federal Financial Reports. Grant recipients submit performance reports 
through the e-Grants system, and FEMA may withhold future awards and 
fund drawdowns if these reports are delinquent. (See figure 3.) 

Figure 3: Upward Flow of Programmatic and Financial Performance Data 
for the Assistance to Firefighters Grants (AFG): 

[Refer to PDF for image: illustration] 

Data flow: 

Collection of Performance Data: 
Primary recipients, Fire Departments and Emergency Medical Services 
Organizations, submit the Program Performance Reports (PPR) and 
Federal Financial Reports (FFR) to FEMA using the E-Grants system and 
report fire data to the National Fire Incident Reporting System 
(NFIRS). 

Flows to: 

Receipt and Review of Performance Data: 
FEMA/Grant Programs Directorate (GPD) receives the PPR and FFR 
information. GPD’s focus is on the FFRs and administrative performance 
measures. 

Receipt and Review of Performance Data: 
FEMA regional offices use the PPR and FFR data to ensure project 
goals, objectives and performance requirements (program and financial) 
are in accord with the grantee’s work plan, and to provide technical 
assistance to grantees where needed. 

Receipt and Review of Performance Data FEMA/U.S. Fire Administration 
firefighter fatality and injury data from NFIRS. These data are used 
as an outcome measure for AFG. 

Flows to: 

Receipt and Review of Performance Data: 
National Preparedness Assessment Division (NPAD) receives AFG 
performance measure based on NFIRS data. 

Flows to: 

Review and Analysis: 
FEMA (Office of Policy, Program Analysis, and International Affairs -
OPPA) receives performance data based on NFIRS data. AFG does not 
contribute to reporting progress toward agency priority goals. 

National Preparedness Goal: 
AFG performance data may be used to report progress toward the 
National Preparedness Goal in the annual National Preparedness Report. 

Source: GAO analysis of FEMA information. 

[End of figure] 

According to FEMA officials, regional office staff periodically 
monitor AFG recipients to ensure that the project goals, objectives, 
performance requirements, timelines, milestones, budgets, and other 
related program criteria are being met. FEMA regional staff monitor 
the AFG grantees by desk-based review, on-site monitoring visits, or 
both. Monitoring involves the review and analysis of proposed work 
plans compared with actual progress toward financial drawdowns and 
compliance with administrative processes, policies, and activities. 
Monitoring results in identifying areas where technical assistance, 
corrective actions, and other support may be needed. According to FEMA 
officials, the only specific outcome measure for AFG is firefighter 
fatality and injury data. These data are from the National Fire 
Incident Reporting System (NFIRS).[Footnote 38] While FEMA reviews 
program activity, aggregated analysis of AFG specific outputs is not 
readily available. 

FEMA's Reliance on Self-Reported Data and Lack of a Data Verification 
Tool Present Reliability Concerns: 

OMB's Circular No. A-11 directs agencies to have a data validation 
plan for performance reporting that uses verification and validation 
techniques including an assessment by agency heads of reliability and 
completeness of reported performance data. Agencies should describe 
how they are ensuring the accuracy and reliability of the data used to 
measure progress towards performance goals. Circular No. A-11 makes 
reference to two of our reports when defining verification as the 
process of checking or testing performance data to assess for errors 
and validation as ensuring that data are free of systematic error or 
bias and that what is intended to be measured is actually 
measured.[Footnote 39] According to the guidance in Circular No. A-11, 
data limitations can lead to bad decisions resulting in lower 
performance or inaccurate performance assessments, and data 
limitations can include imprecise measurement and recordings, 
incomplete data, and inconsistencies in data collection procedures. In 
addition, we have previously found that maintaining consistent data 
over time requires using consistent data collection procedures for 
ongoing data collections.[Footnote 40] Consistent verification and 
validation of performance data support the general accuracy and 
reliability of performance information, reduce the risk of inaccurate 
performance data, and provide a sufficient level of confidence to the 
Congress and the public that the information presented is credible as 
appropriate to its intended use. 

According to FEMA officials, neither EMPG nor AFG currently has a 
standardized tool or process to validate program data. EMPG and AFG 
recipients provide self-reported performance data to FEMA headquarters 
and the 10 FEMA regions. Some performance reporting reviews, which 
FEMA officials explained may serve as performance information 
verification, are conducted at the regional level. However, the 
regions are inconsistent in their approaches to verifying EMPG and AFG 
performance information from the primary recipients. As previously 
mentioned, we reported in a March 2011 testimony that FEMA officials 
said the evaluation efforts that collected data on national 
preparedness capabilities were useful, but they said the data were 
limited by reliability and measurement issues related to the lack of 
standardization in the collection of the data.[Footnote 41] 

The current absence of a formal established data verification 
procedure, as directed by Circular No. A-11, could lead to the 
collection of erroneous performance data. 

All EMPG primary grantees are required to report on progress toward 
the EMPG program priorities. However, all of the reporting documents 
contain self-reported data on states' general level of preparedness 
with respect to core capabilities that support the NPG. Verification 
and validation of EMPG data take place on a case-by-case and region-by-
region basis. FEMA regional officials told us they carry out desk and 
file reviews. On a quarterly basis, they may compare the application 
work plan with performance reports to track actual versus planned 
progress and they may look at the quarterly financial reports to check 
on grantee drawdown of the grant obligation. Additionally, depending 
on the region, desk reviews may be completed on a 1-or 2-year cycle. 
Other performance reporting mechanisms, such as the Biannual Strategy 
Implementation Reports, may be pulled for review; however, according 
to FEMA regional officials, this is not a consistent practice across 
all regions. Regional officials also said that as part of their 
monitoring responsibilities, they are sometimes present at emergency 
planning exercises and meetings conducted by primary recipients and 
they consider this a type of performance data verification and 
validation. A FEMA official said that, FEMA has conducted two 
nationwide reviews required by Congress that provide the status of 
catastrophic planning in all the states and the largest urban areas, 
as well as support data validation for these efforts. The first review 
took place in 2006, with a subsequent review in 2010; a date has not 
been set for the next review.[Footnote 42] 

FEMA officials said that the AFG grants management reporting system 
does not have a data validation tool and because of the high number of 
recipients, along with numerous data points to be verified on each AFG 
recipient's Semi-Annual Programmatic Performance Report, data 
validation is a challenge.[Footnote 43] Officials also said that AFG 
performance data verification takes place on a case-by-case basis 
(individually) at the FEMA regional level. One FEMA regional official 
reported that in their region, reviews are all site visits--both 
scheduled and unscheduled. According to a FEMA headquarters official, 
FEMA is beginning to apply a risk-based approach to AFG monitoring. 
Specifically, regional analysts will visit grantees who received large 
grant awards, as well as grantees who are implementing region-wide 
projects. 

AFG Lacks a Performance Assessment System with Detailed Outcome 
Measures: 

As discussed above, we found that the only outcome measure currently 
available for AFG is the firefighter fatality and injury data 
available through NFIRS, which provides one measure for the program's 
performance. AFG and NFIRS staff are working to create a new system 
for performance assessment. We have previously reported on the general 
attributes of appropriate grant performance measures, noting that 
measures should be linked to agency goals and missions; be clearly 
stated; include measurable targets; and be objective, reliable, and 
balanced. 

The Fire Grants Reauthorization Act (the act) of 2012, signed into law 
in January 2013, requires FEMA to establish a performance assessment 
system that includes quantifiable performance metrics to evaluate the 
extent to which grants awarded are furthering the purposes of AFG, 
including protecting the health and safety of the public and 
firefighting personnel.[Footnote 44] The act also requires that the 
Administrator of FEMA submit to the Committee on Homeland Security and 
Governmental Affairs, the Committee on Science and Technology, and the 
Committee on Transportation and Infrastructure a report that provides 
information on the performance assessment system developed, using the 
performance metrics developed, and an evaluation of the effectiveness 
of the grants awarded by September 30, 2013 and each year thereafter 
through 2017. If the system, which FEMA is currently developing, 
incorporates Circular No. A-11's requirements for data validation and 
verification, it should enable FEMA to identify the specific impacts 
and outcomes related to AFG. 

The act's requirements are consistent with issues raised in a 2007 
report published by a National Academy of Public Administration (NAPA) 
panel and that are still present today.[Footnote 45] The NAPA panel 
was formed and its report issued at the request of DHS and AFG. The 
report assesses AFG's strategic position and offers suggestions on how 
FEMA might plan and manage based on performance. NAPA's report was an 
effort to help identify potential new strategic directions for AFG and 
to provide it with advice on how it could most effectively plan, 
manage, and measure its accomplishments.[Footnote 46] The NAPA report 
concluded, among other things, that AFG program metrics focused on 
inputs, financial management, and characteristics of the grant 
recipients rather than on accomplishing long-term goals or achieving 
intermediate improvements in public safety. NAPA provided an extensive 
analysis and made several recommendations, noting that there was no 
prescribed way for AFG grantees to analyze or report on whether they 
had realized the expected outcomes for the use of the grant funds. The 
report also included metrics that were proposed during stakeholder 
interviews and focus groups and recommended by the panel for 
consideration by AFG. 

Conclusions: 

Effectively tracking and reporting on the performance of grants is 
critical for agencies to determine whether their program goals are 
being achieved and to ensure transparency and accountability for 
federal grant spending. Accordingly, GPRAMA requires that federal 
agencies collect and validate performance data used to report progress 
toward performance goals. OMB Circular No. A-11 provides detailed 
guidance for the agency goal management process, as well as the 
related performance data verification and validation process which 
supports both reporting progress toward goals and data reliability 
efforts in general. We found that Justice and FEMA program offices 
collect performance data through various mechanisms for the selected 
grant programs. However, grant performance data for FEMA's EMPG and 
AFG programs are self-reported by states, fire houses, or emergency 
medical services organizations as part of their overall preparedness 
assessment reports, and FEMA regional officials have varied and 
inconsistent approaches to data verification for these programs. The 
lack of a consistent approach to verifying data could lead to the 
collection of erroneous performance data. Additionally, while FEMA is 
collecting AFG performance data, it does not have a performance 
measurement system with quantifiable performance metrics linked to 
agency goals, clearly stated measurable targets, and reliable data. 
Such a system, which is being developed by FEMA, would help it to 
analyze or report on whether expected outcomes have been realized for 
the use of the grant funds. 

Recommendation for Executive Action: 

To improve the collection and validation of performance data at FEMA, 
we recommend that the Administrator of FEMA take the following action: 

* Ensure, in accordance with the guidance in OMB Circular No. A-11, 
that there are consistent procedures in place at the program office 
and regional level to promote verification and validation of grant 
performance data that allow the agency to attest to the reliability of 
EMPG and AFG grant data used for reporting progress toward goals. 

Agency Comments and Our Evaluation: 

We provided a draft of this report for comment to the Assistant 
Attorney General for Administration, Department of Justice, the 
Director of the Office of Management and Budget, and the Secretary of 
the Department of Homeland Security. Justice had no technical or 
agency comments on our report. OMB provided technical comments which 
we incorporated as appropriate. DHS provided comments in a letter 
signed by the Director of the Departmental GAO-OIG Liaison Office, 
which we have reprinted in Appendix III, and in technical comments, 
which were incorporated into the report as appropriate. DHS concurred 
with our recommendation and stated that FEMA would explore effective 
and affordable ways to verify and validate grant performance data for 
EMPG and AFG. 

We are sending copies of this report to the interested congressional 
committees, the Assistant Attorney General for Administration at 
Justice, the Director of OMB, the Secretary of Homeland Security and 
other interested parties. In addition, the report is available at no 
charge on the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-6806 or czerwinskis@gao.gov. Contact points 
for our Offices of Congressional Relations and Public Affairs may be 
found on the last page of this report. Major contributors to this 
report are listed in appendix IV. 

Signed by: 

Stanley J. Czerwinski: 
Director, Strategic Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

This report examines (1) the extent to which the Office of Management 
and Budget (OMB) provides guidance to agencies on how to collect, 
validate, and report performance information that demonstrates 
progress toward agency-wide and federal goals; and (2) how selected 
agencies collect and validate information on the performance of 
selected grant programs and use that information to report progress 
toward their agency-wide performance goals. 

We interviewed officials at OMB's Office of Performance and Personnel 
Management and the Office of Federal Financial Management, who were 
responsible for developing and disseminating guidance to executive 
agencies on grant performance information collection and validation to 
determine the type and extent of guidance offered by OMB to agencies 
when developing grant performance measurement systems. To determine 
which agencies and grant programs to review, we researched programs 
and agencies that provided a variety of grants in terms of grant size 
(obligation amounts), type (project/formula; categorical/block), 
grantee type (state or local government), and links to agency goals. 
We selected agencies and programs that leveraged and built upon our 
prior knowledge of selected grant programs and related performance 
reporting. We interviewed the officials at the Department of Homeland 
Security (DHS) and Federal Emergency Management Agency (FEMA) who were 
responsible for the Emergency Management Performance Grants and 
Assistance to Firefighters Grants program administration, as well as 
the DHS component that receives FEMA grant performance data to 
understand how grant performance data are collected, validated and 
used to report progress toward goals and manage programs. We also 
interviewed officials from the FEMA regions which allowed us to 
corroborate information and understand program management and data 
validation procedures for the related grant programs. We also 
interviewed the officials from the Department of Justice who manage 
the Justice Assistance Grant and Juvenile Accountability Block Grants 
programs, as well as Justice officials from the Office of Justice 
Programs and the Justice Management Division to better understand 
collection, validation, and use of grant performance data to report 
progress toward goals and manage programs. 

We conducted a literature review to identify performance management 
issues raised, including those in our previous reports, on the 
selected grant programs at Justice and FEMA or by their Offices of 
Inspectors General, the Congressional Research Service, or the 
Congressional Budget Office. We also reviewed the Government 
Performance and Results Act (GPRA) of 1993,[Footnote 47] as updated by 
the GPRA Modernization Act (GPRAMA) of 2010,[Footnote 48] and hearings 
related to GPRAMA, along with relevant federal goals as well as agency 
goals established through GPRAMA to determine any link to selected 
grant programs. Additionally, we reviewed government websites that 
report on government-wide performance, grant obligations, and grant 
types to identify relevant federal government-wide and agency goals 
established through GPRAMA and to select agencies and programs that 
leveraged and built on our prior knowledge of selected grant programs 
and related performance reporting. We reviewed relevant Justice and 
FEMA websites and grant solicitations and related agency documents to 
understand grant performance measurement requirements for the selected 
grant programs. 

We conducted this performance audit from May 2012 to June 2013 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: The National Preparedness Goal Mission Areas and Core 
Capabilities: 

Presidential Policy Directive 8: National Preparedness (PPD-8) 
describes the nation's approach to preparing for the threats and 
hazards that pose the greatest risk to the security of the United 
States. The nation's security and resilience posture is described 
through the core capabilities comprised by the five mission areas. 

Core Capabilities: 

Core capabilities are essential for the execution of each of the five 
mission areas (see below). To assess both the nation's capacity and 
gaps, each core capability includes capability targets for which 
measures will be developed. The core capabilities and capability 
targets are not exclusive to any single level of government or 
organization, but rather require the combined efforts of the whole 
community. 

Table 1: Core Capabilities by Mission Area: 

Core capabilities common to all mission areas: 
* Planning; 
* Public Information and Warning; 
* Operational Coordination. 

Mission Area: Prevention: 
* Forensics and Attribution; 
* Intelligence and Information Sharing[A]; 
* Interdiction and Disruption[A]; 
* Screening, Search, and Detection[A]. 

Mission Area: Protection: 
* Access Control and Identity Verification; 
* Cybersecurity; 
* Intelligence and Information Sharing[A]; 
* Interdiction and Disruption[A]; 
* Screening, Search, and Detection[A]; 
* Physical Protective Measures; 
* Risk Management for Protection Programs and Activities; 
* Supply Chain Integrity and Security. 

Mission Area: Mitigation: 
* Community Resilience; 
* Long-term Vulnerability Reduction; 
* Risk and Disaster Resilience Assessment; 
* Threats and Hazard Identification. 

Mission Area: Response: 
* Critical Transportation; 
* Environmental Response/Health and Safety; 
* Fatality Management Services; 
* Infrastructure Systems[B]; 
* Mass Care Services; 
* Mass Search and Rescue Operations; 
* On-scene Security and Protection; 
* Operational Communications; 
* Public and Private Services and Resources; 
* Public Health and Medical Services; 
* Situational Assessment. 

Mission Area: Recovery: 
* Economic Recovery; 
* Health and Social Services; 
* Housing; 
* Infrastructure Systems[B]; 
* Natural and Cultural Resources. 

Source: National Preparedness Report, March 30, 2012. 

Note: Planning, Public Information and Warning, and Operational 
Coordination are core capabilities common to all mission areas. 

[A] Intelligence and Information Sharing, Interdiction and Disruption, 
and Screening, Search, and Detection are core capabilities for both 
Prevention and Protection: 

[B] Infrastructure Systems is a core capability for both the Response 
and Recovery Mission Areas. 

[End of table] 

[End of section] 

Appendix III: Comments from the Department of Homeland Security: 

Homeland Security: 

June 5, 2013: 

Stanley J. Czerwinski: 
Director, Strategic Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Re: Draft Report GA0-13-552, "Grant Performance: Justice and FEMA
Collect Performance Data for Selected Grants, but Action Needed to 
Validate FEMA Performance Data" 

Dear Mr. Czerwinski: 

Thank you for the opportunity to review and comment on this draft 
report. The U.S. Department of Homeland Security (DHS) appreciates the 
U.S. Government Accountability Office's (GAO's) work in planning and 
conducting its review and issuing this report. 

The Department is pleased to note GAO's recognition that the Federal 
Emergency Management Agency (FEMA) has actions underway to improve the 
collection, analysis, and use of grant performance information to 
better identify any potential unnecessary duplication within and 
across programs. 

The draft report contained one recommendation, with which the 
Department concurs. Specifically, GAO recommended that the FEMA 
Administrator: 

Recommendation: Ensure, in accordance with the guidance in OMB 
Circular No. A-11, that there are consistent procedures in place at 
the program office and regional level to promote verification and 
validation of grant performance data that allows the agency to attest 
to the reliability of EMPG and AFG grant data used for reporting 
progress toward goals. 

Response: Concur. The FEMA Grant Programs Directorate and National 
Preparedness Assessment Division will explore effective and affordable 
ways to verify and validate grant performance data in accordance with 
OMB Circular A-11 Section 260.9, such as ensuring that supporting 
documentation for existing self-reported performance measures is 
maintained and readily available for the EMPG program. For the AFG 
program, the National Defense Authorization Act for Fiscal Year 2013 
(NDAA) (PL 112-239) amended Section 33 of the Federal Fire Prevention 
and Control Act of 1974 (15 U.S.C. § 2229), to reauthorize the AFG 
program. One of the key provisions in the NDAA was to require the 
establishment of a metrics based performance assessment system to 
evaluate the program. 

When establishing this performance assessment system, FEMA will 
explore ways to incorporate verification and validation of self-
reported data. Estimated Completion Date: September 30, 2014.
Again, thank you for the opportunity to review and comment on this 
draft report. Technical comments were previously provided under 
separate cover. Please feel free to contact me if you have any 
questions. We look forward to working with you in the future. 

Sincerely, 

Signed by: 

Jim H. Crumpacker: 
Director: 
Departmental GAO-01G Liaison Office: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Stanley J. Czerwinski, (202) 512-6806 or czerwinskis@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Thomas M. James, Assistant 
Director, Keith O'Brien, analyst-in-charge, and Laurel Beedon made 
major contributions to this report. David Mauer, Joy Booth, Elizabeth 
Curda, Chris Keisling, and Sarah Veale provided key guidance 
throughout the engagement. Other key contributors included Cynthia 
Saunders, Amy Bowser, Robert Gebhart, Kim McGatlin, Beryl Davis, and 
Donna Miller. 

[End of section] 

Footnotes: 

[1] Pub.L. No. 111-5, 123 Stat. 115 (Feb. 17, 2009). 

[2] GAO, Grants to State and Local Governments: An Overview of Federal 
Funding Levels and Selected Challenges, [hyperlink, 
http://www.gao.gov/products/GAO-12-1016] (Washington, D.C.: Sept. 25, 
2012); and Grants Management: Enhancing Performance Accountability 
Provisions Could Lead to Better Results, [hyperlink, 
http://www.gao.gov/products/GAO-06-1046] (Washington, D.C.: Sept. 29, 
2006). 

[3] A formula grant is allocated to eligible recipients based on 
formula factors, such as state population, established in statutory 
and regulatory provisions. 

[4] A project grant provides funding for fixed or known periods of 
specific projects. 

[5] Findings included in this report are not generalizable beyond the 
four grant programs selected for this review. 

[6] Pub. L. No. 103-62, 107 Stat. 285 (Aug. 3, 1993). 

[7] Pub. L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011). 

[8] GAO reports include, National Preparedness: FEMA Has Made Progress 
in Improving Grant Management and Assessing Capabilities, but 
Challenges Remain, [hyperlink, 
http://www.gao.gov/products/GAO-13-456T] (Washington, D.C.: Mar. 19, 
2013); Managing Preparedness Grants and Assessing National 
Capabilities: Continuing Challenges Impede FEMA's Progress, 
[hyperlink, http://www.gao.gov/products/GAO-12-526T] (Washington, 
D.C.: Mar. 20, 2012); Homeland Security: DHS Needs Better Project 
Information and Coordination among Four Overlapping Grant Programs, 
[hyperlink, http://www.gao.gov/products/GAO-12-303] (Washington, D.C.: 
Feb. 28, 2012); Measuring Disaster Preparedness: FEMA Has Made Limited 
Progress in Assessing National Capabilities, [hyperlink, 
http://www.gao.gov/products/GAO-11-260T] (Washington, D.C.: Mar. 17, 
2011); and Recovery Act: Department of Justice Could Better Assess 
Justice Assistance Grant Program Impact, [hyperlink, 
http://www.gao.gov/products/GAO-11-87] (Washington, D.C.: Oct. 15, 
2010). 

[9] GAO, Government Performance: Lessons Learned for the Next 
Administration on Using Performance Information to Improve Results, 
[hyperlink, http://www.gao.gov/products/GAO-08-1026T] (Washington, 
D.C.: July 24, 2008); Managing for Results: Enhancing Agency Use of 
Performance Information for Management Decision Making, [hyperlink, 
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9, 
2005); and Executive Guide: Effectively Implementing the Government 
Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 
1996). 

[10] OMB Circular No. A-11, Preparation, Submission, and Execution of 
the Budget (Washington, D.C.: Aug. 3, 2012). 

[11] JAG and JABG awards are provided to all states, the District of 
Columbia, Guam, American Samoa, the Commonwealth of Puerto Rico, the 
Virgin Islands, and the Northern Mariana Islands. 

[12] The federal fiscal year ends September 30 of each year. 

[13] The AFG program comprises three distinct financial assistance 
programs: Assistance to Firefighters Grants (AFG); Staffing for 
Adequate and Emergency Response; Fire Prevention and Safety. The focus 
of this report is the AFG. 

[14] The National Preparedness Report (NPR) released in March 2012 
describes the National Preparedness Goal and its five mission areas. 
The mission areas include 31 core capabilities essential to 
preparedness. The NPR offers an assessment of the level of 
preparedness related to each of these 31 core capabilities. The NPG 
was developed as a part of the Presidential Policy Directive 8 (PPD-
8). (See appendix II.) 

[15] GAO, Fire Grants: FEMA Has Met Most Requirements for Awarding 
Fire Grants, but Additional Actions Would Improve Its Grant Process, 
[hyperlink, http://www.gao.gov/products/GAO-10-64] (Washington, D.C.: 
Oct. 30, 2009). 

[16] [hyperlink, http://www.gao.gov/products/GAO-11-87] 

[17] [hyperlink, http://www.gao.gov/products/GAO-12-526T] and 
[hyperlink, http://www.gao.gov/products/GAO-11-260T] 

[18] [hyperlink, http://www.gao.gov/products/GAO-13-456T] and 
[hyperlink, http://www.gao.gov/products/GAO-12-303] 

[19] Every 2 years GPRAMA requires agency heads in those agencies 
defined under the Chief Financial Officers Act of 1990, or those 
otherwise determined by OMB, to designate a subset of priority goals 
from the performance goals in their agency performance plans. These 
goals should reflect the highest priorities of the agency as 
determined by the agency head. 

[20] Agency Performance Plans establish performance goals to define 
the level of performance to be achieved during the year in which the 
plan is submitted and the next fiscal year. These plans are made 
available on a public website and cover each program activity set 
forth in the agency budget. 

[21] OMB Circular No. A-102, Grants and Cooperative Agreements with 
State and Local Governments (Washington, D.C.: Oct. 7, 1994, further 
amended Aug. 29, 1997); and OMB Circular No. A-110, Uniform 
Administrative Requirements for Grants and Agreements with 
Institutions of Higher Education, Hospitals, and Other Non-Profit 
Organizations (Washington, D.C.: Nov. 19, 1993, further amended Sept. 
30, 1999) both also set forth requirements for federal grant 
performance monitoring. 

[22] GPRAMA established in law the Performance Improvement Council 
(PIC), chaired by OMB's Deputy Director for Management and composed of 
PIOs from various federal agencies. Among other responsibilities, the 
PIC is charged with facilitating the exchange of successful 
performance management practices among agencies and assisting OMB in 
implementing certain GPRAMA requirements. Our April 2013 report 
discusses the role of the PIC in facilitating the exchange of best 
practices and improving agency program management and performance - 
GAO, Managing for Results: Agencies Have Elevated Performance 
Management Leadership Roles but Additional Training Is Needed, 
[hyperlink, http://www.gao.gov/products/GAO-13-356] (Washington, D.C.: 
Apr. 16, 2013). 

[23] JAG purpose areas are: ? law enforcement programs; prosecution 
and court programs; prevention and education programs; corrections and 
community corrections programs; drug treatment and enforcement 
programs; planning, evaluation, and technology improvement programs; 
and crime victim and witness programs (other than compensation). 

[24] According to Justice officials, they refer to the edit checks as 
data validation and verification rules. 

[25] The Grants Management System is a web-based application that 
provides support for the application, award, and management of awards 
at OJP. 

[26] The Bureau of Justice Assistance Strategic Plan for fiscal years 
2013 to 2016 has five major focus areas: reduction of violent crime, 
the improvement of community safety, and support for public safety 
officers; reduction of recidivism and prevention of unnecessary 
confinement; integration of evidence-based, research-driven strategies 
into the day-to-day operations of BJA and the programs BJA administers 
and supports; increasing program effectiveness with a renewed emphasis 
on data analysis, information sharing, and performance management; and 
ensuring organizational excellence. 

[27] In fiscal year 2012, JABG purpose areas were: graduated 
sanctions, facilities, hiring court staff/pretrial services, hiring 
prosecutors, funding for prosecutors, training law enforcement/court 
personnel, gun courts, drug courts, juvenile records, information 
sharing, accountability-based programs, risk/needs assessments, school 
safety, restorative justice, court/probation programming, hiring 
detention/corrections staff, and reentry. 

[28] The Title II Formula Grants Program supports state and local 
efforts to plan, establish, operate, coordinate, and evaluate projects 
directly or through grants and contracts with public and private 
agencies for the development of more effective education, training, 
research, prevention, diversion, treatment, and rehabilitation 
programs in the area of juvenile delinquency and programs to improve 
the juvenile justice system. 

[29] The AFG Fire Department funding priorities for 2012 are: fire 
operations and firefighter safety, and firefighting vehicles. The 2012 
EMPG program objectives include: approved emergency plans; completion 
of Threat and Hazard Identification and Risk Assessment; development 
and maintenance of multi-year Training and Exercise Plans; and 
targeted training and verification of personnel capability. 

[30] For 2012 to 2013, DHS's three agency priority goals are: 
strengthen aviation security counterterrorism capabilities by using 
intelligence-driven information and risk-based decisions; improve the 
efficiency of the process to detain and remove criminal aliens from 
the United States; and ensure resilience to disasters by strengthening 
disaster preparedness and response capabilities. 

[31] Although EMPG and AFG performance measures are not always 
included in states' SPRs, FEMA has developed management activity 
performance measures that assess management practices for the 
preparedness grant programs including EMPG and AFG. For example, these 
measures include the percent of preparedness grant awards closed 
within 90 days and the percent of Federal Financial Reports submitted 
on time. 

[32] FEMA developed the Comprehensive Preparedness Guide 201: Threat 
Hazard Identification and Risk Assessment Guide to provide a 
comprehensive approach for identifying and assessing risks and 
associated impacts. 

[33] Section 652(c) of the Post-Katrina Emergency Management Reform 
Act of 2006 (Pub. L. No. 109-295) requires any state that receives 
federal preparedness assistance to submit an SPR to FEMA. 

[34] The Quarterly Performance Progress Report is submitted to the 
region through the Non-Disaster Grants Management System (ND Grants 
System). 

[35] The Biannual Strategy Implementation Report is a reporting 
requirement submitted by states to FEMA on the progress of certain 
grants. 

[36] The Federal Financial Report (also known as form SF 425) is 
submitted quarterly through the Payment and Reporting System. 

[37] The core capabilities contained in the NPG are highly 
interdependent and require FEMA to use existing preparedness networks 
and activities, improve training and exercise programs, promote 
innovation, and ensure that the administrative, finance, and logistics 
systems are in place to support these capabilities. 

[38] NFIRS is a data base of fire incident information initiated and 
supported by the U.S. Fire Administration. Fire departments report 
their fire incident information to states or directly to NFIRS. 

[39] GAO, Performance Plans: Selected Approaches for Verification and 
Validation of Agency Performance Information, GAO/GGD-99-139 
(Washington, D.C.: July 1999); and The Results Act: An Evaluator's 
Guide to Assessing Agency Annual Performance Plans, GAO/GGD-10.1.20 
(Washington, D.C.: April 1998). 

[40] GAO, Federal Telework: Program Measurement Continues to Confront 
Data Reliability Issues, [hyperlink, 
http://www.gao.gov/products/GAO-12-519] (Washington, D.C.: Apr. 19, 
2012). 

[41] [hyperlink, http://www.gao.gov/products/GAO-11-260T].  

[42] Department of Homeland Security, Nationwide Plan Review: Phase I 
Report (Washington, D.C.: 2006); and Department of Homeland Security, 
Nationwide Plan Review: Fiscal Year 2010 Report to Congress 
(Washington, D.C.: 2010). 

[43] According to FEMA officials, AFG has more than 3,000 primary 
recipients a year. 

[44] Pub. L. No. 112-239, Div. A, Title XVIII, Subtitle A, § 1803, 126 
Stat. 1632, 2100 (Jan. 2, 2013) (15 U.S.C. § 2229(p)(2)). 

[45] NAPA is a nonprofit, independent coalition of public management 
and organizational leaders. 

[46] James Kunde, Paul D. Brooks, Glenn Corbett, Harry Hatry, Bruce D. 
McDowell, Darrell W. Stephens, Assistance to Firefighters Grant 
Program: Assessing Performance (Washington, D.C.: NAPA 2007). 

[47] Pub. L. No. 103-62, 107 Stat. 285 (Aug. 3, 1993). 

[48] Pub. L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011). 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]