This is the accessible text file for GAO report number GAO-09-273
entitled 'Aviation Security: Federal Air Marshal Service Has Taken
Actions to Fulfill Its Core Mission and Address Workforce Issues, but
Additional Actions Are Needed to Improve Workforce Survey' which was
released on February 13, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Requesters:
United States Government Accountability Office:
GAO:
January 2009:
Aviation Security:
Federal Air Marshal Service Has Taken Actions to Fulfill Its Core
Mission and Address Workforce Issues, but Additional Actions Are Needed
to Improve Workforce Survey:
GAO-09-273:
GAO Highlights:
Highlights of GAO-09-273, a report to congressional requesters.
Why GAO Did This Study:
By deploying armed air marshals onboard selected flights, the Federal
Air Marshal Service (FAMS), a component of the Transportation Security
Administration (TSA), plays a key role in helping to protect
approximately 29,000 domestic and international flights operated daily
by U.S. air carriers. GAO was asked to examine (1) FAMS’s operational
approach or “concept of operations” for covering flights, (2) to what
extent this operational approach has been independently evaluated, and
(3) the processes and initiatives FAMS established to address workforce-
related issues. GAO analyzed documented policies and procedures
regarding FAMS’s operational approach and a July 2006 classified report
based on an independent evaluation of that approach. Also, GAO analyzed
employee working group reports and other documentation of FAMS’s
processes and initiatives for addressing workforce-related issues, and
interviewed the FAMS Director, other senior officials, and 67 air
marshals (selected to reflect a range in levels of experience). This
report is the public version of a restricted report (GAO-09-53SU)
issued in December 2008.
What GAO Found:
Because the number of air marshals is less than the number of daily
flights, FAMS’s operational approach is to assign air marshals to
selected flights it deems high risk—such as the nonstop, long-distance
flights targeted on September 11, 2001. In assigning air marshals, FAMS
seeks to maximize coverage of flights in 10 targeted high-risk
categories, which are based on consideration of threats,
vulnerabilities, and consequences.
In July 2006, the Homeland Security Institute, a federally funded
research and development center, independently assessed FAMS’s
operational approach and found it to be reasonable. However, the
institute noted that certain types of flights were covered less often
than others. The institute recommended that FAMS increase randomness or
unpredictability in selecting flights and otherwise diversify the
coverage of flights within the various risk categories. As of October
2008, FAMS had taken actions (or had ongoing efforts) to implement the
Homeland Security Institute’s recommendations. GAO found the
institute’s evaluation methodology to be reasonable.
To address workforce-related issues, FAMS’s previous director, who
served until June 2008, established a number of processes and
initiatives—such as working groups, listening sessions, and an internal
Web site—for agency personnel to provide anonymous feedback to
management on any topic. These efforts have produced some positive
results. For example, FAMS revised its policy for airport check-in and
aircraft boarding procedures to help protect the anonymity of air
marshals in mission status, and FAMS adjusted its flight scheduling
process for air marshals to support a better work-life balance. The air
marshals GAO interviewed expressed satisfaction with FAMS efforts to
address workforce-related issues. Further, the current FAMS Director,
after being designated in June 2008 to head the agency, issued a
broadcast message to all employees, expressing a commitment to continue
applicable processes and initiatives. Also, FAMS has plans to conduct a
workforce satisfaction survey of all employees every 2 years, building
upon an initial survey conducted in fiscal year 2007. Although the 2007
survey indicated positive changes since the prior year, it was answered
by 46 percent of the workforce, well short of the 80-percent response
rate that the Office of Management and Budget (OMB) encourages for
ensuring that results reflect the views of the target population. OMB
guidance gives steps, such as extending the cut-off date for
responding, that could improve the response rate of future surveys.
Also, several of the 2007 survey questions were ambiguous, and response
options were limited. Addressing these design considerations could
enhance future survey results.
What GAO Recommends:
To improve future workforce satisfaction surveys, GAO recommends that
the FAMS Director take actions to ensure that (1) survey questions and
response options are clearly worded and structured and (2) the response
rate is as high as possible. TSA agreed.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-273]. For more
information, contact Steve Lord at (202) 512-4379 or lords@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
The Federal Air Marshal Service's Operational Approach to Achieving Its
Core Mission Is Based on Risk-Related Factors:
An Independent Assessment Concluded That the Federal Air Marshal
Service's Approach for Achieving Its Core Mission Was Reasonable;
Recommendations for Enhancing the Approach Are Being Implemented:
The Federal Air Marshal Service Has Taken Positive Actions to Address
Issues Affecting Its Workforce and to Help Ensure Continued Progress:
Conclusions:
Recommendation for Executive Action:
Agency Comments:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: OMB's Program Assessment Rating Tool Results in 2003 and
2008 for the Federal Air Marshal Service:
Appendix III: Excerpts from the Homeland Security Institute's July 2006
Report:
Appendix IV: Performance Measures for Assessing the Federal Air Marshal
Service:
Appendix V: Working Groups Formed by the Federal Air Marshal Service to
Address Various Issues:
Appendix VI: FAMS Workforce Satisfaction Survey:
Appendix VII: Comments from the Department of Homeland Security:
Appendix VIII: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Homeland Security Institute's Recommendations (July 2006
Report) to Improve the Federal Air Marshal Service's Approach for
Achieving Its Core Mission:
Table 2: FAMS 2007 Workforce Satisfaction Survey Questions That
Combined Two or More Issues:
Table 3: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Purpose and Design:
Table 4: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Strategic Planning:
Table 5: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Management:
Table 6: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Results/Accountability:
Table 7: Overview of the Federal Air Marshal Service's Key Performance
Measures (as of Fiscal Year 2008):
Table 8: Overview of Federal Air Marshal Service Working Groups and
Their Purposes:
Figures:
Figure 1: Federal Air Marshals Are Trained for a Unique Working
Environment:
Abbreviations:
CONOPS: Concept of Operations:
DHS: Department of Homeland Security:
FAMS: Federal Air Marshal Service:
ICE: U.S. Immigration and Customs Enforcement:
OMB: Office of Management and Budget:
PART: Program Assessment Rating Tool:
PDA: personal digital assistant:
SAC: special agent-in-charge:
TSA: Transportation Security Administration:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
January 14, 2009:
Congressional Requesters:
With a shared responsibility for protecting the approximately 29,000
domestic and international flights operated daily by U.S. commercial
passenger air carriers, the Federal Air Marshal Service (FAMS) is a key
component of the Transportation Security Administration's (TSA) layered
approach for aviation security.[Footnote 1] FAMS's overall mission is
to help secure the nation's aviation system by detecting, deterring,
and defeating hostile acts targeting U.S. air carriers, airports,
passengers, and crews. In particular, FAMS's core mission is to deploy
trained and armed federal air marshals--a workforce generally
characterized as the nation's most mobile law enforcement officers--to
provide an on-board security presence on selected flights.[Footnote 2]
In response to the September 11, 2001, terrorist attacks and pursuant
to the Aviation and Transportation Security Act, FAMS's mission was
expanded to provide coverage of domestic as well as international
flights, with priority given to nonstop, long-distance flights such as
those targeted on September 11.[Footnote 3] As a result, the agency's
cadre of air marshals grew significantly. [Footnote 4] Nonetheless,
because the total number of air marshals is less than the approximately
29,000 domestic and international flights operated daily by U.S.
commercial passenger air carriers, FAMS routinely must determine which
flights are to be provided an on-board security presence. To facilitate
making these decisions, FAMS developed an operational approach--
commonly referred to as the agency's concept of operations--for
deploying air marshals on selected flights.
In addition to flight-coverage decisions, FAMS also faces challenges in
addressing various operational and quality-of-life issues that affect
the ability of air marshals to carry out the agency's mission--issues
that have been noted by our prior work and also have received extensive
media coverage.[Footnote 5] Such issues range, for example, from
maintaining anonymity during aircraft boarding procedures to mitigating
the various health concerns associated with frequent flying.
You requested that GAO evaluate FAMS's concept of operations and
workforce-related issues. This report answers the following questions:
* What is FAMS's operational approach for achieving its core mission of
providing an onboard security presence for flights operated by U.S.
commercial passenger air carriers?
* To what extent has FAMS's operational approach for achieving its core
mission been independently assessed?
* To what extent does FAMS have processes and initiatives in place to
address issues that affect the ability of its workforce to carry out
its mission?
This report is a public version of the restricted report (GAO-09-53SU)
that we provided to you on December 8, 2008. DHS and TSA deemed some of
the information in the restricted report as sensitive security
information, which must be protected from public disclosure. Therefore,
this report omits this information, such as the specific details
associated with FAMS's concept of operations. Although the information
provided in this report is more limited in scope, it addresses the same
questions as the restricted report. Also, the overall methodology used
for both reports is the same.
In studying FAMS's operational approach, we reviewed relevant
legislation regarding FAMS's mission, and we analyzed the agency's
policies and other documentation regarding the strategy and concept of
operations for carrying out that mission, including the procedures for
assigning or scheduling air marshals to selected flights. Also, we
reviewed the results of an evaluation conducted in 2003 by the Office
of Management and Budget (OMB), which utilized its Program Assessment
Rating Tool (PART) to assess the management and performance of FAMS.
[Footnote 6] Further, we also reviewed the follow-on reassessment of
FAMS that OMB conducted in 2008.
To determine the extent to which FAMS's operational approach has been
independently assessed, we reviewed a July 2006 classified report
prepared by the Homeland Security Institute based on its independent
evaluation of FAMS's concept of operations.[Footnote 7] Our engagement
team included a social science analyst and an economist with experience
in risk assessment, who used generally accepted social science research
standards in reviewing the Homeland Security Institute's report. Also,
we interviewed applicable Homeland Security Institute officials to
enhance our understanding of the evaluation's scope, methodology,
findings, and recommendations. Based on our review and discussion, we
determined the Homeland Security Institute's report to be sufficiently
reliable for the purposes of our work. Further, we reviewed FAMS
documentation--and interviewed the Director of FAMS and other senior
officials at the agency's headquarters--regarding the status of efforts
to address recommendations made by the Homeland Security Institute and
any related initiatives involving strategic planning and the agency's
concept of operations. We also reviewed two additional Homeland
Security Institute reports, which FAMS provided to us in September
2008. One of the reports detailed the Homeland Security Institute's
analysis regarding requirements for an improved mission scheduling tool
for FAMS,[Footnote 8] and the other report presented the institute's
benchmark analysis that compared FAMS's workday rules[Footnote 9] and
practices against those of similar occupations involving frequent air
travel and the related operational challenges, including fatigue and
other human factors.[Footnote 10]
Regarding processes and initiatives to address issues affecting the
ability of air marshals to carry out the agency's mission, we reviewed
prior reports--including our September 2005 report on planning and
controls for FAMS.[Footnote 11] We also reviewed FAMS documentation
regarding various working groups and other initiatives that FAMS had
established to address issues that affect the ability of air marshals
to carry out the agency's mission. For criteria in reviewing the
agency's documentation regarding these efforts, we drew on our prior
work regarding leading organizations and the best practices for
strategically managing human capital.[Footnote 12] Further, we
interviewed the Director of FAMS and other senior officials at agency
headquarters, and we visited 11 of the agency's 21 field offices, where
we interviewed managers and a total of 67 air marshals. We selected the
11 field offices and the 67 air marshals based on nonprobability
sampling.[Footnote 13] Regarding field office selections, we considered
various factors, such as geographic location and the involvement of
local management in agencywide working groups to address issues
affecting air marshals. Our selections of air marshals were made to
encompass a variety of experience levels. Because we selected a
nonprobability sample of FAMS field offices to visit and air marshals
to interview, the information we obtained in these visits and
interviews cannot be generalized either to all 21 field locations or to
all air marshals in the offices we visited. However, the interviews
provided us a broad overview of issues important to air marshals. We
also reviewed documentation regarding the implementation and results of
a workforce satisfaction survey that FAMS conducted in 2007. Our
engagement team, which included social science analysts with extensive
survey research experience, reviewed the questionnaire used in the
survey for clarity and the related response options for
appropriateness. Also, we discussed with FAMS officials the extent to
which efforts were made to obtain an overall response rate as high as
possible. As criteria to guide our review of the survey results, we
used OMB's Standards and Guidelines for Statistical Surveys (September
2006) and related guidance applicable to federal surveys.
We conducted this performance audit from April 2007 to December 2008 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. More details about the scope
and methodology of our work are presented in appendix I.
Results in Brief:
FAMS's operational approach (or concept of operations) is based on
assessments of threat, vulnerability, and consequences to help ensure
that high-risk flights operated by U.S. commercial air carriers--such
as the nonstop, long-distance flights targeted on September 11, 2001--
are given priority coverage by federal air marshals. Since it is not
feasible for federal air marshals to cover all of the approximately
29,000 domestic and international flights operated daily by U.S.
commercial passenger air carriers, FAMS aims to assign air marshals to
selected, higher-risk flights. Under this approach, FAMS seeks to
maximize coverage of high-risk flights in 10 targeted critical flight
categories, such as flights to and from the national capital region and
long-distance flights from various other locations. FAMS relies on a
scheduling process for determining the most efficient flight
combinations that will allow air marshals to cover the desired flights.
According to FAMS officials, the overall coverage goals and the
corresponding flight schedules of air marshals are subject to
modification at any time based on changing threat information and
intelligence.
In July 2006, the Homeland Security Institute, a federally funded
research and development center, concluded that FAMS's operational
approach for providing an on-board security presence on high-risk
flights was reasonable. However, the Homeland Security Institute noted
that certain types of flights were covered less often than others. The
Homeland Security Institute recommended that FAMS increase randomness
or unpredictability in selecting flights and to otherwise diversify the
coverage of flights. As of October 2008, FAMS had implemented or had
ongoing efforts to implement the institute's recommendations. For
example, FAMS is developing an automated decision-support tool for
selecting flights; this effort is expected to be completed by December
2009. We reviewed the Homeland Security Institute's evaluation
methodology and found it to be reasonable.
To better understand and address operational and quality-of-life issues
affecting the FAMS workforce, the agency's previous director--who
served in that capacity from March 2006 to June 2008--established
various processes and initiatives, including 36 issue-specific working
groups composed of managers and air marshals. The previous director
also established listening sessions that provided a forum for employees
to communicate directly with senior management and an internal Web site
for agency personnel to provide anonymous feedback to management. These
efforts have produced some positive results. For example, FAMS amended
its policy for airport check-in and flight boarding procedures
(effective May 15, 2008) to better ensure the anonymity of air marshals
in mission status[Footnote 14] and also adjusted its flight scheduling
process for air marshals to support a better work-life balance. All of
the 67 air marshals we interviewed in 11 field offices commented
favorably about these efforts and credited the leadership of the
previous FAMS Director. To reinforce a shared vision for workforce
improvements and to sustain forward progress, the current FAMS Director
has expressed a commitment to continuing applicable processes and
initiatives.
Also, FAMS has plans to conduct a workforce satisfaction survey of all
employees every 2 years, building upon an initial survey conducted in
fiscal year 2007. A majority (79 percent) of the respondents to the
2007 survey indicated that there had been positive changes from the
prior year, although the overall response rate (46 percent) constituted
less than half of the FAMS workforce and was substantially less than
the 80 percent encouraged by OMB in its guidance for federal surveys
that require its approval. According to the OMB guidance, a high
response rate increases the likelihood that the views of the target
population are reflected in the survey results. Although the FAMS
workforce survey did not require OMB approval, we believe that OMB
guidance provides direction in planning, designing, and implementing
surveys. While FAMS made some efforts to obtain responses to the 2007
survey, other methods outlined in the OMB guidance, such as extending
the cut-off date for responding, could improve the response rate of
future FAMS surveys.[Footnote 15] Also, 7 of the 60 questions in the
2007 survey questionnaire combined two or more issues, which could
cause respondents to be unclear on what issue to address and result in
potentially misleading responses. Moreover, none of the 60 questions in
the 2007 survey questionnaire provided for response options such as
"not applicable" or "no basis to judge"--responses that would be
appropriate when respondents had little or no familiarity with the
topic in question. Thus, the 2007 survey's results may not provide a
complete assessment of employees' satisfaction.
To increase the usefulness of biennial workforce satisfaction surveys,
we are recommending that the FAMS Director take steps to ensure that
the surveys are well designed and that additional efforts are
considered for obtaining the highest possible response rates. In
November 2008, in written comments on a draft of our restricted report,
the Department of Homeland Security and TSA agreed with our
recommendation and noted that FAMS was in the initial stages of
formulating the next workforce satisfaction survey, which included
plans to implement the recommendation.
Background:
Originally established as the Sky Marshal program in the 1970s to
counter hijackers, the Aviation and Transportation Security Act
expanded FAMS's mission and workforce in response to the September 11,
2001, terrorist attacks and mandated the deployment of federal air
marshals on high-security risk flights. Within the 10-month period
immediately following September 11, 2001, the number of air marshals
grew significantly. Also, during years following the 2001 attacks, FAMS
underwent various organizational transfers. Initially, FAMS was
transferred within the Department of Transportation from the Federal
Aviation Administration to the newly created TSA. In March 2003, FAMS
moved, along with TSA, to the newly established Department of Homeland
Security. In November 2003, FAMS was transferred to U.S. Immigration
and Customs Enforcement (ICE). Then, about 2 years later, FAMS was
transferred back to TSA in the fall of 2005.[Footnote 16]
FAMS is one layer among multiple layers of aviation security. For
example, prospective passengers are prescreened against applicable
records in the Terrorist Screening Center's consolidated watch list.
[Footnote 17] Passengers and baggage are also physically screened. Air
marshals generally are characterized as being the last line of defense
within the layered aviation-security framework. In this regard, FAMS
officials stressed that air marshals constitute the only in-flight
security layer deployed on the basis of risk.[Footnote 18]
FAMS deploys thousands of federal air marshals to a significant number
of daily domestic and international flights. In carrying out this core
mission of FAMS, air marshals are deployed in teams to various
passenger flights.[Footnote 19] Such deployments are based on FAMS's
concept of operations, which guides the agency in its selection of
flights to cover. Once flights are selected for coverage, FAMS
officials stated that they must schedule air marshals based on their
availability,[Footnote 20] the logistics of getting individual air
marshals in position to make a flight, and applicable workday rules.
At times, air marshals may have ground-based assignments. On a short-
term basis, for example, air marshals participate in Visible Intermodal
Prevention and Response (VIPR) teams, which provide security nationwide
for mass transit systems other than aviation.[Footnote 21] Also, air
marshals participate in Joint Terrorism Task Forces led by the Federal
Bureau of Investigation.
Good marksmanship is considered a necessity for air marshals,
particularly given the unique environment of the core mission--the
relatively tight confines of an airplane, coupled with the presence of
numerous passengers ("bystanders") and the possibility of air
turbulence that creates an unstable "shooting platform" (see figure 1).
Thus, according to TSA, air marshals have the highest marksmanship
standard in the federal government and also must be recertified on
their firearm every quarter.
Figure 1: Federal Air Marshals Are Trained for a Unique Working
Environment:
[Refer to PDF for image]
Series of four photographs depicting air marshals in training.
Source: GAO and the Federal Air Marshal Service (FAMS).
Note: Faces are intentionally blurred.
[End of figure]
To preserve their anonymity on covered flights, air marshals are to
blend in with other passengers by dressing appropriately and performing
their duties discreetly without drawing undue attention.
The Federal Air Marshal Service's Operational Approach to Achieving Its
Core Mission Is Based on Risk-Related Factors:
FAMS's operational approach (concept of operations) is based on risk-
related factors, such as assessments of threat, vulnerability, and
consequences. FAMS is guided by the provisions of the Aviation and
Transportation Security Act that specify the deployment of federal air
marshals on flights presenting high-security risks, such as nonstop,
long-distance flights targeted on September 11, 2001. FAMS seeks to
maximize coverage of high-risk flights by establishing coverage goals
for 10 targeted critical flight categories. In order to reach these
coverage goals, FAMS uses a scheduling process to determine the most
efficient flight combinations that will allow air marshals to cover the
desired flights. FAMS management officials stressed, however, that the
overall coverage goals and the corresponding flight schedules of air
marshals are subject to modification at any time based on changing
threat information and intelligence.
Risk-Related Factors Are Considered in Assigning Air Marshals to
Flights:
Following the attacks of September 11, 2001, FAMS developed a risk-
based concept of operations for deploying air marshals on U.S.
commercial passenger air carriers. Because there are many more U.S. air
carrier flights each day than can be covered by air marshals, FAMS
relies on the methodology outlined in its concept of operations to
assign air marshals to flights with the highest security risks. Under
this approach, FAMS considers the following risk-related factors to
categorize each of the approximately 29,000 domestic and international
flights operated daily by U.S. commercial passenger air carriers into
risk categories--high risk or lower risk:[Footnote 22]
* Threat (intelligence): Available strategic or tactical information
affecting aviation security is considered.[Footnote 23]
* Vulnerabilities: Although FAMS's specific definition is deemed to be
sensitive security information, DHS defines "vulnerability" as a
physical feature or operational attribute that renders an entity open
to exploitation or susceptible to a given hazard.
* Consequences: FAMS recognizes that flight routes over certain
geographic locations involve more potential consequences than other
routes.
FAMS attempts to assign air marshals to provide an on-board security
presence on as many of the flights in the high-risk category as
possible. However, other considerations can make covering only high-
risk flights impractical from a scheduling perspective and potentially
predictable to an adversary. Therefore, for purposes of scheduling
efficiency and adversary uncertainty, FAMS may deploy some air marshals
on lower-risk flights.
Air Marshals Are Deployed on Flights to Meet Coverage Goals for
Targeted Critical Flight Categories; the Concept of Operations Provides
Flexibility for Responding to Changing Threats:
* FAMS has established a scheduling process intended to maximize the
coverage of high-risk flights and meet the agency's desired coverage
goals for 10 targeted critical flight categories.[Footnote 24] FAMS's
Domestic Planning Branch (within the Systems Operation Control
Division) is responsible for scheduling air marshals to domestic
missions. During the course of a year, the Domestic Planning Branch
must prepare schedules for 13 roster periods of 28 days each. According
to FAMS officials, each 28-day schedule takes approximately 3 weeks to
prepare. The Domestic Planning Branch prepares each domestic schedule
using an automated scheduling tool.[Footnote 25] As part of the
scheduling process, each FAMS field office is responsible for making
available a specific percentage of their air marshals on a daily basis
to cover targeted critical flights (both domestic and international
flights) in the roster periods. FAMS utilizes the automated scheduling
tool to determine the most efficient flight "pairings" of departure and
return flights that will bring an air marshal back to his or her
starting point and that would be within the parameters for mission
assignment and rest for the air marshal. FAMS officials also perform
other checks on the fairness or appropriateness of the schedules, such
as ensuring that certain flights are not being covered repeatedly by
the same air marshals.
FAMS officials noted that the schedules for deploying air marshals are
altered as needed to cover specific, high-threat flights. For example,
in August 2006, FAMS increased its coverage of international flights in
response to the discovery, by authorities in the United Kingdom, of
specific terrorist threats directed at flights from Europe to the
United States. However, the officials added that a shift in resources
of this type can have consequences because of the limited number of air
marshals. FAMS officials noted that international missions require more
resources than domestic missions partly because the trips are of longer
duration.
An Independent Assessment Concluded That the Federal Air Marshal
Service's Approach for Achieving Its Core Mission Was Reasonable;
Recommendations for Enhancing the Approach Are Being Implemented:
In its 2003 PART review of FAMS, OMB concluded that an independent
evaluation should be conducted to assess FAMS's performance related to
aspects of the agency's concept of operations--particularly aspects
involving flight coverage risk categories, the distribution of covered
flights, and target levels of coverage.[Footnote 26] The Homeland
Security Institute, a federally funded research and development center,
performed this evaluation and issued a final report in July 2006. The
report concluded that FAMS's approach for achieving its core mission of
providing an onboard security presence for flights, as detailed in the
agency's concept of operations, was reasonable and made several
recommendations for enhancements.[Footnote 27] The Homeland Security
Institute recommended, for example, that FAMS increase randomness or
unpredictability in selecting flights and otherwise diversify the
coverage of flights within various risk categories. As of October 2008,
FAMS had implemented or had ongoing efforts to implement the
recommended enhancements.
In a July 2006 report, the Homeland Security Institute specifically
noted the following regarding FAMS's overall approach to flight
coverage:
* FAMS applies a structured, rigorous approach to analyzing risk and
allocating resources.
* The approach is reasonable and valid.
* No other organizations facing comparable risk-management challenges
apply notably better methodologies or tools.
As part of its evaluation methodology, the Homeland Security Institute
examined the conceptual basis for FAMS's approach to risk analysis.
Also, the institute examined FAMS's scheduling processes and analyzed
outputs in the form of "coverage" data reflecting when and where air
marshals were deployed on flights. Further, the Homeland Security
Institute developed and used a model to study the implications of
alternative strategies for assigning resources. We reviewed the
Homeland Security Institute's evaluation methodology and generally
found it to be reasonable.
In a 2008 PART reassessment of FAMS, OMB also reported that the
Homeland Security Institute's evaluation employed quality evaluation
methods and was comprehensive in scope. Further, OMB noted that an
interagency steering group--which was convened by the Homeland Security
Institute and met in conference in April 2006--also had reviewed FAMS's
concept of operations and considered it to be reasonable. In addition
to FAMS and Homeland Security Institute participants, the interagency
steering group consisted of representatives from various law
enforcement and counterterrorism agencies, which included the Federal
Bureau of Investigation, U.S. Customs and Border Protection, the
Transportation Security Administration, the Federal Aviation
Administration, the Homeland Infrastructure Threat and Risk Assessment
Center,[Footnote 28] U.S. Northern Command/North American Aerospace
Defense command,[Footnote 29] and the National Counterterrorism Center.
In its July 2006 report, the Homeland Security Institute made several
recommendations for enhancing FAMS's approach for deploying air
marshals on flights. As presented in table 1, FAMS had implemented or
had ongoing efforts to implement all of the recommended enhancements,
as of October 2008.
Table 1: Homeland Security Institute's Recommendations (July 2006
Report) to Improve the Federal Air Marshal Service's Approach for
Achieving Its Core Mission:
Topic: Definition of vulnerability;
Discussion, specific recommendations, and implementation status (as of
October 2008):
Discussion: According to the Homeland Security Institute, FAMS's
definition of "vulnerability" is inconsistent with traditional risk-
based methodologies;
Recommendation: FAMS should engage the aviation security community to
better understand vulnerabilities surrounding aircraft configuration or
possible gaps in the air defense system;
Implementation status: FAMS officials stated that they consulted with
the aviation security community and believe that the agency's
definition of vulnerability is appropriate for risk analyses relevant
to the mission of FAMS. However, the officials also noted that a new
automated decision-support tool for selecting flights is being
developed with assistance from the Homeland Security Institute and will
incorporate consideration of more traditional aspects of vulnerability.
FAMS expects development of this tool to be completed by the end of
calendar year 2009.
Topic: Filtering process;
Discussion, specific recommendations, and implementation status (as of
October 2008):
Discussion: The Homeland Security Institute reported that FAMS's
interpretation and implementation of impact (consequences) and the
subsequent "filtering" process heavily bias the agency's allocation
decisions;
Recommendation: FAMS should engage the intelligence and security
communities to validate these basic assumptions concerning threat and
consequence;
Implementation status: FAMS has modified its flight filtering approach
to open up more flights for potential coverage. In addition, the
Homeland Security Institute, as part of its ongoing work with FAMS
after issuance of the July 2006 report, continued to examine FAMS's
approach to flight coverage and will provide FAMS a report by the end
of calendar year 2009.
Topic: Allocation process;
Discussion, specific recommendations, and implementation status (as of
October 2008):
Discussion: The Homeland Security Institute found that the allocation
algorithm used by FAMS is biased and may not be consistent with
effective risk reduction;
Recommendation: FAMS should continue pursuing next-generation
scheduling and analysis tools to better quantify scheduling efficiency
and risk-reduction effectiveness;
Recommendation: To compensate for a lack of resources and to deny
predictability, FAMS should integrate randomness (unpredictability) in
its scheduling process;
Implementation status: As another part of its ongoing work, the
Homeland Security Institute is assisting FAMS in modifying the current
scheduling tool to provide more randomness in flight scheduling. In
July 2008, the institute provided FAMS a report detailing requirements
for a new scheduling tool. The institute's report recommended that FAMS
continue to use its present automated scheduling tool but expand the
capabilities of the software for determining flight risk and allocating
resources based on that risk. In addition, the institute recommended an
approach for using the scheduling tool to increase the randomness of
flights selected for coverage--that is, an approach designed to
randomly select some flights in each risk category while selecting
other flights in the respective category based on efficient flight
pairings and the need for scheduling efficiency. The institute also
recommended that FAMS continue to research methods for incorporating
more randomness into its scheduling process. FAMS officials indicated
that these ongoing efforts with the institute, along with the
modifications to the filtering process mentioned above, will help
address the randomness issue. FAMS officials stated that a date for
final completion of the new scheduling tool is uncertain since it is
dependent on the agency's future budgets.
Topic: Performance measures;
Discussion, specific recommendations, and implementation status (as of
October 2008):
Discussion: The Homeland Security Institute reported that FAMS's
primary performance measure-- average coverage rates--can mask
weaknesses in coverage patterns (variance)[A];
Recommendation: FAMS should develop performance measures to track the
distribution or coverage across certain categories of flights;
Implementation status: FAMS contracted with the Homeland Security
Institute to develop, as part of its ongoing work, proposals for
alternative performance measures for FAMS to consider. The institute's
final report is due to FAMS by the end of calendar year 2008.
Topic: Decision-support tools;
Discussion, specific recommendations, and implementation status (as of
October 2008):
Discussion: During the course of its analysis, the Homeland Security
Institute noted that FAMS's decision makers did not have a tool for
evaluating the resource implications of different threat scenarios and
alternative coverage schemes;
Recommendation: FAMS should consider building a decision-support tool,
along the lines of the model that the Homeland Security Institute
developed based on risk balancing, to facilitate a system-wide view of
resource decisions;
Implementation status: FAMS, with the assistance of the Homeland
Security Institute, is developing an automated decision-support tool
that will be linked to its flight-scheduling system. FAMS expects the
new decision-support tool to consider every flight over critical
infrastructure and include input for vulnerability of different types
of aircraft and airports. Development of the decision-support tool is
anticipated to be completed by the end of calendar year 2009. In the
interim, FAMS is basing its resource-allocation decisions partly on
manual processes whereby FAMS officials review the threat environment
and determine the most beneficial mix of international and domestic
flights, while observing budgetary and personnel constraints.
Source: Homeland Security Institute; GAO analysis.
Note: The "discussion" and the "recommendation" narratives are excerpts
from the Homeland Security Institute's July 2006 report. For more
details about the institute's report, see appendix III.
[A] Appendix IV presents more details about FAMS's key performance
measure.
[End of table]
In reference to the core mission of FAMS, the Homeland Security
Institute's recommendations regarding two processes--the filtering
process for selecting flights and the allocation process for assigning
air marshals to flights--are particularly important. To address the
institute's recommendations, FAMS officials stated that a broader
approach to filtering flights has been implemented--an approach that
opens up more flights for potential coverage, provides more diversity
and randomness in flight coverage, and extends flight coverage to a
variety of airports.
The Homeland Security Institute's ongoing work has also resulted in two
reports delivered to FAMS in July 2008. One of the reports detailed the
institute's analysis regarding requirements for a next-generation-
mission scheduling tool for FAMS,[Footnote 30] and the other report
presented the institute's benchmark analysis that compared FAMS's
workday rules and practices against those of similar occupations
involving frequent air travel and the related operational challenges,
including fatigue and other human factors.[Footnote 31] Also, in
September 2008, the Homeland Security Institute provided FAMS a
classified report assessing the deterrent effects of the agency's
approach to flight coverage. Further, based on its continuing work, the
institute expects to provide FAMS one additional final report by the
end of calendar year 2008--a report regarding potential enhancements to
performance measures.[Footnote 32]
The Federal Air Marshal Service Has Taken Positive Actions to Address
Issues Affecting Its Workforce and to Help Ensure Continued Progress:
To identify and address issues affecting the ability of its workforce
to successfully carry out its mission, FAMS has implemented various
communication-oriented processes or initiatives--including 36 issue-
specific working groups--that have produced some positive results. For
instance, FAMS has revised and documented certain policies--including
the policy related to aircraft check-in and boarding procedures--to
better protect air marshals' anonymity. In addition, FAMS has modified
its mission scheduling processes and implemented a voluntary lateral
transfer program to address certain issues regarding air marshals'
quality of life--and has plans to further address health issues
associated with varying work schedules and frequent flying. As an
additional initiative to help determine the effectiveness of
management's actions to address issues affecting air marshals, FAMS
conducted a workforce satisfaction survey of all staff in late fiscal
year 2007. A majority (79 percent) of the respondents to the survey
indicated that there had been positive changes from the prior year,
although the overall response rate (46 percent) constituted less than
half of the workforce. The 46 percent response rate was substantially
less than the 80 percent rate encouraged by OMB in its guidance for
federal surveys that require its approval. According to the OMB
guidance, a high response rate increases the likelihood that the views
of the target population are reflected in the survey results. Obtaining
a higher response rate to FAMS's future surveys, which the agency plans
to conduct every 2 years, and modifying the structure of some
questions, could enhance the surveys' potential usefulness by, for
instance, providing a more comprehensive basis for assessing employees'
attitudes and perspectives. All 67 of the air marshals we interviewed
in 11 field offices attributed progress under these efforts largely to
the "tone at the top," particularly the commitment exhibited by the
former FAMS Director who served in his position from March 2006 to June
2008. To reinforce a shared vision for workforce improvements and
sustain forward progress, the current FAMS Director has expressed a
commitment to continuing applicable processes and initiatives.
The Federal Air Marshal Service Implemented Various Processes or
Initiatives to Address Issues Affecting Its Workforce:
Our prior work has shown that leading organizations commonly sought
their employees' input on a periodic basis--by, for example,
establishing working groups or task forces, convening focus groups, and
conducting employee satisfaction surveys--and used that input to adjust
their human capital approaches.[Footnote 33] Starting in March 2006,
the then-serving FAMS Director implemented several communication
processes or initiatives to better understand and address issues facing
the agency's workforce. Chief among these processes or initiatives were
issue-specific working groups established to study, analyze, and
address a variety of issues ranging from mission, organizational, and
operational topics to workforce satisfaction and quality-of-life
concerns. Initially, based on his knowledge of issues facing the
organization when he assumed the leadership position in March 2006, the
FAMS Director established 12 working groups. Subsequently, based on
feedback from these initial groups and other sources regarding issues
of concern, the number of working groups expanded to 36 (see app. V).
Each working group typically included a special agent-in-charge, a
subject matter expert, air marshals, and mission support personnel from
the field and headquarters. FAMS management directed working group
members to define each group's purpose, analyze specific issues,
develop short-and long-term recommendations and determine their
financial feasibility. As a final product, FAMS management expected
each working group to submit a report, including recommendations, to
the FAMS executive staff for managerial consideration.[Footnote 34]
According to FAMS management, the working groups typically disband
after submitting a final report. FAMS management stressed, however,
that applicable groups could be reconvened or new groups established as
needed to address relevant issues.
In addition to the working groups, other processes or initiatives
implemented by FAMS management to address workforce issues or otherwise
improve management-workforce communication include the following:
* Field office focus groups--Each of the 21 FAMS field offices
organized a local focus group composed of representatives from the
respective office's air marshal squads and at least one mission support
staff. All members serve on a rotating basis, and the groups are to
meet at least quarterly to discuss issues of concern to the local
workforce and bring these issues to the attention of the applicable
field office's special agent-in-charge.
* Field office visits by the FAMS Director--In 2006, the FAMS Director
began visiting field offices and holding informal gatherings with air
marshals, outside the presence of local managers, to discuss their
questions and concerns.
* Listening sessions--FAMS senior management established forums to
allow direct communication between FAMS senior management and various
personnel. In 2006, the FAMS Director and Deputy Directors conducted
these sessions weekly in headquarters and the field offices with a
total of 10 to14 staff selected for each meeting. In 2007, this format
changed from weekly to monthly sessions and included larger groups of
FAMS personnel.
* Dinners with the Director--In 2006, the FAMS Director began holding
weekly dinners to meet with air marshals transiting through the
Washington, D.C., area. These dinners provide an opportunity for air
marshals to speak personally with the director about any questions or
concerns. The FAMS Deputy Director and one assistant director also
attend these dinners with selected air marshals.
* Director's e-mail in-box[Footnote 35]--FAMS established an e-mail in-
box for agency personnel to provide feedback to the FAMS Director. At
any time, air marshals--whether at headquarters, in a field office, or
deployed on mission--can send their insights, ideas, suggestions, and
solutions to the FAMS Director.
* Anonymous Web site--FAMS established an internal Web site for agency
personnel to provide anonymous feedback to FAMS management on any
topic.
* Ombudsman Position--FAMS management assigned an air marshal to the
position of Ombudsman in October 2006. According to FAMS management,
the Ombudsman provides confidential, informal, and neutral assistance
to employees to address workplace-related problems, issues, and
concerns. FAMS reported that, in fiscal year 2007 (the first full year
of the position), the Ombudsman handled 67 cases, and, through the
first three quarters of fiscal year 2008, an additional 54 cases.
FAMS officials estimated that, as of October 2008, more than one-fourth
of the agency's employees had participated in one or more of these
activities which encompass the various working groups and other
processes and initiatives.[Footnote 36]
Based on input provided by the working groups and information obtained
through the other processes and initiatives, FAMS has taken or is
planning to take actions to address issues that affect the ability of
air marshals to carry out the agency's mission. As discussed in the
following sections, these actions address operational issues, such as
check-in and boarding procedures that affect air marshals' anonymity as
well as quality-of-life and health issues.
To Help Preserve the Anonymity of Air Marshals, the Federal Air Marshal
Service Revised Its Policy and Procedures for Check-In and Boarding of
Aircraft:
To preserve their anonymity on covered flights, federal air marshals
are to blend in with other passengers by dressing appropriately and
performing their duties discreetly without drawing undue attention. In
past years, air marshals frequently asserted that the check-in and
boarding policy and procedures established by FAMS compromised their
anonymity by requiring repeated interactions with airline personnel. In
September 2005, we reported that the full extent of incidents that air
marshals encounter was unknown because FAMS lacked adequate management
controls for ensuring that such incidents were recorded, tracked, and
addressed.[Footnote 37]
Accordingly, to facilitate management of incidents that affect air
marshals' ability to operate discreetly during their missions, our
September 2005 report recommended that FAMS take the following four
actions:
* Develop a means for recording all incidents reported to the Mission
Operations Center that affect air marshals' ability to operate
discreetly and criteria for determining which incidents require federal
air marshals to complete a mission report.
* Develop a means for tracking and retrieving data on mission reports
to enable FAMS to analyze and monitor reported and systemic incidents.
* Establish written policies and procedures for reviewing and
addressing reported incidents.
* Establish a means for providing feedback on the status and outcome of
FAMS mission reports to the federal air marshals who submit them.
FAMS has taken steps to address all four of these recommendations and
also address the related feedback received from air marshals through
various working groups and other initiatives. In October 2005, FAMS
issued a written directive establishing policies and procedures for
reporting and managing mission incidents.[Footnote 38] In November
2005, we reported that we had reviewed the directive and believed that
it addressed two of our recommendations--the first and the third
recommendations.[Footnote 39] More recently, in March 2008, FAMS issued
an addendum to its written directive establishing a means for providing
feedback on the status and outcome of FAMS mission reports to the
federal air marshals who submit them (fourth recommendation).
Additionally, FAMS revised its policy and procedures regarding
interaction with airline personnel during the check-in and boarding
processes in order to better protect the anonymity of air marshals in
mission status.[Footnote 40] To help ensure effective implementation,
the new policy and procedures have been incorporated into TSA's
Aircraft Operator Standard Security Program, which specifies
requirements that domestic passenger air carriers must implement as
part of their TSA-approved security programs. According to FAMS
officials, the recent update constitutes the first time that the
Aircraft Operator Standard Security Program guidance specifically
includes a section regarding the boarding of federal air marshals.
Through use of a database created in fiscal year 2006 to track mission
incidents, FAMS senior executive staff noted that analysis and
monitoring are conducted daily of reported incidents, including those
that could compromise the ability of air marshals to operate discreetly
(second recommendation). The first management report detailing overall
incident patterns and trends was produced in July 2008. Going forward,
FAMS officials stated that reports would be produced quarterly to allow
management to review patterns or trends regarding mission incidents and
the effectiveness of the new policy and procedures.
The Federal Air Marshal Service Modified Its Dress Code and Hotel
Policies to Further Protect Air Marshals' Anonymity:
To further protect the anonymity of air marshals while on missions, and
in response to air marshals' feedback and the working groups'
recommendations, FAMS management revised the dress code policy and the
hotel policy for air marshals in August 2006 and February 2007,
respectively. The revisions allow air marshals greater discretion in
selecting appropriate attire to wear on missions and choosing hotels
for overnight trips. Before the revisions, air marshals reported that
the dress code policy was too restrictive and forced them to dress too
formally for certain flights, such as those to vacation-oriented
destinations. According to the air marshals, this restrictive policy
resulted in their standing out from the other passengers, a situation
that compromised their anonymity. Similarly, before being revised,
FAMS's hotel policy directed air marshals to stay at certain hotels on
overnight missions so that they could be located easily by management
in an emergency. Additional considerations of FAMS management for
restricting the hotel selection were to ensure that air marshals were
able to stay at hotels within per diem rates and would have ready
access to transportation between the hotel and the airport. Air
marshals expressed concerns that repeatedly staying at the same hotels
risked exposing their anonymity. The revised policy allows air marshals
to select their own hotels, provided the hotels are within per diem
rates and have adequate transportation options. To alleviate concerns
of FAMS management about being able to contact air marshals in an
emergency, the revised policy requires air marshals to report their
hotel locations via the FAMS intranet.
All 67 of the air marshals we interviewed in the 11 field offices we
visited said that the revised dress code and hotel policies adequately
addressed their concerns.
The Federal Air Marshal Service Is Planning to Replace Its Inventory of
Personal Digital Assistant Communication Devices by Early Fiscal Year
2009:
FAMS has described the agency's personal digital assistant (PDA)
communication device as being a lifeline for air marshals. The current
device carried by air marshals is intended to function as a cell phone
and personal computer and allow users to place phone calls, access the
Internet, send e-mails, pull up basic Microsoft Word documents, store
documents, and submit reports. However, the findings of FAMS's
applicable working groups indicated that the current PDA communication
device has proven unreliable. Similarly, all 67 of the air marshals we
interviewed in 11 field offices stated that they had experienced
problems with their PDA device while on missions. Examples of problems
reported by air marshals included dropped calls or lost signals in
certain geographical areas, limited audio quality and durability, and
lack of ability to send certain required documents (such as time and
attendance reports). Another reported problem was the frequent freezing
or locking of the PDA device, which then necessitated use of a
cumbersome reset process. As a result of such problems, air marshals
reported that the PDA device has hindered their ability to communicate
effectively with management while in mission status. Additionally, the
air marshals we interviewed commented that the current PDA device is
relatively large and bulky, which potentially contributes to loss of
anonymity.
In response to air marshals' feedback and the working groups'
recommendations, FAMS is taking steps to procure new PDA communication
devices and distribute them to air marshals. Furthermore, according to
FAMS officials, the procurement contract for the new PDA devices will
provide for a 2-year replacement cycle. In the interim, to improve
voice communication capabilities pending arrival of the new devices,
FAMS officials reported that it issued new cell phones to air marshals
in June 2008. The officials noted, however, that air marshals still
must rely on the current PDA device for non-voice functions, such as
sending and receiving e-mail messages and documents, until the new PDA
devices are available.
To Address Quality-of-Life and Health Issues, the Federal Air Marshal
Service Has Implemented Changes to Mission Scheduling and Enlisted the
Assistance of Outside Experts:
In reference to quality-of-life and health issues, mission scheduling
constitutes the most significant concern of air marshals, according to
feedback that FAMS management received from working groups and other
communication processes and initiatives. To be fully effective, air
marshals must be healthy, fit, and alert. However, FAMS's Medical
Issues Working Group reported that air marshals have experienced
various types of health issues--poor physical fitness as well as
musculoskeletal injuries and upper respiratory infections--that may
potentially be attributable to frequent flying and the overall nature
of their jobs. The working group noted various challenges to ensuring
that air marshals have adequate sleep, exercise, and recovery time. A
contributing factor noted is that the agency's automated scheduling
tool historically has lacked the capability to consistently program an
air marshal's daily start and end times throughout a roster period,
which makes normal sleep patterns difficult to maintain and often
results in fatigue. For instance, an air marshal may have been
scheduled to begin some days at 5 a.m. and other days at 10 a.m., with
unpredictable ending times because of flight delays. In addition to
inconsistent shifts, the Medical Issues Working Group noted that air
marshals are subject to long hours--including arriving home late on a
Friday and then having to depart early the following Monday morning.
These types of schedules, according to the working group, make allowing
adequate time for workouts and maintaining healthy eating habits
difficult and also limit the amount of time available to take care of
family and personal needs.
To address these scheduling issues, FAMS has implemented or is planning
to implement various changes:
* Mission exchange program--This program, which FAMS initially piloted
in 2006 and is now available to all 21 field offices, allows air
marshals within the respective field office to exchange mission days
based on a demonstrated need, such as medical issues or family-related
issues. For instance, an air marshal with an 8 a.m. mission start time
and a 9 a.m. medical appointment could exchange shifts with another air
marshal for a later mission start time. The program is intended to
reduce the amount of unscheduled leave taken by air marshals and
otherwise mitigate the hardships or other effects associated with
FAMS's current policy of requiring air marshals to submit requests for
annual leave 38 to 66 days in advance.
* Preset ending time and 60-hour rule--In September 2006, FAMS
instituted a change to its mission-scheduling policy. The change is
designed to help ensure that air marshals complete their mission
flights by a preset time on the day before a regular day off (or the
day before scheduled annual leave) and not begin a new mission until
receiving a minimum of 60 hours rest.[Footnote 41] For example, if an
air marshal's regular days off are Saturday and Sunday, and this
individual's mission ended on Friday evening, the next mission
assignment (on Monday) would begin no earlier than Monday morning.
* Limit on number of flight days--In April 2007, FAMS implemented
another change in mission-scheduling policy designed to distribute
flight days equitably and improve the balance between work and personal
life for air marshals. Specifically, under the new policy, each air
marshal's total flight days are targeted to not exceed 18 days per
roster period and 200 days annually.
* More rest time after completing extended international missions--Also
in April 2007, FAMS issued guidance to field offices to make every
attempt at increasing rest time for air marshals after completing an
extended international mission. Under this guidance, air marshals
returning from an international mission are to be given a non-flight
day as their next duty day when any one of the following three
conditions apply: (1) the return flight exceeds 10 hours in the air,
(2) the flight crossed the international date line, or (3) the overall
mission (round-trip flights plus overnight stays) was 4 days or longer
in duration. Depending on an air marshal's schedule, a non-flight day
could be a training day, a regular day off, or a non-mission status
day.[Footnote 42]
* More consistent start times--FAMS is currently developing a
modification to its scheduling tool to provide a consistent, defined
scheduling window (encompassing, for example, 3 hours) for air marshals
in mission status to report for duty during a 7-day period. Under the
planned modification, for instance, FAMS schedulers would assign an air
marshal to flights departing during 5:00 a.m. to 8:00 a.m., 9:00 a.m.
to 12:00 p.m., or other 3-hour window during the week. FAMS officials
stated that this modification, which is intended to provide more
consistent start times for each air marshal throughout the applicable
week, should be completed and ready for pilot testing by the middle of
calendar year 2009.
The 67 air marshals we interviewed in 11 field offices generally
expressed satisfaction with the various enhancements to mission
scheduling, although most (43) mentioned that implementation of the
mission exchange program was still evolving.
To more specifically address the health implications of flight
scheduling, several efforts were recently completed or are planned. For
instance, the Homeland Security Institute conducted a benchmark
analysis and assessment of fatigue issues related to air marshals and
issued a report to FAMS in July 2008.[Footnote 43] In its analysis, the
institute compared FAMS's workday rules against other occupations--
largely in the aviation realm--that face challenges involving frequent
travel, jet lag, long work hours, rotating shifts, and the stress of
maintaining a schedule across multiple flights and airports. The
Homeland Security Institute noted that although no other occupation is
identical to that of air marshals, meaningful comparisons were made
with similar occupations, such as commercial airline pilots and cargo
pilots and law enforcement officers working in aviation (e.g., U.S.
Marshals Service aviation enforcement officers responsible for
transporting prisoners). In its July 2008 report, the Homeland Security
Institute noted that while stress and fatigue issues are a part of all
organizations and cannot be entirely eliminated, air marshals are
provided considerable blocks of rest within their schedules, when
assessed against similar occupations. Overall, the institute reported
that the results of the benchmark analysis showed that air marshals are
provided above-average time to recuperate from duty days.
Further, the institute noted that FAMS has taken various steps,
including implementation of the mission exchange program, to improve
aspects of mission scheduling. In addition, in October 2008, FAMS
officials informed us that the agency has funded a contract with the
National Institute of Justice to implement FAMS-specific research
regarding mission scheduling, work-rest cycles, fatigue, and
performance.
To Further Address Quality-of-Life Issues, the Federal Air Marshal
Service Has Implemented a Voluntary Lateral Transfer Program:
According to FAMS officials, air marshals frequently cited the need for
a voluntary lateral transfer program during listening group sessions
and dinners with the director. FAMS working groups that examined
quality-of-life issues also reported that the agency would benefit from
implementing a transfer program for air marshals to express interest in
relocating to another field office. Thus, in October 2006, FAMS
management implemented a voluntary lateral transfer program. Under the
program, an air marshal in good standing may request a transfer for up
to three field offices, ranked by order of preference, and FAMS
management is to make decisions based on the number of vacancies in
each office and the seniority of air marshals who apply for transfer.
In December 2006, FAMS announced that 176 air marshals had been
selected, during the first phase of the program, for transfer to new
offices within 60 days. In the second phase, which occurred in the
spring of 2007, FAMS management made transfer offers to 40 air
marshals--all of whom accepted. In the third and most recent phase,
which occurred in the spring of 2008, FAMS management made offers to 48
air marshals--of whom 45 accepted. FAMS expects to continue offering
voluntary transfer opportunities during open seasons in the spring of
each year.
The Federal Air Marshal Service Conducted a Workforce Satisfaction
Survey in 2007; the Potential Usefulness of Future Surveys Could Be
Enhanced by Improving the Response Rate and Redesigning Certain
Questions:
In late fiscal year 2007, FAMS conducted a workforce satisfaction
survey of all staff--not just air marshals--to help determine issues
affecting the ability of agency personnel to perform their jobs--and to
obtain feedback on the effectiveness of measures already taken by
management to address relevant issues. [Footnote 44] The 2007 survey
questionnaire consisted of a total of 60 questions that covered 13
topics--senior leadership; supervisor/management; resources and
technology; training and education; career development; policies and
procedures; employee involvement and autonomy; rewards and recognition;
communication; safety, health, and medical issues; work and family
life; organizational commitment; and job satisfaction.[Footnote 45]
According to FAMS management officials, the survey provided useful
information on quality-of-life and other issues affecting the ability
of air marshals and other agency personnel to perform their jobs. In
addition, the officials reported that survey results indicated that
employees generally were pleased with the policy changes and other
actions implemented by management to address relevant issues. For
example, although the 2007 workforce satisfaction survey had an overall
response rate (46 percent) that constituted less than half of the FAMS
workforce, 79 percent of the respondents indicated that there had been
positive changes from the prior year. Regarding future plans, FAMS
expects to administer a workforce satisfaction survey every 2 years.
FAMS officials stated that a purpose of the initial workforce
satisfaction survey was to establish a baseline for use in comparing
the results of future surveys.
In reviewing the 2007 survey's implementation and results, we made
several observations that are important for enhancing the potential
usefulness of future surveys. First, as noted previously, the overall
response rate was 46 percent. FAMS officials expressed satisfaction
with this response rate given the highly mobile nature of their
workforce. The FAMS officials also noted that the 46 percent response
rate was similar to the response rates for other federal workforce
satisfaction surveys. However, the 46 percent response rate was
substantially less than the 80 percent rate OMB encourages for federal
surveys that require its approval. Although internal workforce surveys
such as the one conducted by FAMS do not require OMB approval, we
believe the OMB standards and guidance provide relevant direction on
planning, designing, and implementing high-quality surveys--including
the need to obtain a high response rate to increase the potential that
survey responses will accurately represent the views of the survey
population.[Footnote 46]
Specifically, the OMB guidance stipulates that agencies must design
surveys to achieve the highest practical rates of response to ensure
that the results are representative of the target population and that
they can be used with confidence as input for informed decision-making.
OMB encourages agencies to obtain at least an 80 percent response rate,
and its guidance states that response rates are an important indicator
of the potential for nonresponse bias, which could affect the accuracy
of a survey's results. For instance, survey estimates may be biased if
the individuals who choose to participate (respondents) differ
substantially and systematically in some way from those who choose not
to participate (nonrespondents). In general, a higher response rate
increases the likelihood that any bias problem is decreased, resulting
in the views and characteristics of the target population being more
accurately reflected in the survey's results. Thus, for any federal
survey that must be approved by OMB, applicable guidelines stipulate
that an analysis for possible nonresponse bias must be conducted if the
final response rate is less than 80 percent.[Footnote 47]
Regarding the 46 percent response rate for the 2007 survey, FAMS
management reported that an analysis of potential nonresponse bias was
conducted by comparing various demographic data provided by the
respondents to the FAMS workforce as a whole. Based on the analysis of
the available demographic data, FAMS concluded that nonresponse bias
did not exist as the respondents were representative of the entire
workforce. Although the analysis conducted by FAMS was a useful effort,
the potential for a nonresponse bias still exists given that over half
of the FAMS workforce did not respond to the survey.
As noted previously, concerns about nonresponse bias could be avoided
or mitigated by obtaining a higher response rate. FAMS employees were
given 3 weeks (August 23 through September 14, 2007) to complete the
2007 workforce satisfaction survey. According to FAMS management, even
though all employees (not just nonrespondents) were sent four messages
reminding them of the deadline for completing the voluntary survey, the
final overall response rate was 46 percent. We believe that other
widely acknowledged methods, outlined in OMB guidance, could improve
the response rate of future FAMS surveys. These methods include, for
example, promoting awareness of the survey through outreach efforts
with groups of prospective respondents and extending the cut-off date
for responding to the survey. Also, monitoring questionnaire returns
and targeting extra follow-up efforts to air marshals in particular
field locations that have comparatively low response levels could help.
Additional observations we made in reviewing the 2007 workforce
satisfaction survey's questionnaire involve the sentence structure of
certain questions and the response options. Generally, any question
that combines two or more issues--but does not provide for separate or
respective answers--can cause uncertainty about how to respond if the
answer to each issue is different.[Footnote 48] Table 2 lists the seven
2007 workforce satisfaction survey questions that used these types of
sentence structures. For instance, regarding the senior leadership of
FAMS, question 3 cites two concepts ("visions" and "initiatives"), as
well as two actions ("shared" and "supported") associated with these
concepts. However, the response options did not account for the fact
that experiences could be different with each of these concepts and
actions. Similarly, question 10 addresses the reliability of equipment
used by agency personnel and cites four different devices. However, the
response options did not account for the fact that experiences could be
different with each of these devices.
Table 2: FAMS 2007 Workforce Satisfaction Survey Questions That
Combined Two or More Issues:
Survey question topic, number, and wording:
Topic: Senior Leadership;
Number: 3;
Wording: I believe that the Director's visions and initiatives are
consistently shared and supported by my field management.
Topic: Resources and technology;
Number: 10;
Wording: Generally, the equipment I use (e.g., firearm, computer, cell
phone, personal digital assistant, etc.) to perform my job works
properly.
Topic: Training and education;
Number: 16;
Wording: Generally, I am satisfied with the content and variety of job-
related training I receive in my office.
Topic: Communication;
Number: 36;
Wording: FAMS policies and procedures are clearly communicated and easy
to understand."
Topic: Safety, health, and medical issues;
Number: 43;
Wording: I am satisfied that FAMS management is concerned for the
health and safety of employees and is working continuously to offer
improved services.
Topic: Safety, health, and medical issues;
Number: 44;
Wording: I have been provided information and resources to take
personal responsibility for my health and wellness as it relates to my
job (e.g., proper diet, fitness, sufficient rest).
Topic: Work and family life;
Number: 51;
Wording: I have seen improvement in quality of work life and family
life as a result of the recommendations from the Director's working
groups.
Source: FAMS 2007 workforce satisfaction survey.
Note: See appendix VI.
[End of table]
Also, none of the 60 questions in the 2007 workforce satisfaction
survey provided for response options such as "not applicable" or "no
basis to judge"--responses that would be appropriate when respondents
had little or no familiarity with the topic in question. Not providing
response options such as "not applicable" or "no basis to judge" could
lead to potentially misleading question responses. In the interest of
being compliant, respondents might be compelled to give a response,
such as "neutral," to a question when they actually have no opinion due
to either non-applicability or lack of familiarity with the question
topic. While it might be assumed that all individuals being surveyed
should be familiar with the topic of all questions, this might not be
the case and will not be known unless the questionnaire contains the
relevant response options. For example, question 39 (see app. VI),
reads as follows: "I am satisfied that the work-related concerns I
address with management are addressed appropriately." As written, this
sentence assumes that every employee has raised work-related concerns
with management. If a respondent had never expressed work-related
concerns with management, this individual might not know how to
respond, given the question's existing response options.
Thus, based on the sentence structure of certain questions and the
response options, the results from the 2007 survey may provide an
incomplete assessment of employees' perspectives and attitudes to FAMS
management. Regarding our observations on the design of survey
questions and response options, FAMS officials stated that limited
personnel resources precluded investing more time in development of the
survey questionnaire and that the survey had served a useful purpose in
providing information on issues of concern to be more fully explored
through other communication processes or initiatives. Nonetheless, in
developing future survey instruments, designing questions to avoid
these types of ambiguities could provide FAMS management with
information that is more focused and complete. Although we recognize
that FAMS has a variety of other processes and initiatives--in addition
to the customer satisfaction survey--for identifying and addressing
workforce issues, customer satisfaction surveys can be particularly
useful given that they are distributed to all employees and provide for
anonymity of respondents. Further, the design considerations that we
discussed involve relatively minor technical aspects that could be
addressed with a minimal investment of personnel resources.
The Federal Air Marshal Service Has Committed to Addressing Workforce
Issues by Sustaining Applicable Processes and Initiatives:
As highlighted in our prior work, agency leaders in best practice
organizations view people as an important enabler of agency performance
and recognize the need for sustained commitment to strategically manage
human capital.[Footnote 49] In developing approaches to managing the
workforce, leaders of best practice agencies seek out the views of
employees at all levels. Involving employees in the planning process
helps agencies to develop goals and objectives that incorporate
frontline insights and perspectives about operations. Further, such
involvement can also serve to increase employees' understanding and
acceptance of organization goals and objectives and improve motivation
and morale. Our work has shown that leading organizations commonly
sought their employees' input on a periodic basis and used that input
to adjust their human capital approaches. Among other means, the
organizations collected feedback by convening focus groups, providing
opportunities for employees to participate in working groups or task
forces, and conducting employee satisfaction surveys.
As discussed earlier in this report, FAMS has implemented a variety of
processes and initiatives to address workforce issues by soliciting the
views of front-line staff across the agency. Several key improvements
in FAMS policies and procedures have resulted from these efforts. Among
other improvements, for example, FAMS amended its policy for flight
check-in and boarding procedures to better ensure the anonymity of air
marshals in mission status. Also, the various processes and initiatives
have helped to improve agency morale, according to the federal air
marshals we interviewed. Moreover, agency officials noted that the
processes and initiatives represented a significant commitment in
management time and resources.
In our view, fostering continued progress in addressing workforce
issues at FAMS is important. The current FAMS Director, after being
designated in June 2008 to head the agency, issued a broadcast message
to all employees, expressing a commitment to continue applicable
processes and initiatives, including the working group process,
listening sessions, field office visits, and the internal Web site for
agency personnel to provide anonymous feedback to management on any
topic. More recently, in response to our inquiry, FAMS's Chief of Staff
reported in October 2008 that the various communications processes and
initiatives "have become an institutionalized and positive aspect" of
the agency's culture. Also, the Chief of Staff noted that FAMS was in
the process of establishing an agencywide national advisory council--
with representatives from headquarters and all field offices--to
further enhance communication and outreach efforts, promote greater job
satisfaction, and improve organizational effectiveness through
cooperative problem solving and replication of best practices.
Conclusions:
Federal air marshals are an important layer of aviation security. Thus,
it is incumbent upon FAMS management to have sound management processes
in place for identifying and addressing the challenges associated with
sustaining the agency's operations and addressing workforce quality-of-
life issues. FAMS, to its credit, has established a number of processes
and initiatives--including a workforce satisfaction survey--to address
various operational and quality-of-life issues that affect the ability
of air marshals and other FAMS personnel to perform their aviation-
security mission. Consistent with the human capital practices of
leading organizations, the current FAMS Director has expressed a
commitment to continuing relevant processes and initiatives for
identifying and addressing workforce concerns, maintaining open lines
of communications, and sustaining forward progress. Although the
workforce satisfaction survey is only one of a number of processes or
initiatives used by FAMS to identify and address workforce issues, such
surveys play an important role given their agencywide scope and the
provision for anonymous responses. A higher response rate and more
clearly structured questions and response options could add to the
usefulness of this effort.
Recommendation for Executive Action:
To facilitate continued progress in identifying and addressing issues
that affect the ability of FAMS personnel to perform the agency's
aviation-security mission, we recommend that the FAMS Director take
appropriate actions to increase the usefulness of the workforce
satisfaction surveys that FAMS plans to conduct biennially. Such
actions could include, for example, ensuring that the survey questions
and the answer options are clearly structured and unambiguous and that
additional efforts are considered for obtaining the highest possible
response rates.
Agency Comments:
We provided a draft of our restricted report for comment to the
Department of Homeland Security and TSA. In November 2008, in written
comments, the Department of Homeland Security and TSA agreed with our
recommendation and noted that FAMS was in the initial stages of
formulating the next workforce satisfaction survey, which included
plans to implement the recommendation. Also, the Department of Homeland
Security and TSA commented that our key findings and recommendation
will facilitate continued progress in identifying and addressing issues
that affect the ability of FAMS personnel to perform the agency's
aviation security mission. The full text of the department's and TSA's
written comments is reprinted in appendix VII.
As arranged with your office, we plan no further distribution until 30
days after the date of this report. At that time, we will send copies
of this report to interested congressional committees and
subcommittees. We will also make copies available to others upon
request.
If you or your staff have any questions about this report or wish to
discuss the matter further, please contact me at (202) 512-4379 or
lords@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Other key contributors to this report are listed in appendix
VIII.
Signed by:
Stephen M. Lord:
Acting Director, Homeland Security and Justice Issues:
List of Congressional Requesters:
The Honorable Susan M. Collins:
Ranking Member:
Committee on Homeland Security and Governmental Affairs:
United States Senate:
The Honorable Daniel K. Akaka:
United States Senate:
The Honorable Bennie G. Thompson:
Chairman:
Committee on Homeland Security:
House of Representatives:
The Honorable Sheila Jackson-Lee:
House of Representatives:
The Honorable Christopher P. Carney:
House of Representatives:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
Objectives:
This report addresses the following three principal questions:
* What is the Federal Air Marshal Service's operational approach for
achieving its core mission of providing an onboard security presence
for flights operated by U.S. commercial passenger air carriers?
* To what extent has the Federal Air Marshal Service's operational
approach for achieving its core mission been independently assessed?
* To what extent does the Federal Air Marshal Service have processes
and initiatives in place to address issues that affect the ability of
its workforce to carry out its mission?
Scope and Methodology:
Initially, to obtain contextual and overview perspectives regarding the
principal questions, we reviewed information available on the Web sites
of relevant federal entities--the Department of Homeland Security, the
Transportation Security Administration (TSA), and the Federal Air
Marshal Service (FAMS). To obtain additional perspectives regarding
FAMS's mission and operations--and issues affecting its workforce--we
conducted a literature search to identify relevant reports or studies
and other publicly available information, including news media
articles. In particular, we focused on reviewing congressional studies,
Inspector General reports, and our previous reports. These included the
following:
* U.S. House of Representatives, Committee on the Judiciary, Plane
Clothes: Lack of Anonymity at the Federal Air Marshal Service
Compromises Aviation and National Security (Washington, D.C.: May 25,
2006).
* GAO, Aviation Security: Federal Air Marshal Service Could Benefit
from Improved Planning and Controls, GAO-05-884SU (Washington, D.C.:
Sept. 29, 2005). The report is restricted (not available to the public)
because it contains sensitive security information. The public version
of the report is [hyperlink, http://www.gao.gov/products/GAO-06-203]
(Nov. 28, 2005).
* U.S. Department of Homeland Security, Office of Inspector General,
Review of Alleged Actions by Transportation Security Administration to
Discipline Federal Air Marshals for Talking to the Press, Congress, or
the Public, OIG-05-01 (Washington, D.C.: Nov. 2004).
* GAO, Budget Issues: Reprogramming of Federal Air Marshal Service
Funds in Fiscal Year 2003, [hyperlink,
http://www.gao.gov/products/GAO-04-577R] (Washington, D.C.: Mar. 31,
2004).
* GAO, Aviation Security: Federal Air Marshal Service Is Addressing
Challenges of Its Expanded Mission and Workforce, but Additional
Actions Needed, [hyperlink, http://www.gao.gov/products/GAO-04-242]
(Washington, D.C.: Nov. 19, 2003).
This report is the public version of a restricted report that we
provided to congressional requesters in December 2008.[Footnote 50]
Further details about the scope and methodology of our work regarding
each of the three principal questions are presented in the following
sections, respectively.
The Federal Air Marshal Service's Operational Approach for Achieving
Its Core Mission:
In addressing this topic, we reviewed relevant legislation regarding
FAMS's mission and organizational structure. In particular, we reviewed
a provision of the Aviation and Transportation Security Act that:
* requires the deployment of federal air marshals on passenger airline
flights and:
* specifically requires the deployment of federal air marshals on every
flight determined to present high security risks.[Footnote 51]
We analyzed FAMS documentation regarding the agency's strategy and
concept of operations for carrying out its mission. Also, we reviewed
the results of an evaluation conducted in 2003 by the Office of
Management and Budget (OMB), which utilized its Program Assessment
Rating Tool (PART) to assess the management and performance of FAMS and
concluded that key aspects of program design needed to be independently
assessed.[Footnote 52] Further, we reviewed the follow-on PART-related
reassessment of FAMS that OMB conducted in 2008 (see app. II).
Independent Assessment of the Federal Air Marshal Service's Operational
Approach:
We reviewed the July 2006 classified report prepared by the Homeland
Security Institute based on its independent evaluation of FAMS's
concept of operations.[Footnote 53] Our engagement team included a
social science analyst and an economist with experience in risk
assessment, who used generally accepted social science research
standards in reviewing the institute's report. Also, we interviewed
applicable Homeland Security Institute officials to enhance our
understanding of the evaluation's scope, methodology, findings, and
recommendations. Based on our review and discussion, we determined this
report to be sufficiently reliable for the purposes of our work.
Further, we reviewed FAMS documentation--and interviewed the Director
of FAMS and other senior officials at the agency's headquarters--
regarding the status of efforts to address recommendations made by the
Homeland Security Institute and any related initiatives involving
strategic planning and the agency's concept of operations.[Footnote 54]
We also reviewed two additional Homeland Security Institute reports,
which FAMS provided to us in September 2008. One of the reports
detailed the institute's analysis regarding requirements for a next-
generation mission scheduling tool for FAMS,[Footnote 55] and the other
report presented the institute's benchmark analysis that compared
FAMS's workday rules[Footnote 56] and practices against those of
similar occupations involving frequent air travel and the related
operational challenges, including fatigue and other human factors.
[Footnote 57]
The Federal Air Marshal Service's Processes and Initiatives for
Addressing Issues That Affect Its Workforce:
Regarding operational or tactical issues as well as quality-of-life
issues that affect the ability of air marshals to carry out the
agency's mission, we reviewed published reports, including our
September 2005 report (GAO-05-884SU) as well as news media accounts of
relevant issues. We also reviewed FAMS documentation regarding various
working groups (see app. V) and other initiatives that FAMS had
established to address issues that affect the ability of air marshals
to carry out the agency's mission. In particular, we reviewed the final
report (if available) produced by the respective working group. For
criteria in reviewing the agency's documentation regarding these
efforts, we drew on our prior work regarding leading organizations and
the best practices for strategically managing human capital.[Footnote
58]
Further, we interviewed the Director of FAMS and other senior officials
at agency headquarters, and we visited 11 of the agency's 21 field
offices, where we interviewed managers and a total of 67 air marshals.
[Footnote 59] We selected the 11 field offices and the 67 air marshals
based on nonprobability sampling.[Footnote 60] In selecting the 11
field offices, we considered various factors, such as geographic
location of the offices and the involvement of local management in
agencywide working groups to address issues affecting air marshals.
At each of the 11 field offices, we first reviewed available work-
related information about individual air marshals, such as their
starting dates with FAMS and their involvement in ground-based
assignments or any agencywide working groups.[Footnote 61] Based on
these factors, we selected and interviewed 6 to 7 air marshals at each
of the 11 field offices. Specifically, we selected 6 air marshals at
each of 10 field offices and 7 air marshals at the remaining office.
Our selections were made to encompass a variety of experience levels.
Also, at each field office, rather than meeting separately with each
individual, we conducted the interviews of the selected air marshals in
group settings to encourage a wide array of perspectives, whether
corroborating or contradictory. We conducted our interviews at the
field offices during a 7-month time period, July 2007 through January
2008.
Because we selected a nonprobability sample of FAMS field offices to
visit and air marshals to interview, the information we obtained in
these visits and interviews cannot be generalized either to all 21
field locations or to all air marshals in the offices we visited.
However, the visits and interviews provided us a broad overview of
issues important to air marshals.
We reviewed documentation regarding the implementation and results of a
workforce satisfaction survey that FAMS conducted in 2007. Our
engagement team, which included social science analysts with extensive
survey research experience, reviewed the questionnaire used in the
survey for clarity and the related response options for appropriateness
(see app. VI). Also, we discussed with FAMS officials the extent to
which efforts were made to obtain an overall response rate as high as
possible. As criteria to guide our review of the survey results, we
used the following OMB guidance:
* Standards and Guidelines for Statistical Surveys (September 2006).
* Questions and Answers When Designing Surveys for Information
Collections (Jan. 20, 2006).
We conducted this performance audit from April 2007 to December 2008 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: OMB's Program Assessment Rating Tool Results in 2003 and
2008 for the Federal Air Marshal Service:
The Office of Management and Budget's (OMB) Program Assessment Rating
Tool (PART) consists of a standard series of 25 questions intended to
determine the strengths and weaknesses of federal programs. The 25
total questions cover 4 broad topics--(1) program purpose and design,
(2) strategic planning, (3) program management, and (4) program
results/accountability.
This appendix provides an overview of OMB's PART-based assessments of
the Federal Air Marshal Service (FAMS) conducted in 2003 and 2008.
Additionally, for each of the 25 questions used in the PART-based
assessments, the appendix compares OMB's 2003 and 2008 answers and
summarizes OMB's narrative findings (explanation and evidence). Also,
when the answers in 2003 and 2008 differed for a particular question,
the appendix briefly explains the basis for the respective answer.
More detailed information regarding the 2003 and 2008 OMB PART
assessments of FAMS can be found on OMB's Web site: [hyperlink,
http://www.ExpectMore.gov].
Overview of the 2003 and 2008 Assessments of the Federal Air Marshal
Service:
OMB's PART assessments of federal agencies provide performance ratings
that indicate how effectively tax dollars are spent. Following an
assessment, OMB assigns an agency one of five possible overall ratings:
* Effective. Programs rated "effective" set ambitious goals, achieve
results, are well-managed, and improve efficiency.
* Moderately effective. A "moderately effective" rating indicates a
program that sets ambitious goals and is well-managed but needs to
improve its efficiency or address other problems in the programs'
design or management in order to achieve better results.
* Adequate. An "adequate" rating describes a program that needs to set
more ambitious goals, achieve better results, improve accountability,
or strengthen its management practices.
* Ineffective. An "ineffective" rating indicates a program that fails
to use tax dollars effectively and is unable to achieve results because
of a lack of clarity regarding the program's purpose or goals, poor
management, or some other significant weakness.
* Results not demonstrated. A "results not demonstrated" rating
indicates that a program has been unable to develop acceptable
performance goals or collect data to determine whether it is
performing.
In OMB's 2003 PART assessment, FAMS received a rating of "results not
demonstrated" because at that time FAMS did not have measurable
results. Additionally, OMB cited strategic planning deficiencies that
included the absence of baselines, targets, and time frames associated
with performance goals and performance measurements. OMB further noted
the absence of a second long-term outcome measure,[Footnote 62] proxy
measures with respect to deterrence, and an efficiency measure.
In OMB's 2008 PART assessment, FAMS received a rating of "moderately
effective." Regarding the improved rating, OMB recognized the Homeland
Security Institute's independent evaluation, which endorsed FAMS's
concept of operations. Also, OMB noted that FAMS had addressed other
deficiencies by developing the following performance measures:[Footnote
63]
* A second long-term outcome measure--the level of public confidence in
air marshals' ability to promote aviation security--which is reflective
of FAMS's purpose.
* Proxy measures of deterrence, such as air marshals' average annual
rate of accuracy in firearms requalification testing.
* Efficiency measures, such as (a) cost per flight per air marshal and
(b) percentage of air marshals meeting the targeted number of flying
days per year.
Section-by-Section Details of the 2003 and 2008 Assessments of the
Federal Air Marshal Service:
The results of PART's 25 questions in reference to the 2003 and 2008
assessments of FAMS are presented in tables 3 through 6, specifically:
* The 5 questions in table 3 cover program purpose and design.
* The 8 questions in table 4 cover strategic planning.
* The 7 questions in table 5 cover program management.
* The 5 questions in table 6 cover program results/accountability.
As presented in tables 3 through 6, the narrative discussion
(explanation and evidence) is our summary of OMB's key points. If
needed for purposes of clarifying the respective topic or ensuring
accuracy, we used additional or alternative wording to summarize OMB's
findings. Also, in a few instances, we updated the information as
appropriate.
Table 3: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Purpose and Design:
Section 1 questions, explanation, and evidence:
Question 1.1: Is the program purpose clear?;
Explanation and Evidence: The 2008 assessment noted that FAMS's purpose
is to promote confidence in the nation's civil aviation system through
the effective deployment of air marshals to detect, deter, and defeat
hostile acts targeting U.S. air carriers, airports, passengers, and
crews. Also, the assessment noted the following:
* 49 U.S.C. § 114(q)[A] authorizes the Secretary of Homeland Security--
and by delegation, the Administrator of the Transportation Security
Administration (TSA)--to designate employees of TSA as federal law
enforcement officers. The authority provides that these officers may
carry a firearm, make arrests, and seek and execute warrants for arrest
or seizure of evidence;
* Section 1303 of the Implementing Recommendations of the 9/11
Commission Act of 2007 amended the Homeland Security Act of 2002[B] to
authorize the Secretary of the Department of Homeland Security to
develop Visible Intermodal Prevention and Response teams[C] for
augmenting the security of any mode of transportation at any U.S.
location;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 1.2: Does the program address a specific and existing problem,
interest, or need?
Explanation and Evidence: The 2008 assessment stated that President
Bush ordered the rapid expansion of FAMS as a result of the September
11th attacks in order to restore public confidence in air travel and
prevent further attacks. Also, the assessment cited the following:
* In establishing the Department of Homeland Security (DHS), the
Homeland Security Act of 2002 noted that FAMS is critical to aviation
security;
* The National Strategy for Aviation Security categorizes protection of
the air domain as a vital national interest and cites federal air
marshal presence on commercial passenger aircraft as among the efforts
responsible for enhancing aviation security on an ongoing basis;
* A message to DHS employees from Secretary Chertoff, dated August 10,
2006, stated that FAMS would expand mission coverage for U.K.-U.S.
flights as a component of the U.S. government's response to the
Heathrow plot[D] and DHS's responsibility to protect travelers;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 1.3: Is FAMS designed so that it is not redundant or
duplicative of any federal, state, local, or private effort?
Explanation and Evidence: Regarding the "No" answer, the 2008
assessment indicated that the design of aviation security includes
layers of redundancies and duplicate security measures to reinforce
each and every layer for the strongest level of security. However, the
assessment also noted the following:
* As a critical layer of aviation security, FAMS fills an important
role in detecting, deterring, and defeating criminal terrorist
activities on aircraft. For international flights, only air marshals
are authorized to fly armed;
* Although the presence of other armed law enforcement officers and
Federal Flight Deck Officers[E] on domestic flights may at times
augment air marshals, the distinctive differences inherent in the FAMS
program prevent redundancy or excessive overlap;
Answer: 2003: Yes;
Answer: 2008: No.
Question 1.4: Is the program design free of major flaws that would
limit the program's effectiveness or efficiency?
Explanation and Evidence: According to the 2003 assessment, key aspects
of the program design needed to be independently assessed and
validated. The 2008 assessment recognized that the Homeland Security
Institute conducted an independent evaluation (document is classified)
of the development and implementation of FAMS's concept of operations
in 2006. Given FAMS's objectives and performance goals, the institute
found that FAMS applies a valid approach to analyzing risk and
allocating resources on the basis of threat, vulnerability, and
consequence;
Answer: 2003: No;
Answer: 2008: Yes.
Question 1.5: Is the program effectively targeted, so program resources
reach intended beneficiaries and/or otherwise address the program's
purpose directly?
Explanation and Evidence: According to the 2003 assessment, FAMS
considered the entire flying public to be the intended beneficiaries of
the air marshal security program. According to the 2008 assessment, the
application of FAMS's concept of operations effectively targeted
resources by prioritizing and scheduling missions on the basis of risk.
The assessment noted that FAMS structures deployments so that program
benefits reach highest risk beneficiaries at the micro level, while
deterrence as a macro-level benefit reaches the flying public
(collectively) and other intended beneficiaries;
Answer: 2003: NA[F];
Answer: 2008: Yes.
Source: OMB's 2003 and 2008 PART reviews of FAMS.
[A] The provision cited, which authorizes the Under Secretary of
Transportation for Security (now the Assistant Secretary, TSA) to
designate employees of TSA or other federal agencies to serve as law
enforcement officers, has subsequently been redesignated §114(p).
[B] Section 1303 does not specifically amend the Homeland Security Act
but does provide specific authority to the Secretary, acting through
the Assistant Secretary, TSA, to establish Visible Intermodal
Prevention and Response teams.
[C] After the March 2004 train bombings in Madrid, TSA created and
deployed Visible Intermodal Prevention and Response teams to enhance
security on U.S. rail and mass transit systems nationwide. Comprised of
federal air marshals, surface transportation security inspectors,
transportation security officers, behavior detection officers, and
explosives detection canines, the teams are intended to work with local
security and law enforcement officials to supplement existing security
resources, provide a deterrent presence and detection capabilities, and
introduce an element of unpredictability to disrupt potential terrorist
activities.
[D] The Heathrow plot was a terrorist plan to blow up 10 commercial
airplanes bound for the United States from the United Kingdom. On
August 10, 2006, British law enforcement arrested 24 suspects regarding
the plot to smuggle liquid explosive materials aboard the planes and
assemble bombs.
[E] TSA developed the Federal Flight Deck Officer Program as an
additional layer of aviation security. See Pub. L. No. 107-296, §
1402(a), 116 Stat. 2135, 2300-05 (2002) (codified as amended at 49
U.S.C. § 44912). Under the program, eligible flight crew members--a
pilot, flight engineer, or navigator assigned to the flight--are
authorized to use firearms to defend against an act of criminal
violence or air piracy attempting to gain control of an aircraft.
Federal Flight Deck Officers are trained by FAMS on the use of
firearms, use of force, legal issues, defensive tactics, the psychology
of survival, and program standard operating procedures.
[F] Not applicable.
[End of table]
Table 4: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Strategic Planning:
Section 2 questions, explanation, and evidence:
Question 2.1: Does the program have a limited number of specific long-
term performance measures that focus on outcomes and meaningfully
reflect the purpose of the program?
Explanation and Evidence: The 2008 assessment noted that FAMS developed
two long-term outcome measures:
* The number of successful terrorist and other criminal attacks
initiated from commercial passenger aircraft cabins on flights in
identified risk categories;
* The level of public confidence in air marshals' ability to promote
aviation security;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 2.2: Does the program have ambitious targets and time frames
for its long-term measures?
Explanation and Evidence: According to the 2003 assessment, FAMS's long-
term measures were under development. The 2008 assessment recognized
FAMS's two defined targets:
* The number of successful terrorist and other criminal attacks
initiated from commercial passenger aircraft cabins on flights in
identified risk categories. OMB noted that this target was ambitious
because it quantified zero for all time frames;
* The level of public confidence in air marshals' ability to promote
aviation security. OMB noted that FAMS quantified ambitious targets for
a 5-year time frame;
Answer: 2003: No;
Answer: 2008: Yes.
Question 2.3: Does the program have a limited number of specific annual
performance measures that demonstrate progress toward achieving the
program's long-term measures?
Explanation and Evidence: The 2008 assessment stated that FAMS has two
annual measures:
* The average annual rate of accuracy in air marshals' firearms
requalification. Because a very high level of firearms proficiency is
required for air marshals to successfully defeat terrorist and criminal
attacks onboard commercial passenger aircraft, the associated rate of
accuracy is one proxy outcome for the number of successful attacks;
* The percentage level in meeting FAMS's coverage targets for each
individual category of identified risk.[A] With U.S. air carriers
providing over 29,000 flights daily, the ability of FAMS to identify
high-risk flights for air marshal deployment is one proxy outcome for
the number of successful attacks;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 2.4: Does the program have baselines and ambitious targets and
time frames for its annual measures?
Explanation and Evidence: According to the 2003 assessment, FAMS's
annual measures were under development. The 2008 assessment noted the
following:
* FAMS has the highest firearm requirement for rate of accuracy (85
percent) of any federal law enforcement agency. No other federal law
enforcement agency is known to afford its officers a smaller margin of
error (15 percent) on its agencywide course of fire. Further, FAMS
strives for an even higher standard--that is, to achieve an average
annual practical pistol course score of 95 percent among all air
marshals;
* FAMS established baselines and annual targets to measure the
percentage of flights (in the various risk categories) covered by air
marshals. The risk categories are based on vulnerability, threat, and
consequence. The flight-coverage targets are highly ambitious (never
striving for less than 100 percent) because FAMS, in accordance with
its concept of operations, pursues flight coverage in the risk
categories as a high priority;
Answer: 2003: No;
Answer: 2008: Yes.
Question 2.5: Do all partners (including grantees, sub-grantees,
contractors, cost-sharing partners, etc.) commit to and work toward the
annual and/or long-term goals of the program?
Explanation and Evidence: The 2008 assessment noted the following:
* Air carriers and foreign air marshal programs are key partners upon
whom FAMS relies to support performance goals. FAMS interacts daily
with the airlines to ensure that both planned and emergent reservation
requirements are met in order to ensure both ticketing and tactically
optimal seating for air marshals on high-risk flights;
* Section 4017 of the National Intelligence Reform Act of 2004[B]
encouraged President Bush to pursue aggressively international
agreements with foreign governments to allow the maximum deployment of
air marshals on international flights. As a consequence of numerous
such bilateral agreements, U.S. air marshals cover U.S. flights into
and out of multiple foreign countries. In some instances, the partner
country has air marshals, whose deployment into the United States on
foreign airlines is also provided for in the bilateral agreement. These
bilateral partnerships reduce the threat of terrorism against aircraft
of both nations;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 2.6: Are independent and quality evaluations of sufficient
scope and quality conducted on a regular basis or as needed to support
program improvements and evaluate effectiveness and relevance to the
problem, interest, or need?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough for its effectiveness to be assessed.
OMB recommended that FAMS obtain an independent evaluation of its
methods for risk analysis and resource allocation. The 2008 assessment
noted that FAMS contracted with the Homeland Security Institute for an
evaluation and that the institute's report, titled Federal Air Marshal
Service Risk Assessment, was completed in July 2006. According to OMB,
the institute's independent evaluation was comprehensive in its scope;
Answer: 2003: NA[C];
Answer: 2008: Yes.
Question 2.7: Are budget requests explicitly tied to accomplishment of
the annual and long-term performance goals and are the resource needs
presented in a complete and transparent manner in the program's budget?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough to present the resource needs in a
complete and transparent manner in FAMS's budget. According to the 2008
assessment, FAMS did not yet meet the requirement to put forth budget
justifications that define explicitly the relationship between
increments/decrements in program performance and increases/decreases in
funding levels. OMB noted that FAMS is working with independent experts
of the Homeland Security Institute to develop a means of transparently
quantifying and communicating the performance implications (outputs and
outcomes) that various resource levels (inputs) would drive;
Answer: 2003: NA[C];
Answer: 2008: No.
Question 2.8: Has the program taken meaningful steps to correct its
strategic planning deficiencies?
Explanation and Evidence: According to the 2003 assessment, FAMS
received the rating of "results not demonstrated" because FAMS was
unable to develop acceptable performance goals or collect data to
determine whether it was performing. Specifically, OMB cited strategic
planning deficiencies that included the absence of baselines, targets,
and time frames associated with performance goals and performance
measurements. Additionally, the 2003 assessment cited the absence of a
second long-term outcome measure, proxy measures with respect to
deterrence, and efficiency measures;
* According to the 2008 PART review, FAMS took steps to correct these
strategic planning deficiencies by developing the following:
- A second long-term outcome measure--the level of public confidence in
air marshals' ability to promote aviation security--which is reflective
of FAMS's purpose;
- Proxy measures of deterrence, such as air marshals' average annual
rate of accuracy in firearms requalification testing;
- Efficiency measures, such as (a) cost per flight per air marshal and
(b) percentage of air marshals meeting the targeted number of flying
days per year;
- Baselines, targets, and timelines associated with performance goals
and performance measurements;
Answer: 2003: Yes;
Answer: 2008: Yes.
Source: OMB's 2003 and 2008 PART reviews of FAMS.
[A] In light of recent OMB guidance regarding the identification and
constitution of proxy measures, FAMS believes that the coverage for
categories of risk, while not a stand-alone outcome measure, is
actually a "proxy" outcome for the number of successful attacks
measure. That is, the ability to identify and deploy air marshals on
higher risk flights reduces the likelihood of a catastrophic outcome.
[B] The provision cited is part of the Intelligence Reform and
Terrorism Prevention Act of 2004.
[C] Not applicable.
[End of table]
Table 5: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Management:
Section 3 questions, explanation, and evidence:
Question 3.1: Does the agency regularly collect timely and credible
performance information, including information from key program
partners, and use it to manage the program and improve performance?
Explanation and Evidence: According to the 2008 assessment, FAMS
collects performance information every roster period (the period of
mission scheduling that corresponds to two pay periods) and applies the
optimal volume of air marshals where the most risk lies. Also, OMB
noted that, with respect to partners, FAMS actively requests from
international air marshal programs their increased coverage on foreign-
flagged flights to the United States;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 3.2: Are federal managers and program partners (grantees, sub-
grantees, contractors, cost-sharing partners, etc.) held accountable
for cost, schedule, and performance results?
Explanation and Evidence: According to the 2008 assessment:
* FAMS's key managers are subject to the departmentally standardized
DHS senior executive service pay-for-performance system. More
specifically, the FAMS Director, Deputy Director, Assistant Directors,
senior executive service-level Deputy Assistant Directors, and senior
executive service-level Special-Agents-in-Charge each enter into an
annual executive performance agreement under that system;
* The performance agreements serve as vehicles for identifying (in
writing) the managers who are responsible for achieving key program
results and for clearly defining each manager's standards of
accountability for achieving those results. FAMS's contract awards and
renewals do consider past performance;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 3.3: Are all funds (federal and partners') obligated in a
timely manner and spent for the intended purpose?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough to assess obligation data on a timely
basis. The 2008 assessment noted the following:
* According to sample fund status reports, the fiscal year 2007
financial plan, and the unobligated balance chart, FAMS obligated its
funds in a timely manner and for intended purposes;
* FAMS's financial analysts reviewed and certified all procurement
requests to ensure that funds were available and that planned actions
comply with established appropriation law standards;
* As a result of these steps, FAMS achieved annual obligation rates of
at least 99 percent;
Answer: 2003: NA[A];
Answer: 2008: Yes.
Question 3.4: Does the program have procedures (e.g., competitive
sourcing/cost comparisons, information technology improvements,
appropriate incentives) to measure and achieve efficiencies and cost
effectiveness in program execution?
Explanation and Evidence: The 2008 assessment noted that FAMS gauges
efficiency and cost effectiveness by measuring the following:
* The percentage of federal air marshals providing the targeted number
of flying days per year;
* The cost per flight per air marshal;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 3.5: Does the program collaborate and coordinate effectively
with related programs?
Explanation and Evidence: The 2008 assessment noted the following:
* Air marshals, TSA security officers (i.e., behavior detection
officers), and airport police departments use FAMS's Tactical
Information Sharing System to report suspicious activities;
* Additionally, managing overseas air marshals' deployments requires
strong partnerships with both U.S. government entities (e.g.,
Department of State, U.S. Embassy, and TSA Transportation Security Area
Representative) and each foreign locality's host government
(particularly foreign law enforcement authorities). Partnerships
address the diplomatic, security, and logistical matters required to
accomplish FAMS's international mission deployments;
Answer: 2003: Yes;
Answer: 2008: Yes.
Question 3.6: Does the program use strong financial management
practices?
Explanation and Evidence: According to the 2003 assessment, FAMS
appeared to manage financial resources properly. The 2008 assessment
stated that FAMS and TSA use strong financial management practices;
however, in the annual DHS audit (for which DHS has contracted with
KPMG), the auditors reported two TSA-wide material control
weaknesses[B] for fiscal year 2007; one of these material control
weaknesses, regarding undelivered order balances, involved FAMS;
Answer: 2003: Yes;
Answer: 2008: No.
Question 3.7: Has the program taken meaningful steps to address its
management deficiencies?
Explanation and Evidence: The 2008 assessment noted that FAMS
participates in a system for evaluating and correcting program-level
management deficiencies. Specifically:
* The program's field offices undergo, on a 3-year cycle, inspections
by the TSA Office of Inspection, whose mission is to ensure the
integrity and effectiveness of TSA employees and programs;
* The inspectors determine whether management deficiencies are present,
and recommend courses of action to address any deficiencies identified;
Answer: 2003: Yes;
Answer: 2008: Yes.
Source: OMB's 2003 and 2008 PART reviews of FAMS.
[A] Not applicable.
[B] A material weakness is a significant deficiency or combination of
significant deficiencies that results in more than a remote likelihood
that a material misstatement of the financial statements will not be
prevented or detected.
[End of table]
Table 6: OMB's 2003 and 2008 PART Review of FAMS--Detailed Assessment
of Program Results/Accountability:
Section 4 questions, explanation, and evidence:
Question 4.1: Has the program demonstrated adequate progress in
achieving its long-term outcome performance goals?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough to assess its long-term outcome
performance goals. The 2008 assessment noted that:
* FAMS made substantial and measurable progress in achieving its long-
term performance goals to (1) detect, deter, and defeat hostile acts
targeting U.S. air carriers, airports, passengers, and crews and (2)
improve confidence in the security of our nation's civil aviation
system;
* FAMS met its targets for the number of successful terrorist and other
criminal attacks initiated from commercial passenger aircraft cabins on
flights in identified risk categories. The absence of these successful
attacks is the intended result of the program's activities and is of
direct, significant benefit to the public;
* With respect to its public confidence goal, the program has some
measurable evidence of public confidence in air marshals;
Answer: 2003: No;
Answer: 2008: Small extent.
Question 4.2: Does the program (including program partners) achieve its
annual performance goals?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough to achieve its annual performance
goals. The 2008 assessment noted the following regarding annual
performance goals:
* The percentage level in meeting FAMS's coverage targets for each
individual category of identified risk; FAMS exceeded its target in
fiscal year 2006 and performed under its target in fiscal year 2007;
* The average annual rate of accuracy in federal air marshals' firearms
requalification scores is a new measure, baselined in fiscal year 2007.
FAMS met the target in the first and second quarters of fiscal year
2008; Answer: 2003: No; Answer: 2008: Small extent.
Question 4.3: Does the program demonstrate improved efficiencies or
cost effectiveness in achieving program performance goals each year?
Explanation and Evidence: According to the 2003 assessment, FAMS had
not been in existence long enough to demonstrate improved efficiencies
or cost effectiveness. The 2008 assessment noted that FAMS demonstrated
cost efficiency by optimizing the allocation of federal air marshal
teams to missions;
Answer: 2003: No;
Answer: 2008: Large extent.
Question 4.4: Does the performance of this program compare favorably to
other programs, including government, private, etc., that have a
similar purpose and goals?
Explanation and Evidence: According to the 2003 assessment, OMB
directed FAMS to have an independent evaluation of its program
performance with respect to flight coverage risk categories, the
distribution of covered flights, and target levels of coverage.
According to the 2008 assessment, the Homeland Security Institute's
independent evaluation of FAMS considered other programs whose main
business line is deterrence and/or protection via risk management.
According to the institute, no other organizations that face a similar
challenge apply significantly better tools or methodologies;
Answer: 2003: No;
Answer: 2008: Yes.
Question 4.5: Do independent and quality evaluations of this program
indicate that FAMS is effective and achieving results?
Explanation and Evidence: As noted previously, following the 2003
assessment, OMB directed FAMS to have an independent evaluation of
program performance. The 2008 assessment noted that the Homeland
Security Institute conducted an independent evaluation of FAMS in 2006
and determined that FAMS ensures effective risk reduction by applying a
reasonable and valid approach to analyzing risk and allocating
resources. Additionally, the assessment noted that an interagency
steering group--convened by the Homeland Security Institute and
consisting of representatives from an array of law enforcement and
counter-terrorism agencies--also considered FAMS's concept of
operations to be reasonable;
Answer: 2003: No;
Answer: 2008: Large extent.
Source: OMB's 2003 and 2008 PART reviews of FAMS.
[End of table]
[End of section]
Appendix III: Excerpts from the Homeland Security Institute's July 2006
Report:
The Homeland Security Institute, a federally funded research and
development center, was established to assist the Department of
Homeland Security in addressing relevant issues that require
scientific, technical, and analytical expertise. The institute--after
conducting an evaluation of the Federal Air Marshal Service's approach
for achieving the agency's core mission of providing an onboard
security presence for flights operated by U.S. commercial passenger air
carriers--issued its final report in July 2006.
This appendix presents quoted excerpts that substantially replicate the
executive summary in the Homeland Security Institute's July 2006
report.
Background and Tasking for an Independent Evaluation:
"The Federal Air Marshal Service (FAMS) challenge to reduce risk in the
aviation domain is daunting. U.S. commercial passenger carriers make
roughly 28,000 domestic and international flights each day. These
flights canvas the globe and originate, terminate, or fly in proximity
to thousands of critical facilities. The FAMS must evaluate which
flights it will defend and to what extent. It cannot cover every
flight.
"In response to an Office of Management and Budget (OMB) direction, the
FAMS asked HSI [Homeland Security Institute] for an independent
evaluation of its methods for analyzing risk and allocating resources.
In particular, it asked HSI to determine if its risk management
processes and the application of its concept of operations (CONOPS) to
scheduled commercial flights were valid."
Evaluation Approach Used by the Homeland Security Institute:
"We defined 'validation' as a test of whether or not the FAMS risk
management processes and the outcome of those processes are reasonable
and consistent externally with stated guidance and internally with its
own CONOPS. Our analysis involved three tasks. First, we examined the
conceptual basis for the FAMS approach to risk analysis. Second, we
examined the FAMS scheduling process and analyzed the output of that
process in the form of 'coverage' data, i.e., when and where air
marshals were deployed on flights. Third, we developed and employed a
basic quantitative model to study the implications of alternative
strategies for assigning resources."
Findings Reported by the Homeland Security Institute:
"Based on our analysis, we find that the FAMS applies a valid approach
to analyzing risk and allocating resources. In particular, its approach
is reasonable given the scarcity of resources and the guidance it has
received. It assesses risk as a function of threat, vulnerability, and
consequence and employs a filtering process along with an allocation
tool to optimize resource allocation. Moreover, the FAMS seeks to
strengthen risk management processes by improving its scheduling tools
and analytical techniques. We did not find any other organizations that
face a similar challenge and apply significantly better methodologies
or tools."
Additional Considerations Reported by the Homeland Security Institute:
"During our analysis, we identified [the following] five issues that
the FAMS should address itself or in conjunction with the
Transportation Security Administration (TSA), the Department of
Homeland Security (DHS), and the broader intelligence and security
communities."
Vulnerability:
"The FAMS definition of vulnerability … is inconsistent with
traditional risk-based definitions, which focus on the probability that
an attack will succeed. It shifts the focus away from other potential
vulnerabilities. [Bold text: We recommend that the FAMS reconsider its
approach to vulnerability and engage the aviation security community on
this issue."; end bold text] (The emphasis is in the original.)
Filtering Process:
"The FAMS understanding of consequence and its subsequent 'filtering'
process ...bias its allocation decision. To focus limited resources,
the FAMS filters flights according to ...[various factors].
"Guidance in the form of legislation and departmental memoranda
following 9/11 directed FAMS to focus on flights that present 'high
security risks.' But, ultimately, that guidance was ambiguous and could
be outdated. [Bold text: These fundamental assumptions concerning risk,
on which it [FAMS] allocates resources, warrant interagency review by
the broader intelligence and security community."; end bold text] (The
emphasis is in the original.)
Allocation Process:
"The FAMS filtering process defines 'high risk' and directs its efforts
toward flights fitting those characteristics. Its allocation process--
a modified version of the SABRE software used by airlines to schedule
flight crews--attempts to cover the maximum number of high risk flights
within fixed resources. … The scheduling tool requires manual
involvement to recognize and modify scheduling solutions, which may not
be consistent with effective risk reduction.
"Our analysis of one month of FAMS coverage data reveals ...[some
concerns.] [Bold text: To compensate for a lack of resources and deny
predictability, the FAMS should integrate randomness into its
allocations."; end bold text] (The emphasis is in the original.)
"Contrary to the popular use of the term 'random,' allocating resources
in such a way does not mean choosing them haphazardly or without a
plan. The overall probability distribution for a group of comparable
aircraft can be chosen based on risk analysis. For instance, the FAMS
may choose to cover [a determined percentage of] flights in and out of
[a particular geographic region]. But the tactical allocation decision
concerning a specific flight must be random and converge around the
overall category average over time. A terrorist group may be able to
discern the overall category average through effective, long-term
surveillance but will never know conclusively whether or not the flight
it plans to hijack will be covered on a particular day."
Performance Measures:
"The FAMS primary performance measure--average coverage rates--can mask
weaknesses in coverage patterns. In particular, they can mask a
situation in which certain flights within a category of comparable
flights are heavily covered while others are rarely if ever covered.
[Bold text: Accordingly, the FAMS should develop performance measures
to track coverage consistency.; end bold text] One example involves
tracking coverage deviation, defined as the average difference between
the individual coverage rates of each flight in a comparable category
and the overall category coverage rate." (The emphasis is in the
original.)
Decision Support Tools:
"During the course of our analysis, we noted that FAMS decision makers
did not have a tool for evaluating the resource implications of
different threat scenarios and alternative coverage schemes. The SABRE
scheduler is not flexible enough to support quick-response analyses.
[Bold text: The FAMS should build a simple decision-support tool, along
the lines of the model we developed based on risk balancing, to
facilitate a system-wide view of resource decisions."; end bold text]
(The emphasis is in the original.)
"Such a tool would augment, not replace its scheduling tool, by
allowing decision makers to look across the entire aviation system and
investigate the resource implications of alternative allocation
scenarios. In other words, how easily could the FAMS adapt to a
different threat environment? Could it significantly increase the
number of marshals aboard specific flights? Where might those resources
come from? How would these changes affect coverage elsewhere?"
[End of section]
Appendix IV: Performance Measures for Assessing the Federal Air Marshal
Service:
In 2003, the Office of Management and Budget (OMB) used its Program
Assessment Rating Tool (PART) to assess the management and performance
of the Federal Air Marshal Service (FAMS).[Footnote 64] At that time, a
key performance measure for assessing FAMS was based on coverage of
targeted critical flights under various risk categories. This measure
is still applicable currently; however, its designation has been
changed from an output measure[Footnote 65] to an outcome measure.
[Footnote 66] Moreover, this performance measure--the coverage of
targeted critical flights--is now considered a proxy indicator[Footnote
67] regarding air marshals' ability to defeat an attempted attack.
Also, in further response to the findings of OMB's 2003 assessment,
FAMS established two additional outcome measures, one of which serves
as another proxy indicator of air marshals' ability to defeat an
attempted attack:
* The additional proxy outcome measure is the average annual rate of
accuracy in air marshals' firearms requalification testing.
* The additional, non-proxy outcome measure is based on a national
survey of households to determine the level of public confidence in air
marshals' ability to promote aviation security.
These updated measures have been approved by the Department of Homeland
Security--and also were approved in 2008 by OMB during its PART-based
reassessment of FAMS. An overview of FAMS's updated performance
measures is presented in table 7.
Table 7: Overview of the Federal Air Marshal Service's Key Performance
Measures (as of Fiscal Year 2008):
Performance measure: Number of successful terrorist and other criminal
attacks initiated from commercial passenger aircraft cabins in
identified risk categories;
Type of measure[A]: Outcome;
Term: Long-term;
Description: FAMS expects that, on 100 percent of covered flights, air
marshals will detect and defeat terrorist or criminal attackers to
prevent hostile control of the aircraft. By virtue of meeting this
measure's targets, air marshals deter onboard attacks. The annual
target for this measure is zero successful attacks. Since September 11,
2001, to date, no attacks have occurred on board covered flights.
Performance measure: Percentage level in meeting coverage targets for
each individual category of identified risk;
Type of measure[A]: Outcome;
Term: Annual;
Description: This measure reflects the performance levels of FAMS
coverage of targeted critical flights based upon a risk-based
management approach involving impact (geographical location),
vulnerability (aircraft destructive potential), threats, and
intelligence relative to the availability of resources. FAMS deploys
air marshals to targeted critical flights under 10 individual risk
categories that are specified in FAMS's concept of operations.
According to OMB, this outcome measure is a proxy indicator regarding
the number of successful attacks performance measure, given that
deployment of air marshals on higher risk flights reduces the
likelihood of a catastrophic attack. For this measure, FAMS has
established its target to be 100 percent coverage of identified high-
risk flights in each risk category. FAMS exceeded this goal in fiscal
year 2006 (101.7 percent) and fell short of the target in fiscal year
2007 (96.2 percent) because of an increased emphasis on international
flights resulting from the August 2006 terrorist plot to attack flights
from the United Kingdom to the United States.
Performance measure: Average annual rate of accuracy in federal air
marshals' firearms requalification;
Type of measure[A]: Outcome;
Term: Annual;
Description: As a requirement for the job, each individual air marshal
must score at least 85 percent on quarterly firearms requalification
testing. However, as a stretch or ambitious goal, FAMS established a
baseline precision rate of 95 percent in 2007, and FAMS is retaining
this rate as an annual goal through 2011. According to OMB, the
firearms precision requirement is reflective of FAMS's ultimate ability
to defeat an attempted attack. Thus, as a second proxy outcome measure,
FAMS calculates an average, annualized precision rate that encompasses
the firearms requalification scores of all air marshals.[B]
Performance measure: Level of public confidence in federal air
marshals' ability to promote aviation security;
Type of measure[A]: Outcome;
Term: Long-term/annual;
Description: Beginning in fiscal year 2008, the U.S. Department of
Transportation's Bureau of Transportation Statistics included in its
National Household Transportation Survey a question regarding
respondents' confidence in federal air marshals' ability to defend an
aircraft and passengers from individuals with hostile intentions.
Performance measure: Percentage of air marshals meeting the targeted
number of flying days per year;
Type of measure[A]: Efficiency;
Term: Annual;
Description: In the spring of 2007, FAMS implemented a policy to target
the number of flying days per air marshal to 18 days per 28-day roster
period and 200 days per year. As a measure of efficiency for fiscal
year 2007, FAMS established a target of 75 percent of all air marshals
meeting the expected number of annual flying days; the target is to
increase to 77.5 percent by fiscal year 2011. According to OMB, the
actual rate achieved in fiscal year 2007 was 74 percent.
Performance measure: Cost per flight per air marshal;
Type of measure[A]: Efficiency;
Term: Long-term/annual;
Description: This measure reflects FAMS's total annual costs (full-time
equivalent personnel costs, per diem costs, and program management
costs) divided by the number of flights with an on-board air marshal
presence during the year and further divided by the number of air
marshals per flight. The result is the cost to the government per
flight per air marshal.[C]
Source: Office of Management and Budget.
[A] OMB defines an "efficiency" measure as one that captures a
program's ability to carry out its activities and achieve results (an
outcome or output) relative to resources (an input such as cost).
[B] The firearms requalification proxy measure is reflective of the
ability of air marshals to defeat an attack on board an aircraft, which
is the parent outcome measure.
[C] OMB did not publish the targeted and actual costs making up this
efficiency measure because the data are considered sensitive security
information.
[End of table]
[End of section]
Appendix V: Working Groups Formed by the Federal Air Marshal Service to
Address Various Issues:
In March 2006, the Director of the Federal Air Marshal Service (FAMS)
communicated to employees his intention to establish working groups to
examine a variety of issues ranging from mission, organizational, and
operational topics to workforce satisfaction and quality-of-life
concerns. Two months later, in May 2006, the director communicated to
FAMS employees that 12 working groups had been established, with each
chaired by a field office special agent-in-charge (SAC) and that
subject matter experts from the field and headquarters were available
to assist in an advisory role.
Subsequently, the number of working groups increased to a total of 36.
Table 8 categorizes the 36 working groups and briefly summarizes the
purposes of each.
Also, regarding the status of the 36 working groups as of October 1,
2008, FAMS officials reported the following (see notes to table 8):
* 18 working groups (table note a): Each of these working groups had
completed its work and given a final report to FAMS management. Each
report had been reviewed by FAMS executives and then distributed to
agency employees via a broadcast message from the FAMS Director. If
applicable, the broadcast message also presented management's responses
to any recommendations made by the respective report.
* 9 working groups (table note b): Each of these working groups had
completed its work and given a final report to FAMS management. The
reports were undergoing review by FAMS executives.
* 5 working groups (table note c): Each of these working groups had yet
to complete its work and give a final report to FAMS management.
* 4 working groups (note d): Each of these working groups is to remain
ongoing. As such, final reports are not expected to be issued; rather,
each group will present its findings when applicable and by appropriate
means.
Table 8: Overview of Federal Air Marshal Service Working Groups and
Their Purposes:
Issue categories: Mission, organizational, and operational issues;
Working groups and their purposes (first meeting date):
Corporate Analysis Working Group[A] (April 4, 2006);
* Review and evaluate the current processes and structure of FAMS;
* Develop a recommended organizational structure to address a broader
range of responsibilities transferred to FAMS based on the recent
Transportation Security Administration functional integration effort;
* Position the organization to achieve its strategic objectives over a
5-year period.
Building a Law Enforcement Culture Working Group[B] (May 3, 2006);
* Assess FAMS's organizational culture;
* Determine if existing culture meets expectations of employees and
stakeholders;
* Develop recommendations to promote and sustain a culture that
represents the values, customs, and traditions that best reflect
organizational expectations;
* Provide recommendations to best position FAMS for sustained growth
and operations during the next 5-year period.
Process Management Working Group[A] (May 16, 2006);
* Review and analyze the current process management components of FAMS
to include strategic planning efforts, investment management, financial
management, and administrative practices;
Operational Management Issues Working Group[B] (May 17, 2006);
* Evaluate operational options in acknowledgment of the expanded role
of FAMS within the Department of Homeland Security and the
Transportation Security Administration, in the context of risk-based
management and in coordination with federal, state, and local law
enforcement counterparts;
Project Management Working Group[B] (May 24, 2006);
* Examine the project management process and the tools and technology
available to improve or enhance the process;
* Examine how goals, objectives, and strategies are identified and how
milestones are established and monitored;
Tactical Policy Working Group[A] (June 21, 2006);
* Examine and review FAMS's tactical policies and procedures to include
boarding procedures, dress code policies, transit within the airport
environment, and hotel-choice policies for overnight stays;
* Develop enhancements or revisions that are needed to FAMS's current
policies or operational procedures;
Federal Flight Deck Officer Working Group[B] (August 1, 2006);
* Review and analyze the existing focus committee recommendations for
credentials/badges, checkpoint requirements, weapon issues (including
transport, storage, and qualification), communication protocols,
training, liaison issues with the airline associations and carriers,
and develop associated implementation strategies;
Firearms Working Group[A] (September 6, 2006);
* Conduct research and analysis to define the requirements for a future
firearm acquisition to include the possibility of recommending an
alternative firearm or a firearm to accommodate the range of federal
air marshal physical sizes;
* Review the ammunition that is currently used for the duty firearm;
* Examine the off-duty firearm requirements and policy;
* Review the current firearm policies and procedures and determine if
any revisions are required;
Training Manual Working Group[C] (October 24, 2006);
* Examine various training instruction documentation and, if
appropriate, recurrent instruction practices at select field offices;
Explosives Security Specialist Working Group[A] (October 31, 2006);
* Discuss the explosives security specialists' current duties,
responsibilities, and training requirements and any other associated
issues within the Explosives Division and the field offices;
* Determine if any of the recently acquired programs within FAMS would
benefit from involvement from the explosives specialists and, in turn,
benefit the explosives specialists with enhanced experience;
* Develop a series of recommended proposals to further define the
explosives security specialists' duties and responsibilities;
Canine Program Working Group[B] (December 5, 2006);
* Review and discuss canine program operations, including the following
elements: (1) canine team composition and operations for deployment at
applicable airports; (2) cooperative agreement content; (3) roles of
the Federal Security Directors,[E] the Assistant Federal Security
Director for Law Enforcement, and SACs regarding canine team operations
and coordination; and (4) additional issues as determined by working
group members;
SAC Advisory Council Working Group[D] (December 6, 2006);
* Examine and discuss a wide range of issues that impact FAMS's
operations and provide recommended actions to the FAMS executive staff.
[Note: Agency documentation explained that members of the SAC Advisory
Council are uniquely positioned to provide extensive knowledge of
operational issues that are critically important to executing the
mission of FAMS.];
Explosives Security Specialist Sub-Working Group[A] (August 28, 2008);
* Review recommendations previously made by the Explosives Security
Specialist Working Group;
International Mission Working Group[A] (February 6, 2008);
* Examine and discuss FAMS's international deployment procedures and
concepts currently in use;
Joint Terrorism Task Force Working Group[C] (March 6, 2008);
* Examine and discuss the various procedures and issues pertaining to
FAMS personnel serving on Joint Terrorism Task Force assignments;
Scheduling Practices and Operations Working Group[C] (8/5/08);
* Review and examine various scheduling practices and protocols.
Issue categories: Technology issues;
Working groups and their purposes (first meeting date):
Technology Working Group[B] (May 24, 2006);
* Examine FAMS's technological initiatives and requirements to ensure
that the organization's needs are met now and in the future;
Operational Communications Group[D] (November 9, 2007);
* Examine FAMS's operational and technical requirements for land mobile
radio communications to support a number of initiatives, including
Visible Intermodal Prevention and Response teams and other operations.
Issue categories: Human capital issues;
Working groups and their purposes (first meeting date):
Workforce Satisfaction, Recruitment, and Retention Working Group[A]
(April 12, 2006);
* Examine FAMS's recruitment and retention practices and develop
strategies and recommendations to enhance those efforts;
* Identify key issues that affect FAMS's employee workforce
satisfaction and morale issues as the organization continues to mature
over a 5-year period;
Quality of Life, Scheduling, and Transfers Working Group[A] (April 25,
2006);
* Examine the quality-of-life and associated issues involving mission
scheduling and manpower issues;
* Examine the current transfer/change-of-station program.
Career Path and Promotional Opportunities Working Group[A] (May 2,
2006);
* Examine and assess the career path/promotional opportunity process
for the federal air marshal and civilian population of FAMS;
* Identify recommendations to further develop/enhance career
development and promotional opportunities within the organization;
* Examine managerial succession planning efforts and develop
recommendations to best position FAMS for sustained growth and
operations during the next 5-year period;
Recruitment and Retention of Women & Family-Friendly Initiatives
Working Group[A] (Aug. 9, 2006);
* Identify and analyze those issues that impact the recruitment and
retention of women;
* Identify creative family-friendly initiatives that will positively
impact quality of life and retention of employees;
Mission Support Employee Working Group[B] (October 24, 2006);
* Examine various issues and policies that specifically pertain to
civilians and range from the current structure of mission support
employee positions to professional development training to enhance
recruitment and retention;
Hardship Transfer Policy Working Group[D] (October 25, 2006);
* Develop a hardship transfer policy that is (1) consistent, fair, and
non-discriminatory and (2) tailored for the needs of a law enforcement
organization and its officers;
Awards and Recognition Working Group[A] (November 7, 2006);
* Review the current awards and recognition programs and policies of
the Transportation Security Administration and FAMS;
* Develop recommended actions, as appropriate, to modify the current
policies and practices;
Diversity Working Group[A] (January 17, 2007);
* Discuss and examine the range of diversity-related concepts and
issues that impact FAMS's personnel and operations;
* Focus on the continued development of a culture within FAMS that
leverages the diversity of the workforce and identifies and utilizes
the range of employee talent to carry out FAMS' mission/operations;
Employee Outreach Working Group[C] (February 2, 2007);
* Examine critical incident stress management models for law
enforcement to explore their potential benefit for implementation
within FAMS;
* Analyze traumatic incidents (specifically suicides) that have
occurred to date within FAMS to identify commonalities and lessons
learned across the cases, if any;
* Develop specific goals, objectives, and parameters associated with an
organization-wide resource;
* Consult relevant scientific literature and professional expertise.
Issue categories: Communications and relationships issues;
Working groups and their purposes (first meeting date):
Communications Working Group[A] (June 13, 2006);
* Review and evaluate the current communication and media relationships
practices (from headquarters to the field offices, within and among
headquarters directorates, and within and among field offices);
* Gather input from the other working groups that have addressed
communication issues and prepare recommendations to design
communication and media practices and strategies for FAMS;
Stakeholder Relationships Working Group[B] (June 14, 2006);
* Review and analyze the current stakeholder relationships with
entities such as Transportation Security Administration, Department of
Homeland Security, Office of Management and Budget, Congress, and
various law enforcement entities;
* Determine how FAMS can best leverage those engagements to ensure a
viable, sustainable organization;
Assistant Federal Security Director for Law Enforcement Working
Group[A] (June 27, 2006);
* Evaluate the current work activities of the position's workforce and
then identify the core roles and responsibilities of that position;
* Codify the Assistant Federal Security Director for Law Enforcement's
reporting procedures and the relationships between that position, the
Federal Security Directors, and the field office SACs;
Federal Air Marshal Issue Working Group[A] (October 3, 2006);
* Review current federal air marshal issue groups within FAMS and
benchmark against similar types that are operational in other law
enforcement organizations;
* Prepare a federal air marshal issue group charter that details
objectives, membership, and standard operating practices and procedures
to initiate and sustain a group within each field office;
Policy Issues Working Group[D] (January 16, 2007);
* Identify and resolve conflicts between FAMS's policies (including
legacy policies) and Transportation Security Administration management
directives. [Note: This working group has transitioned to having
periodic meetings between FAMS and TSA officials. No final report will
be issued];
Integrated Conflict Management System Working Group[B] (August 14,
2007);
* Develop a proposal for implementing (including training) an
integrated conflict management system within FAMS;
* Ensure that the proposal considers practical, operational, and
organizational culture factors and also recognizes practices that are
already in place;
* Propose recommendations concerning the roles and responsibilities
associated with system coordinators' functions at the field offices and
headquarters' divisions.
Issue categories: Medical issues;
Working groups and their purposes (first meeting date): Medical Issues
Working Group[A] (October 11, 2006);
* Examine and review medical studies pertaining to the effects of
flying in the aviation environment;
* Examine the number and types of claims for injuries and illnesses as
documented in FAMS' Office of Workers' Compensation Program;
* Identify and recommend best practices from the military and aviation
industry.
Issue categories: Administrative issues;
Working groups and their purposes (first meeting date):
Administrative Manual Working Group[C] (November 7,2006);
* Develop an administrative manual to provide a resource for FAMS
standard operating procedures and policies;
Time and Attendance Forms Working Group[A] (April 19, 2007);
* Discuss and examine time and attendance sheets currently in use by
all FAMS field offices;
* Make recommendations concerning which form(s) the group identified as
a best practice to be replicated service wide--or, if no such form
exits, make recommendations concerning a new form or modification to an
existing one;
* Discuss and make recommendations concerning methods by which the time
and attendance reporting process could become more streamlined and
ensure consistency regarding reporting and pay.
Source: Federal Air Marshal Service.
[A] According to FAMS officials, as of October 1, 2008, (1) the working
group had completed its work and given a final report to FAMS
management, (2) the report had been reviewed by FAMS executives and
then distributed to agency employees via a broadcast message from the
FAMS Director, and (3) if applicable, the broadcast message also
presented management's responses to any recommendations made by the
report.
[B] According to FAMS officials, as of October 1, 2008, the working
group had completed its work and given a final report to FAMS
management, and the report was undergoing review by FAMS executives.
[C] According to FAMS officials, as of October 1, 2008, the working
group had yet to complete its work and give a final report to FAMS
management.
[D] According to FAMS officials, the working group is to remain
ongoing, and a final report is not expected to be issued. Rather, the
group will present its findings when applicable and by appropriate
means.
[E] A Federal Security Director is the top-ranking TSA official
responsible for security at each of the nation's commercial airports.
[End of table]
[End of section]
Appendix VI: FAMS Workforce Satisfaction Survey:
FAMS Workforce Satisfaction Survey:
On behalf of the Director and the Office of Law Enforcement [OLE]/FAMS
Office of Workforce Planning and Management, thank you for taking the
time to complete this survey. The purpose of this survey is to capture
information regarding workforce satisfaction at OLE/FAMS. The
information obtained by this survey will be used by OLE/FAMS leadership
to assess the current levels of workforce satisfaction for the purposes
of planning, policy development, and program enhancement. In addition,
the data obtained by this survey will be used to evaluate current
OLE/FAMS workforce satisfaction initiatives and strategies. Please take
the time to carefully complete this survey. Your input will remain
confidential and is vital to making OLE/FAMS a premier law enforcement
organization. Thank you for your help.
For each item, choose the response that best reflects your experience
at OLE/FAMS.
Senior Leadership:
1. I have trust in the abilities of OLE/FAMS senior leaders and
executives (e.g., Director, Deputy Director, Assistant Directors,
Deputy Assistant Directors) to lead the organization.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
2. OLE/FAMS executive leaders are receptive to organizational change.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
3. I believe that the Director’s visions and initiatives are
consistently shared and supported by my field management.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Supervisor/Management:
4. My first-line supervisor is competent in doing his/her job.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
5. My supervisor is receptive to change.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
6. My discussions with my supervisor about my performance are
worthwhile.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
7. I have discussed a career development plan with my supervisor.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
8. My supervisor shows concern for my career progress.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Resources & Technology:
9. I have adequate equipment, supplies, and materials to accomplish my
duties.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
10. Generally, the equipment I use (e.g., firearm, computer, cell
phone, PDA [Personal Digital Assistant], etc.) to perform my job works
properly.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
11. The equipment I use is sufficiently easy to operate.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
12. I am generally satisfied with the quality of OLE/FAMS physical
facilities (e.g., workspaces, training facilities, physical fitness
areas, firearms ranges, etc.)
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
13. I am generally satisfied with the availability of OLE/FAMS physical
facilities (e.g., workspaces, training facilities, physical fitness
areas, firearms ranges, etc.)
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Training & Education:
14. I receive the training I need to do my job.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
15. I am satisfied with the frequency/amount of training I receive in
my office.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
16. Generally, I am satisfied with the content and variety of job-
related training I receive in my office.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
17. I am satisfied with the continuing education opportunities offered
by my job.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
18. OLE/FAMS supports continuing education opportunities relevant to my
job.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
19. My work schedule affords me the opportunity to pursue continuing
education.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
20. I am encouraged by my supervisors and managers to seek training and
educational opportunities.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Career Development:
21. Overall, I am satisfied with the progress I have made toward my
career goals.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
22. There are sufficient opportunities for career advancement at
OLE/FAMS.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
23. In my present position, I have a clearly understood career path.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
24. Promotions to supervisory levels in OLE/FAMS are based on merit.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
25. In OLE/FAMS, the selection criteria for promotion are clear.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Policies & Procedures:
26. OLE/FAMS’ written policies support (and do not hinder) mission
accomplishment.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
27. The local policies and procedures of my office support mission
accomplishment.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
28. I am able to stay updated and am informed about the latest policies
and procedures.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Employee Involvement & Autonomy:
29. I have effective channels to voice my opinion regarding work -
related issues (e.g., working groups, listening sessions, e-mail
suggestion box, etc.)
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
30. I am empowered to use my professional discretion in daily execution
of my duties.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
31. I am provided sufficient opportunities to participate in important
decisions affecting my work.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Rewards & Recognition:
32. Outstanding performance is recognized in my office.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
33. I am satisfied with the promotion practices of OLE/FAMS.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
34. In my office, monetary rewards (i.e., cash awards, in-position
increases, etc.) are tied to performance.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
35. I am generally satisfied with my pay.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Communication:
36. OLE/FAMS’ policies and procedures are clearly communicated and easy
to understand.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
37. I am satisfied with communication within my office (e.g., FO [Field
Office], branch, division, directorate).
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
38. There are mechanisms in place which allow me to freely express my
comments, concerns, and suggestions without fear of retaliation.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
39. I am satisfied that the work-related concerns I address with
management are addressed appropriately.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
40. I have enough information to do my job well.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Safety, Health & Medical Issues:
41. I feel that my job-related stress is manageable.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
42. I am generally satisfied with OLE/FAMS programs related to employee
safety, health, and wellness.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
43. I am satisfied that OLE/FAMS management is concerned for the health
and safety of employees and is working continuously to offer improved
services.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
44. I have been provided information and resources to take personal
responsibility for my health and wellness as it relates to my job
(e.g., proper diet, fitness, sufficient rest).
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
45. I feel that medical information relevant to my job is communicated
to me.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Work & Family Life:
46. I am able to effectively balance my work with my personal/family
life.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
47. My family is supportive of my career with OLE/FAMS.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
48. Current initiatives (e.g., Voluntary Lateral Transfer Program, new
office openings, etc.) have a positive effect on quality of work
life/family life.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
49. OLE/FAMS leadership has implemented positive changes affecting
scheduling.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
50. I am satisfied that OLE/FAMS is exploring initiatives to improve
quality of life/family life.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
51. I have seen improvement in quality of work life and family life as
a result of the recommendations from the Director’s Working Groups.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
52. I have seen positive changes made in OLE/FAMS in the last year.
Organizational Commitment:
53. I am proud to work for OLE/FAMS.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
54. I find my values are similar to OLE/FAMS values.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
55. I feel a sense of loyalty to OLE/FAMS.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
56. I am likely to stay at OLE/FAMS for the next 12 months.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Job Satisfaction:
57. The work I do is important.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
58. I find my work challenging and interesting.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
59. Generally speaking, I am very satisfied with my job.
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
60. I like the kind of work I do (e.g., my current duties and
assignment).
Strongly agree:
Agree:
Neutral:
Disagree:
Strongly disagree:
Comments Section:
Please use this section to provide more specific information for any of
the above questions.
Reasons for Staying with OLE/FAMS:
Indicate the importance of each of the following factors in your
reasons for staying with OLE/FAMS.
Money (e.g., salary and benefits);
Schedule (e.g., number of hours, flexibility);
Duty location (e.g., FO, HQ [headquarters], SOCD [Systems Operation
Control Division], etc.);
OLE/FAMS mission (e.g., belief in importance of mission);
Career Opportunities (e.g., promotion potential, career growth);
Other:
[End of section]
Appendix VII: Comments from the Department of Homeland Security:
U.S. Department of Homeland Security:
Washington, DC 20528:
November 26, 2008:
Stephen M. Lord:
Acting Director, Homeland Security and Justice Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Mr. Lord:
Thank you for the opportunity to review and provide comments on the
Government Accountability Office's (GAO) draft report GAO-04-53SU
titled, Federal Air Marshal Service Has Taken Actions to Fulfill he
Core Mission and Address Workforce Issues, but Additional Actions Are
Needed to Improve Workforce Survey (440601). The Department of Homeland
Security (DHS) appreciates GAO's effort in conducting this study and we
believe that the findings and recommendation will facilitate continued
progress in identifying and addressing issues that affect the ability
of Federal Air Marshal Service (FAMS) personnel to perform the Agency's
aviation security mission.
In April 2007, GAO initiated its assessment of the FAMS. The inquiry
lasted approximately 17 months, concluding in October 2008, During this
time, FAMS provided GAO with full access to FAMS' Executive Management,
as well as access to the subject-matter experts involved in the
coordination of programs that affect the entire organization. FAMS
briefed GAO on numerous occasions regarding the 36 working groups and
other initiatives that were implemented to address workforce issues.
During this period, GAO also conducted site visits and interviews at 11
of the FAMS' 21 field offices. During these visits. GAO had full access
to the managers and was given access to interview air marshals of their
choosing. This was the most in-depth inquiry conducted by an outside
entity.
DHS concurs with GAO's finding that the FAMS has successfully taken
actions to fulfill core mission and accepts the finding as validation
of the efforts undertaken to improve FAMS morale and performance.
Additionally, DHS/TSA is pleased to note GAO's endorsements of:
* FAMS' Concept of Operation (consideration of risk-related factors
deployed based on coverage goals and flexibility for responding to
changing threats);
* Management's efforts to address employee concerns and improve
workforce satisfaction (Working Group Initiative, Field Office Focus
Groups, Field Office Visits by the Director, Listening Sessions,
Dinner/Breakfast with the Director. Director's and Anonymous Suggestion
In-Boxes, assignment of an agency Ombudsman, and establishment of the
Voluntary Lateral Transfer Program);
* Steps taken to preserve anonymity of air marshals (revision of hotel
and dress code policies); and;
* Accomplishments in improving the quality of life for agency employees
{Mission Exchange Program, 6 p.m. and 60-hour rule, limit on number of
flights per days, more rest time following international missions, and
more consistent start times).
GAO made one recommendation that "the FAMS Director take appropriate
actions to increase the usefulness of the workforce satisfaction
surveys that the FAMS plans to conduct biennially. Such actions could
include, for example, ensuring that the survey questions and the answer
options are clearly structured and unambiguous and that additional
efforts are considered for obtaining the highest possible response
rates."
We generally concur with the aforementioned recommendation. At present,
FAMS is in the initial stages of formulating the next workforce
satisfaction survey and plans to implement the suggestions.
Once again, DHS appreciates the opportunity to review and provide
comments and we look forward to working with you on future homeland
security issues.
Sincerely,
Signed by:
Jerald E. Levine Director:
Departmental GAO/OIG Liaison Office:
[End of section]
Appendix VIII GAO Contact and Staff Acknowledgments:
GAO Contact:
Stephen M. Lord, Acting Director, Homeland Security and Justice Issues,
(202) 512-4379 or lords@gao.gov:
Staff Acknowledgments:
In addition to the contacts named above, Danny Burton and John Hansen
(Assistant Directors) and Michael Harmond (Analyst-in-Charge) managed
this assignment.
David Alexander, Chuck Bausell, Arturo Cornejo, Wendy Dye, Stuart
Kaufman, and Courtney Reid made significant contributions to the work.
Tom Lombardi provided legal support.
Katherine Davis provided assistance in report preparation.
[End of section]
Footnotes:
[1] FAMS is one layer among the multiple layers of aviation security.
For example, prospective passengers are prescreened against applicable
records in the consolidated watch list managed by the Federal Bureau of
Investigation's Terrorist Screening Center. Also, at TSA checkpoints in
airports across the nation, passengers and baggage are physically
screened. Air marshals generally are characterized as being the last
line of defense within this layered aviation-security framework.
[2] Air marshals are included in the GS-1801 federal law enforcement
officer job series, as defined by the Office of Personnel Management.
Law enforcement personnel with this series designation engage in
general inspection and compliance activities. See GAO, Federal Law
Enforcement: Survey of Federal Civilian Law Enforcement Functions and
Authorities, [hyperlink, http://www.gao.gov/products/GAO-07-121]
(Washington, D.C.: Dec. 19, 2006).
[3] See Pub. L. No. 107-71, § 105, 115 Stat. 597, 606-08 (2001)
(codified as amended at 49 U.S.C. § 44917).
[4] The specific number of federal air marshals is classified.
[5] For example, see GAO, Aviation Security: Federal Air Marshal
Service Could Benefit from Improved Planning and Controls, GAO-06-203
(Washington, D.C.: Nov. 28, 2005).
[6] PART consists of a standard series of questions intended to
determine the strengths and weaknesses of federal programs. The PART
questions cover four broad topics--(1) program purpose and design, (2)
strategic planning, (3) program management, and (4) program results/
accountability. See GAO, Program Evaluation: OMB's PART Reviews
Increased Agencies' Attention to Improving Evidence of Program Results,
[hyperlink, http://www.gao.gov/products/GAO-06-67] (Washington, D.C.:
Oct. 28, 2005).
[7] The Homeland Security Institute is a federally funded research and
development center established pursuant to the Homeland Security Act of
2002. See Pub. L. No. 107-296, § 312, 116 Stat. 2135, 2176, as amended.
The Institute's mission is to assist the Department of Homeland
Security address relevant issues requiring scientific, technical, and
analytical expertise. In response to OMB's 2003 PART review of FAMS,
which concluded that key aspects of program design needed to be
independently assessed, FAMS contracted for an independent evaluation
to be conducted by the Homeland Security Institute.
[8] Homeland Security Institute, Analysis for the Next-Generation
Federal Air Marshal Service (FAMS) Mission Scheduling & Notification
System (MSNS), final report (July 10, 2008).
[9] "Workday rules" refer to the parameters that FAMS uses for
assigning air marshals to flights.
[10] Homeland Security Institute, Federal Air Marshal Service (FAMS)
Risk-Based Resource Allocation Study: Human Factors Analysis, final
report (July 10, 2008).
[11] GAO, Aviation Security: Federal Air Marshal Service Could Benefit
from Improved Planning and Controls, GAO-05-884SU (Washington, D.C.:
Sept. 29, 2005). The report is restricted (not available to the public)
because it contains sensitive security information. The public version
of the report is [hyperlink, http://www.gao.gov/products/GAO-06-203]
(Nov. 28, 2005).
[12] See GAO, A Model of Strategic Human Capital Management,
[hyperlink, http://www.gao.gov/products/GAO-02-373SP] (Washington,
D.C.: Mar. 15, 2002).
[13] Nonprobability sampling is a method of sampling where observations
are selected in a manner that is not completely random, generally using
specific characteristics of the population as criteria. Results from a
nonprobability sample cannot be used to make inferences about an entire
population because some elements of the population being studied had no
chance or an unknown chance of being selected as part of the sample.
[14] FAMS's changes to check in and boarding procedures concern air
marshals' interactions with airline personnel. FAMS's policy continues
to require air marshals to adhere to established TSA regulations and
locally established airport procedures.
[15] FAMS officials attributed the 2007 survey's 46-percent response
rate to the highly mobile nature of the agency's workforce and also
noted that this rate is consistent with the response rates for similar
workforce satisfaction surveys in the federal government. FAMS
officials added that they view the workforce survey as one of a variety
of tools--along with the numerous issue-specific working groups and
other processes and initiatives--for raising issues of concern and
providing feedback on actions taken by management.
[16] The transfer of FAMS to ICE was premised partly on the assumptions
that (1) air marshals would be afforded a broader career path by cross-
training with ICE's investigative division and (2) ICE's special agents
could provide a surge capability by serving as supplemental air
marshals, if needed. See GAO, Aviation Security: Federal Air Marshal
Service Is Addressing Challenges of Its Expanded Mission and Workforce,
but Additional Actions Needed, [hyperlink,
http://www.gao.gov/products/GAO-04-242] (Washington, D.C.: Nov. 19,
2003).
[17] The Terrorist Screening Center, managed by the Federal Bureau of
Investigation, maintains a consolidated watch list of known or
appropriately suspected terrorists and sends records from the list to
agencies to support terrorism-related screening. TSA currently provides
airlines with portions of the Terrorist Screening Center's watch list-
-the No Fly and Selectee lists--and directs them to screen the names of
passengers to identify those who may pose threats to aviation. In
general, individuals on the No Fly List are to be precluded from
boarding an aircraft, and individuals on the Selectee List are to
receive additional physical screening prior to boarding an aircraft.
See GAO, Terrorist Watch List Screening: Opportunities Exist to Enhance
Management Oversight, Reduce Vulnerabilities in Agency Screening
Processes, and Expand Use of the List, [hyperlink,
http://www.gao.gov/products/GAO-08-110] (Washington, D.C.: Oct. 11,
2007).
[18] Other in-flight security layers not employed based on risk include
hardened cockpit doors and armed flight crew members who are part of
TSA's Federal Flight Deck Officer program.
[19] The specific number of air marshals assigned to an on-board team,
whether for a domestic flight or an international flight, may vary
depending on such factors as duration of the flight, the type of
aircraft, the departure and destination cities, and awareness of
specific threat information.
[20] In determining air marshals' availability, FAMS officials stated
that they must consider such factors as training requirements, other
ground-based duties, and annual leave plans.
[21] After the March 2004 train bombings in Madrid, TSA created and
deployed Visible Intermodal Prevention and Response teams to enhance
security on U.S. rail and mass transit systems nationwide. Comprised of
federal air marshals, surface transportation security inspectors,
transportation security officers, behavior detection officers, and
explosives detection canines, the teams are intended to work with local
security and law enforcement officials to supplement existing security
resources, provide a deterrent presence and detection capabilities, and
introduce an element of unpredictability to disrupt potential terrorist
activities.
[22] FAMS's criteria for determining high-risk flights are classified.
[23] FAMS considers "threat" and "intelligence" as separate risk-
related factors.
[24] FAMS's coverage rate goals for the 10 targeted critical flight
categories are classified.
[25] According to FAMS officials, international mission schedules are
prepared by Systems Operation Control Division officials (other than
those in the Domestic Planning Branch) who use manual methods because
of added complexities, such as the need to coordinate with foreign
governments and make reservations at overseas locations.
[26] Appendix II presents more details about the results of OMB's 2003
PART review of FAMS and the follow-on reassessment conducted in 2008.
[27] Much of the specific information in the report is classified.
[28] The center was established by the Department of Homeland Security.
The center's intelligence analysts and infrastructure specialists work
to identify the threat to critical infrastructures, vulnerabilities and
interdependencies, and the overall risk inherent in any potential
attack against critical infrastructure.
[29] The U.S. Northern Command provides command and control of
Department of Defense homeland defense efforts and coordinates defense
support of civil authorities. The North American Aerospace Defense
Command is the binational American and Canadian command responsible for
the air defense of North America and maritime warning for the United
States and Canada.
[30] Homeland Security Institute, Analysis for the Next-Generation
Federal Air Marshal Service (FAMS) Mission Scheduling & Notification
System (MSNS), final report (July 10, 2008).
[31] Homeland Security Institute, Federal Air Marshal Service (FAMS)
Risk-Based Resource Allocation Study: Human Factors Analysis, final
report (July 10, 2008).
[32] The Institute noted that it is coordinating its work on
performance measures with OMB.
[33] GAO, A Model of Strategic Human Capital Management, [hyperlink,
http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: Mar. 15,
2002).
[34] Regarding the status of the 36 working groups as of October 1,
2008, FAMS officials stated that (a) 18 working groups had completed
their work and submitted final reports, which have been distributed by
the FAMS Director to the workforce; (b) 9 working groups had submitted
final reports that were still under review by FAMS management; (c) 5
working groups had yet to complete their work and prepare final
reports; and (d) 4 working groups are to remain ongoing and present
findings when applicable and by appropriate means.
[35] FAMS also refers to the e-mail in-box as the "working group e-mail
address."
[36] In addition to the working groups and other processes and
initiatives, in late fiscal year 2007, FAMS conducted the first in a
series of biennial surveys of its workforce. FAMS expected the 2007
survey to help identify issues affecting the ability of air marshals
and other agency personnel to perform their jobs and also to help
management assess the effectiveness of actions taken to address various
issues. We discuss the 2007 survey in greater detail later in this
report.
[37] GAO-05-884SU (Sept. 29, 2005), which is a restricted report (not
available to the public) because it contains sensitive security
information. The public version of the report is [hyperlink,
http://www.gao.gov/products/GAO-06-203] (Nov. 28, 2005).
[38] FLD 7100, Subject: FAMS Reporting System Policy and Procedure
(Oct. 18, 2005).
[39] [hyperlink, http://www.gao.gov/products/GAO-06-203] (Nov. 28,
2005).
[40] FAMS's changes to check in and boarding procedures concern air
marshals' interactions with airline personnel. FAMS's policy continues
to require that air marshals adhere to established TSA regulations and
locally established airport procedures.
[41] A "regular day off" refers to each of the 2 days during the 7-day
week when an air marshal will not be reporting to work. For example, if
an air marshal worked a Monday to Friday schedule, the regular days off
would be Saturday and Sunday.
[42] According to FAMS workday rules, an air marshal in "non-mission
status" is at work performing administrative or other tasks and is not
expected to fly while in this status. Under workday rules, air marshals
generally have 11 days in non-mission status each year. In addition,
air marshals are not expected to fly during training days (20 days
annually).
[43] Homeland Security Institute, Federal Air Marshal Service (FAMS)
Risk-Based Resource Allocation Study: Human Factors Analysis, final
report (July 10, 2008).
[44] FAMS has conducted two other surveys of employees. One of these
was conducted earlier in 2007 and focused on communications and
operational concepts. The other survey, conducted in 2006, focused on
quality-of-life issues.
[45] See appendix VI for a copy of the survey questionnaire. In
addition to the 60 substantive questions, the survey questionnaire also
had 10 questions that solicited demographic information from the
respondents.
[46] The OMB guidance governs federal agency surveys of the public at
large or outside individuals, groups, or organizations, such as local
government entities. The FAMS workforce survey was administered
internally to gather information from the agency's employees.
[47] Office of Management and Budget, Standards and Guidelines for
Statistical Surveys (Washington, D.C.: September 2006), p. 16; and,
Office of Management and Budget, Office of Information and Regulatory
Affairs, Questions and Answers When Designing Surveys for Information
Collections (Washington, D.C.: Jan. 20, 2006), p. 56.
[48] Specifically, a question containing the word "and" raises a
concern about whether the respective portions of the question on either
side of the conjunction are sufficiently related to be considered
similar or are really different topics or concepts. Such questions are
called "double barreled" within the social science community.
[49] GAO, A Model of Strategic Capital Management, [hyperlink,
http://www.gao.gov/products/GAO-02-373SP] (Washington, D.C.: March
2002).
[50] GAO, Aviation Security: Federal Air Marshal Service Has Taken
Actions to Fulfill Its Core Mission and Address Workforce Issues, but
Additional Actions Are Needed to Improve Workforce Survey, GAO-09-53SU
(Washington, D.C.: Dec. 8, 2008).
[51] Pub. L. No. 107-71, § 105, 115 Stat. 597, 606-08 (2001) (codified
as amended 49 U.S.C. § 44917).
[52] PART consists of a standard series of questions intended to
determine the strengths and weaknesses of federal programs. The PART
questions cover four broad topics--(1) program purpose and design, (2)
strategic planning, (3) program management, and (4) program results/
accountability. See GAO, Program Evaluation: OMB's PART Reviews
Increased Agencies' Attention to Improving Evidence of Program Results,
GAO-06-67 (Washington, D.C.: Oct. 28, 2005).
[53] The Homeland Security Institute is a federally funded research and
development center established pursuant to the Homeland Security Act of
2002. See Pub. L. No. 107-296, § 312, 116 Stat. 2135, 2176, as amended.
The institute's mission is to assist the Department of Homeland
Security address relevant issues requiring scientific, technical, and
analytical expertise. In response to OMB's 2003 PART review of FAMS,
which concluded that key aspects of program design needed to be
independently assessed, FAMS contracted for an independent evaluation
to be conducted by the Homeland Security Institute.
[54] Appendix III presents quoted excerpts that substantially replicate
the executive summary in the Homeland Security Institute's July 2006
report. Appendix IV presents an overview of performance measures for
assessing FAMS.
[55] Homeland Security Institute, Analysis for the Next-Generation
Federal Air Marshal Service (FAMS) Mission Scheduling & Notification
System (MSNS), final report (July 10, 2008).
[56] "Workday rules" refer to the parameters that FAMS uses for
assigning air marshals to flights. As applicable to non-overnight
missions, for example, FAMS tries to assign air marshals to flights (or
combinations of flights) that will return the air marshals back home
during a scheduled 10-hour workday.
[57] Homeland Security Institute, Federal Air Marshal Service (FAMS)
Risk-Based Resource Allocation Study: Human Factors Analysis, final
report (July 10, 2008).
[58] See GAO, A Model of Strategic Human Capital Management,
[hyperlink, http://www.gao.gov/products/GAO-02-373SP (Washington, D.C.:
Mar. 15, 2002).
[59] The field offices we visited are not listed in this report because
TSA considers field office locations to be Sensitive Security
Information.
[60] Nonprobability sampling is a method of sampling where observations
are selected in a manner that is not completely random, generally using
specific characteristics of the population as criteria. Results from a
nonprobability sample cannot be used to make inferences about an entire
population because some elements of the population being studied had no
chance or an unknown chance of being selected as part of the sample.
[61] Examples of ground-based assignments for air marshals included
work in a field office's operations or training sections or in an
airport liaison position. These assignments are temporary with varying
lengths of service.
[62] At the time of OMB's 2003 PART assessment, FAMS had one long-term
outcome measure: "Number of successful terrorist and other criminal
attacks initiated from commercial passenger aircraft cabins with air
marshal coverage."
[63] Appendix IV presents more detailed information about FAMS's
performance measures.
[64] PART consists of a standard series of questions intended to
determine the strengths and weaknesses of federal programs. The PART
questions cover four broad topics--(1) program purpose and design, (2)
strategic planning, (3) program management, and (4) program results/
accountability. See GAO, Program Evaluation: OMB's PART Reviews
Increased Agencies' Attention to Improving Evidence of Program Results,
[hyperlink, http://www.gao.gov/products/GAO-06-67] (Washington, D.C.:
Oct. 28, 2005).
[65] OMB defines an "output" measure as one pertaining to the internal
activities of a program, such as the products or services delivered.
[66] Regarding types of performance measures, OMB defines an "outcome"
measure as pertaining to the events or conditions of direct importance
to the public/beneficiary that are external to the program.
[67] According to OMB, programs that have difficulty defining a
quantifiable outcome measure may adopt a "proxy" outcome measure or
indicator. These proxy measures need not meet the external event or
condition criterion for a standalone outcome measure. The proxy measure
need not be a public good unto itself, but rather be linked to the
public good that is captured by the parent outcome measure--which is,
in this case, the number of successful attacks.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: