This is the accessible text file for GAO report number GAO-09-603
entitled 'Public Transportation: FTA's Triennial Review Program Has
Improved, But Assessments of Grantees' Performance Could Be Enhanced'
which was released on July 30, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Highways and Transit, Committee on
Transportation and Infrastructure, House of Representatives:
United States Government Accountability Office:
GAO:
June 2009:
Public Transportation:
FTA's Triennial Review Program Has Improved, But Assessments of
Grantees' Performance Could Be Enhanced:
GAO-09-603:
GAO Highlights:
Highlights of GAO-09-603, a report to Subcommittee on Highways and
Transit, Committee on Transportation and Infrastructure, House of
Representatives.
Why GAO Did This Study:
The Federal Transit Administration (FTA) oversees about $5.5 billion in
federal funds each year to transit agencies serving urban areas
(grantee), in part through its triennial review program, which
evaluates grantee adherence to federal requirements at least once every
3 years. GAO recommended in a 1998 oversight report that FTA improve
the program.
The subcommittee requested that GAO review this program. GAO identified
(1) the extent to which triennial reviews indicate that grantees met
applicable federal requirements from fiscal years 2000 through 2008;
(2) the strengths and weaknesses of the triennial review process; and
(3) FTA’s performance measures for the triennial review and the extent
to which they meet key attributes of successful performance measures.
GAO addressed these objectives by analyzing oversight data on 424
grantees that had three triennial reviews, reviewing triennial review
reports and guidance, assessing FTA’s performance measures; and
interviewing FTA headquarters and regional officials, contractors who
conduct the reviews, and grantees.
What GAO Found:
GAO’s analysis of FTA’s triennial review oversight data found that over
two-thirds of the 424 grantees analyzed have not consistently improved
overall performance in terms of meeting more federal requirements from
fiscal years 2000 through 2008. Fifty-one percent of grantees had mixed
results in meeting requirements and 17 percent consistently met fewer
requirements; while 31 percent consistently met more requirements—one
of the goals of the triennial review program. (See figure 3).
Executives from three grantees that met most requirements attributed
their performance to, among other things, having job descriptions that
link employee responsibilities to the triennial review—a practice they
said contributed to a culture of accountability. During the same time,
grantees had the greatest number of findings in 5 of 23 triennial
review areas, including the procurement and drug and alcohol testing
areas. While FTA helps grantees address findings, additional efforts to
identify the underlying causes and the severity of findings could
further benefit grantees.
FTA’s triennial review program uses some strong management practices—
having a well-defined process, using an information system to monitor
grantees, and issuing reports timely. Still, two areas could be
strengthened. First, while FTA is legislatively required to conduct a
complete review of grantees’ adherence to federal requirements at least
once every 3 years, GAO identified a few instances where documentation
does not clearly show that FTA reviewed all requirement areas. For
example, 10 triennial review reports for 2008 showed that the drug and
alcohol program area was “not reviewed.” FTA’s practice is to review
all areas, regardless of documentation, but because FTA’s guidance is
not clear about how to document the review of areas where FTA has
conducted a related special review in the prior two years, a few
grantees may not be reviewed for 5 years. FTA plans to revise its
guidance to avoid ambiguity. Second, FTA is aware of the burden
oversight reviews place on grantees and works to limit this burden.
However, in a limited number of cases, FTA did not coordinate its
special oversight reviews with the triennial review schedule, which may
place undue burden on a few grantees receiving multiple oversight
reviews in the same fiscal year.
FTA’s two timeliness performance measures for assessing the triennial
review program—(1) closing 80 percent of grantees’ deficient findings
within 30 days of their due date and (2) issuing 95 percent of the
final triennial review reports within 30 days of completing a review—
meet some, but not all key attributes of successful measures. (See
figure 6). Although both measures link throughout the organization,
have measurable targets, are clearly stated, and do not overlap, the “
close findings” measure does not meet the objectivity and reliability
attribute. For example, data inaccuracies in past “close findings” data
raised questions about the reliability of the measure. Also, both
measures do not assess the core program activity to evaluate grantees’
performance or governmentwide priorities, such as the quality of the
triennial review program, and thus, as a whole, are not balanced,
making it difficult for managers to not overemphasize one priority at
the expense of others.
What GAO Recommends:
GAO recommends that FTA analyze oversight data to enhance grantees’
performance, strengthen the triennial review process, and improve
performance measures. The Department of Transportation reviewed a draft
of this report and generally agreed with its contents and with GAO’s
recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/products/GAO-09-603]. For more
information, contact Phillip R. Herr at (202) 512-2834 or
herrp@gao.gov.
[End of section]
Contents:
Letter:
Background:
FTA's Triennial Review Data Show Many Grantees Are Not Making Progress
Toward Meeting Federal Requirements, and FTA Could Do More to Improve
Grantees' Performance:
The Triennial Review Program Uses Some Strong Management Practices, but
Can Be Improved in a Few Areas:
FTA's Two Triennial Review Performance Measures Meet Some, But Not All
Key Attributes of Successful Performance Measures:
Conclusions:
Recommendations for Executive Action:
Agency Comments:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: Description of FTA's 23 Triennial Review Areas:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Description of FTA's Primary Oversight Reviews:
Table 2: FTA Categories Used to Document Triennial Review Results:
Table 3: Nine Key Attributes of Successful Performance Measures:
Table 4: FTA's Descriptions of the Requirements in Each of the
Triennial Review Areas:
Figures:
Figure 1: Key Information about FTA's 10 Regions:
Figure 2: Major Steps in the Triennial Review Process:
Figure 3: Grantees' Performance, as Measured by the Change in the
Number of Findings from Fiscal Years 2000 through 2008:
Figure 4: The Range in the Average Number of Findings After Three
Triennial Reviews for Each of the 424 Grantees from Fiscal Years 2000
through 2008:
Figure 5: Percentage of Triennial Review Findings for the Top Five
Areas, from Fiscal Years 2000 through 2008:
Figure 6: Comparison of FTA Triennial Review Performance Measures to
Key Attributes:
Abbreviations:
FTA: Federal Transit Administration:
grantee: recipient of Urbanized Area Formula Program grants:
OTRAK: Oversight Tracking System:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
June 30, 2009:
The Honorable Peter A. DeFazio:
Chairman:
The Honorable John J. Duncan, Jr.
Ranking Member:
Subcommittee on Highways and Transit:
Committee on Transportation and Infrastructure:
House of Representatives:
Public transportation systems are playing an increasingly important
role in the nation's overall transportation system. Public transit
increases mobility for millions of Americans in communities both large
and small, provides congestion relief, promotes safe travel, and helps
reduce our country's dependence on foreign oil. The Department of
Transportation's Federal Transit Administration (FTA) fosters the
development and maintenance of bus, subway, trolley, passenger ferry
boat, and other public transportation systems through several grant
programs to transit agencies serving both rural (population less than
50,000) and urban communities (population 50,000 or more). A large
portion of FTA's grants are formula grants to transit agencies in urban
areas across the country.[Footnote 1] This Urbanized Area Formula
Program received 42 percent of FTA grant funds--about $22 billion from
fiscal years 2004 through 2009.[Footnote 2] In addition, Congress
recently appropriated an additional $6 billion for the Urbanized Area
Formula Program in the American Recovery and Reinvestment Act of 2009.
[Footnote 3]
Federal law requires the Secretary of Transportation to review and
evaluate completely the performance of each recipient of Urbanized Area
Formula Program grants (grantee) that receives urbanized area formula
funds under this section in complying with statutory and administrative
requirements, at least once every 3 years.[Footnote 4] This evaluation--
commonly known as the "triennial review"--is one of the primary means
FTA uses to evaluate to what extent Urbanized Area Formula Program
grantees are meeting federal requirements. Grantees must comply with a
variety of requirements to safeguard federal funds and help ensure safe
transportation, among other things.
As the subcommittee prepares for the 2009 surface transportation
reauthorization, we assessed, at your request, FTA's triennial review
program. Specifically, we identified (1) to what extent FTA's triennial
reviews indicate that grantees met applicable federal requirements from
fiscal years 2000 through 2008, (2) the strengths and weaknesses of the
triennial review process, and (3) FTA's performance measures for the
triennial review and to what extent FTA's measures meet key attributes
of successful performance measures.
To identify the extent to which triennial reviews indicate that
grantees are meeting applicable federal requirements from fiscal years
2000 through 2008, we obtained and analyzed data from FTA's Oversight
Tracking System (OTRAK)--the official electronic information system
used for tracking and monitoring oversight activities. Specifically, we
conducted various analyses of oversight data on 424 of the
approximately 600 urbanized area formula fund grantees. These 424
grantees received exactly 3 triennial reviews from fiscal years 2000
through 2008--the period of time for which OTRAK data was available.
[Footnote 5] We found most OTRAK data sufficiently reliable for our
purposes, but have reliability concerns about closed findings data used
for performance measurements, which we discuss later in this report. To
identify the strengths and weaknesses of the triennial review process,
we reviewed federal guidance for improving grant accountability,
previous GAO reports on oversight, FTA's oversight review order, the
Triennial Review Contractors' Guide, oversight review schedules, and
the contents of over 200 triennial review reports for fiscal years 2007
and 2008.[Footnote 6] We also attended FTA's new grantee triennial
review workshop and observed one of FTA's 2008 triennial reviews. To
identify performance measures for the triennial review and the extent
to which they meet key attributes of successful performance measures,
we reviewed the key attributes of successful performance measures, as
described in our past work on the Government Performance and Results
Act of 1993 and FTA's strategic and annual performance plans.[Footnote
7] Specifically, we compared each of FTA's performance measures against
each of the key attributes to determine the extent to which they fully
met, partially met, or did not meet all key attributes. Finally, we
interviewed FTA headquarters officials, staff in all 10 FTA regional
offices, representatives of the four contracting firms that conduct the
triennial reviews, and professional transportation associations. We
also selected 10 grantees and interviewed executives from these
grantees to discuss their performance, as assessed by the triennial
review. We selected these grantees on several factors, including their
past performance, as assessed by triennial reviews; having both a
triennial review and another FTA oversight review in the same fiscal
year; and demographic features, such as size and geographic region.
We conducted this performance audit from June 2008 to June 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. Further details about our
objectives, scope, and methodology appear in appendix I.
Background:
As one of 10 administrations within the Department of Transportation,
FTA provides financial assistance to transit agencies to develop new
systems and improve, maintain, and operate existing systems in all 50
states, the District of Columbia, and some U.S. territories. FTA
maintains its headquarters in Washington, D.C., with 10 regional
offices throughout the continental United States, and employs about 500
of the approximately 60,000 employees of the department. Figure 1
summarizes key information about FTA's 10 regions, including the total
amount and funds for the Urbanized Area Formula Program.
Figure 1: Key Information about FTA's 10 Regions:
[Refer to PDF for image: U.S. map]
The following data is depicted on the map:
Region: 1;
Regional office: Boston, Massachusetts;
Number of staff[A]: 13;
Total funds[B] (in millions): $482;
Funds for the Urbanized Area Formula Program[B](in millions): $237.
Region: 2;
Regional office: New York City, New York;
Number of staff[A]: 19;
Total funds[B] (in millions): $2,093;
Funds for the Urbanized Area Formula Program[B](in millions): $1,246.
Region: 3;
Regional office: Philadelphia, Pennsylvania;
Number of staff[A]: 23;
Total funds[B] (in millions): $1,548;
Funds for the Urbanized Area Formula Program[B](in millions): $675.
Region: 4;
Regional office: Atlanta, Georgia;
Number of staff[A]: 24;
Total funds[B] (in millions): $1,125;
Funds for the Urbanized Area Formula Program[B](in millions): $633.
Region: 5;
Regional office: Chicago, Illinois;
Number of staff[A]: 24;
Total funds[B] (in millions): $1,278;
Funds for the Urbanized Area Formula Program[B](in millions): $635.
Region: 6;
Regional office: Fort Worth, Texas;
Number of staff[A]: 18;
Total funds[B] (in millions): $690;
Funds for the Urbanized Area Formula Program[B](in millions): $334.
Region: 7;
Regional office: Kansas City, Missouri;
Number of staff[A]: 11;
Total funds[B] (in millions): $230;
Funds for the Urbanized Area Formula Program[B](in millions): $92.
Region: 8;
Regional office: Denver, Colorado;
Number of staff[A]: 10;
Total funds[B] (in millions): $309;
Funds for the Urbanized Area Formula Program[B](in millions): $120.
Region: 9;
Regional office: San Francisco, California;
Number of staff[A]: 25;
Total funds[B] (in millions): $2,110;
Funds for the Urbanized Area Formula Program[B](in millions): $1,288.
Region: 10;
Regional office: Seattle, Washington;
Number of staff[A]: 14;
Total funds[B] (in millions): $667;
Funds for the Urbanized Area Formula Program[B](in millions): $254.
Region: Total;
Number of staff[A]: 181;
Total funds[B] (in millions): $10,533;
Funds for the Urbanized Area Formula Program[B](in millions): $5,515.
Source: GAO analysis of FTA data.
Note: Dollar amounts may not add to totals due to rounding.
[A] Full-time equivalents for fiscal year 2009.
[B] Funds are for fiscal year 2008 and are in total obligations
(legally binding government financial commitments).
[End of figure]
In 1994, FTA laid out a common framework for regional and headquarters
staff to administer oversight reviews in FTA Order 5400.1.[Footnote 8]
That order, which remains in effect today, describes the process for
conducting various oversight reviews and taking corrective actions. In
addition, the order established an internal working group called the
Oversight Review Council to develop oversight review schedules,
coordinate review schedules to avoid unnecessary duplication and
overlap, and achieve a comprehensive, uniform, and cohesive approach,
among other things.
The triennial review provides FTA with a valuable opportunity to
routinely evaluate the use of federal funds for about 600 urban area
grantees at least once every three years--or about 200 grantees each
year. It covers many areas, such as procurement, financial management,
and drug and alcohol programs. Appendix II contains FTA's descriptions
of all 23 triennial review areas. FTA also uses a similar, general
review to oversee grantees that receive rural area formula funds--a
state management review.
In addition to the more general triennial and state management reviews,
FTA conducts special oversight reviews with a narrowed subject matter
focus, such as procurement system reviews, which help ensure that
grantees (especially 30 grantees that receive large amounts of federal
transit funds) are purchasing buses and trains using competitive
bidding practices, as well as special civil rights reviews, which help
ensure that grantees are, among other things, providing public
transportation to the disabled and adhering to equal employment
opportunity practices. Grantees also experience non-FTA reviews, such
as single audits that examine grantees' financial statements, inspector
general audits, and various state audits. Table 1 describes FTA's
primary oversight reviews.
Table 1: Description of FTA's Primary Oversight Reviews:
Oversight Review: General Reviews: Triennial Review;
Description: Congress mandated the triennial review in 1983. This
review evaluates urbanized area formula grantees' performance at least
once every three years in carrying out transit programs, including
specific reference to adhere to statutory and administrative
requirements.
Oversight Review: General Reviews: State Management Review;
Description: FTA began these reviews in fiscal year 1996. This review
examines states' implementation of FTA's grants for rural areas and the
elderly and disabled, typically once every three years.
Oversight Review: Selected Special Reviews: Civil Rights Reviews;
Description: Civil Rights Reviews began in the 1980s, according to
officials in FTA's Office of Civil Rights. They now include four
discretionary oversight reviews assessing grantees' compliance with the
Americans with Disabilities Act of 1990, Title VI regulations,
Disadvantaged Business Enterprise requirements, and Equal Employment
Opportunity requirements.
Oversight Review: Selected Special Reviews: Drug and Alcohol Audit
Program Review;
Description: The Omnibus Transportation Employee Testing Act of 1991
required the Department of Transportation to implement drug testing. A
discretionary review of selected transit agencies to ensure regulations
governing substance abuse management programs are followed through its
drug and alcohol testing of safety-sensitive transit employees.
Oversight Review: Selected Special Reviews: Financial Management
Oversight Review;
Description: FTA began financial reviews in 1992. This review evaluates
grantees' financial management systems to determine whether they are
sufficient to process federal funds.
Oversight Review: Selected Special Reviews: Procurement System Review;
Description: Since 1992, FTA's procurement system review examines
grantees' procurement systems for compliance with competitive bidding
and other federal requirements. FTA conducts procurement reviews,
primarily on the 30 largest grantees, every 3 years.
Oversight Review: Selected Special Reviews: State Safety Oversight
Review;
Description: FTA began these reviews in fiscal year 1996. A review in
which FTA, other federal agencies, states, and rail transit agencies
collaborate to ensure the safety and security of rail transit systems,
such as any light and heavy rail systems, trolleys, and cable cars.
Sources: GAO and FTA reports.
[End of table]
The triennial review covers 23 triennial review areas, but given the
need to review all of these areas, the time constraints (estimated at
about 130 hours for each review), and in order to reduce the oversight
burden upon grantees, FTA does not conduct an in-depth document review
or inspection. For example, FTA normally samples no more than 15
financial records during site visits, conducts a spot check of facility
and maintenance records, and may only inspect a few buses and subway
cars during the review. However, according to FTA officials, when an
area of concern or deficiency is noted, FTA will expand the review to
examine these areas more in-depth. Figure 2 summarizes the major steps
in the triennial review process.
Figure 2: Major Steps in the Triennial Review Process:
[Refer to PDF for image: illustrated list]
Scheduling:
* Headquarters assigns contractors to all 10 regions and specific
individual grantees;
* Contractors schedule with regions the desk reviews and site visits to
grantees;
* Contractors notify regions of upcoming triennial review workshops and
grantees are encouraged to attend.
Desk reviews:
* Contractors collect data on grantees from FTA information systems;
* Contractors review grantee-specific documents at regional office;
* Contractors interview regional office staff on issues of emphasis for
each grantee;
* Contractors prepare customized agenda package for each grantee.
Customized agenda package:
* Contractors identify questions to be asked during the site visit;
* Contractors provide questions for grantee response;
* Contractors identify documents needed to complete review.
Site visit and draft report:
* Contractors conduct entrance conference with grantee;
* Contractors visit grantee facilities, inspect accounting records, and
resolve issues based on grantee input:
* Contractors determine deficient findings:
* Contractor draft report and submit to regional staff for review;
* Contractors conduct exit conference with grantee and provide draft
report for grantee review and comment.
Final report:
* Contractors provide regional staff with the final report detailing
all deficient findings;
* Regions send the final report to the grantees.
Corrective action:
* Contractors and regional staff sets a due date by which grantees must
take corrective action;
* Regions monitor the corrective action, provide technical assistance,
and close out the finding.
Sources: FTA contractors’ guide and 2008 Booz Allen Hamilton report.
[End of figure]
The triennial review also gives FTA an opportunity to provide technical
assistance to individual grantees--assistance specifically aimed to
help individual grantees resolve specific deficient findings resulting
from triennial reviews. FTA also helps grantees prepare for the
triennial review by holding annual triennial review workshops
throughout the country and offering formal training courses focused on
specialized subject areas through the National Transit Institute--a
training institute funded by an FTA grant.[Footnote 9]
While officials from both FTA headquarters and regional offices are
ultimately responsible for implementing the triennial reviews, they
have increasingly relied on contractors since the program was created
in the early 1980s. After we reported in 1992 that FTA had not fully
utilized contractors for oversight activities, it has, since 1996, used
contractors to conduct all triennial reviews. FTA, through its Office
of Program Management, awarded multiple-year, firm, fixed-price
contracts totaling about $4 million each year to four contractors to
conduct the fiscal years' 2008 through 2012 triennial reviews.
To assist contractors with the reviews, FTA developed the Triennial
Review Contractors' Guide. FTA has expanded the contractors' guide from
its original 29 pages to over 200 pages in 2008. The contractors' guide
provides the questions to ask grantees for each of the 23 triennial
review areas, a list of grantees' documents that contractors are to
examine, and a method that contractors are to use for determining
whether grantees are meeting federal requirements.
The contractors' guide assists contractors with the decision-making
process about what constitutes a deficient finding by providing
condition statements for the 23 triennial review areas. For example,
according to the contractors' guide, if the grantee has included the
lobbying prohibitions clause (that is, the prohibition from using
federal funds to lobby for additional federal funds) in its procurement
solicitations, then the grantee is not deficient; otherwise, the
grantee is deficient. The contractors' guide also contains suggested
corrective actions if a grantee is found deficient, (e.g., the grantee
must certify to FTA that it will have subgrantees, contractors, and
subcontractors comply with the lobbying prohibitions requirement.) In
another example, if FTA conducted a special procurement system review
(see table 1) in the prior two fiscal years or if one is scheduled for
the year of the grantee's triennial review, then a review of the
procurement area is not necessary and a notation is made in the
triennial review report as "not reviewed." Subsequently, contractors
enter predetermined, deficiency codes into OTRAK that correspond to
explanations for any deficient findings, herein called findings. Once
entered, FTA regional offices track and monitor the finding until the
grantee addresses it. Table 2 shows the four categories FTA uses to
document triennial review results.
Table 2: FTA Categories Used to Document Triennial Review Results:
Category: Not Deficient;
Description: Used when the contractor or FTA staff was satisfied with
the information provided by the grantee.
Category: Deficient;
Description: Findings that represent areas for which the grantee is not
adhering to federal requirements. For these findings, the grantee must
take corrective action, usually within 30, 60, 90, or 120 days after
the final report is issued.
Category: Not Applicable;
Description: Matters that do not pertain to the grantee. For example,
when reviewing the maintenance requirements, if a grantee has no
vehicles or equipment under warranty, findings pertaining to warranty
claims are not applicable.
Category: Not Reviewed;
Description: Used when the contractor did not review certain areas
during the triennial review because the grantee had recently
experienced a special oversight review in the prior two years, or was
scheduled for one in the year of the grantee's triennial review.
Source: FTA's contractors' guide.
[End of table]
FTA's Triennial Review Data Show Many Grantees Are Not Making Progress
Toward Meeting Federal Requirements, and FTA Could Do More to Improve
Grantees' Performance:
From fiscal years 2000 through 2008, triennial review data show many
grantees are not making progress toward meeting federal requirements,
as indicated by not consistently reducing grantees' findings with
federal requirements. While FTA uses several tools to help grantees
meet more requirements, especially in areas with the greatest number of
findings, it could do more to improve grantees' performance by
identifying the underlying causes and by objectively noting the
severity of the finding.
Over Two-Thirds of the 424 Grantees We Analyzed Did Not Consistently
Meet More Federal Requirements From Fiscal Years 2000 through 2008, But
Practices Used by Some Grantees Could Improve Other Grantees'
Performance:
Our analysis of OTRAK data shows that 51 percent of 424 grantees had
mixed results in terms of meeting more triennial review requirements
from fiscal years 2000 through 2008, as demonstrated by both increases
and decreases in their numbers of findings over three consecutive
reviews. Over the same time period, 17 percent of grantees consistently
met fewer requirements from one review to the next, as indicated by a
consistent increase in the numbers of findings in their triennial
reviews.[Footnote 10] Thirty-one percent of grantees consistently met
more federal requirements over time--one of the goals of the triennial
review--as demonstrated by a consistent decrease in their numbers of
findings over three consecutive reviews. Finally, 1 percent of grantees
met the same number of requirements in all three reviews. Figure 3
summarizes the performance in terms of meeting federal requirements for
424 of the approximately 600 current urban area formula fund grantees,
as measured by the change in the number of findings in three triennial
reviews from fiscal years 2000 through 2008.
Figure 3: Grantees' Performance, as Measured by the Change in the
Number of Findings from Fiscal Years 2000 through 2008:
[Refer to PDF for image: pie-chart]
No increases or decreases: 1% (6 grantees);
Increase: 17% (71 grantees);
Decrease: 31% (133 grantees);
Both increases and decreases: 51% (214 grantees).
Source: GAO analysis of FTA data for 424 grantees.
[End of figure]
While comparing the change in the numbers of findings over time does
not show improvement across all grantees, a different analysis of OTRAK
data shows that some grantees averaged fewer than two findings per
triennial review, which we identify for this analysis as high-
performing grantees.[Footnote 11] Figure 4 illustrates the range in the
average number of findings after three triennial reviews for each of
the 424 grantees from fiscal years 2000 through 2008.
Figure 4: The Range in the Average Number of Findings After Three
Triennial Reviews for Each of the 424 Grantees from Fiscal Years 2000
through 2008:
[Refer to PDF for image: vertical bar graph]
The graph plots the Grantees (424 total) against the average number of
findings per triennial review. The number of findings range from 0 to a
high of 23.
Twenty high-performing grantees averaged fewer than 2 findings per
triennial review.
Source: GAO analysis of FTA data for 424 grantees.
[End of figure]
This analysis of OTRAK data shows that about 31 of 424 grantees
averaged fewer than 2 of 229 possible findings per triennial review.
[Footnote 12] Twenty-six of these grantees provide transportation
services to small or medium-sized urban communities, located in various
parts of the country.[Footnote 13]
Three of the 10 grantees we interviewed were high-performing grantees.
[Footnote 14] Executives representing these three grantees implemented
some management practices that they believe contributed to their high
performances. One management practice included linking the job
descriptions of pertinent individuals to the triennial review--a
practice that grantee executives said contributed to a culture of
accountability. Specifically, the controller's and grants
administrator's job descriptions for the large grantee explicitly
required these individuals to be responsible for the triennial review.
[Footnote 15] An assistant director of public works, responsible for
managing the transit system of a small grantee, was required to help
implement FTA requirements, among other things. In addition, according
to the chief executive officer of the medium grantee, the officer set
verbal job expectations with managers of zero findings, which the
officer believed contributed to the high performance in the triennial
review.
Another practice implemented by 2 of the 3 high-performing grantees was
to initiate external reviews by independent consultants to assess their
operations, which contributed to, or is expected to sustain, their high
performance. Executives from a large grantee we interviewed attributed
their high performance to an external review conducted several years
ago that focused on the grantee's readiness for the triennial review.
Despite having zero findings in their last two consecutive triennial
reviews, executives from the medium grantee recently completed an
external maintenance review that they believe should help them achieve
their goal of operating an efficient transit system and that is
necessary to remain good stewards of public funding.
FTA officials recognize the importance of sharing best practices among
grantees. As such, it advocates a grantee mentoring system where
regional offices, within resource and budget restraints, initiate,
encourage and direct grantees to team with recognized grantee leaders
in order to improve grantee performance. FTA officials also indicated
that they address mentoring during their triennial review workshops.
FTA Helps Grantees Meet More Requirements, Especially in Five Review
Areas with Most Findings; However, FTA Does Not Identify the Underlying
Causes of Findings:
When we analyzed OTRAK data from fiscal years 2000 through 2008 by
area, we found that the 424 grantees had the greatest numbers of
findings in five review areas--procurement, drug and alcohol program,
maintenance, technical, and satisfactory continuing control areas.
[Footnote 16] These five areas accounted for about 54 percent of all
findings over the last nine years. This may not be unexpected since we
also found a positive relationship between the number of questions
asked for certain areas and the number of findings. Specifically,
grantees tended to have higher numbers of findings in those areas where
the contractors' guide requires contractors to ask more questions when
compared to other areas. For example, for fiscal year 2008, contractors
asked grantees 20 questions in the procurement area, as compared to 2
or 3 questions in other areas, such as school bus and half-fare, which
had far fewer findings.[Footnote 17] Figure 5 shows the percentage of
triennial review findings for the top 5 areas, compared to the other 18
areas for the same 424 grantees over the nine-year period.
Figure 5: Percentage of Triennial Review Findings for the Top Five
Areas, from Fiscal Years 2000 through 2008:
[Refer to PDF for image: pie-chart]
Satisfactory Continuing Control: 9%;
Technical: 10%;
Maintenance: 10%;
Drug and Alcohol Program: 11%;
Procurement: 14%;
Other 18 Review Areas: 46%.
Source: GAO analysis of FTA data for 424 grantees.
[End of figure]
FTA has also analyzed OTRAK data on past triennial reviews, identified
similar areas with the greatest number of findings, and provided
resources at a national level to help grantees address the findings in
these areas. According to several end of year reports, FTA analysis of
deficiency codes related to the procurement area showed grantees
lacking justification for non-competitive awards, lacking open
competition, having no contract administration system, and having no
written record of procurement history. For the drug and alcohol area,
deficiency codes pointed to grantees having policies that were out-of-
date or lacked required elements, and conducting random drug and
alcohol tests below the federally required level. To help grantees
improve their performance in areas with the greatest number of
findings, FTA has provided an increasing amount of technical assistance
to individual grantees; workshops that focus on, among other things,
procurement related matters; and training. In its 2003 end of year
report, FTA noted that a decline in findings from fiscal year 2000 to
2003 indicated the effectiveness of its efforts to improve grantees'
performance.
We could not definitively determine whether, or to what degree, FTA's
technical assistance, workshops, and training improved grantees'
performance because while workshops increased from 2005 to 2008, so did
the total number of findings. Other factors could have contributed to a
reduction of grantees' findings over the last nine years.[Footnote 18]
The reduction in findings could have been caused by contractors
reporting that they did not review certain areas for some grantees that
had recently experienced a special oversight review or by FTA
conducting a pilot study in 2005 to streamline the triennial review
process, which also resulted in some areas not being completely
reviewed. Further, grantees, as well as FTA headquarters and regional
officials, cite a shift over the years in how FTA conducts the
triennial reviews--from a "gotcha" auditing approach to a customer-
service approach, helping grantees quickly resolve findings during the
reviews and lessening the need to report the findings.
Although FTA analyzes deficiency codes in OTRAK data to determine
problem areas, these codes alone do not describe the underlying causes
for the findings. Generally accepted government auditing standards
recommend that federal agencies, such as FTA, identify, to the extent
possible, the underlying cause in developing findings as this provides
persuasive evidence on the factor(s) responsible for the problem.
[Footnote 19] Deficiency codes pointing to grantees lacking open
competition or not having a written record of their procurement
history, for example, is not an underlying cause, as grantees' ability
to meet procurement requirements may be caused by grantees not having
sufficient or knowledgeable staff to initiate a competitive contracting
process. If FTA would identify these as common underlying causes, it
could prescribe, as part of the necessary corrective action, that
grantees attend specific procurement training courses. Knowing the
common underlying causes of grantees' findings could provide FTA with
valuable information to help many grantees find solutions to correct
their findings and improve grantees' long-term performance. FTA
officials stated that during the workshops, they plan to emphasize ways
to identify systemic deficiencies.
Officials expressed concern about their authority to use oversight
funds for the purpose of undertaking additional efforts to help the
grantees meet or exceed requirements by identifying the common
underlying causes for less than satisfactory levels of performance.
Officials told us that they have the authority to use oversight funds
for providing individual grantees with technical assistance--
assistance necessary to satisfactorily address specific findings
resulting from the grantee's triennial review. However, they
acknowledged that they can help grantees develop capabilities to
improve their long-term performance in other ways, such as National
Transit Institute courses, that do not require using oversight funds.
FTA Plans to Improve the Data It Collects on the Severity of Findings:
While FTA collects data on the numbers, the areas, and the reasons for
the findings by the deficiency codes in OTRAK, the information it
collects on severity of findings is not objective. OTRAK requires that
contractors and FTA regional staff enter a level of severity--"high,"
"medium," and "low"--for each finding. However, individuals entering
severity data into OTRAK labeled almost 98 percent of findings as
medium, which makes it difficult for FTA to prioritize technical
assistance in problem areas. According to FTA officials, contractors or
regional staff made judgments about the level of severity because FTA
has not developed definitions from which to make this judgment. This
could mean the information on severity is not consistent across the
country. FTA officials said that they plan to improve their data
collection efforts for the fiscal year 2010 triennial reviews by
establishing definitions for the three levels of severity, which could
provide more meaningful findings data and add significant value to the
information FTA collects to target its technical assistance for the
purposes of improving grantees' performance.
The Triennial Review Program Uses Some Strong Management Practices, but
Can Be Improved in a Few Areas:
Strengths of the triennial review include a well-defined process, the
increased use of an electronic tracking system to monitor grantees'
performance, increased timeliness in issuing triennial review reports,
and the willingness of FTA staff to seek out opportunities for
continuous improvement. However, we found a few areas where FTA could
improve the triennial review process by ensuring and documenting
grantees receive a complete review at least once every three years and
by coordinating the schedules of multiple oversight reviews in the same
fiscal year.
The Triennial Review Program Has A Well-Defined Process and Continues
to Improve:
One strength of the triennial review program is that the process is
well defined. FTA Order 5400.1 and the Triennial Review Contractors'
Guide detail the roles and responsibilities for those involved in the
triennial review, including FTA headquarters, regional offices,
contractors, and grantees. These documents outline the process,
identify the questions that contractors are to ask grantees, specify
the types of documents that grantees are to provide to FTA, and
describe any follow up required of regional staff. Preparing policies
and procedures that outline individual responsibilities meets an
important element of strong federal grant accountability best
practices.[Footnote 20]
Another strength is an increased use of an information system to track
and monitor grantees. In 1998, we criticized FTA for not making full
use of its information system because its regional staff had not been
uniformly updating the system and its headquarters staff was not
enforcing its use as a management tool.[Footnote 21] FTA upgraded OTRAK
in 2003 from a local-area computer network to a Web-based system, which
allows headquarters and regions to share information and track
oversight activities more efficiently. FTA has continued to
incrementally update this system. In addition, the contractors' guide
directs contractors to input the findings from each review into OTRAK
and all regional staff and contractors we spoke with reported using
OTRAK.[Footnote 22] Performance information, such as that recorded in
OTRAK, helps agencies identify, prioritize, and manage potential at-
risk grantees.[Footnote 23]
FTA has strengthened its triennial review reporting process by issuing
more timely triennial review reports than in previous years. In 1998,
we criticized FTA for taking an average of 182 days after the site
visit to process its 1995 and 1996 triennial review reports and
recommended that FTA follow its established 90-day time frame.[Footnote
24] FTA subsequently established a performance measure in 2001 of
issuing 95 percent of triennial review reports within 30 days of
completing a review. FTA has significantly improved its performance in
issuing reports in recent years. FTA reported it issued between 82 to
94 percent of triennial review reports within 30 days of completing the
review from fiscal year 2004 through 2008. During that same time
period, only 2 of 902 reports were issued more than 60 days after the
review.
Finally, FTA has looked for ways to continuously improve by contracting
with Booz Allen Hamilton to examine its oversight review processes and
recommend ways to improve them. In January 2008, Booz Allen Hamilton
issued its final report, outlining four areas of improvement: program
planning, program execution, program evaluation, and change
management.[Footnote 25] We found that FTA is taking steps to respond
to recommendations in these areas, including developing a strategic
approach for prioritizing each of the Booz Allen Hamilton
recommendations and developing a Web-based, master calendar feature in
OTRAK that is expected to improve the visibility of future oversight
review schedules to assist both FTA headquarters and regional staff in
coordinating the oversight review schedules.
FTA Could Better Ensure and Document that Grantees Receive a Complete
Review at Least Once Every Three Years:
FTA is required to conduct a complete review and evaluation of
grantees' adherence with statutory and administrative requirements at
least once every three years.[Footnote 26] In reviewing 198 fiscal year
2008 triennial review reports, we identified, but could not confirm, 33
instances where FTA may not have completely reviewed and evaluated
grantees in all areas. Specifically, contractors recorded in the 2008
triennial review reports that certain areas were "not reviewed" because
FTA had conducted special oversight reviews on these areas in the prior
two years--fiscal years 2006 or 2007. For example, contractors recorded
in the triennial review reports of ten grantees that the drug and
alcohol program area was "not reviewed" during the 2008 triennial
review because these grantees had received special drug and alcohol
program reviews in fiscal year 2006. Because the area was not reviewed
during the 2008 triennial review, these grantees may not be reviewed in
this area for 5 years (this area was last reviewed during the 2006
special oversight review and will not be reviewed again until the 2011
triennial review). While FTA's contractors' guide describes the
flexibility to not review certain areas, FTA regional office staff
provides triennial review contractors with the approval to not review
certain areas when a special oversight review has been conducted in the
prior two years or is scheduled in the fiscal year of the triennial
review.[Footnote 27]
FTA officials indicate that their practice is to update its review and
evaluation of an area having an oversight review in the prior two
years. However, the guidance does not always instruct contractors to
document this update in the triennial review reports to help FTA ensure
it meets the statutory requirement of a complete review at least once
every three years. As an example, the contractors' guide describes such
a process for reviewing and evaluating the planning area, telling
contractors to review previous planning certification review reports
during the triennial review desk review, verify the status of any
findings, and determine if corrective actions have been implemented on
schedule.[Footnote 28] The contractors' guide does not provide similar
guidance for other areas, including the drug and alcohol program area,
and we did not see--nor could FTA provide us with--any such
documentation in the 2008 triennial review reports. In commenting on
our draft report, FTA said it would revise applicable guidance,
including the contractors' guide for fiscal year 2010, to avoid
ambiguity and ensure consistency within the triennial review program.
FTA headquarters officials indicated that regional officials and
contractors discuss all 23 areas during the triennial review process
and identify concerns about the grantees. Specifically, headquarters
officials explained that regional staff and contractors discuss the
results of special oversight reviews conducted in the prior two years,
such as any outstanding findings from the prior reviews, and reach
consensus on the extent to which a grantee would be reviewed in the
related area during the triennial review, ranging from an area not
being reviewed at all to a more detailed review in the area, depending
on the situation. However, officials acknowledged that the results of
those discussions are not always documented in the triennial review
reports, especially in regards to the extent to which areas are to be
reviewed or not reviewed during the triennial review. Because of this
lack of documentation, we could not confirm in the 33 instances noted
above whether or to what extent all 23 areas were reviewed and
evaluated.
By allowing flexibility in the contractors' guide to not review certain
areas, FTA may not be ensuring that all areas are completely reviewed
and evaluated at least once every three years, as required by statute.
In addition, because grantees can and do experience findings during
triennial reviews, despite having special oversight reviews in the
prior two years, updating previously conducted special oversight
reviews is important. We identified seven instances where grantees had
received special oversight reviews in the financial management and
civil rights areas in fiscal years 2006 and 2007, yet during the 2008
triennial review, findings in these areas resurfaced. Finally, proper
evaluation and documentation provides future reviewers with a more
transparent audit trail of findings and gives FTA managers the
confidence that grantees were completely reviewed and evaluated in all
23 areas at least once every three years. Officials plan to refine and
use OTRAK to document, among other things, the results of special
oversight reviews that occurred in the years between triennial reviews,
which would help ensure grantees adhere to requirements.
Uncoordinated Scheduling of Multiple Oversight Reviews May Unduly
Burden Some Grantees:
We found a limited number of instances in which grantees may have been
unduly burdened by uncoordinated scheduling of multiple oversight
reviews in the same fiscal year. We found that 23 of 198 grantees (12
percent) receiving a triennial review in fiscal year 2008 also received
one or more special oversight reviews in the same year. Twenty-four of
183 grantees (13 percent) received multiple reviews in fiscal year
2007, including one medium sized grantee that underwent a special
procurement system review in January 2007, a drug and alcohol program
review in February 2007, and a triennial review in June 2007. [Footnote
29] The same grantee also underwent two special civil rights reviews--
an Americans with Disabilities review in October 2007 and a
Disadvantaged Business Enterprises review in November of 2007.
One grantee with whom we spoke said that although the triennial review
is beneficial, the multiple oversight reviews it received in the same
fiscal year were burdensome, redundant and delayed project schedules
and cost the grantee about $18,000 to $20,000. Four other grantees
provided information showing they spent about 100 to 300 hours
preparing for and hosting 2-to 3-day triennial reviews involving
sometimes over 20 individuals and many departments. Some grantees also
said that special oversight reviews continue for two or more weeks and
require even more grantee time and attention.
Representatives from the American Public Transportation Association--
an association that represents many grantees--also said that two or
more FTA oversight reviews in the same year could be burdensome.
According to the association representatives and grantees with whom we
spoke, multiple reviews in the same year are especially burdensome for
smaller grantees, whose employees tend to be responsible for multiple
review areas. For example, a small grantee told us that the triennial
review alone taxed its two employees who were responsible for
addressing many of the areas in the triennial review. Large grantees,
on the other hand, typically have several employees in a division
responsible for one triennial review area, so the burden of multiple
reviews may be less taxing for them than for small grantees.
The decision-making body responsible for developing and coordinating
FTA's oversight review schedules is the Oversight Review Council; but,
according to FTA officials, the Oversight Review Council is not
fulfilling these responsibilities, as it was inactive on and off in
recent years because of management changes and priorities. FTA Order
5400.1 requires the Oversight Review Council to develop and coordinate
various oversight review schedules to avoid unnecessary duplication and
overlap, among other things.[Footnote 30] The Council was reinstituted
in 2007 after management changes, and continues to redefine its roles
and responsibilities. Further, FTA is aware of the burden oversight
reviews place on grantees and works to limit this burden.
Without the Oversight Review Council taking responsibility for
developing and coordinating oversight review schedules, FTA offices
developed their own oversight processes, which, in some instances,
could lead to grantees receiving a triennial review and a special
review in the same fiscal year. For example:
* FTA's goal is to conduct special procurement reviews every three
years on the 30 grantees that receive large amounts of federal transit
funds.
* The Offices of Civil Rights and Safety and Security conduct their own
special oversight reviews, including reviews of grantees that have had
civil rights complaints filed against them. According to officials from
the Civil Rights office, after notifying the regions and others of the
schedules, the regions are to notify headquarters about any scheduling
conflicts.
* The drug and alcohol review schedules are sometimes made without
informing regional staff because this information is sensitive.
[Footnote 31]
* Other FTA officials who conduct special oversight reviews enter their
scheduled review date of grantees in OTRAK. However, regional staff
said they are sometimes not aware of special review scheduling.
[Footnote 32]
In receiving urban area formula funds, grantees agree to the
congressionally-mandated triennial review at least once every three
years. Some grantees are subject to other FTA oversight reviews. Having
an effective body to coordinate the strategic planning of special
oversight reviews, in coordination with the triennial review, could
help increase FTA's oversight coverage of grantees, while at the same
time, minimize the undue burden of uncoordinated multiple oversight
reviews on some grantees. FTA's new Web-based, master calendar feature
in OTRAK is expected to improve the visibility of future oversight
review schedules.
FTA's Two Triennial Review Performance Measures Meet Some, But Not All
Key Attributes of Successful Performance Measures:
FTA established two performance measures to assess timeliness of steps
in the triennial review program: (1) 80 percent of triennial review
findings are to be closed within 30 days of their due date ("Close
Findings Timely"); and (2) 95 percent of the final triennial review
reports are to be issued within 30 days of the completion of the review
("Issue Reports Timely").[Footnote 33] While these measures meet some,
they do not meet all key attributes of successful measures, as
described in our past work on the Government Performance and Results
Act of 1993.[Footnote 34] Further, FTA lacks performance measures that
assess other core elements of FTA's triennial review program, including
the outcome and quality of the reviews. FTA officials recognize the
limits of their current performance measures and have begun efforts to
enhance them. Table 3 summarizes the 9 key attributes we used to
analyze FTA's performance measures.[Footnote 35]
Table 3: Nine Key Attributes of Successful Performance Measures:
Key Attribute: Linkage;
Definitions: Measure is aligned with division and agencywide goals and
mission and is clearly communicated throughout the organization.
Key Attribute: Measurable target;
Definitions: Measure has a numerical goal.
Key Attribute: Clarity;
Definitions: Measure is clearly stated and the name and definition are
consistent with the methodology used to calculate it.
Key Attribute: Limited overlap;
Definitions: Measure should provide new information beyond that
provided by other measures.
Key Attribute: Objectivity;
Definitions: Measure is reasonably free from significant bias or
manipulation.
Key Attribute: Reliability;
Definitions: Measure produces the same result under similar conditions.
Key Attribute: Core program activities;
Definitions: Measures cover the activities that an entity is expected
to perform to support the intent of the program.
Key Attribute: Governmentwide priorities;
Definitions: Measures should cover governmentwide priorities, such as
quality, timeliness, and outcome.
Key Attribute: Balance;
Definitions: Balance exists when the suite of measures ensures that an
organization's various priorities are covered.
Source: GAO summary.
Note: For an expanded explanation of these key attributes, see appendix
I of GAO-03-143.
[End of table]
Figure 6 compares FTA's triennial review performance measures to these
key attributes, showing that they meet some but not all attributes.
Both measures meet 4 of 9 attributes; that is, they link to agency
goals, provide a measurable target, are clear, and have limited overlap
with other measures. The Issue Reports Timely measure meets 2
additional attributes by demonstrating objectivity and reliability.
Figure 6: Comparison of FTA Triennial Review Performance Measures to
Key Attributes:
[Refer to PDF for image: illustrated table]
Performance measure: Close finding timely;
Key Attribute: Linkage;
Status: Key attribute met.
Performance measure: Close finding timely;
Key Attribute: Measurable target;
Status: Key attribute met.
Performance measure: Close finding timely;
Key Attribute: Clarity;
Status: Key attribute met.
Performance measure: Close finding timely;
Key Attribute: Limited oversight[A];
Status: Key attribute met.
Performance measure: Close finding timely;
Key Attribute: Objectivity;
Status: Key attribute not met.
Performance measure: Close finding timely;
Key Attribute: Reliability;
Status: Key attribute not met.
Performance measure: Close finding timely;
Key Attribute: Core program activities[B];
Status: Key attribute not met.
Performance measure: Close finding timely;
Key Attribute: Government-wide priorities;
Status: Key attribute partially met.
Performance measure: Close finding timely;
Key Attribute: Balance[A];
Status: Key attribute not met.
Performance measure: Issue reports timely;
Key Attribute: Linkage;
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Measurable target;
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Clarity;
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Limited oversight[A];
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Objectivity;
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Reliability;
Status: Key attribute met.
Performance measure: Issue reports timely;
Key Attribute: Core program activities[B];
Status: Key attribute not met.
Performance measure: Issue reports timely;
Key Attribute: Government-wide priorities;
Status: Key attribute partially met.
Performance measure: Issue reports timely;
Key Attribute: Balance[A];
Status: Key attribute not met.
Source: GAO analysis of FTA data.
[A] This attribute applies to the overall suite of measures, rather
than to the measures individually.
[B] The core program activities of the triennial review program is to
ensure grantees meet applicable federal requirements.
[End of figure]
* Linkage. We found FTA's two triennial review performance measures
link throughout the organization--from FTA regional offices, through
headquarters, and to the department level. Both performance measures
aligned with the Department of Transportation's strategic goal of
organizational excellence in the FTA Annual Performance Plan, FTA
regional business and strategic plans, and individual performance
appraisals.[Footnote 36] In addition, FTA officials--including
officials in the Performance Management Division, the Office of Program
Management, and all 10 regional offices--confirmed in interviews that
performance goals were communicated across the agency. Such
communication is important to ensure FTA officials understand what the
organization is trying to achieve and the goals it seeks to reach.
* Measurable target. Both performance measures also have measurable
numeric targets--given in percentages--that allow FTA officials to
easily assess whether overall goals and objectives were achieved.
Numeric targets facilitate assessments of whether objectives were
achieved because comparisons can be easily made between projected
performance and actual results. FTA reports meeting the Close Findings
Timely measure in fiscal years 2004 through 2007; and in fiscal year
2008, closed 74 percent of findings within 30 days. Although FTA
reports never meeting the Issue Reports Timely measure in fiscal years
2004 through 2008, it has greatly improved the timeliness with which it
issues triennial review reports. From fiscal years 2004 through 2008,
FTA nearly met its goals in 4 of the 5 years.[Footnote 37]
* Clarity. We found the performance measures are clearly stated and
each has a definition that is consistent with the methodology used to
calculate it. The Issue Reports Timely formula is the percentage of
reports issued within 30 days of the final conduct of the review, and
the Close Findings Timely formula is the number of findings closed
within 30 days of the due date, divided by the sum of the number of
findings closed plus the number of findings open that are 30 days past
the due date. Clear measures provide transparency and allow managers to
accurately assess FTA's performance in issuing triennial review reports
and closing findings in a timely manner.
* Limited overlap. Both program performance measures have unique names,
definitions, and do not overlap. One measure evaluates the timeliness
of issuing reports, while the other assesses the timeliness of closing
findings. The presence of distinct performance measures prevent
managers from sifting through unnecessary or redundant information.
* Objectivity. We found the Issue Reports Timely measure is objective
because the data elements used to calculate it are free from
significant bias. However, our analysis of the Close Findings Timely
measure found that its objectivity is limited because decisions to
close findings, establish due dates for when grantees must address
findings, and extend, if necessary, any due dates, is done largely at
the discretion of regional staff, in consultation with grantees, and,
when appropriate, headquarters officials. This presents opportunities
to influence this measure because regional staff may use different
criteria for when to close or extend deadlines. While FTA officials
recently met to discuss a mandatory process to uniformly close findings
across all FTA regions, the current lack of criteria reduces the
transparency of FTA's performance in closing findings. For example, an
FTA official stated that the Closing Findings Timely performance
measure could result in the unintended consequence of prematurely
closing findings. The annual performance appraisals and resulting
compensation for regional staff, including the Regional Administrators,
are based, in part, on meeting the Close Finding Timely performance
measure. While we did not find widespread instances when findings were
prematurely closed, an official from the civil rights office stated
there have been cases when regional staff closed triennial review
findings in OTRAK and requested that the civil rights office monitor
the grantees' progress in resolving them.
* Reliability. We found the Issue Reports Timely measure is reliable,
as FTA has standard procedures for collecting data, and based on our
examination of individual triennial review reports, accurately stated
the number of reports issued in the most recent end of year report for
fiscal year 2008. However, our analysis of data used to calculate the
Close Finding Timely performance measure found data inaccuracies and,
therefore, the measure does not meet the key attribute of reliability.
After examining closed findings data we found several instances where
OTRAK reported findings open more than seven years after they were due
for closure.[Footnote 38] FTA officials familiar with the database said
that these inaccuracies resulted from FTA regional staff not entering
data into OTRAK after the transition to the new Web-based information
system in 2003 and said, but could not document, that the findings were
likely closed. However, we also found OTRAK data entry errors from
fiscal years 2005 through 2007--after FTA had transitioned to the Web-
based system--where the database showed almost 200 findings were open
more then one year after they were due for closure. Headquarters
officials said, but could not document, that these findings are
inaccurately listed as open in OTRAK and were likely closed at the
regional level. Officials further stated they plan to re-examine the
regional process for closing findings and issue guidance to improve
this process. Finally, the 2008 Booz Allen Hamilton report noted
instances of missing data in OTRAK because some regions were using
stand alone spreadsheets instead of fully using the database to report
closing findings.[Footnote 39] Officials acknowledged that entering
closed findings data into the database can be problematic, especially
for inexperienced regional staff not familiar with OTRAK. Officials
believe that a 2007 decision to tie regional staff performance
appraisals to data entered into OTRAK should provide an incentive to
ensure future data are reliable. In addition, officials said they are
emphasizing the importance of recording data into OTRAK to regional
offices and contractors.[Footnote 40] Without reliable data, FTA
managers cannot be confident in the extent to which performance goals
have been achieved, including the timeliness in closing findings.
* Core program activities. Both triennial review performance measures
do not meet the triennial review program's core program activity,
evaluating whether grantees meet federal requirements over time. While
FTA's existing performance measure of Close Findings Timely may
increase adherence to federal requirements by encouraging grantees to
address findings, it does not provide an overall program level
assessment of grantees' performance over time. In addition, FTA
measures annual grantee compliance information in Triennial Review
Program end-of-year reports, but has not developed performance measures
that set targets to evaluate progress. Therefore, FTA does not assess
whether the triennial review program is increasing grantees' ability to
continuously meet federal requirements. FTA could measure the core
program activity of the triennial review program in various ways, such
as assessing grantees' performance at the end of each review year or
assessing grantees' long term performance, as our previous analysis of
findings data shows. See figure 3.
* Governmentwide priorities. FTA's current performance measures
partially address governmentwide priorities by ensuring that FTA
reports on its timeliness in conducting reviews and in addressing
corrective actions. However, the measures do not evaluate other
governmentwide priorities, including the outcome of the reviews--the
extent to which grantees are meeting federal requirements, or the
quality of the triennial reviews.[Footnote 41]
- Outcome. The performance measures assess the output of the triennial
review program--the closing of findings and issuing of final reports--
but, not the outcome--grantees' ability to meet federal
requirements.[Footnote 42] As previously stated in the discussion of
core program activities, outcome performance measures--such as
grantees' performance over time, or annual progress against a
measurable target--can help evaluate the results of the triennial
review programs. Without such a measure, FTA is limited in the extent
to which it can effectively ascertain and document the extent to which
the triennial review program is improving grantees' performance in
meeting federal requirements and FTA managers have limited information
on program performance, making the possibility of achieving program
goals less likely.
- Quality. FTA has no performance measures for assessing the quality of
the triennial review program. FTA officials help ensure quality by
checking triennial review reports to determine whether the contractors
made reasonable findings for grantees, determining what actions
grantees must take to resolve findings, and attending some site visits
to oversee contractors. However, FTA officials are unsure how to
develop a measure for evaluating the quality of the triennial review
program. In addition, Booz Allen Hamilton reported that FTA
headquarters and field staff do not have a common understanding of what
constitutes a quality review or how it could be measured. Without
common quality standards, FTA cannot evaluate the overall quality of
the program nor ensure the results and objectives of the triennial
review program are being met. Performance measures related to quality
could assess the implementation of triennial reviews, including a way
to oversee contractors and test the overall quality of reviews. Many
federal agencies periodically review their programs to ensure they meet
quality standards. FTA officials recognize the benefits of such
performance measures and plan to enhance its current measures and
standards to assess the quality and value added of the triennial review
program to help ensure that the results and objectives of the triennial
review program are being met.
* Balance. FTA's triennial review program performance measures address
only one of three governmentwide priorities and, therefore, are not
balanced--an attribute that applies to the overall suite of measures,
rather than the measures individually. Balanced performance measures
allow managers to understand what other triennial review aspects are
working well and why. Adding outcome and quality performance measures
will help balance the existing timeliness measures and could allow
managers to better set triennial review priorities by not
overemphasizing one or two priorities at the expense of others. FTA
officials recognize current performance measures do not assess some
aspects of the triennial review program and are undertaking efforts to
enhance them.
Conclusions:
Strong FTA oversight of federal surface transportation grant programs
helps ensure that grantees meet federal requirements, provide safe
public transportation, and safeguard from misuse the billions of
dollars that Congress appropriated for surface transportation programs
to help America recover from the economic downturn and the billions
more that Congress plans to reauthorize for surface transportation.
FTA's efforts over the past decade to improve the congressionally-
mandated triennial review--the primary means for overseeing hundreds of
grantees--have produced concrete results. For example, FTA now
regularly collects oversight data on grantees, issues the vast majority
of triennial review reports within 30 days of the review (a significant
improvement since our last report in 1998), and provides grantees with
technical assistance to address problem areas. Nonetheless, FTA should
undertake additional efforts to further improve the triennial review
program.
Although one of the goals of the triennial review program is to assist
grantees in meeting more federal requirements over time, our analysis
of OTRAK data shows many grantees with mixed success in reducing their
numbers of findings over three triennial reviews from fiscal years 2000
through 2008; and 17 percent of the grantees we reviewed met fewer
federal requirements in their latest triennial review than they did in
their two previous reviews. While FTA analyzes apparent problem areas,
its analysis does not provide information on the underlying causes of
findings and information on the level of severity for each finding.
Without conducting further analyses of its OTRAK data and better
understanding the underlying causes of grantees' performance in
triennial reviews, FTA may be missing valuable opportunities to
increase its effectiveness in helping grantees improve their
performance.
The triennial review process could be improved in two respects. First,
when a grantee has recently received or is scheduled to receive a
special oversight review, the flexibility currently allowed in the
contractors' guide could lead to confusion about how to document that
all 23 triennial review areas are completely reviewed at least once
every three years. Until FTA revises its guidance to remove the "not
reviewed" category, require an updated evaluation of areas reviewed in
the prior two years of the triennial review, and describe how to
document triennial review reports with the updates, FTA does not
provide assurance that grantees are being assessed equally and are
receiving a complete review and evaluation at least once every three
years, as required. Second, until the Oversight Review Council performs
its role as oversight review coordinator or delegates this role to
another FTA body, uncoordinated, multiple oversight reviews may not
effectively increase FTA's coverage of grantees and may place undue
burden on a few grantees.
Finally, although FTA's two triennial review performance measures meet
some of the key attributes we have previously reported are necessary
for a successful program, the Close Findings Timely measure could be
improved to minimize the possibility for manipulation and increase the
reliability of closed finding data. Without an internal control process
to verify whether grantees have satisfactorily resolved findings, FTA
cannot determine the validity of reported results for the Close
Findings Timely performance measure. Without reliable data, FTA may be
unable to determine the validity of reported results and make
appropriate decisions. FTA also lacks performance measures for
evaluating the outcome and quality of the triennial review program.
Such performance measures are essential because they can help decision
makers determine areas for improvement and increase the likelihood of
achieving program goals, such as increasing the number of grantees that
meet federal requirements.
Recommendations for Executive Action:
To support the triennial review program's goal of having grantees
consistently meet more federal requirements, the Secretary of
Transportation should direct the FTA Administrator:
1. To further analyze and use OTRAK data on the triennial review to
help grantees improve and sustain their performance. As part of this
analysis, FTA should consider and identify, when necessary: (1) high
performing grantees and their management practices, among other things;
(2) the areas in which problem findings occur, and the underlying
causes of these findings; and (3) the level of severity for each
finding.
2. To help ensure that a complete performance review and evaluation of
each grantee is conducted at least once every three years, in
accordance with the statutory requirement, revise the contractors'
guide to remove the "not reviewed" category, require an updated
evaluation of areas reviewed in the prior two years of the triennial
review, and describe how contractors should document the triennial
review reports with updates to other reviews, such as special oversight
reviews conducted in the prior two years.
3. To strengthen the triennial review process by ensuring that the
Oversight Review Council, or another body, carries out the
responsibilities specified in FTA Order 5400.1 to be actively involved
in coordinating oversight review schedules in order to increase FTA's
oversight coverage of grantees while, at the same time, minimizing the
undue burden of multiple oversight reviews on grantees.
4. To improve the objectivity and reliability of the "Close Findings
Timely" performance measure by:
* developing an internal controls process to verify grantees have
satisfactorily resolved findings before closing them;
* continuing to emphasize the use of OTRAK across all regions and
developing a process to close out triennial reviews in OTRAK upon
grantees' completion of corrective actions.
5. To meet standards set forth in previous GAO work, based on the
Government Performance and Results Act of 1993, and develop performance
measures to assess:
* the outcomes of the triennial review program, such as a method for
evaluating improvements in grantee performance in meeting more federal
requirements over time; and:
* the quality of the triennial review through improved contractor
oversight, testing, or inspection.
In addition, FTA should ensure additional outcome and quality
performance measures are balanced with the existing measures for
assessing the timeliness of the triennial review program so that one
priority is not emphasized at the expense of others.
Agency Comments:
The Department of Transportation reviewed and commented on a draft of
this report. The department generally agreed with the report's findings
and recommendations. It also suggested that we provide more information
on FTA's efforts to review grantees at least once every three years and
coordinate its oversight reviews, which we added, as appropriate. The
Department also provided a technical correction, which we have
incorporated.
As agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from the report date. At that time, we will send copies to appropriate
congressional committees, to the Secretary of Transportation, and to
appropriate FTA officials. The report will also be available at no
charge on the GAO Web site at [hyperlink, http://www.gao.gov].
If you have any questions about this report, please contact me at (202)
512-2834 or herrp@gao.gov. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made major contributions to this
report are listed in appendix III.
Signed by:
Phillip R. Herr:
Director, Physical Infrastructure Issues:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
To determine the extent to which the triennial reviews indicate
grantees met federal requirements from fiscal years 2000 through 2008,
we reviewed documentation on the Federal Transit Administration's (FTA)
Oversight Tracking System (OTRAK)--the official electronic information
system for tracking and monitoring oversight activities. Documents
included a system description, requirements, and background; reference
guides; data entry controls and processes; data dictionary; deficiency
codes; and the users' manual. We requested and obtained OTRAK findings
data by grantee and by area from fiscal years 2000 through 2008. Data
prior to 2000 was not available. We sorted the data to, among other
things, determine the extent to which grantees' performance in meeting
federal requirements increased, decreased, both increased and
decreased, or remained constant in three triennial reviews; which
grantees averaged fewer than two findings in their triennial reviews;
and which areas accounted for the greatest numbers of findings. We did
not have direct access to the OTRAK database, but took several steps to
assess the reliability of the data. We performed electronic testing to
identify missing data, obtained and reviewed the queries that FTA used
to generate the data that we requested, and discussed with FTA and its
OTRAK contractors the results of the information. Based on these tests,
we excluded 150 grantees because they did not meet our criteria, having
exactly three triennial reviews in the last nine years and excluded 27
other grantees because, according to FTA officials, they received
triennial reviews for multiple triennial review cycles, more than 3
fiscal years, or both as a result of triennial reviews that were either
canceled or closed without findings.
To address the strengths and weaknesses of the triennial review
process, we reviewed federal guidance for improving grant
accountability, federal internal control standards, GAO reports on
performance measures, FTA Order 5400.1, triennial review reports of
numerous grantees, and a 2008 Booz Allen Hamilton report of FTA's
oversight programs. We also reviewed and compared schedules for the
triennial and other special oversight reviews by grantee from fiscal
years 2006 through 2008 to determine which grantees experienced both a
triennial review and at least one special oversight review in fiscal
year 2008. In addition, we examined specific areas reviewed or not
reviewed during fiscal years 2007 and 2008 triennial review reports and
compared the areas to completed or planned special oversight reviews.
We attended the triennial review of the Los Angeles County Metropolitan
Transit Authority in September 2008. We also attended a FTA triennial
workshop for new grantees in Kansas City, Missouri in July 2008.
We selected 10 grantees with a range of findings, from various FTA
regions, having special oversight reviews in fiscal years 2006, 2007,
or 2008, and from different population sizes--small (50,000-199,999),
medium (200,000 to 999,999), or large (1 million or more) urban
centers. We interviewed these grantees in order to understand grantees'
practices that contributed to them averaging fewer than 2 findings per
review, understand the burden multiple reviews had on grantees'
operations, and understand if the triennial review unduly burdened
grantees with different size operations. The following lists the 10
grantees interviewed:
* Southwest Ohio Regional Transportation Agency, Cincinnati, Ohio:
* Spokane Transit Authority, Spokane, Washington:
* St. Joseph Transit, St. Joseph, Missouri:
* Utah Transit Agency, Salt Lake City, Utah:
* Southeastern Pennsylvania Transportation Authority, Philadelphia,
Pennsylvania:
* Santa Monica Blue Bus, Santa Monica, California:
* Wichita Transit, Wichita, Kansas:
* Hampton Roads Transit, Hampton, Virginia:
* City of Hattiesburg, Mississippi:
* Santa Clara Valley Transportation Authority, San Jose, California:
With respect to this and our other reporting objectives, we interviewed
officials from the following organizations:
* FTA's Associate Administrator for Program Management:
* FTA's Office of Civil Rights:
* FTA's Office of Safety and Security:
* FTA's Office of Program Management:
* FTA's Office of Performance Management:
* All FTA regional offices:
* All four triennial review contractors (Reid Consulting LLC, CDI/DCI
Joint Venture, Advanced Systems Technical and Management Inc., and
Interactive Elements LLC):
* Booz Allen Hamilton:
* The American Public Transportation Association:
* Community Transportation Association of America.
To determine FTA's performance measures for the triennial review and to
what extent FTA's performance measures meet nine key attributes, we
obtained, reviewed, and analyzed, among other documents: GAO reports
describing the importance of establishing effective performance
measures, GAO guidance on assessing the strengths and weaknesses of
federal agencies' performance measures and identifying the key
attributes of successful performance measures, FTA triennial review end-
of-year reports, and FTA performance appraisal templates. We also spoke
with FTA management to identify the current performance measures for
evaluating the success of the triennial review program, the reasons for
them, and when they were established. We then compared FTA's two
performance measures to each of the nine key attributes in order to
determine the extent to which and why they either fully met, partially
met, or did not meet all attributes.
[End of section]
Appendix II: Description of FTA's 23 Triennial Review Areas:
Table 4: FTA's Descriptions of the Requirements in Each of the
Triennial Review Areas:
FTA Designated Area: Legal;
FTA's Description of the Requirements in Each Area: The grantee must be
eligible and authorized under state and local law to request, receive,
and dispense FTA funds and to execute and administer FTA-funded
projects. The authority to take all necessary action and responsibility
on behalf of the grantee must be properly delegated and executed.
FTA Designated Area: Financial;
FTA's Description of the Requirements in Each Area: The grantee must
demonstrate the ability to match and manage FTA grant funds, maintain
and operate federally-funded facilities and equipment, and conduct an
annual independent organization-wide audit in accordance with the
provisions of OMB Circular A-133.
FTA Designated Area: Technical;
FTA's Description of the Requirements in Each Area: The grantee must be
able to implement the Urbanized Area Formula Grant Program of Projects
in accordance with the grant application, Master Agreement, and all
applicable laws and regulations, using sound management practices.
FTA Designated Area: Satisfactory Continuing Control;
FTA's Description of the Requirements in Each Area: The grantee must
maintain control over real property, facilities, and equipment and
ensure that they are used in transit service.
FTA Designated Area: Maintenance;
FTA's Description of the Requirements in Each Area: The grantee must
keep federally-funded equipment and facilities in good operating order.
FTA Designated Area: Procurement;
FTA's Description of the Requirements in Each Area: FTA grantees will
use their own procurement procedures that reflect applicable state and
local laws and regulations, provided that the process ensures
competitive procurement and that the procedures conform to applicable
federal law.
FTA Designated Area: Disadvantaged Business Enterprise (DBE);
FTA's Description of the Requirements in Each Area: The grantee must
comply with the policy of the Department of Transportation that DBEs
are ensured nondiscrimination in the award and administration of
Department of Transportation-assisted contracts. Grantees also must
create a level playing field on which DBEs can compete fairly for
Department of Transportation-assisted contracts; ensure that only firms
that fully meet eligibility standards are permitted to participate as
DBEs; help remove barriers to the participation of DBEs; and assist the
development of firms that can compete successfully in the marketplace
outside the DBE program.
FTA Designated Area: Buy America;
FTA's Description of the Requirements in Each Area: Per "Buy America"
law, federal funds may not be obligated unless steel, iron, and
manufactured products used in FTA-funded projects are produced in the
United States, unless FTA has granted a waiver, or the product is
subject to a general waiver. Rolling stock must have sixty percent
domestic content and final assembly must take place in the United
States.
FTA Designated Area: Suspension/debarment;
FTA's Description of the Requirements in Each Area: To protect the
public interest and prevent fraud, waste, and abuse in federal
transactions, persons or entities, which by defined events or behavior,
potentially threaten the integrity of federally administered programs,
are excluded from participating in FTA-assisted programs.
FTA Designated Area: Lobbying;
FTA's Description of the Requirements in Each Area: Recipients of
federal grants and contracts exceeding $100,000 must certify compliance
with restrictions on lobbying before they can receive funds. In
addition, grantees are required to impose the lobbying restriction
provisions on their contractors.
FTA Designated Area: Planning/Program of Projects;
FTA's Description of the Requirements in Each Area: The grantee must
participate in the transportation planning process in accordance with
FTA requirements; Safe, Accountable, Flexible, Efficient Transportation
Equity Act: A Legacy for Users requirements; and the Metropolitan and
Statewide Planning Regulations.
FTA Designated Area: Title VI;
FTA's Description of the Requirements in Each Area: The grantee must
ensure that no person in the United States shall, on the grounds of
race, color, or national origin, be excluded from participating in, or
denied the benefits of, or be subject to discrimination under any
program, or activity receiving federal financial assistance.
FTA Designated Area: Public Comment Process;
FTA's Description of the Requirements in Each Area: The grantee is
expected to have a written copy of a locally developed process to
solicit and consider public comment before raising a fare or carrying
out a major reduction of transportation services.
FTA Designated Area: Half fare;
FTA's Description of the Requirements in Each Area: Grantees must
ensure that elderly persons and persons with disabilities, or an
individual presenting a Medicare card will be charged, during non-peak
hours for transportation using or involving a facility or equipment of
a project financed under Section 5307, not more than 50 percent of the
peak hour fare.
FTA Designated Area: Americans with Disabilities Act;
FTA's Description of the Requirements in Each Area: Titles II and III
of the Americans with Disabilities Act of 1990 provide that no entity
shall discriminate against an individual with a disability in
connection with the provision of transportation service. The law sets
forth specific requirements for vehicle and facility accessibility and
the provision of service, including complementary paratransit service.
FTA Designated Area: Charter bus;
FTA's Description of the Requirements in Each Area: FTA grantees are
prohibited from using federally-funded equipment and facilities to
provide charter service except on an incidental basis and when one or
more of applicable exceptions set forth in the charter service
regulation applies.
FTA Designated Area: School bus;
FTA's Description of the Requirements in Each Area: FTA grantees are
prohibited from providing exclusive school bus service unless it
qualifies under specified exceptions. In no case can federally-funded
equipment or facilities be used.
FTA Designated Area: National Transit Database;
FTA's Description of the Requirements in Each Area: The grantee must
collect, record, and report financial and non-financial data in
accordance with the Uniform System of Accounts and updated with the
National Transit Database Reporting Manual.
FTA Designate Areas: Safety and Security;
FTA's Description of the Requirements in Each Area: Any recipient of
Urbanized Area Formula Grant Program funds must annually certify that
it is spending at least one percent of such funds for transit security
projects or that such expenditures for security systems are not
necessary.
FTA Designated Area: Drug-Free workplace;
FTA's Description of the Requirements in Each Area: FTA grantees are
required to maintain a drug-free workplace for all employees and to
have an ongoing drug-free awareness program.
FTA Designated Area: Drug and Alcohol Program;
FTA's Description of the Requirements in Each Area: Grantees receiving
FTA funds under Capital Grant (Section 5309), Urbanized Area Formula
Grant (Section 5307), or Non-Urbanized Area Formula Grant (Section
5311). Programs must have a drug and alcohol testing program in place
for all safety-sensitive employees.
FTA Designated Area: Equal Employment Opportunity;
FTA's Description of the Requirements in Each Area: The grantee must
ensure that no person in the United States shall on the grounds of
race, color, creed, national origin, sex, age, or disability be
excluded from participating in, or denied the benefits of, or be
subject to discrimination in employment under any project, program, or
activity receiving federal financial assistance from the federal
transit laws.
FTA Designated Area: Intelligent Transportation Systems (ITS)
Architecture;
FTA's Description of the Requirements in Each Area: ITS projects funded
by the Highway Trust Fund and the Mass Transit Account must conform to
the National ITS Architecture, as well as to Department of
Transportation adopted ITS Standards.
Source: FTA's contractors' guide.
[End of table]
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Phillip R. Herr, (202) 512-2834 or herrp@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Catherine A. Colwell (Assistant
Director), Matthew A. Cook, Elizabeth H. Curda, Bert Japikse, Delwen A.
Jones, Grant M. Mallie, Steve Martinez, Amanda K. Miller, Joshua H.
Ormond, Kelly L. Rubin, and Sandra E. Sokol made key contributions to
this report.
[End of section]
Footnotes:
[1] Formula grants, as the name suggests, are apportioned among urban
areas by a statutory formula based on population data and statistics
for transit service and ridership.
[2] Congressional Research Service Report for Congress, Public Transit
Program Issues in Surface Transportation Reauthorization, (Washington,
D.C., 2008).
[3] Public L. No. 111-5, Title XII, 123 Stat. 115 (2009).
[4] 49 U.S.C. § 5307(h)(2).
[5] We excluded 177 other grantees because they did not meet our
criteria of having 1 triennial review every 3 years and a total of
exactly 3 triennial reviews from 2000 through 2008.
[6] Grant Accountability Project, Guide to Opportunities for Improving
Grant Accountability, (Washington, D.C., 2005) and GAO, Mass Transit
Grants: If Properly Implemented, FTA Initiatives Should Improve
Oversight, [hyperlink, http://www.gao.gov/products/GAO/RCED-93-8],
(Washington, D.C.: Nov. 19, 1992).
[7] GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing
Season Performance Measures, [hyperlink,
http://www.gao.gov/products/GAO-03-143], (Washington, D.C.: Nov. 22,
2002).
[8] FTA Order 5400.1, Oversight Reviews, November 1, 1994. FTA
officials stated they plan to update the order by September 30, 2009.
[9] Congress authorized the establishment of the institute under the
Intermodal Surface Transportation Efficiency Act of 1991 (Pub.L. No.
102-240, § 6022, 105 Stat. 1914) to develop, promote, and deliver
training and education programs for the public transit industry.
[10] In our analysis, we categorized grantees as consistently meeting
fewer requirements over time if they had greater than or equal numbers
of findings in each successive review. Similarly, we categorized
grantees as consistently meeting more federal requirements if they had
less than or equal numbers of findings in each successive review.
[11] Those grantees that we identified as high performing may have had
small increases in the numbers of findings over time. However, because
they averaged fewer than 2 findings per review, we consider them high
performing. For example, one high-performing grantee had 0, 2, and 3
findings in three consecutive reviews, an average of 1.7 findings.
[12] The 229 possible findings represent the number of deficiency codes
for the fiscal year 2008 review. The total number of deficiency codes
in fiscal years 2006 and 2007 did not vary by more than 6.
[13] For the purposes of our review, we generally define small, medium,
and large by the size of urban population that the grantee services.
Small grantees serve 50,000 to 199,999 people, medium grantees serve
200,000 to 999,999 and large grantees serve 1 million or more people.
In addition, we found little correlation between the size of grantee
and the total number of findings.
[14] These grantees vary in size and by the FTA regions in which they
are located.
[15] The controller's job description mentioned the triennial review,
in general, while the grants administrator's job description required
the individual to prepare and coordinate with other agency departments
any documentation required for the triennial review.
[16] The technical area requires grantees to implement the Urbanized
Area Formula Grant Program in accordance with their grant application
and agreement and applicable laws and regulations using sound
management practices. The satisfactory continuing control area assesses
grantees' controls over real property, facilities, and equipment, and
ensures that they are used in transit services.
[17] FTA grantees are prohibited from providing exclusive school bus
service. Grantees also must ensure that elderly persons, persons with
disabilities, and individuals presenting a Medicare card will be
charged during non-peak hours for transportation, not more than 50
percent of the peak hour fare.
[18] According to Baldrige Award Examiners' Feedback, in response to
FTA's application for the Malcolm Baldrige National Quality Award, FTA
did not provide examiners with data showing that its efforts improved
grantee performance.
[19] GAO, Government Auditing Standards: July 2007 Revision,
[hyperlink, http://www.gao.gov/products/GAO-07-731G] (Washington, D.C.,
July 2, 2007).
[20] Grant Accountability Project, Guide to Opportunities for Improving
Grant Accountability, October 2005.
[21] GAO, Mass Transit: Grants Management Oversight Improving, but
Better Follow-Up Needed on Grantees' Noncompliance, [hyperlink,
http://www.gao.gov/products/GAO/RCED-98-89], (Washington, D.C.: Apr. 3,
1998).
[22] A 2008 Booz Allen Hamilton report noted, however, that some
regions are not entering data into OTRAK. Further, consultants who
wrote the 2008 report stated that Region 3 had data missing in the
closed finding data element--a key element in evaluating FTA's
triennial review program.
[23] Grant Accountability Project, Guide to Opportunities for Improving
Grant Accountability, October 2005.
[24] GAO, [hyperlink, http://www.gao.gov/products/GAO/RCED-98-89].
[25] Booz Allen Hamilton, Blueprint for Continuous Improvement of FTA
Oversight Review Program, (Washington, D.C., 2008).
[26] 49 U.S.C. § 5307 (h)(2).
[27] For example, the contractors' guide states that if a Drug and
Alcohol Program special review has been conducted within the past two
fiscal years or if one is scheduled for the fiscal year in which the
triennial review is being conducted, then "a review of the Drug and
Alcohol Program area is not necessary and a finding of 'Not Reviewed
(NR) should be made."
[28] Planning certification reviews are another oversight review that
applies to metropolitan planning organizations.
[29] The 24 grantees included large grantees such as those serving New
York City and Chicago, as well as smaller grantees serving the Topeka,
Kansas and Pueblo, Colorado communities.
[30] For the purposes of this analysis, we defined overlap as a
triennial review and one or more special reviews in the same fiscal
year. Neither the FTA oversight review order nor the contractors' guide
defines unnecessary duplication or overlap. Further, FTA headquarters
and regions have their own definitions.
[31] As we previous reported, drug and alcohol reviews include audits
of grantees and employ unique methods for collecting samples. See GAO,
Motor Carrier Safety: Improvements to Drug Testing Programs Could
Better Identify Illegal Drug Users and Keep Them off the Road, GAO-08-
600 (Washington, D.C.: May 15, 2008).
[32] According to FTA officials, they believe notification letters sent
to the regions and grantees give sufficient notice of special oversight
reviews scheduled for grantees.
[33] In addition to these performance measures for the triennial
review, FTA tracks several metrics annually, such as contract
deliverables (including total reviews completed and new grantee
workshops conducted), total findings per review, total findings per
year, trends for most frequent findings, impact of selected findings in
key areas, case studies of specific triennial reviews, findings per
review by region, and findings per review area. However, these metrics
are not compared to a goal or standard and we do not consider them
performance measures. For the purposes of this report, we will refer to
both the measures and goals used for the triennial review as
"performance measures."
[34] GAO, Tax Administration: IRS Needs to Further Refine Its Tax
Filing Season Performance Measures, [hyperlink,
http://www.gao.gov/products/GAO-03-143], (Washington, D.C.: Nov. 22,
2002).
[35] All attributes are not equal. Failure to have a particular
attribute does not necessarily indicate the measure is not useful;
rather, it may indicate an opportunity to strengthen or supplement.
[36] The FTA annual performance plan includes a variation of the Issue
Reports Timely goal for several FTA regional offices as a way of
achieving the Department of Transportation's strategic goal of
organizational excellence. For example, Region 10 uses a 90 percent
goal for issuing reports. Regions 1 and 8 amended the goal to state "30
days after site review or receipt of grantees' comments to the initial
draft."
[37] FTA issued reports within 30 days 94, 82, 94, 91, and 89 percent
of the time from fiscal years 2004 through 2008, respectively.
[38] According to FTA officials, errors in the data element associated
with closed findings did not impact other data elements, such as number
of findings by grantee, region, and review area.
[39] Booz Allen Hamilton, Blueprint for Continuous Improvement of FTA
Oversight Review Program, (Washington, D.C., 2008).
[40] FTA officials noted they do no intend to correct inaccurate data
from the past.
[41] Booz Allen Hamilton--a consultant that recently reviewed FTA's
oversight program--also found that triennial review performance
measures do not contain balance, noting that there were limited
measures to evaluate quality, consistency, and the effectiveness of
oversight activities.
[42] Outputs are the direct products and services delivered by a
program and outcomes are the results of those products and services.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: