This is the accessible text file for GAO report number GAO-10-335
entitled 'Runaway And Homeless Youth Grants: Improvements Needed in
the Grant Award Process' which was released on May 10, 2010.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to Congressional Committees:
United States Government Accountability Office:
GAO:
May 2010:
Runaway And Homeless Youth Grants:
Improvements Needed in the Grant Award Process:
GAO-10-335:
GAO Highlights:
Highlights of GAO-10-335, a report to congressional committees.
Why GAO Did This Study:
The Department of Health and Human Services (HHS) awards grants to
provide shelter and services to runaway and homeless youth through the
Basic Center, Transitional Living and Street Outreach Programs. In
response to a mandate for a review of the grant award process for
these programs in the Reconnecting Homeless Youth Act of 2008 (Pub. L.
No. 110-378), GAO examined (1) grant announcements and application
requirements, (2) technical assistance for grant applicants, (3) how
grant award decisions are made, and (4) notification of grant award
decisions. GAO reviewed requirements, documents, and records
associated with this process for fiscal years 2007 and 2008, observed
the grant evaluation portion of this process, and interviewed
applicants, peer reviewers, and agency officials.
What GAO Found:
Based on GAO’s review of past grant announcements for these programs,
GAO found that the criteria upon which grant applications were
evaluated were not clearly identified or presented in a single
location in the announcement. Rather, GAO found that criteria were
scattered throughout various sections of the announcement, had
multiple labels, and were not presented in an orderly manner. As a
result, applications that did not address the criteria from all
sections were likely to receive lower evaluation scores, decreasing
their chances of receiving a grant.
HHS provides technical assistance to potential applicants for runaway
and homeless youth grants, as required by statute. Of the 20
applicants GAO interviewed who sought technical assistance, 17 were
satisfied with the help they received.
Grant award decisions are primarily based on the results of the peer
review process, and internal controls in place to ensure that
applications are evaluated consistently were not always adequate. GAO
found weaknesses in four out of the six procedures the agency relies
on to ensure consistent evaluation of applications. For example,
although HHS policy requires peer reviewers to be experts in the field
of runaway and homeless youth programs, about one-quarter of the
reviewers who evaluated applications for 2009 Street Outreach grants
had little or no experience in this area.
With regard to notification of grant award decisions, GAO found that
they have not always been communicated to applicants in a timely
manner, which can delay the start of new programs and present planning
challenges for existing ones. GAO also found that the information in
notification letters to applicants who were not awarded grants was not
always clear or complete.
Figure: HHS’s Grant Programs for Runaway and Homeless Youth:
[Refer to PDF for image: illustration]
Basic Center:
Provides food, clothing, shelter, health care, and counseling services
for runaway and homeless youth under age 18.
Transitional Living/maternity Group Homes: Provides runaway and
homeless youth ages 16 through 21 with up to 18 months of residential
services.
Street Outreach:
Designed to prevent the sexual abuse and exploitation of runaway and
homeless youth and to treat any already abused or exploited.
Source: GAO analysis of ACF documents.
[End of figure]
What GAO Recommends:
HHS should take steps to ensure a fair and transparent award process,
including clarifying evaluation criteria; selecting better qualified
reviewers; better documenting agency guidance to reviewers and its
rationale for grant decisions; and providing this rationale to
applicants in a timely manner.
HHS agreed to improve its process in response to all of GAO’s
recommendations except the ones related to clarifying evaluation
criteria and better documenting agency guidance to reviewers.
View [hyperlink, http://www.gao.gov/products/GAO-10-335] or key
components. For more information, contact Kay Brown at (202) 512-7215
or brownke@gao.gov.
[End of section]
Contents:
Letter:
Background:
Grant Announcements Have Not Always Provided Clear and Concise
Information:
ACF Provides Technical Assistance That Applicants Found Helpful:
ACF's Process for Determining Grant Awards May Not Ensure Consistent
Decisions:
Notification of Grant Award Decisions Have Not Always Been Timely, and
Notices Are Not Always Clear:
Conclusions:
Recommendations:
Agency Comments:
Appendix I: Administration for Children and Families' Notification
Letters to Grant Applicants:
Appendix II: Comments from the Department of Health and Human
Services' Administration for Children and Families:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Funding for Runaway and Homeless Youth Program, Fiscal Years
2002 to 2008 (Dollars in thousands):
Table 2: Numbers of Applications for 2007 and 2008 Runaway and
Homeless Youth Grants and Dispositions:
Table 3: Examples of Guidance Provided to Peer Reviewers for
Evaluation of Applications:
Table 4: Examples of Internal Controls and Related Weaknesses:
Figures:
Figure 1: HHS's Grant Programs for Runaway and Homeless Youth:
Figure 2: HHS's Runaway and Homeless Youth Grant Award Process:
Figure 3: Runaway and Homeless Youth Grants Peer Review Process:
Figure 4: Depiction of Scattered Criteria in Different Sections of the
Grant Announcement:
Figure 5: Timelines of Key Dates for the Grant Award Process, Fiscal
Years 2007 and 2008:
Abbreviations:
ACF: Administration for Children and Families:
FY: fiscal year:
FYSB: Family and Youth Services Bureau:
HHS: Health and Human Services:
NIH: National Institutes of Health:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
May 10, 2010:
The Honorable Patrick J. Leahy:
Chairman:
The Honorable Jeff Sessions:
Ranking Member:
Committee on the Judiciary:
United States Senate:
The Honorable George Miller:
Chairman:
The Honorable John P. Kline:
Ranking Member:
Committee on Education and Labor:
House of Representatives:
According to the Department of Health and Human Services (HHS),
thousands of U.S. youth run away from home, are asked to leave their
homes, or become homeless each year.[Footnote 1] Without shelter and
guidance, these youth are vulnerable to exploitation and involvement
in illicit activities, such as selling drugs and prostitution. In 2008
and 2009, HHS awarded more than $100 million each year in
discretionary grants to providers of shelter and services for this
vulnerable population through three programs for runaway and homeless
youth.[Footnote 2] The Basic Center and Transitional Living Programs
fund short-term and longer-term shelter for youth, respectively, and
the Street Outreach Program funds services to prevent sexual abuse and
exploitation of these youth.
Because only about one out of three homeless and runaway youth grant
applications is approved, it is particularly important that the agency
have systems in place to help ensure consistent and transparent grant
award decisions. The Reconnecting Homeless Youth Act of 2008 mandated
a GAO review of HHS's process for awarding certain Runaway and
Homeless Youth Act grants.[Footnote 3] This report responds to that
mandate by addressing the following questions: (1) How clear are grant
announcements and application documents and requirements? (2) How
useful do applicants find the technical assistance they receive to
assist them with applying for grants? (3) How are grant award
decisions made and to what extent does this process comply with
federal requirements? (4) To what extent are grant award decisions
communicated to applicants in a timely and clear manner in accordance
with federal requirements?
To answer these questions, we reviewed relevant laws, regulations,
grant announcements, applications and other agency documents, and we
interviewed key agency officials and contractors. To determine
applicants' views on the usefulness of technical assistance, we
randomly selected and interviewed 24 out of the 590 applicants that
competed for fiscal year 2008 grants. This sample included applicants
who were awarded grants and applicants who were not. We also randomly
selected and interviewed 6 peer reviewers out of approximately 170 who
evaluated applications for at least one of the three grant programs
for fiscal year 2008. Additionally, we interviewed representatives
from the National Network for Youth, an organization that represents
providers of services to youth and families. To determine how grant
award decisions are made, we analyzed agency data and documents
related to peer review evaluations and final award decisions for the
three programs for fiscal years 2007 and 2008. We also observed the
peer review process for the Street Outreach Program in June 2009.
[Footnote 4] To determine the extent to which the grant award process
complies with federal requirements, we compared the grant award
process to relevant requirements in the law, HHS regulations and
written guidance, and internal control standards for the federal
government.[Footnote 5] To determine the extent to which grant award
decisions are communicated in a timely and clear manner in accordance
with federal requirements, we reviewed a random sample of 69
notifications that were sent to applicants for at least one of the
three programs for fiscal year 2008 grants. Additionally, we
established a timeline for each grant program's award process for
fiscal years 2007 and 2008.
We conducted this performance audit from March 2009 through April 2010
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.
Background:
Program Descriptions:
The Runaway and Homeless Youth Act[Footnote 6], as amended, authorizes
federal funding in the form of discretionary grants for three programs
to assist runaway and homeless youth. These programs are administered
by the Family and Youth Services Bureau (FYSB) within HHS's
Administration for Children and Families (ACF).[Footnote 7] The three
programs--the Basic Center Program, Transitional Living Program, and
Street Outreach Program[Footnote 8]--enable local public and private
organizations and shelters in all 50 states and the U.S. territories
to compete for grants that allow them to serve runaway, homeless, and
sexually exploited youth who may be on the streets and in need of
shelter or longer-term support.
The Basic Center Program provides temporary shelter, counseling, and
other services to runaway and homeless youth under the age of 18.
Basic Center grants are awarded competitively to providers and may be
awarded for a period of up to 3 years.[Footnote 9] The Transitional
Living Program provides homeless youth ages 16 through 21 with longer-
term residential services for up to 18 months.[Footnote 10] These
services include such things as counseling and education in basic life
skills, interpersonal skills, educational advancement, job attainment
skills, and physical and mental health care. Transitional Living
grants are awarded competitively to providers and may be awarded for
up to 5 years. The Street Outreach Program provides education,
treatment, counseling, and referrals for runaway, homeless youth under
the age of 18 who have been subjected to or are at risk of being
sexually abused and exploited. Street Outreach grants may be awarded
for up to 3 years. See figure 1 for key aspects of these three
programs.
Figure 1: HHS's Grant Programs for Runaway and Homeless Youth:
[Refer to PDF for image: illustration]
Basic Center:
Provides food, clothing, shelter, health care, and counseling services
for runaway and homeless youth under age 18.
Funding, FY 2007: $48.3 million;
Funding FY 2008: $52.9 million.
Transitional Living/maternity Group Homes: Provides runaway and
homeless youth ages 16 through 21 with up to 18 months of residential
services.
Funding, FY 2007: $39.5 million;
Funding FY 2008: $43.3 million.
Street Outreach:
Designed to prevent the sexual abuse and exploitation of runaway and
homeless youth and to treat any already abused or exploited.
Funding, FY 2007: $15.0 million;
Funding FY 2008: $17.3 million.
Source: GAO analysis of ACF documents.
[End of figure]
For fiscal years 2002 through 2009, funding for these programs has
been just over $100 million in total, with Basic Center funding
representing the largest dollar amount authorized of the three grant
programs. Funding for these programs over the past several years is
shown in table 1.
Table 1: Funding for Runaway and Homeless Youth Program, Fiscal Years
2002 to 2008 (Dollars in thousands):
Program: Basic Center;
2001: $48,338;
2002: $48,288;
2003: $48,298;
2004: $49,171;
2005: $48,786;
2006[A]: $48,265;
2007[A]: $48,298;
2008[B]: $52,860.
Program: Transitional Living;
2001: $20,740;
2002: $39,736;
2003: $40,505;
2004: $40,260[D];
2005: $39,938[C];
2006[A]: $39,511[D];
2007[A]: $39,539[C,D];
2008[B]: $43,268[D].
Program: Street Outreach;
2001: $14,999;
2002: $14,999;
2003: $15,399;
2004: $15,302;
2005: $15,178;
2006[A]: $15,017;
2007[A]: $15,027;
2008[B]: $17,221.
Program: Total;
2001: $84,127;
2002: $103,023;
2003: $104,202;
2004: $104,733;
2005: $103,902;
2006[A]: $102,793;
2007[A]: $102,864;
2008[B]: $113,349.
Source: CRS Report to the Congress, Runaway and Homeless Youth:
Demographics and Programs, RL33785, March 19, 2009 (Washington, D.C.).
Note: Basic Center Program and Transitional Living Program funding are
distributed under the Consolidated Runaway and Homeless Youth Program.
Street Outreach Program funds are distributed separately.
[A] The fourth Continuing Resolution for the FY2007 budget (Pub. L.
No. 110-5) generally funded programs at their FY2006 levels. However,
the FY2006 funding total for the Runaway and Homeless Youth Program
was slightly lower than the FY2007 total because of an additional
transfer of funds from the Runaway and Homeless Youth Program accounts
to an HHS sub-agency.
[B] The FY2008 appropriations include a 1.7 percent across-the-board
rescission on Labor-HHS-Education programs.
[C] The Department of Health and Human Services, in consultation with
Congress, will allocate the $97.2 million for the Basic Center Program
and Transitional Living Program between the two programs.
[D] Includes funding for the Maternity Group Home component.
[End of table]
Grant Award Process:
HHS's grant award process for Runaway and Homeless Youth grants is
comprised of several major steps. HHS's Family and Youth Services
Bureau, which is referred to as the Program Office, and the Grants
Management Office are responsible for carrying out and overseeing this
process. Some of the steps in the grant award process are performed by
contractors on behalf of the agency, and one step in the process is
performed by panels of peer reviewers selected by the agency to
evaluate grant applications. The grant award process consists of the
major steps as illustrated in figure 2.
* Grant Announcement: Each fiscal year, the agency develops and
publishes a grant announcement for each grant program. Announcements
provide the information potential applicants need to determine if they
are eligible to apply and the instructions on how to complete and
submit their application. In addition, they include the criteria used
to evaluate applications.
* Technical Assistance: Each announcement lists the technical service
providers responsible for providing technical assistance to potential
applicants to help them understand the announcement requirements.
Technical assistance can be provided through a webinar, seminar,
information on the Web, or upon request.
* Application Submission: Applications may be submitted electronically
via Grants.gov or by hard copy via mail or other delivery service.
Applicants may also hand deliver their application to the agency's
contractor responsible for receiving the applications. The deadline
for submitting applications is usually 45 to 60 days after an
announcement is published.
* Application Pre-Screening: Applications are prescreened to determine
whether they meet two requirements. Applications are eliminated from
review if they are received after the deadline or if they request more
funding than the maximum amount specified in the announcement.
* Peer Review of Applications: Applications remaining after pre-
screening are submitted to peer review panels. A panel generally
consists of three peer reviewers, who apply the evaluation criteria
contained in the announcement to applications and score the
applications, and a panel chair responsible for facilitating consensus
of the peer review panel.
Peer reviewers assign points to each application, based upon specific
criteria that are outlined in the announcement. The points are added
up and the applicant's average score is derived. This score, ranging
from 0 to 100, becomes the basis for the ranked listing of applicants
which the agency uses in its award decisions. Because applicants whose
score places them below the total available funding limit may be
denied a grant, a single point can make a difference between awarding
a grant and denying a grant.
* Final Grant Award Decisions: Taking into account peer review panel
scores and comments for each application and, in some cases, other
factors, the Program Office and the Grants Management Office make the
final award decisions. These decisions are documented in the final
funding award decision memos, which contain the listing of all
applicants, ranked by their scores, and the final award decisions.
* Notification of Award Decisions: Each applicant is sent a letter
that communicates the grant award decision. Successful applicants are
notified before letters are sent to unsuccessful applicants.
Figure 2: HHS's Runaway and Homeless Youth Grant Award Process:
[Refer to PDF for image: illustration]
Performed by HHS Program Office:
Drafts and approves grant criteria and announcements;
Publishes grant opportunities on the Web.
Applicant applies for grant:
Receives applications and sets up panels to review them (Performed by
contractors);
Panels score applications against published criteria.
Grants Management Office:
Makes final grant decisions and certifies process was competitive;
Notifies unsuccessful grant applicants (Performed by contractors); or;
Prepares award package for grant recipients:
Applicant receives notification letter.
Source: GAO analysis of ACF documents.
[End of figure]
Evaluation of Applications:
According to the HHS Grants Policy Statement, each discretionary grant
application, including those for Runaway and Homeless Youth Grants,
must receive an objective, "…advisory review...by a minimum of three
unbiased reviewers with expertise in the programmatic area for which
applications are submitted." To meet this requirement, the agency
relies on the peer review process, in which three reviewers convene to
evaluate and score applications based on the criteria outlined in the
announcement. The peer reviewers are defined by the agency as experts
in the field of runaway and homeless youth programs. Figure 3 provides
an overview of the peer review process.
Figure 3: Runaway and Homeless Youth Grants Peer Review Process:
[Refer to PDF for image: illustration]
Agency identifies all qualified applications from among submissions;
Agency selects peer reviewers and panel chairs from HHS roster;
Peer review panel: Reviewers score each application individually, then
meet as a group with the chair to agree on a final score.
Agency assigns a set of applications to each panel (each panel reviews
about 10 applications);
Panel scores each application according to the grant criteria and
summarizes the strengths and weaknesses of each grant proposal.
HHS makes the final decision on grant awards using scores from the
panels and the total available funds as a guide.
Funding awarded using cut-off score based on the total funds available
for each type of grant.
Source: GAO analysis of ACF documents.
[End of figure]
HHS awarded grants to about one-quarter of the applicants that applied
in 2007 and 2008, as shown in table 2.
Table 2: Numbers of Applications for 2007 and 2008 Runaway and
Homeless Youth Grants and Dispositions:
Program: Basic Center;
Applications received: FY 2007: 230;
Applications received: FY 2008: 209;
Applications funded: FY 2007: 130;
Applications funded: FY 2008: 120;
Applications not funded: 100;
Applications not funded: FY 2008: 89.
Program: Transitional Living;
Applications received: FY 2007: 281;
Applications received: FY 2008: 338;
Applications funded: FY 2007: 72;
Applications funded: FY 2008: 85;
Applications not funded: 209;
Applications not funded: FY 2008: 253.
Program: Street Outreach;
Applications received: FY 2007: 186;
Applications received: FY 2008: 144;
Applications funded: FY 2007: 86;
Applications funded: FY 2008: 21;
Applications not funded: 100;
Applications not funded: FY 2008: 123.
Program: Total;
Applications received: FY 2007: 697;
Applications received: FY 2008: 691;
Applications funded: FY 2007: 288;
Applications funded: FY 2008: 226;
Applications not funded: 409;
Applications not funded: FY 2008: 465.
Source: GAO analysis of ACF documents.
[End of table]
Grant Announcements Have Not Always Provided Clear and Concise
Information:
Based on the grant announcements we reviewed and our observation of
the peer review process, the criteria upon which grant applications
would be evaluated were not clearly defined in a single location in
the announcement. Rather, we found that criteria were scattered
throughout various sections of the announcement, had multiple labels,
and were not presented in an orderly manner in a single location. For
example, for the 2009 Street Outreach Program grant competition, grant
applications were evaluated and scored based on how well they
addressed criteria contained in three different sections of the
announcement. First, applicants must address the "Program
Requirements," found in Section 1 of the announcement. Second,
applicants must address the "Project Description," found in Section 4.
Third, applicants must address the "Evaluation Criteria" found in
Section 5. However, only the "Evaluation Criteria" section of the
announcement explicitly described how their responses would be
evaluated and scored. Because the applicant must address criteria
contained in different sections of the announcement, if applicants
focused primarily on responding to the "Evaluation Criteria" they may
not have adequately addressed information in the other sections. If
the applicants focused only on the "Evaluation Criteria," these
applicants could have received lower scores, which would have
decreased their likelihood of being awarded grants. Figure 4
represents the various locations where we found descriptions of
criteria.
Figure 4: Depiction of Scattered Criteria in Different Sections of the
Grant Announcement:
[Refer to PDF for image: illustration]
Funding Opportunity Description:
Program requirement: Describes elements grantees must have to operate
their program, such as operations, services, and record keeping.
Program requirements appear in two places within section.
Award information:
Eligibility information:
Application/Submission information:
Project description: A comprehensive description of the proposed
project, its goals, and how the project would be implemented.
Application Review Information:
Evaluation criteria: Explains how the application will be judged,
including the maximum number of points an applicant can receive for
each criterion.
Award Administration Information:
Agency Contacts:
Other Information:
Source: GAO analysis of ACF documents.
[End of figure]
Additionally, peer reviewers we interviewed noted that consolidation
of criteria in the announcement into a single location would aid them
in their evaluation of applications by reducing the time it would take
to review the application because they would not need to look in
multiple places in the application for information.[Footnote 11]
During our observation of the 2009 Street Outreach Program grant
competition, we found that the agency provided detailed guidance to
peer reviewers to help them evaluate and score applications. This
guidance, which was not available to applicants, consolidated
information from various sections of the announcement. The federal
officials instructed reviewers to focus on specific information when
evaluating and scoring applications. Because applicants did not have
this detailed guidance, which combined information from various parts
of the announcement, applicants may not have had full knowledge of
what information was critical to receiving a high score. Table 3 shows
examples of the guidance provided to peer reviewers during the 2009
Street Outreach Program grant competition.
Table 3: Examples of Guidance Provided to Peer Reviewers for
Evaluation of Applications:
Criteria (from announcement): Describes a clear need for the proposed
project through a discussion of the conditions of youth and families
in the area to be served;
Guidelines (from other parts of the announcement): Does the
application describe the conditions of youth and families in the area
to be served that would identify and support the need for the project?
Does the application describe need for assistance by identifying the
physical, economic, social, financial, institutional, and/or other
problems requiring a solution?
Criteria (from announcement): Describes emergency preparedness and
management plan by addressing steps to be taken in care of a local or
national situation that poses risk to the health and safety of program
staff and youth;
Guidelines (from other parts of the announcement): Does the emergency
preparedness plan include prevention, preparedness, response, and
recovery efforts/activities?
Does the plan include strategies for addressing evacuation, security,
food, medical supplies and notification of youth families?
What is the identified alternative location to deliver services in the
case of fire or loss of use of the facility?
What is the plan to notify FYSB of evacuation plans when they are
executed?
Source: GAO analysis of 2009 Street Outreach Program grant competition
documents.
[End of table]
ACF Provides Technical Assistance That Applicants Found Helpful:
ACF provides technical assistance to potential applicants for runaway
and homeless youth grants, as required by statute.[Footnote 12]
Technical assistance is generally defined as providing expertise or
support to applicants and grantees for the purpose of strengthening
their capabilities for providing shelter and support services for
runaway and homeless youth. In fiscal years 2006 and 2007, the agency
provided technical assistance to potential applicants through its
regional network of 10 providers, and listed these providers in its
announcements.
However, beginning in September 2007, ACF centralized its technical
assistance in order to provide more consistent technical assistance
for all applicants, regardless of where they were located. At that
time, the agency entered into cooperative agreements with the
University of Oklahoma to provide technical assistance nationwide.
[Footnote 13]
Through its providers, the agency coordinates technical assistance,
which generally consists of a pre-application conference (webinar)
covering the application requirements such as the project description,
eligibility, and the evaluation criteria, among other things. After
the conference, a recording and transcript is posted on the agency Web
site.
Potential applicants may also ask specific questions of the contacts
listed on the announcement. These contacts include the Program Office
officials and technical assistance providers. If the technical
assistance providers cannot answer the questions, they coordinate with
agency staff to obtain responses that are then posted to the Web site.
The technical assistance providers also arrange seminars on broader
topics related to runaway youth, such as mental health, crisis
intervention, and skills training.
Most of the applicants we interviewed who received technical
assistance under both systems reported that they found it helpful. For
example, 17 of the 20 applicants who sought technical assistance were
satisfied with the help they received. However, three of these
applicants said they prefer the technical assistance provided by their
regional providers because of such things as the regional assistance
being more "hands-on," the regional staff being more responsive and
accessible, and the regional staff being more knowledgeable of local
programs. Agency officials noted that the agency has moved toward
centralized approach to gain a more consistent approach to the
technical assistance it provides.
ACF's Process for Determining Grant Awards May Not Ensure Consistent
Decisions:
ACF's Peer Review Process Has Weak Internal Controls:
ACF's process for determining which grant applicants will be awarded
grants is primarily based on the results of the peer review process,
which has weak internal controls to ensure that applications are
evaluated consistently. According to GAO standards, internal controls
should provide reasonable assurance that the agency's objectives, such
as providing grants to the most qualified providers, are being
achieved.[Footnote 14] Ideally, internal controls should be
continuous, built-in components of the agency's processes, and should
provide reasonable assurance that the grant award process works as it
is designed to work. Our review of ACF's grant award process found
that, while the agency has a number of internal controls in place to
help ensure consistent application of evaluation criteria across
reviewers and across panels, some of these controls are limited in
their effectiveness. For example, we found weaknesses in four out of
six internal controls related to the grant award process, as shown in
table 4.
Table 4: Examples of Internal Controls and Related Weaknesses:
Internal control: 1. Peer reviewers responsible for evaluating grant
applications should be experts in the field of runaway and homeless
youth programs;
Weakness: Some peer reviewers had little or no related expertise in
the field.
Internal control: 2. The agency holds an orientation session for all
reviewers and panel chairs on the first day of panels. Additionally,
it holds a meeting for panel chairs on how to apply the evaluation
criteria. The agency also holds a session for new reviewers;
Weakness: Not all peer reviewers attended these meetings.
Internal control: 3. Detailed guidance is provided to peer reviewers
to aid them in the evaluation of applications;
Weakness: Not all reviewers followed this guidance because it was not
always found in the criteria section of the announcement. This could
lead to inconsistency in scoring applications.
Internal control: 4. Federal officials review each panel's scores and
narrative comments and provide each panel with feedback to help
improve the application of criteria;
Weakness: This feedback is not documented in a permanent record.
Source: GAO analysis of ACF documents and observation of peer review
process.
[End of table]
First, ACF does not always select peer reviewers whose qualifications
comply with the standards outlined in HHS policy. The policy states
that each application for runaway and homeless youth grants must
receive an objective, advisory review by a minimum of three unbiased
reviewers with expertise in the programmatic area for which
applications are submitted. Furthermore, the announcements we reviewed
stated that grant application reviewers should be experts in the field
of runaway and homeless youth programs. However, we found that HHS
considered students, school teachers, business consultants, and
television and media workers as qualified peer reviewers. Our review
of resumes of all the peer reviewers and chairs for 2009 Street
Outreach Program grants found that many had professional and volunteer
experiences that were not always directly related to runaway and
homeless youth programs. Based on the resumes of 76 peer reviewers, we
found that 26 peer reviewers had direct experience with runaway and
homeless youth programs listed on their resume, and another 31 had
indirect experience, such as social work, teaching, or grant
reviewing. However, 19 did not appear to have any of the relevant
knowledge and expertise in runaway and homeless youth programs
required by HHS policy. Three of these 19 reviewers were identified as
"youth reviewers" in their resumes. One agency official responsible
for the grant review process during 2009 explained that HHS interprets
its policies governing peer reviewer qualifications broadly and
accepts all related experience. He also noted that HHS encourages the
use of "youth reviewers" for its peer review panels.
Second, during our observation of the 2009 Street Outreach Program
grant competition, we found that the meetings for peer reviewers and
chairs were not mandatory. These meetings included an orientation
session, panel chair meetings, and new reviewer meetings. Meetings for
panel chairs are particularly important for helping to ensure
consistent evaluations across panels because in these meetings, all
panel chairs agree on how to apply the evaluation criteria. However,
we observed that some panel chairs did not attend these meetings, and,
therefore, their panels may not have applied the evaluation criteria
in the same manner as panels whose chairs had attended the meetings.
Similarly, new reviewers were permitted to miss the new reviewers'
meetings and still participate in the reviews, which could also
increase the risk of inconsistent application of evaluation criteria.
Third, we observed that the agency provided detailed guidance to peer
reviewers to aid them in evaluating applications. The detailed
guidance provided to reviewers explaining the evaluation criteria has
led to variation in application of criteria by review panels. For
example, when we observed peer review panel deliberations for the 2009
Street Outreach grants, we found that peer review panels varied in the
way they applied the criterion for evaluation of emergency evacuation
plans. The announcement's "Evaluation Criteria" section contained the
following evaluation criterion related to emergency plans:
The application "describes the emergency preparedness and management
plan by addressing steps to be taken in case of a local or national
situation that poses risk to the health and safety of program staff
and youth."
At the panel session, agency officials told panels that they should
also apply all of the information in the detailed guidance they were
given, which included information in the "Program Requirements"
section of the announcement. Federal officials advised peer reviewers
that they should score the application on the following information:
"Grantees must immediately provide notification to FYSB when
evacuation plans are executed."
As a result, peer review panels that followed the guidance gave lower
scores to applicants that did not specifically indicate that they
would notify the agency when an evacuation occurred. One peer review
panel we observed, however, did not give lower scores when this was
not specified in an application. These peer reviewers said that they
did not think it was fair to assign lower scores in these cases
because the more detailed information about evacuation requirements
was not listed in the "Evaluation Criteria" section of the
announcement.
Additionally, we interviewed peer reviewers who participated in panels
for 2008 runaway and homeless youth grants. Three of the six peer
reviewers we interviewed told us they observed variations in the way
panels applied the criteria. Reviewers said that the 2008 Transitional
Living Program announcements contained evaluation criteria requiring
applicants to provide background checks for all staff members who
would be working with youth. However, the peer reviewers told us that
the guidance provided to peer reviewers by the agency during that
review process further specified that these background checks must be
conducted in accordance with local, state, and national requirements.
According to the peer reviewers we interviewed, this could have led to
variation in how this aspect of the application was evaluated by
different panels. Given that the peer review score is the key factor
in determining grant awards, inconsistent evaluation criteria across
panels can have a significant impact on whether an applicant is
awarded a grant or not.
The fourth control weakness we observed during our review of the 2009
Street Outreach Program grant competition was that agency officials
did not keep a permanent record of their comments and feedback to peer
review panels during their oversight of the peer review process, which
introduced further potential for inconsistent application of
evaluation criteria. Agency officials review the panel's scores and
narrative comments for each application during the peer review process
before they are finalized. The officials visit panels as peer
reviewers deliberate and respond to their questions, and provide
feedback to chairpersons on their panel's evaluations. Agency
officials told us their review and feedback is meant to ensure that
all panels apply the evaluation criteria in the same way.[Footnote 15]
However, the federal officials we observed did not record this
information in a permanent record. Instead, the officials provided
their feedback to the chair via comments written on post-it notes.
This lack of permanent documentation of federal official feedback to
peer review panels makes it difficult for the agency to ensure that it
is providing consistent guidance to panels and responding to problems
across panels in the same way during the peer review process.
No weaknesses were apparent in two of the six internal controls--(1)
the provision of standard training materials to peer reviewers prior
to panel sessions, and (2) the presence of federal officials on site
during panel sessions to respond to questions from, and communicate on
a daily basis with panels.
The Basis for Denying Grants Is Not Always Documented:
Final funding decision memos used to internally document grant award
decisions for 2007 and 2008 did not contain supporting information
regarding why applications with high scores were not funded. Final
decisions regarding grant awards are determined by HHS's Program
Office and Grants Management Office, taking into account the review
panels' scores and narrative comments for each application. According
to HHS policy and guidance, the agency has the discretion to deny a
grant to an applicant who would otherwise receive one based on the
results of the peer review score alone. The agency is permitted to use
its discretion to deny grants based on other reasons, such as the
agency's concerns about the applicant's program or about the
concentration of service providers in the applicant's location, which
is referred to as concerns about "geographic distribution" of
services. However, the agency does not always clearly document the
rationale for its decision to deny a grant based on "geographic
distribution" of services.[Footnote 16] When grants were denied for
geographic reasons in 2007 and 2008, we found that the final funding
decision memos did not clearly describe the details surrounding such
denials, such as the number of other programs that exist in the same
locale, the services they provide, or the numbers of youth they serve.
Such details could support or justify a denial for geographic reasons.
Without fully documenting and permanently recording its rationale for
exercising its discretion to deny grants to highly scored applicants,
the agency decision-making process is not transparent.
Notification of Grant Award Decisions Have Not Always Been Timely, and
Notices Are Not Always Clear:
Notification Time Frames Can Present Planning Challenges for Some
Applicants:
Grant award decisions are not always communicated in a timely manner,
which may present planning challenges for some applicants. According
to one ACF official, successful applicants are generally notified at
the end of the federal fiscal year. Based on our review of grant
documents for fiscal years 2007 and 2008, we found that for all but
the 2008 Transitional Living Program grants, this was true, regardless
of when the announcement closed or when the funding decisions were
made. For example, applications for the 2008 Basic Center Program were
due in February 2008 and were evaluated and scored in March; however,
applicants were not notified of their award status until September, 6
months later. HHS policy does not indicate when notification letters
should be distributed to applicants, but according to an ACF official,
awards to successful applicants are made by September 30 because most
new programs are expected to start on or before October 1. Given the
proximity of the notification date to program start date, some
successful applicants with new programs we spoke with told us that the
September notification timeframe did not allow enough preparation time
to hire staff and secure the resources needed to provide services. See
figure 5 for the timeline of dates for key events for the fiscal year
2007 and 2008 grant award process.
Figure 5: Timelines of Key Dates for the Grant Award Process, Fiscal
Years 2007 and 2008:
[Refer to PDF for image: 2 timelines]
Basic Center, 2007 grant program:
Publication: March 20;
Deadline: April 23;
Panel: May 20-25;
Decision: August 7;
Notification: September 28.
Basic Center, 2008 grant program:
Publication: December 18, 2007;
Deadline: February 19;
Panel: March 9-14;
Decision: September 23;
Notification: September 30.
Transitional Living/Group Maternity Homes, 2007 grant program:
Publication: March 13;
Deadline: May 8;
Panel: July 15-20;
Decision: August 27;
Notification: September 28.
Transitional Living/Group Maternity Homes, 2008 grant program[A]:
Publication: May 29, 2007;
Deadline: July 30, 2007;
Panel: September 9-14, 2007;
Decision: December 13, 2007;
Notification: January 1, 2008.
Street Outreach, 2007 grant program:
Publication: April 3;
Deadline: June 4;
Panel: July 8-13;
Decision: August 14;
Notification: August 31.
Street Outreach, 2008 grant program:
Publication: June 3;
Deadline: July 18;
Panel: August 10-15;
Decision: September 3;
Notification: September 30.
Source: GAO analysis of ACF documents.
[A] The agency shifted its funding schedule for the Transitional
Living Program in 2008.
[End of figure]
Notification delays also create planning issues for ongoing programs
that are not awarded new grants and, as a result, need to develop
contingency plans for continuing or discontinuing services. Since
unsuccessful applicants are notified of their grant award status after
successful applicants have been notified, an applicant whose previous
grant is about to expire may experience planning problems if
notifications are delayed. Delays in notifying unsuccessful applicants
may not give applicants adequate time to react to not being awarded a
new grant. In the event that funding is denied or discontinued,
earlier notification of award decisions could help providers properly
plan.
According to an ACF official, there is nothing in policy that
prohibits notifying an applicant as soon as award decisions have been
made. The official told us that delays in sending out notification
letters are linked to the timeliness of writing and issuing the
announcement. According to this official, announcements must be
reviewed by many departments within the agency, and, therefore, the
turnaround time is not as timely as it could be. However we found that
even after the announcements were published and closed, applicants
were still not notified of their award status for several months. For
example, for the 2008 Transitional Living grant, regardless of when
the announcement was published, applicants were not notified of their
award, until close to 4 months after the panels had completed
evaluating the applications. Similarly, notifications of decisions
related to 2008 Basic Center grants were not sent out until about 7
months after the panels.
In addition to the challenges applicants experienced due to
notification delays, the agency created additional planning challenges
for applicants when it unexpectedly changed the timing of the funding
cycle for the Transitional Living Program in fiscal year 2008 without
notifying applicants of this change in a timely manner. The
announcement stated that ACF anticipated making grant awards in the
first quarter of fiscal year 2008, which would have been from October
through December of 2007. However, the grant award start date was
changed to March 2008 after this announcement was published. According
to an agency official, the original start date was moved in an effort
to spread out the timing of peer review panels for each of the three
runaway and homeless youth programs and other activities that were
scheduled to occur around the same time during the summer months. As a
result of moving the cycle start date--from October to March--some
successful applicants were without federal funding for several months
between the end of the previous grant cycle and the new grant award
start date.
Runaway and homeless youth service providers have also raised concerns
to their congressional representatives about the timeliness of
notifications. Specifically, we reviewed nine complaint letters that
were sent to congressional representatives regarding runaway and
homeless youth grants applications in 2007 and 2008. One letter,
representing six providers, stated that notification delays created
planning problems for service providers who were not able to develop
contingency planning for either the continuation or discontinuation of
their programs. ACF responded to the complaint by noting that it
offers funding for successful applicants to recoup some of the costs
that programs incurred due to the delay.
In addition, the National Network for Youth, an organization that
represents providers of services to youth and families also noted that
the timeliness of notifications has been an issue of concern for its
membership. In particular, some service providers have raised issues
about the difficulties receiving timely communications from ACF
concerning grant awards.
Notification Letters to Applicants Are Not Always Clear:
All of the successful applicants we spoke with felt that their
notification letters were clear and contained sufficient information;
however, unsuccessful applicants were not all satisfied with the
clarity and completeness of the information presented in their
letters. The standard letter to unsuccessful applicants may list
several possible "other factors" for the denial, beyond their peer
review panel score, without any indication of which of the reasons
listed in the standard notice applied to their application. See
appendix I for a standard letter. The "other factors" include:
* "comments of reviewers and government officials,"
* "staff evaluation and input,"
* "geographic distribution," and:
* "audit reports and previous program performance."
Some unsuccessful applicants told us the letter did not contain enough
information for them to understand why their application was denied.
In particular, some applicants told us that they did not understand
what the agency meant by geographic distribution, which was the basis
for denying grants to at least eight applicants during fiscal years
2007 and 2008. Officials told us that "geographic distribution" means
that an applicant was denied because the geographical area their
program would serve is already served by another runaway and homeless
youth service provider. The agency does not keep a record to document
detailed information that would support or justify a denial for
geographic reasons, such as the number or names of programs that exist
in the same locale, the services these programs provide, or the
numbers of youth they serve. As a result, it is not possible to verify
that denying a grant based on "geographic distribution" was justified.
Applicants who want further explanation of their award decisions may
request additional information along with their scores from ACF
through a Freedom of Information Act request.[Footnote 17] An ACF
official told us that it would be difficult to provide all
unsuccessful applicants more information supporting the denial
decision based on other factors such as "geographic distribution" in
notification letters because of limited resources. However, the
resources needed to provide such information may be small, given that
"geographic distribution" was the basis for denying grants to only a
small number of applicants during fiscal years 2007 and 2008.
Moreover, based upon our review of decision notices sent to applicants
who were screened out of the competition due to late submissions or
improper funding requests, we found contradictory language that may
confuse applicants. Specifically, the letter states that "the limited
availability of funds permitted us to select only the highest scoring
applications that also met all of the eligibility requirements,"
leaving the impression that the application, in these cases, had been
evaluated and scored by a peer review panel. However, applications
that are screened-out of the process before the peer review session
are not evaluated or scored. When we pointed out this statement to the
agency, officials agreed the language could be confusing to applicants.
Conclusions:
The runaway and homeless youth grant programs provide much needed
services to a vulnerable population and the number of applications far
exceed the number of grants that can be awarded with available
funding. To ensure that ACF awards these grants to the most capable
applicants, its award process must be fair and transparent.
Without clearly organized evaluation criteria in grant announcements,
applicants can have difficulty determining what their applications
will be evaluated on. Furthermore, without consistent evaluation of
applications in the process, there cannot be a level playing field for
all applicants. All peer reviewers must have the required programmatic
expertise, or not all applicants are evaluated by their peers.
Additionally, unless all peer reviewers attend meetings at panel
review sessions; these meetings cannot help ensure consistent
evaluation of applications. Without documentation of ACF comments to
peer review panels during the review process there is also a risk that
the evaluation process will not be consistent. Moreover, without fully
documenting the rationale for denying grants to highly scored
applicants, agency grant award decisions are not transparent.
Once the grant award decisions are made, it is incumbent on ACF to
notify applicants of decisions in a timely manner and provide them
with clear and specific information about, in particular, decision not
to grant awards. Without such notification, applicants may experience
planning challenges and not fully understand the reasons they were
denied grants.
Recommendations:
To enhance transparency and fairness in the grant award process, and
improve grantees ability to plan for services, we recommend that the
Secretary of Health and Human Services direct the Assistant Secretary
for the Administration for Children and Families to take the following
seven actions:
* Clearly identify in grant announcements all the criteria that peer
reviewers will use to evaluate and score applications, and ensure that
peer reviewers use only those criteria during the peer review process.
* Select peer reviewers with expertise in the programmatic area for
which they are evaluating grant applications.
* Make all meetings for peer reviewers, including those for new
reviewers and chairs, mandatory.
* Document and maintain records of ACF comments to peer review panels
during the review process.
* Document the specific reasons for denying grants to high-scoring
applicants in favor of other applicants for the agency record.
* Provide clear information to applicants about the specific reasons
their applications were not approved.
* Notify applicants about the outcome of their applications as soon as
grant award decisions are made.
Agency Comments:
We provided a draft of this report to the Department of Health and
Human Services for review and comment; these appear in appendix II. In
its comments, HHS disagreed with our recommendation to review and
revise announcements to ensure that all evaluation criteria listed be
clearly labeled as evaluation criteria and be contained in a single
section of the announcement. HHS maintains that all of the criteria
used to evaluate and score applications are contained in section 5 of
the announcement. However during the peer review process we observed,
in addition to evaluating and scoring applications based on criteria
specified in the "Evaluation Criteria" section of the announcement
(section 5), some of the panels evaluated and scored applications
based on criteria from two other sections of the announcement. Given
the difference between the agency's response to our recommendation and
what we observed, we are revising our recommendation to highlight the
need to ensure that the all criteria used to evaluate and score
applications are clearly identified to applicants and peer reviewers,
and that peer reviewers use only those criteria when evaluating and
scoring applications.
With regard to our recommendation to select peer reviewers with
expertise in the program for which they are evaluating grant
applications, HHS commented that the agency has elected to accept
reviewers who are knowledgeable of the risk factors faced by runaway
and homeless youth, and that many professional disciplines often
intersect with runaway and homeless youth. However, we found that in
the past the agency has used individuals that would not be expected to
have relevant expertise, such as television and media workers. Noting
our concern in this area, the agency indicated that they plan to take
steps to ensure that all reviewers possess the knowledge and expertise
in the particular program for which they are reviewing grant
applications. In the event of a shortage of reviewers, the agency
intends to staff panels with at least one peer reviewer with extensive
relevant knowledge, which would continue to differ from the current
policy that grants must receive an objective, advisory review by a
minimum of "three" unbiased reviewers with expertise in the
programmatic area for which applications are submitted. We agree that
professionals in varied disciplines could have sufficient expertise to
serve as reviewers and recognize that it may be difficult for the
agency to find enough reviewers with expertise in a particular
program. As a result, we are clarifying our recommendation to include
those that have expertise in the programmatic area for which they are
evaluating grant applications, and not a specific program.
Regarding our recommendations to make peer review meetings mandatory,
HHS indicated that all meetings for peer reviewers and chairs are
already mandatory but due to unforeseen factors, it is not always
possible for all reviewers to attend. Indeed, during our observation
of a peer review session, not all reviewers and chairs attended the
meetings. Moreover, at the time, agency officials told us that
attendance at these meetings was not explicitly mandatory, but highly
encouraged. They also indicated that attendance was not enforced and
attendance records were not maintained. In response to this
recommendation, the agency indicated that they plan to officially
notify all reviewers and chairpersons participating in future reviews
that all training is mandatory. In the event some reviewers and
chairpersons are not able to attend the mandatory training sessions
due to unforeseen circumstances, the agency intends to offer "make up"
sessions.
HHS did not provide comments on our recommendation to maintain records
of ACF comments to peer reviewer panels during the review process.
However they agreed with our recommendation to document the specific
reasons for denying grants to high-scoring applicants in favor of
other applicants. HHS commented that the agency plans to include more
details concerning geographic distribution in the letters to
applicants who are denied grants for this reason. While these efforts
would be in line with our recommendation; the details supporting such
decisions must be consistently documented in the agency's records to
support the information provided to applicants in their letters.
In response to our recommendation to provide clear information to
applicants about specific reasons their applications were not
approved, HHS stated that in accordance with ACF policy and
procedures, every unsuccessful applicant is entitled to an explanation
of why their application was not funded. In addition, the agency noted
that, upon request, the Program Office will provide a debriefing to
applicants. However, letters sent to unsuccessful applicants should
clearly note that applicants may request a debriefing by the Program
Office regarding specific reasons why their application was not
funded. Currently, letters to unsuccessful applicants do not include
this information. In addition, it is important to revise the language
in letters to applicants that are screened out of the grant
competition that implies their application was evaluated and scored.
Finally, HHS agrees with our recommendation to notify applicants about
the outcome of their application as soon as grant award decisions are
made. As part of the grant application process, the agency plans to
explain to applicants that final grant decisions depend on the results
of the grant award negotiations between ACF and the prospective
grantees. We recognize that these grants are discretionary and that
final award decisions involve negotiations that may take time.
However, every effort should be made to complete negotiations and
notify both successful and unsuccessful applicants as quickly as
possible. To enable applicants to efficiently and effectively manage
their programs, it is important for applicants to receive their
notices in a timely manner.
HHS also provided technical comments, which we incorporated into the
report as appropriate.
We are sending copies of this report to the Secretary of HHS, relevant
congressional committees, and other interested parties. In addition,
the report will be made available at no charge on GAO's Web site at
[hyperlink, http://www.gao.gov].
If you or your staffs have any questions about this report, please
contact Kay E. Brown at (202) 512-7215 or brownke@gao.gov. GAO staff
who made major contributions to this report are listed in appendix III.
Signed by:
Kay E. Brown:
Director, Education, Workforce, and Income Security:
[End of section]
Appendix I: Administration for Children and Families' Notification
Letters to Grant Applicants:
<>
<>
<>
<>
<>
<>
RE: Grant Application: <>.
Dear <>:
Thank you for submitting an application in response to the February
23, 2007, Grants.gov publication of the Basic Center Discretionary
Grants program announcement (HHS-2004-ACF-ACYF-CY-0063). We appreciate
the considerable effort that went into the preparation of your
application.
I regret to inform you that your application was deemed non-responsive
and was not considered under this funding announcement. The limited
availability of funds permitted us to select only the highest scoring
applications that also met all of the eligibility requirements.
Please note Part III (Eligibility Information) of the program
announcement, Section 3, Other (Disqualification Factors) which
determined your application's non-responsiveness.
__________Applications that exceed the ceiling amount will be deemed
non-responsive and will not be considered for funding under this
announcement; or;
__________Any application that fails to satisfy the deadline
requirements referenced in Section IV.3 will be deemed non-responsive
and will not be considered for funding under this announcement.
I also want to take this opportunity to tell you about a new Federal
website entitled Grants.gov. Grants.gov allows organizations to
electronically find and apply for all Federal competitive grants.
Grants.gov is THE single access point for over 900 grant programs
offered by the 26 Federal grant-making agencies. At http://Grants.gov
you can register to receive electronic notification of all future
federal funding opportunities, including the FY 2007 Basic Center
Program Announcement.
Sincerely,
Daphne Weeden, Director:
Division of Discretionary Grants:
Family and Youth Services Bureau:
cc: Contact:
[End of letter]
September 28, 2008:
<>
<>
<>
<>
<>
In reference to grant application <>.
Dear <>:
Thank you for submitting an application in response to the June 3,
2008, Grants.gov publication of the Street Outreach Program
Discretionary Grants program announcement (HES-2008-ACF-ACYF-Y0-0128).
A panel of non-federal experts who have broad experience in the
subject areas covered by the announcement carefully reviewed all of
the applications received in response to the Street Outreach program
announcement. The reviewers evaluated the applications against the
published criteria, and their assessments were the principal basis for
the final funding decision on each application.
Funding decisions have been made, and I regret to inform you that your
application was not selected for funding. The amount of funds
available permitted us to select only a limited number of applications
for funding. I hope the enclosed summary of your application's
strengths and weaknesses will be of assistance to you in preparing
future applications.
Please note that some high ranking applications were not awarded in
accordance with Part V.2 (Review and Selection Process) of the program
announcement, which states, "Highly ranked applications are not
guaranteed funding because of other factors are taken into
consideration. These include, but are not limited to geographic
distribution, previous program performance of applicants, compliance
with grant terms under previous HHS grants, audit reports, an
applicant's progress in resolving any final audit disallowance on
previous FYSB or other Federal agency grants.
We appreciate the effort that went into the preparation of your
application and look forward to your participation in future
competitions.
Sincerely,
Signed by:
Curtis 0. Porter:
Acting Associate Commissioner:
Family and Youth Services Bureau:
Enclosure:
[End of section]
Appendix II: Comments from the Department of Health and Human
Services' Administration for Children and Families:
Department Of Health & Human Services:
Office Of The Secretary:
Assistant Secretary for Legislation:
Washington, DC 20201:
April 30 2010:
Kay E. Brown, Director:
Education, Workforce, and Income Security Issues:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20548:
Dear Ms. Brown:
Enclosed are the Departments comments on the U.S. Government
Accountability Office's (GAO) draft report entitled: "Runaway and
Homeless Youth Grants: Improvements Needed in the Grant Award
Process"(GAO-10-335).
The Department appreciates the opportunity to comment on this report
before its publication.
Sincerely,
Signed by:
Andrea Palm:
Acting Assistant Secretary for Legislation:
Enclosure:
[End of letter]
General Comments Of The Department Of Health And Human Services (HHS)
On The Government Accountability Office's (GAO) Draft Report Entitled,
"Runaway And Homeless Youth Grants: Improvements Needed In The Grant
Award Process" (GA0-10-335):
The Department appreciates the opportunity to comment on this
Government Accountability Office (GAO) draft report.
GAO Recommendation:
To enhance transparency and fairness in the grant award process, and
improve grantees ability to plan for services, we recommend that the
Secretary of Health and Human Services direct the Assistant Secretary
for the Administration for Children and Families to take the following
seven actions:
* Review and revise announcements to ensure that all evaluation
criteria listed in the announcements are clearly labeled as evaluation
criteria and contained in a single section of the announcement.
* Select peer reviewers with expertise in the program for which they
are evaluating grant applications.
* Make all meetings for peer reviewers, including those for new
reviewers and chairs, mandatory.
* Document and maintain records of ACF comments to peer review panels
during the review process.
* Document the specific reasons for denying grants to high scoring
applicants in favor of other applicants for the agency record.
* Provide clear information to applicants about the specific reasons
their applications were not approved.
* Notify applicants about the outcome of their application as soon as
grant award decisions are made.
ACF Response to GAO's Recommendation:
GAO found that the criteria upon which grant applications were
evaluated were not clearly identified or presented in a single
location in the announcement. ACF uses a standard template for its
Funding Opportunity Announcements (F0As). Section V of the template
entitled "Applicant Review Information" is where the criteria for
evaluation are listed. This section is the only section in the FOA
that contains the points for each of the criteria to be evaluated.
Section V of the Street Outreach Program announcement contained the
review and scoring information for applications submitted. Each
element of Section V contained the criterion of the element along with
the points assigned to each element. This was the only section of the
announcement where the application was scored based upon the
applicant's response to each criterion.
With respect to the recommendation regarding peer reviewers, given the
relatively small universe of runaway and homeless youth programs
nationally and that FYSB has historically excluded reviewers from
organizations applying for funding in the same competition, FYSB has
elected to accept reviewers who are knowledgeable of the risk factors
faced by runaway and homeless youth. Often these individuals are
persons who may work in a Basic Center or Transitional Living Program
and are responsible for reviewing Street Outreach applications. It is
important to realize that while a person may not work directly in a
street outreach program, if they work in a Transitional Living or
Basic Center Program, they do have outreach experience as both
programs have outreach components.
FYSB has also used retired Runaway and Homeless Youth grantee staff
that may at the time of the reviews serve as consultants. Furthermore,
the Bureau has been committed to using college students as reviewers,
some of whom may have been runaway and/or homeless youth themselves.
These youth offer valuable insight into the youth service needs of
this vulnerable population.
FYSB understands that many professional disciplines often intersect
with runaway and homeless youth and have not barred them from
participating in the review process. These disciplines include
educators, social workers, therapists, juvenile probation officers and
retired law enforcement officers.
Noting GAO's concern in this area, FYSB will make a concerted effort
to ensure that all reviewers possess extensive knowledge in the field
of the specific runaway and homeless youth program for which they are
reviewing. However, where this is not possible due to the lack of
available reviewers, FYSB will work to ensure that each panel has at
least one person on it with extensive knowledge and experience in the
particular program for which they are reviewing.
All orientation and meeting sessions are mandatory. However, due to
challenges in travel schedules and other unforeseen factors, it has
not always been possible for all reviewers and chairpersons to attend
the sessions when they are given.
FYSB has attempted to mitigate potential problems associated with
reviewers and chairpersons inability to attend the training sessions
in a timely manner by hosting webinar trainings prior to the on-site
reviews. FYSB staff also attempt to meet with any individuals who may
have missed a meeting in order to bring them up to speed on the
information prior to the review panels convening.
Noting GAO's concern, FYSB will officially notify all reviewers and
chairpersons participating in future reviews that all training is
mandatory. Furthermore, in the event that some reviewers and
chairpersons are not able to attend the mandatory training sessions
due to unforeseen circumstances, FYSB will offer make-up sessions
prior to them joining their review panels.
FYSB will work to include greater details concerning geographic
distribution in the letter to unsuccessful applicants. Furthermore,
the letter will direct applicants to the FYSB website to in order that
they may view the list of grantees funded nationally as well as
identify existing grantees in their State and local community.
In accordance with ACF policy and procedures, every unsuccessful
applicant is entitled to an explanation of the reasons why the
application was not funded. No Fund letters contain a summary of the
strengths and weaknesses of the application. Upon request, applicants
may request a debriefing by the Program Office.
ACF's current practice and policy is to notify applicants as soon as
grant award decisions have been made. ACF agrees that grantees should
be notified about the outcome of their applications as soon as grant
award decisions are made. Award decisions sometimes change as a result
of the negotiations. There is a possibility that an applicant could
deny funding or that ACF would reduce the amount of funding for an
award, which would then make funds available to add additional awards.
As part of the grant application process, ACF will explain that final
grant decisions depend on the results of the grant award negotiations
between ACF and the prospective grantee, so the results will not be
known until the negotiation process is completed.
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Kay E. Brown (202) 512-7215 or brownke@gao.gov:
Staff Acknowledgments:
In addition to the contact named above, Clarita Mrena (Assistant
Director) and Jacqueline Harpp (Analyst-in-Charge) managed all aspects
of the assignment up to report production. Anna Kelley managed the
report production and Vernette Shaw made significant contributions in
all aspects of the work. Lisa Fisher and Jennifer McDonald also made
significant contributions to this report. Additionally, Walter Vance
and Minette Richardson provided technical support in design and
methodology. James Rebbe provided legal support and Susannah Compton
assisted in message and report development. James Bennett assisted
with visual communications.
[End of section]
Footnotes:
[1] The exact number of children and youth that run away or become
homeless is unknown due to the transient nature of this population and
the lack of a consistent definition of a "runaway or homeless"
individual. See CRS Report for Congress, Runaway and Homeless Youth:
Demographics and Programs, RL33785 (Washington, D.C.: Mar. 19, 2009).
[2] According to the HHS Grants Policy Statement, discretionary grants
are those that permit the federal government, according to specific
authorizing legislation, to exercise judgment in selecting the
recipients. Discretionary grants are generally made following a
competitive process.
[3] Pub. L. No. 110-378, § 9.
[4] Approximately 180 applications were reviewed by 19 peer review
panels during that competition.
[5] Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[6] The Runaway and Homeless Youth Act was enacted as Title III of the
Juvenile Justice and Delinquency Prevention Act of 1974 (Pub. L. No.
93-415).
[7] The Administration for Children and Families (ACF) is 1 of 11
federal agencies within HHS.
[8] The Street Outreach Program is formally known as the Education and
Prevention Services to Reduce Sexual Abuse of Runaway, Homeless, and
Street Youth Program.
[9] Basic Center funds are allotted to states on the basis of their
relative population of youth less than 18 years of age. The term
"state" includes the District of Columbia and Puerto Rico. U. S.
territories also receive funding based on their population of youth.
[10] Transitional Living Program also includes grants for maternity
group homes targeted to young mothers and their children to meet the
needs of this population.
[11] Another HHS agency, The National Institutes of Health (NIH)
conducted a peer review self-study in 2007 to improve the quality and
transparency of its grants review process for research grants and
cooperative agreements. As a result of the study, NIH shortened the
length of its applications and aligned it with specific review
criteria to clearly identify for applicants the most important
requirements to address and reduce the burden of review for reviewers.
[12] The statutory requirement calls for HHS to provide "informational
assistance to potential grantees." 42 U.S.C. § 5714a HHS terms the
assistance it provides as "technical assistance."
[13] The University of Oklahoma established and operates the Runaway
and Homeless Youth Training and Technical Assistance Center (RHYTTAC)
to provide these services.
[14] Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[15] They also noted that this is a method for the agency to determine
how individual reviewers are performing and if they should be selected
for future reviews.
[16] The agency uses this discretion to ensure that services are
geographically distributed throughout the nation and to increase the
capacity of services to communities with a high concentration of
runaway and homeless youth.
[17] The Freedom of Information Act (5 U.S.C. § 552) generally
provides that any person has a right, enforceable in court, to obtain
access to federal agency records, except to the extent they are
protected from public disclosure.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: