This is the accessible text file for GAO report number GAO-10-125 
entitled 'Juvenile Justice: DOJ Is Enhancing Information on Effective 
Programs, but Could Better Assess the Utility of This Information' 
which was released on December 17, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Subcommittee on Crime, Terrorism, and Homeland 
Security, Committee on the Judiciary, House of Representatives: 

United States Government Accountability Office: 
GAO: 

December 2009: 

Juvenile Justice: 

DOJ Is Enhancing Information on Effective Programs, but Could Better 
Assess the Utility of This Information: 

GAO-10-125: 

GAO Highlights: 

Highlights of GAO-10-125, a report to the Chairman, Subcommittee on 
Crime, Terrorism, and Homeland Security, Committee on the Judiciary, 
House of Representatives. 

Why GAO Did This Study: 

State juvenile justice systems face critical problems when it comes to 
juvenile delinquency issues such as reentry—when offenders return home 
from incarceration—and substance abuse. GAO was asked to review 
juvenile reentry and substance abuse program research and efforts by 
the Department of Justice’s (DOJ) Office of Juvenile Justice and 
Delinquency Prevention (OJJDP) to provide information on effective 
programs (i.e., whether a program achieves its intended goal) and cost-
beneficial programs (i.e., whether the benefits of programs exceeded 
their costs). This report addresses (1) expert opinion and available 
research on these types of reentry and substance abuse programs, (2) 
the extent to which OJJDP assesses its efforts to disseminate 
information on effective programs, and (3) OJJDP’s plans to accomplish 
its research and evaluation goals. GAO, among other things, reviewed 
academic literature, and OJJDP’s dissemination efforts and research 
goals. GAO also interviewed OJJDP officials and a nonprobability sample 
of 26 juvenile justice experts selected based on their experience with 
juvenile reentry and substance abuse issues. 

What GAO Found: 

The majority of the juvenile justice reentry and substance abuse 
experts GAO interviewed cited evidence that shows cognitive behavioral 
therapy—programs that help individuals change their beliefs in order to 
change their behavior—and family therapy—programs that treat juveniles 
by focusing on improving communication with family members—are 
effective and cost beneficial when addressing reentry and substance 
abuse issues. For example, two juvenile reentry experts cited studies 
showing that 1 year after participating in a cognitive behavioral 
therapy program, participants were less likely to commit another 
offense than nonparticipants. Additionally, experts cited a study that 
reported that a family therapy program provides about $80,000 in 
savings per participant when accounting for savings from a decline in 
crime, such as the cost the police would have incurred. Most experts 
indicated that there was limited evidence on the effectiveness and cost 
benefits of reentry programs, such as aftercare—programs that assist 
juvenile offenders in returning to their communities during the reentry 
process—and substance abuse programs, such as drug courts—specialized 
courts that provide programs for substance-abusing juveniles and their 
families. 

GAO reviewed two OJJDP efforts that provide information on effective 
programs across the range of juvenile justice issues, the National 
Training and Technical Assistance Center (NTTAC) and the Model Programs 
Guide. OJJDP has mechanisms in place to regularly assess the utility of 
the information provided by NTTAC, but does not have such a mechanism 
for the guide. OJJDP ensures the utility of NTTAC’s information through 
evaluations in accordance with federal guidelines that highlight the 
importance of regularly soliciting feedback from users. However, OJJDP 
could better ensure the utility of the information disseminated by the 
Model Programs Guide by having a mechanism in place to solicit regular 
feedback from members of the juvenile justice field—for example, 
program practitioners—that is specifically related to the guide. 

OJJDP has articulated research and evaluation goals to support its 
mission of improving the juvenile justice system and is developing 
plans to assist in meeting these goals. OJJDP is required under the 
Juvenile Justice and Delinquency Prevention Act, as amended, to publish 
an annual program plan that describes planned activities under accounts 
authorized for research and evaluation activities, among other things. 
Additionally, the Office of Management and Budget (OMB) recommended 
that OJJDP develop a comprehensive evaluation plan for juvenile justice 
programs. While OJJDP has not published an annual program plan since 
2002, in December of 2009, it issued a proposed plan for public comment 
and aims to publish the final program plan once public comments are 
incorporated. Additionally, although the office has considered 
developing a comprehensive evaluation plan to address OMB 
recommendations, it had not previously done so because of a lack of 
resources. However, OJJDP is committed to developing a comprehensive 
evaluation plan once the program plan is finalized. 

What GAO Recommends: 

GAO recommends that OJJDP develop a cost-effective mechanism to 
regularly solicit and incorporate feedback from the juvenile justice 
field on the usefulness of information in the Model Programs Guide. DOJ 
concurred with GAO’s recommendation. 

View [hyperlink, http://www.gao.gov/products/GAO-10-125] or key 
components. For more information, contact Eileen Larence at (202) 512-
8777 or larencee@gao.gov. 

[End of section] 

Contents: 

Letter: 

Scope and Methodology: 

Background: 

Experts Cite Evidence from Available Research Indicating That Cognitive 
Behavioral Therapy and Family Therapy Programs Are Effective and Cost 
Beneficial When Addressing Reentry and Substance Abuse Issues: 

OJJDP Has Mechanisms in Place to Ensure Training and Technical 
Assistance Meet Users' Needs, but Regular Feedback on the Model 
Programs Guide Would Help OJJDP Better Assess Information Utility: 

Finalizing a Program Plan and Developing a Comprehensive Evaluation 
Plan Would Help OJJDP Achieve Its Research and Evaluation Goals and Use 
Its Limited Resources Effectively: 

Conclusions: 

Recommendation for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Juvenile Justice Reentry and Substance Abuse Experts We 
Interviewed: 

Appendix II: Juvenile Justice Experts' Views on Factors That Can Help 
Programs Achieve Intended Outcomes: 

Appendix III: OJJDP's Enacted Appropriations for Fiscal Years 2007 
through 2009: 

Appendix IV: Juvenile Reentry and Substance Abuse Programs OJJDP Funded 
through Discretionary Grants: 

Appendix V: Experts' Opinions of Reentry Programs That Lack Conclusive 
Evidence of Effectiveness: 

Appendix VI: Experts' Opinions of Substance Abuse Programs That Lack 
Conclusive Evidence of Effectiveness: 

Appendix VII: Additional OJJDP Efforts to Disseminate Information about 
Effective Juvenile Justice Programs: 

Appendix VIII: Comments from the Department of Justice: 

Appendix IX: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Types of Juvenile Reentry Programs: 

Table 2: Types of Juvenile Substance Abuse Programs: 

Table 3: Examples of Net Benefits of Program Interventions within 
Reentry and Substance Abuse Program Types We Reviewed from Four Studies 
Cited by Experts: 

Table 4: OJJDP Funds Authorized and Used for Research and Evaluation, 
Fiscal Year 2008: 

Table 5: Juvenile Justice Experts We Interviewed about Juvenile Reentry 
or Substance Abuse Programs: 

Table 6: Juvenile Justice Enacted Appropriations, Fiscal Years 2007 
through 2009: 

Table 7: OJJDP Juvenile Reentry and Substance Abuse Programs Funded 
through Discretionary Grants, Fiscal Years 2007 through 2009: 

Table 8: Additional OJJDP Efforts to Disseminate Information on 
Effective Programs: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

December 17, 2009: 

The Honorable Robert C. Scott: 
Chairman: 
Subcommittee on Crime, Terrorism, and Homeland Security: 
Committee on the Judiciary: 
House of Representatives: 

Dear Mr. Chairman: 

Although the juvenile arrest rate is near its lowest point in two 
decades, critical problems affecting states' juvenile justice systems 
remain, such as recidivism--the act of committing new offenses after 
having been arrested or convicted of a crime--and substance abuse--a 
pattern of use of illegal, prescription, or nonprescription drugs 
leading to significant impairment in functioning. Reentry, the return 
of juvenile offenders[Footnote 1] from residential facilities back into 
their communities, aims to reduce recidivism by using programs that 
promote the effective reintegration of juvenile offenders and assist 
them in acquiring the life skills needed to succeed and become law- 
abiding citizens.[Footnote 2] According to the Department of Justice 
(DOJ), approximately 100,000 juvenile offenders are held in residential 
facilities in the United States on a given day, and most of these 
juveniles will likely leave these facilities and return home to their 
communities each year.[Footnote 3] Further, DOJ has estimated that 62 
percent of juveniles in residential facilities self-reported having had 
at least one prior commitment in the juvenile justice system.[Footnote 
4] Additionally, substance abuse plays a significant role in juvenile 
crime. 

In addition, in a 2008 annual survey of members conducted by the 
Federal Advisory Council on Juvenile Justice,[Footnote 5] reentry of 
offenders into communities and schools (28 states) and substance abuse 
(21 states) were two of the top three programmatic issues most 
frequently reported as affecting these states' juvenile justice 
systems.[Footnote 6] Specifically, concerns were noted about the 
minimal number of programs available to help juveniles reenter 
communities, the large number of inexperienced practitioners operating 
programs, and the limited substance abuse treatment options. Related to 
research and policy, the most common issue state officials cited was 
the need to identify effective practices for juvenile programs (23 
states) followed by the need to enhance the states' capacities to 
conduct juvenile research and collect data about juvenile programs (20 
states). Respondents also cited the need for federal agencies and state 
governments to collaborate on conducting research and evaluations to 
provide a better explanation of (1) practices that have been evaluated 
and have been shown to be effective, (2) the need for more training 
from federal agencies on how to conduct evaluations, and (3) increased 
funding to establish evaluation procedures (23 states). 

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is 
DOJ's office charged with providing national leadership, coordination, 
and resources to help prevent and respond to juvenile delinquency and 
victimization. OJJDP supports states and communities in their efforts 
to develop and implement effective programs to prevent delinquency and 
intervene after a juvenile has offended, among other things. For 
example, from fiscal years 2007 through 2009, Congress provided OJJDP 
almost $1.1 billion to use for grants to states, localities, and 
organizations for a variety of juvenile justice programs. In support of 
its mission, the office also disseminates information to states and 
communities related to a number of juvenile justice issues, including 
effective reentry and substance abuse programs, and funds research and 
evaluations of juvenile justice programs. 

States and localities develop and implement programs that specifically 
target facilitating reentry and reducing substance abuse; therefore, it 
is important that federal, state, and local agencies that provide 
grants for programs, as well as practitioners operating programs, have 
information about which ones are effective and cost beneficial. Having 
such information will better position these agencies to help ensure 
that federal, state, and local funds are well spent. In general, 
effectiveness is determined through program evaluations, which are 
systematic studies conducted to assess how well a program is working-- 
that is, whether a program produced its intended results or effects. 
Additionally, cost-benefit analyses can help determine if the dollar 
value of a program's success--such as a reduction in recidivism-- 
exceeds the cost of the program. To help ensure the effective use of 
grant funds for juvenile reentry and juvenile substance abuse programs, 
you asked us to review the available research as well as OJJDP's 
efforts to provide information about effective programs to the juvenile 
justice field, that is, program practitioners and communities. In 
addition, you asked us to provide information on OJJDP's research and 
evaluation planning. Specifically, this report addresses the following 
questions: 

* What do expert opinion and available research indicate about the 
types of juvenile reentry programs and juvenile substance abuse 
programs that are effective or cost beneficial? 

* To what extent does OJJDP have efforts under way to disseminate 
information about effective juvenile justice programs and assess the 
utility of the information it is providing through these efforts? 

* To what extent does OJJDP have plans in place to accomplish its 
juvenile justice research and evaluation goals? 

Scope and Methodology: 

To determine what experts and the available research indicate about the 
types of reentry programs and substance abuse programs that are 
effective or cost beneficial for juvenile offenders, we reviewed 
relevant literature, studies, and federal resources for juvenile 
justice programs, and interviewed federal officials and 26 juvenile 
justice experts.[Footnote 7] Specifically, to identify the types of 
programs to review, we conducted a literature search for studies and 
articles, including evaluations of juvenile reentry and juvenile 
substance abuse programs in the United States that were published from 
May 30, 1999, through May 30, 2009. We chose this time frame, the past 
10 years, because it provided us with an overview of the available 
research, including unpublished and ongoing studies, which assesses the 
effectiveness of reentry and substance abuse programs. We also 
consulted with OJJDP officials who coordinate research on juvenile 
justice programs and Department of Health and Human Services officials 
who oversee substance abuse and adolescent programs to obtain their 
recommendations for repositories--online databases that contain 
information on effective programs--and research studies and relevant 
Web sites for identifying types of reentry and substance abuse 
programs.[Footnote 8] Using these recommendations, information from 
relevant literature, and categories of program types used by OJJDP's 
Model Programs Guide, we identified five types of juvenile justice 
programs that are used to address reentry issues and five types of 
programs that are used to address substance abuse issues for juvenile 
offenders.[Footnote 9] 

Specifically related to substance abuse, we focused on substance abuse 
programs that involved relapse prevention treatment for juvenile 
offenders with substance abuse histories.[Footnote 10] After consulting 
with experts and reviewing the literature, we excluded juvenile alcohol 
abuse programs and substance abuse programs for the general juvenile 
population as well as at-risk juveniles who are prone to, but have not 
yet developed, substance abuse problems. For instance, we excluded 
after school or recreation programs, conflict resolution programs, and 
school or classroom programs. While all of these programs may have a 
substance abuse component, this component is not designed to address 
juvenile offenders' actual substance abuse problems. 

After identifying the types of programs to be reviewed, we looked at 
online databases, academic research, and professional organizations to 
select subject matter experts--researchers and practitioners--to obtain 
their views on the types of programs that have been shown to be 
effective or cost beneficial and the basis they used for making such 
determinations. We specifically identified researchers who focus on 
juvenile reentry issues or substance abuse issues and practitioners who 
operate programs that address these issues. We chose 26 experts to 
interview as a result of this process. Specifically, we selected 13 
individuals with expertise related to juvenile reentry programs, 7 
individuals with expertise related to juvenile substance abuse 
programs, and 6 individuals with both juvenile reentry and substance 
abuse program expertise. We selected these experts based on several 
criteria, including their employment histories related to juvenile 
reentry and substance abuse programs and the number of years they spent 
studying, evaluating, or managing programs addressing juvenile reentry 
or substance abuse issues. We evaluated their experience by reviewing 
the studies the researchers had completed and determining the 
experience the practitioners had managing the types of juvenile reentry 
and substance abuse programs selected for our review. See appendix I 
for the list of experts we interviewed. 

We asked these experts to provide their views about the effectiveness 
of program types (e.g., drug courts), rather than about the 
effectiveness of individual intervention programs (e.g., a specific 
drug court program that was implemented in one county).[Footnote 11] 
Because the Model Programs Guide, like other online repositories, 
contains information about the effectiveness of individual intervention 
programs, it does not provide information about the effectiveness of 
program types. As a result, we were interested in obtaining the 
experts' consolidated views of the effectiveness of program types. We 
also asked the experts to identify other program types--in addition to 
those that we explicitly asked about--that they considered to be 
effective or cost beneficial, but no additional program types were 
mentioned. In addition, we asked the experts to identify factors that 
in their view could help programs to achieve intended outcomes, such as 
reducing participants' recidivism, which are summarized in appendix II. 
While the results of these interviews cannot be generalized to reflect 
the views of all experts knowledgeable about juvenile reentry or 
substance abuse programs, we believe the interviews provided us with a 
good overview of the available research and valuable information about 
what program types are considered to be effective by subject matter 
experts. In addition, while we did not assess the methodological rigor 
of studies and evaluations in our review, we corroborated expert 
testimony by reviewing and summarizing the studies or evaluations that 
experts cited as the basis for their opinions. We also provided the 
experts with a summary of their opinions to review in order to ensure 
that we correctly captured their views. 

To identify the extent to which OJJDP has efforts under way to 
disseminate information about effective juvenile justice programs and 
assess the extent to which OJJDP ensures the utility of the information 
provided, we reviewed documentation, such as OJJDP's annual reports 
outlining information dissemination efforts, OJJDP publications, and a 
contract related to disseminating training information on effective 
programs. We interviewed knowledgeable OJJDP officials, such as the 
Training Coordinator and communications policy personnel, about OJJDP's 
efforts to disseminate information about effective programs. We 
selected two of OJJDP's efforts through which it disseminates 
information about effective programs--the Model Programs Guide and the 
National Training and Technical Assistance Center (NTTAC), which 
provides training and support to the juvenile justice field in 
identifying and implementing effective programs--because they provide 
information about effective programs across the range of issue areas in 
which OJJDP is involved, including reentry and substance abuse 
programs. We then compared these efforts to guidance articulated by the 
Office of Justice Programs (OJP), which oversees OJJDP, and in prior 
GAO reports that stresses the importance of assessing whether the 
information disseminated is meeting the needs of its users.[Footnote 
12] We also interviewed representatives from the two organizations that 
manage these two information dissemination efforts. Additionally, we 
asked the 26 juvenile reentry and substance abuse experts we 
interviewed about their views regarding OJJDP's information 
dissemination efforts and their opinions about the effectiveness of 
these efforts. Although their views cannot be generalized to the entire 
juvenile justice field, we believe that the experts provided us with a 
good overview of the utility of the information disseminated by OJJDP. 
We did not contact recipients of the information OJJDP disseminates for 
their views on the usefulness of the information provided because of 
the large volume of recipients and the resulting cost that would be 
incurred to obtain this input. 

To assess the extent to which OJJDP has plans in place for its research 
and evaluation efforts, we reviewed relevant laws related to the 
office's role in supporting research and evaluations of juvenile 
justice programs. We also reviewed relevant DOJ and OJJDP 
documentation, such as annual reports and strategic plans that contain 
information on OJJDP's research and evaluation goals and plans. We 
interviewed cognizant OJJDP officials about the office's planning 
efforts related to research and evaluation. We also reviewed criteria 
found in standard practices for program management and our prior 
products that highlight the importance of developing plans to meet 
goals and help ensure that resources are used effectively, and then 
compared these criteria to OJJDP's stated plans.[Footnote 13] 
Additionally, we analyzed OJJDP funding and staff data for fiscal years 
2005 through 2009 to better understand the resources the office has had 
available to support its evaluation activities. We chose these years 
because they provide the most recent overview of OJJDP's research and 
evaluation funding. 

We conducted this performance audit from April 2009 through December 
2009 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

Within states' juvenile justice systems, reentry aims to promote the 
effective reintegration of juvenile offenders back into communities 
upon release from residential facilities. Reentry is a process that 
incorporates a variety of programs to assist juvenile offenders in the 
transition from residential facilities to communities. In addition, 
reentry is intended to assist juvenile offenders in acquiring the life 
skills needed to succeed in the community and become law-abiding 
citizens and can incorporate the use of education, mental health, drug 
rehabilitation, or vocational training programs. While reentry begins 
after a juvenile is released back into the community, to help ensure a 
seamless transition, a reentry process begins after sentencing, then 
continues through incarceration, and into the period of release back 
into the community. According to OJJDP, juvenile justice practitioners 
and researchers believe that providing supervision and services to 
juvenile offenders returning to the community will reduce the high rate 
of recidivism among these juveniles. Several types of programs address 
juvenile reentry issues, as described in table 1. 

Table 1: Types of Juvenile Reentry Programs: 

Aftercare: programs that focus on the delivery of services and 
supervision that start while a juvenile is incarcerated to assist 
juvenile offenders in returning to their communities during the reentry 
process. These programs prepare juvenile offenders to return to the 
community by establishing collaboration with the community and 
marshaling its resources to help ensure that juvenile offenders receive 
services that address their individual needs, such as treatment for a 
substance abuse problem. 

Cognitive behavioral therapy: programs that are designed to identify 
and provide juveniles the skills to change thoughts and behavior that 
contribute to their problems. 

Reentry courts: specialized courts that manage the return of juvenile 
offenders to the community after they are released from residential 
facilities. A court manages reentry by using its authority to direct 
resources to support the offender's return to the community and promote 
positive behavior, among other things. For example, the court may 
oversee a juvenile's release into the community by assigning a judge to 
meet with the juvenile once a month. 

Vocational/job training: programs that provide juveniles with 
employment opportunities and are intended to improve juveniles' social 
and educational functioning by, for example, increasing earnings, 
raising self-esteem, and instilling a positive work ethic. 

Wraparound/case management: a strategy designed to keep delinquent 
youth at home and out of institutions whenever possible. This strategy 
involves making an array of individualized services and support 
networks available to juveniles, rather than requiring them to enroll 
in structured treatment programs, which may not address individual 
needs. 

Source: OJJDP’s Model Programs Guide. 

[End of table] 

Substance abuse includes, but is not limited to, the use or abuse of 
illegal drugs (e.g., heroin), prescription drugs, and nonprescription 
drugs (e.g., over-the-counter medications available without a 
prescription, such as cough suppressant). Treatment of substance abuse 
may occur in a variety of different settings, such as in clinics on an 
outpatient basis or at a hospital. Treatment can also occur in short- 
and long-term residential facilities that range from secure 
environments where juveniles' activities are physically restricted, to 
group homes, which are nonsecure settings where juveniles live and 
receive services in a homelike environment. According to the Center for 
Substance Abuse Treatment, juveniles with addictions to substances can 
be helped through programs that specifically target the factors 
associated with substance abuse--such as a family history of such 
abuse.[Footnote 14] For example, substance abuse intervention programs, 
such as cognitive behavioral therapy and family therapy, aim to change 
a juvenile's behavior by focusing on improving a juvenile's response to 
situations that contributed to prior substance abuse. Substance abuse 
intervention programs can be provided to juvenile offenders throughout 
the juvenile justice system: after sentencing, during incarceration, 
and after release back into the community. Whether treatment occurs 
while a juvenile is incarcerated or after the juvenile is released into 
the community, according to OJJDP, effective intervention programs can 
help addicted juveniles to overcome their substance abuse, lead crime- 
free lives, and become productive citizens. Table 2 describes types of 
programs--in addition to cognitive behavioral therapy and wraparound/ 
case management, which are discussed in table 1--that address juvenile 
substance abuse issues. 

Table 2: Types of Juvenile Substance Abuse Programs: 

Drug courts: specialized courts established within and supervised by 
juvenile courts to provide intervention programs, such as cognitive 
behavioral therapy or family therapy, for substance-abusing juveniles 
and their families. 

Family therapy: programs using trained therapists to treat juvenile 
offenders with substance abuse problems by including families of 
juveniles in the treatment, focusing on improving communication and 
interactions among family members, as well as improving overall 
relationships between juveniles and their families. 

Mentoring: programs that establish a relationship between two or more 
people over a prolonged period of time, where an older, more 
experienced individual provides support and guidance to a juvenile. The 
goal of mentoring is for the juvenile to develop positive adult 
contact, thereby reducing risk factors, such as exposure to juveniles 
who use substances, while increasing positive factors, such as 
encouragement for abstaining from substance use. 

Source: OJJDP’s Model Programs Guide. 

[End of table] 

The Juvenile Justice and Delinquency Prevention Act (JJDPA) established 
OJJDP in 1974.[Footnote 15] As the only federal office charged 
exclusively with preventing and responding to juvenile delinquency and 
with helping states improve their juvenile justice systems, OJJDP 
supports its mission through a variety of activities. For example, 
OJJDP administers a wide variety of grants to states, territories, 
localities, and public and private organizations through formula, 
block, and discretionary grant programs; provides training and 
technical assistance; produces and distributes publications and other 
products containing information about juvenile justice topics; and 
funds research and evaluation efforts.[Footnote 16] In fiscal year 
2009, the total appropriation for juvenile justice programs was about 
$374 million. See appendix III for more detailed information on OJJDP's 
enacted appropriations for fiscal years 2007 through 2009. 

OJJDP, through its various grant programs, has provided funding to 
states and organizations to support juvenile reentry and substance 
abuse programs, although the JJDPA does not specifically require OJJDP 
to fund them. States generally have the authority to determine how 
formula and block grants are allocated and may use these funds to 
support a range of program areas, including programs specifically for 
reentry or substance abuse. For example, from fiscal years 2007 through 
2008, OJJDP reported that states used approximately $7.1 million in 
applicable formula and block grant funds for programs that target 
reentry and $19 million in formula and block grant funds for programs 
that target substance abuse, representing approximately 1.8 percent and 
4.5 percent, respectively, of such funding for those years. 
Additionally, from fiscal years 2007 through 2009, OJJDP awarded a 
total of approximately $33 million in discretionary grants through four 
juvenile reentry grant programs and three substance abuse grant 
programs. Specifically, in the area of reentry, OJJDP awarded a total 
of $25.4 million to 38 grantees under 4 programs, and in the area of 
substance abuse, OJJDP awarded a total of $7.6 million to 15 grantees 
under 3 programs. See appendix IV for more detail on funding for these 
reentry and substance abuse programs. 

Experts Cite Evidence from Available Research Indicating That Cognitive 
Behavioral Therapy and Family Therapy Programs Are Effective and Cost 
Beneficial When Addressing Reentry and Substance Abuse Issues: 

Of the five reentry program types we reviewed, reentry experts reported 
that there is evidence from available research that cognitive 
behavioral therapy reduces recidivism. While experts cited a lack of 
evidence demonstrating that wraparound/case management, aftercare, and 
vocational/job training were effective in achieving results, such as a 
reduction in recidivism, they generally provided positive views on the 
potential results of these three types of programs, based on their own 
experience or knowledge of them.[Footnote 17] Similarly, of the five 
substance abuse program types we reviewed, juvenile substance abuse 
experts reported that there is evidence from available research that 
cognitive behavioral therapy along with family therapy are effective at 
reducing recidivism and show successful results at reducing substance 
abuse. However, expert opinions regarding other substance abuse program 
types, such as drug courts, mentoring, and wraparound/case management, 
were mixed, with experts stating that these program types could be 
effective, they were ineffective, or there was not enough evidence to 
determine effectiveness. Furthermore, both reentry and substance abuse 
experts cited studies indicating that cognitive behavioral therapy and 
family therapy programs are cost beneficial; however, the experts cited 
limited evidence for determining the costs and benefits of the other 
programs we reviewed. 

Reentry Experts Cited Evidence That Cognitive Behavioral Therapy Is 
Effective at Reducing Recidivism but Concluded That Other Programs Lack 
the Evidence Necessary to Determine Their Effectiveness: 

Experts Cited Evidence from Available Research Demonstrating That 
Cognitive Behavioral Therapy Is Effective at Reducing Recidivism: 

Eleven of the 12 experts we interviewed who provided comments based on 
their knowledge and experience with cognitive behavioral therapy stated 
that evidence from available research shows that these programs can be 
effective at reducing recidivism.[Footnote 18] Cognitive behavioral 
therapy intervention programs are designed to identify and provide 
juveniles with the skills to change thoughts and behaviors that 
contribute to their problems. The underlying principle of these 
programs is that thoughts affect emotions, which then influence 
behaviors. These intervention programs combine two kinds of 
psychotherapy--cognitive therapy[Footnote 19] and behavioral therapy. 
[Footnote 20] The strategies of cognitive behavioral therapy have been 
used to, among other things, prevent the start of a problem behavior--
such as violence and criminal activity--or stop the problem behavior 
from continuing. A juvenile offender can receive this type of 
intervention program after sentencing, throughout incarceration, or 
after returning to the community. For example, a cognitive behavioral 
therapy intervention program may provide individual and family services 
to treat a juvenile offender who has mental health and substance abuse 
issues. The treatment can occur during the juvenile's transition from 
incarceration back into the community and help the juvenile lower the 
risk of recidivism, connect the family with appropriate community 
support, assist the juvenile in abstaining from drugs, and improve the 
mental health of the juvenile. 

Based on their assessment of the available research, these 11 experts 
stated that cognitive behavioral therapy programs have been shown to be 
effective. Experts identified two meta-analyses[Footnote 21] of 
cognitive behavioral therapy programs that demonstrated effectiveness. 
[Footnote 22] One such study concluded that effective cognitive 
behavioral therapy programs are characterized by the low proportion of 
juveniles who dropped out of the program, as well as the close 
monitoring of the quality of the treatment and adequate training for 
the providers. In addition, this same study also found that 12 months 
after treatment, the likelihood of a juvenile who received cognitive 
behavioral therapy not recidivating was about one and a half times 
greater than for a juvenile who did not receive the therapy. This study 
also reported that the effects of cognitive behavioral therapy were 
greater for offenders who had a higher risk of recidivism than those 
with a lower risk. Specifically, the best results, in terms of 
recidivism reductions, occurred when high-risk offenders received more 
intensive treatment that targeted criminal thinking patterns. A second 
study also reported that among therapeutic interventions, such as skill 
building, cognitive behavioral therapy was most effective at reducing 
recidivism.[Footnote 23] The 12th expert stated that the particular 
cognitive behavioral therapy intervention program he was using-- 
aggression replacement training®--had not been evaluated at his 
particular program site, so he could not draw conclusions as to its 
effectiveness.[Footnote 24] 

Reentry Experts Indicated Positive Views of Wraparound/Case Management, 
Aftercare, and Vocational/Job Training Programs, but Stated That These 
Programs Lacked Evidence to Demonstrate Effectiveness: 

Despite having generally positive views on the results of wraparound/ 
case management, aftercare, and vocational/job training programs based 
on their experience or knowledge of these programs, reentry experts 
reported a lack of evaluations that show conclusive evidence about the 
effectiveness of these programs. Specifically, of the nine experts who 
provided comments on wraparound/case management programs, eight offered 
positive opinions about these programs. For example, two of these 
experts commented that wraparound/case management can be successful at 
reducing recidivism, depending on the quality and availability of 
services provided to juveniles. However, two of these eight experts 
also stated that there was a lack of evaluations demonstrating the 
effectiveness of wraparound/case management programs. One of these 
experts pointed us to a study on a specific wraparound/case management 
intervention program, Wraparound Milwaukee, that showed potentially 
promising results related to a reduction in recidivism rates for 
juvenile offenders.[Footnote 25] However, another expert cautioned that 
initial evaluations of wraparound/case management programs did not 
conclusively demonstrate the effectiveness of wraparound/case 
management programs.[Footnote 26] Finally, the ninth expert stated that 
in her experience, wraparound/case management interventions are not 
effective because, for example, juveniles are placed into these 
interventions based on the availability of program staff and resources 
rather than program services being tailored to the individual needs of 
each juvenile. 

In addition, 7 of the 15 experts who commented about aftercare programs 
opined that aftercare interventions are important reentry programs, in 
part, because they link a juvenile with the community and provide 
regular contact with a caseworker. However, 6 other experts said there 
was inconclusive evidence to determine whether these programs can be 
effective in achieving results. Three of these experts based their 
opinions on an evaluation of the Intensive Aftercare Program[Footnote 
27] that showed inconclusive results about program effectiveness. 
[Footnote 28] Specifically, the study found no evidence that the 
program had its intended impact of reducing recidivism among juveniles 
who were released back into the community under supervision in the 
three states that piloted the program. However, the evaluation did find 
that the three states that implemented the Intensive Aftercare Program 
model successfully incorporated most of its core features, which 
prepared juveniles to transition back into the community. For instance, 
these states created new Intensive Aftercare Programs--specific 
treatment programs that among other things, prepared juveniles for 
increased responsibility in the community, facilitated interaction with 
the community, and worked with the juveniles' schools and families. The 
state programs had a large percentage of juveniles involved in various 
treatment services. Despite the inconclusive results of the study, 1 
expert credited the aftercare program model with addressing the issue 
of juveniles interacting with multiple probation officers throughout 
the entire reentry process because aftercare programs, in general, 
assign one probation officer to a juvenile as a consistent point of 
contact. The remaining 2 of 15 experts opined that aftercare 
intervention programs had not been shown to be effective at achieving 
desired results because, for example, the treatment a juvenile receives 
depends on the services available in the community. 

With respect to vocational/job training programs, 10 of the 11 reentry 
experts who commented on these programs expressed positive opinions 
about the programs' potential outcomes but noted that there had been 
limited research conducted to demonstrate their effectiveness. 
Specifically, experts noted that vocational/job training programs could 
be beneficial if they were applied to older juveniles and if they led 
to those juveniles getting jobs. The remaining expert said there is 
little evidence to demonstrate the effectiveness of these intervention 
programs. For a more detailed description of reentry experts' opinions 
about these program types, see appendix V. 

Substance Abuse Experts Cited Evidence That Cognitive Behavioral 
Therapy and Family Therapy Are Effective at Reducing Recidivism and Can 
Help to Reduce Substance Abuse, but Said That Evidence of Effectiveness 
Was Lacking for the Other Programs: 

Experts Cited Evidence from Available Research Showing Cognitive 
Behavioral and Family Therapies Effectively Reduce Recidivism and 
Demonstrate Success at Reducing Substance Abuse: 

All of the 13 substance abuse experts we interviewed stated that based 
on available research, cognitive behavioral therapy effectively reduces 
recidivism and has demonstrated success at reducing substance abuse. 
Experts cited six studies to support their opinions, two of which were 
the same studies cited by reentry experts that demonstrate that 
cognitive behavioral therapy is effective at reducing 
recidivism.[Footnote 29] Two of the substance abuse experts noted that 
few studies have been conducted to determine whether an intervention 
program is effective at specifically reducing substance abuse. However, 
3 experts also noted that within the last decade, newly emerging 
research has shown promising results with respect to cognitive 
behavioral therapy program types reducing substance abuse. For example, 
these experts pointed us to three studies that report that juveniles 
who participated in these programs showed reductions in marijuana 
use.[Footnote 30] 

Twelve of the 13 substance abuse experts we interviewed who provided 
comments based on their knowledge and experience stated that family 
therapy programs are effective at reducing recidivism or decreasing 
substance use.[Footnote 31] Family therapy uses trained therapists to 
treat juvenile offenders with substance abuse problems by including 
families of juveniles in the treatment and focusing on improving 
communication and interactions among family members and improving 
overall relationships between juveniles and their families. This type 
of therapy focuses on the family as it is the primary and sometimes 
only source for emotional support, moral guidance, and self-esteem for 
juveniles. Family habits, such as failing to set clear expectations for 
children's behavior, poor monitoring and supervision, and severe and 
inconsistent discipline can often lead to juveniles engaging in 
delinquency and substance abuse, according to OJJDP. For example, 
family drug use often results in adolescent drug use. 

Based on their assessment of the available research, these 12 experts 
provided positive opinions about the effectiveness of family therapy, 
and 7 of these experts cited 9 studies that support their opinions. 
[Footnote 32] These studies demonstrated, for example, that 
multisystemic therapy--a family therapy intervention program that helps 
parents identify strengths and develop natural support systems (e.g., 
extended family, neighbors, friends, and church members)--is an 
effective intervention program for reducing recidivism and substance 
use because, for example, juveniles who participated in multisystemic 
therapy programs engaged in significantly less criminal activity than 
did nonparticipants. Specifically, multisystemic therapy participants 
had fewer average convictions per year for violent crimes than those 
juveniles who did not participate in the program. Additionally, 
analyses of drug tests demonstrated significantly higher rates of drug 
abstinence for program participants than for nonparticipants. One study 
also showed that participants in functional family therapy, another 
family therapy intervention program, had 50 percent reductions in 
substance use as compared to juveniles who did not participate in the 
program.[Footnote 33] 

Experts Had Mixed Views on Effectiveness of Drug Courts, Mentoring, and 
Wraparound/Case Management Types of Programs: 

According to the 10 experts who commented on drug courts, 5 stated that 
there is a lack of evidence to determine program effectiveness, while 
another expert stated that drug courts are ineffective types of 
programs because they expose first-time offenders to more serious drug 
users. The remaining 4 experts stated that drug courts can be effective 
if, for example, they are combined with other effective intervention 
programs, such as multisystemic therapy. Similarly, according to the 8 
experts who commented on mentoring, 1 stated that there are too few 
evaluations to determine effectiveness, while 4 stated that mentoring 
programs alone are ineffective or unsuccessful at achieving desired 
results and that mentoring intervention programs are more effective at 
preventing at-risk juveniles from engaging in delinquent behavior. 
[Footnote 34] However, 3 experts thought mentoring intervention 
programs could be effective if the programs adhere to certain factors 
that have been evaluated and shown to be effective, such as the mentor 
being properly trained.[Footnote 35] Finally, experts also had mixed 
views on the effect of wraparound/case management types of programs. Of 
the 11 experts who commented on these programs, 7 experts stated that 
wraparound/case management is effective or can be effective if, for 
example, wraparound/case management is combined with another 
intervention program that has been evaluated and has shown to be 
effective, such as cognitive behavior therapy. Conversely, 4 experts 
either stated that these programs are ineffective because, for example, 
the intervention programs lack follow-through as there are no 
consequences if a juvenile does not show up for treatment, or there is 
not sufficient evidence to determine effectiveness. For a more detailed 
description of substance abuse experts' opinions about these program 
types, see appendix VI. 

Reentry and Substance Abuse Experts Cited Available Research That 
Indicates Cognitive Behavioral and Family Therapies Are Cost 
Beneficial, but Provided Limited Evidence of the Costs and Benefits of 
the Other Program Types: 

While program evaluations establish if a program is effective in 
producing its intended results or effects--such as a reduction in 
recidivism--cost-benefit analyses use program evaluation to determine 
if the dollar value of a program's benefits exceeds the costs to 
deliver the program. For example, if a program evaluation shows that an 
intervention program reduces the number of offenses committed by 
juveniles from three to one, a cost-benefit analysis would first 
determine a dollar value for each of the offenses. Then, the cost- 
benefit analysis would estimate whether the savings of going from three 
offenses to one offense is more or less costly than the amount of money 
required to deliver the intervention program, as compared to an 
alternative program the juvenile would have received. The intervention 
may not always be more expensive than the alternative. For example, if 
the alternative is incarceration, the intervention program may be less 
expensive--meaning that the intervention program can be cost beneficial 
even if it does not result in a reduction of offenses. By applying the 
same cost-benefit analysis techniques to evaluations of different 
program types, decision makers can make comparisons among alternatives 
and determine which program types offer the greatest benefits for the 
least cost. The results of a cost-benefit analysis are often 
represented as a net benefit, meaning total benefits minus total cost. 

Of the 26 reentry and substance abuse experts we interviewed, 19 
provided information related to the cost benefits of the reentry and 
substance abuse program types in our review. These 19 experts 
identified five cost-benefit analyses of juvenile justice programs 
consisting of four meta-analyses[Footnote 36] and one systematic 
review.[Footnote 37] The studies demonstrate that various cognitive 
behavioral therapy and family therapy intervention programs are cost 
beneficial because they are effective at reducing crime and are 
expected to produce more benefits than costs compared to the 
alternative. For example, in one study, the authors reviewed several 
program interventions that fall into the family therapy program type, 
such as multisystemic therapy and multidimensional treatment foster 
care.[Footnote 38] The authors analyzed three program evaluations of 
multidimensional treatment foster care and found that this intervention 
can be expected to reduce crime outcomes by 22 percent. Based on this 
reduction in crime, the authors of the study predict that the 
intervention provides about $80,000 worth of benefits per participant. 
This dollar value reflects the savings per participant that result from 
a decrease in criminal activity, including savings to crime victims, 
police and sheriff's office costs, and juvenile detention costs, among 
others. 

The four studies cited by the experts show mixed or inconclusive 
results for drug courts, vocational/job training, and mentoring program 
types. For example, one study found that juvenile drug courts are cost 
beneficial because they are expected to have a net benefit of $4,622 
per program participant. The other studies could not determine drug 
courts' cost-effectiveness because they either did not include program 
evaluations of drug court programs or they found mixed results in the 
program evaluations analyzed and therefore could not determine the net 
benefits. In addition, two studies found that there are too few 
evaluations of vocational/job training or mentoring in juvenile justice 
programs to calculate if the benefits of these program types outweigh 
the costs. The remaining program types in our review--wraparound/case 
management, aftercare, and reentry courts--were not analyzed in these 
studies. Table 3 presents a summary of these studies. 

Table 3: Examples of Net Benefits of Program Interventions within 
Reentry and Substance Abuse Program Types We Reviewed from Four Studies 
Cited by Experts: 

Examples of the net benefits for interventions within program types: 

Studies cited by experts: Evidence-Based Public Policy Options to 
Reduce Crime and Criminal Justice Costs: Implications in Washington 
State (in 2007 dollars); 
Cognitive behavioral therapy[A]: $23,015; 
Family therapy[B]: $88,953; 
Drug courts: Findings are mixed; 
Vocational/job training: Too few recent evaluations; 
Mentoring: Too few evaluations to date. 

Studies cited by experts: Benefits and Costs of Prevention and Early 
Intervention Programs for Youth (in 2003 dollars); 
Cognitive behavioral therapy[A]: $8,805; 
Family therapy[B]: $24,290; 
Drug courts: Not included in study; 
Vocational/job training: Not included in study; 
Mentoring: $5,073. 

Studies cited by experts: Evidence-Based Public Policy Options to 
Reduce Future Prison Construction, Criminal Justice Costs, and Crime 
Rates (in 2006 dollars); 
Cognitive behavioral therapy[A]: $14,660; 
Family therapy[B]: $77,798; 
Drug courts: $4,622; 
Vocational/job training: Too few recent evaluations; 
Mentoring: Too few evaluations to date. 

Studies cited by experts: The Economic Return on Pennsylvania's 
Investment in Research-Based Programs: A Cost-Benefit Assessment of 
Delinquency Prevention in Pennsylvania (in 2007 dollars); 
Cognitive behavioral therapy[A]: Not included in study; 
Family therapy[B]: $79,331; 
Drug courts: Not included in study; 
Vocational/job training: Not included in study; 
Mentoring: Not included in study. 

Source: GAO analysis of four studies cited by experts. 

Notes: Dollar values indicate the highest net benefits for 
interventions for which net benefits were calculated (i.e., for which 
both cost and benefit data were available) and that experts categorized 
within the program types analyzed in this report. 

Net benefits are discounted, or adjusted, to reflect that costs are 
incurred when a juvenile initially receives the intervention, but 
savings may not result for many years. 

Net benefits estimates for (1) Evidence-Based Public Policy Options to 
Reduce Crime and Criminal Justice Costs: Implications in Washington 
State, (2) Benefits and Costs of Prevention and Early Intervention 
Programs for Youth, and (3) Evidence-Based Public Policy Options to 
Reduce Future Prison Construction, Criminal Justice Costs, and Crime 
Rates are for expected savings per program participant in Washington 
state. 

The Economic Return on Pennsylvania’s Investment in Research-Based 
Programs: A Cost-Benefit Assessment of Delinquency Prevention in 
Pennsylvania shows estimated net benefits per program participant in 
Pennsylvania. 

In cases where there are sizable differences in methodologies between 
studies, the benefits and costs reported cannot be directly compared. 
For example, there can be differences in the methodological criteria 
for inclusion, the surveys of literature can cover different time 
periods or differing program types, or values for benefits and costs 
can either come from external literature or can be drawn from the 
studies themselves. 

[A] Net benefits are for aggression replacement training®, which is an 
intervention program categorized as cognitive behavioral therapy by the 
experts. 

[B] Net benefits are for multidimensional treatment foster care, which 
is an intervention program categorized as family therapy by the 
experts. 

[End of table] 

In addition, seven experts also commented on reentry and substance 
abuse programs that were not included in the cited studies. For 
example, three experts opined that wraparound/case management programs 
may eventually be proven to be cost beneficial, based on preliminary 
research and evaluations. For example, one expert cited an unpublished 
study of a wraparound program pilot project that showed that recidivism 
of program participants was low, and that program costs were 
approximately 60 percent of the costs of incarcerating juveniles. 
Additionally, although experts did not cite cost-benefit analyses of 
aftercare program types, four reentry experts stated that such programs 
could be cost beneficial if the intervention program being delivered is 
effective because the cost of incarceration is so high. Three experts 
we interviewed stressed that even though some intervention programs 
that have been shown to be effective are expensive, if they reduce 
recidivism, they might be cost beneficial because of the high cost of 
incarcerating juveniles. 

OJJDP Has Mechanisms in Place to Ensure Training and Technical 
Assistance Meet Users' Needs, but Regular Feedback on the Model 
Programs Guide Would Help OJJDP Better Assess Information Utility: 

Consistent with the JJDPA, OJJDP has several efforts under way to 
disseminate information about effective juvenile justice programs. Two 
of these efforts--NTTAC and the Model Programs Guide--provide 
information about effective programs for a range of juvenile justice 
issues, including reentry and substance abuse issues. Consistent with 
federal guidelines for ensuring the utility of information, OJJDP has 
established mechanisms to ensure that the information provided through 
its training and technical assistance efforts meets the needs of the 
juvenile justice field. However, OJJDP could better ensure the 
usefulness of the information it disseminates through the Model 
Programs Guide by having a mechanism in place to solicit regular 
feedback specifically related to the guide from the juvenile justice 
field. 

OJJDP Disseminates Information about Effective Juvenile Justice 
Programs through Several Efforts: 

According to the JJDPA, OJJDP is authorized, but is not required, to 
provide information about juvenile justice issues and programs and to 
provide training and technical assistance to help the juvenile justice 
field implement and replicate such programs.[Footnote 39] In accordance 
with this authority and its mission to support states and communities 
in their efforts to develop and implement effective juvenile justice 
programs, OJJDP disseminates information related to these programs 
through a range of efforts, from those designed to meet the needs of 
the juvenile justice field as a whole to those that focus on effective 
programs in a specific issue area, such as gang prevention or girls' 
delinquency. OJJDP distributes the broadest range of information on 
juvenile justice topics through the Juvenile Justice Clearinghouse 
(Clearinghouse).[Footnote 40] Through its services, the Clearinghouse 
offers, among other things, the latest research findings and 
statistics, publications on juvenile justice issues and programs, 
announcements of funding opportunities, and other resources prepared by 
a variety of researchers in juvenile justice. As part of its efforts, 
the Clearinghouse responds to requests for information about effective 
programs by directing users to OJJDP efforts that develop and 
disseminate information about effective programs, such as NTTAC and the 
Model Programs Guide. Thus, we focused on NTTAC and the Model Programs 
Guide because they provide information about effective programs across 
the range of issue areas in which OJJDP is involved, including reentry 
and substance abuse programs. OJJDP also disseminates information about 
effective programs in specific issue areas through various centers, 
such as the National Youth Gang Center and the Underage Drinking 
Enforcement Center. For a more detailed discussion of these centers and 
other information dissemination efforts that focus on specific issues, 
see appendix VII. 

NTTAC was established in 1995, in part to provide information about 
effective juvenile justice programs--such as programs that address 
issues related to reentry and substance abuse--through its training and 
technical assistance efforts.[Footnote 41] According to OJJDP, NTTAC 
works to promote the use of effective programs in the field through 
training and technical assistance programs. Additionally, NTTAC 
develops training materials and resources, and customizes the 
information included in its curricula in an effort to best meet the 
needs of its training and technical assistance recipients. In terms of 
its efforts specifically related to program effectiveness, NTTAC 
provides training and technical assistance for members of the juvenile 
justice field on how to develop and sustain effective programs, and to 
help the field understand programs that are effective for various 
juvenile populations, such as juveniles with mental health issues or 
female offenders. 

The Model Programs Guide is an online database that contains summary 
information about approximately 200 juvenile justice programs, from 
prevention programs to reentry programs. It is designed to help 
practitioners and communities identify and implement prevention and 
intervention programs that have been evaluated and have been shown to 
be effective.[Footnote 42] Programs in the Model Programs Guide may 
focus on a range of issues, including delinquency, violence, youth gang 
involvement, substance abuse, or academic issues, and can include, but 
are not limited to, delinquency prevention, community service, drug 
courts, or family therapy. To be included in the Model Programs Guide, 
programs are reviewed and rated along several dimensions, including 
such factors as whether an evaluation of the program established a 
causal association between the treatment and the outcome. Users can 
search the Model Programs Guide to find programs that meet their 
specific needs. For example, users can look for a program that has been 
shown to be effective for juveniles with substance abuse problems who 
are first-time offenders, or they can search for a program that has 
been shown to be effective for juveniles involved in gang activities 
who are reentering the community. 

Evaluations and Needs Assessment Help OJJDP Ensure Usefulness of 
Information Provided by Training and Technical Assistance Efforts, but 
OJJDP Could Better Ensure the Utility of the Model Programs Guide's 
Information through Regular Feedback: 

In accordance with federal guidelines from OJP and prior GAO work, 
OJJDP has mechanisms in place to regularly conduct evaluations and is 
currently conducting a needs assessment to ensure the usefulness of the 
information provided by its training and technical assistance efforts. 
However, OJJDP could better ensure the utility of the information 
provided by the Model Programs Guide by establishing a mechanism to 
solicit regular feedback from the juvenile justice field. We have 
previously reported on the importance of regularly soliciting feedback 
to assess user needs and satisfaction.[Footnote 43] Specifically, we 
have reported that without feedback, an agency lacks valuable 
information from its users and is hindered in its ability to make 
improvements to information products that are relevant to users. 
Additionally, OJP has published Information Quality Guidelines for its 
bureaus, including OJJDP, that highlight the importance of ensuring the 
utility of information to be disseminated to the public by continuously 
monitoring information needs, among other things.[Footnote 44] 

OJJDP has mechanisms in place to regularly assess the usefulness of the 
information disseminated by NTTAC to ensure that it meets the needs of 
the juvenile justice field. Specifically, OJJDP has established an 
evaluation process for NTTAC that is designed to collect the data 
necessary to regularly assess the outcome and impact of the training 
and technical assistance NTTAC provides to improve the quality of the 
information it disseminates. Officials at NTTAC explained that after 
every training or technical assistance event, all participants are 
given an evaluation form to complete. This form is intended to capture 
feedback from participants about the quality of the event, as well as 
feedback about the referrals and resources NTTAC provides. Other 
evaluation forms are also available on NTTAC's Web site so that users 
can provide feedback about NTTAC's services, as well as feedback about 
the utility of the Web site. NTTAC then follows up with a sample of 
these respondents for more in-depth feedback. According to NTTAC 
officials, NTTAC analyzes the data collected from these forms and then 
provides them to OJJDP. These officials stated that OJJDP receives this 
information on at least a quarterly basis, and uses the information to 
make changes to existing curricula and guide future curriculum 
development, among other things. In accordance with OJP guidelines and 
prior GAO work that highlights the importance of assessing user needs, 
these evaluation efforts allow OJJDP to regularly monitor the 
usefulness of the information it disseminates in order to develop or 
modify its information products. 

In addition, OJJDP is conducting a needs assessment to solicit 
additional information about the utility of the information it 
disseminates through NTTAC's training and technical assistance efforts. 
NTTAC is administering the needs assessment and, according to NTTAC 
officials, it is designed to determine the training and technical 
assistance that would be most helpful to the field. Specifically, the 
needs assessment is soliciting feedback from members of the juvenile 
justice field about OJJDP's existing efforts. It is also requesting 
information regarding issues of interest to the field, any current 
training or technical assistance needs, and the specific challenges 
that the juvenile justice field is facing in its work. OJJDP officials 
stated that they intend to use the results of the needs assessment to 
influence the development of training and technical assistance 
activities and curricula and the content of national conferences and 
workshops. 

OJJDP's efforts to conduct evaluations and a needs assessment are 
consistent with comments we received from our expert interviews. We 
asked all 26 of the juvenile reentry and substance abuse experts we 
interviewed to comment on OJJDP's overall efforts to disseminate 
information about effective programs to the juvenile justice field. 
Thirteen experts provided responses, and while they did not comment 
specifically on NTTAC or the Model Programs Guide, they commented on 
the utility of the information OJJDP provides in general about 
effective programs. Ten of 13 experts had negative opinions of how 
useful the information OJJDP disseminates is to members of the juvenile 
justice field. For example, 1 expert stated that practitioners often do 
not have the time to read research data disseminated by OJJDP, which 
prevents them from being able to effectively use it in their work. The 
expert added that it would be more useful if OJJDP disseminated 
information that was practical and could be applied in the field. In 
addition, 2 of these 10 experts suggested that it would be helpful for 
OJJDP to obtain feedback from members of the juvenile justice field 
about what types of information they would find useful. Thus, OJJDP's 
needs assessment should help to address this concern. The remaining 3 
experts who commented on OJJDP's information dissemination efforts had 
generally positive opinions, stating that the information is useful to 
researchers and practitioners. 

With respect to the Model Programs Guide, although OJJDP has ad hoc 
mechanisms in place to solicit feedback about the information it 
provides, it does not solicit this feedback on a regular basis or use 
feedback to help ensure that the information disseminated by the Model 
Programs Guide is useful to the field, in accordance with federal 
guidelines. For example, the Model Programs Guide's Program Director 
gives several presentations about the guide each year at juvenile 
justice conferences. Officials who operate the Model Programs Guide 
stated that following these presentations, they request verbal feedback 
from participants. Officials also stated that they regularly receive 
unsolicited feedback through the e-mail address that is listed on the 
Model Programs Guide's Web site, which they respond to on a case-by- 
case basis. Additionally, officials said that they collect feedback 
about the Model Programs Guide through an annual e-mail survey that is 
sent to the program points of contact listed on the guide to obtain 
updated program information. Although these efforts to solicit feedback 
about the Model Programs Guide provide OJJDP with some information from 
its users, according to OJJDP officials, because the guide does not 
have a systematic feedback mechanism, information received cannot be 
analyzed on an aggregate level in order to regularly assess how the 
juvenile justice field views the utility of the information provided by 
the Model Programs Guide. Further, while the annual e-mail survey can 
help OJJDP confirm that the program information featured in the Model 
Programs Guide is accurate, it does not provide information about 
whether the guide is useful to the field as a whole since OJJDP sends 
the survey's request for comments about the Model Programs Guide to a 
portion of the juvenile justice field whose programs are already 
published in the guide, which means that the comments it receives about 
the Model Programs Guide do not necessarily reflect the opinions of the 
juvenile justice field as a whole. 

OJJDP officials agreed that they had not established a systematic 
mechanism to obtain feedback from the field regarding the usefulness of 
the Model Programs Guide and recognized that such a mechanism would be 
useful to have in place. Officials also stated that NTTAC's needs 
assessment might be used as a model to build in more consistent 
mechanisms for feedback for the office's broader efforts. Because NTTAC 
uses evaluations and is taking steps to conduct a needs assessment to 
monitor the information needs of the juvenile justice field, OJJDP is 
in a better position to help ensure that the information provided by 
NTTAC is useful to the juvenile justice field. Recognizing that, 
although there is a cost associated with gathering and analyzing 
feedback data, establishing a cost-effective mechanism to regularly 
solicit feedback about the Model Programs Guide should provide OJJDP 
with the information necessary to assess whether the information 
provided by this tool is useful to the juvenile justice field. 

Finalizing a Program Plan and Developing a Comprehensive Evaluation 
Plan Would Help OJJDP Achieve Its Research and Evaluation Goals and Use 
Its Limited Resources Effectively: 

OJJDP has articulated research and evaluation goals to support its 
mission of promoting effective programs and improving the juvenile 
justice system. According to OJJDP, one of its three main goals is to 
promote improvements in juvenile justice and facilitate the most 
effective allocation of resources by conducting research to understand 
how the juvenile justice system works in serving children and 
families.[Footnote 45] Under the JJDPA, OJJDP is required to publish an 
annual program plan that describes planned activities that are under 
accounts authorized for research and evaluation activities and that 
demonstrate promising initiatives, among other things.[Footnote 46] 
This plan is required to be published annually in the Federal Register 
for public comment, and is to describe the activities the Administrator 
intends to carry out under parts D and E, the appropriations accounts 
that in general are available for research and the development of new 
programs and initiatives, respectively.[Footnote 47] Specifically, 
according to the JJDPA, the Administrator must take into account the 
public comments received during the 45-day period and develop and 
publish a final plan before December 31 of each fiscal year, describing 
the particular activities that the Administrator intends to carry out 
under parts D and E. While OJJDP has not published an annual program 
plan since 2002, it issued a proposed plan in the Federal Register to 
solicit public comment in December 2009.[Footnote 48] OJJDP aims to 
publish the final version once public comments are incorporated, in 
accordance with the JJDPA's requirements.[Footnote 49] Although the 
annual program plan is required to describe the particular activities 
the Administrator intends to carry out under parts D and E of the 
JJDPA, the proposed program plan includes the office's priorities with 
respect to all discretionary funding, including its research and 
evaluation efforts. According to the Acting Administrator, this will, 
in part, provide complete transparency for all such funding. According 
to OJJDP, the development and publication of the annual program plan is 
a first step that will lead to a comprehensive evaluation plan as the 
annual program plan outlines the agency's overall research and 
evaluation goals. 

Additionally, the Office of Management and Budget's (OMB) fiscal year 
2006 Program Assessment Rating Tool found that juvenile justice 
programs would benefit from evaluations of their effectiveness but 
noted that such evaluations are difficult and expensive to do.[Footnote 
50] As a result, OMB recommended that OJP develop a comprehensive 
evaluation plan for juvenile justice programs to obtain better 
information about the programs' impacts.[Footnote 51] Although OMB's 
recommendation was directed at OJP, OJP and OJJDP officials stated that 
because OJJDP is the office within OJP required to conduct juvenile 
justice evaluations, it is that office's responsibility to develop this 
evaluation plan. 

In addition to the above requirement and recommendation, federal 
guidelines stipulate the importance of developing a plan to achieve 
agency goals. As established in the standard practices for program 
management, specific goals of an agency must be conceptualized and 
defined in a plan.[Footnote 52] Specifically, this plan is to contain a 
description or road map of how the goals and objectives are to be 
achieved, including identifying the needed resources and target 
milestones or time frames for achieving desired results. We have also 
reported on the importance of planning research and evaluation efforts, 
in part to ensure that goals are met and resources are used 
effectively.[Footnote 53] 

OJJDP's Research Coordinator stated that such a road map or plan for 
conducting research and evaluation would help better target the 
agency's research and evaluation efforts toward achieving their goals. 
However, from 2006 to 2009, OJJDP had not developed such a plan, 
primarily because of resource constraints. According to this official, 
in lieu of having a comprehensive evaluation plan in place to guide its 
research and evaluation efforts, the office's efforts are influenced by 
a number of factors, including whether Congress directs the agency to 
conduct research in a particular area or whether ideas are generated 
internally by staff or externally by members of the juvenile justice 
field. For example, OJJDP staff responsible for the mentoring area may 
generate ideas about how available research funds could be used, for 
example, by evaluating a particular type of mentoring program. In 
addition, the office may receive recommendations from the Federal 
Advisory Council on Juvenile Justice or feedback from others in the 
juvenile justice field. While these factors have influenced OJJDP's 
research and evaluation efforts, they have not provided a framework for 
helping the office meet its research and evaluation goals. Therefore, 
once the program plan is finalized, OJJDP intends to develop a 
comprehensive evaluation plan in accordance with OMB recommendations to 
provide direction and priorities for its research and evaluation 
efforts. According to the Acting Administrator, OJJDP intends to use 
this comprehensive evaluation plan to better align and target available 
discretionary funds toward achieving its research and evaluation goals. 

In addition to having a road map to help ensure it meets its goals, it 
is important for OJJDP to have a comprehensive plan that lays out how 
the office will evaluate its juvenile justice programs. Such a plan 
would help to ensure that its limited resources are being used 
effectively. This is important because OJJDP does not currently receive 
dedicated funding for research and instead must make trade-off 
decisions to balance funding to implement programs with funding to 
evaluate which programs are effective. The office has not received 
dedicated research funding since fiscal year 2005 when it received $10 
million for its part D appropriations account--the appropriations 
account specifically available for research and evaluation efforts. 
[Footnote 54] Without part D funding, OJJDP has relied on funds it has 
set aside from its other appropriation accounts to fund its research 
and evaluation activities.[Footnote 55] Specifically, as shown in table 
4, OJJDP is authorized by the appropriations act to set aside up to 10 
percent of certain appropriations accounts for its research and 
evaluation efforts. In fiscal year 2008, the last year for which set-
aside funding data are available, the appropriations act authorized 
OJJDP to set aside over $23 million for research and evaluation. 
However, according to OJJDP, the office set aside approximately $11 
million. OJJDP officials stated that this was, in part, because the 
JJDPA requires and the agency wants to ensure that sufficient funds are 
available to the states for grant programs.[Footnote 56] In addition, 
officials explained that some of OJJDP's accounts are transferred to 
other program offices, such as the Office of Community Oriented 
Policing Services, so research funds are not deducted from those 
accounts.[Footnote 57] Of the over $11 million that OJJDP did set 
aside, officials reported that the office used nearly $8 million (or 70 
percent) for research and evaluation. Table 4 shows the amounts 
authorized to be set aside by the annual appropriations act, as well as 
the amounts actually set aside and used by OJJDP. 

Table 4: OJJDP Funds Authorized and Used for Research and Evaluation, 
Fiscal Year 2008: 

OJJDP appropriations account: Part A - Concentration of Federal 
Efforts[C]; 
Issue area funded by the account[A]: Cooperation and coordination 
between federal agencies involved in juvenile justice issues; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $65,800; 
Funds set aside by OJJDP: 0; 
Funds used by OJJDP: 0. 

OJJDP appropriations account: Part B - State Formula Grants[D]; 
Issue area funded by the account[A]: Grants to state and local 
governments for juvenile delinquency programs; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $7,426,000; 
Funds set aside by OJJDP: $5,940,800; 
Funds used by OJJDP: v3,170,011. 

OJJDP appropriations account: Part D - Research, Evaluation, Technical 
Assistance and Training[E]; 
Issue area funded by the account[A]: Research and evaluation; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: 0; 
Funds set aside by OJJDP: 0; 
Funds used by OJJDP: 0. 

OJJDP appropriations account: Part E - Developing, Testing, and 
Demonstrating Promising New Initiatives and Projects[F]; 
Issue area funded by the account[A]: Development and demonstration of 
new projects and initiatives; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: N/A; 
Funds set aside by OJJDP: N/A; 
Funds used by OJJDP: N/A. 

OJJDP appropriations account: Youth Mentoring Grants; 
Issue area funded by the account[A]: Mentoring programs; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $7,000,000; 
Funds set aside by OJJDP: $907,620; 
Funds used by OJJDP: v907,479. 

OJJDP appropriations account: Title V - Local Delinquency Prevention 
Incentive Grants[G]; 
Issue area funded by the account[A]: Tribal programs, gang prevention, 
alcohol prevention, other local delinquency programs; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $320,000; 
Funds set aside by OJJDP: $320,000; 
Funds used by OJJDP: $124,623. 

OJJDP appropriations account: Secure Our Schools[H]; 
Issue area funded by the account[A]: Improved security at schools; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $1,504,000; 
Funds set aside by OJJDP: 0; 
Funds used by OJJDP: 0. 

OJJDP appropriations account: Victims of Crime Act - Improving the 
Investigation and Prosecution of Child Abuse Programs; 
Issue area funded by the account[A]: Training and technical assistance 
for professionals involved in investigating, prosecuting, and treating 
issues related to child abuse; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $1,692,000; 
Funds set aside by OJJDP: 0; 
Funds used by OJJDP: 0. 

OJJDP appropriations account: Juvenile Accountability Block Grant; 
Issue area funded by the account[A]: Juveniles already involved in the 
juvenile justice system; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $5,170,000; 
Funds set aside by OJJDP: $4,136,000; 
Funds used by OJJDP: $3,626,917. 

OJJDP appropriations account: Project Childsafe[I]; 
Issue area funded by the account[A]: Safe firearm handling and storage 
practices; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: 0; 
Funds set aside by OJJDP: 0; 
Funds used by OJJDP: 0. 

OJJDP appropriations account: Total; 
Maximum funds authorized by the appropriations act for research and 
evaluation 10 percent of amount appropriated)[B]: $23,177,800; 
Funds set aside by OJJDP: $11,304,420; 
Funds used by OJJDP: $7,829,030. 

Sources: Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 
121 Stat. 1844, 1911-12 (2007); the JJDPA; and GAO analysis of OJJDP 
data. 

[A] The appropriation act allows OJJDP to set aside funds for research, 
evaluation, and statistics activities designed to benefit the programs 
or activities authorized. Consolidated Appropriations Act, 2008, Pub. 
L. No. 110-161, 121 Stat. 1844, 1911-12 (2007). As such, the accounts 
from which OJJDP sets aside funds for research and evaluation are tied 
to specific issue areas, and therefore all of the set-asides must be 
used for research, evaluation, and statistics activities designed to 
benefit these specific issue areas. For example, the mentoring 
appropriation account can only be used for research and evaluation 
activities related to mentoring programs. 

[B] The appropriations statute for fiscal year 2008 provides that not 
more than 10 percent of each amount appropriated may be used for 
research, evaluation, and statistics activities that benefit the 
programs or activities authorized, and not more than 2 percent for 
training and technical assistance. Consolidated Appropriations Act, 
2008, Pub. L. No. 110-161, 121 Stat. 1844, 1911-12 (2007). These 
provisions applied to appropriation accounts for juvenile justice 
programs, but did not apply to amounts appropriated for grants and 
projects authorized by sections 261 and 262, part E, of the JJDPA—which 
is the account available for developing, testing, and demonstrating 
promising new initiatives and projects. See id. 

[C] According to OJJDP officials, historically, OJJDP has elected not 
to set aside funds from part A as the appropriation supports the 
Coordinating Council on Juvenile Justice and Delinquency Prevention—a 
council created, in part, to coordinate all federal juvenile offender 
programs in cooperation with state and local juvenile justice programs. 

[D] Under the formula grants program, funds are to be allocated among 
the states on the basis of relative population of people under age 18; 
however, the JJDPA also sets base amounts for awards to the states, 
depending on the aggregate amount appropriated for the program each 
year and taking into account amounts allocated to the states for fiscal 
year 2000. 42 U.S.C. § 5632. 

[E] Funds for part D have not been appropriated since fiscal year 2005. 

[F] The provision in the appropriations act allowing the office to set 
aside up to 10 percent of each amount appropriated is not applicable to 
grants and projects authorized by sections 261 and 262, part E, of the 
JJDPA and is authorized for developing, testing, and demonstrating 
promising new initiatives and projects. Consolidated Appropriations 
Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 1911-12 (2007). 

[G] As described in app. III, the appropriation account for Title V 
programs includes funds that are carved out for specific purposes. 
Pursuant to the appropriations act, these amounts are not available for 
set-asides, as they are under sections 261 and 262, part E, of the 
JJDPA, whereas funds may be set aside from the remaining amounts 
available for Title V delinquency prevention programs. Consolidated 
Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 1911-12 
(2007). 

[H] According to OJJDP officials, the office does not set aside funds 
from this account because the entire appropriation is transferred to 
the Office of Community Oriented Policing Services—a component of DOJ 
responsible for helping state and local law enforcement agencies 
prevent crime. 

[I] In fiscal year 2008, Project Childsafe was not appropriated funds 
as part of the appropriations for juvenile justice programs. 

[End of table] 

Additionally, all of the set-asides from these four accounts must be 
used for research, evaluation, and statistics activities designed to 
benefit the juvenile justice issues that the accounts specify. For 
example, set-aside funds from the youth mentoring grant appropriation 
account must be used to research or evaluate mentoring programs. For 
other accounts, OJJDP can elect to fund research and evaluation efforts 
in a number of different areas. For example, under Juvenile 
Accountability Block Grants, OJJDP provides funds to states and units 
of local government to strengthen the juvenile justice system. The 
states can use these funds for 17 different purpose areas, including 
establishing programs to help the successful reentry of juvenile 
offenders from state and local custody in the community or for hiring 
staff or developing training programs for detention and corrections. 
Consequently, there are limits on the amount of funds OJJDP can divert 
to research and evaluation and on its discretion over how to use of 
some of these funds. In fiscal year 2008, the appropriation act allowed 
OJJDP to set aside more than $23 million that could be dedicated to 
research and evaluation efforts on numerous eligible programs. Because 
OJJDP has to decide how to split set aside funds between supporting 
state and local program implementation and program evaluation, in 
accordance with federal guidelines, a comprehensive evaluation plan 
that in part identifies its funding resources could help OJJDP make 
this determination. 

According to OJJDP, the office has spent several years considering 
developing a plan to provide a road map for how it would meet its 
research and evaluation goals. However, officials stated that it has 
been difficult to complete a comprehensive evaluation plan to fulfill 
OMB's Program Assessment Rating Tool recommendation because they have 
not had the resources available--that is, funding and staffing--to 
develop the plan. Specifically, because funds have not been 
appropriated for part D since fiscal year 2005, OJJDP has not had a 
dedicated source of funding that could be used to develop a 
comprehensive evaluation plan or to fund the research identified by 
such a plan. Additionally, in 2003, OJJDP reorganized its divisions 
and, as part of this, dissolved its research division, as well as the 
training and information dissemination units. According to OJJDP, the 
intention of the former Administrator who implemented this 
reorganization was to better integrate these functions throughout the 
agency. OJJDP officials stated that those staff who were dedicated to 
research and evaluation work were reassigned to other divisions. 
Although some of these staff retained the research projects they had at 
the time, they also assumed new grant management duties. Also, over the 
past 8 fiscal years, OJJDP's overall authorized staffing level has 
decreased from 95 to 76. Specifically, those staff dedicated to 
research and evaluation decreased from 10 in fiscal year 2002 to 3.5 in 
fiscal year 2009. According to OJJDP officials, the reduction in staff 
who were dedicated to research and evaluation has strained the staffing 
resources that could be used for developing a comprehensive evaluation 
plan. 

Although OJJDP cited funding and staffing constraints, the Acting 
Administrator has made developing a comprehensive evaluation plan a 
priority and the office is committed to moving forward with developing 
this plan. Following through with its planning efforts will help OJJDP 
to meet its research and evaluation goals and better ensure that its 
resources are being used effectively as stipulated by federal 
guidelines. 

Conclusions: 

As the juvenile justice field--including states and local communities-
-works to implement programs to lower juvenile recidivism rates and 
address juvenile substance abuse, it is important that the field has 
information about which programs have been shown to be effective 
through program evaluations. The importance of OJJDP's goal to research 
and evaluate programs to improve juvenile delinquency underscores the 
need for a comprehensive plan to evaluate juvenile justice programs, 
one that identifies resources to be committed to its research and 
evaluation efforts and outlines the details of how OJJDP will 
accomplish its research and evaluation goals. OJJDP efforts to publish 
a fiscal year 2010 program plan in December are positive steps in 
developing the comprehensive evaluation plan that officials have said 
they are committed to developing. Having such a plan will provide OJJDP 
with a road map to help ensure that it meets its research and 
evaluation goals, uses its limited resources effectively, and 
contributes to identifying effective programs to help support states 
and localities. With respect to OJJDP's efforts to disseminate 
information about effective programs, NTTAC's efforts to regularly 
assess the needs for the information it is disseminating through 
training and technical assistance are important to helping OJJDP assess 
the utility of its efforts and make appropriate improvements. We also 
recognize that OJJDP's efforts to conduct a needs assessment could help 
provide important information to NTTAC that can be used in conjunction 
with its evaluation efforts. Consistent with federal guidelines from 
OJP and prior GAO reports, assessing the utility of the information 
disseminated through OJJDP's Model Programs Guide is also critical to 
ensuring that such information meets the needs of the juvenile justice 
field so the field can better implement effective programs. Having a 
mechanism in place to regularly solicit feedback from the field about 
the usefulness of the Model Programs Guide would better position OJJDP 
to assess whether the information it is disseminating through the guide 
on effective programs regularly meets the needs of its users. 

Recommendation for Executive Action: 

To help ensure that OJJDP's Model Programs Guide is regularly meeting 
user needs and providing the most helpful information on effective 
programs, consistent with federal guidelines, we recommend the 
Administrator of OJJDP develop a cost-effective mechanism for regularly 
soliciting and incorporating feedback from the juvenile justice field 
on the usefulness of the information provided in its Model Programs 
Guide. 

Agency Comments and Our Evaluation: 

We provided a copy of this report to the Attorney General for review 
and comment. On December 3, 2009, OJP provided written comments, which 
are reprinted in appendix VIII. OJP stated that it agreed with our 
recommendation and intends to develop a mechanism for regularly 
soliciting and incorporating feedback from the juvenile justice field 
on the usefulness of the information provided in its Model Programs 
Guide by March 31, 2010. OJP also provided technical comments, which we 
incorporated as appropriate. 

We are sending copies of this report to the Attorney General, selected 
congressional committees, and other interested parties. The report also 
is available at no charge on the GAO Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact Eileen Larence at (202) 512-8777 or larencee@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. Key contributors to this 
report are listed in appendix IX. 

Sincerely yours, 

Signed by: 

Eileen R. Larence: 
Director, Homeland Security and Justice Issues: 

[End of section] 

Appendix I: Juvenile Justice Reentry and Substance Abuse Experts We 
Interviewed: 

For the purposes of our review, we selected a total of 26 experts to 
interview--13 of whom had expertise related to juvenile reentry 
programs, 7 of whom had expertise related to juvenile substance abuse 
programs, and 6 of whom had both juvenile reentry and substance abuse 
program expertise. See table 5 for a list of these experts. 

Table 5: Juvenile Justice Experts We Interviewed about Juvenile Reentry 
or Substance Abuse Programs: 

Experts on reentry programs: 

* David Altschuler, John Hopkins University: 
* Troy Armstrong, California State University: 
* Richard Dembo, University of South Florida: 
* Lynn Ellsworth, Creative Strategy Group and Columbia University: 
* Jim Heafner, McLaughlin Youth Center: 
* Peter Jones, Temple University: 
* Barry Krisberg, National Council on Crime and Delinquency: 
* Kevin Minor, Eastern Kentucky University: 
* Shelli Rossman, Urban Institute: 
* Richard Steele, Juvenile Court Judges' Commission, State of 
Pennsylvania: 
* Mercer Sullivan, Rutgers University: 
* Christy Visher, University of Delaware: 
* Richard Wiebush, National Council on Crime and Delinquency: 

Experts on substance abuse programs: 

* Paul Boxer, Rutgers University: 
* Laurie Chassin, Arizona State University: 
* Michael Dennis, Chestnut Health Systems: 
* Scott W. Henggeler, Medical University of South Carolina: 
* Randolph Muck, Substance Abuse and Mental Health Services 
Administration[A]: 
* Lawrence Murray, Columbia University: 
* John Roman, The Urban Institute: 

Experts on both reentry programs and substance abuse programs: 

* Steve Aos, Washington State Institute for Public Policy: 
* Peter Greenwood, Association for the Advancement of Evidence-Based 
Practice: 
* Nancy Jainchill, National Development and Research Institutes, Inc.: 
* Bruce Kamradt, Milwaukee County: 
* Doug Kopp, Washington State Juvenile Rehabilitation Administration: 
* Mark Lipsey, Vanderbilt University: 

Source: GAO. 

[A] During our interview with Randolph Muck, other researchers were 
present, including H. Westley Clark, Director, Substance Abuse and 
Mental Health Services Administration. 

[End of table] 

[End of section] 

Appendix II: Juvenile Justice Experts' Views on Factors That Can Help 
Programs Achieve Intended Outcomes: 

Of the 26 reentry and substance abuse experts we interviewed, 22 
experts--as well as research we reviewed--identified several factors 
that can help programs achieve intended outcomes, that is, be 
effective.[Footnote 58] The following factors, while not an exhaustive 
list of items for programs to consider when implementing juvenile 
justice intervention programs, were the most frequently cited by the 
experts we interviewed: 

* maintaining fidelity to the program; 

* selecting, training, and retaining qualified providers; 

* conducting needs-based assessments to provide individualized 
treatment; and: 

* improving juvenile program participation by engaging and motivating 
juvenile and family involvement. 

While incorporating these factors into reentry or substance abuse 
programs does not guarantee that any particular intervention program 
will be successful, existing programs that have been evaluated and 
found to be effective have generally included these factors in their 
designs or implementation. 

Maintaining Fidelity to the Program: 

According to 17 of the 22 experts, maintaining fidelity to the program 
as it was intended to be implemented can help programs achieve their 
intended objectives. This factor focuses on ensuring that core program 
services or intervention components are delivered as they were 
designed, that is, with fidelity. For example, for cognitive behavioral 
therapy, this would entail that core intervention components, such as 
cognitive and social skills training, were provided exactly as they 
were designed to each participant.[Footnote 59] According to one 
expert's research, the degree to which an intervention program is 
delivered with fidelity is closely related to its effects on 
recidivism. Another expert concurred, stating that the more closely 
core program services or intervention components are implemented as 
they were designed, the more the intervention program will reduce 
recidivism rates. For example, one expert emphasized the importance of 
maintaining fidelity to the program when replicating the model within a 
specific community. In particular, another expert explained that some 
therapists tend to substitute their own preferred treatment techniques 
instead of using those prescribed by the intervention program, which 
can affect how effective a program is at reducing recidivism. This is 
particularly true if the intervention program being delivered is a 
program that has been evaluated and found to be effective. Furthermore, 
one of these experts stated that the specific model chosen has less of 
an effect on intended outcomes than the manner in which it is 
delivered. As another expert explained it, a weaker intervention--one 
that has not been evaluated and proven to be effective--may result in 
decreased recidivism rates, for example, if it is implemented as 
designed, while an effective intervention program that is implemented 
poorly may have little or no effect on intended outcomes. 

Selecting, Training, and Retaining Qualified Providers: 

According to 19 of the 22 experts, selecting, training, and retaining 
qualified providers can help intervention programs achieve intended 
outcomes. For example, the quality of the services that cognitive 
behavioral therapy delivers depends, in part, on the provider's ability 
and whether the provider has been trained on the specific therapies and 
components of the intervention program. Three of these experts noted 
that if providers are not appropriately trained in the therapy or 
intervention being implemented, they may not provide the program as it 
was intended, or as one of them noted, may substitute their own 
preferred treatment techniques for those prescribed by the intervention 
program. As a result, the providers' failure to deliver the 
intervention program as it was designed reduces the ability of the 
program to achieve intended outcomes. Furthermore, many intervention 
programs utilize providers who have certain educational or clinical 
experience, such as having a background in mental health or being a 
licensed practitioner for the specific therapy being implemented. One 
of the 19 experts we interviewed also mentioned the importance of 
gaining the support of the juvenile justice community, as well as 
agencies' program management, in the selection and training of 
providers. 

Conducting Needs-Based Assessments to Provide Individualized Treatment: 

According to 18 of the 22 experts, by assessing a juvenile's specific 
treatment needs, program providers can better design intervention 
programs that will be targeted to a juvenile's individual situation. 
For example, 4 experts noted that this can help intervention programs 
achieve intended outcomes because individualized treatment is more 
likely to affect participants' individual outcomes since it takes into 
account differences such as age, gender, culture, environment, and 
problem severity. One expert noted that individualized treatment 
ensures that juveniles do not receive unnecessary treatment, which in 
some instances may produce harmful results. According to this expert, 
providing juveniles who do not have substance abuse problems some 
programs, such as cognitive behavioral therapy, may lead to harmful 
results because these juveniles are exposed to others who have more 
serious addictions. Four experts also noted that using needs-based 
assessments to develop individualized treatment plans can be more cost 
beneficial than using standard treatment plans. Specifically, as one of 
these experts noted, this is because individualized treatment plans can 
help ensure that costly interventions are not provided to juveniles who 
do not need extensive services. In addition, 5 experts stated that 
conducting a risk-based assessment is important to determining which 
juveniles are at higher risk of reoffending in order to focus 
programming efforts on them.[Footnote 60] One of these experts cited a 
study that shows that targeting specific treatment needs of offenders 
is correlated with recidivism outcomes, that is, providing targeted 
treatment needs is generally related to lower recidivism.[Footnote 61] 

Improving Juvenile Participation by Engaging and Motivating Juvenile 
and Family Involvement: 

According to 16 of the 22 experts, engaging and motivating juvenile and 
family involvement can help to improve a juvenile's program 
participation, thereby helping intervention programs to achieve 
intended outcomes. For example, 1 expert noted that successful programs 
rely on staff members to gain the trust of juvenile offenders. These 
programs also recognize that juveniles may experience program fatigue 
because they are participating in numerous programs and that motivation 
may become an issue. In addition, this expert noted that after being 
released into the community, juveniles and their families may not be 
motivated to participate in intervention programs. Additionally, 
research has shown that encouraging families to participate in the 
juvenile's treatment program can reduce family risk factors for 
delinquency.[Footnote 62] Eleven experts also mentioned that motivating 
juvenile offenders and their families to participate can assist 
juveniles in successfully completing an intervention program. Two of 
these experts noted that by involving family members in treatment, some 
issues that may contribute to juvenile dropout rates, such as a history 
of traumatic stress and family members who also abuse substances, can 
be addressed within an intervention program.[Footnote 63] 

[End of section] 

Appendix III: OJJDP's Enacted Appropriations for Fiscal Years 2007 
through 2009: 

To prevent and respond to juvenile delinquency and help states improve 
their juvenile justice systems, the Office of Juvenile Justice and 
Delinquency Prevention (OJJDP) administers a wide variety of grants to 
states, territories, localities, and public and private organizations 
through formula, block, and discretionary grant programs.[Footnote 64] 
The office also provides training and technical assistance, produces 
and distributes publications and other products containing information 
about juvenile justice topics, and funds research and evaluation 
efforts. Table 6 shows funding by fiscal year from 2007 through 2009 
for the appropriation accounts for juvenile justice programs. 

Table 6: Juvenile Justice Enacted Appropriations, Fiscal Years 2007 
through 2009 (Dollars in thousands): 

Funding by fiscal year: 

Line item: Part A - Concentration of Federal Efforts[B]; 
2007: $703; 
2008: $658; 
2009[A]: 0. 

Line item: Part B - State Formula Grants; 
2007: $78,978; 
2008: $74,260; 
2009[A]: $75,000. 

Line item: Part D--Research, Evaluation, Technical Assistance, and 
Training; 
2007: 0; 
2008: 0; 
2009[A]: 0. 

Line item: Part E - Developing, Testing, and Demonstrating Promising 
New Initiatives and Projects; 
2007: $104,674; 
2008: $93,835; 
2009[A]: $82,000. 

Youth Mentoring Grants; 
2007: $9,872; 
2008: $70,000; 
2009[A]: $80,000. 

Line item: Title V - Local Delinquency Prevention Incentive Grants[C]; 
2007: $64,171; 
2008: $61,100; 
2009[A]: $62,000; 

Line item: Title V - Local Delinquency Prevention Incentive Grants[C]; 
Tribal Youth; 
2007: ($9,872); 
2008: ($24,680); 
2009[A]: ($24,680); 

Line item: Title V - Local Delinquency Prevention Incentive Grants[C]; 
Gang Prevention; 
2007: (v14,100); 
2008: ($18,800); 
2009[A]: ($25,000); 

Line item: Title V - Local Delinquency Prevention Incentive Grants[C]; 
Alcohol Prevention; 
2007: ($25,000); 
2008: ($10,000); 
2009[A]: ($25,000). 

Line item: Project Childsafe[D]; 
2007: $987; 
2008: 0; 
2009[A]: 0. 

Line item: Secure Our Schools[E]; 
2007: $14,808; 
2008: $15,040; 
2009[A]: 0. 

Line item: Victims of Crime Act--Improving Investigation and 
Prosecution of Child Abuse Program[F]; 
2007: $14,808; 
2008: $16,920; 
2009[A]: $20,000. 

Line item: Juvenile Accountability Block Grant Program[G]; 
2007: $49,361; 
2008: $51,700; 
2009[A]: $55,000. 

Total: 
2007: $338,362; 
2008: $383,513; 
2009[A]: $374,000. 

Source: OJJDP funding data. 

Note: According to OJJDP, the office’s overall budget also includes 
funds that are transferred from other appropriations accounts; for 
example, Juvenile Drug Court programs are administered by OJJDP, but 
the funding for these programs is provided by the Bureau of Justice 
Assistance—a component of the Department of Justice that in part 
provides funding, training, and technical assistance in support of 
national, state, and local efforts to prevent crime, drug abuse, and 
violence. 

[A] Not included in this table is the American Recovery and 
Reinvestment Act of 2009, Pub. L. No. 111-5, 123 Stat. 115, 130, which 
provided an additional $225 million under the State and Local Law 
Enforcement Assistance account that the conference report directed be 
used over 2 years for competitive, peer-reviewed grants to units of 
state, local, and tribal governments and to national, regional, and 
local nonprofit organizations to support critical nurturing and 
mentoring of juveniles, among other things. H.R. Rep. No. 111-16, at 
419. 

[B] According to the Office of Justice Program’s fiscal year 2010 
congressional budget submission, the Concentration of Federal Efforts 
program promotes interagency cooperation and coordination among federal 
agencies with responsibilities in the area of juvenile justice, as 
authorized by part A of the Juvenile Justice and Delinquency Prevention 
Act, as amended. 

[C] These three amounts are carved out from the Title V appropriations 
account in accordance with the annual appropriations acts. The 
remaining funds are used for Title V delinquency prevention programs 
for juveniles who have had contact with the juvenile justice system, as 
well as nonoffenders—juveniles who have not yet had contact with the 
juvenile justice system but are in need of preventive services. 

[D] Project Childsafe is a nationwide program to promote safe firearms 
handling and storage practices through the distribution of safety 
education messages and free gun-locking devices. 

[E] The Secure Our Schools program provides discretionary grants to 
states, local governments, and Indian tribes to provide improved 
security, such as placement and use of metal detectors, at schools and 
on school grounds. 

[F] The Victims of Crime Act—Improving Investigation and Prosecution of 
Child Abuse program provides training and technical assistance to 
professionals involved in investigating, prosecuting, and treating 
issues related to child abuse. 

[G] Under the Juvenile Accountability Block Grant Program, OJJDP 
provides funds to states and units of local government to strengthen 
the juvenile justice system. These funds can be used for 17 different 
purpose areas, including establishing programs to help the successful 
reentry of juvenile offenders from state and local custody in the 
community or for hiring or training programs for detention and 
corrections personnel. 

[End of table] 

[End of section] 

Appendix IV: Juvenile Reentry and Substance Abuse Programs OJJDP Funded 
through Discretionary Grants: 

From fiscal years 2007 through 2009, OJJDP allocated approximately $33 
million through discretionary grants to four juvenile reentry grant 
programs and three juvenile substance abuse programs. See table 7 for a 
description of these reentry and substance abuse programs and the 
amounts OJJDP awarded to grantees. 

Table 7: OJJDP Juvenile Reentry and Substance Abuse Programs Funded 
through Discretionary Grants, Fiscal Years 2007 through 2009: 

Reentry programs: 

Grant program: High-Risk Youth Offender Reentry and Family 
Strengthening Initiatives; 
Description: The purpose of this program is to facilitate the 
successful transition of juvenile offenders to their families and 
communities following confinement in a juvenile residential facility. 
In 2007, OJJDP awarded a total of $15 million to 19 grantees to develop 
programs and strategies aimed at helping juvenile offenders who pose 
significant public safety risks stay out of the juvenile justice 
system, for example, by improving their family relationships. 

Grant program: Second Chance Juvenile Mentoring Initiative; 
Description: The purpose of this program is to support the successful 
and safe transition of juvenile offenders from residential facilities 
to their communities. In 2009, OJJDP awarded a total of $4.7 million to 
11 grantees to develop, implement, and expand mentoring programs and 
transitional services that specifically match juveniles with mentors 
during the juveniles' confinement to facilitate successful community 
reintegration and reduce recidivism. 

Grant program: Second Chance Act Youth Offender Reentry Initiative; 
Description: The purpose of this program is to assist jurisdictions 
characterized by large numbers of juvenile offenders returning to their 
communities after release from residential facilities and reduce the 
rate of recidivism for these juvenile offenders. In 2009, OJJDP awarded 
a total of nearly $3.7 million to five grantees to develop projects 
aimed at providing juvenile offenders with services, including 
vocational and job placement services, substance abuse treatment, 
family strengthening practices, and mentors who work with juvenile 
offenders during confinement and after reentry into the community. 

Grant program: Tribal Juvenile Detention and Reentry Green 
Demonstration Program; 
Description: The purpose of this program is to provide federally 
recognized tribes with the funds to support program services for tribal 
juveniles residing in, or soon to be released from, tribal juvenile 
detention centers. In 2009, OJJDP awarded a total of $2 million to 
three grantees to develop programs that provide services, such as needs 
assessments, vocational training, substance abuse treatment, family 
strengthening, and aftercare to help successfully reintegrate the 
juveniles into the tribal community. 

Substance abuse programs: 

Grant program: Juvenile Drug Court/Reclaiming Futures Program[A]; 
Description: The purpose of this program is to build the capacity of 
states, state courts, local courts, units of local government, and 
Indian tribal governments to develop and establish juvenile drug courts 
adopting the Reclaiming Futures model. This model embodies three 
essential elements for juvenile offenders who are abusing substances: 
(1) designing a system to coordinate services, (2) involving the 
community in creating new opportunities, and (3) improving treatment 
services for drug and alcohol abuse. In 2007, OJJDP awarded nearly $1.3 
million to three grantees for programs that applied the Reclaiming 
Futures model to their juvenile drug courts by helping youth meet 
educational goals, identifying juveniles requiring substance abuse 
treatment, and effectively engaging youth in treatment by increasing 
the number and range of available options. 

Grant program: Substance Abuse Prevention and Intervention Programs; 
Description: The purpose of these programs is to enhance juvenile 
justice, child protection, and delinquency prevention by funding 
substance abuse programs or strategies that focus on change at the 
individual, family, and community levels. In 2007, OJJDP awarded $4.7 
million to nine grantees for programs that aim to prevent or reduce 
juvenile delinquency associated with substance abuse problems by 
offering a range of services, from community-based interventions, such 
as academic assistance and job skills training, to individual treatment 
services, such as counseling or therapy. These programs incorporate the 
use of prevention and intervention programs that use innovative 
approaches or that have been evaluated and have been shown to be 
effective, such as wraparound/case management or family therapy, as 
well as those that target both substance-abusing juveniles and their 
families. 

Grant program: Brief Interventions and Referrals to Treatment for 
Juvenile Courts and Juvenile Drug Courts[A]; 
Description: The purpose of this program is to increase the provision 
and effectiveness of outpatient treatment for adolescents with 
substance abuse disorders by funding juvenile courts and juvenile drug 
courts to adopt and expand a combination of two interventions for 
adolescents in outpatient treatment settings. These interventions are 
(1) motivational enhancement therapy, an intervention program designed 
to change behavior by helping clients understand their ambivalence and 
achieve lasting changes for a range of problematic behaviors, and (2) 
cognitive behavioral therapy. In 2008, OJJDP awarded nearly $1.6 
million to three grantees for juvenile drug court programs that adopt 
and expand the use of these two therapies. 

Source: GAO analysis of OJJDP grant programs and funding data. 

[A] According to OJJDP, funding for the Juvenile Drug Court/Reclaiming 
Futures Program and Brief Interventions and Referrals to Treatment for 
Courts and Juvenile Drug Courts is provided by the Bureau of Justice 
Assistance but administered by OJJDP. In addition, OJJDP has funded 
family drug court programs and training and technical assistance 
efforts for drug courts. 

[End of table] 

[End of section] 

Appendix V: Experts' Opinions of Reentry Programs That Lack Conclusive 
Evidence of Effectiveness: 

Experts Provided Positive Views about Wraparound/Case Management 
Programs, but Could Not Cite Evidence from Available Research 
Indicating That They Are Effective at Reducing Recidivism: 

Of the 19 reentry experts we interviewed, 9 had specific experience or 
knowledge related to wraparound/case management and 8 experts had 
positive comments about the effectiveness of this program.[Footnote 65] 
In general, wraparound/case management interventions involve making an 
array of individualized services and support networks available to 
juveniles, rather than requiring them to enroll in treatment programs 
that may not address individual needs. According to OJJDP, the goal of 
wraparound/case management programs is to keep delinquent juveniles at 
home and out of institutions whenever possible. The basic elements that 
constitute a wraparound program include, among other things, (1) a 
collaborative, community-based interagency team responsible for 
designing, implementing, and overseeing the intervention program in a 
given jurisdiction; (2) care coordinators who are responsible for 
helping juveniles create customized treatment programs, among other 
things; (3) juvenile and family teams consisting of family members and 
community members who work together to ensure the juvenile's needs are 
met at home, at school, and in the community; and (4) a plan of care 
developed and updated by all members of the juvenile and family teams 
that identifies the juvenile's strengths and weaknesses, targets 
specific goals such as improved performance in school, and outlines how 
to achieve them. 

Of these nine experts, eight provided positive opinions of the results 
of wraparound/case management intervention programs. For example, an 
expert commented about how in one specific wraparound intervention 
program, a single case manager is assigned to a juvenile and is 
responsible for determining the services the juvenile is to receive 
based on his or her specific needs, instead of enrolling the juvenile 
into a treatment program that may not be as beneficial for the 
juvenile. Two of these eight experts noted that there was a lack of 
evaluations demonstrating effectiveness of these intervention programs 
but pointed us to a study[Footnote 66] on a specific wraparound/case 
management intervention program, Wraparound Milwaukee, that showed 
potentially promising results related to a reduction in recidivism 
rates for juvenile offenders.[Footnote 67] However, these experts 
stressed that this study alone did not conclusively demonstrate the 
effectiveness of wraparound/case management programs. The ninth expert 
stated that in her experience, wraparound/case management interventions 
are not effective because, for example, a juvenile is placed into this 
intervention program based on the availability of program staff and 
resources rather than program services being tailored to the individual 
needs of the juvenile. Additionally, three of the nine experts 
cautioned that these intervention programs are difficult to implement 
because of such issues as a lack of quality services or low retention 
of juveniles and their families in the intervention being provided. 
Specifically, one of these experts noted that the quality of wraparound 
services can vary depending on a community's resources. In addition, 
another expert emphasized the importance of obtaining buy-in from 
diverse service providers who may be used to working on their own, such 
as within the welfare, foster care, and public school systems. 

Reentry Experts Cited Lack of Conclusive Evidence Based on Available 
Research and Mixed Views of the Effectiveness of Aftercare Programs at 
Reducing Recidivism: 

Of 19 reentry experts we interviewed, 15 had specific experience or 
knowledge related to aftercare programs and 6 cited a lack of 
conclusive evidence of effectiveness of the program type. Aftercare 
intervention programs are intended to prepare juvenile offenders to 
return to the community during the reentry process by focusing on the 
delivery of services and supervision that start while juveniles are 
incarcerated and continue after they return to their communities. 
Specifically, aftercare programs collaborate with the community and 
marshal its resources to help ensure that juvenile offenders receive 
services that address their individual needs, such as treatment for a 
substance abuse problem. These intervention programs focus on changing 
individual behavior thereby preventing further delinquency. For 
example, an aftercare program might incorporate the use of techniques 
from an intervention therapy, such as motivational enhancement therapy, 
to engage juvenile offenders in treatment and increase their commitment 
to change. 

Of these 15 experts, 7 offered positive opinions regarding aftercare 
intervention programs, based on their own experience or knowledge of 
the intervention programs. For example, 1 expert noted that if 
aftercare included intervention programs that were proven to be 
effective, used assessment tools that identified the individual needs 
of the juvenile, and implemented the therapies as they were designed, 
then the aftercare intervention program should be effective at reducing 
recidivism rates. Although these experts could not provide examples of 
studies that had been conducted to show evidence of the effectiveness 
of the intervention programs, all 7 of them agreed based on their own 
experience or knowledge that aftercare interventions are important 
reentry programs, in part, because they link the juvenile with his or 
her community and provide regular contact with a caseworker. 
Additionally, 3 of these 7 experts stated that aftercare could be 
effective depending on the intervention programs used and if they were 
delivered as intended. For example, they said that if aftercare 
includes intervention programs that have proven to be effective, such 
as cognitive behavioral therapy, and identifies the individual needs of 
the juvenile, the programs can reduce recidivism. However, 6 of the 15 
reentry experts said there was inconclusive evidence to determine 
whether these programs can be effective in achieving results. Three of 
these experts based their opinions on an evaluation of the Intensive 
Aftercare Program that showed inconclusive results about program 
effectiveness. Specifically, the study[Footnote 68] found no evidence 
that the program had its intended impact of reducing recidivism among 
juveniles who were released back into the community under supervision 
in the three states that piloted the program.[Footnote 69] However, the 
evaluation did find that the three states that implemented the 
Intensive Aftercare Program model did successfully incorporate most of 
its core features, which prepared juveniles to transition back into the 
community. For instance, these states created new Intensive Aftercare 
Program-specific treatment programs that among other things, prepared 
juveniles for increased responsibility in the community, facilitated 
interaction with the community, and worked with the juveniles' schools 
and families. The state programs had a large percentage of juveniles 
involved in various treatment services. Despite the inconclusive 
results of the study, one expert credited aftercare programs with 
addressing the issue of juveniles having to deal with different 
probation officers throughout the reentry process because, in general, 
aftercare programs assign one probation officer to a juvenile as a 
consistent point of contact. The evaluation also stated that in order 
for the general aftercare model to be effective, it must not only 
provide supervision and services after a juvenile's release into the 
community, but also focus on preparing a juvenile for release. The 
remaining 2 experts opined that aftercare intervention programs had not 
been shown to be effective at achieving desired results because, for 
example, the treatment a juvenile receives depends on what services are 
actually available in the community. 

Experts Indicated Potential Positive Outcomes for Vocational or Job 
Training Programs: 

Of the 19 reentry experts we interviewed, 11 had specific experience or 
knowledge related to vocational or job training programs and indicated 
potential positive outcomes for these programs. According to OJJDP, 
providing juveniles with employment opportunities during reentry is a 
common strategy used to try to reduce future criminal behavior. 
Vocational or job training intervention programs are intended to 
improve juveniles' social and educational functioning by, for example, 
increasing earnings, raising self-esteem, and instilling a positive 
work ethic. Juveniles can participate in vocational/job training 
intervention programs while they are incarcerated and after they return 
to the community. 

Of the 11 reentry experts, 10 of them had positive comments based on 
their experience or knowledge of the program type. Specifically, they 
said that vocational/job training programs were potentially beneficial, 
in part, if they were applied to older juveniles and if they led to 
those juveniles getting jobs. The remaining expert said that there is 
little evidence to demonstrate the effectiveness of these intervention 
programs. 

Experts Cited No Evidence for Effectiveness of Reentry Courts: 

Reentry courts are specialized courts that manage the return of 
juvenile offenders to the community after they are released from 
residential facilities.[Footnote 70] The court manages reentry by using 
its authority to direct resources to support the offender's return to 
the community and promote positive behavior, among other things. For 
example, a reentry court would oversee a juvenile's release into the 
community by assigning a judge to meet with the juvenile once a month. 
The judge would actively engage the supervising authority, such as a 
parole officer, in assessing the juvenile's progress. The judge would 
also oversee sanctions for violations as well as rewards, like early 
release from parole, for successful achievement of goals, such as 
successfully completing a cognitive behavioral therapy intervention 
program. 

Of the 19 reentry experts we interviewed, 2 provided comments related 
to reentry courts and had differing opinions on their effectiveness. 
One had a negative impression of the courts, stating that the reentry 
courts do not provide more to a juvenile than a probation officer 
would. The other commented that he considers concepts encompassed in 
reentry courts, such as intensity of supervision, to be a best practice 
when it comes to reentry programs. However, neither was aware of any 
evaluations of these types of courts.[Footnote 71] 

[End of section] 

Appendix VI: Experts' Opinions of Substance Abuse Programs That Lack 
Conclusive Evidence of Effectiveness: 

Experts Views Were Divided on the Effectiveness of Drug Courts, with 
Half Indicating That Additional Evidence Is Needed to Determine 
Effectiveness: 

Of the 13 substance abuse experts we interviewed, 10 had specific 
experience or knowledge related to drug courts that resulted in mixed 
views of the effectiveness of this program type.[Footnote 72] Juvenile 
drug courts are specialized courts established within and supervised by 
juvenile courts to provide intervention programs, such as cognitive 
behavioral therapy or family therapy, for substance-abusing juveniles 
and their families. Juvenile offenders assigned to drug courts are 
identified by a juvenile court as having problems with alcohol or 
drugs. The drug court judge maintains close oversight of each case 
through frequent--often weekly--status hearings with the individuals 
involved. The judge both leads and works as a member of a team that can 
comprise representatives from juvenile justice, social services, school 
and vocational training programs, law enforcement, probation, the 
prosecution, and the defense. Together, the team determines how best to 
address the substance abuse and related problems of the juvenile and 
his or her family. 

Specifically, of these 10 experts, 5 experts described drug courts as 
having insufficient evidence to determine program effectiveness. For 
example, 2 experts mentioned that while some studies show drug courts 
reducing substance abuse while juveniles were under court supervision, 
the results did not last after juveniles were no longer being 
supervised by the courts. Another expert stated that since drug courts 
tend to be used for juveniles who have their first or second contact 
with the juvenile justice system, they are ineffective at achieving 
desired results because they expose these first-time offenders to peers 
who have more serious substance abuse addictions and therefore might 
influence them to continue to abuse substances. By contrast, the 
remaining 4 experts stated that drug courts can be effective at 
achieving desired results such as reducing substance abuse if, for 
example, the juvenile is sent to a community where there are 
intervention programs offered that have been evaluated and have been 
shown to be effective, such as cognitive behavioral therapy or family 
therapy intervention programs. One expert cited a study to support the 
opinion that drug courts supplemented with multisystemic therapy 
[Footnote 73] resulted in a decrease in substance abuse by juvenile 
offenders.[Footnote 74] 

Experts Report Mixed Views on the Effectiveness of Mentoring Programs: 

Of the 13 substance abuse experts we interviewed, 8 had specific 
experience or knowledge related to mentoring intervention programs that 
resulted in mixed views of their effectiveness. Mentoring programs 
consist of a relationship between two or more people over a prolonged 
period of time, where an older, more experienced individual provides 
support and guidance to a juvenile. The goal of mentoring is for the 
juvenile to develop positive adult contact, thereby reducing risk 
factors, such as exposure to juveniles who use substances, while 
increasing positive factors, such as encouragement for abstaining from 
substance use. In the substance abuse field, juveniles in need of 
sobriety are teamed with older sponsors to serve as positive role 
models in helping them become sober. 

Of these eight experts, four stated that mentoring programs are 
ineffective or unsuccessful at achieving desired results, such as 
reducing substance abuse, and that these intervention programs are more 
effective at preventing at-risk juveniles from engaging in delinquent 
behavior.[Footnote 75] Also, one expert stated that there have been too 
few evaluations conducted on mentoring programs to make a general 
statement about the relative benefits of mentoring. Conversely, three 
experts stated that mentoring programs are effective or can be 
effective if, for example, mentors are trained or if mentoring is 
combined with another intervention program that has been evaluated and 
has been shown to be effective, such as multisystemic therapy. 

Experts Provided Positive Views about the Potential Results of 
Wraparound/Case Management, but Cited Limited Evidence of Its 
Effectiveness: 

Of the 13 substance abuse experts we interviewed, 11 had specific 
experience or knowledge related to wraparound/case management 
intervention programs that resulted in mixed views of the program type. 
Of these 11 experts, 7 stated that wraparound/case management is 
effective or can be effective if, for example, it is combined with 
another intervention program that has been evaluated and has shown to 
be effective, such as multisystemic therapy. Although these experts had 
limited evidence demonstrating the effectiveness of wraparound/case 
management, 2 experts cited two studies that show potentially promising 
results related to a reduction in recidivism.[Footnote 76] For example, 
one study showed that juveniles in wraparound/case management receive a 
number of individualized services, such as mental health treatment for 
those juveniles who struggle with emotional issues. However, this study 
stressed that it is difficult to evaluate wraparound/case management in 
a controlled way since treatment plans are individualized for each 
juvenile. The other 4 experts stated that wraparound/case management 
intervention programs are ineffective because, for example, the 
intervention programs lack follow-through as there are no consequences 
if a juvenile does not show up for treatment, or there is not yet 
sufficient evidence to determine their effectiveness. 

[End of section] 

Appendix VII: Additional OJJDP Efforts to Disseminate Information about 
Effective Juvenile Justice Programs: 

In addition to the National Training and Technical Assistance Center 
and the Model Programs Guide, OJJDP disseminates information about 
effective programs through a variety of other efforts. Specifically, 
the office has developed mechanisms to disseminate information related 
to effective programs in specific issue areas, such as youth gang 
activity, disproportionate minority contact, and girls' delinquency, as 
described in table 8. 

Table 8: Additional OJJDP Efforts to Disseminate Information on 
Effective Programs: 

Effort: National Youth Gang Center; 
Description: OJJDP established the National Youth Gang Center in 1995 
to assist policymakers, practitioners, and researchers in their efforts 
to reduce youth gang involvement and crime by contributing information, 
resources, practical tools, and expertise toward the development and 
implementation of effective gang prevention, intervention, and 
suppression strategies. For example, the center continually identifies 
and reviews current gang literature and uses an automated database to 
compile gang-related legislation. The center also conducts assessments 
of the scope and characteristics of youth gang activity in the United 
States; develops and disseminates resources for practitioners and 
communities; and provides training and technical assistance in support 
of community-based prevention, intervention, and suppression efforts. 
For example, the center annually collects and analyzes gang-related 
data from law enforcement agencies across the nation. 

Effort: Disproportionate Minority Contact Reduction Database; 
Description: OJJDP established the Disproportionate Minority Contact 
Reduction Database in 2006 to address the issue of a disproportionate 
number of minorities coming into contact with the juvenile justice 
system. The database is a searchable database that assists 
jurisdictions in identifying effective programs that may prove useful 
to address disproportionate minority contact. The database provides 
guidelines for the juvenile justice field to assist practitioners in 
the field in choosing strategies and developing intervention plans. 

Effort: Girls Study Group; 
Description: OJJDP established the Girls Study Group in 2004 to further 
the juvenile justice field's understanding of female juvenile offending 
and to identify effective strategies for preventing and reducing female 
juvenile involvement in delinquency and violence. The Girls Study Group 
conducted a literature search and reviewed 61 girls' delinquency 
programs to identify risk and protective factors for girls' delinquency 
and to identify effective programs or strategies for preventing and 
reducing girls' delinquency. In July 2009, we reported on the findings 
of the Girls Study Group and OJJDP's efforts related to addressing the 
group's findings. Among other things, we reported that the study group 
found no effective programs for preventing or reducing girls' 
delinquency, and we recommended that OJJDP develop and implement a plan 
for responding to the findings of the Study Group.[A]. 

Effort: Underage Drinking Enforcement Training Center; 
Description: OJJDP established the Underage Drinking Enforcement 
Training Center in 1999 to build leadership capacity and increase the 
effectiveness of states and local communities in their efforts to 
enforce underage drinking laws, prevent underage drinking, and 
eliminate the consequences associated with alcohol use by underage 
youth. The center works to achieve this goal by providing a wide 
variety of practical training and technical assistance services, such 
as electronic seminars where national experts, researchers, and 
representatives from OJJDP give presentations on a specific areas of 
interest, for example, school substance abuse policies or the effects 
of alcohol on adolescent brain development. 

Source: GAO analysis of OJJDP information. 

[A] GAO, Juvenile Justice: Technical Assistance and Better Defined 
Evaluation Plans Will Help to Improve Girls’ Delinquency Programs, 
[hyperlink, http://www.gao.gov/products/GAO-09-721R] (Washington, D.C.: 
July 24, 2009). 

[End of table] 

[End of section] 

Appendix VIII: Comments from the Department of Justice: 

U.S Department of Justice: 
Office of Justice Programs: 
Office of the Assistant Attorney General: 
Washington, D.C. 20531: 

December 3, 2009: 

Ms. Eileen R. Larence: 
Director, Homeland Security and Justice Issues: 
Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Larence: 

Thank you for the opportunity to comment on the draft Government 
Accountability Office (GAO) report entitled "Juvenile Justice: DOJ Is 
Enhancing Information on Effective Programs, but Could Better Assess 
the Utility of This Information" (GAO-10-125). The Office of Justice 
Programs agrees with the Recommendation for Executive Action, which is 
restated in bold text below and is followed by our response. 

To help ensure that OJJDP's Model Programs Guide is regularly meeting 
user needs and providing the most helpful information on effective 
programs, consistent with federal guidelines, we recommend the 
Administrator of OJJDP develop a cost-effective mechanism for regularly 
soliciting and incorporating feedback from the juvenile justice field 
on the usefulness of the information provided in its Model Programs 
Guide. 

By March 31, 2010, the Office of Juvenile Justice and Delinquency 
Prevention (OJJDP) will develop and implement an ongoing mechanism to 
solicit regular feedback from the juvenile justice field on the 
usefulness of the information provided in its Model Programs Guide. 
OJJDP's goal will be to implement a process which ensures that its 
Model Programs Guide is responsive to the needs of users, accurately 
represents the featured programs, and promotes the use of evidence-
based program of practices in juvenile justice. 

If you have any questions regarding this response, you or your staff 
may contact Maureen A. Henneberg, Director, Office of Audit, 
Assessment, and Management, on (202) 616-3282. 

Sincerely, 

Signed by: 
Laurie O. Robinson: 
Assistant Attorney General: 

cc: 

Beth McGarry: 
Deputy Assistant Attorney for Operations and Management: 

Jeffrey Slowikowski: 
Acting Administrator Office of Juvenile Justice and Delinquency 
Prevention: 

Maureen Henneberg: 
Director: 
Office of Audit, Assessment, and Management: 

Richard P. Theis: 
Audit Liaison: 
Department of Justice: 

[End of section] 

Appendix IX: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Eileen R. Larence, (202) 512-8777 or larencee@gao.gov: 

Acknowledgments: 

In addition to the contact named above, Mary Catherine Hult, Assistant 
Director; David Alexander; Elizabeth Blair; Ben Bolitzer; Carissa 
Bryant; Katherine Davis; Sean DeBlieck; Allyson Goldstein; Rebecca 
Guerrero; Jared Hermalin; Dawn Locke; Lisa Shibata; Janet Temko; and 
Delia Zee made key contributions to this report. 

[End of section] 

Footnotes: 

[1] A juvenile offender is a youth under the age of 18 who is too young 
to be tried as an adult, and who has been found to have committed an 
offense that would be criminal if committed by an adult. 

[2] Residential facilities house juveniles who are awaiting 
adjudication (i.e., resolution of a case by a judge) or have been 
adjudicated for an offense, or juveniles who are removed from their 
homes because of nondelinquency issues, such as being victims of child 
abuse. 

[3] This number is based on data from the Census of Juveniles in 
Residential Placement (the Census), which takes 1-day population counts 
of juveniles placed in residential facilities across states. The number 
of juveniles placed in residential facilities ranged from about 108,000 
in 1999 to around 93,000 in 2006, based on the most recently conducted 
Census. The Census does not capture data on juveniles held in adult 
prisons or jails nor does it include facilities exclusively intended 
for drug or mental health treatment, even though such facilities may 
house juvenile offenders. 

[4] This estimate, the most recent available as reported in the Office 
of Juvenile Justice and Delinquency Prevention's (OJJDP) 2006 National 
Report, is derived from the 2003 Survey of Youth in Residential 
Placement, which consists of anonymous interviews with a large, 
nationally representative sample of juvenile offenders placed, or 
committed, in residential facilities. The universe for this survey is 
the population of juvenile offenders ages 10 to 20 in all facilities 
included in the 2003 Census of Juveniles in Residential Placement. 
There is no national recidivism rate for juveniles since juvenile 
justice systems vary across states. Department of Justice, Office of 
Juvenile Justice and Delinquency Prevention, Juvenile Offenders and 
Victims: 2006 National Report (Washington, D.C., 2006). 

[5] The Federal Advisory Committee on Juvenile Justice is an advisory 
body established by the Juvenile Justice and Delinquency Prevention Act 
of 1974, as amended, to, among other things, advise the President and 
Congress on state perspectives regarding the operation of OJJDP and on 
federal legislation pertaining to juvenile justice and delinquency 
prevention. 42 U.S.C. § 5633(f). The committee consists of appointed 
representatives from advisory groups in each of the 50 states, the 
District of Columbia, and the 5 U.S. territories. As part of its 
mandated responsibilities, the committee submits two annual reports-- 
one to the President and Congress and the second to the OJJDP 
Administrator. These reports are informed by data gathered through an 
annual request for information from the individual state and territory 
advisory groups. Specifically, 47 of 55 states and territories 
(responses were only requested from 4 of the 5 U.S. territories) 
responded to the 2008 request. 

[6] The most frequently cited issue was mental health assessment and 
treatment (38 states). Respondents cited concerns about the lack of 
resources and funding available for mental health programs, the minimal 
number of treatment services available, the minimal number of trained 
staff operating programs, and the limited mental health treatment 
options. 

[7] GAO defines an expert as a person who is recognized by others who 
work in the same subject matter area as having knowledge that is 
greater in scope or depth than that of most people working in that 
area. The expert's knowledge can come from education, experience, or 
both. 

[8] Repositories and Web sites we identified include OJJDP's Model 
Programs Guide, Substance Abuse and Mental Health Service 
Administration's National Registry of Evidence-based Programs and 
Practices, University of Colorado's Blueprints for Violence Prevention, 
and the National Institute of Justice. 

[9] We used program types outlined in the Model Programs Guide because 
it describes a range of potential program types that juvenile offenders 
may encounter in the juvenile justice system. Specifically, the Model 
Programs Guide is a database that in part is designed to provide 
information on individual intervention programs within these program 
types. The Model Programs Guide categorizes programs by the type of 
program being administered (e.g., vocational training or family 
therapy), the stage at which the program is administered (e.g., 
prevention, during incarceration, or after return to the community), 
and the issue area the program addresses (e.g., gangs or substance 
abuse). 

[10] According to the Substance Abuse and Mental Health Services 
Administration, relapse prevention is a variety of interventions 
designed to teach individuals who are trying to maintain behavior 
changes how to anticipate and cope with the problem of relapse. 

[11] For the purposes of this review, we define an intervention program 
as a specific activity, treatment, therapy, or service funded at the 
local, state, or federal level that is intended to address the reasons 
behind a juvenile's delinquent behavior and prevent the juvenile from 
committing increasingly serious offenses. 

[12] Department of Justice, Office of Justice Programs, OJP Information 
Quality Guidelines (Washington, D.C.: 2007). GAO, U.S. Public 
Diplomacy: Actions Needed to Improve Strategic Use and Coordination of 
Research, [hyperlink, http://www.gao.gov/products/GAO-07-904] 
(Washington, D.C.: July 18, 2007), and Transportation Research: 
Opportunities for Improving the Oversight of DOT's Research Programs 
and User Satisfaction with Transportation Statistics, [hyperlink, 
http://www.gao.gov/products/GAO-06-917] (Washington, D.C.: Aug. 15, 
2006). 

[13] Program management standards we reviewed are reflected in the 
Project Management Institute's The Standard for Program Management © 
(2006). Also, see GAO, South Florida Ecosystem Restoration: A Strategic 
Plan and a Process to Resolve Conflicts Are Needed to Keep the Effort 
on Track, [hyperlink, http://www.gao.gov/products/GAO/T-RCED-99-170] 
(Washington, D.C.: Apr. 29, 1999); South Florida Ecosystem Restoration: 
Substantial Progress Made in Developing a Strategic Plan, but Actions 
Still Needed, [hyperlink, http://www.gao.gov/products/GAO-01-361] 
(Washington, D.C.: Mar. 27, 2001); Great Lakes: A Coordinated Strategic 
Plan and Monitoring System Are Needed to Achieve Restoration Goals, 
[hyperlink, http://www.gao.gov/products/GAO-03-999T] (Washington, D.C.: 
July 16, 2003); and [hyperlink, 
http://www.gao.gov/products/GAO-06-917], 7, 11. 

[14] The mission of the Center for Substance Abuse Treatment of the 
Substance Abuse and Mental Health Services Administration, within the 
Department of Health and Human Services, is to promote the quality and 
availability of community-based substance abuse treatment services for 
individuals and families who need them. The center works with states 
and community-based groups to improve and expand existing substance 
abuse treatment services. 

[15] 42 U.S.C. § 5611. 

[16] In general, formula and block grant awards provide funds to states 
in accordance with statutory requirements. OJJDP allocates some formula 
and block grants to states on the basis of states' juvenile 
populations, while others may be awarded at a fixed level to all 
states. Discretionary grants provide funds to states, units of local 
government, and organizations to administer programs. OJJDP awards 
discretionary grants to recipients through an application process or 
based on congressional direction. The term "state" means any state of 
the United States, the District of Columbia, the Commonwealth of Puerto 
Rico, the Virgin Islands, Guam, American Samoa, and the Commonwealth of 
the Northern Mariana Islands. 42 U.S.C. § 5603. 

[17] In addition to the four program types discussed, we also reviewed 
reentry courts. However, because only 2 of 19 experts provided comments 
related to reentry courts, we included their comments in app. V. 

[18] Twelve out of 19 reentry experts provided specific comments on 
cognitive behavioral therapy as a program type. Not all of the reentry 
experts provided comments on each program type as their comments were 
based on their particular area(s) of expertise. 

[19] Cognitive therapy concentrates on thoughts, assumptions, and 
beliefs and encourages the recognition and change of thoughts that 
contribute to problem behaviors, such as violence, criminal activity, 
substance use, or school-based problem behaviors. For instance when a 
student has trouble completing a math problem and automatically thinks, 
"I'm stupid, I'm not a good student, I can't do math," the student is 
encouraged to replace the negative thoughts with more realistic 
thoughts, such as "this problem is difficult, I'll ask for help." 

[20] Behavioral therapy concentrates on specific actions and 
environments that either change or maintain behaviors. For instance, 
when people try to stop smoking they are often encouraged to change 
their daily habits--for example, instead of having their daily coffee 
upon waking, which may trigger the urge to have a cigarette, they are 
encouraged to take a morning walk. 

[21] A meta-analysis is a study that is a systematic synthesis of 
quantitative research results. In other words, a meta-analysis of a 
substance abuse or reentry intervention program involves conducting a 
literature search for all studies that have been conducted on that 
program, evaluating the methodological quality of those studies, and 
then systematically combining the findings of the studies using 
statistical procedures to calculate the overall effect the program has 
on the various outcomes, such as recidivism and substance abuse. 

[22] (1) N.A. Landenberger and M. W. Lipsey, "The Positive Effects of 
Cognitive-Behavioral Programs for Offenders: A Meta-Analysis of Factors 
Associated with Effective Treatment," Journal of Experimental 
Criminology, vol. 1 (2005). (2) M.W. Lipsey, "The Primary Factors that 
Characterize Effective Interventions with Juvenile Offenders: A Meta- 
Analytic Overview," Victims and Offenders, vol. 4 (2009). 

[23] Skill-building programs assist juveniles in developing skills to 
control their behavior or enhance their ability to participate in 
society in a positive manner. 

[24] Aggression replacement training® is an intervention program 
designed to alter the behavior of chronically aggressive juveniles, 
such as those who have continual problems managing their anger. The 
goal of the program is to improve skill competence, anger control, and 
moral reasoning. 

[25] Bruce Kamradt, Stephen Gilbertson, and Nancy Lynn, "Wraparound 
Milwaukee: Program Description and Evaluation," Outcomes for Children 
and Youth with Emotional and Behavioral Disorders and Their Families, 
ed. Michael H. Epstein et al. (Austin: ProEd, 2005), 307. 

[26] Wraparound Milwaukee is an intervention program for juveniles with 
serious emotional, behavioral, and mental health needs and for their 
families. This intervention program attempts to meet the mental health, 
substance abuse, social service, and other supportive needs of 
juveniles in the Milwaukee community by identifying the specific 
personal, community, and professional services each family needs to 
care for a juvenile with special needs. 

[27] The Intensive Aftercare Program was a program initiative funded by 
OJJDP from 1987 until 2000 that was intended to reduce recidivism among 
juveniles who are released back into the community under supervision. 

[28] Department of Justice, Office of Juvenile Justice and Delinquency 
Prevention, Implementation and Outcome Evaluation of the Intensive 
Aftercare Program (Washington, D.C.: March 2005). 

[29] (1) Steve Aos, Marna Miller, and Elizabeth Drake, Evidence-Based 
Public Policy Options to Reduce Future Prison Construction, Criminal 
Justice Costs, and Crime Rates (Olympia, Wash.: Washington State 
Institute for Public Policy, October 2006). (2) M. Dennis, S. Godley, 
G. Diamond, F. Tims, T. Babor, J. Donaldson, H. Liddle, J. Titus, Y. 
Kaminer, C. Webb, N. Hamilton, and R. Funk, "The Cannabis Youth 
Treatment Study: Main findings from two randomized trials," Journal of 
Substance Abuse Treatment, vol. 27 (2004). (3) Landenberger and Lipsey, 
"Positive Effects of Cognitive-Behavioral Programs," 451. (4) Lipsey, 
"Primary Factors," 124. (5) H. B. Waldron and C. W. Turner, "Evidence- 
Based Psychosocial Treatments for Adolescent Substance Abuse," Journal 
of Clinical Child & Adolescent Psychology, vol. 37, no. 1 (2008). (6) 
H. B. Waldron and Y. Kaminer, "On the Learning Curve: The Emerging 
Evidence Supporting Cognitive-Behavioral Therapies for Adolescent 
Substance Abuse," Addiction, vol. 99 (2004). 

[30] (1) Waldron and Kaminer, "On the Learning Curve: The Emerging 
Evidence Supporting Cognitive-Behavioral Therapies for Adolescent 
Substance Abuse," 93. (2) Waldron and Turner, "Evidence-Based 
Psychosocial Treatments for Adolescent Substance Abuse," 238. (3) 
Dennis et al., "The Cannabis Youth Treatment Study: Main findings from 
two randomized trials," 197. 

[31] Not all 13 of the substance abuse experts provided comments on 
each program type as their comments were based on their particular 
area(s) of expertise. 

[32] (1) Aos et al., Benefits and Costs. (2) Dennis et al., "The 
Cannabis Youth Treatment Study: Main findings from two randomized 
trials," 197. (3) P. Greenwood, "Prevention and Intervention Programs 
for Juvenile Offenders," The Future of Children, vol. 18, no. 2 (2008). 
(4) A. Sheidow and S. Henggeler, "Multisystemic Therapy with Substance 
Using Adolescents: A Synthesis of the Research," Crossing Frontiers 
(Brighton: Pavilion Publishing, 2008). (5) S. Schoenwald, D. Ward, S. 
Henggeler, S. Pickrel, and H. Patel, "Multisystemic Therapy Treatment 
of Substance Abusing or Dependent Adolescent Offenders: Costs of 
Reducing Incarceration, Inpatient, and Residential Placement," Journal 
of Child and Family Studies, vol. 5, no. 4 (1996). (6) L. Stambaugh, S. 
Mustillo, B. Burns, R. Stephens, B. Baxter, D. Edwards, and M. DeKraai, 
"Outcomes From Wraparound and Multisystemic Therapy in a Center for 
Mental Health Services System-of-Care Demonstration Site," Journal of 
Emotional and Behavioral Disorders, vol. 15, no. 3 (2007). (7) S. 
Henggeler, W. Clingempeel, M. Brondino, and S. Pickrel, "Four-Year 
Follow-up of Multisystemic Therapy with Substance-Abusing and Substance-
Dependent Juvenile Offenders," Journal of the American Academy of Child 
and Adolescent Psychiatry, vol. 41, no. 7 (2002). (8) M. Lipsey and F. 
Cullen, "The Effectiveness of Correctional Rehabilitation: A Review of 
Systematic Reviews," The Annual Review of Law and Social Science, vol. 
3 (2007). (9) Waldron and Turner, "Evidence-Based Psychosocial 
Treatments for Adolescent Substance Abuse," 238. 

[33] Functional family therapy is a family therapy intervention program 
that among other things, attempts to reduce and eliminate problem 
behaviors, such as substance abuse, through improving communication 
skills, parenting, problem solving, and conflict management while 
increasing the family's exposure to community resources to prevent a 
reoccurrence of substance use. 

[34] At-risk juveniles are youths who, because of certain 
characteristics or experiences, are statistically more likely than 
other youths to encounter certain problems, such as legal, social, 
financial, educational, emotional, and health problems. 

[35] Experts we interviewed noted that several factors could help a 
program be effective at achieving its goals, such as ensuring that 
juveniles receive programs based on their individual needs and ensuring 
that staff are well trained to correctly implement a specific program. 
See app. II for a summary of these most frequently cited factors. 

[36] The four meta-analyses cited by experts were (1) Steve Aos, Marna 
Miller, and Elizabeth Drake, Evidence-Based Public Policy Options to 
Reduce Future Prison Construction, Criminal Justice Costs, and Crime 
Rates (Olympia, Wash.: Washington State Institute for Public Policy, 
October 2006); (2) Aos et al., Benefits and Costs; (3) Elizabeth Drake, 
Steve Aos, and Marna Miller, "Evidence-Based Public Policy Options to 
Reduce Crime and Criminal Justice Costs: Implication in Washington 
State," Victims and Offenders, vol. 4 (2009); and (4) Damon Jones, 
Brian K. Bumbarger, Mark T. Greenberg, Peter Greenwood, and Sandee 
Kyler, The Economic Return on PCCD's Investment in Research-Based 
Programs: A Cost-Benefit Assessment of Delinquency Prevention in 
Pennsylvania (University Park, Pa.: The Prevention Research Center for 
the Promotion of Human Development, Pennsylvania State University, 
2008). 

[37] C. McDougall, M. Cohen, R. Swaray, and A. Perry, "Benefit-Cost 
Analyses of Sentencing," Campbell Systematic Reviews (March 2008). A 
systematic review is a summary of the results of available research on 
a given topic. The systematic review examined available literature 
about sentencing of both adult and juvenile offenders to determine how 
many cost-benefit analyses of these programs exist and the quality of 
the studies that had been done. Because the systematic review did not 
examine the specific program types in this report, we did not include 
the review's assessments of the costs and benefits of sentencing 
options in this report. 

[38] Multidimensional treatment foster care is an intervention that 
places juveniles with specially trained foster parents instead of in 
residential facilities. 

[39] 42 U.S.C. § 5661(e). 

[40] The Clearinghouse was established in 1979 to provide individuals 
and organizations with access to a comprehensive collection of 
information and resources on juvenile justice topics in a centralized 
location. The Clearinghouse is responsible for coordinating OJJDP's 
mailing list, newsletter, publications and editorial services, library 
and reference services, and Web site maintenance. It also coordinates 
such efforts as the Juvenile Justice Listserv--a listserv that provides 
information on juvenile justice and other youth service-related 
publications, funding opportunities, and events; News-at-a-Glance-- 
OJJDP's bimonthly newsletter; as well as OJJDP's library services, 
among other things. The Clearinghouse is a component of the National 
Criminal Justice Research Service, which offers justice and substance 
abuse information to support research, policy, and program development 
by providing access to published reports, research findings, and other 
information products. 

[41] According to OJJDP, NTTAC is one of many providers that OJJDP uses 
for its training and technical assistance efforts. Whereas NTTAC 
provides information on a wide range of juvenile justice issues, 
OJJDP's other providers deliver specialized training and technical 
assistance on specific topics, such as gang prevention through the 
National Youth Gang Center. 

[42] The Model Programs Guide is one of several online databases that 
provide information about programs that have been evaluated and may 
have been shown to be effective. In general, these online databases 
provide information about specific intervention programs (e.g., 
multisystemic therapy) rather than types of programs (e.g., drug courts 
or mentoring). At the time this report was issued, GAO was in the 
process of issuing another report that indicated that these online 
databases use different criteria for determining whether a program is 
effective. See GAO, Program Evaluation: a Variety of Rigorous Methods 
Can Help Identify Effective Interventions, [hyperlink, 
http://www.gao.gov/products/GAO-10-30] (Washington, D.C.: Nov. 23, 
2009). 

[43] [hyperlink, http://www.gao.gov/products/GAO-07-904] and 
[hyperlink, http://www.gao.gov/products/GAO-06-917]. 

[44] OJP developed the Information Quality Guidelines in accordance 
with Office of Management and Budget (OMB) guidance, which is found in 
Office of Management and Budget, Guidelines for Ensuring and Maximizing 
the Quality, Objectivity, Utility, and Integrity of Information 
Disseminated by Federal Agencies, 67. Fed. Reg. 8,452 (Feb. 22, 2002). 
Utility, as defined by OMB, refers to the usefulness of the information 
to the intended users. 

[45] OJJDP has established three goals that it states constitute the 
major elements of a sound policy for juvenile justice and delinquency 
prevention. In addition to its goal for research and evaluation, 
OJJDP's remaining two goals are (1) to promote delinquency prevention 
and early intervention efforts and (2) to foster the use of community- 
based programs and services for juvenile offenders. 

[46] 42 U.S.C. § 5614(b)(5). 

[47] Under part D, OJJDP is authorized to conduct research, evaluation, 
and technical assistance, among other things. 42 U.S.C. §§ 5661-62. 
Under part E, OJJDP is authorized to make grants for developing, 
testing, and demonstrating promising new initiatives and programs. 42 
U.S.C. §§5665-66. OJJDP has awarded funds under part E pursuant to 
congressional direction for a number of years. For example, the 
explanatory statement accompanying the fiscal year 2008 appropriations 
act directed that funds appropriated for part E programs be provided to 
specific grantees. Explanatory statement accompanying the Consolidated 
Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844 (2007). 
For fiscal year 2009, the appropriations act directed that funds 
appropriated under part E be provided to specific grantees, as 
specified in the explanatory statement. Omnibus Appropriations Act, 
2009, Pub. L. No. 111-8, 123 Stat. 524, 581. 

[48] 74 Fed. Reg. 62,821 (Dec. 1, 2009). OJJDP officials did not 
provide a reason why the office has not published annual program plans 
for fiscal years 2003 through 2009. 

[49] See 42 U.S.C. § 5614(b)(5). 

[50] The Program Assessment Rating Tools aimed to assess and improve 
program performance so that the federal government could achieve better 
results. Office of Management and Budget, "Juvenile Justice Programs," 
2006 Performance and Accountability Report (2006), [hyperlink, 
http://www.whitehouse.gov/omb/expectmore/detail/10003813.2006.html] 
(accessed May 19, 2008). 

[51] The other two recommendations included in OMB's review were to (1) 
make juvenile justice programs' performance results available to the 
public through program publications and the Internet, and (2) include 
performance information in budget submissions to better link resources 
requested to program performance goals. 

[52] Project Management Institute, The Standard for Program Management 
© (2006). 

[53] [hyperlink, http://www.gao.gov/products/GAO-03-999T], [hyperlink, 
http://www.gao.gov/products/GAO-01-361], and [hyperlink, 
http://www.gao.gov/products/GAO/T-RCED-99-170]. 

[54] OJP's Performance Budget shows that OJP has not requested part D 
funding for OJJDP since fiscal year 2007. 

[55] Appropriations statutes for fiscal years 2007 through 2009 
provided that OJJDP may use not more than 10 percent of each amount 
appropriated for research, evaluation, and statistics activities that 
benefit the programs or activities authorized, and not more than 2 
percent of each appropriated amount for training and technical 
assistance. See, e.g., Consolidated Appropriations Act, 2008, Pub. L. 
No. 110-161, 121 Stat. 1844, 1911-12 (2007). This provision applied to 
appropriation accounts under juvenile justice programs, but did not 
apply to amounts appropriated for grants and projects authorized by 
sections 261 and 262, part E, of the JJDPA--which is the account 
available for developing, testing, and demonstrating promising new 
initiatives and projects. See, e.g., id. 

[56] Under the formula grants program, funds are to be allocated among 
the states on the basis of relative population of people under age 18; 
however, the JJDPA also sets base amounts for awards to the states, 
depending on the aggregate amount appropriated for the program each 
year and taking into account amounts allocated to the states for fiscal 
year 2000. 42 U.S.C. § 5632. 

[57] The Office of Community Oriented Policing Services is a component 
of DOJ responsible for helping state and local law enforcement agencies 
prevent crime. 

[58] Four experts did not provide comments on any factors. 

[59] Cognitive skills include training on general thinking and decision-
making skills, such as to stop and think before acting, generate 
alternative solutions, evaluate consequences, and make decisions about 
appropriate behavior. Social skills include training in pro-social 
behaviors, interpreting social cues, and taking other persons' feelings 
into account, among others. 

[60] Risk factors for reoffending can include antisocial attitudes and 
peer associations, self-control and self-management skills, and drug 
dependencies. 

[61] Four additional experts commented on assessing an individual 
juvenile's treatment needs, but did not expand on its significance. 

[62] For example, family risk factors for delinquency can include 
patterns of high family conflict, having a parent who has been involved 
in the criminal justice system, and sibling delinquent behavior. 

[63] Five additional experts commented on engaging and motivating 
juvenile and family involvement, but did not expand on its 
significance. 

[64] OJJDP formula and block grant awards provide funds to states in 
accordance with statutory requirements. OJJDP allocates some formula 
and block grants to states on the basis of states' juvenile 
populations, while others may be awarded at a fixed level to all 
states. OJJDP discretionary grants provide funds to states, units of 
local government, and organizations to administer programs. OJJDP 
awards discretionary grants to recipients through an application 
process or based on congressional direction. The term "state" means any 
state of the United States, the District of Columbia, the Commonwealth 
of Puerto Rico, the Virgin Islands, Guam, American Samoa, and the 
Commonwealth of the Northern Mariana Islands. 42 U.S.C. § 5603. 

[65] Not all of the reentry experts provided comments on each program 
type as their comments were based on their particular area(s) of 
expertise. 

[66] Kamradt et al., "Wraparound Milwaukee: Program Description and 
Evaluation," Outcomes for Children and Youth with Emotional and 
Behavioral Disorders and Their Families, 307. 

[67] Wraparound Milwaukee is an intervention program for juveniles with 
serious emotional, behavioral, and mental health needs and for their 
families. This intervention program attempts to meet the mental health, 
substance abuse, social service, and other supportive needs of 
juveniles in the Milwaukee community by identifying the specific 
personal, community, and professional services each family needs to 
care for a juvenile with special needs. 

[68] Wiebush et al., "Implementation and Outcome Evaluation of the 
Intensive Aftercare Program." 

[69] The Intensive Aftercare Program was implemented in three states: 
Nevada, Colorado, and Virginia. In Nevada, evaluators found no 
significant or substantive differences in recidivism between youth 
involved in the Intensive Aftercare Program and youth assigned to a 
control group receiving traditional services--such as education and 
individual and group counseling--except that youth in the Intensive 
Aftercare Program group were more likely to be charged with violating 
parole because of increased monitoring. Evaluators could not draw 
conclusions about Colorado and Virginia because of implementation 
issues, such as small samples of available participants in those 
states. 

[70] Residential facilities are correctional facilities that house 
juveniles who are awaiting adjudication (i.e., a trial in a juvenile 
court) or have been adjudicated for an offense, or juveniles who are 
removed from their homes because of nondelinquency issues, such as 
being victims of child abuse. 

[71] None of the remaining 17 experts we interviewed could cite studies 
demonstrating the effectiveness of reentry courts or had opinions on 
this program type. 

[72] Not all of the substance abuse experts provided comments on each 
program type as their comments were based on their particular area(s) 
of expertise. 

[73] Multisystemic therapy is a type of family therapy that helps 
parents identify strengths and develop natural support systems (e.g., 
extended family, neighbors, friends, and church members). 

[74] Sheidow and Henggeler, "Multisystemic Therapy with Substance Using 
Adolescents: A Synthesis of the Research." 

[75] At-risk juveniles are youth who, because of certain 
characteristics or experiences, are statistically more likely than 
other youths to encounter certain problems, such as legal, social, 
financial, educational, emotional, and health problems. 

[76] (1) Stambaugh et al., "Outcomes From Wraparound and Multisystemic 
Therapy in a Center for Mental Health Services System-of-Care 
Demonstration Site." (2) Bruce Kamradt, Stephen A. Gilbertson, and 
Nancy Lynn, "Wraparound Milwaukee Program Description and Evaluation," 
Outcomes for Children and Youth with Emotional and Behavioral Disorders 
and Their Families: Programs and Evaluation Best Practices, 2nd ed. 
(Austin: ProEd, 2005). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: