Justice Outcome Evaluations: Design and Implementation of Studies Require More NIJ Attention
GAO-03-1091
Published: Sep 24, 2003. Publicly Released: Oct 08, 2003.
Skip to Highlights
Highlights
Policy makers need valid, reliable, and timely information on the outcomes of criminal justice programs to help them decide how to set criminal justice funding priorities. In view of previously reported problems with selected outcome evaluations managed by the National Institute of Justice (NIJ), GAO assessed the methodological quality of a sample of completed and ongoing NIJ outcome evaluation grants.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Justice | The Attorney General should instruct the Director of NIJ to conduct a review of its ongoing outcome evaluation grants--including those discussed in this report--and develop appropriate strategies and corrective measures to ensure that methodological design and implementation problems are overcome so the evaluations can produce more conclusive results. Such a review should consider the design and implementation issues we identified in our assessment in order to decide whether and what type of intervention may be appropriate. If, based on NIJ's review, it appears that the methodological problems cannot be overcome, NIJ should consider refocusing the studies' objectives and/or limiting funding. |
NIJ developed and implemented an Evaluation Research Oversight Plan to ensure that evaluations stay on track, issues are highlighted, and solutions, where feasible, are proposed. To be responsive to GAO's recommendation, NIJ now identifies evaluation research projects that require monitoring at a high intensity level, as opposed to normal. The oversight plan calls for project monitors to discuss the high intensity projects at semi-annual meetings. Among the topics for discussion are changes in evaluation plan, methodology, or design, and problems in implementation. Corrective actions taken are to be incorporated into an annual report. NIJ provided information on each of its high intensity evaluation research grants for the period January 2005 through June 2005.
|
| Department of Justice | The Attorney General should instruct the Director of NIJ to continue efforts to respond to our March 2002 recommendation that NIJ assess its evaluation process with the purpose of developing approaches to ensure that future outcome evaluation studies are funded only when they are effectively designed and implemented. The assessment could consider the feasibility of such steps as obtaining more information about the availability of outcome data prior to developing a solicitation for research; requiring that outcome evaluation proposals contain more detailed design specifications before funding decisions are made regarding these proposals; and more carefully calibrating NIJ monitoring procedures to the cost of the grant, the risks inherent in the proposed methodology, and the extent of knowledge in the area under investigation. |
We recommended that NIJ continue efforts to assess its evaluation process with the purpose of developing approaches to ensure that future outcome evaluation studies are funded only when they are effectively designed and implemented. NIJ has reviewed its guidance for grant applications and added selection criteria and reporting requirements, as well as stated that funding priority would be given to rigorous evaluation research designs. NIJ also developed a multi-part corrective action plan to implement our recommendation which, among other things, calls for assessing the feasibility of proposed outcome evaluations when appropriate and puts in place a mechanism for monitoring all current and future evaluation research grants. In addition, NIJ prepared an Evaluation Research Oversight Plan that included nine steps to be taken by NIJ personnel (specifically, Evaluation Division staff, project monitors, division chiefs, and the Acting director of the Office of Research and Evaluation) to monitor NIJ outcome evaluations and help ensure that they are effectively designed and implemented. We believe the steps taken by NIJ will facilitate program planning and provide the Congress and others with more meaningful information based on better designed and implemented evaluation studies.
|
Full Report
Topics
Crime preventionCrimesEvaluation methodsFederal aid for criminal justiceProgram evaluationStrategic planningDomestic violenceLaw enforcementCrimeData collection