Justice Impact Evaluations: One Byrne Evaluation Was Rigorous; All Reviewed Violence Against Women Office Evaluations Were Problematic
Highlights
Discretionary grants awarded under the Bureau of Justice Assistance's (BJA) Byrne Program help state and local governments make communities safe and improve criminal justice. Discretionary grants awarded under BJA's Violence Against Women Office (VAWO) programs are aimed at improving criminal justice system responses to domestic violence, sexual assault, and stalking. The National Institute of Justice (NIJ) awarded $6 million for five Byrne Program and five VAWO discretionary grant program evaluations between 1995 and 2001. Of the 10 programs evaluated, all five VAWO evaluations were designed to be both process and impact evaluations of the VAWO programs. Only one of the five Byrne evaluations was designed as an impact evaluation and the other four evaluations were process evaluations. GAO's in-depth review of the four impact evaluations since fiscal year 1995 showed that only one of these--the evaluation of the Byrne Children at Risk Program--was methodologically sound. The other three evaluations, all of which examined VAWO programs, had methodological problems.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Justice | The Attorney General should instruct the Director of NIJ to assess the two VAWO impact evaluations that are in the formative stage to address any potential methodological design and implementation problems and, on the basis of that assessment, initiate any needed interventions to help ensure that the evaluations produce definitive results. |
Our in-depth review of four impact evaluations done under grants awarded by DOJ's National Institute of Justice (NIJ) showed that only one--an evaluation of a grant awarded under DOJ's Byrne Program--was methodologically sound. The other three NIJ evaluations, all of which examined grants awarded by DOJ's Violence Against Women Office (VAWO), had methodological problems. GAO recommended that the Attorney General instruct the Director of NIJ to assess the two VAWO impact evaluations that were in the formative stage to address any potential methodological design and implementation problems and, on the basis of that assessment, initiate any needed interventions to help ensure that the evaluations produce definitive results. In 2004, NIJ took steps to assess the two VAWO impact evaluations and upon reviewing their methodological design, decided that it was not feasible to structure the evaluations so that they could produce definitive results.
|
| Department of Justice | The Attorney General should instruct the Director of NIJ to assess its evaluation process with the purpose of developing approaches to ensure that future impact evaluation studies are effectively designed and implemented so as to produce definitive results. |
Our in-depth review of the impact evaluations conducted by the Byrne Program and Violence Against Women Office (VAWO) of the National Institute of Justice (NIJ) demonstrated that only one of four studies conducted between fiscal years 1995-2001 that had progressed beyond the formative stage proven methodologically sound. This raised concerns about whether these impact evaluations will produce definitive results. As a result, GAO recommended that the Attorney General instruct the Director of NIJ assess its evaluation process with the purpose of developing approaches to ensure that future impact evaluation studies are effectively designed and implemented so as to produce definitive results. In 2003, NIJ responded to this recommendation by reviewing its current guidance for grant applications. As a result of this review, NIJ added additional selection criteria and reporting requirements. For example, additional selection criteria included a utility component; applicants must be able to demonstrate the relevance of the evaluation design and analyses to the information needs of policy and practice audiences; likely contributions to substantive knowledge base; and a plan for communicating results. In addition, intermediate reports are now an additional reporting requirement, and additional topics must be covered in the final report. This will facilitate future program planning and provide the Congress and others with more meaningful, measurable results.
|