This is the accessible text file for GAO report number GAO-02-641T entitled 'Violence Against Women Office: Problems with Grant Monitoring and Concerns about Evaluation Studies' which was released on April 16, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office: GAO: Testimony: Before the Subcommittee on Crime and Drugs, Committee on the Judiciary, U.S. Senate: For Release on Delivery: Expected at 10:15 a.m., EDT: Tuesday, April 16, 2002: Violence Against Women Office: Problems with Grant Monitoring and Concerns about Evaluation Studies: Statement of Laurie E. Ekstrand: Director, Justice Issues: GAO-02-641T: Mr. Chairman and Members of the Subcommittee: Thank you for inviting me to testify about our recent work concerning the Violence Against Women Office (VAWO). This work has specifically focused on monitoring activities and impact evaluations related to VAWO's discretionary grant programs and is part of a body of recent work concerning monitoring and evaluation of grants by a number of Office of Justice Program's (OJP) bureaus and offices. Monitoring and evaluation are the activities that identify whether programs are operating as intended, whether they are reaching those that should be served, and ultimately whether they make a difference. In other words, these are major elements of assessing results. Our recent work has shown a need for improvement in VAWO grant monitoring and in the evaluations that are intended to assess the impacts of VAWO programs. Background: VAWO was created in 1995 to carry out certain programs created under the Violence Against Women Act of 1994.[Footnote 1] The Victims of Trafficking and Violence Prevention Act of 2000 reauthorized most of the existing VAWO programs and added new programs.[Footnote 2] VAWO's mission is to lead the national effort to end violence against women, including domestic violence, sexual assault, and stalking. VAWO programs seek to improve criminal justice system responses to these crimes by providing support for law enforcement, prosecution, courts, and victim advocacy programs across the country. In addition, programs are to enhance direct services for victims, including victim advocacy, emergency shelter, and legal services. VAWO also addresses violence against women issues internationally, including working to prevent trafficking in persons. VAWO is one of seven program offices and five bureaus in OJP.[Footnote 3] VAWO's discretionary grant programs have grown substantially since its inception in 1995. Data provided by OJP showed that, between fiscal years 1995 and 2000, the yearly number of VAWO discretionary grant awards increased about 362 percent—-from 92 in fiscal year 1996, the first full year of funding, to 425 in fiscal year 2000. In addition the yearly dollar amount of VAWO discretionary grant awards increased about 940 percent-—from just over $12 million in fiscal year 1996, the first full year of funding, to about $125 million in fiscal year 2000. Appendix II shows the number of yearly VAWO discretionary grant awards for fiscal year 1995 through fiscal year 2000. Appendix BI shows the dollar amount of VAWO discretionary grant awards, adjusted to constant fiscal year 2000 dollars, over the same period. Problems with VAWO Discretionary Grant Monitoring: The monitoring of grant activities is a key management tool to help ensure that funds awarded to grantees are being properly spent. In November 2001, in response to a request by Senators Grassley and Sessions, we reported that grant files for discretionary grants awarded by VAWO often lacked the documentation necessary to ensure that the required monitoring activities occurred.[Footnote 4] Our review of grant files for a representative sample of VAWO discretionary grants active in all of fiscal years 1999 and/or 2000 showed that: * VAWO grant files did not always contain requisite grant monitoring plans. When monitoring plans were in the files, grant managers did not consistently document their monitoring activities, such as site visits, according to the plans they developed. * A substantial number of VAWO grant files did not contain progress and financial reports sufficient to cover the entire grant period, contrary to OJP guidelines. Furthermore, VAWO grantee progress and financial reports were often submitted late by grantees. These reports are an important tool to help managers and grant monitors determine if grantees are meeting program objectives and financial commitments. * VAWO grant files did not always contain the required closeout documents-—key documents by which OJP ensures that, among other things, the final accounting of federal funds have been received. We also found that, because documentation about monitoring activities was not readily available, VAWO was not positioned to systematically determine staff compliance with monitoring requirements and assess overall performance. Although VAWO officials said that they met with grant managers weekly to discuss any grant problems or monitoring issues, VAWO did not (1) have an overall system to track monitoring activities, other than site visits and (2) appear to be routinely using OJP-wide data on late progress reports and financial reports. Furthermore, the lack of systematic data associated with program monitoring activities and the documentation problems we observed raised questions about whether VAWO was positioned to measure its performance consistent with the Government Performance and Results Act (GPRA) of 1993. Specifically, we pointed out that, in DOJ's Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan, DOJ failed to recognize the serious limitations associated with inconsistent documentation and the lack of systematic monitoring data in measuring whether VAWO was achieving its goals for formula and discretionary grants—especially since the Report and Plan stated that VAWO would rely on grant monitoring data to measure its performance. We concluded that neither OJP, VAWO, nor GAO can determine the level of monitoring performed by grant managers as required by OJP and the comptroller general's internal control standards, which call for documentation of all transactions and other significant events to ensure that management directives are being carried out.[Footnote 5] We recommended that VAWO review why documentation problems occurred and take steps to resolve these problems. Too Early to Gauge Efforts to Resolve Grant Monitoring Problems: VAWO and OJP officials have acknowledged that they need to take steps to resolve some of the problems associated with grant monitoring, but it is too early to tell if these steps will be effective. For example, in response to our report, the assistant attorney general said that VAWO had begun to develop both an internal monitoring manual that would include procedures for developing monitoring plans using a risk- based assessment tool. They also said they have developed a management information system that will eventually track the submission of progress and financial reports. Furthermore, while we were developing our report, VAWO officials said that they were not satisfied with the performance measures they used to gauge their performance under GPRA because they did not believe they are meaningful for measuring program outcomes. They said that they are working with other OJP officials and an outside contractor to develop new measures and hope to have them available for the fiscal year 2003 performance plan. It is also important to note that VAWO's efforts to address grant monitoring problems need to be viewed in the context of OJP efforts in this area. Our recent related reports discussed grant monitoring problems at other OJP organizations, such as the Bureau of Justice Assistance (BJA) and the Office of Juvenile Justice and Delinquency Programs (OJJDP),[Footnote 6] and pointed out that, over the last few years, we and others, including OJP, have identified various grant monitoring problems among OJP bureaus and offices. We discussed how OJP had begun to work with bureaus and offices to resolve some of the problems it and others have identified, including OJP efforts to develop an automated grants management system as a way to standardize and streamline the grant process. Our report concluded that OJP efforts to automate the grant management process, particularly in regard to grant monitoring, holds some promise if OJP takes steps to ensure that all monitoring activities are consistently recorded and maintained in a timely manner. We also said that current and future efforts will be futile unless OJP and its bureaus and offices, such as VAWO, periodically test grant manager compliance with OJP requirements and take corrective action when needed to enforce those requirements. We recommended that OJP (1) study and recommend ways to establish an approach to systematically test or review grant files to ensure consistent documentation across OJP and (2) explore ways to electronically compile and maintain documentation of monitoring activities to facilitate more consistent documentation, more accessible management oversight, and sound performance measurement. In January 2002, in response to our report, the assistant attorney general said that OJP agreed that it needs to develop more consistent documentation of monitoring activities. She said that among other things, OJP has created a chief information officer position charged with planning and implementing an agencywide grant management system. According to the assistant attorney general, the new system is envisioned to produce reports in response to informational requests, provide information pertaining to grantees and all resources provided by OJP, and maintain information from the opening to the closing of a grant award. Although the assistant attorney general said that OJP will consider the comptroller general's internal control standards in taking these steps, it is unclear whether the new system will include the full range and scope of monitoring activities carried out by grant managers in VAWO and other OJP organizations. Concerns About Evaluation Studies of VAWO Discretionary Grant Programs: We have also recently issued a report on work undertaken for Senators Grassley and Sessions that addressed the methodological rigor of impact evaluations of three VAWO discretionary grant programs. [Footnote 7] During fiscal years 1995 through 2001, the National Institute of Justice (NIJ) awarded about $4 million for five VAWO discretionary grant program evaluations that were intended to measure the impact of the VAWO programs.[Footnote 8] Our in-depth review of the three program evaluations that had progressed beyond the formative stage showed that all three had methodological problems that raised concerns about whether the evaluations will produce definitive results. More specifically, our report stated that, although program evaluation is an inherently difficult task, in all three VAWO evaluations, the effort was particularly arduous because of variations across grantee sites in how the programs are implemented. Our concerns about these efforts included problems with both evaluation design and implementation. In particular, VAWO sites participating in the impact evaluations had not been shown to be representative of their programs, thereby limiting the evaluators' ability to generalize results. Further, the lack of nonprogram participant comparison groups hindered evaluators' ability to minimize the effects of factors that are external to the program and isolate the impact of the program alone. While in some situations, other means (other than comparison groups) can be effective in isolating the impact of a program from other factors, in these evaluations, effective alternative methods were not used. In addition, data collection and analytical problems (e.g., related to statistical tests, assessment of change) compromised the evaluators' ability to draw appropriate conclusions from the results. We concluded that, despite great interest in assessing the results of OJP's discretionary grant programs, it can be extremely difficult to design and execute evaluations that will provide definitive information. We further concluded that, given that NIJ spends millions of dollars to evaluate OJP grant programs, including those within VAWO, more up-front attention to the methodological rigor of these evaluations will increase the likelihood that they will produce meaningful results for policymakers. We recommended that the attorney general direct the NIJ director to assess the two VAWO impact evaluations still in the formative stage to address any design methodology and implementation problems and, on the basis of that assessment, initiate any needed interventions to help ensure that the evaluations produce definitive results. We further recommended that the director of NIJ be instructed to assess its evaluation process to develop approaches to ensure that future impact evaluation studies are designed and implemented to produce definitive results. The assistant attorney general commented that she agreed with the substance of our recommendations and has begun or plans to take steps to address them. It is still too early to tell whether these actions will be effective in preventing or resolving the problems we identified, but they appear to be steps in the right direction. In summary, since its inception, VAWO has grown substantially both in terms of the number of discretionary grants awarded and dollars awarded for those discretionary grants—increasing the importance of ensuring that its grantees are achieving intended results. Unfortunately, the lack of good data from monitoring activities and impact evaluations leaves us with very little basis to assess program results. Both VAWO and OJP have indicated a commitment to making improvements, citing reorganization plans and the anticipated management information system as the foundation for improved grants management, including improvements in monitoring and evaluation. But, reorganization and management information systems are only tools and are only as good as the management that wields them. Commitment to improvement and oversight are needed to ensure progress. Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions that you or other members of the subcommittee may have. For further information regarding this testimony, please contact Laurie E. Ekstrand or John F. Mortin at (202) 512-8777. Individuals making key contributions to this testimony included Wendy C. Simkalo, Jared A. Hermalin, and Chan My J. Battcher. [End of section] Appendix I: OJP Organization Chart: [Refer to PDF for image: organization chart] Top level: Office of Assistant Attorney General: * American Indian & Alaska Native Affairs Desk; * Equal Employment Opportunity Office. Second level, reporting to Office of Assistant Attorney General: * Violence Against Women Office; * Corrections Program Office; * Drug Courts Program Office; * Executive Office for Weed & Seed; * Office for Domestic Preparedness; * Office of Police Corps & Law Enforcement Education; * Office of Budget, & Management Services; * Office of Administration; * Office for Civil Rights; * Office of the Comptroller; * Office of General Counsel; * Office of Congressional & Public Affairs; * Bureau of Justice Assistance; * Bureau of Justice Statistics; * National Institute of Justice; * Office of Juvenile Justice & Delinquency Prevention; * Office for Victims of Crime. Note: The organization chart is current as of March 2002. Source: Prepared by GAO based on OJP documentation. [End of figure] [End of section] Appendix II: Number of Yearly VAWO Discretionary Awards, Fiscal Years 1995-2000: [Refer to PDF for image: vertical bar graph] Number of Awards (approximated from graph): Fiscal Year 1995: 20; Fiscal Year 1996: 80; Fiscal Year 1997: 245; Fiscal Year 1998: 350; Fiscal Year 1999: 330; Fiscal Year 2000: 425. Source: OJP Office of the Comptroller. [End of figure] [End of section] Appendix III: Dollar Amount of VAWO Discretionary Awards, Fiscal Years 1995-2000: [Refer to PDF for image: vertical bar graph] Amount of Awards (approximated from graph): Fiscal Year 1995: $1,000,000; Fiscal Year 1996: $10,000,000; Fiscal Year 1997: $70,000,000; Fiscal Year 1998: $115,000,000; Fiscal Year 1999: $90,000,000. Fiscal Year 2000: $125,000,000. Note: The award amounts for each fiscal year are adjusted to constant fiscal year 2000 dollars. Source: OJP Office of the Comptroller. [End of figure] [End of section] Footnotes: [1] Title IV of the Violent Crime Control and Law Enforcement Act of 1994 (P.L. 103-322). [2] P.L. 106-386. [3] OJP’s five bureaus are Bureau of Justice Assistance, Bureau of Justice Statistics, National Institute of Justice, Office of Juvenile Justice and Delinquency Prevention, and Office for Victims of Crime. OJP’s seven program offices are American Indian and Alaska Native Affairs Desk, Violence Against Women Office, Executive Office for Weed and Seed, Corrections Program Office, Drug Courts Program Office, Office for Domestic Preparedness, and Office of Police Corps and Law Enforcement Education. Appendix I shows OJP’s current organizational structure. [4] U.S. General Accounting Office, Justice Discretionary Grants: Byrne Program and Violence Against Women Office Grant Monitoring Should Be Better Documented, [hyperlink, http://www.gao.gov/products/GAO-02-025] (Washington, D.C.: Nov. 28, 2001). [5] U.S. General Accounting Office, Internal Control: Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.1.3.1] (Washington, D.C.: Nov. 1999). [6] U.S. General Accounting Office, Juvenile Justice: OJJDP Reporting Requirements for Discretionary and Formula Grantees and Concerns About Evaluation Studies, [hyperlink, http://www.gao.gov/products/GAO-02-23] (Washington, D.C.: Oct. 30, 2001). [7] U.S. General Accounting Office, Justice Impact Evaluations: One Byrne Evaluation Was Rigorous; AU Reviewed Violence Against Women Office Evaluations Were Problematic, [hyperlink, http://www.gao.gov/products/GAO-02-309] (Washington, D.C.: Mar. 7, 2002). [8] Impact evaluations are designed to assess the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. [End of section]