This is the accessible text file for GAO report number GAO-13-682T entitled 'Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures' which was released on September 10, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Statement for the Record to the Subcommittee on Bankruptcy and the Courts, Committee on the Judiciary, U.S. Senate: For Release on Delivery: Expected at 10:30 a.m. EDT: Tuesday, September 10, 2013: Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures: Statement of David C. Maurer, Director: Homeland Security and Justice: GAO-13-862T: Mr. Chairman and Members of the Subcommittee: I appreciate the opportunity to submit this statement on our work on case-related workload measures for district court and courts of appeals judges. My statement today is based on work completed and reported in 2003[Footnote 1] and discussed in testimony on June 17, 2008,[Footnote 2] and September 30, 2009.[Footnote 3] My statement is focused exclusively on these workload measures; we have not assessed or taken a position on the Judicial Conference's pending request for additional judgeships. Biennially, the Judicial Conference, the federal judiciary's principal policy-making body, assesses the judiciary's needs for additional judgeships.[Footnote 4] If the conference determines that additional judgeships are needed, it transmits a request to Congress identifying the number, type (such as courts of appeals or district courts), and location of the judgeships it is requesting. In assessing the need for additional judgeships, the Judicial Conference considers a variety of information, including responses to its biennial survey of individual courts, temporary increases or decreases in case filings, and other factors specific to an individual court. However, the Judicial Conference's analysis begins with the quantitative case-related workload measures it has adopted for the district courts and courts of appeals--weighted case filings and adjusted case filings, respectively. These two measures recognize, to different degrees, that the time demands on judges are largely a function of both the number and complexity of the cases on their dockets. Some types of cases may demand relatively little time and others may require many hours of work. Generally, each case filed in a district court is assigned a weight representing the average amount of judge time the case is expected to require. The weights are relative to one another: the higher the case weight, the greater the time the case would be expected to require. For example, on average, a case with a relative weight of 2.0 would be expected to require twice as much judge time as a case with a weight of 1.0. In the courts of appeals, all case filings are weighted equally at 1.0, except for pro se case filings--those in which one or both parties are not represented by an attorney--which are discounted. Using these measures, individual courts whose past case-related workload meets the threshold established by the Judicial Conference may be considered for additional judgeships. These thresholds are 430 weighted case filings per authorized judgeship for district courts and 500 adjusted case filings per three-judge panel of authorized judgeships for courts of appeals (courts of appeals judges generally hear cases in rotating panels of three judges each). Authorized judgeships are the total number of judgeships authorized by statute for each district court or court of appeals. The Judicial Conference relies on these quantitative workload measures to be reasonably accurate measures of judges' case-related workload. Whether these measures are reasonably accurate rests in turn on the soundness of the methodology used to develop them. This statement summarizes information and recommendations from our 2003 report and related updates to the methodology including (1) whether the judiciary's quantitative case-related workload measures were reasonably accurate measures of district judges' and courts of appeals judges' case-related workload, and (2) actions the judiciary has taken to implement recommendations from our 2003 report. Our 2003 report and 2008 and 2009 testimonies were based on the results of our review of documentation provided by the Federal Judicial Center (FJC) and the Administrative Office of the U.S. Courts (AOUSC) on the history and development of the case-related workload measures and interviews with officials in each organization. The scope of our work did not include how the Judicial Conference used these case-related workload measures to develop any specific request for additional district or courts of appeals judgeships. In addition, in August 2013, we met with FJC and AOUSC officials to discuss any updates that have been made to the methodologies for developing case- related workload measures. While we obtained some updated information for the purposes of this statement, we did not evaluate these methodologies or the judiciary's efforts in this area. This work was conducted in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: The demands on judges' time are largely a function of both the number and complexity of the cases on their dockets. To measure the case- related workload of district court judges, the Judicial Conference has adopted weighted case filings. The purpose of the district court case weights was to create a measure of the average judge time that a specific number and mix of cases filed in a district court would require. Importantly, the weights were designed to be descriptive, not prescriptive--that is, the weights were designed to develop a measure of the national average amount of time that judges actually spent on specific cases, not to develop a measure of how much time judges should spend on various types of cases. Moreover, the weights were designed to measure only case-related workload. Judges have noncase- related duties and responsibilities, such as administrative tasks, that are not reflected in the case weights. With few exceptions, such as cases that are remanded to a district court from the court of appeals, each civil or criminal case filed in a district court is assigned a case weight. For example, in the 2004 case weights--which are still in use--drug possession cases are weighted at 0.86, while civil copyright and trademark cases are weighted at 2.12. The total annual weighted filings for a district are determined by summing the case weight associated with all the cases filed in the district during the year. A weighted case filing per authorized judgeship is the total annual weighted filings divided by the total number of authorized judgeships. The Judicial Conference uses weighted filings of 430 or more per authorized judgeship as an indication that a district may need additional judgeships. Thus, for example, a district with 460 weighted filings per authorized judgeship, including newly requested judgeships, could be considered for an additional judgeship. However, the Judicial Conference does not consider a district for additional judgeships, regardless of its weighted case filings, if the district does not request any additional judgeships. Accuracy of Judiciary's Workload Measures Could Not Be Determined: 1993 District Case Weights Reasonably Accurate, but Accuracy of 2004 Case Weights Could Not Be Statistically Determined: In our 2003 report, we found the district court case weights approved in 1993 to be a reasonably accurate measure of the average time demands a specific number and mix of cases filed in a district court could be expected to place on the district judges in that court. The methodology used to develop the weights used a valid sampling procedure, developed weights based on actual case-related time recorded by judges from case filings to disposition, and included a measure (standard errors) of the statistical confidence in the final weight for each weighted case type. Without such a measure, it is not possible to objectively assess the accuracy of the final case weights. At the time of our 2003 report, the Subcommittee on Judicial Statistics of the Judicial Conference's Judicial Resources Committee had approved the research design for revising the 1993 case weights, with a goal of having new weights submitted to the Judicial Resources Committee for review in the summer of 2004. The design for the new case weights relied on three sources of data for specific types of cases: (1) data from automated databases identifying the docketed events associated with the cases; (2) data from automated sources on the time associated with courtroom events for cases, such as trials or hearings; and (3) consensus of estimated time data from structured, guided discussion among experienced judges on the time associated with noncourtroom events for cases, such as reading briefs or writing opinions. As we reported in 2009, according to FJC, the subcommittee wanted a study that could produce case weights in a relatively short period of time without imposing a substantial record-keeping burden on district judges.[Footnote 5] FJC staff provided the subcommittee with information about various approaches to case weighting, and the subcommittee chose an event-based method--that is, a method that used data on the number and types of events, such as trials and other evidentiary hearings, in a case. The design did not involve the type of time study that was used to develop the 1993 case weights. Although the proposed methodology appeared to offer the benefit of reduced judicial burden (no time study data collection), potential cost savings, and reduced calendar time to develop the new weights, we had two areas of concern--the challenge of obtaining reliable, comparable data from two different data systems for the analysis and the limited collection of actual data on the time judges spend on cases. First, the design assumed that judicial time spent on a given case could be accurately estimated by viewing the case as a set of individual tasks or events in the case. Information about event frequencies and, where available, time spent on the events would be extracted from existing administrative databases and reports and used to develop estimates of the judge time spent on different types of cases. For event data, the research design proposed using data from two databases (one of which was new in 2003 and had not been implemented in all district courts) that would have to be integrated to obtain and analyze the event data. FJC proposed creating a technical advisory group to address this issue. In August 2013, FJC officials told us that the process of integrating the two data systems, though labor-intensive, was successful and resulted in accurate data. However, we have not reviewed the integration process for the two data systems, so we cannot determine the effectiveness of this process or whether accurate data resulted. Second, we reported that the research design did not require judges to record time spent on individual cases, as was done for the 1993 case weights. Actual time data would be limited to that available from existing databases and reports on the time associated with certain courtroom events and proceedings for different types of cases. However, a majority of district judges' time is spent on case-related work outside the courtroom. The time required for noncourtroom events-- and some courtroom events that did not have actual time data available--would be derived from structured, guided discussion of groups of 8 to 13 experienced district court judges in each of the 12 regional circuits (about 100 judges in all).[Footnote 6] The judges would develop estimates of the time required for different events in different types of cases within each circuit using FJC-developed "default values" as the reference point for developing their estimates. These default values would be based in part on the existing case weights and, in part, on other types of analyses. Following the meetings of the judges in each circuit, a national group of 24 judges (2 from each circuit) would consider the data from the 12 circuit groups and develop the new weights. The accuracy of judges' time estimates is dependent upon the experience and knowledge of the participating judges and the accuracy and reliability of the judges' recall about the average time required for different events in different types of cases--about 150 if all the case types in the 1993 case weights were used. In 2003, we found that these consensus data could not have been used to calculate statistical measures of the accuracy of the resulting case weights. Thus, we concluded that the planned methodology did not make it possible to objectively, statistically assess how accurate the new case weights are--weights whose accuracy the Judicial Conference relies upon in assessing judgeship needs. In August 2013, AOUSC officials stated that, since 2005, for purposes of determining the need for an additional authorized judgeship, a district's weighted case filings per authorized judgeship is calculated by including the potential additional judgeship. For example, if a district had total weighted filings of 4,600 and 9 authorized judgeships, and it planned to request 1 additional judgeship, its weighted filings per authorized judgeship, for purposes of the judgeship request process, would be 460. Without including the potential additional judgeship in the calculation, the weighted case filings would be about 511. AOUSC officials stated in August 2013 that the judiciary adopted the proposed methodology in 2004 and does not have plans to update the 2004 district court case weights. Accuracy of Courts of Appeals Case-Related Workload Measure Could Not Be Assessed: In 2003, we found that the principal quantitative measure the Judicial Conference used to assess the need for additional courts of appeals judgeships was adjusted case filings. The measure is based on data available from standard statistical reports for the courts of appeals. The adjusted filings workload measure is not based on any empirical data regarding the time that different types of cases required of appellate judges. The Judicial Conference's policy is that courts of appeals with adjusted case filings of 500 or more per 3-judge panel may be considered for 1 or more additional judgeships. Courts of appeals generally decide cases using constantly rotating 3-judge panels. Thus, if a court had 12 authorized judgeships, those judges could be assigned to four panels of 3 judges each. In assessing judgeship needs for the courts of appeals, the conference may also consider factors other than adjusted filings, such as the geography of the circuit or the median time from case filings to disposition.[Footnote 7] Essentially, the adjusted case filings workload measure counts all case filings equally, with two exceptions. First, cases refiled and approved for reinstatement are excluded from total case filings.[Footnote 8] Second, pro se cases are weighted at 0.33, or one- third as much as other cases, which are weighted at 1.0. For example, a court with 600 total pro se case filings in a year would be credited with 198 adjusted pro se case filings (600 x 0.33). Thus, a court of appeals with 1,600 filings (excluding reinstatements)--600 pro se cases and 1,000 non-pro se cases--would be credited with 1,198 adjusted case filings (198 discounted pro se cases plus 1,000 non-pro se cases). If this court had 6 judges (allowing two panels of 3 judges each), it would have 599 adjusted case filings per 3-judge panel, and, thus, under Judicial Conference policy, could be considered for an additional judgeship. The current court of appeals workload measure, which, AOUSC officials stated, was adopted in 1996, represents an effort to improve the previous measure. In our 1993 report on judgeship needs assessment, we found that the restraint of individual courts of appeals, not the workload standards, seemed to have determined the actual number of appellate judgeships the Judicial Conference requested.[Footnote 9] At the time the current measure was developed and approved, using the new benchmark of 500 adjusted case filings resulted in judgeship numbers that closely approximated the judgeship needs of the majority of the courts of appeals, as the judges of each court perceived them. The current courts of appeals case-related workload measure principally reflects a policy decision using historical data on filings and terminations. It is not based on empirical data regarding the judge time that different types of cases may require. On the basis of the documentation we reviewed for our 2003 report, we determined that there was no empirical basis for assessing the potential accuracy of adjusted case filings as a measure of case-related judge workload. Judiciary Has Taken No Action to Address Our 2003 Recommendations: In our 2003 report, we recommended that the Judicial Conference of the United States: * update the district court case weights using a methodology that supports an objective, statistically reliable means of calculating the accuracy of the resulting weights, and: * develop a methodology for measuring the case-related workload of courts of appeals judges that supports an objective, statistically reliable means of calculating the accuracy of the resulting workload measures and that addresses the special case characteristics of the Court of Appeals for the D.C. Circuit. Neither of these recommendations has been implemented, and in August 2013, AOUSC officials stated that the judiciary does not have plans to update the 2004 district court case weights or the 1996 court of appeals adjusted filings weights. With regard to our 2003 recommendation for updating the district court case weights, we reported that FJC agreed that the method used to develop the new case weights would not permit the calculation of standard errors, but that other methods could be used to assess the integrity of the resulting case weight system.[Footnote 10] In response, we noted that the Delphi technique to be used for developing out-of-court time estimates was most appropriate when more precise analytical techniques were not feasible and the issue could benefit from subjective judgments on a collective basis. More precise techniques were available for developing the new case weights and were to be used for developing new bankruptcy court case weights. In our 2003 report, we also concluded that the methodology the Judicial Conference decided to begin in June 2002 for the revision of the bankruptcy case weights offered an approach that could be usefully adopted for the revision of the district court case weights.[Footnote 11] The bankruptcy court methodology used a two-phased approach. First, new case weights were to be developed based on time data recorded by bankruptcy judges for a period of weeks--a methodology very similar to that used to develop the bankruptcy case weights that existed in 2003 at the time of our report. The accuracy of the new case weights could be assessed using standard errors. The second part represents experimental research to determine if it is possible to make future revisions of the weights without conducting a time study. The data from the time study could be used to validate the feasibility of this approach. If the research determined that this was possible, the case weights could be updated more frequently with less cost than required by a time study. We concluded in 2003 that that approach could provide (1) more accurate weighted case filings than the design developed and used for the development of the 2004 district court case weights, and (2) a sounder method of developing and testing the accuracy of case weights that were developed without a time study. However, we have not reviewed the effectiveness of this methodology or confirmed whether the judiciary implemented this approach. With regard to our recommendation on improving the case-related workload measure for the courts of appeals, the Chair of the Committee on Judicial Resources commented that the workload of the courts of appeals entails important factors that have defied measurement, including significant differences in case-processing techniques. We recognized that there were significant methodological challenges in developing a more precise workload measure for the courts of appeals. However, we stated that using the data available, neither we nor the Judicial Conference could have assessed the accuracy of adjusted case filings as a measure of the case-related workload of courts of appeals judges. The Ranking Member of the Subcommittee on Bankruptcy and the Courts has requested that we conduct a full review of the case-related workload measures for district court and courts of appeals judges, including a follow-up on our 2003 recommendations. Such a review will allow us to evaluate the judiciary's methodology and efforts over the last 10 years. Mr. Chairman, this concludes my statement for the record. Contact and Staff Acknowledgments: For further information about this statement, please contact David C. Maurer, Director, Homeland Security and Justice Issues, on (202) 512- 9627 or maurerd@gao.gov. In addition to the contact named above, the following individuals also made major contributions to this testimony: Chris Currie, Acting Director; David P. Alexander, Assistant Director; Brendan Kretzschmar; Jean M. Orland; Rebecca Kuhlmann Taylor; and Janet G. Temko. [End of section] Footnotes: [1] GAO, Federal Judgeships: The General Accuracy of the Case-Related Workload Measures Used to Assess the Need for Additional District Court and Courts of Appeals Judgeships, [hyperlink, http://www.gao.gov/products/GAO-03-788R] (Washington, D.C., May 30, 2003). [2] GAO, Federal Judgeships: General Accuracy of District and Appellate Judgeship Case-Related Workload Measures, [hyperlink, http://www.gao.gov/products/GAO-08-928T] (Washington, D.C., June 17, 2008). [3] GAO, Federal Judgeships: The General Accuracy of District and Appellate Judgeship Case-Related Workload Measures, [hyperlink, http://www.gao.gov/products/GAO-09-1050T] (Washington, D.C., Sept. 30, 2009). [4] The Chief Justice of the United States presides over the conference, which consists of the chief judges of the 13 courts of appeals, a district judge from each of the 12 regional circuits, and the chief judge of the Court of International Trade. The conference meets twice a year. [5] [hyperlink, http://www.gao.gov/products/GAO-09-1050T]. [6] In August 2013, FJC officials stated that, for the 2004 case weight methodology, there were 15 criminal case event types and 13 civil case event types. Actual event frequency data from administrative databases and reports were used in the case weight methodology for all criminal and civil case event types. Actual event time data from administrative databases and reports were used for 4 of the criminal and civil case event types, such as a jury trial. Time estimates from the judge discussion process were used for 11 criminal case event types and 9 civil case event types--including all 6 noncourtroom events, such as preparation for trial. [7] At the time of our 2003 report, FJC had suggested that adjusted case filings may not be an appropriate measure for the Court of Appeals for the D.C. Circuit, given the distinctive characteristics of the administrative agency appeals that were a major source of that court's caseload. Details on the FJC analysis for the D.C. Circuit can be found in our 2003 report: [hyperlink, http://www.gao.gov/products/GAO-03-788R]. In August 2013, AOUSC reported that the judiciary does not have plans to develop an authorized judgeship workload formula specific to the Court of Appeals for the D.C. Circuit. [8] Such cases were dismissed for procedural defaults when originally filed, but "reinstated" to the court's calendar when the case was later refiled. The number of such cases, as a proportion of total cases, is generally small. [9] GAO, Federal Judiciary: How the Judicial Conference Assesses the Need for More Judges, [hyperlink, http://www.gao.gov/products/GAO/GGD-93-31] (Washington, D.C., Jan. 29, 1993). [10] We have not reviewed in detail the materials FJC has posted on its website with regard to the methodology actually used to develop the revised case weights approved in 2004. However, those materials indicate that FJC essentially followed the design we reviewed and that standard errors were not computed for the final weights. [11] See GAO, Federal Bankruptcy Judges: Weighted Case Filings as a Measure of Judges' Case-Related Workload, [hyperlink, http://www.gao.gov/products/GAO-03-789T] (Washington, D.C., May 22, 2003). [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]