This is the accessible text file for GAO report number GAO-03-11 entitled 'Public Schools: Insufficient Research to Determine Effectiveness of Selected Private Education Companies' which was released on October 29, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products’ accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. Report to the Ranking Minority Member, Subcommittee on the District of Columbia, Committee on Appropriations, House of Representatives: United States General Accounting Office: GAO: October 2002: Public Schools: Insufficient Research to Determine Effectiveness of Selected Private Education Companies: Selected Private Education Companies: GAO-03-11: Contents: Letter: Results in Brief: Background: Education Management Companies Have Multifaceted Programs; District Schools Varied In Their Implementation: Limited Research Exists of the Effectiveness of These Companies’ Programs: Agency Comments: Appendix I: Objectives, Scope, and Methodology: Objectives: Scope and Methodology: Appendix II: Studies and Other Information Sources Considered: Appendix III: Comments from the Department of Education: Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Acknowledgments: Related GAO Products: Table: Table 1: Profiles of Three Education Management Companies Operating in the District of Columbia: United States General Accounting Office: Washington, DC 20548: October 29, 2002: The Honorable Chaka Fattah Ranking Minority Member Subcommittee on the District of Columbia Committee on Appropriations House of Representatives: Dear Mr. Fattah: In recent years, local school districts and traditional public schools have taken various initiatives to improve failing schools. For example, school districts and charter schools are increasingly contracting with private, for-profit companies to provide a range of education and management services to schools. To date, there has been debate regarding the effectiveness of such companies in managing public schools. These companies generally offer schools services in areas such as school organization, instruction, technology, and professional development. In the District of Columbia, some public schools currently contract with three such companies: Edison Schools, Mosaica Education, and Chancellor Beacon Academies. As agreed with your office, we (1) identified the characteristics of their programs and determined the extent to which District schools managed by them have used their programs and (2) determined what is known about the effectiveness of these companies’ educational programs, as measured primarily by student achievement. To address these issues, we reviewed relevant research on charter and traditional public schools managed by for-profit educational management companies as well as documents and materials provided by the companies. In addition, we observed an on-site review of one school’s program which was conducted for District oversight authorities. We also interviewed officials of the companies that manage District public schools, officials of the District’s oversight authorities, and representatives of the schools, as well as officials of the Department of Education, other education experts and advocates. Finally, we reviewed evaluations concerning the three companies operating in the District that met the following criteria: included comparison groups and measurement of performance over time, and focused on academic achievement, parental satisfaction, parental involvement, or school climate. We assessed the quality of the evaluations’ research designs, reviewed them for threats to validity and determined whether we had confidence in their conclusions. We conducted our work between January and September 2002 in accordance with generally accepted government auditing standards. Results in Brief: Edison, Mosaica, and Chancellor Beacon have programs that consist of both management services, such as personnel, and educational services, which they offer to schools across the nation; in the District, most of the schools managed by these companies have either adopted selected elements of their companies’ programs or chosen other educational programs. Each company provides services such as curriculum, assessments, parental involvement opportunities, and student and family support. They also offer a variety of organizational options, including smaller class and school sizes, as well as longer school days and school years. All of the companies allow their schools some flexibility in adapting their programs to local circumstances. The extent to which the District schools implemented all the elements of these companies’ educational programs varied. For example, 6 of the 10 schools managed by these companies had either partially implemented the company’s curriculum or had not implemented that curriculum at all. Some schools have opted to customize the company’s educational program; other schools have left in place the educational program of a company that formerly managed them. In school year 2001-02, all 10 District schools managed by these companies were charter schools with predominantly poor and minority student populations; most enrolled elementary and middle school students. Little is known about the effectiveness of these companies’ programs on student achievement, parental satisfaction, parental involvement, or school climate because few rigorous studies have been conducted. While the companies publish year-to-year comparisons of standardized test scores to indicate that students in schools they manage are making academic gains, they do not present data on comparable students who are not in their programs, a necessary component of a program effectiveness study. An effectiveness study attempts to isolate the effect a program has on specific outcomes by, among other things, comparing outcomes for students in company managed schools with those of a comparable group of traditional public school students, tracking students over time, testing students before and after exposure to the company’s program, and controlling for differences between these groups. Of five studies we identified concerning the three companies operating in the District, four focused only on one company. Moreover, of the five studies, one-- based on one school in Florida--was rigorous enough to allow confidence in the findings about the program’s effectiveness in that school. This particular study, using two analytical techniques, found no difference between students in the company’s program and other students. The remaining studies had methodological limitations that precluded assessments about a company’s effect on student achievement, parental satisfaction, parental involvement, and school climate. Additional research on one company’s program is planned by an organization with experience in conducting educational evaluations. Background: During the last decade, a new kind of entity has emerged in public education: the for-profit provider of education and management services. Historically, school districts have contracted with private companies for noninstructional services, such as transportation and food service, and have also relied on contractors in some cases to provide limited instructional services to specified populations. Until recently, public schools have generally not contracted for the comprehensive programs of educational and management services that these companies typically offer. In recent years, the options available to public schools considering contracting with private companies have steadily grown. Today, approximately 20 major companies manage public schools.[Footnote 1] Nationally, it is estimated that these companies as well as other smaller companies serve over 300 schools out of the nation’s approximately 92,000 public schools. Although these companies manage public schools at all grade levels, most such privately managed public schools are elementary and middle schools. In these public schools, companies generally provide the same kinds of educational and management services that school districts do for traditional public schools. Educational services typically include a curriculum as well as a range of services designed to enhance or support student achievement, such as professional development opportunities for teachers, opportunities for parental involvement and school environments that aim to facilitate student support. Management services typically include personnel, payroll, and facilities management. Although these are the services that are typically offered to schools, companies also may adapt their services to respond to the preferences or needs of individual schools. For example, while some companies offer a particular curriculum or educational approach, others appear more willing to work with the curriculum the school or school district has already adopted.[Footnote 2] Typically, companies provide their services to public schools in one of two ways. First, they can contract directly with school districts to manage traditional public schools; such schools are known as “contract schools.” Second, they can manage charter schools, which are public schools that receive a degree of autonomy and freedom from certain school district requirements in exchange for enhanced accountability. Generally, charter schools are run by individual boards of trustees, which in most states and the District of Columbia have the authority to decide whether to contract with a private company.[Footnote 3] Both contract schools and charter schools remain public schools, however, and are generally subject to federal and state requirements for public schools in areas such as the application of standardized tests and special education. While the reasons public schools turn to private companies vary, the potential to increase student achievement appears to be one factor. In particular, according to certain experts and company officials we spoke to, school districts that seek a company’s help often do so with the expectation of raising achievement in struggling or failing schools. While management services appear to be especially important for charter schools that contract with such companies, charter schools also consider the potential to raise student achievement or a particular educational approach consistent with the school’s mission, according to school officials and experts we spoke with. Both types of schools that seek these companies’ assistance--struggling schools and charter schools--appear concentrated in urban areas. Further, several of the major companies reportedly serve a predominantly disadvantaged urban and minority student population. Recent changes in federal law have implications for the role played by these companies in public schools. The No Child Left Behind Act of 2001[Footnote 4] requires that schools that fail to meet state student achievement standards for 5 consecutive years must be restructured by implementing one or more alternative governance actions. One of the alternatives available to states and districts is to contract with an education management company.[Footnote 5] Three companies currently operate in the District of Columbia: Edison Schools, Mosaica Education, and Chancellor Beacon Academies. Edison began operating its first District school in 1998, and Mosaica and Chancellor Beacon first contracted with the District schools they manage in 2001. Throughout this report, these companies will generally be discussed in this order. Mergers and acquisitions are common among such companies. In 2001, Edison acquired nine schools nationwide through a merger with LearnNow. In the same year, Mosaica acquired nine schools nationwide through its acquisition of Advantage Schools. In addition, Chancellor and Beacon merged into a single company. Such changes can have several outcomes: in some cases, the company may operate schools that continue to use the educational program of another company; in other cases, the school may consider adopting the educational program of the new company or terminating the contract. Education Management Companies Have Multifaceted Programs; District Schools Varied in Their Implementation: The companies that operate public schools in the District of Columbia offer management and educational services as part of their programs; the extent to which District schools managed by these companies implemented all of the components of the companies’ programs varied. All of these companies offer programs that include management and educational services, such as curricula that integrate technology and professional development opportunities for teachers. Of the 10 District schools managed by these companies, 4 had completely implemented their company’s program. In school year 2001-02, all 10 District schools managed by these companies were charter schools with predominantly poor and minority student populations; most enrolled elementary and middle school students. Similar to traditional public schools, the District schools managed by these companies were required to be open to all students, up to their enrollment limits, and to meet District standards in areas such as health, safety, standardized testing, and compliance with federal special education requirements. All Three Companies Offer Management and Educational Services to Schools Nationally: The three for-profit companies that operate in the District of Columbia[Footnote 6]--Edison, Mosaica, and Chancellor Beacon--share common elements in terms of the management and educational services they offer to schools nationwide as well as those company officials described as distinctive. Each of the three companies generally offers similar management services. For example, all three offer management services such as personnel, payroll and facilities management, services that can be important for charter schools. In addition, the three companies employ some common approaches designed to improve student achievement. All three companies offer an extended school day and year. All three integrate technology in their educational programs. For example, all three offer students access to classroom computers. Similarly, all organize schools into smaller units to facilitate their tracking of students’ progress. All three provide summer training to teachers as well as other forms of professional development. Additionally, all have activities designed to involve and support parents and students. For example, each company uses parent satisfaction surveys. Experts we spoke to noted that these same approaches were being used in some other public schools. Finally, officials of all three companies stated that their companies contributed positively to school climate--a sense of mission and an environment conducive to learning--and cited aspects of school climate such as a safe and orderly school environment and teacher motivation. In addition to the characteristics they had in common, company officials identified others they believed were distinctive. These include, for example, their programs’ curriculum and instruction as well as the ability to provide economies of scale, develop community partnerships, and provide strong administrative support. As Table 1 shows, all three companies provided their services to schools in multiple states in 2001-02. Table 1: Profiles of Three Education Management Companies Operating in the District of Columbia: Year company was established; Edison: 1992; Mosaica: 1997; Chancellor Beacon: 2001[A]. Number of states[B] where company operated schools in school year 2001- 02; Edison: 24; Mosaica: 8; Chancellor Beacon: 11. Number of schools company operated nationwide in school year 2001-02; Edison: 136; Mosaica: 21; Chancellor Beacon: 82. Number of schools company operated in the District of Columbia in school year 2001-02; Edison: 6; Mosaica: 2; Chancellor Beacon: 2. [A] Year of merger of Chancellor Academies (originally founded in 1999) and Beacon Education Management (founded in 1993). [B] Including the District of Columbia. Source: Edison, Mosaica, and Chancellor Beacon. [End of table] According to Edison officials, its program has a number of distinctive characteristics.[Footnote 7] The first of these is its curriculum, which emphasizes basic skills, especially reading as the basis for future learning. It also includes enrichment in areas such as world languages (e.g., Spanish) and art. Edison’s basic skills curriculum includes components developed by Edison, such as a remedial reading program, and other components that Edison states are supported by research, such as Chicago Math and the Success for All reading program. Instructional methods are a second characteristic of Edison’s program. Edison schools use a variety of instructional methods. One of these, direct instruction, relies on repetition and drill. Other methods use projects, small groups, and individualized lessons. A third characteristic of Edison schools is their use of assessments. According to Edison officials, their program uses frequent assessments and the results of these assessments are promptly provided to teachers to assess student needs and provide appropriate additional help. “Systems and scale” is another key characteristic of Edison schools according to company officials. The company views its schools as part of a national system linked by a common purpose, and because of the system’s size, the company says it is able to purchase supplies at lower costs. Mosaica officials also identified certain distinctive characteristics of their company’s program.[Footnote 8] The first is the program’s curriculum, which has two parts. According to Mosaica officials, its morning program features instruction in traditional subjects such as reading and math. In the afternoon, students use Paragon--Mosaica’s own curriculum. According to company officials, Paragon stresses multidisciplinary learning, uses projects to emphasize the humanities, and recognizes students’ different learning styles. For example, students may use their reading, math, and social studies learning to build a pyramid or a Viking ship and thus study a period of history. According to company officials, projects accommodate a variety of learning styles--for example, some students learn visually, others by performing. Community involvement is a second key characteristic of Mosaica’s program. Company officials say that Mosaica brings community support into the school by networking with various community organizations. According to company officials, this provides its schools with access to additional resources. Chancellor Beacon officials also identified distinctive characteristics of their program. One is their willingness to customize their educational program to meet the needs and preferences of local schools. For example, in response to community interest, some Chancellor Beacon schools feature a cultural heritage element in the curriculum while one of its schools emphasizes the environment. Chancellor Beacon’s own curriculum was recently finalized in July 2002 and is based on an integration of the curricula of Chancellor and Beacon before they merged. One component of its curriculum is Core Knowledge--a program that expects students to master specific content in language arts, history, geography, math, science and fine arts. Other components emphasize ethics, morality and community volunteerism. A second key characteristic of Chancellor Beacon’s program is its operational support, according to company officials. These officials told us that in focusing on operational support, Chancellor Beacon allows schools to focus on academics. While the Chancellor Beacon program emphasizes customization as a key characteristic, the other two companies also allow schools to modify their programs. For example, in its reading program, Edison allows schools some flexibility regarding what books to read and in what order. In addition, up to one-fourth of its curriculum can be determined by the local school. Similarly, Mosaica allows its schools to use different approaches or materials in their morning session. Of the 10 District Schools, 4 Completely Implemented the Company’s Educational Program While 6 Schools Selected Elements of Their Companies’ Programs or Chose Other Programs: While all of the 10 District schools managed by the companies during the 2001-02 school year obtained management services from these companies, the schools were more selective in implementing the companies’ educational programs. Of the 10 District schools, 4 have completely implemented the companies’ educational programs and 6 have adopted selected elements of their companies’ programs or chosen other programs, typically those of a previous company. A key factor that helps explain the difference between the programs the companies offer and what has been implemented by District schools is that recent mergers and acquisitions have led to changes in management companies in these 6 schools; these schools have generally left in place the educational programs of the companies that formerly managed them. Four schools, all managed by Edison, implemented the company’s educational program completely, according to company officials. These 4 schools all opened in 1998 as the result of a partnership between Friendship House, a nonprofit community organization serving District children and youth since 1904, and Edison. According to a Friendship House official, these schools completely implemented Edison’s program because they saw it as complementing their own goals. One of these schools--a high school--has supplemented the Edison program by developing a program to expose certain students to college through campus visits and workshops for parents. Six District schools adopted selected elements of their companies’ educational programs or chose other educational programs. These 6 schools include 2 schools managed by Edison, 2 by Mosaica, and 2 by Chancellor Beacon. All 6 schools have had recent changes in management companies as a result of mergers or acquisitions. The 2 schools that received services from Edison have opted to retain the curriculum already in place at the schools, rather than adopt the Edison program.[Footnote 9] In 2001, Edison bought LearnNow, the company that formerly provided services to the 2 schools. According to an Edison official knowledgeable about the schools formerly managed by LearnNow, the primary difference between the companies’ curricula was in elementary language arts, for which LearnNow preferred a different reading program than Success for All, which the Edison program uses in its other schools. The 2 schools managed by Mosaica have adopted some elements of the company’s educational program, and have plans to adopt more by 2003. In 2001, Mosaica bought Advantage, the company that formerly managed these schools. Both schools retained an instructional approach put in place by the previous company. This approach--direct instruction--emphasizes drill and repetition. By school year 2003, both schools expect to use direct instruction during the morning session and Paragon in the afternoon. The 2 schools managed by Chancellor Beacon both had distinct curricula in place before being managed by this company; one has combined its existing curriculum with elements of Chancellor Beacon’s, and the other has left its existing curriculum in place. The school that has adopted elements of Chancellor Beacon’s curriculum has done so by integrating the company’s language arts and math curriculum with the school’s existing curriculum, according to company officials.[Footnote 10] This school, which serves at-risk youth, had a curriculum called expeditionary learning, which focuses on learning through field trips and experiences. The other Chancellor Beacon school opted to retain its existing basic-skills curriculum, relying instead on the company’s management services and selected educational services, such as assessments. Chancellor Beacon officials support the schools’ choices regarding what company components to adopt. Company and school officials identified several reasons why these 6 schools did not completely implement the current company’s educational program, opting instead to continue with an existing curriculum. These included continuity for students, the company’s flexibility with regard to local customization, and the right of charter school boards to make broad curriculum decisions.[Footnote 11] The 10 schools in the District managed by these companies shared certain characteristics and served similar student populations in 2001- 02. All were public charter schools governed by their own boards and accountable to District oversight authorities. Most (9) were combined schools spanning elementary and middle school grades. As public schools, they were required to accept any student who applied, up to their enrollment limit. Their student populations were substantially minority and poor: 92 to 100 percent African American and 48 to 95 percent receiving free or reduced school lunch.[Footnote 12] All served some students with special needs, such as learning disabilities: in 9 of the schools, the percentage ranged from 5 to 13 percent, and in one school, 32 percent of the student population had special needs. All but one served no or very few students with limited English proficiency; at the remaining school, students with limited English proficiency represented about 12 percent of all students enrolled. Limited Research Exists of the Effectiveness of These Companies’ Programs: Little rigorous research exists on the effectiveness of the three educational management companies--Edison, Mosaica, and Chancellor Beacon--in the schools they manage across the country; as a result, we cannot draw conclusions about the effect that these companies’ programs have on student achievement, parental satisfaction, parental involvement, or school climate. Students in company managed schools have demonstrated academic progress, but more research is needed to determine if this improvement is directly the result of the companies’ programs and if this progress is different from that of comparable students in traditional public schools. We reviewed five studies that addressed student achievement, but only one was conducted in a way that allows an assessment of the effect the company’s program had on student achievement in one school. The remaining studies had methodological limitations that precluded such assessments. In an effort to learn more about effectiveness, Edison has recently commissioned RAND, a nonprofit research organization that has evaluated educational reforms, to complete a study to assess its program’s impact. Determining the effect of an educational company’s program can be challenging for researchers. Ideally, evaluations of program effectiveness should involve a comparison of outcomes for one group exposed to a particular program with outcomes of a second group not exposed to the program. Some evaluations assign participants randomly to one group or the other to increase the likelihood that the two groups are roughly equivalent on all characteristics that could affect outcomes. This technique of random assignment is often problematic in educational research because public school enrollment is generally based on residency requirements. Therefore the most common way to compare student achievement results from two different groups of students is to ensure the groups are similar in a number of ways, including socioeconomic status, ethnicity, and performance on prior academic assessments. In addition to controlling for the effects of these background characteristics, it is critical to follow the performance of students over time, preferably before any group has been exposed to the program, and at least one point thereafter.[Footnote 13] It is also beneficial to analyze individual student data, rather than grade or school-level averages, to account for individual differences and to factor in the effects of missing data. Within the context of rigorous educational program evaluations, various measurements can be used to capture a student’s performance on standardized tests. According to several experts, it is important to examine both the percent of students in a particular grade or school making yearly gains and the distribution of these gains across ability levels to ensure that students of all achievement levels are demonstrating academic growth. Another point of interest relates to the length of time students participate in a particular program. Some experts claim that students will exhibit greater gains the longer they participate in a program. However, it is particularly challenging to design studies that address this claim, because educational companies are still a relatively new phenomenon. We identified five studies concerning the three companies operating in the District that met the criteria for our review: inclusion of comparison groups, measurement over time, and focus on academic achievement, parental satisfaction, parental involvement, or school climate.[Footnote 14] All of the studies addressed the effectiveness of schools managed by Edison. One study also addressed the effectiveness of schools managed by all three private companies--Edison, Mosaica, and Chancellor Beacon.[Footnote 15] We were unable to identify any rigorous studies that included analysis of District public schools managed by any of these three companies.[Footnote 16] Of the studies included in our review, four studies addressed only outcomes related to student achievement, while one study addressed student achievement and other outcomes such as parental satisfaction and school climate.[Footnote 17] Only one of the studies, A Longitudinal Study of Achievement Outcomes in a Privatized Public School: A Growth Curve Analysis, based on one Edison school in Miami-Dade County, Florida, was conducted in a way that allows an assessment of the program’s effect on student achievement. This study followed individual student standardized test scores over a 3-year period and found that Edison students progressed at similar rates to those in the traditional Miami-Dade County Public Schools (MDCPS); this finding is not generalizable to other schools managed by Edison or any other private company. The study was designed to ensure that the Edison students were similar to the random sample of students drawn from MDCPS in terms of school grade, socioeconomic status, as indicated by the percent eligible for free/reduced price lunch, ethnicity, and achievement levels, as indicated by comparability in test scores prior to students enrolling in the Edison school. The study employed two different analytical techniques and both resulted in the finding that the Edison students progressed at similar rates to the traditional public school students. Several methodological techniques that would have strengthened its overall findings could have been employed. These include controlling more specifically for school-level differences between the participating students as well as better ensuring the two groups of students remained equivalent despite study dropouts (subsequently referred to as attrition). Differences in the composition of these groups, after attrition, could affect the test score results. This study did not examine the effect of this company’s program on parental satisfaction, parental involvement, or school climate. Significant limitations in the other four studies preclude our making assessments of the effectiveness of schools managed by Edison, Chancellor Beacon, or Mosaica that were included in the studies.[Footnote 18] These limitations included use of comparison groups that did not adequately control for differences between the students in the company’s schools and the students in traditional public schools, instances where achievement data were not available for all students, and lack of adjustment for high attrition rates. Company officials report that one way to determine if their programs are effective is to assess whether students demonstrate academic growth as evidenced by improvement on standardized tests. There is evidence to support the assertion that students enrolled in schools managed by Chancellor Beacon, Mosaica, and Edison have demonstrated academic improvement from one point in time to another, but it is important to determine if these gains are specifically the result of company programs. Additional research is in progress. Edison commissioned RAND to evaluate Edison schools across the country. Where possible, RAND plans to compare the scores of individual Edison students to those of traditional public schools students with similar characteristics. Since it is often difficult to gather individual level student data, RAND will also compare Edison data, either at the grade or school level, to publicly available state data at that same level. RAND expects to publish its findings in 2004. Agency Comments: We received written comments on a draft report from the Department of Education. These comments are presented in appendix III. Education stated that there are insufficient data on the effectiveness of private education companies. Education also stated that it encourages others’ evaluation efforts. We also received comments from an expert on private education companies, the authors of the MDCPS study that we assessed, the District of Columbia Board of Education, the District of Columbia Public Charter School Board, as well as Edison Schools, Mosaica Education, and Chancellor Beacon Academies. These comments were also incorporated where appropriate. We are sending a copy of this report to the Secretary of Education, the District of Columbia Board of Education, the District of Columbia Public Charter School Board, Edison Schools, Mosaica Education, and Chancellor Beacon Academies. We will make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please call me at (202) 512-7215. Other contacts and contributors to this report are listed in appendix IV. Sincerely yours, Marnie S. Shaul, Director Education, Workforce, and Income Security Issues: Signed by Marnie S. Shaul [End of section] Appendix I: Objectives, Scope, and Methodology: Objectives: The objectives of our review were to (1) identify the characteristics of the for-profit educational management companies operating in the District and determine the extent to which District schools managed by these companies have used their programs and (2) determine what is known about the effectiveness of these programs, as measured primarily by student achievement. We conducted our work between January and September 2002, in accordance with generally accepted government auditing standards. Scope and Methodology: To identify the characteristics of the programs offered by for-profit companies operating in the District, and determine the extent to which District public schools managed by them have used their programs, we interviewed company officials, representatives of the 10 schools, as well as officials of the District’s chartering authorities. We collected information on the companies from their Web sites and obtained technical comments from the companies on the descriptions of their programs. We also contacted education experts and advocates to obtain both their recommendations on research regarding the three for- profit companies and information on any research they might have conducted on the companies. We also acquired information on the companies by reviewing relevant research summaries. We also observed an on-site review of one school’s program conducted for District oversight authorities. To determine what is known about the effectiveness of these programs, we collected, reviewed, and analyzed information from available published and unpublished research on the effect on student achievement, parental satisfaction, parental involvement, and school climate of the three companies managing schools in the District. We also spoke with RAND officials about the design and methods of their current evaluation of Edison Schools. To identify relevant research, we followed three procedures: (1) interviewed experts to find out what studies were completed or in the process of being completed on the effectiveness of company programs; (2) conducted library and Internet searches; and (3) reviewed bibliographies of studies that focused on the effectiveness of company programs. We reviewed studies concerning the three companies operating in the District that met the following criteria: included comparison groups and measurement over time, and focused on academic achievement, parental satisfaction, parental involvement, or school climate. Our final list of studies for review consisted of five studies, as listed in appendix II. We did not identify any studies that evaluated the effect of these three programs in District schools. Two GAO social scientists examined each study to assess the adequacy of the samples and measures employed, the reasonableness and rigor of the statistical techniques used to analyze them, and the validity of the results and conclusions that were drawn from the analyses. For selected studies, we contacted the researchers directly when we had questions about their studies. [End of section] Appendix II: Studies and Other Information Sources Considered: In order to identify research that explicitly addresses the effect on student achievement, parental satisfaction, parental involvement, or school climate of the three companies managing schools in the District, we interviewed experts to determine what studies were completed or in the process of being completed, conducted library and Internet searches, and reviewed bibliographies of studies that focused on the effect of these companies’ programs on student achievement. Although five studies met our criteria for review (inclusion of comparison groups, measurement over time, and focus on academic achievement, parental satisfaction, parental involvement, or school climate), we cannot draw conclusions, due to methodological weaknesses, from the four studies listed below.[Footnote 19] Conclusions from A Longitudinal Study of Achievement Outcomes in a Privatized Public School: A Growth Curve Analysis were presented in the text. * Miron, Gary and Brooks Applegate. An Evaluation of Student Achievement in Edison Schools Opened in 1995 and 1996. Kalamazoo, Michigan: The Evaluation Center, Western Michigan University, December 2000. * Miron and Applegate analyzed both individual and aggregate level data and compared improvements in the test scores of 10 Edison schools with those of comparison schools, districts, states, and national norms, where applicable. However, significant weaknesses prevented conclusive statements on the effects of Edison schools. These weaknesses included limitations in the available data, such as incompleteness and inconsistency, high attrition rates, and the lack of corresponding adjustments for attrition. * Horn, Jerry and Gary Miron. An Evaluation of the Michigan Charter School Initiative: Performance, Accountability, and Impact. Kalamazoo, Michigan: The Evaluation Center, Western Michigan University, July 2000. * Horn and Miron examined the percentage of students earning a passing grade on achievement tests in individual charter schools in Michigan in comparison with the percentage passing in the districts where these schools were located.[Footnote 20] The analysis included schools managed by Edison, Mosaica, and Beacon. Weaknesses included inadequate controls for differences between the students in charter schools and their host districts, no consideration of attrition rates, and the likelihood that analyses were often based on a small number of students. * American Federation of Teachers. Trends in Student Achievement for Edison Schools, Inc.: The Emerging Track Record. Washington, D.C.: October 2000. * Researchers examined school and grade-level achievement data from 40 Edison schools in eight states and compared it to data gathered from school districts and other schools. Weaknesses included insufficient information about the methodology employed by the states, including construction of comparison groups and matching techniques, and a lack of analysis of attrition rates. * Gomez. Ph.D., Joseph and Sally Shay, Ph.D. Evaluation of the Edison Project School. Final Report, 1999-00 (portions related to parental satisfaction and involvement, and school climate). Office of Evaluation and Research, Miami-Dade County Public Schools (MDCPS), April 2001.[Footnote 21] * Gomez and Shay examined responses from surveys MDCPS had administered to parents and teachers from both the Edison school and the control group. However, the outcomes related to parental satisfaction and involvement were measured with single-item survey questions that do not seem to capture the full context of the concepts. School climate was measured with a single-item question on a teacher survey and with school archival data.[Footnote 22] Shay and Gomez did not report whether any differences are statistically significant, in part because they acknowledged it would be inappropriate to conduct tests of significance on single-item questions. Therefore, there is no evidence to determine whether Edison school parents were more satisfied or involved than those in the control group, or whether the Edison school improved school climate. We are aware of other studies and reports that address the effect of Chancellor Beacon Academies, Mosaica Education, and Edison Schools on academic achievement, parental satisfaction, parental involvement, or school climate; however, the following are examples that did not meet the criteria for inclusion in our review. * District of Columbia Public Charter School Board. School Performance Reports. Washington, D.C.: August 2001. * Department of Research, Evaluation, and Assessment, Minneapolis Public Schools. Edison/PPL School Information Report 2000-2001. Minneapolis, Minnesota: 2001. * Department of Administration, Counseling, Educational and School Psychology, Wichita State University. An Independent Program Evaluation for the Dodge-Edison Partnership School: First Year Interim Report. Wichita, Kansas: 1996. * Missouri Department of Elementary and Secondary Education. Charter School Performance Study: Kansas City Charter Schools. Jefferson City, Missouri: 2001. * Company-provided information such as annual reports and school performance reports. Other sources of general information included school district websites and other educational services, such as Standard and Poor’s School Evaluation Services and the National Association of Charter School Authorizers’ Educational Service Provider Information Clearinghouse. [End of section] Appendix III: Comments from the Department of Education: UNITED STATES DEPARTMENT OF EDUCATION: THE DEPUTY SECRETARY: October 8, 2002: Ms. Marnie S. Shaul: Director, Education, Workforce and Income Security Issues, General Accounting Office: Washington, DC 20548: Dear Ms. Shaul: Thank you for the opportunity to review and comment on the draft report, “Public Schools: Insufficient Research to Determine Effectiveness of Selected Private Education Companies.” Across the country, we see how public education is being transformed by expanding the options and choices available to parents. For example, when the school year opened last month, there were over 2,700 charter schools operating in 39 States nationwide, an impressive feat considering the charter school movement is just a decade old. Charter schools, which are public schools of choice that operate free from many State and local district rules in exchange for heightened accountability, allow a variety of different groups to enter the public education market and manage schools. Some of these new managers include local community groups and businesses, as well as private education management organizations (EMOs), like those identified in your report. As you also indicate in your report, many of these EMOs also operate traditional public schools on a contract basis. Are these companies effective in boosting student achievement? Unfortunately, as your study details, there is insufficient data on the performance of EMOs due to the relatively small number of schools they manage and other methodological constraints. The only way to learn more about charter schools, alternate governance arrangements (such as schools managed by EMOs), and other reforms is through further experimentation and research. Those of us in public education should embrace these exciting new innovations that hold out the promise of boosting student achievement. Unfortunately, we know that too many of our schools are not performing at the levels they should, so we must explore alternatives while rigorously examining their results. Parents too must be free to choose which schools best suit their children’s needs. I applaud the work of States and local communities in broadening the choices available to parents and experimenting with new, innovative approaches to the delivery of public education. In the future, I know there will be a deeper body of research available and I encourage the GAO and other organizations to continue their evaluation efforts. Together, we will work to ensure that no child is left behind. Sincerely, William D. Hansen Signed by Bill Hansen Appendix IV: GAO Contacts and Staff Acknowledgments: GAO Contacts: Harriet Ganson, (202) 512-7042 Chris Morehouse, (202) 512-7214: Acknowledgments: In addition to those named above, Rebecca Ackley and N. Kim Scotten made key contributions to this report. Jay Smale, Michele Fejfar, Kevin Jackson, Sara Ann Moessbauer, and Shana Wallace provided important methodological contributions to the review of research. Patrick Dibattista and Jim Rebbe also provided key technical assistance. [End of section] Related GAO Products: School Vouchers: Characteristics of Privately Funded Programs. GAO-02- 752. Washington, D.C.: September 26, 2002. School Vouchers: Publicly Funded Programs in Cleveland and Milwaukee. GAO-01-914. Washington, D.C.: August 31, 2001. Charter Schools: Limited Access to Facility Financing. GAO/HEHS-00-163. Washington, D.C.: September 12, 2000): Charter Schools: Federal Funding Available but Barriers Exist. HEHS-98- 84. Washington, D.C.: April 30, 1998. Charter Schools: Recent Experiences in Accessing Federal Funds. T-HEHS-98-129. Washington, D.C.: March 31, 1998. Charter Schools: Issues Affecting Access to Federal Funds. T-HEHS-97- 216. Washington, D.C.: September 16, 1997. Private Management of Public Schools: Early Experiences in Four School Districts. GAO/HEHS-96-3. Washington, D.C.: April 19, 1996. Charter Schools: New Model for Public Schools Provides Opportunities and Challenges. GAO/HEHS-95-42. Washington, D.C.: January 18, 1995. School-Linked Human Services: A Comprehensive Strategy for Aiding Students At Risk of School Failure. GAO/HRD-94-21. Washington, D.C.: December 30, 1993. FOOTNOTES [1] In this context, “major” refers to those companies that manage at least 3 schools and operate in multiple states. [2] In fact, not all companies offer a particular curriculum or educational approach, focusing instead on management services. For example, Arizona-based ABS concentrates on management services like recruitment, personnel, payroll, and facilities management. [3] However, in Arizona, private companies may run charter schools directly. [4] P.L. 107-110, Jan. 8, 2002. [5] Others include reconstituting by replacing teachers, principals, or both; converting to public charter school status; and turning over school operation to the state educational agency. [6] In school year 2001-02, a fourth for-profit company, Richard Milburn Academies, continued to operate a District public charter school, pending a final decision by District oversight authorities to revoke its charter. The school closed in June 2002. [7] Edison officials describe the educational approach as consisting of “10 fundamental” characteristics: school organization, extended school day and year, curriculum, instructional methods, assessments, professional teaching environment (including professional development), use of technology, partnership with families, instructional programs that reflect community interests, and the benefits of “systems and scale.” [8] Mosaica officials describe the educational approach as consisting of “seven pillars” or characteristics: the Paragon curriculum, professional development, use of technology, extended school day and year, parental involvement, community involvement, and school climate. [9] According to an Edison official, as of September 2002, Edison and one of these 2 schools were planning a transition that would completely discontinue Edison’s services to the school and return the school to management by its own board of trustees. The services that Edison provided to both schools in the 2001-02 school year were limited to management services, such as payroll. [10] This school’s existing educational approach is tied to a 3-year federal comprehensive school reform demonstration grant. The grant expires at the end of the 2004 school year. [11] In addition, according to officials of the agencies that oversee District charter schools, substantial curriculum changes must be officially approved. [12] Data were not available for all schools; these ranges are based on 8 schools (percent African American) and 9 schools (percent receiving subsidized lunch). [13] Experts varied with regard to how many years of data are sufficient for analysis. Generally, experts we spoke to indicated that 3 to 5 years of data would be sufficient for analysis. [14] While the study, Achievement Performance Report: Dallas-Edison Partnership Schools 2001-02 (Dallas Division of Evaluation and Accountability, Dallas Independent School District, 2002), met the criteria for inclusion, we were unable to review it because it was published after completion of the review. [15] The evaluation included 18 schools managed by Beacon and was completed prior to the merger of Chancellor and Beacon. [16] One study compares academic achievement in District charter schools and traditional public schools, but does not distinguish charter schools managed by private companies. See Jeffrey Henig et al. Growing Pains: An Evaluation of Charter Schools in the District of Columbia; 1999-2000. (Washington, D.C.: The Center for Washington Area Studies, The George Washington University, 2001). [17] We did not review the student achievement portion of the Miami- Dade County Public Schools Evaluation because the same data were analyzed, with very similar results, by the same researcher in another study in the review. [18] Please see appendix II for the studies and other information sources considered for our review. [19] While the study, Achievement Performance Report: Dallas-Edison Partnership Schools 2001-02 (Dallas Division of Evaluation and Accountability, Dallas Independent School District, 2002), met the criteria for inclusion, we were unable to review it because it was published after completion of the review. [20] Analyses of individual schools were included in an appendix provided upon request from the authors. We included this study in our review because it explicitly addressed the results of schools managed by the three companies in this report, unlike other studies that did not disaggregate school results by company. [21] We only examined the portion of this study that relates to the effect of the Edison school on parental satisfaction and involvement and school climate. We examined the data and analysis that related to the academic achievement piece in Dr. Shay’s dissertation, A Longitudinal Study of Achievement Outcomes in a Privatized Public School: A Growth Curve Analysis, reported in the text. [22] This includes data on students’ attendance rates, student mobility, indoor and outdoor suspensions, and the teacher-student ratio. GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: