Highly Touted Charter School Study Doesn't Stand Up To Scrutiny
Reviewer finds serious statistical flaws, bias in research on NYC charter schools
Contact: Teri Battaglieri, (248) 444-7071; firstname.lastname@example.org
The report, How New York City Charter Schools Affect Achievement, was written by Caroline Hoxby, Sonali Murarka, and Jenny Kang. When it was released, it was enthusiastically and uncritically embraced by charter advocates as well as a number of major media outlets across the country as being the definitive word on charter school effectiveness.
Because of the proclaimed importance of the new report, Sean Reardon, an expert on research methodology, was asked to review the report's strengths and weaknesses for the Think Twice think tank review project. Reardon, like the report's lead author Hoxby, is a professor at Stanford University.
The Hoxby report estimates the effects on student achievement of attending a New York City charter school rather than a traditional public school. A key finding, repeated in press reports throughout the U.S., compares the cumulative effect of attending a New York City charter school for nine years (from kindergarten through eighth grade) to the magnitude of average test score differences between students in Harlem and the wealthy New York community of Scarsdale. The report estimates this cumulative effect at roughly 66% of the "Scarsdale-Harlem gap" in English and roughly 86% of the gap in math.
In his review, Reardon observes that the report "has the potential to add usefully to the growing body of evidence regarding the effectiveness of charter schools." New York charter schools' use of randomized lotteries to admit students to charter schools offers the possibility that the study of those schools can roughly approximate laboratory conditions.
But Reardon points out that the report's key findings are grounded in an unsound analysis--an inappropriate set of statistical models--and that the report's authors never provide crucial information that would allow readers to more thoroughly evaluate "its methods, results, or generalizability."
Reardon's review notes these shortcomings in the report:
Reardon explains that, as a result of the flaws in the report's statistical analysis, the report "likely overstates the effects of New York City charter schools on students' cumulative achievement, though it is not possible -- given the information missing from the report -- to precisely quantify the extent of overestimation" This, as well as the lack of detailed information in the report to assess the extent of that bias, make it impossible for readers to know whether the report's estimated charter school effects are in fact valid.
"Policymakers, educators, and parents should therefore not rely on these estimates until the bias issues have been fully investigated and the analysis has undergone rigorous peer review."
Find Sean Reardon's review as well as a link to the NYC report by Hoxby and her colleagues on the web at: http://www.greatlakescenter.org.
About The Think Twice Project
The mission of the Great Lakes Center is to improve public education for all students in the Great Lakes region through the support and dissemination of high quality, academically sound research on education policy and practices.
Visit the Great Lakes Center Web Site at: http://www.greatlakescenter.org