http://www.greatlakescenter.org
http://www.greatlakescenter.org
Campaign to Market Florida's Grab-bag of Reforms is an Exercise in Misused Data Heritage Report is latest in series of misleading analyses Contact: Teri Battaglieri (248) 444-7071; greatlakescenter@greatlakescenter.org EAST LANSING, Mi. (November 30, 2010) – Did a collection of Florida education policies—ranging from grade retention to school choice and virtual schools—improve achievement and narrow the test-score gap? A recent Heritage Foundation report is part of a larger campaign to convince us that the answer to that question is "yes"—but a new review finds serious flaws in the Heritage report which render it of little value to policymakers. The Heritage report, Closing the Racial Achievement Gap: Learning from Florida's Reforms, was reviewed for the Think Twice think tank review project by Madhabi Chatterji, a professor of educational measurement and evaluation at Teachers College at Columbia University who has conducted considerable research of Florida schools. The review was produced by the National Education Policy Center (NEPC), housed at the University of Colorado at Boulder School of Education, with funding from the Great Lakes Center for Education Research and Practice. The Heritage report, authored by Matthew Ladner and Lindsey Burke, contends that Florida's "far-reaching" education policies have caused test scores to increase and the achievement gap to narrow. In particular, the report focuses on fourth-grade reading scores on the National Assessment of Educational Progress (NAEP). These research claims are also made by Dr. Ladner, the Vice President of Research at Arizona's free-market Goldwater Institute, in a dozen other reports and articles similar to the Heritage report reviewed by professor Chatterji. The claims, however, do not withstand scrutiny. "The report's key conclusions are unwarranted and insufficiently supported by research," Chatterji states in her review. Most importantly, she points out the very direct effects of the state's grade-retention policy, causing the report's comparisons to be largely meaningless. By analogy, consider growth in height instead of growth in test scores. If two states wanted to measure the average height of their fourth graders, but one state (Florida) first identified the shortest 20% of third graders and held them back to grow an additional year before measurement, the study's results would not be useful. That, in brief, is the key problem that professor Chatterji identifies with the Heritage report. Florida's retention policy, instituted in 2002, focuses on third graders, who are held back when their reading scores are low. The Heritage report focuses on NAEP fourth-grade reading scores. Low scoring readers—mostly black and Hispanic—were screened out of grade 4 tests, which resulted in inflated and erroneous fourth grade scores. The review also points out that NAEP scores at other grade levels and even NAEP scores in fourth-grade math do not show the same jump as NAEP fourth grade reading. That is, the report cherry-picks the best data. Also, even if scores in Florida are in fact increasing, the report's methods are too weak to allow for a causal inference. The report uses only descriptive test score trends to compare states and then make sweeping generalizations. Moreover, many other changes occurred in Florida during the period analyzed, including the phasing in of one of the nation's most ambitious class-size reduction reforms—yet the report never mentions these other possible causes of any improvements. "In sum, the report's analyses are highly biased and of very limited value," Chatterji concludes. "The major elements of Florida's education reform policies are in need of continuing and more careful examination, individually and collectively, before they can be recommended for wider policy adoption." "Notwithstanding the clear problems with Dr. Ladner's analysis, he has done an excellent job in marketing," says Kevin Welner, professor of education at the University of Colorado and director of NEPC, "He has repackaged his analysis for at least seven individual states, showing each state's purported failings relative to Florida. This is how policy gets made—and it isn't pretty." Find Madhabi Chatterji's review on the Great Lakes Center website at: http://greatlakescenter.org/docs/Think_Twice/TT_Chatterji_Fla.pdf Find Closing the Racial Achievement Gap: Learning from Florida's Reforms by Matthew Ladner and Lindsey Burke on the web at: The Think Twice think tank review project, a project of the National Education Policy Center, provides the public, policy makers and the press with timely, academically sound reviews of selected think tank publications. The project is made possible in part by the generous support of the Great Lakes Center for Education Research and Practice. The review is also available on the National Education Policy Center website at: ### |
The mission of the Great Lakes Center is to improve public education for all students in the Great Lakes region through the support and dissemination of high quality, academically sound research on education policy and practices. Visit the Great Lakes Center Web Site at: http://www.greatlakescenter.org |