ELL Report Asks a Question It Can't Answer
CEP publication has significant weaknesses, according to new review
EAST LANSING, Mi., (May 19, 2010) – A recent Center on Education Policy (CEP) report asks, Has Progress Been Made in Raising Achievement for English Language Learners? and finds that states have generally made progress in increasing the achievement of English Language Learners (ELLs) under No Child Left Behind. But a review of that report finds that the conclusions drawn are not supported by the evidence presented.
In his review, MacSwan, finds two principal flaws with the study. The first is that the report fails to adequately account for the fact that the language in which a person is taught can be a serious source of measurement error for ELL achievement test scores.
The more sweeping problem, however, is in the source of data that the CEP report uses to draw its conclusions. The report relies on the percentages of ELLs in each year's cohort for a given grade level who meet NCLB-related adequate yearly progress (AYP) standards. Because the makeup of those cohorts changes each year, "drawing growth-related conclusions from [these data] is inappropriate," MacSwan writes.
Moreover, while the report's title, "Has progress been made in raising achievement for English language learners?" asks readers to infer a direct causal relationship between changes in reported percentages of ELLs meeting AYP and improvements in academic achievement for ELLs, no such inference can legitimately be drawn from these data. And the report all but ignores a wide range of plausible, competing explanations for changes in scores over time other than actual improvement of achievement levels.
"A causal inference would require minimally that competing explanations be eliminated," MacSwan writes. "While the report notes these potential complications in the data, they are apparently not taken seriously enough to prevent the authors from drawing very specific conclusions about the achievement trajectories of ELLs."
The result is a report that "is not a useful guide for policy or practice related to ELLs," MacSwan concludes. "The data analyses presented do not provide useful insight into ELL progress, and no judgments can be made regarding the relative success of current federal policy based on the report."
Find Jeff MacSwan's review and a link to the CEP report on the web at: http://www.greatlakescenter.org
Think Tank Research Quality: Lessons for Policy Makers, the Media, and the Public, a new book based on the work of the Think Twice/ Think Tank Review Project, is now available at: http://www.greatlakescenter.org/docs/think_twice_research_quality/Order_Form.pdf
About The Think Twice Project
The mission of the Great Lakes Center is to improve public education for all students in the Great Lakes region through the support and dissemination of high quality, academically sound research on education policy and practices.
Visit the Great Lakes Center Web Site at: http://www.greatlakescenter.org