July 2, 2007





Methodology is too weak to support finding that student achievement has increased since passage of NCLB


Contact: Teri Battaglieri (248) 444-7071 (email)

    John T. Yun (805) 893-2342 (email)

    Kevin Welner (303) 492-8370 (email)


EAST LANSING, Mich.— A report released last month by the Center on Education Policy (CEP) is being used to argue that student achievement has increased since the implementation of the federal No Child Left Behind law. A review of this report finds that it suffers from important weaknesses and that the wording of numerous findings and key conclusions imply a much stronger connection between NCLB and increased achievement than can be substantiated by the data.


The report, Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind was reviewed for the Think Twice think tank review project by John T.Yun, a professor at the University of California at Santa Barbara.


The CEP’s report has already received widespread attention from the news media, including front-page coverage in the Washington Post. The U.S. Secretary of Education immediately pointed to the report as confirming NCLB’s success. As reviewer Yun notes, the report “is likely to be cited often in the upcoming debate on the reauthorization of the No Child Left Behind Act (NCLB).”


While Yun credits the report with attempting “to carefully analyze the complex issue of test score improvement before and after the implementation of NCLB in 2002,” he describes how shortcomings in the data and analyses may have “resulted in a much more optimistic picture of the impact of the legislation than the data warrant.” Additionally, Yun says, while the report’s title may convey the impression that it seeks to examine the direct impact of NCLB on student achievement, the report itself acknowledges that an analysis accomplishing this goal may be impossible. The possible effects of NCLB cannot be disentangled from the possible effects of the large number of other state and local policies aimed at raising achievement during the same period of time.


The review from Professor Yun contends that the most useful and important finding in the new report is its explanation of current weaknesses in state data availability. The authors had great difficulty in obtaining and analyzing state-level achievement data that should be readily available; until this situation is improved, researchers such as those at CEP will be faced with many of the same obstacles encountered here. But Yun stresses that this finding was given far too little attention and should not have been overshadowed by the problematic analyses of student achievement.


Regarding the student achievement analyses, Yun does credit the report with offering thoughtful approaches and concludes that the report does represent progress toward more comprehensive examination of the outcomes of the law. He also commends the report for cautionary notes that help readers understand some of its limitations—yet not the three that he identifies as most serious:





These limitations seem particularly salient given the report’s prompt interjection into the public debate. U.S. Secretary of Education of Education Margaret Spellings, for instance, used the report to argue for NCLB reauthorization. Her official statement said, “This study confirms that [NCLB] has struck a chord of success with our nation’s schools and students. … We know the law is working, so now is the time to reauthorize [it].”


Commenting on this new review in light of such reaction the report, Professor Kevin Welner of the University of Colorado at Boulder, co-director of the Think Twice project, focused on the issue of publicity. “The methodological problems pointed out by Professor Yun are important and should be carefully considered by any policy maker or researcher who makes use of the study. But the bigger problem here seems to be in the packaging and subsequent publicity. Neither the data nor the analyses in the report are anywhere near strong enough to meaningfully support the report’s title, ‘Answering the question that matters most,’ nor can the report support the sort of puffery we see from Secretary Spellings.”


Find the complete review by John Yun as well as a link to the Center on Education

Policy report at:


About Think Twice

The Think Twice project provides the public, policy makers and the press with timely, academically sound reviews of selected think tank publications. It is a collaboration of the Education Policy Studies Laboratory at Arizona State University and the Education and the Public Interest Center at the University of Colorado at Boulder and is funded by the Great Lakes Center for Education Research and Practice.


The mission of the Great Lakes Center for Education Research and Practice is to identify, develop, support, publish and widely disseminate empirically sound research on education policy and practices designed to improve the quality of public education for all students within the Great Lakes Region. 

Visit the Great Lakes Center website at: