Mathematica Study of Alternative Teacher Certification Does Not Apply to Vast Majority of American Classrooms, Experts Say
March 10, 2009

They Note Methodological Limitations and Caution against Relying on the Report to Inform Teacher Certification Policy

Contact: Patrick Richards – (703-237-2554); Patrick@exemplarpr.com

EAST LANSING, Mi., (March 10, 2009) – Despite the headlines announcing that a recent Mathematica Policy Research evaluation found alternative teacher certification to be as effective as traditional teacher preparation programs, a closer look by several education researchers shows that such conclusions are not supported by the study’s actual data.

Commissioned by the federal Institute of Education Sciences, the Mathematica study reports that teachers from most alternative certification programs have similar impact on student achievement as teachers from traditional certification routes.  But the report’s conclusions are based on a selective interpretation of results from a sample of schools and teachers that is dissimilar from the general population of schools in the United States, two new critiques have found.

One review of the Mathematica report was prepared by Sean P. Corcoran, assistant professor of educational economics at New York University, and Jennifer L. Jennings, who will join the NYU faculty in the fall. They found significant methodological limitations that call into question the sweeping findings announced by Mathematica. They also found the Mathematica data themselves often suggest the opposite finding: that the alternatively certified teachers who were studied generally produced poorer outcomes than do the traditionally certified teachers in the sample.

The full review is available at http://www.greatlakescenter.org as part of the Think Twice think tank review project funded by the Great Lakes Center for Education Research and Practice.

Corcoran and Jennings point out that policymakers and other readers of the Mathematica report tend to look for simple answers to simple questions such as “Is alternative teacher certification a bad thing or a good thing, and if alternative certification does no harm, is traditional teacher certification even necessary?”

“Notwithstanding suggestions to the contrary in the report’s press release or executive summary, this report is unable to provide a satisfactory or general answer to either of those questions,” they conclude.

Specifically, the review found the Mathematica study:

• Did not fully report and acknowledge in its conclusions the many analyses from the study finding that traditionally trained teachers outperformed alternative route teachers in both math and reading. 
• Has a research design that favors finding few significant differences between groups, most notably its small sample size, sampling methods, and failure to distinguish the “treatments” that alternative certification and traditional certification teachers provided (meaning that members of the two compared groups had substantially overlapping preparation experiences).
• Is relevant only to a very limited population of teachers in schools that hire many alternatively certified teachers, and is not generalizable to most states, districts, and schools that do not allow such programs and are more selective in their hiring.

The review found that the study sample includes teachers with an average of three years experience and is heavily weighted towards grades K-2, which comprise 71 percent of the study’s teachers. The average school in the study is a high-poverty, high-minority urban school, and the traditionally certified teachers who are the comparison group in the study are only those employed by these disadvantaged schools. As a result, alternatively certified teachers are compared only with teachers that prior research has demonstrated are, on average, less qualified than other traditionally certified teachers.

“Unfortunately, none of the results found in this study can be used to meaningfully inform the policy debate over alternative certification because they cannot be generalized to the larger population of schools and teachers,” Corcoran said.

The review also took issue with Mathematica’s selected emphasis of results, highlighting those results suggesting no difference between alternatively and conventionally certified teachers while minimizing results showing alternatively certified teachers doing significantly less well than other teachers in many categories. For example, alternatively certified teachers still taking coursework were significantly less effective in teaching reading and mathematics than their traditionally certified counterparts; and alternatively certified teachers from California were significantly less effective in teaching mathematics than traditionally prepared teachers.

“We want to be careful here not to overstate the findings—because in reality the findings are minimal,” wrote Corcoran and Jennings. “This is for good reason: if one set out to design a study that would find no statistically significant differences between the achievement of students taught by traditionally and alternatively certified teachers, this is precisely the study one would have designed.”

Aaron Pallas, professor of education and sociology at Teachers College, Columbia University, concurred with the findings, noting that “The design of this study successfully precludes it from addressing the most salient policy questions about alternative teacher certification – but we get a pretty clean estimate of the relative effectiveness of pairs of traditional-route and alternate-route teachers that are not representative of any population of teacher education programs, teachers, or schools.”

These concerns are also reinforced by a second review of the Mathematica research, conducted by Linda Darling-Hammond, the Charles Ducommun Professor of Education at Stanford University, as part of a policy brief on teacher preparation programs generally. In her analysis, Darling-Hammond pointed to a number of similar methodological flaws in the Mathematica study.  She also noted that, when students’ fall-to-spring scores are examined, Mathematica’s data show that alternatively certified teachers from “low-coursework” programs – most of them from Texas schools – actually lowered student achievement noticeably – hardly a standard to which other schools should aspire. Students of “high-coursework” alternatively certified teachers did only slightly better. In all cases, the students of traditionally certified teachers gained more or declined less than the alternatively certified teachers, but none of these already low-achieving students did well enough to begin to close the achievement gap.

Characterizing the outcomes of these programs as a “race to the bottom,” Darling-Hammond also reviewed the results of studies showing substantial gains for students of teachers with stronger qualifications and much higher-quality training.

“If we truly want our children to engage in a race to the top, we must create substantially higher levels of teacher effectiveness, particularly in those classrooms that have been left the furthest behind,” Darling-Hammond said. “The poor overall outcomes reported by Mathematica for these overwhelmingly disadvantaged schools show that these children are being abandoned by a system that has not invested in the quality of teachers needed to substantially improve their learning.  This race to the bottom cannot be halted through alternative certification programs that simply seek to ‘do no more harm’ – or at least not much more harm – than other relatively poor training. Our students deserve far better than that. We need to invest in replicating the high-quality programs that achieve better results.”

Darling-Hammond’s review, “Educational Opportunity and Alternative Certification: New Evidence and New Questions,” can be found on the Stanford Center for Opportunity Policy in Education (SCOPE) website, http://edpolicy.stanford.edu/.

About The Think Twice Project
The Think Twice project provides the public, policy makers and the press with timely, academically sound reviews of selected think tank publications. It is a collaboration of the Education Policy Studies Laboratory at Arizona State University and the Education and the Public Interest Center at the University of Colorado at Boulder and is funded by the Great Lakes Center for Education Research and Practice.

###