‘Widget Effect’ Report Draws Praise for Analysis of Teacher Evaluation Systems
August 5, 2009

Critique of evaluation systems is strong, but omissions in methodology and failure to report on promising strategies are weaknesses, according to reviewers

Contact: Teri Battaglieri, (517) 203-2940; greatlakescenter@greatlakescenter.org
Raymond L. Pecheone (650) 723-4106; pecheone@stanford.edu

EAST LANSING, Mi., Aug. 5, 2009 – A recent report from the New Teacher Project describes serious problems with teacher evaluation systems. The report, based on survey results from 12 school districts in four states, urges changes to establish more effective systems that will be able to discern the highest (and lowest) quality teachers. A new review of that report finds it well grounded and praises the report’s policy recommendations as sound and consistent with prior research. The review does, however, note that the report lacks important details about its own methods and fails to consider or to draw upon any promising teacher evaluation strategies in current use.

The report, The Widget Effect: Our National Failure to Acknowledge and Act on Teacher Differences, was reviewed for the Think Twice think tank review project by Raymond Pecheone, Co-Executive Director of the Stanford University School Redesign Network and Director of the Performance Assessment for California Teachers, and Ruth Chung Wei, Director of Assessment Research and Development at the Stanford University School Redesign Network.

Pecheone and Wei credit The Widget Effect with presenting an extensive analysis of teacher evaluation systems in 12 districts across four states. The report is based on surveys of current and former teachers as well as administrators; on an examination of teacher evaluation records in the districts surveyed; and on interviews with school district leaders, school board officials, labor relations and human resources experts, and union leaders. Its extensive reporting of the data collected enables the authors “to build a compelling case for the inadequacy of the current teacher evaluation processes, uses, and policies in the four states,” Pecheone and Wei write.

The report falls short, however, in its omission of key details that might help readers to understand the sampling choices. The reviewers write: “it is unclear … how and why particular districts were selected, and whether they represent the range of teacher evaluation practices being implemented in school districts and states across the United States.” Omissions in the report’s description of its methodology (e.g., sampling strategy and survey response rates) and its sample lead to questions about the generalizability of the findings.

Pecheone and Wei also point out that the report does not consider important prior research in two areas: (1) why many evaluation systems don’t work well (for example, because they rely on administrators with limited time and pedagogical expertise, tools and processes that have a limited relationship with effective teaching); and (2) how some innovative and effective evaluation systems do work.

The latter omission is particularly important, because considering these evaluation systems and research about those systems would have allowed the authors to draw upon current successes in designing their recommendations.

Transforming the system rather than tinkering around the edges, Pecheone and Wei conclude, will require broader thinking and a commitment to provide much greater investments and support for innovation to build, test, and audit evaluation systems that can stand up to public scrutiny and be practically feasible.

Find the review by Raymond Pecheone and Ruth Chung Wei and a link to The Widget Report at: http://www.greatlakescenter.org.

About The Think Twice Project
The Think Twice project provides the public, policy makers and the press with timely, academically sound reviews of selected think tank publications. It is a collaboration of the Education Policy Studies Laboratory at Arizona State University and the Education and the Public Interest Center at the University of Colorado at Boulder and is funded by the Great Lakes Center for Education Research and Practice.

###