2013 Think Twice Reviews
Think Twice is one of the nation's first efforts to serve as a watchdog to review think tank research on public education issues and policies, ensuring that published work meets the quality and standards of university scholarship. As think tank research becomes increasingly important reference sources in public policy debates, media and other critics have called for increased scrutiny to ensure validity and objectivity (click here to see related stories).
The goal of the Think Twice project is to provide the public, policy makers and the press with timely academically sound reviews of selected think tank publications.
Click Here To Read Our Legal Statement
 |
Think Tank Research Quality: Think Tank Research Quality, edited by Kevin Welner of the University of Colorado at Boulder, Patricia H. Hinchey of Pennsylvania State University, Alex Molnar of Arizona State University, and independent researcher Don Weitzman, offers clearly written, jargon-free expert reviews of studies on topics such as vouchers, charter schools, and alternative teacher certification. Friends of The Great Lakes Center receive 20% off phone, fax or mail-in orders...not valid online. |
Recently Released
May 9, 2013
ALEC's Report Card Receives Failing Marks
Policymakers should insist on referencing evidence-based research
Reports & Reviews for 2013
| Report Reviewed: | Report Card on American Education: Ranking State K-12 Performance, Progress, and Reform (18th edition) |
| Publisher/Think Tank: | American Legislative Exchange Council (ALEC) |
| The 18th edition of the American Legislative Exchange Council’s (ALEC) Report Card draws on ratings from market-orientated advocacy groups to grade states in areas such as allowances for charter schools, availability of vouchers, and permissiveness for homeschoolers. | |
| Think Twice Review Date: | May 9, 2013 |
| Reviewer: | Christopher Lubienski, University of Illinois T. Jameson Brewer, University of Illinois |
| This review finds that ALEC’s grades reflect an explicit ideological agenda absent any evidence of the effectiveness of promoted policies. This report provides little or no usefulness to policymakers. |
|
  |
|
| Report Reviewed: | KIPP Middle Schools: Impacts on Achievement and Other Outcomes |
| Publisher/Think Tank: | Mathematica Policy Research (MPR) |
| A recent study from Mathematica Policy Research (MPR) suggests that KIPP (Knowledge Is Power Program) charter middle schools may actually boost test-score growth by as much as eight months to eleven months over a three year period. | |
| Think Twice Review Date: | April 30, 2013 |
| Reviewer: | Gregory Camilli, University of Colorado Boulder |
| Professor Camilli notes that the study was carefully planned and executed. Despite the careful execution, Camilli's review cautions that the results appear to be overstated for two reasons. (1) Translating outcomes into "months" of additional learning is an inexact science and can lead to absurd results if taken literally. (2) Reported measures of effectiveness that take attrition into account are smaller than the estimates used to draw conclusions about the effectiveness of KIPP. |
|
| |
|
| Report Reviewed: | Mayoral Governance and Student Achievement: How Mayor-Led Districts are Improving School and Student Performance |
| Publisher/Think Tank: | Center for American Progress (CAP) |
| A recent report from the Center for American Progress claims that mayor-led districts may use resources more strategically and that mayor-controlled districts have seen increases in student achievement. The report also states that moving to a mayoral-controlled district can also help spur innovation and advancement. | |
| Think Twice Review Date: | April 23, 2013 |
| Reviewer: | Katrina E. Bulkley, Montclair State University |
| Reviewer Katrina E. Bulkley finds that the report offers useful information about the shift to mayoral-led districts and the challenges that may arise. However, Bulkley found several limitations during her review, which make the report useless for serious policy discussions. |
|
| |
|
| Report Reviewed: | Evaluation of Teach for America in Texas Schools |
| Publisher/Think Tank: | Edvance Research (Funded by Teach For America) |
| This study by Edvance Research, with funding from Teach For America, claimed that TFA corps members and alumni had outsized impact on middle schools in several Texas districts. | |
| Think Twice Review Date: | April 9, 2013 |
| Reviewer: | Ed Fuller, Penn State University Nathan Dadey, University of Colorado Boulder |
| Fuller and Dadey find that while the findings appear large enough to be relevant to public policy, several issues related to the report’s statistical model make it likely that the actual size of the TFA teacher effects differ than what is found in the report. The expert reviewers also have found cause for concern in sample construction, matching procedures and interpretations. Because of those limitations, Fuller and Dadey caution that the report does not provide solid evidence that either TFA teachers or TFA alumni have a measured effect on student test scores. |
|
| |
|
| Report Reviewed: | Does Sorting Students Improve Scores? An Analysis of Class Composition |
| Publisher/Think Tank: | National Bureau of Economic Research, Courtney A. Collins and Li Gan |
| Despite the vast majority of research into tracking or ability grouping of students, a recently released report from the National Bureau of Economic Research (NBER) purports that sorting students by ability improves outcomes for low-achieving and high-achieving students alike. | |
| Think Twice Review Date: | April 2, 2013 |
| Reviewer: | Dr. Carol Corbett Burris, South Side High School Katherine Allison, University of Colorado Boulder |
| The report suffers from several research defects and reviewers find that this working paper should not be used to inform policy regarding tracking or grouping practices. The reviewers further find “The authors go beyond their data and analyses when they conclude that schools should sort students as a cost-free method to improve student achievement.” |
|
| |
|
| Report Reviewed: | The School Staffing Surge: Decades of Employment Growth in America’s Public Schools, Part II |
| Publisher/Think Tank: | The Friedman Foundation for Educational Choice |
| The report disaggregates trends in K-12 hiring for individual states, presenting ratios comparing the number of administrators and other non-teaching staff to the number of teachers or students. | |
| Think Twice Review Date: | March 26, 2013 |
| Reviewer: | Joydeep Roy, Visiting Professor Teachers College – Columbia University & Senior Economist for the New York City Independent Budget Office |
| A new review finds that the report fails to attempt to benchmark hiring against each state’s needs and circumstances. Neither report explores the causes and consequences of faster employment growth. Part II is devoid of any serious policy implications. |
|
| |
|
| Report Reviewed: | Estimating the effect of leaders on public sector productivity: The case for school principals AND “School Leaders Matter” |
| Publisher/Think Tank: | National Bureau of Economic Research Education Next |
| This report attempted to estimate how much “high” and “low” effective principals affect student achievement. The study also explored patterns of change in the composition of schools’ teaching staffs, as well as the movement of principal talent across schools. | |
| Think Twice Review Date: | March 5, 2013 |
| Reviewer: | Margaret Terry Orr, Bank Street College of Education |
| In the review by Terry Orr, Orr confirms “that principals have a positive, independent influence on achievement and that the size of this influence varies by school poverty rates.” On the other hand, Orr also found methodological flaws which raise questions about sample sizes and the validity of such analysis. Orr cautions that the report has limited utility to guide policy and practice. This is concerning, because multiple states have adopted policies that include similar estimates of principal effectiveness relative to changes in student test scores. |
|
| |
|
Report Reviewed: |
State Policy Report Card |
| Publisher/Think Tank: | StudentsFirst |
| This report assigned letter grades to states based on its preferred policies regarding school choice, defined contribution pension programs, private vouchers, test-based accountability and centralized control of public schools. | |
| Think Twice Review Date: | February 20, 2013 |
| Reviewer: | Sherman Dorn, University of South Florida Ken Libby, University of Colorado Boulder |
| “Report cards” that grade states on their education policies assign rankings that vary tremendously, depending on the political ideology of the grader, according to a new review released today by the Think Twice think tank review project. Sherman Dorn of the University of South Florida and doctoral student Ken Libby of the University of Colorado Boulder discovered great variation in a recent StudentsFirst state policy report card. The review was produced by the National Education Policy Center with funding from the Great Lakes Center for Education Research and Practice. | |
| |
|
| Report Reviewed: | Charter School Performance in Michigan |
| Publisher/Think Tank: | Center for Research on Education Outcomes (CREDO) at Stanford University |
| The CREDO researchers analyzed differences in student performance at charter schools and traditional public schools in Michigan. Charter advocates trumpeted the new CREDO Michigan study as a “smashing success” for charter schools. | |
| Think Twice Review Date: | February 12, 2013 |
| Reviewer: | Andrew Maul, University of Colorado Boulder |
| Andrew Maul of the University of Colorado Boulder reviewed the report and found the study itself has both strong and weak elements. The new study estimates that students in charter schools in Michigan experience 0.06 standard deviations more academic growth than comparison students in traditional public schools. As Maul points out, “This is equivalent to saying that about a tenth of one percent of the variation in academic growth is associated with school type.” Such a finding of almost no difference between charters and non-charters is very much in line with the overall body of past research. Some studies suggest slight benefits, some suggest slight harm, and many show no difference. Maul goes on to point out “significant reasons for caution in interpreting the study’s results.” |
|
| |
|
Report Reviewed: |
Two Culminating Reports From The MET Project |
| Publisher/Think Tank: | The Bill and Melinda Gates Foundation |
| The Measures of Effective Teaching Project (MET) is an ambitious, multi-year study of thousands of teachers in six school districts. This review is of two of the final research papers dealing with the impact of student assignment on teacher evaluations and how teacher evaluation measures are best combined. | |
| Think Twice Review Date: | January 31, 2013 |
| Reviewer: | Jesse Rothstein, University of California Berkeley William Mathis, University of Colorado Boulder |
| The MET study does little to settle longstanding debates over how to best evaluate teachers comprehensively. The study’s results were inconclusive and provide little usable guidance for real-world decisions. | |
| |
|
| Report Reviewed: | The Education Choice and Competition Index: Background and Results 2012 |
| Publisher/Think Tank: | Brown Center on Education Policy at the Brookings Institution Grover J. "Russ" Whitehurst and Sarah Whitfield |
| This report highlights the findings of a self-developed choice index and calls for expansion of market-based school reforms. | |
| Think Twice Review Date: | January 29, 2013 |
| Reviewer: | David Garcia, Mary Lou Fulton Teachers College, Arizona State University |
| The report and its choice index fail to advance education policy decisions and only rehash old arguments for unregulated school choice. David Garcia, who reviewed the report, found it makes no substantiated connection between satisfaction and the purported benefits arising from market-based education policies. The review also found weaknesses in the index’s subjective scoring system, as well as a lack of research behind the report’s recommendations, making the Brookings’ product fall short of being an effective policy tool. |
|
| |
|
Report Reviewed: |
Failure is Not an Option: How Principals, Teachers, Students, and Parents from Ohio’s High-Achieving, High Poverty Schools Explain Their Success |
| Publisher/Think Tank: | Public Agenda |
| The report attempts to explain how principals, teachers, students, and parents sustain effective practices and what helps them to weather tough times. The report was compiled through interviews with nine high-achieving, high-poverty schools in Ohio and offers a description of key attributes and possible recommendations. | |
| Think Twice Review Date: | January 15, 2013 |
| Reviewer: | Mark Paige, UMass Dartmouth |
| Paige’s review finds that the report fails to make clear connections between the recommendations and key attributes of these high-achieving, high-poverty schools. The report fails to specify how these attributes were derived from the interviews. According to Paige, the report’s biggest deficiency is that the recommendations fail to propose remedies or explicitly address poverty and equity needs of schools. The recommendations are common sense, but the proposals are not sufficiently grounded in either the study’s own data or in the larger body of research. | |
| |
|
Previous Reports & Reviews:
Think Twice Reviews Released In 2012
Think Twice Reviews Released In 2011
Think Twice Reviews Released In 2010
Think Twice Reviews Released In 2009
Think Twice Reviews Released In 2008
Think Twice Reviews Released In 2007
Think Twice Reviews Released In 2006
Previous Weekly Reports:
Current Weekly Report
2012 Think Twice Weekly Reports
2011 Think Twice Weekly Reports
2010 Think Twice Weekly Reports
2009 Think Twice Weekly Reports
Five "Honorees" of Bunkum Awards Announced for their Contributions to Sub-Par Education Research
Two New Think Tank Reviews Published … Elsewhere
Examining the Funding and Activities of Free Market Education Think Tanks
US Think-Tanks: Casualties In The War of Ideas
Far-Right "Think Tanks" (Propaganda Mills) – Who Are Those Guys?
Let The Buyer Beware [ Executive Summary | Full Report ]
