2013 Think Twice Reviews
Think Twice is one of the nation's first efforts to serve as a watchdog to review think tank research on public education issues and policies, ensuring that published work meets the quality and standards of university scholarship. As think tank research becomes increasingly important reference sources in public policy debates, media and other critics have called for increased scrutiny to ensure validity and objectivity (click here to see related stories).
The goal of the Think Twice project is to provide the public, policy makers and the press with timely academically sound reviews of selected think tank publications.
Reports & Reviews for 2013
|Report Reviewed:||An Opportunity Culture for All : Making teaching a highly paid, high-impact profession|
|Publisher/Think Tank:||Public Impact|
|Authors:||Emily Ayscue Hassel and Bryan C. Hassel|
|A report from Public Impact calls for a sweeping restructuring of the teaching profession. Public Impact in its consulting work promotes school turnarounds, charter schools, and market-based reforms.|
|Think Twice Review Date:||December 12, 2013|
|Reviewer:||Patricia Hinchey, Penn State University|
|Patricia Hinchey finds that the report lacks any empirical grounding. This report is based on nothing more than speculation and has nothing to recommend it as a guide to sound policy.
|Report Reviewed:||Understanding Illinois’ broken education funding system: A primer on General State Aid|
|Publisher/Think Tank:||Illinois Policy Institute|
|Authors:||Ted Dabrowski, Josh Dwyer, and John Klingner|
|This report claims that Illinois’ school funding system is “broken” and should be replaced with a system where “parents control the flow and distribution of money.”|
|Think Twice Review Date:||November 21, 2013|
|Reviewer:||Reviewer: Amy Ellen Schwartz, New York University
Sarah Cordes, New York University
|An academic review by Amy Ellen Schwartz and Sarah Cordes finds little evidence to support its claims or preferred solution. The review finds that there is little in the way of evidence to point to any real “flaws” in Illinois’ current funding system. The reviewers find that the report does provide a number of government generated statistics to support its claims, but the report is thin on other relevant literature. Schwartz and Cordes say, “Thus, the report’s usefulness for evaluating the current education finance system in Illinois is limited.” More analysis is needed to fully evaluate Illinois’ school funding system.
|Report Reviewed:||School Choice and School Performance in the New York City Public Schools: Will the Past be Prologue?|
|Publisher/Think Tank:||Brown Center on Education Policy at Brookings|
|Authors:||Grover J. “Russ” Whitehurst and Sarah Whitfield|
|Whitehurst and Whitfield attempt to use aggregate findings to make the case that expansion of school choice and school competition in NYC schools is a major factor in increased graduation rates and enhanced performance on state tests of academic achievement. The authors’ recommendations also include calls for more general improvements in instruction, teacher quality, and accountability systems.|
|Think Twice Review Date:||November 19, 2013|
|Reviewer:||Patricia Burch, University of Southern California|
|A review finds that the report is lacking in many ways, including a very slender review of relevant research in the area of school choice and competition. It was also found that the report selectively presents the data, which has the effect of creating an overly positive picture of the reports referenced. The majority of the report’s recommendations are not linked to the analyses presented in the paper. The usefulness of the report is limited by the lack of support provided for the recommendations. In general, the report offers little in the way of useful guidance for policymakers.
|Report Reviewed:||“Why the Gap? Special Education and New York City Charter Schools"|
|Publisher/Think Tank:||Center for Reinventing Public Education (CRPE) Manhattan Institute for Policy Research (MIPR)|
|Authors:||Marcus A. Winters|
|This study attempts to ascertain why a disparity exists in special education enrollment rates in New York City charter schools. The study cautions against recent attempts to address the special education gap through enrollment targets.|
|Think Twice Review Date:||November 7, 2013|
|Reviewer:||Julie Mead, University of Wisconsin-Madison|
|A review finds that the report raises issues about application and transfer patterns in charter schools, yet it fails to provide useful results to adequately inform policymakers. The report neglects any review of related literature and ignores alternate explanations for the statistical patterns found. The use of a restricted, non-representative data set further limits the usefulness of the findings and conclusions that are drawn. The report does confirm the existence of enrollment disparities between charter and traditional public schools in New York City and the growth in these disparities over time. While the report sheds light on a complex issue, the evidence presented simply does not justify the elimination of accountability efforts to reduce enrollment disparities.
|Report Reviewed:||“Building the Possible: The Achievement School District’s Presentation in Milwaukee” & “The Recovery School District’s Presentation in Milwaukee”|
|Publisher/Think Tank:||The Achievement School District (ASD) & the Recovery School District (RSD)|
|Authors:||Elliot Smalley, Chief of Staff for Achievement School District
Patrick Dobard, Superintendent of the Recovery School District
|Portfolio districts, such as the Recovery School District (RSD) and the Achievement School District (ASD), remove schools from the governance of locally elected school boards and superintendents and place them into districts that answer to a state authority. The presenters aimed to encourage other districts to adopt such governance structures although no rigorous research has yet examined their effectiveness.|
|Think Twice Review Date:||October 8, 2013|
|Reviewer:||Elizabeth DeBray, University of Georgia Huriya Jabbar, University of California Berkeley|
|A review of the presentations found that the presentations accurately depict the strategies employed by these two ‘districts,’ but do not address whether such strategies are effective or efficient. Neither presentation offers a sophisticated discussion of the data or the claims made. The presentations fail to provide the research based needed for policymakers.
|Report Reviewed:||Were all those standardized tests for nothing? The lessons of No Child Left Behind|
|Publisher/Think Tank:||American Enterprise Institute|
|The report, from a noted conservative think tank, attempts to isolate the effects of NCLB’s threat of sanctions on underperforming schools. The authors infer that federal pressure and punishment are promising policy avenues.|
|Think Twice Review Date:||August 27, 2013|
|Reviewer:||Bruce Fuller, University of California Berkeley|
|A review found that the report is a worthwhile study of the effectiveness of punitive sanctions on student achievement in one state (North Carolina). The report offers recommendations for improving NCLB, but those recommendations are not tied to the report’s empirical analysis. Readers in search of an effective federal role will remain uninformed.
|Report Reviewed:||National Charter School Study 2013|
|Publisher/Think Tank:||Center for Research on Education Outcomes (CREDO) at Stanford University|
|The Center for Research on Education Outcomes (CREDO) at Stanford University examined charter schools in 26 states and New York City. Its primary findings were: (a) a small positive effect of being in a charter school on reading scores and no impact on math scores; and (b) a relative improvement in average charter school quality since CREDO’s 2009 study.|
|Think Twice Review Date:||July 16, 2013|
|Reviewer:||Andrew Maul, University of Colorado Boulder
Abby McClelland, University of Colorado Boulder
|Reviewers find significant reasons for caution in interpreting the study’s results. The review points out that the study itself shows only a tiny real impact on the part of charter schools: “less than one hundredth of one percent of the variation in test performance is explainable by charter school enrollment.” Even setting aside its analytical flaws, the study merely confirms that charter schools perform on par with traditional public schools.
|Report Reviewed:||Weighted Student funding for California|
|Publisher/Think Tank:||Reason Foundation, Lisa Snell|
|The Reason Foundation, a Libertarian think tank, recently issued a brief on California Gov. Jerry Brown’s school finance reform plan, which endorses Brown’s proposal and then uses it to advance its own ideas for reducing the role of school districts and for giving school principals greater autonomy over spending.|
|Think Twice Review Date:||June 18, 2013|
|Reviewer:||Dr. Bruce D. Baker, Rutgers University|
|According to the review, the Reason report advocates modifying Gov. Brown’s proposal, to link the state funds to individual children, so that “money follows the child” regardless of which school he or she is enrolled in. Baker finds that revising Brown’s proposal with the Reason Foundation recommendations has the potential to more equitably distribute funding across local public school districts. "Yet no data are presented or evaluated to support these claims," according to Baker. The report offers little guidance for policymaking.
|Report Reviewed:||Middle Class or Middle of the Pack: What can we learn when benchmarking U.S. schools against the world’s best|
|Publisher/Think Tank:||Publisher/Think Tank: America Achieves|
|America Achieves report calls for additional international testing. The authors of the report consider recent PISA results a "wake up call to America’s middle class" and contend in response that U.S. high schools should start taking part in a new international test that is to compare the academic knowledge and skills of 15-year-olds- around the world.|
|Think Twice Review Date:||June 4, 2013|
|Reviewer:||Martin Carnoy, Stanford University|
|This review finds no evidence to support the report’s arguments. Martin Carnoy, Stanford University, finds that the America Achieves report “is not grounded in research but rather is an assertion that measurement, by itself, is an effective reform tool.” And it makes that assertion without ever explaining how the test in question would be linked to curricula or strategies tailored to teaching mathematics and science, or to specific teacher professional development strategies. “Thus the report is of no utility to policymakers,” Carnoy concludes.
|Report Reviewed:||Report Card on American Education: Ranking State K-12 Performance, Progress, and Reform (18th edition)|
|Publisher/Think Tank:||American Legislative Exchange Council (ALEC)|
|The 18th edition of the American Legislative Exchange Council’s (ALEC) Report Card draws on ratings from market-orientated advocacy groups to grade states in areas such as allowances for charter schools, availability of vouchers, and permissiveness for homeschoolers.|
|Think Twice Review Date:||May 9, 2013|
|Reviewer:||Christopher Lubienski, University of Illinois
T. Jameson Brewer, University of Illinois
|This review finds that ALEC’s grades reflect an explicit ideological agenda absent any evidence of the effectiveness of promoted policies. This report provides little or no usefulness to policymakers.
|Report Reviewed:||KIPP Middle Schools: Impacts on Achievement and Other Outcomes|
|Publisher/Think Tank:||Mathematica Policy Research (MPR)|
|A recent study from Mathematica Policy Research (MPR) suggests that KIPP (Knowledge Is Power Program) charter middle schools may actually boost test-score growth by as much as eight months to eleven months over a three year period.|
|Think Twice Review Date:||April 30, 2013|
|Reviewer:||Gregory Camilli, University of Colorado Boulder|
|Professor Camilli notes that the study was carefully planned and executed. Despite the careful execution, Camilli's review cautions that the results appear to be overstated for two reasons. (1) Translating outcomes into "months" of additional learning is an inexact science and can lead to absurd results if taken literally. (2) Reported measures of effectiveness that take attrition into account are smaller than the estimates used to draw conclusions about the effectiveness of KIPP.
|Report Reviewed:||Mayoral Governance and Student Achievement: How Mayor-Led Districts are Improving School and Student Performance|
|Publisher/Think Tank:||Center for American Progress (CAP)|
|A recent report from the Center for American Progress claims that mayor-led districts may use resources more strategically and that mayor-controlled districts have seen increases in student achievement. The report also states that moving to a mayoral-controlled district can also help spur innovation and advancement.|
|Think Twice Review Date:||April 23, 2013|
|Reviewer:||Katrina E. Bulkley, Montclair State University|
|Reviewer Katrina E. Bulkley finds that the report offers useful information about the shift to mayoral-led districts and the challenges that may arise. However, Bulkley found several limitations during her review, which make the report useless for serious policy discussions.
|Report Reviewed:||Evaluation of Teach for America in Texas Schools|
|Publisher/Think Tank:||Edvance Research (Funded by Teach For America)|
|This study by Edvance Research, with funding from Teach For America, claimed that TFA corps members and alumni had outsized impact on middle schools in several Texas districts.|
|Think Twice Review Date:||April 9, 2013|
|Reviewer:||Ed Fuller, Penn State University
Nathan Dadey, University of Colorado Boulder
|Fuller and Dadey find that while the findings appear large enough to be relevant to public policy, several issues related to the report’s statistical model make it likely that the actual size of the TFA teacher effects differ than what is found in the report. The expert reviewers also have found cause for concern in sample construction, matching procedures and interpretations. Because of those limitations, Fuller and Dadey caution that the report does not provide solid evidence that either TFA teachers or TFA alumni have a measured effect on student test scores.
|Report Reviewed:||Does Sorting Students Improve Scores? An Analysis of Class Composition|
|Publisher/Think Tank:||National Bureau of Economic Research,
Courtney A. Collins and Li Gan
|Despite the vast majority of research into tracking or ability grouping of students, a recently released report from the National Bureau of Economic Research (NBER) purports that sorting students by ability improves outcomes for low-achieving and high-achieving students alike.|
|Think Twice Review Date:||April 2, 2013|
|Reviewer:||Dr. Carol Corbett Burris, South Side High School
Katherine Allison, University of Colorado Boulder
|The report suffers from several research defects and reviewers find that this working paper should not be used to inform policy regarding tracking or grouping practices. The reviewers further find “The authors go beyond their data and analyses when they conclude that schools should sort students as a cost-free method to improve student achievement.”
|Report Reviewed:||The School Staffing Surge: Decades of Employment Growth in America’s Public Schools, Part II|
|Publisher/Think Tank:||The Friedman Foundation for Educational Choice|
|The report disaggregates trends in K-12 hiring for individual states, presenting ratios comparing the number of administrators and other non-teaching staff to the number of teachers or students.|
|Think Twice Review Date:||March 26, 2013|
|Reviewer:||Joydeep Roy, Visiting Professor Teachers College – Columbia University & Senior Economist for the New York City Independent Budget Office|
|A new review finds that the report fails to attempt to benchmark hiring against each state’s needs and circumstances. Neither report explores the causes and consequences of faster employment growth. Part II is devoid of any serious policy implications.
|Report Reviewed:||Estimating the effect of leaders on public sector productivity: The case for school principals AND “School Leaders Matter”|
|Publisher/Think Tank:||National Bureau of Economic Research
|This report attempted to estimate how much “high” and “low” effective principals affect student achievement. The study also explored patterns of change in the composition of schools’ teaching staffs, as well as the movement of principal talent across schools.|
|Think Twice Review Date:||March 5, 2013|
|Reviewer:||Margaret Terry Orr, Bank Street College of Education|
|In the review by Terry Orr, Orr confirms “that principals have a positive, independent influence on achievement and that the size of this influence varies by school poverty rates.” On the other hand, Orr also found methodological flaws which raise questions about sample sizes and the validity of such analysis. Orr cautions that the report has limited utility to guide policy and practice. This is concerning, because multiple states have adopted policies that include similar estimates of principal effectiveness relative to changes in student test scores.
|State Policy Report Card|
|This report assigned letter grades to states based on its preferred policies regarding school choice, defined contribution pension programs, private vouchers, test-based accountability and centralized control of public schools.|
|Think Twice Review Date:||February 20, 2013|
|Reviewer:||Sherman Dorn, University of South Florida
Ken Libby, University of Colorado Boulder
|“Report cards” that grade states on their education policies assign rankings that vary tremendously, depending on the political ideology of the grader, according to a new review released today by the Think Twice think tank review project. Sherman Dorn of the University of South Florida and doctoral student Ken Libby of the University of Colorado Boulder discovered great variation in a recent StudentsFirst state policy report card. The review was produced by the National Education Policy Center with funding from the Great Lakes Center for Education Research and Practice.|
|Report Reviewed:||Charter School Performance in Michigan|
|Publisher/Think Tank:||Center for Research on Education Outcomes (CREDO) at Stanford University|
|The CREDO researchers analyzed differences in student performance at charter schools and traditional public schools in Michigan. Charter advocates trumpeted the new CREDO Michigan study as a “smashing success” for charter schools.|
|Think Twice Review Date:||February 12, 2013|
|Reviewer:||Andrew Maul, University of Colorado Boulder|
|Andrew Maul of the University of Colorado Boulder reviewed the report and found the study itself has both strong and weak elements. The new study estimates that students in charter schools in Michigan experience 0.06 standard deviations more academic growth than comparison students in traditional public schools. As Maul points out, “This is equivalent to saying that about a tenth of one percent of the variation in academic growth is associated with school type.” Such a finding of almost no difference between charters and non-charters is very much in line with the overall body of past research. Some studies suggest slight benefits, some suggest slight harm, and many show no difference. Maul goes on to point out “significant reasons for caution in interpreting the study’s results.”
|Two Culminating Reports From The MET Project|
|Publisher/Think Tank:||The Bill and Melinda Gates Foundation|
|The Measures of Effective Teaching Project (MET) is an ambitious, multi-year study of thousands of teachers in six school districts. This review is of two of the final research papers dealing with the impact of student assignment on teacher evaluations and how teacher evaluation measures are best combined.|
|Think Twice Review Date:||January 31, 2013|
|Reviewer:||Jesse Rothstein, University of California Berkeley
William Mathis, University of Colorado Boulder
|The MET study does little to settle longstanding debates over how to best evaluate teachers comprehensively. The study’s results were inconclusive and provide little usable guidance for real-world decisions.|
|Report Reviewed:||The Education Choice and Competition Index: Background and Results 2012|
|Publisher/Think Tank:||Brown Center on Education Policy at the Brookings Institution
Grover J. "Russ" Whitehurst and Sarah Whitfield
|This report highlights the findings of a self-developed choice index and calls for expansion of market-based school reforms.|
|Think Twice Review Date:||January 29, 2013|
|Reviewer:||David Garcia, Mary Lou Fulton Teachers College, Arizona State University|
|The report and its choice index fail to advance education policy decisions and only rehash old arguments for unregulated school choice. David Garcia, who reviewed the report, found it makes no substantiated connection between satisfaction and the purported benefits arising from market-based education policies. The review also found weaknesses in the index’s subjective scoring system, as well as a lack of research behind the report’s recommendations, making the Brookings’ product fall short of being an effective policy tool.
|Failure is Not an Option: How Principals, Teachers, Students, and Parents from Ohio’s High-Achieving, High Poverty Schools Explain Their Success|
|Publisher/Think Tank:||Public Agenda|
|The report attempts to explain how principals, teachers, students, and parents sustain effective practices and what helps them to weather tough times. The report was compiled through interviews with nine high-achieving, high-poverty schools in Ohio and offers a description of key attributes and possible recommendations.|
|Think Twice Review Date:||January 15, 2013|
|Reviewer:||Mark Paige, UMass Dartmouth|
|Paige’s review finds that the report fails to make clear connections between the recommendations and key attributes of these high-achieving, high-poverty schools. The report fails to specify how these attributes were derived from the interviews. According to Paige, the report’s biggest deficiency is that the recommendations fail to propose remedies or explicitly address poverty and equity needs of schools. The recommendations are common sense, but the proposals are not sufficiently grounded in either the study’s own data or in the larger body of research.|
These articles and/or reports are copyrighted material, the use of which has not been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of educational issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the U.S. Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information go to: http://www.law.cornell.edu/uscode/17/107.shtml. If you wish to use copyrighted material from this site for purposes of your own that go beyond fair use, you must obtain permission from the copyright owner.