2014 Think Twice Reviews
Think Twice is one of the nation's first efforts to serve as a watchdog to review think tank research on public education issues and policies, ensuring that published work meets the quality and standards of university scholarship. As think tank research becomes increasingly important reference sources in public policy debates, media and other critics have called for increased scrutiny to ensure validity and objectivity (click here to see related stories).
The goal of the Think Twice project is to provide the public, policy makers and the press with timely academically sound reviews of selected think tank publications.
september 16, 2014
Reports & Reviews for 2014
|Report Reviewed:||Seeds of Achievement: AppleTree’s Early Childhood D.C. Charter Schools|
|Publisher/Think Tank:||Pioneer Institute|
C. Stillings Candal
|This report, funded by the Pioneer Institute and written by Cara Stillings Candal, presents a case, based on anecdotes and basic data, that the AppleTree model, a well-regarded early childhood program in the Washington D.C. area, is “unusually effective” and should be replicated across the nation.|
|Think Twice Review Date:||September 16, 2014|
|Reviewer:||W. Steven Barnett and Cynthia E. Lamy, National Institute for Early Education Research|
|In the review, Barnett and Lamy found that the program described in this case closely resembles the research-based approaches found in other highly effective publicly funded preschool models, but there are limitations to making the kind of claims that the report does to support expansion of preschool charters.
|Report Reviewed:||The Productivity of Public Charter Schools|
|Publisher/Think Tank:||Department of Education Reform (DER) at the University of Arkansas|
|Author:||Meagan Batdorff, Larry Maloney, Jay F. May, Sheree T. Speakman, Patrick J. Wolf, Albert Cheng|
|The authors of the report claim that this is the first national study of productivity of charter schools relative to traditional public schools. The report utilizes finding from the National Assessment of Educational Progress (NAEP) and “revenues received” to support that charter schools spend less per pupil than traditional public schools and produce achievement as good or superior to that of traditional public schools.|
|Think Twice Review Date:||August 19, 2014|
|Reviewer:||Gene V. Glass, University of Colorado Boulder and Arizona State University|
|In his review, Gene V. Glass found the report inaccurately utilizes NAEP test results and that the assessment of expenditures is based on questionable data, leaving readers with little evidence on which to base any valid conclusions. The report and its conclusions rest on shaking ground and suffer from multiple sources of invalidity, rendering it useless for educators or policymakers.
|Report Reviewed:||Preparing Principals to Raise Student Achievement: Implementation and Effects of the New Leaders Program in Ten Districts|
|Publisher/Think Tank:||RAND Corporation|
Susan M. Gates, Laura S. Hamilton, Paco Martorell, Susan Burkhauser, Paul Heaton, Ashley Pierson, Matthew Baird, Mirka Vuollo, Jennifer J. Li, Diana Catherine Lavery, Melody Harvey, Kun Gu
|A recently released RAND report, an evaluation of the New Leaders principal preparation program, attempted to determine the program’s impact on student test scores. It concluded that New Leaders principals were slightly more effective overall than non-New Leaders principals. The report was based on a statistical analysis of student achievement data on standardized tests.|
|Think Twice Review Date:||June 26, 2014|
|Reviewer:||Edward J. Fuller, Penn State University|
|Ed Fuller (Penn State University) finds no known statistical approach that can identify the independent effect of principals. In addition, Fuller finds the overall conclusion to be problematic. Specifically, he cites problems with effect sizes, the use of multiple VAMs, peer effects, and an unbalanced presentation of positive findings among his cautions for the report’s usefulness. Regarding the results, Fuller finds inconsistencies across years, grade levels, and sites and that the positive results were very small. Encouragingly, Fuller finds the study provides a thoughtful approach to studying principal and principal preparation program (PPP) effectiveness. “Those seeking to evaluate principals and especially PPPs will find the study quite informative.”
|Report Reviewed:||Expanding the Education Universe: A Fifty-State Strategy for Course Choice|
|Publisher/Think Tank:||Thomas B. Fordham Institute|
|Based on the presumed success of school choice programs,a new Fordham Institute report called Expanding the Education Universe: A Fifty-State Strategy for Course Choice seeks to take choice a step further. Each student would design his or her own program of online and off-line courses, choosing their courses from a marketplace of curriculum developed by for-profit and not-for-profit vendors, as well as school districts or other public entities.|
|Think Twice Review Date:||June 19, 2014|
|Reviewer:||Patricia Burch, University of Southern California|
|In the review, the authors found that the report simply assumes that school choice is effective, and that it represents sound public and educational policy. Course choice is presented as a logical extension of that model. In reviewing the literature on the topic of course choice, the authors turned up a dearth of existing research to either support or refute the claims made in the report. The reviewers note that the report is more of "an advocacy piece" which seeks to advance "course choice." In conclusion, the authors find "The piece lacks the empirical evidence and sophisticated discussion necessary for a serious policy proposal."
|Report Reviewed:||The State Education Agency: At the Helm, Not the Oar|
|Publisher/Think Tank:||Bellwether Education Partners & The Thomas B. Fordham Institute|
Andrew Smarick & Juliet Squire
|A recent report from the Fordham Institute and Bellwether Education Partners argues that states should get out of the reform business directly and hand it over to an "ecosystem of nonprofit organizations." It proposes a governance overhaul to replace the role of State Education Agencies in school improvement. The report outlines the responsibilities of SEAs and suggests that they struggle to deliver. It cautions against strengthening SEAs to meet new accountability obligations.|
|Think Twice Review Date:||June 17, 2014|
|Reviewer:||Peter W. Cookson, Jr., Georgetown University|
|An academic review finds the report is well written, but reveals that the findings and recommendations are not based on original research. In his review, Peter W. Cookson, Jr. found the report fails to support its findings or recommendations with the in-depth research needed to support the sweeping claims in the report. "There is no real substantiation of their claims about the failings of SEAs or the efficacy of their solutions; it relies on the preponderance of secondary evidence as the authors' have collected it." Cookson further states: "The methods used in this report are more akin to advocacy journalism than scholarship."
|Report Reviewed:||New York City's Children First: Lessons in School Reform|
|Publisher/Think Tank:||Center for American Progress|
|A recent report from the Center for American Progress summarizes and evaluates the public education reforms during the Bloomberg mayoral era in NYC. The report suggests that positive change resulted from the Children First reforms and makes several policy recommendations. The report describes four aspects of reform during the Bloomberg era, and makes recommendations for NYC and other urban districts. It focuses on top-down school reforms from the mayor's office, rather than on instructional changes inside classrooms.|
|Think Twice Review Date:||June 3, 2014|
|Reviewer:||Randal Reback, Barnard College|
|An academic review finds that the report does an impressive job summarizing the reforms into a single, short narrative. However, several of the claims overhype the success of small high schools and charter schools. In his review, Randall Reback found several misleading claims and endorsements for the success of New York's small school movement. Regarding these claims, he says: "It is thus difficult to separate whether successful small schools were successful due to their ‘smallness' or due to other, correlated factors." He also finds that the report exaggerates student performance outcomes for both charter schools and traditional public schools. For example, he discusses how the report fails to mention concerns about inflated high school graduation rates.
|Report Reviewed:||Time to Improve: How Federal Policy Can Promote Better Prepared Teachers and School Leaders|
|Publisher/Think Tank:||New America Foundation|
Melissa Tooley and Laura Bornfreund
|The report recommends federal regulation of teacher-education programs on the basis of how well their graduates' pupils score on standardized tests.|
|Report Reviewed:||Fast Start: Training Better Teachers Faster, with Focus, Practice, and Feedback|
Ana Menezes and Adam Maier
|The report recommends giving new teachers the tools they need to be successful and proposes a five-week pre-service preparation model. The recommendations include several strategic changes: (1) a narrower curriculum focused only on the most essential teaching skills; (2) an emphasis on practicing skills instead of just learning about them; and (3) intensive coaching that provides regular, specific feedback on changes to instructional techniques.|
|Think Twice Review Date:||May 29, 2014|
|Reviewer:||William J. Mathis, University of Colorado Boulder|
|In his review, Mathis finds that neither report is grounded in research. He also singles out problems in each report. Time to Improve ignores a large body of recent experience and research that cautions against its proposal to the extent that the use of standardized tests scores to assess the quality of teacher preparation programs. Mathis calls the proposed changes in Fast Start "basic survival training" and not teacher preparation. Fast Start fails to show how proposed changes are superior to other approaches.
|Report Reviewed:||Charter School Funding: Inequity Expands Department of Education Reform (DER) at the University of Arkansas|
|Publisher/Think Tank:||Department of Education Reform (DER) at the University of Arkansas|
|Author:||Meagan Batdorff, Larry Maloney, Jay F. May, Sheree T. Speakman, Patrick J. Wolf, Albert Cheng|
|The authors of the report claim large and growing inequities between district funding provided through state, local, federal and other sources and charter school revenues from those same sources, even after accounting for differences in student needs.|
|Think Twice Review Date:||May 20, 2014|
|Reviewer:||Bruce Baker, Rutgers University|
|In his review, Bruce Baker finds that the report has one overarching flaw that invalidates all of its findings and conclusions, "the report displays complete lack of understanding of intergovernmental fiscal relationships, which results in the blatantly erroneous assignment of ‘revenues' between charters and district schools." Baker further states that the report ignores district funding that passes through district schools to charter schools in most states. The serious flaws in the report invalidate its conclusions and any subsequent return-on-investment comparisons claiming they're a better deal because they receive less funding and yet perform as well if not better than traditional public schools.
|Report Reviewed:||The Effect of Co-locations on Student Achievement in NYC Public Schools|
|Publisher/Think Tank:||The Manhattan Institute|
Marcus A. Winters
|The report from the Manhattan Institute investigated the effect of colocations of charter schools and traditional public schools on a narrow range of standardized test scores. It focused on students' test scores and looked at fourth through eighth grade ELA and math standardized assessment gains. Colocation was not found to have a significant impact on students' test scores.|
|Think Twice Review Date:||April 15, 2014|
|Reviewer:||Tina Trujillo and Marialena Rivera, UC-Berkeley|
|An academic review of the report finds that the report omits important details about its analysis. Additionally, the report does not build on existing research or background knowledge on colocations or related topics. Furthermore, the report expressly neglects to consider other important outcomes related to students' socio-emotional development, safety, health, and broader academic experiences. The report does little to help policymakers and practitioners evaluate the effects of colocation on students' educational experiences and outcomes.
|Report Reviewed:||Measuring the Impacts of Teachers I (NBER No. 19423) & II (NBER No. 19424)|
|Publisher/Think Tank:||National Bureau of Economic Research (NBER)|
|Author:||Raj Chetty, John N. Friedman, and Jonah E. Rockoff (both papers)|
|This NBER working paper - split in two parts - studied the impact of teachers on long-term outcomes using value-added scores. The reports addressed a key question: Are teachers' impacts on students' test scores ("value added") a good measure of their quality? Economists Raj Chetty, John Friedman, & Jonah Rockoff claim in a two-part paper that higher value-added scores for teachers lead to greater economic success for their students later in life. The paper drew widespread media and politicians' attention, including the president's.|
|Think Twice Review Date:||April 10, 2014|
|Reviewer:||Moshe Adler, Columbia University & Empire State College, SUNY|
|Moshe Adler of the department of Urban Planning at Columbia University and the Harry Van Arsdale, Jr. Center for Labor Studies at Empire State College, SUNY reviewed the reports for the Think Twice think tank review project. In his review Adler found five problems that invalidate the paper's main claim: (1) the paper fails to mention the existence of a crucial conflicting result that was reported in an earlier version of the same paper; (2) the paper claims that there was insufficient data to investigate whether teacher value-added has an effect on income at any age after the age of 28. This claim is untrue; (3) the method used to calculate the effect of teacher value-added on income at age 28 is biased and misleading; (4) the paper makes assumptions that inflate its main result, but the evidence contradicts these assumptions; and (5) the studies that the paper cites in support of its methodology don't actually provide that support.
|Report Reviewed:||The Economic Benefits of New York City's Public School Reforms, 2002-2013|
|Publisher/Think Tank:||Sonecon, Inc.|
Robert J. Shapiro, Sonecon, Inc. &
|An estimated $74 billion impact was attributed to the reforms (based on earnings of students who graduated under the reforms – who might not have otherwise – and on property values). The report assumes that higher graduation rates and charter school availability increased residential property values in the city.|
|Think Twice Review Date:||March 31, 2014|
|Reviewer:||Sean P. Corcoran, New York University|
|Professor Sean Corcoran writes, "While such estimates are always an exercise in some level of speculation, this report relies on highly inappropriate assumptions to reach its conclusions." Breaking down the math of the authors, Corcoran calculates that the impact on property values attributed to the Bloomberg-era educational reforms is comparable to "two-thirds of the entire increase in residential property values between 2007 and 2013." Corcoran found that many NYC public school students did experience " real educational and economic gains" during Bloomberg's time in office, but the estimates that the Sonecon report makes, he concludes, "are pure fantasy."
|Report Reviewed:||Updating Career and Technical Education for the 21st Century|
|Publisher/Think Tank:||The Lexington Institute|
|Author:||Kristen Nye Larson|
|A recent publication from the Lexington Institute makes the claim that schools are failing to unleash the potential of Career and Technical Education (CTE). The report claims that too many CTE programs offered by today's high schools are outdated and fail to produce 21st century workplace skills. The report showcased effective CTE models underway in the U.S. and suggests elements for replication in other schools. It identified three strategies for replication: (1) partnerships with businesses; (2) improved accountability measures; and (3) innovative curriculum options.|
|Think Twice Review Date:||March 18, 2014|
|Reviewer:||Marisa Saunders and Jaime L. Del Razo, Annenberg Institute for School Reform at Brown University|
|A review of the report found that it does not identify or produce evidence on the proposed practices. Furthermore, it fails to provide rationale for why the recommended practices are key to improving CTE experiences or opportunities for students. More important, the review finds that the report focuses on workforce readiness and does not identify postsecondary readiness as a critical component. It reinforces the belief that CTE and a college prep curriculum are mutually exclusive. The reviewers note: "The report is limited in its usefulness, both in practice and policy. It both over-reaches and under-reaches." The report makes broad claims that are "superficial, inconsistent, and lack a coherency."
|Report Reviewed:||A New Frontier: Utilizing Charter Schooling to Strengthen Rural Education|
|Publisher/Think Tank:||Bellwether Education Partners; J. A. and Kathryn Albertson Foundation|
|A recent document from Bellwether Education Partners argues for expanding charter schools into rurally populated states. It recommends opening the rural charter market and removing barriers to expansion. It highlights several state-level policy recommendations and offers five directives to policymakers: (1) replace barriers to growth with smart, flexible policies; (2) provide flexibility from teacher certification rules; (3) provide fair funding; (4) make facilities accessible; and (5) leverage technology.|
|Think Twice Review Date:||March 13, 2014|
|Reviewer:||Craig Howley, Ohio University|
|In a review, Ohio University's Craig Howley finds that the document provides no articulated rationale for its recommendations. Citing missing research and slanted representations, Howley declares it "useless as a source of objective information." The document employed very little peer-reviewed research and no reference was found to any peer-reviewed empirical research in rural education. Howley notes that the methods employed by New Frontier are entirely rhetorical, based on unstated assumptions that charter schools provide positive outcomes in all regards. Also, Howley finds the document "obscures rather than illuminates consideration of rural education, making it more harmful than helpful." In summarizing the usefulness of New Frontier, Howley says, "The document is useful only to those who seek to expand charter schools into rural areas."
|Report Reviewed:||Pluck & Tenacity: How Five Private Schools in Ohio Have Adapted to Vouchers|
|Publisher/Think Tank:||The Thomas B. Fordham Institute|
|This report highlights perceived challenges facing private schools brought on by the introduction of vouchers and calls for policymakers to strengthen Ohio's voucher programs. It praises the private schools profiled and cautions against state accountability programs that may "trample" on private schools' ability to provide a unique learning experience.|
|Think Twice Review Date:||March 6, 2013|
|Reviewer:||Christopher Lubienski and T. Jameson Brewer, University of Illinois|
|Overall, the reviewers found the report to be a case-study in how to engage in a slanted selection and interpretation of research evidence. The reviewers note that the report is supported by a total of three endnote references "cherry-picked" to support a contested policy agenda. "Thus, the report is grounded in a twice-skewed and intellectually dishonest view of the research on vouchers and their academic outcomes."
|Report Reviewed:||A Legal Lever for Enhancing Productivity|
|Publisher/Think Tank:||George W. Bush Institute|
|Author:||Elizabeth Ettema, Krishanu Sengupta, and Sandy Kress|
|This report examines the extent to which the Texas education system is efficient. Using a five-factor definition of efficiency, the authors argue that in key areas – teacher training, teacher evaluation, teacher pay-setting, and use of instructional materials – the Texas education system is unlikely to be efficient or cannot demonstrate efficiency.|
|Think Twice Review Date:||February 27, 2014|
|Reviewer:||Clive Belfield, Queens College, City University of New York|
|Dr. Clive Belfield found that the authors failed to prove that the Texas education system is inefficient. The review noted positively on the theoretical framework (x-efficiency) used to expand the concept of efficiency beyond the common economic definition. However, the report suffers from a lack of evidence and sufficient analysis to draw the conclusions made. The report is of limited use for policymakers and education professionals.
|Report Reviewed:||Should Charter Schools Pay Rent? Implications for Staffing and Growth|
|Publisher/Think Tank:||Manhattan Institute|
|This report from the Manhattan Institute examined the potential impact of requiring co-located charter schools to pay rent in NYC. The report reflects concerns of charter advocates and operators regarding potential policy changes under Bill de Blasio, New York's new mayor. The report claims that charging rent to co-located charters in NYC would have triggered an average budget deficit of 10.7 percent at those schools. The report also proposes that paying rent could cripple the co-located charters' growth in NYC.|
|Think Twice Review Date:||February 20, 2014|
|Reviewer:||Bruce Baker, Rutgers University|
|Bruce Baker finds that the report only presents a handful of poorly documented tables and graphs listing potential budget deficits, speculative layoffs, and average proficiency rates of co-located and non-co-located charter schools. Baker says that the report's greatest weakness is in its assumption that there is no possible downside when resources are transferred from city schools to charter schools. The report assumes that subsidies benefit charter schools and halting these subsidies harms charters and benefits no one. More importantly, Baker finds that the report ignores the broader and more complex policy questions of what it takes to manage a balanced and equitable system of schooling options.
|Report Reviewed:||Transfer Incentives for High-Performing Teachers: Final Results from a Multisite Randomized Experiment|
|Publisher/Think Tank:||Mathematica Policy Research (MPR)|
|Authors:||Steven Glazerman, Ali Protik, Bing-ru Teh, Julie Bruch, and Jeffery Max|
|The Teacher Transfer Initiative (TTI) actively encouraged 1,500 "high-performing" elementary teachers in 10 urban school districts to transfer to low-performing schools in return for a total stipend of $20,000 distributed over two years. The teachers were all identified as having students who were in the top 20 percent in their districts, based on a value-added measure of standardized test scores. The assumption was that better teachers would get better results on standardized tests wherever they teach.|
|Think Twice Review Date:||January 28, 2014|
|Reviewer:||Donald Gratz, Curry College|
|The results of this extensive study are not encouraging, writes reviewer Donald Gratz, who found several flaws that limit the report's usefulness for policy consideration. In his review, Gratz identifies several flaws. Among these: (1) the study does not consider the potential impact of other relevant school factors that affect teacher success, such as school leadership; and (2) while it considers the overall cost of the program, it only compares that cost to class size reduction, a generally expensive program. The most significant problem, Gratz says, is that this two-year study was too short to draw conclusions either about teacher retention or student achievement, limiting its impact and usefulness
|Report Reviewed:||Fixing Classroom Observations: How Common Core Will Change the Way We Look at Teaching|
|Publisher/Think Tank:||TNTP, Student Achievement Partners|
|This report suggests that many states and school districts are rolling out new teacher evaluation systems that will not succeed in improving instruction because they have not been updated to reflect the Common Core. The report asserts that new evaluation systems have not remedied practical challenges of classroom observations. It calls for increased emphasis on lesson content and a paring down of observation rubrics – to make them more focused and clear.|
|Think Twice Review Date:||January 21, 2014|
|Reviewer:||Jennie Whitcomb, University of Colorado Boulder|
|Dr. Jennie Whitcomb, associate dean for teacher education at the University of Colorado Boulder, concludes that simply streamlining the instruments and paying greater attention to what content is taught are unlikely to address the core problems surrounding teacher evaluations. Rather than solving implementation problems facing schools, Whitcomb says, "The report appears to be a sales-pitch for TNTP's soon-to-be-launched observation tools focused on Common Core lesson content."
Previous Reports & Reviews:
Think Twice Reviews Released In 2013
Think Twice Reviews Released In 2012
Think Twice Reviews Released In 2011
Think Twice Reviews Released In 2010
Think Twice Reviews Released In 2009
Think Twice Reviews Released In 2008
Think Twice Reviews Released In 2007
Think Twice Reviews Released In 2006
Previous Weekly Reports: