ABOUT   DONATE   SUBSCRIBE

ABOUT   DONATE   SUBSCRIBE

Facebook Twitter LinkedIn rss
Great Lakes Center Logo

2016 Think Twice Reviews

Think Twice is one of the nation's first efforts to serve as a watchdog to review think tank research on public education issues and policies, ensuring that published work meets the quality and standards of university scholarship. As think tank research becomes increasingly important reference sources in public policy debates, media and other critics have called for increased scrutiny to ensure validity and objectivity (click here to see related stories).

The goal of the Think Twice project is to provide the public, policy makers and the press with timely academically sound reviews of selected think tank publications.

Click Here To Read Our Legal Statement

Think Tank Research Quality

Think Tank Research Quality:
Lessons for Policy Makers, the Media, and the Public

Think Tank Research Quality, edited by Kevin Welner of the University of Colorado at Boulder, Patricia H. Hinchey of Pennsylvania State University, Alex Molnar of Arizona State University, and independent researcher Don Weitzman, offers clearly written, jargon-free expert reviews of studies on topics such as vouchers, charter schools, and alternative teacher certification. Friends of The Great Lakes Center receive 20% off phone, fax or mail-in orders...not valid online.

Reports & Reviews for 2016

Report Reviewed:

Squeezing the Public School Districts: The Fiscal Effects of Eliminating the Louisiana Scholarship Program

and

The Fiscal Effect of Eliminating the Louisiana Scholarship Program on State Education

Publisher/Think Tank: Department of Education Reform (DER) at the University of Arkansas
Author: Corey A. DeAngelis & Julie R. Trivitt
Two recent papers from the Department of Education Reform (DER) at the University of Arkansas predicted the budgetary consequences of ending the Louisiana Scholarship Program (LSP). The two closely allied papers concluded that terminating the LSP would not relieve pressure on public spending in Louisiana. The first paper uses an economic model to conclude that terminating the LSP would increase costs. While, the second paper confirms the findings from the first paper and advances that the findings would apply to almost all school districts in Louisiana.
 
Think Twice Review Date: September 29, 2016
Reviewer: Clive Belfield, Queens College, City University of New York
A new review of the papers says that the findings are reasonable, but they fail to make a convincing case. In his review Belfield says, “These papers are useful in that they apply an economic model to understanding the consequences of voucher systems.” However, he says that the papers do not make a fully convincing case. Additionally, Belfield questions the omission of contradictory evidence provided by the state’s Legislative Fiscal Office. Belfield concludes, “There may be savings or additional expenditures, depending on several key parameters which have not been precisely estimated.” Regardless of the economic consequences, Belfield adds, “the poor performance of LSP might in itself be justification for termination.”


Report Reviewed: A 21st Century School System in the Mile-High City
Publisher/Think Tank: Progressive Policy Institute
Author: David Osborne
A recent report from the Progressive Policy Institute advocated for expanded use of the “portfolio model” of school governance. The report drew on contemporary reforms adopted by the Denver Public Schools. It claimed that reforms enacted by the Denver Public Schools (viz., portfolio model, school closings, and charter expansion) positively impacted student test scores. The goal was to highlight Denver as a success story that should be copied elsewhere.
 
Think Twice Review Date: July 26, 2016
Reviewer: Terrenda White, University of Colorado Boulder
An academic review by Terrenda White, University of Colorado Boulder, finds that the report oversteps its claims in an attempt to positively portray Denver’s reforms. The review raises serious concerns about the claims made. Most notably, she finds that the only data presented were simple charts that lacked any conventional statistical analysis. Regarding the report’s use of research literature, White finds that the report drew primarily from advocacy publications from partisan foundations. In her conclusion, White says: “Ironically, the report celebrates Denver’s portfolio strategy and its expansion of charter schools as a model of 21st century reform at a time when other cities have discovered that charters are not panaceas.”


Report Reviewed: Measuring School Turnaround Success
Publisher/Think Tank: Public Impact and The Center on School Turnaround (WestEd)
Author: Cassie Lutterloh, Jeanette P. Cornier, & Bryan C. Hassel
A report produced by Public Impact for WestEd’s Center on School Turnaround asserted that because there is a lack of a shared definition of school turnaround success for states and districts, it is difficult for school districts to learn from one another. The report claimed to provide a model for defining school turnaround success. The report bases its model framework for determining turnaround success on three school-level measures: (1) proficiency in reading and math on state assessments; (2) growth in reading and math on state assessments; and (3) graduate rate (for high schools).
 
Think Twice Review Date: July 20, 2016
Reviewer: Tina Trujillo, University of California Berkeley; and
Marialena Rivera, Texas State University
A review of the report found following shortcomings: (1) Sufficient research evidence for the claims made is not presented in the report; (2) It lacks sound methodological techniques; and (3) It omits several rigorous, peer-reviewed studies that contradict a majority of its proposals. The review concludes that the proposed model continues the trend of relying on flawed, test-centered strategies.


Report Reviewed: Bang For The Buck: Which Public Schools In Milwaukee Produce The Best Outcomes Per Dollar Spent?
Publisher/Think Tank: Wisconsin Institute for Law and Liberty (WILL)
Author: Will Flanders and C.J. Szafir
This report from the Wisconsin Institute for Law and Liberty (WILL) attempted to estimate efficiency scores for schools in Milwaukee and make the case for greater autonomy for charter schools. The scores were used in an attempt to draw conclusions about the relative “efficiency” of Milwaukee’s charter and traditional public schools. The report used a simple analysis of math and science scores divided by per pupil funding. A second analysis from the report generated “efficiency scores” by using a regression model, which accounted for selected demographic characteristics. The authors of the report claimed that schools with greater autonomy were more efficient than schools with less autonomy.
 
Think Twice Review Date: July 12, 2016
Reviewer: Casey Cobb, University of Connecticut
A review of the report found the strong claims about the relative efficiencies to be too weak to serve any useful function for policymakers. Five major problems arise from the report’s approach, Casey Cobb finds: (1) Test scores do not comprehensively represent the purposes of schools; (2) The report does not address threats to the validity of its assumption that there is uniform financial accounting across schools and types; (3) The analytic description of the study is incomplete, making interpretation difficult; (4) “Autonomy” is never really defined—it is just used as a loose term implying independence—so autonomous behavior is assumed by virtue of their charter status. The report then makes strong but unmeasured claims about the superior “efficiency” of charter schools based on their having this greater autonomy; and (5) While the report’s analysis controls for some school demographic characteristics, it does not appear to adjust for selection effects; effects that could prove fatal to their conclusions.


First Document Reviewed: A Win-Win Solution: The Empirical Evidence on School Choice
Publisher/Think Tank: Friedman Foundation for Educational Choice
Author: Greg Forster
Second Document Reviewed: The Participant Effects of Private School Vouchers across the Globe: A Meta-Analytic and Systematic Review
Publisher/Think Tank: Department of Education Reform (DER) at the University of Arkansas
Author(s): M. Danish Shakeel, Kaitlin P. Anderson, & Patrick J. Wolf
Two reports claimed to offer empirical support of the effectiveness of school vouchers. The reviewed reports include: A Win-Win Solution: The Empirical Evidence on School Choice from the Friedman Foundation for Educational Choice; and The Participant Effects of Private School Vouchers across the Globe: A Meta-Analytic and Systematic Review from the Department of Education Reform (DER) at the University of Arkansas. The reports focused on randomized studies of the effects of school vouchers on educational outcomes, concluding that school vouchers have positive effects.
 
Think Twice Review Date: June 30, 2016
Reviewer: Christopher Lubienski, University of Illinois Urbana-Champaign
An academic review released today finds the reports have serious problems and errors. Christopher Lubienski, University of Illinois Urbana-Champaign, reviewed both reports for the Think Twice think tank review project. Lubienski concludes: “Both reports are marred by a number of serious problems and errors, including misrepresentations of the research literature, a failure to acknowledge the limitations of their approaches, not addressing the shortcomings of the theoretical underpinnings of vouchers, and the use of methods that bias the selections of the studies they utilize.” He adds, “Together, their manifold serious flaws undercut the trustworthiness and usefulness of these reports.”


Report Reviewed: School Spending and Student Achievement in Michigan: What’s the Relationship?
Publisher/Think Tank: Mackinac Center for Public Policy
Author: Ben DeGrow and Ed Hoang
A report released in April by the Mackinac Center for Public Policy made the claim that spending more on Michigan schools doesn’t increase achievement. The report asserted that there is little or no relationship between student achievement and marginal increases in school spending in the state. The report characterizes spending levels in Michigan as “high,” but never substantiates the claim. In summary, the report argues that spending increases in Michigan would provide little or no gain in student achievement, and that spending on increased salaries or reduced class sizes would be inefficient or ineffective.
 
Think Twice Review Date: May 13, 2016
Reviewer: Bruce D. Baker, Rutgers University
An academic review of the Mackinac Center report finds that the report clashes with existing research about the positive impact of funding nationally and in Michigan. In his review, professor Bruce D. Baker says that the report lacks evidence to substantiate the claims. According to Baker, the report "wrongly assumes that all Michigan districts are now high spending and that none could benefit from any marginal increase to funding; it fails to evaluate thoroughly the overall level of spending in context, nor does it adequately consider whether and to what extent spending varies across children and contexts within Michigan."


Report Reviewed: 2016 Brown Center Report on American Education — Part II: Tracking and Advanced Placement
Publisher/Think Tank: Brown Center on Education Policy at Brookings
Author: Tom Loveless
Based on the logic that states with more tracking tend to have more students passing AP exams in high school, a recent report from the Brown Center on Education Policy at the Brookings Institution claimed that tracking in eighth grade would promote greater equity. The Brown Center report, authored by Tom Loveless, used state-level data from the NAEP to describe a positive association between tracking in eighth grade and larger percentages of high-scoring AP test takers. Based on the findings, the author claims that tracking is beneficial for high-achieving students.
 
Think Twice Review Date: April 26, 2016
Reviewer: Marshall Jean, University of Chicago
An academic review of the report questions the impact of tracking policies on low-achieving students, and calls attention to the potential for tracking practices to exacerbate existing educational inequalities. Ultimately, the reviewer, Marshall Jean (University of Chicago) finds the claim that tracking has the potential for promoting equity to be “dubious.” He concludes, “unless tracking systems are implemented carefully and coincide with substantial supports for struggling students, students assigned to low-ranking tracks are likely to be harmed.”


Report Reviewed: The Policy Framework for Online Charter Schools
Publisher/Think Tank: Center for Reinventing Public Education (CRPE)
Author: Rosa Pazhouh, Robin Lake, Larry Miller
A study from the Center for Reinventing Public Education (CRPE), an organization that often advocates for charter schools, attempted to provide an in-depth analysis of policy features across the states that allow for online charter schools. The report concluded that online charter schools should use a separate regulatory framework than brick-and-mortar charter schools.  The report is part of a series of three, which were sponsored by the Walton Foundation.
 
Think Twice Review Date: April 18, 2016
Reviewer: Gary Miron, Western Michigan University
An academic review of the report finds that it potentially offers a solid contribution for policymaking. In his review of the report, professor Gary Miron, Western Michigan University, describes it as a well-organized description of policy features, and that the policy recommendations generally, but not always, follow well from the study’s evidence.


Report Reviewed: The School Choice Voucher: A “Get Out of Jail” Card?
Publisher/Think Tank: Department of Education Reform, University of Arkansas
Author: Corey DeAngelis and Patrick J. Wolf
The authors of the working paper used data from the Milwaukee Parental Choice Program (MPCP) to compare crimes processed through the Wisconsin courts for program participants and a matched sample of students. Although most of these comparisons showed no association, the report finds that some subgroups of MPCP students were less likely to commit crimes as adults.
 
Think Twice Review Date: April 6, 2016
Reviewer: Clive Belfield, Queens College, City University of New York (CUNY)
An academic review of the paper finds that it has several weaknesses and does not support the implication that voucher programs caused a reduction in crime. Clive Belfield, Queens College (CUNY) is very clear in his review, the findings from the paper should not warrant any strong claims of voucher effects on crime. Belfield raises the following concerns: (1) the study’s results are highly variable, with most comparisons showing statistically insignificant results; (2) the methods used do not justify a causal inference; and (3) the evidence, despite the paper’s title, is associational, not causal. In short, Belfield cautions that a better interpretation of the report would be that vouchers and crime are, in fact, not correlated.


Report Reviewed: Learning about Learning: What Every New Teacher Needs to Know
Publisher/Think Tank: National Council on Teacher Quality (NCTQ)
Author: Laura Pomerance, Julie Greenberg and Kate Walsh
As part of a series of reports chronicling teacher education, the National Council on Teacher Quality (NCTQ) attempted to investigate the textbooks used in teacher prep coursework. The report contended that textbooks assigned in methods coursework fail to provide teacher candidates with proper preparation. The NCTQ report is based on six strategies adapted from a practice guide provided by the What Works Clearinghouse, which is funded by the U.S. Department of Education’s Institute of Education Sciences (IES). The report offers recommendations for textbook publishers, teacher education programs, and state departments of education.
 
Think Twice Review Date: March 24, 2016
Reviewer: P.L. Thomas, Furman University; and Christian Z. Goering, University of Arkansas
However, the reviewers note that the report is not grounded in a comprehensive examination of the literature on teaching methods. Moreover, the report relies on a single source, which itself was based on a narrow set of research studies. The reviewers find that, despite posing an interesting question, NCTQ’s report falls short because it uses a narrow set of criteria and applies it in a misleading and superficial way. They conclude: “this report in no way justifies taking the conclusions or recommendations seriously when determining policy or practice.”


Report Reviewed: SchoolGrades.org
Publisher/Think Tank: Manhattan Institute for Policy Research
Author: Jacob L. Vigdor and Josh B. McGee
The Manhattan Institute for Policy Research launched a website, SchoolGrades.org, which aimed to provide a means to compare how well America’s schools prepare students in core subjects. The website attempts to evaluate and assign letter grades to schools using reading and math test scores. The website claims that the school grades found on the site allow parents to compare local schools against schools in other countries using a four-step process.
 
Think Twice Review Date: March 10, 2016
Reviewer: Jaime L. Del Razo, Annenberg Institute for School Reform & Brown University
The review finds that the unsubstantiated norming involved in the process of creating school grades is too tenuous and the results are overly extrapolated, which diminishes their value. A technical analysis also finds that the website fails to explain how: (1) international scores are equated to a national standard created by the website; (2) letter grades are determined; and (3) free and reduced lunch counts were used to make socioeconomic adjustments. In his conclusion, reviewer Jaime L. Del Razo indicates that the site fails on two grounds; technical and philosophical. He says, “the Manhattan Institute’s website fails to advance policy not only on the technical shortcomings of its efforts but most importantly, for not appreciating the [broader] purposes of education.”


Report Reviewed: Lessons From State Performance on NAEP: Why Some High-Poverty Students Score Better Than Others
Publisher/Think Tank: Center for American Progress
Author: Ulrich Boser and Catherine Brown
A report from the Center for American Progress (CAP) attempted to examine the impact of standards-based policies between 2003 and 2013. The report explored whether states’ adoption of standards-based policies predicts low-income students’ NAEP achievement trends in fourth and eighth grade math and reading. The report attempted to highlight that the difference between high and low scoring states was the adoption of high standards, specifically, the Common Core State Standards.
 
Think Twice Review Date: February 25, 2016
Reviewer: Sharon L. Nichols, University of Texas San Antonio
A review finds that it employs inappropriate research methods, fails to adequately define its approach, and reports only incomplete findings from its analyses. Additionally, according to the review, the report does not adequately describe variables or analytic methods, and the data and methods used do not allow for any causal findings. The review also finds that while the report claims to analyze changes across five separate two-year intervals, it only reports findings for 2009-2011. And the positive results are statistically significant only at the generally unacceptable 0.10 level of significance.


Report Reviewed: Special Education and English Language Learner Students in Boston Charter Schools
Publisher/Think Tank: School Effectiveness & Inequality Initiative (SEII); Department of Economics at MIT
Author: Elizabeth Setren
The School Effectiveness & Inequality Initiative (SEII), housed in the Department of Economics at MIT, released a working paper in December 2015. The report sought to investigate the enrollment and achievement of students with special needs and English language learners (ELLs) in oversubscribed charter schools in Boston. The SEII working paper claimed to “debunk” the common perception that students with special needs and ELLs are underserved in charters. The report focused on Boston charter schools and Boston Public Schools (BPS) that enrolled similar special populations.
 
Think Twice Review Date: February 17, 2016
Reviewer: Julie Mead, University of Wisconsin- Madison; and Mark Weber, Rutgers University
The reviewers found some limitations to the paper’s methods; however, they find the primary claim regarding test score effects to be on solid ground. Additionally, the models used to estimate the effects were also found to be appropriate. The reviewers do point out the following limitations in the working paper: (1) the data and analyses are more limited than readers of the report might be lead to believe; (2) the effects can only be generalized to those students who enter the lottery and comply with their assignment (charter or traditional public school); and (3) there is no context provided to compare the size of reported test score gains, or how those test score gains were realized (e.g., unaccounted peer effects and spending differences). Reviewers Mead and Weber ultimately conclude that, while this report takes an important step in studying how oversubscribed charters may affect the academic achievement of special needs students, a closer examination is needed in order to accurately inform those making education policy.


Report Reviewed: Continued Progress: Promising Evidence on Personalized Learning
Publisher/Think Tank: RAND Corporation
Author: John F. Pane, Elizabeth D. Steiner, Matthew D. Baird, & Laura S. Hamilton
A recent report published by the RAND Corporation focused on three school-wide initiatives funded by the Bill and Melinda Gates Foundation. The report concentrated on approaches touted as “personalized learning.” The report sought to add to the evidence base by examining the effects of school-wide efforts to promote personalized learning and the links between implementation of particular strategies and outcomes.
 
Think Twice Review Date: January 26, 2016
Reviewer: William R. Penuel and Raymond Johnson, University of Colorado Boulder
Encouragingly, the reviewers state that the report did include high-quality elements. Yet, the conclusions about the efficacy of technology-based personalized learning are not warranted by the research presented. Readers should be skeptical of what promise the report’s evidence actually provides for any given model of personalized learning. The reviewers conclude, “the study lacks utility for judging the value of the more disruptive and digital-technology-based personalized learning.” Limitations identified by the review include: (1) sample of treatment schools unrepresentative of the general population of schools; (2) the lack of a threshold in the study for what qualified as implementing “personalized learning” in treatment schools; and (3) that several strategies highlighted were rarely implemented in the studied schools.


Report Reviewed: Smart, Skilled, and Striving: Transforming and Elevating the Teaching Profession
Publisher/Think Tank: Center for American Progress (CAP)
Author: Carmel Martin, Lisette Partelow, and Catherine Brown
This Center for American Progress (CAP) report outlined a vision for elevating and modernizing the teaching profession. The report provided ten recommendations for improving the public perceptions and experiences of teachers. The report argued that if we do not change the perception of the teaching profession, schools would not be able to recruit “high achieving young people” into teaching.
 
Think Twice Review Date: January 7, 2016
Reviewer: Elizabeth J. Meyer, University of Colorado Boulder
In an academic review, Elizabeth J. Meyer, University of Colorado Boulder, notes that elements of the report’s recommendations are likely to be beneficial. However, she also finds that the some of the report’s recommendations could likely have the opposite effect by: (1) increasing the surveillance of teachers; (2) reducing the job security of teachers; (3) evaluating teachers based on students’ test scores; and (4) using merit pay systems. The review also finds that the report relies too heavily on popular rhetoric, sound bites, opinion articles, and advocacy publications. In conclusion, Meyer says, “Other than a review of contemporary issues, the report offers little of substance to advance the teaching profession.”


Report Reviewed: The Integration Anomaly: Comparing the Effects of K-12 Education Delivery Models on Segregation in Schools
Publisher/Think Tank: Friedman Foundation for Educational Choice
Author: Benjamin Scafidi
A report from the Friedman Foundation for Educational Choice claimed that universal school choice offers a solution to increasing school segregation. The report argued that competition unleashed by unrestricted school choice would promote integration. The report suggested that housing integration has not been an effective way to pursue school integration, and it concludes with recommendations for how to structure school choice to achieve integration.
 
Think Twice Review Date: January 5, 2016
Reviewer: Genevieve Siegel-Hawley, Virginia Commonwealth University; & Erica Frankenberg, Penn State University
An academic review of the report from Genevieve Siegel-Hawley, an assistant professor at Virginia Commonwealth University, and Erica Frankenberg, an associate professor at Penn State University finds that the arguments are not based on evidence. The reviewers further find that the analysis of the empirical relationship between school and residential segregation relies on flawed methodological decisions with regard to how to define segregation and divergent trends over time. Those problematic definitions, in turn, yield biased results and prompt the reader to incorrectly assume that housing integration policies will have little bearing on school segregation. Siegel-Hawley and Frankenberg also point out that the report’s use of research literature on school choice is haphazard and incomplete, drawing conclusions either beyond what the research supports or contrary to what research has found.


Report Reviewed: The Effects of Test-based Retention on Student Outcomes Over Time: Regression Discontinuity Evidence from Florida
Publisher/Think Tank: National Bureau of Economic Research (NBER)
Author: Guido Schwerdt and Martin R. West
As part of its working paper series, the National Bureau of Economic Research (NBER) recently released a report examining the outcomes of Florida’s third-grade retention policy. The report concluded, contrary to the conventional wisdom on grade retention, that third-grade retention had positive effects on the following year’s test results, but the effects fade, with no effect on graduation. The report attempted to investigate the impact of a Florida policy, which flags students for retention, to repeat third grade, based on a state-specified cut-score on the Florida Comprehensive Achievement Test. The findings indicated that students just below the threshold (one-third of whom were retained) performed better than those just above the threshold (5% of whom were retained) on next year’s tests.
 
Think Twice Review Date: December 3, 2015
Reviewer: Joseph P. Robinson-Cimpian, University of Illinois at Urbana-Champaign
Overall, the review points out, the methods used have extremely limited generalizability, which is restricted to students at or very near the threshold and directly affected by the policy. Even setting aside the problems generated by confounding retention effects with the effects of other interventions and supports, the findings are not easily generalizable to lower- or higher-achieving students, to other grades, or to other states with similar test-based retention policies.


Report Reviewed: The Hidden Value of Curriculum Reform: Do States and Districts Receive the Most Bang for their Curriculum Buck?
Publisher/Think Tank: Center for American Progress (CAP)
Author: Ulrich Boser, Matthew Chingos, and Chelsea Straus
The Center for American Progress (CAP) issued a recent report that drew on bold conclusions about the high payoff of better textbooks. The report claimed that, compared to other reforms, a switch to better textbooks was a cost-efficient way to improve student achievement. The report pointed to very real problems with textbook adoption, including poor alignment with standards. It also made strong claims about the payoff of schools’ investment in high-quality curriculum materials.
 
Think Twice Review Date: November 19, 2015
Reviewer: Sarah Lubienski, University of Illinois at Urbana-Champaign
The review points to clear merits of the report. However, the review also points out that the report overreaches in several areas. Specifically, the report is based on a single prior study and ignores key findings within the original study. Additionally, the report attempts to compare estimated return on investment (ROI) for textbooks against another study’s calculations, ignoring findings from that report as well. The report’s conclusions are marred by highly optimistic claims about curricular ROI.


Report Reviewed: Do We Already Have Universal Preschool?
Publisher/Think Tank: Brookings Institution
Author: Grover J. (Russ) Whitehurst and Ellie Klein
A Brookings Institution project called “Evidence Speaks” claims in a recent report that advocates exaggerate unmet need as well as the cost of universal pre-kindergarten. Unfortunately, an academic review of the report released today finds the report vastly underestimates unmet need and costs. The report estimates that 69 percent of all four-year-olds already attend preschool and that universal access tops out at 80 percent enrollment. The report estimated that only $2 to $4 billion per year would be needed.
 
Think Twice Review Date: November 12, 2015
Reviewer: W. Steven Barnett, National Institute for Early Education Research (NIEER) at Rutgers University
Unfortunately, an academic review of the report released today finds the report vastly underestimates unmet need and costs. According to the review, both estimates are based on serious factual errors and unfounded assumptions. Regarding the overall usefulness of the report, Professor W. Steven Barnett determines, “The report’s conclusions regarding needs and the costs to meet them are invalid and misleading, and it should not be used as a basis for policymaking.”


Report Reviewed: Ten Years in New Orleans: Public School Resurgence and the Path Ahead
Publisher/Think Tank: Public Impact and New Schools for New Orleans
Author: Christen Holly, Tim Field, Juli Kim, Bryan C. Hassel, Maggie Runyan-Shefa, Michael Stone, and Davis Zaunbrecher
A report published by Public Impact and New Schools for New Orleans attempted to review 10 years of education reforms in post-Katrina New Orleans, and the creation of a “portfolio model.” The report argued that the reform experiment has been an unquestioned success, and other cities should duplicate efforts. Overall, the report lays out a grand vision for the future of public education in New Orleans, which was described by the authors as “America’s first great urban public school system.” It is organized into six themes: governance, schools, talent, equity, community, and funders.
 
Think Twice Review Date: October 21, 2015
Reviewer: Adrienne D. Dixson, University of Illinois at Urbana-Champaign
An academic review of the report from Adrienne D. Dixson finds that the report does little to accurately inform policymakers or practitioners about the current state of public education in New Orleans or the viability of “portfolio” districts. The review found multiple weaknesses that limit the usefulness of the report. In her review, Dixson highlights that the report exaggerates improvements, while downplaying the ways in which the enacted reforms exacerbated inequities in New Orleans. She also notes that the report erroneously presents the reforms as a result of a logical and apolitical process.


Report Reviewed: The Mirage: Confronting the Hard Truth About our Quest for Teacher Development
Publisher/Think Tank: TNTP (previously The New Teacher Project)
A recent report from TNTP, previously The New Teacher Project, argued for changes in the way public school districts think about teacher development. The report identified a divide between teacher professional development and improved teacher evaluation scores. The report offers lessons about professional development in three large school districts and one mid-size charter network. Specifically, the report provides details about teachers’ time investment, how that time is spent, and district costs related to professional development.
 
Think Twice Review Date: September 30, 2015
Reviewer: Heather C. Hill, Harvard Graduate School of Education
Professor Heather C. Hill, Harvard Graduate School of Education, identified several strengths of the report and noted that readers could benefit from the report’s empirical evidence. However, she cautions that several of the reports conclusions are problematic. Weaknesses of the report identified by the reviewer, include: (1) problems with the analysis comparing teachers’ professional development and growth in teacher evaluation scores; (2) differences between academic conventions for calculating the cost of professional development and the methods used in the report; and (3) “hyperbolic” statements presented without sufficient support.


Report Reviewed: Measuring Diversity in Charter School Offerings
Publisher/Think Tank: American Enterprise Institute (AEI)
Author: Michael McShane and Jenn Hatfield
A recent report from the American Enterprise Institute (AEI) rates the diversity of charter school programs in 17 major cities.  The authors of the report advocate for the expansion and deregulation of charter schools since they provide greater program variety and, thus, respond to parental desires. The authors find small to moderate correlations between city demographics and certain types of charter schools. They also find that specialized charters tend to morph into homogenized general schools over time.
 
Think Twice Review Date: September 1, 2015
Reviewer: Arnold Danzig, San Jose State University; and
William J. Mathis, University of Colorado Boulder
The reviewers found several weaknesses: (1) the report claims charter schools provide greater program diversity, but fails to empirically compare charter offerings to traditional public school districts, which can also have diverse offerings; (2) the report claims but does not address how charter schools are hamstrung by red tape; and (3) the report’s findings were based on website descriptions, which can be error-prone. More specifically, the reviewers find that the correlations made in the report are weak, and relied on only 17 cases. Ultimately, the report fails to support its major claims and is of little use to practitioners or policymakers. The reviewers conclude, “Assuming we accept diversity of offerings as a primary policy goal, the report presents no evidence that charter schools do any better or worse than the current mix of public school alternatives.”

 

Report Reviewed: School Closures and Student Achievement: An Analysis of Ohio’s Urban District and Charter Schools
Publisher/Think Tank: Thomas B. Fordham Institute
Author: Deven Carleson and Stéphane Lavertu
This report from the Thomas B. Fordham Institute investigated school closures in Ohio for urban district and charter schools. The report found that test scores of displaced district students showed greater gains in math and reading relative to students from non-closed schools and that displaced charter students showed gains in math but not reading. Among the findings of the report, closed schools were attended by greater percentages of African American students from low-income families.  Additionally, the report noted that displaced students performed better if they transferred to schools with higher levels of performance.
 
Think Twice Review Date: June 16, 2015
Reviewer: Ben Kirshner, University of Colorado Boulder; and Matthew Gaertner, Pearson
An academic review of the report finds that, despite the encouraging results, they leave un-addressed core questions about closure policy. National and international media extoled the report’s findings and suggested that closing schools could improve educational outcomes. However, the reviewers caution that readers should not draw strong or straightforward conclusions from the study. The reviewers cited three primary concerns: (1) availability of higher-quality receiving schools that are easily accessible to students ought to be a precondition for a decision to close a school; (2) what counts as a better schooling option needs to be better defined, with the promise of safe, reliable transportation options; and (3) school closure does not provide a lasting solution to challenges associated with economic and racial segregation found in neighborhoods of concentrated poverty. The reviewers primary concern for the report focuses on the ability of parents and communities to make decisions about schools.

 

Report Reviewed: ESEA Reauthorization: How Can We Build upon No Child Left Behind’s Progress for Students with Disabilities in a Reauthorized ESEA
Publisher/Think Tank: Center for American Progress (CAP)
Author: Chelsea Straus
A report from the Center for American Progress (CAP) links benchmarks for inclusion in state testing and stringent accountability with improved educational outcomes for students with disabilities. While noting that causal claims could not be made, the report then proceeded to try to convince readers of exactly such claims. The report compared 2000 to 2013 NAEP test scores along with NCES national-level data and found increased test scores, decreased dropout rates, and increased graduation rates for students with disabilities, as well as improved outcomes for Black and Hispanic students with disabilities.
 
Think Twice Review Date: June 8, 2015
Reviewer: Edward Fierros & Katherine Cosner, Villanova University
An academic review by Edward Fierros & Katherine Cosner, Villanova University, explains that the causal arguments made in the report are nowhere near possible given the relatively weak data and analyses used in the report. Despite the report’s suggestions that tougher accountability and higher expectations produced these better educational outcomes, the reviewers found little to justify the claims. “While these student outcomes did indeed improve for students with disabilities during this time period,” the reviewers note “the report is wrong to assert that improvements were caused by NCLB or NCLB-type reforms.”


Report Reviewed: Dramatic Action, Dramatic Improvement: the Research on School Turnaround
Publisher/Think Tank: Center for American Progress (CAP)
Author: Tiffany D. Miller and Catherine Brown
The Center for American Progress (CAP) recently released a report advocating for implementation of evidence-based best practices for turning around low-performing schools through the federal School Improvement Grant (SIG) program. The report asserts that it offers five lessons about the most effective, research-based methods for turning around low-performing schools through the federal SIG program.
 
Think Twice Review Date: May 28, 2015
Reviewer: Tina Trujillo, University of California Berkeley
Tina Trujillo, in her review, finds that the report essentially ignores large bodies of research on high-stakes accountability, school improvement, and emerging evidence on school closures and federally funded turnarounds. Specifically, Trujillo cautions that a more inclusive review of extant research “reveals that the federal SIG program’s turnaround policies are based on unwarranted claims and are contradicted by the empirical evidence.” Trujillo’s review identifies that the report failed to meet standards for evidence and analytic transparency, and is of little use to policymakers or practitioners.


Report Reviewed: Increasing Education: What it Will and Will Not Do for Earnings and Earnings Inequality
Publisher/Think Tank: The Hamilton Project
Author: Brad Hershbein, Melissa S. Kearney, and Lawrence H. Summers
A recent report from the Hamilton Project sought to study how public investment in education will further long-term prosperity, economic growth, and individual economic security. The inquiry focused on whether or not a bachelor’s degree or higher education would increase economic prosperity and reduce economic inequality.
 
Think Twice Review Date: May 18, 2015
Reviewer: Marvin Lazerson, Central European University and University of Pennsylvania; and Ryan Pfleger, University of Colorado Boulder
In the review, the authors find that the report and its assertions are straightforward and use an empirically-based simulation for future projections. However, the report has several limitations. Specifically, Marvin Lazerson and Ryan Pfleger find the following insufficiencies: (1) there is little direct evidence in the report to show that increasing educational attainment is the most “efficient and effective,” as the report describes, way to improve prosperity; (2) the data are drawn are only from males, with no attention paid to gender, race, field of study, labor-market conditions, or institutional reputation; and (3) no data were analyzed to evaluate other ways to address economic problems. The reviewers conclude, “Claiming that the primary solution to a wide array of economic problems is to improve ‘human capital,’ the report perpetuates a problematic myth that undervalues alternative ways to address poverty and economic insecurity.”


Report Reviewed: Dramatic Action, Dramatic Improvement: the Research on School Turnaround
Publisher/Think Tank: Center for American Progress (CAP)
Author: Tiffany D. Miller and Catherine Brown
The Center for American Progress (CAP) recently released a report advocating for implementation of evidence-based best practices for turning around low-performing schools through the federal School Improvement Grant (SIG) program. The report asserts that it offers five lessons about the most effective, research-based methods for turning around low-performing schools through the federal SIG program.
 
Think Twice Review Date: May 11, 2015
Reviewer: Tina Trujillo, University of California Berkeley
Tina Trujillo, in her review, finds that the report essentially ignores large bodies of research on high-stakes accountability, school improvement, and emerging evidence on school closures and federally funded turnarounds. Specifically, Trujillo cautions that a more inclusive review of extant research “reveals that the federal SIG program’s turnaround policies are based on unwarranted claims and are contradicted by the empirical evidence.” Trujillo’s review identifies that the report failed to meet standards for evidence and analytic transparency, and is of little use to policymakers or practitioners.


Report Reviewed: Pushed Out? Low-Performing Students and New York City Charter Schools
Publisher/Think Tank: Manhattan Institute
Author: Marcus A. Winters
A Manhattan Institute report by Marcus A. Winters recently investigated claims that New York City charter schools are pushing out low-performing students as a means of inflating academic achievement scores. The report suggested that charter school exit rates were similar to traditional public schools (TPS). In short, the report determines that there is no charter push-out effect for low-performing students in charter schools.
 
Think Twice Review Date: May 4, 2015
Reviewer: Erica Frankenberg, Penn State University
An academic review of the report finds, despite a rich dataset available for the analysis, that the report has little detail and fails to provide guidance to policymakers. Frankenberg, in her review, discovers that: (1) the research design does not address its primary push-out question; (2) the report has little detail; (3) the report does not examine other relevant factors; and (4) there is a substantial gap between the findings and the conclusions of the report. Specifically, Frankenberg notes that the report’s findings show attrition rates for low-achieving students are higher than attrition of higher-performing students. The report’s conclusions that NYC charter schools are not pushing out students are unsupported.


Report Reviewed: Urban Charter School Study
Publisher/Think Tank: Center for Research on Education Outcomes (CREDO) at Stanford University
A recent report from the Center for Research on Education Outcomes (CREDO) at Stanford University attempted to investigate whether charter schools generate better outcomes than traditional public schools (TPS) in urban environments. The report, part of a series of reports on the performance of charter schools relative to TPS, asserts charter schools in urban environments provide a slightly greater test score advantage than those in non-urban environments. The report utilizes a methodological approach similar to previous reports from CREDO, and finds that students in urban charter schools were estimated to score approximately 0.055 standard deviations higher on math tests and 0.039 standard deviations higher on reading tests than their peers in urban TPSs.
 
Think Twice Review Date: April 27, 2015
Reviewer: Andrew Maul, University of California Santa Barbara
An academic review issues concerns with the methodology and reporting of the CREDO study. In a review, Andrew Maul cites the following concerns: (1) the study’s “virtual twin” technique is insufficiently documented; (2) the report’s estimation of growth using “days of learning” requires accepting untested assumptions; and (3) the study includes a number of arbitrary and unexplained analytic choices. All the same, Maul states, “Even setting aside such concerns over analytic methods, the actual effect sizes reported are very small, explaining well under a tenth of one percent the variance in test scores.” In his conclusion, Maul says “The findings of this report cannot be regarded as compelling evidence of the greater effectiveness of charter school compared to traditional public schools, either overall or specifically within urban districts.”


Report Reviewed: Measuring and Understanding Education Advocacy
Publisher/Think Tank: Brown Center on Education Policy at Brookings
Author: Grover J. “Russ” Whitehurst, David Stuit, Claire Graves, and Lauren Shaw
The report looked at the causal influence of organizations for and against education reform in Louisiana, Tennessee, and North Carolina. According to the report, findings indicated that coordination of advocacy groups strengthens their impact on the introduction of policy into the legislative arena, content of legislation, and the votes of members of the legislature. The report also found that the groups’ perceived influence closely tracked outcomes. The report utilized two methodological innovations to measure the impact of advocacy groups on education reform policy: (1) Surveys with Placebo (SwP); and (2) Critical Path Analysis (CPA).
 
Think Twice Review Date: April 16, 2015
Reviewer: Robin Rogers, Queens College, CUNY
Sara Goldrick-Rab, University of Wisconsin-Madison
An academic review of the report finds there is a gap between the evidence and the conclusions presented, however, the methods used in the report may be useful in education policy research. Regarding the overall merits of the report, Rogers and Goldrick-Rab state, “this report is useful primarily for understanding the perceptions of education advocacy groups’ influence and tactics in the three cases studied.” In their review, Rogers and Goldrick-Rab, find that SwP and CPA may be useful in education policy research, but the methods have limitations that are not acknowledged in the report. Additionally, the reviewers suggest the report is further limited because the research is a small case study, included a low response rate, and were based on advocacy groups’ self-reported tactics.


Report Reviewed: Paying the Best Teachers More to Teach More Students
Publisher/Think Tank: Edunomics Lab at Georgetown University
Authors: Marguerite Roza and Amanda Warco
A recent report from the Edunomics Lab at Georgetown University proposed that school districts pay top performing teachers a bonus for accepting additional students into their existing classes. The report claimed that larger classes and reductions to the teaching force would create significant savings. According to the report, districts should pay teachers in the top quartile a bonus for increasing their class size by up to three students. The report considers that students working with more effective teachers would offset any potential sacrifice in student learning.
 
Think Twice Review Date: April 13, 2015
Reviewer: Patricia H. Hinchey, Penn State University
An academic review of the report finds that the report is largely unsubstantiated, ignores what is known about teacher pay, and fails to offer guidance for policymaking. In her review, Penn State’s Patricia Hinchey finds that the report: (1) ignores the technical problem of how the best teachers might be reliably identified; (2) neglects a strong research base that has established a link between class size and student learning; and (3) misrepresents what is known about teacher pay, teacher attitudes, and teacher job satisfaction.

 

Report Reviewed: Separating Fact and Fiction: What You Need to Know about Charter Schools
Publisher/Think Tank: National Alliance for Public Charter Schools (NAPCS)
The report addressed 21 "myths" regarding charter schools, which were rejected. Succinctly, the original report addressed various claims about charter schools in such areas as financial equality of charter schools, lower teacher qualifications, student selection demographics, academic outcomes, segregation, and innovation.
 
Think Twice Review Date: February 23, 2015
Reviewer: Gary Miron, Western Michigan University
Kevin G. Welner, University of Colorado Boulder
and William J. Mathis, University of Colorado Boulder
The reviewers found that the report’s main purpose appears to be the "repetition or 'spinning' of claims voiced by advocacy groups and think tanks that promote privatization and school choice." Furthermore, the reviewers found that it relied almost exclusively on advocacy documents rather than more careful and balanced empirical research, and provides only a superficial examination of any "criticisms" regarding charter schools. The review is organized in a format that lists each of the criticisms identified, and then provides a short commentary based on the extant research literature. Where the original document overlooked research evidence, the reviewers provide readers with a valuable tool to examine charter school criticisms.


Report Reviewed: The Texas Economy and School Choice
Publisher/Think Tank: Texas Public Policy Foundation & Texas Association of Business
Authors: Arthur Laffer, Laffer Associates Investment Research
The Texas Association of Business and the Texas Public Policy Foundation commissioned Laffer Associates to perform an analysis of the Taxpayer Savings Grant Program (TGSP), which is a universal voucher program for Texas. The report theorized that by raising graduation rates, improving education achievement, and thus increasing human capital, the TSGP would create economic growth in Texas.
 
Think Twice Review Date: February 17, 2015
Reviewer: Chris Lubienski & Ee-Seul Yoon, University of Illinois
Chris Lubienski and Ee-Seul Yoon, in their review, highlight two key problems with the report: (1) Arthur Laffer’s assertions about the educational benefits of choice represent a severe overreach with and misapplication of the available research; and (2) the economic estimations are overgeneralized and heavily biased towards those families who already have the wealth to choose and relocate. The reviewers also note that the TSGP could result in further inequities for Texas schoolchildren, as higher income families would be able to supplement their children’s education even further, while devoting fewer resources to low-income families. In their conclusion, Lubienski and Yoon state: “While this report is clearly written to recommend the TSGP to the State of Texas, the lack of comprehensiveness and transparency, as well as the problems in its methodology, literature review and analysis, make it unsuitable as a basis for public policy decisions.”

 

Report Reviewed: Turning Lightning into Electricity: Organizing Parents for Education Reform
Publisher/Think Tank: American Enterprise Institute (AEI)
Author: Andrew P. Kelly
Turning Lightning into Electricity, written by Andrew P. Kelly, presents an inside look at several Education Reform Advocacy Organizations (ERAO) to examine the issue of parent participation and community organizing. The report offers strategy for advocacy groups who support the ERAO agenda to compete at the grassroots level.
 
Think Twice Review Date: February 2, 2015
Reviewer: Mark R. Warren, University of Massachusetts Boston
A review of the report released today finds limitations to the research methods and that the report cannot be considered a reliable study of parent organizing in ERAOs. Mark Warren, in his review, finds that the research methods employed are not adequately explained. Moreover, the research presented does not appear to be systematic or representative of ERAOs. He says: “the sample of ERAO groups is biased toward groups that more highly value long-lasting forms of parent engagement.” According to Warren, the report reduces organizing to a set of tactics to engage parents around an issue or agenda, rather than a democratic practice. “This approach undermines an understanding of community organizing as a profoundly democratic practice rooted in deeply held values of equity and social justice.”


Report Reviewed: No Excuses Charter Schools: A Meta-Analysis of the Experimental Evidence
Publisher/Think Tank: Department of Education Reform (DER) at the University of Arkansas
Authors: Albert Cheng, Collin Hitt, Brian Kasida, & Jonathan N. Mill
The authors of this working paper declared that students attending No Excuses charter schools had significantly improved math and reading scores. No Excuses schools feature high academic standards, strict disciplinary codes, extended instructional time, and targeted supports for low-performing students. The hope is that by having high standards for academics and discipline, No Excuses schools will help close the achievement gap between minority students and their White peers. The authors used findings from 10 studies that utilized experimental methods to estimate achievement outcomes.
 
Think Twice Review Date: January 20, 2015
Reviewer: Jeanne M. Powers, Arizona State University
Jeanne Powers, Arizona State University, found that the working paper is of limited value to policymakers or anyone seeking to understand the advantages or disadvantages of the No Excuses model. Powers, in her review, finds several flaws in the paper: (1) students who apply to charter school lotteries are not representative of all charter school students; (2) the authors did not address student attrition from charter schools; and (3) the results are based on a small sample of schools concentrated in East Coast cities. She says, “As a result, the author’s claim that No Excuses schools can close the achievement gap substantially overstates their findings.” Furthermore, Powers cautions “the current research base is too limited to make conclusions about the effectiveness of No Excuses charter schools.”

 

Report Reviewed: Proposed 2015 Federal Teacher Preparation Regulations
The U.S. Department of Education has released proposed Teacher Preparation Regulations under Title II of the Higher Education Act with a call for public comments through the Federal Register. The comment period closes February 2. The Department of Education claims that the proposed regulations will help ensure teacher-training programs are better preparing educators to succeed in the classroom by requiring states to measure outcomes of how graduates are doing in the classroom. The proposal would require states to assess and rate every teacher preparation program every year with four Performance Assessment Levels (exceptional, effective, at-risk, and low-performing), and states would be required to provide technical assistance to “low-performing” programs. Additionally, programs that do not show improvement could lose state approval, state funding, and federal student financial aid.
 
Think Twice Review Date: January 12, 2015
Reviewer: Kevin K. Kumashiro, University of San Francisco
An academic review of the proposed regulations considers the evidentiary support and identifies concerns. In his review Kevin Kumashiro has identified seven concerns with the proposed regulations. The proposed regulations: (1) will likely burden institutions with costs that are higher than estimated; (2) inaccurately conceptualize the impact and preparedness of teachers apart from systems; (3) mandate an improperly narrow definition of teacher classroom readiness; (4) require a reliance on scientifically discredited test-based accountability and value-added measures for data analysis; (5) disincentivize teachers to work in high-needs schools; (6) could restrict federal funding for students in financial need, and restrict access to the teaching profession for underrepresented groups; and (7) create a narrow view of the purposes of education.


Report Reviewed: Better Data, Better Decisions: Informing School Choosers to Improve Education Markets
Publisher/Think Tank: American Enterprise Institute (AEI)
Authors: John Valant
This report, published by AEI, asserts that choice is seen as a mechanism, and at times a panacea, for better educational quality. The report provides an overview of what families want from schools, where they get information on schools, and how they use information to make decisions. The report concludes that informed choice and the power of the market will produce better educational outcomes.
 
Report Reviewed: A Crisis We Can Solve: Connecticut’s Failing Schools and Their Impact
Publisher/Think Tank: Connecticut Coalition for Achievement Now (ConnCAN)
Authors: No author named
The ConnCAN report makes a case that there is a desperate need to improve school quality in Connecticut’s neediest neighborhoods and promotes charter schools as the best and perhaps only method for increasing the number of high-quality educational seats in Connecticut.
 
Think Twice Review Date: January 5, 2015
Reviewer: Erin McNamara Horvat, Temple University; and David Everington Baugh, Bensalem Township School District, Pennsylvania
A review of the reports finds that the ConnCAN report is more opinion than fact, while the AEI report provides useful information to help parents make informed decisions. However, the reviewers question whether simply having more information to make better decisions is sufficient to improve our educational system. Reviewers Horvat and Baugh

Previous Reports & Reviews:

These articles and/or reports are copyrighted material, the use of which has not been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of educational issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the U.S. Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit to those who have expressed a prior interest in receiving the included information for research and educational purposes. For more information go to: http://www.law.cornell.edu/uscode/17/107.shtml. If you wish to use copyrighted material from this site for purposes of your own that go beyond fair use, you must obtain permission from the copyright owner.