Milwaukee Voucher Report Sound But Of Limited Value
May 28, 2009

Future reports examining voucher program should be improved and clarified

Contact: Teri Battaglieri (517) 203-2940; greatlakescenter@greatlakescenter.org
Casey Cobb (860) 486-0253;  casey.cobb@uconn.edu

EAST LANSING, MI (May 28, 2009)—A recent report on the second year of a five-year evaluation of the Milwaukee school voucher program finds no achievement growth differences between voucher schools and the Milwaukee Public Schools over a one-year period. A new review of the report finds its design and methods sound, but its usefulness is limited by several factors, including a lack of clarity about the representativeness of the voucher schools included in the study.

The MPCP Longitudinal Educational Growth Study Second Year Report was written for the University of Arkansas School Choice Demonstration Project as part of a five-year evaluation of the Milwaukee Parental Choice Program (MPCP).

The report was reviewed for the Think Twice think tank review project by Casey Cobb of the University of Connecticut, an expert in policy analysis and social science research methods.

The Second Year Report’s primary finding was that there were no overall statistically significant differences in achievement growth between a group of students in the voucher schools included in the study and a matched group of students in Milwaukee Public Schools (MPS).

The most substantial concern Cobb raises in his review involves the sample of schools used for this evaluation – and likely for future evaluations. An act of the Wisconsin legislature requires schools accepting vouchered students to administer a standardized test in reading, math and science in grades 4, 8, and 10. It is these test scores that are analyzed in the new report. Because the law doesn’t require any particular exam, different schools administer different tests. The sample of voucher schools included in the report reflects only those using one particular test, the Wisconsin Knowledge and Concepts Examination (WKCE). As a result, Cobb points out in his review, the results “are only generalizable to MPCP schools that administer the WKCE.” The key question, and one that a reader of the report cannot answer, is whether this sample differs in any meaningful way from those schools that use a different exam. For instance, might a given subgroup of schools, such as those run by the Milwaukee Catholic Archdiocese, be excluded from the study because they use a different test?
 
Cobb compliments the report’s authors for being “careful not to overstate the findings of a single-year study on achievement effects.” He also notes that the methodology used “adheres to rigorous principles of social science.” However, he concludes that this second-year report is unlikely to be very useful to policymakers.

Find Casey Cobb’s review on the web at:
http://www.greatlakescenter.org.

This is the third Think Twice review in recent weeks to examine the School Choice Demonstration Project’s Milwaukee reports. Previously, a review by Dr. Clive Belfield found that a report about the Milwaukee voucher program’s fiscal impact offers valuable insights but also questioned the report’s findings regarding fiscal benefits and noted that the report omits some important factors essential to a more complete analysis. And a review by Dr. Gregory Camilli of a report about the Milwaukee voucher program’s competitive effects raised a number of questions about the report’s statistical methods, ultimately concluding that any positive effect of competition is very small, if it exists at all.

Clive Belfield’s review can be found at:
http://greatlakescenter.org/docs/Think_Twice/TT_Belfield_MPCP.pdf

Gregory Camilli’s review can be found at:
http://greatlakescenter.org/docs/Think_Twice/TT_Camilli_MilwVoucher.pdf

About The Think Twice Project
The Think Twice project provides the public, policy makers and the press with timely, academically sound reviews of selected think tank publications. It is a collaboration of the Education Policy Studies Laboratory at Arizona State University and the Education and the Public Interest Center at the University of Colorado at Boulder and is funded by the Great Lakes Center for Education Research and Practice.

###