New research challenges
admissions practices at New York
City’s selective public high schools
October
21, 2008
Contact:
Joshua Feinman - 212-454-7964 (w); 917-596-5946 (c); josh.feinman@db.com
Kevin Welner - (303) 492-8370; kevin.welner@gmail.com
TEMPE, Ariz and BOULDER, Colo. (October
22, 2008) -- This weekend, about 25,000 students in New York City will take
an examination in hopes of gaining admission into one of eight specialized
public high schools. If this year follows past trends, fewer than 20% of all
test-takers will be offered seats at these highly-selective schools. The
system appears straightforward: higher scorers on the test gain admission,
while lower scorers are rejected. But a new study of this system raises questions
about its fairness.
In the policy brief High
Stakes but Low Validity? A Case Study of Standardized Tests and Admissions
into New York City Specialized High Schools, economist
Joshua Feinman, Ph.D., finds evidence that the admissions process may not
produce valid or equitable results. The report is published jointly by the
Education Policy Research Unit at Arizona
State University
and the Education and the Public Interest Center
at the University of Colorado at Boulder.
Feinman and his wife are each graduates of
one of these specialized high schools, and they currently have a daughter
enrolled in one such school. Yet, even though the members of his own family
gained admission into the most competitive of the exam schools, the more
Feinman looked into the technical details of the admissions system, the more
troubled he became.
New
York
state law requires that an exam be the only criterion for admission to the
specialized high schools. The exam used is called the Specialized High
Schools Admissions Test (SHSAT). The next administration of the exam, to
determine admission for September 2009, is this weekend (October 25 &
26), and is provoking the usual frantic nail-biting among thousands of
families throughout New York.
Feinman studied test results from 2005 and
2006 and identified a number of areas of concern. For example, he concluded
that thousands of students may be rejected even though their scores are
"...statistically indistinguishable from thousands who were granted
admission."
Feinman also found that different versions
of the test were used and that test performance -- and therefore the
likelihood of a student gaining admission to a specialized school -- varied
according to which test version a student is fortunate (or unfortunate)
enough to have taken.
Equally troubling is the way the test
scores are calculated. Feinman notes that the scores on the two sections of
the test -- math and verbal -- are combined in a way that is unusual for
standardized tests, resulting in an advantage for a particular subset of
students. Those with a very high score in one section and a much lower score
in the other have a better chance of admission than students with relatively
strong performance in both sections. The rationale for such a system was
never made available, and no evidence was offered to support it.
As Feinman points out, other potential
systems could be just as sensible and valid, or even more so. The chosen
system penalizes students who are equally strong on the math and verbal
sections, and this is something most test-takers probably never realize.
Moreover, the scoring gives an advantage to students whose families can
afford costly test-prep tutoring and can learn how to game the system.
These findings raise important questions
of fairness, Feinman says, especially because the test is the sole arbiter of
admission. He acknowledges that no single admissions criterion would be
perfect. But numerous past studies, he says, "have found that multiple
imperfect criteria, used in tandem, provide better insight into future
student performance than a single imperfect criterion." Largely for that
reason, "virtually all educational institutions use multiple admissions
criteria." He explains the issue as follows:
"All told, on a different day, many
students might have flipped to the other side of the admission/rejection line
by pure chance -- if they'd been assigned a different test version, if the
winds of random variation in test scores had blown a bit differently, if
slightly different but equally logical scoring had been used, or if they'd
been told how the actual scoring system works. Sometimes, all of the test's
idiosyncrasies combine to help one student and harm another... The decisions
here seem arbitrary, especially since there is no validity evidence to
support them."
The last point is what Feinman considers
the most serious problem with the current system: the lack of predictive
validity studies to see how well the test scores predict whatever subsequent
performance is considered important (such as success in the schools or in
college). Because New York
has never conducted such a study, nobody knows how sensible or fair the
current system truly is. Similarly, no studies have examined whether the
SHSAT is skewed for or against specific ethnic or gender groups.
"The whole process flies in the face
of accepted psychometric standards and practice, and reminds us why those
standards and practices were established and should be maintained,"
Feinman writes. "The thousands of students who apply to these select
high schools deserve a properly tested system of determining who gets access
to these prestigious and potentially life-changing educational experiences."
While the Feinman study examines only one
highly selective system of public schools, it has broader implications for
the nation's public schools, whose fates increasingly turn on their students'
performance in high stakes exams that may govern grade advancement,
graduation, and such factors as school funding and the autonomy of school
administrators.
CONTACT:
Joshua Feinman
212-454-7964 (w)
917-596-5946 (c)
josh.feinman@db.com
Kevin Welner, Professor and Director
Education and the Public
Interest Center
University of Colorado
at Boulder
(303) 492-8370
kevin.welner@gmail.com
**********
###
The Education and the Public Interest
Center (EPIC) at the University of Colorado at Boulder
partners with the Education Policy Research Unit (EPRU) at Arizona State
University to produce
policy briefs and think tank reviews. These centers provide a variety of
audiences, both academic and public, with information, analysis, and insight
to further democratic deliberation regarding educational policies.
Visit their website at http://educationanalysis.org
###
**********
|
