After giving this some thought, engaging in discourse with colleagues and peers, I’ve decided that the theme College Board’s major marketing event last month should have been “Missing Opportunity” (click to read my post about that event). That event marked an important moment in which David Coleman and his College Board junta could have revolutionized the SAT by thoughtfully and comprehensively addressing the elements of the SAT that lead to what he called “the culture and practice of test preparation that now surrounds admission exams drives the perception of inequality and injustice in our country.” Instead we got a marketing event, great talking points, and lipstick put on a pig. From all indications the new SAT will be nothing new, nothing different, and imperceptibly less biased against those who don’t have access to SAT prep. There are great articles floating around (I’ll link to a few I like at the bottom of this post) about why the SAT will fail to deliver on its promise of a level playing field, so instead of piling on what’s wrong with the new test I’m going to show you how the College Board (filled with psychometricians and PhDs who are ostensibly far smarter than me and thus begging the question of why did they not address these things themselves) missed out on simple ways of creating an SAT that was less susceptible to test preparation.
Before going on, I have to point out that information about the specifics of the new SAT hasn’t been released, so this entire piece and all others criticizing or praising the new test are based on a high level description of the forthcoming test and are thus about as reliable as predicting the hunting habits of the T-Rex based on fossil evidence. But since, on the modern internet, prognostication and list-making are second only to making gifs and cat videos, I offer to you my contribution to this scholarly endeavor.
3 Ways the New SAT Could Have Minimized the Impact of Test Preparation
1. Make the entire math section Student-Produced Response Questions
The current SAT has 10 of 54 math questions which are not multiple choice. These questions allow the student to “grid in” answers. By making the entire test grid-ins, this would increase the number of possible answer choices per question to 14,256 from the current 5. This change would eradicate any benefit of random guessing, since the odds of a correct guess causing a statistically significant increase in scores would be reduced to almost zero from the current 20%. Not only would the addition of grid-in remove the guessing advantage, it would also remove the most popular test preparation tool: plugging in. Every test preparation company teaches some form of plugging in, a strategy that is only applicable and effective on a multiple choice algebra test. The removal of multiple choice options would strip the test prep industry of one of the best tools for changing scores.
2. Remove the time pressure
Several studies support the notion that performance on tests is negatively impacted by time constraints, especially for low income and minority groups (though this is not proven). By remaining a highly speeded test (one that creates pressure to finish in time), the SAT remains susceptible to test preparation significantly impacting performance. Many test preparation strategies are designed to allow the test-taker to complete the item (fancy testing talk for what us laypeople call questions) not just correctly, but in the most efficient manner, thus allowing that test-taker time to work on additional questions. Removal of the time pressure would negate or minimize this tool for the test preparation industry.
Removal of the time pressure would also align the SAT more with the manner in which tests are most commonly delivered in school and on statewide tests. Both teacher-created assessments and statewide tests typically provide generous amounts of time for completion, rather than allowing only for the exact amount of time it would take the top performing students to complete the test. The approach taken by the SAT further advantages top performers and those who receive test preparation.
If the College Board wanted to decrease test preparation’s impact, they would increase the amount of time to allow all test-takers the opportunity to finish the test, without the mad panicked bubbling that generally occurs in the last few minutes of a section.
3. Enforce their own rules of fair usage of scores
Another way the College Board could easily and
quickly change the impact of test preparation is to enforce the rules they have so long given lip service to. College Board consistently messages the psychometrically valid ideal that a single test score is more properly indicative of a performance range rather than an absolute number. When you consider the standard error of measure, and the standard error difference as well, it becomes clear that no SAT score should stand as a single marker of ability.
For years institutions have been “encouraged” to understand the statistical meanings of these terms and take them into account when making decisions. However, in truth many schools use SAT scores as hard cutoff for admissions and monetary awards.
If College Board wanted to positively impact opportunity, they would create and enforce more stringent rules against the misuse of test scores. They could also simply have changed the SAT scoring scale to convey fewer perceived levels of distinction. Many students who I’ve tutored over the years have paid good money in order to go from a 590 score to a 600 or a 690 to a 700 because of the perception conveyed by minuscule change.
“Ensure that small differences in test scores are not the basis for rejecting an otherwise qualified applicant.” – College Board guidelines for score usage.
Were the gang at College Board serious about reducing inequities engendered by the mystery surrounding the test and misuse of scores, they could have changed the 200-800 point per section scoring scale to only report scores in 50-point increments. Instead, they continue to report scores in 10 point increments and “warn” schools not to read much into small score differences. Reporting scores in fewer increments would alleviate a lot of the anxiety around minor score improvements for both students and admissions offices, but nope, that not a choice CB made in this latest iteration of the venerable college admission gate-keeping behemoth (sorry Mr. Coleman I know that’s probably over-doing it with the “SAT words”).
Alas, while I hold out hope that some of my changes might have been considered and there was a rationale for not implementing them, I have no real expectation that the College Board will do much besides topical changes to the content of the test. And the changes to the test will simply mean those families with the means to will spend it on test preparation so they can continue to solidify their advantage over those without the means to dedicate hundreds of hours to study or thousands of dollars to preparation. In the meantime Coleman will continue to tout Khan Academy as the great equalizer, though Khan will be the only source of test preparation that doesn’t ever mention timing strategies or plugging in.
Follow up with me in the week after April 16th
when samples of the new SAT will be released. I’ll have a chance to fully analyze the test and lay out the test preparation tools that we’ll use to help students “beat” the test.
In case you’re interested, here’s some research referenced above:
- The effects of time limits on the intelligence test performance of Mexican and American subjects
- Time Requirements for the Different Item Types Proposed for Use in the Revised SAT
- Effects of Applying Different Time Limits to a Proposed GRE Writing Test – Powers and Fowles, 1997
- Allocation of Time to Test Items: A Study in Ethnic Differences – Llabre, Froman
- Effect of Extra Time on the GRE Quantitative and Verbal Scores – Bridgemen, et al 2003
- Guidelines on the Uses of College Board Test Scores and Related Data
Here are some of the articles on the change to the SAT and their impact:
- Will the New SAT Boost College Prospects for Black Students? – Dr. Ivory Toldson, The Root
- Plans for New SAT Spark Mixed Reviews – The Chronicle of Higher Education
- Can the New SAT Solve the Test’s Inequality Problem? – The Atlantic
- The SAT-Prep Industry Isn’t Going Anywhere – James S. Murphy (my former colleague) in The Atlantic