We took the public SHSAT practice tests 53 times, deliberately changing student performance. The questions never changed.
Beginning this fall, the SHSAT will be computer adaptive. NYCPS says students will begin with a question or passage set of average difficulty. As they answer questions, the system will continually re-estimate their performance and adjust the difficulty of the questions they receive. A correct answer may lead to a more challenging question. An incorrect answer may lead to one with a lower difficulty level.
That’s how the real SHSAT is supposed to work this fall.
The four online practice tests don’t work that way.
We tried to make the tests change
We wanted to know whether different patterns of student performance would actually cause the public practice tests to change.
We first tested them manually across 12 runs, including the two extremes: every question right and every question wrong. The questions presented did not change.
We then stress-tested that finding with 41 additional controlled runs using an automated test runner we built. We used predetermined response patterns designed to produce substantially different patterns of performance: five right followed by five wrong, five wrong followed by five right, alternating blocks of three, seven and twelve, and other combinations.
Across all 53 runs, changing performance never changed the path through a practice test. Each of the four tests has its own questions, but within each test, the same questions appeared in the same sequence regardless of how we answered them.
We couldn’t produce the adaptive behavior NYCPS describes for the actual SHSAT.
What the practice tests do provide
The practice tests still provide things students need.
Students can practice the new technology-enhanced items (TEIs), learn the TestNav interface, use the digital tools and experience much of the structure of the new test. NYCPS describes the four tests as “fully functional online practice tests” that “mirror the actual exam in both form and content.”
The missing piece is adaptivity.
Because the practice tests follow the same path regardless of performance, educators can’t observe how different levels of performance affect the questions a student encounters. We can’t see how quickly difficulty changes, how much it changes or what a substantially different performance path looks like from the student’s side of the screen.
NYCPS and Pearson don’t claim that these practice tests are adaptive. Educators need to understand that limitation when deciding what the tests can tell them about student readiness.
The practice tests look like the SHSAT. They don’t behave like it.
When the SHSAT was a paper test, a practice test could reproduce the basic testing process reasonably well. Open a booklet. Work through a fixed set of questions. Manage your time. Answer what you can.
An adaptive test introduces something students haven’t had to manage before: the test itself responds to their performance. Questions may begin to feel harder or easier. Students have to manage their pacing and their reaction to apparent changes in difficulty without knowing whether a harder-feeling question means they’re struggling or the test has adjusted upward.
That is something students can practice on other question-adaptive exams. The GMAT, for example, provides adaptive official practice exams that adjust question difficulty based on performance and use the same scoring algorithm as the actual exam.
The public SHSAT practice tests can't reproduce that experience. Students can practice the format, but they can't practice responding to a test whose difficulty and path are changing as they work.
What educators can do this fall
Use the NYCPS practice tests for what they provide best: familiarity with the official testing environment, question types, tools and navigation. The correct/incorrect summary and explanations can also help identify mistakes and areas that need more work.
Other practice tools can fill some of the gaps. Bell Curves’ digital practice tests use the new SHSAT format and add a score report and incorrect-answer analysis so educators can better identify where students are struggling and what to work on next.
However, until an adaptive practice test becomes available, none of these tools, including ours, can reproduce the central feature of the new SHSAT: the test changing in response to the student’s performance.
The online practice tests can show students what the new SHSAT looks like. They cannot yet show students what taking an adaptive SHSAT is actually like.