The easiest test date: is it a myth or reality?
We know some tests are harder than others. Try as they might, the test makers cannot produce two identically challenging exams. Because of this, and because the pool of test takers changes with every exam, some tests have more perfect scorers than others. And on some tests, you can miss two questions and still score perfectly, while on others, missing two questions drops you far down in the point scale.
But what makes for a hard test? Is it just the luck of the draw, or is it a function of whom you take the tests with? Is it the behind-the-scenes magic of scaling, or is it the time-of-year? The old logic goes like this: testing with more seniors must make for a more competitive base, and thus students will face more difficulty achieving a top score. So the trick must be to test with the most inexperienced group, and that must happen during spring, after the seniors are done testing. Right?
The short answer is no. According to our research and a meta-study of analyses, there is no ‘easy’ test date. Though it sounds simplistic, the easiest time to test is when you feel most prepared.
The long answer is a little more complicated. These tests are scaled, and the methods used are opaque. This has led to speculation over ways to game the system by choosing the optimal factors like time and cohort. While those factors do not appear to impact test difficulty, there are variations that are worth noting. Understanding those requires knowledge of exactly what happens in the process of translating a a raw score to a scaled score.
How the Scale was Made
When most parents and students think of a grading scale, they think of a curve: a student’s raw score is mapped to their final score based on their performance compared to the other members of their class.
This is NOT how score scaling works on the ACT or SAT. Students are not being compared to the other students who took the test during their same test date or even their same testing year. If the test was scaled using the usual curve, then the proportion of students who get a great, average, or lackluster score would be the same for every testing day. But we do not see this, instead there are some testing days when more students than expected get a perfect score and others when fewer do.
The ACT compares students to data based on the Academic Skills Study conducted in 1988. This data was weighted and distributed into two categories, all examinees and college-bound examinees, and the outliers were removed. The college-bound data was then used to create the initial score scale. Minor adjustments were then made to this initial scale to ensure that each score fell within a reasonable margin of error, that there were no gaps in the score scale, and that there were not too many raw scores which converted to a single scaled score.
The ACT conducted another study in 1995 to ensure that the introduction of calculators (which had previously been prohibited) to the test did not change the meaning of the 1988 scale. It was found that there was not a significant difference in meaning, and the scale based on the 1988 data remains in use to this day.
The SAT’s score scaling data is based on the 2014 Scaling Study conducted by the College Board. Because the SAT’s process for screening and distributing their data was fairly similar to the one described for the ACT, and for the sake of brevity, I will not go into further detail.
However, for those who are interested in a more in depth look at the scaling process, I recommend reading for themselves Chapter 9 of the ACT Technical Manual or Chapter 6 of the SAT Suite of Assessments Technical Manual, both of which can be found online for free.
All of this is to say that whom a student takes their test with will have absolutely no bearing on their final score. Students are being compared to college-bound students from 2014 or 1988, not their peers. Therefore, there is absolutely no advantage to avoiding test dates which have a high proportion of Seniors (who usually test in September and October). Seniors will not “break the curve”: their high scores will have no impact on the scores of other students.
But Some Tests are Still “Harder” than Others
Even with the misconception that other students can “break the curve” out of the way, many students will still say that certain test dates felt harder than others. This makes sense: the logistics of creating and administering the SAT and ACT make it almost impossible for each test to be the exact same level of difficulty as all the others.
The makers of the SAT and ACT know this, but before getting into what steps the test makers have taken to reduce the effect of this reality on final scores, let’s define what it means for a test to be “hard.”
What students tell their parents that a test was hard, what they really mean is that the test “felt hard”. Unfortunately, this is not a very useful way to think about difficulty. Not only is everyone’s personal metric for what feels difficult different, but something “feeling easy” is too far vague to create a codified definition for what makes a test “easy” or hard”.
A more useful way to think about difficulty is in terms of “how many questions can I miss and still achieve my target score”. If a student can miss more questions than they usually get wrong in practice and achieve their target score, then that is what makes the test easy. Similarly, if a student needs to miss fewer questions than they usually get wrong to reach their target, then that makes the test hard.
Now, recall that test makers have a system for making sure small difference in difficulty between tests do not affect final scaled scores. This system is called Equating, and, ironically, because of it, tests which feel hard for students are actually easy when using the above definition for difficulty, and similarly, tests which feel easy are actually quite difficult.
How the Scale is Made: Equating
Equating is the process that each new official test form must through to make sure that its final converted score has the same meaning as all the other test forms which came before it.
New test forms are equated using a carefully selected sample of national examinees. They are given a set of tests which contain the new test forms as well as one anchor form which has already been equated to previous forms. The new forms are then compared to the anchor form using equipercentile equating, outliers are removed, and the result becomes the raw-to-scaled conversion chart for that specific test form.
The charts which come out of this system are designed so that when a test is easier, more questions need to be answered correctly, and when a test is more difficult, fewer questions need to be answered correctly to achieve the same reported score. In this way, a certain score demonstrates the same amount of skill on the part of the student, regardless of what version of the test they are taking.
For an example of this in practice, take a look at the two raw-to-scaled conversion charts below:
Test A: 2017-2018 ACT Practice Test Conversion Chart
Test B: 2008-2009 ACT Practice Test Conversion Chart
Focusing on the English Section, we see that in order to receive a minimum score of 35, a student can miss 3 questions on Test B but may only miss 2 questions on Test A. This means that if a particular student took both versions of the test and they missed 2 questions on Test A, that student would be expected to miss 3 questions on Test B.
The scale of Test B is easier, because the test itself was harder. Plus, thinking back to our new definition of what “easy” and “difficult” mean, because a student can miss more questions on the English Section of Test B and still receive a certain score, that makes that section of Test B easier, even though the actual material written in that section is more difficult.
So when is the easiest test date then?
The easiest test date is when a student is most prepared.
If a student is performing consistently and can avoid making simple errors such as misreading questions or typing things into their calculator incorrectly, then the exact scale of the test they are taking will not affect their final score. Furthermore, there is no way to predict what the scale of a test date will be before getting the results back. To do so would require knowing what specific questions would be asked on the test, which is problematic for obvious reasons. Therefore, students should focus on taking the test when the timing works for them and when they have reached a point in their preparation where they can achieve their target score.
For a full analysis on the optimal timeframe, and for help on deciding when to take the SAT and ACT, please review our articles:
The best way to improve a student’s score is not for them to worry about which test date will be “easier”. Rather, the best use of a student’s time is to figure out which test is best for them and create test prep schedule far enough in advance that they can improve their score.
For additional information, here are out sources:
By Lauren Thompson & Matt Larriva