I’ve had a healthy number of responses to my question from the previous post. In case you are reading this post without having read the previous one, I shall continue after the fold, because if you read on it will render you ineligible to participate in the little experiment I am conducting.
Every year in Britain, at round about this time of the year, we have the same debate. The GCSE and A-level results come out (these are taken at the ages of 16 and 18, respectively) and they show a modest improvement on the results from the previous year. Some people can be relied upon to seize on this as evidence that exams are getting easier. Others can be relied upon just as strongly to leap to the defence of the latest generation of schoolchildren, praising their hard work, and that of their teachers, and condemning those who, they claim, are trying to belittle it.
So are the exams getting easier? I certainly think that today’s A-level papers in mathematics are much easier than the ones I took, and am planning to write a post in which I shall attempt a detailed demonstration of this, though actually finding the old question papers (which I think I have somewhere) may be more of a challenge than comparing them with today’s papers.
But I thought I’d prepare the ground by taking the questions from two A-level papers and seeing whether people could tell which paper was easier if they did not know anything more than what the questions were. I then had to decide which two papers to choose. What I’d have liked to do is choose two papers from about 15 years apart, but then I had the problem that from time to time the syllabus and exam format gets changed. I wanted to avoid any possibility of somebody’s arguing that an apparently easier syllabus was in fact just as hard because it “developed a different set of skills”. So I needed two papers that were more directly comparable, and I also wanted one of them to be very recent. Unfortunately, this forced me to take the other one to be fairly recent too.
The two I ended up choosing were Further Pure 1 from January 2005 and from June 2010. The board was MEI, which stands for Mathematics in Education and Industry. (Links to these and all the intermediate papers can be found on this page on MEI’s website.) I would have liked to have separate polls for every question, since there was almost a natural bijection between the two papers, but the marks on offer weren’t quite the same so in the end it seemed safer to split the papers up by section and no further.
Here, a few days later, are the results of the polls as they currently stand.
205 people have voted, and their votes are as follows.
A1 is probably easier than A2 … 49 votes
The two sections seem to be of equal difficulty … 49 votes
A2 is probably easier than A1 … 43 votes
I am certain that A2 is easier than A1 … 35 votes
I am certain that A1 is easier than A2 … 29 votes
A1 was Section A from January 2005, and A2 was section A from June 2010.
130 people have voted, and their votes are as follows.
The two sections seem to be of equal difficulty … 33 votes
B1 is probably easier than B2 … 32 votes
I am certain that B1 is easier than B2 … 29 votes
B2 is probably easier than B1 … 26 votes
I am certain that B2 is easier than B1 … 10 votes
B1 was Section B from June 2010 and B2 was Section B from January 2005.
Obviously, these results should be treated with extreme caution, so I’ll just make a few remarks (as uncontentious as possible) and see whether anyone else has anything to say.
1. One would not expect two papers that are just five years apart to be radically different in their level of difficulty.
2. Even if one of these two papers is found to be clearly easier than the other, these are just two papers from one examination board in one subject, so the result could be put down to random variation. (I should say that I chose those two papers because they were at the two ends of the range that was available to me, and not because I made some prior judgment about whether one of them looked easier than the other.)
3. One might argue that even if the two papers were of identical difficulty, they are so similar, as are the ones in between, that candidates taking the exam in 2010 were at a significant advantage over candidates taking it when the format and syllabus had just changed. (I haven’t checked what the papers just before the change were like, so this point isn’t necessarily a valid one.)
4. It is amusing that several people can be certain of a proposition, and several others can be certain of the opposite of that proposition. This phenomenon does have possible explanations: for example, maybe one person finds one kind of question difficult and another finds another kind difficult, so their perceptions of difficulty are genuinely different.
5. The number of people who found 2005A easier than 2010A is exactly equal to the number of people who found 2010A easier than 2005A. However, those who found 2010A easier were a little bit more sure of their judgments.
6. The number of people who found 2010B easier than 2005B was 61, compared with 36 who found 2005B easier than 2010B. Again, those who found 2010B easier were more confident of their judgments: 29 people were certain that 2010B was easier, compared with 10 people who were certain that 2005B was easier.
Based on those results, I find it tempting to conclude that the 2010 paper was genuinely a little bit easier than the 2005 paper. However, I do not claim that this experiment establishes that beyond all reasonable doubt. I also have arguments that are based on the questions themselves, but I would prefer to wait to hear what others think before giving my own judgments. It would also be interesting to conduct further experiments along these lines. I suppose I could just repeat this one: people would know what I was doing, but perhaps that wouldn’t matter too much. If you have suggestions, I’m ready to listen to them.