Multiple-choice questions have become one of the mainstays of assessment at universities across Australia and the world. But is this the best approach? Or even valid?
Central Queensland University has weighed up the advantages and limitations of multiple choice and has decided to abolish them from all exams. This is for two reasons: firstly, because of the potential impact of guessing; and secondly, because of the lack of authenticity in the method of answering the question.
Do they know the answers, or are they guessing?
The multiple-choice test format has its origins in the work of educational psychologist Frederick Kelly and was first used on a wide scale in US Army intelligence testing during the first world war. Since that time, multiple-choice questions have been adopted across many areas of education and testing.
In universities, they are often employed with large enrolment classes. Their principal benefit is their efficiency. Once set, they can be machine-marked, giving a fast turnaround time for results.
This is a major boon for hard-pressed teaching staff, who have tight deadlines. However, it is important to consider the limitations of the multiple-choice format.
The first limitation relates to their mathematical approach: a correct choice for each question is typically given a single mark, with either a zero mark or a fractional negative mark for the other options. However, that correct choice may be the result of either a) knowing or b) guessing, and there is no easy way to separate the two.
Exams in most subjects at university have a pass mark of 50%. To achieve such a mark in a multiple-choice exam with 100 questions and four options (one correct answer and three “distractors”) can be achieved by knowing 33 answers and guessing the remaining 67 questions (on average, scoring a “correct guess” in a quarter of these, 17).
And that’s only the “on average” position. Once you factor in the randomness of guessing and the influence of binomial probabilities, we can only be 95% confident that the student knows, and hasn’t guessed, around one-quarter of their answers.
This gives rise to something best termed “spurious accuracy”, where a student who has scored 49% will fail and another on 50% will pass, but that may simply mean that the second student made a lucky guess on one more question than the first. Is that really the way we want to decide whether students have passed or failed a subject at university, when there is such uncertainly about the validity of the mark?
You can’t choose options A to D in real life
Some of the mathematical and statistical issues with multiple-choice questions can be reduced by raising the minimum mark required for a “pass”. For example, with a requirement of 80% to pass and with appropriate negative marking, the issue of random guessing can be reduced considerably. However, a more fundamental obstacle to their use cannot be so readily overcome; the format of answering multiple-choice questions is not one that many people will meet in the real world, except perhaps if they’re a quiz show contestant on TV, or if they’re completing a marketing survey.
Many university programs pride themselves on their vocational nature, educating students to enter professions such as nursing, engineering, business, medicine and law. Once qualified, graduates will not make decisions based on selecting the right answer from a very short list of options. They will have to form their answers in their minds rather than finding them in such a list. In short, while multiple-choice exam questions can be authentic (for example, based on real-world contexts or case studies), the method of answering cannot.
There’s also a potential ethical problem with multiple-choice tests, where distractors that are partly correct are used. Is it right that teaching staff try to “lead students astray” by setting such distractors? In other types of written exams, students will often gain part marks for knowing some, but not all, of the answer.
And then there’s the complexity of the questions, which is made worse in negative questions, such as “which one of the following is not…”
CQ University’s academic board recently weighed up the benefits and limitations and decided that, rather than trying to “fix the maths”, multiple-choice questions would no longer be used in any examinations. From next month onwards, teaching staff will replace multiple-choice questions with “short answer” formats, where students are asked questions that require them to write a single word or a short phrase as their answer, which staff will then mark.
This will still be efficient in terms of marking, but will remove the issues of picking from a short list. It will provide greater validity and reliability to marks and will also better reflect what happens in the real world once students graduate.