In academia – as in many other contexts – Multiple Choice Questions (MCQs) are a fact of life. We have many situations in which we need to assess a large cohort of learners, and MCQs make this easier.
Here, I don’t want to debate the value of MCQs as an examination technique in general, but talk about how they are written. If we accept that they are part of our work, let’s at least make them not terrible.
There are three very common ways in which MCQs can be obviously bad. I see them all the time.
They are:
- The question only examines surface knowledge
- The distractors are random
- The off-by-one way of writing distractors
Let’s look at each briefly.
The question only examines surface knowledge
This is quite an easy and obvious shortcoming to recognise: Often, MCQs just ask questions about terminology. They do not touch on concepts or deeper understanding. This is a very common trap to fall into for people who are not used to writing MCQs. Writing MCQs that test understanding is harder, and it’s easy to slip into the trap of just testing surface issues.
We have probably all seen it.
When you write your questions, examine each question critically in this light. Strive for meaningful questions.
The distractors are random
Another very common gripe of mine is that the distractors (the incorrect options in MCQs) are chosen at random. Let’s consider an example. Assume we have this question:
Question 54:
What is the output of the following segment of code?
int sum = 0;
for (int i=0; i<=5; i+=2) {
sum = sum + i;
}
System.out.println(sum);
If you speak Java, then you can work out that the answer is 6. So we use 6 as one of the possible options, and add a few random distractors:
Answer:
(A) 1
(B) 896
(C) 6
(D) -4
(E) 5
In this case, if students have a common misconception, they may arrive at an answer that is not on the list. For example, a student who does not understand that this definition makes the loop increment by 2 might assume that the answer is 15. But 15 is not on the list. So the student is implicitly told that they are wrong, and they can discard unlikely options (“Negative? – No.” “More then 800? – No.”) and make a random guess between a few remaining options.
This question is even worse in a way: Another common mistake is not being able to correctly distinguish between <= and < in the loop condition, and getting the answer wrong because of this. The way the question is written, however, it is not even possible to expose this misconception, because both variants happen to result in the same output!
A badly written question.
The off-by-one way of writing distractors
The other very common pattern I see all the time has an example in this question.
Question 31:
Which of the following code segment would execute the stored procedure “getPassword()” located in a database server?
-
CallableStatement cs = con.prepareCall("{call.getPassword()}");
cs.executeQuery();
-
CallabledStatement callable = con.prepareCall("{call getPassword()}");
callable.executeUpdate();
-
CallableStatement cab = con.prepareCall("{call getPassword()}");
cab.executeQuery();
-
Callablestatement cstate = con.prepareCall("{call getpassword()}");
cstate.executeQuery();
The way questions like this are written is usually this: The question setter writes the correct response, and then creates three distractors by introducing one error in each.
It is trivial to reverse-engineer the correct answer from this. To do this, we spot the differences, and for each differences discard the odd one out. For example:
- One difference is that “{call getpassword()}” is written with a dot in the middle once, but without the dot three times. So the dot is the odd one out, we discard Answer 1.
- Another difference: CallableStatement is written with w capital S three times, with lowercase s once. So we discard answer 4.
- Last one: The second line calls executeQuery() three times, and executeUpdate() once. Answer 2 is out.
And voilà – there is our answer: 3. We can do this without any knowledge or understanding of the subject matter. Keep your eyes open for this pattern in MCQ tests. It is astonishing how often you will see this question pattern.
A better example
Here is an example of something you could do:
Question 8:
HashMap<Integer, String> myMap = new HashMap<>();
myMap.put(2, "Tiger");
myMap.put(4, "Lion");
myMap.put(13, "Lion");
myMap.put(2, "Elephant");
myMap.put(3, "Chicken");
System.out.println(myMap.get(2));
When you try to compile and run this code, what will happen?
-
- The code has an error, because the value “Lion” is used twice in the HashMap.
- The code has an error, because the key 2 is used twice in the HashMap.
- The code will run and print “Tiger”.
- The code will run and print “Lion”.
- The code will run and print “Elephant”.
In this case, the question tests an understanding of how maps work.
- Option 1 tests the possible misconception that values cannot appear twice in a map.
- Keys, on the other hand, are unique in a map. Option 2 tests the misconception that they cannot be replaced by a new entry.
- Option 3 tests the misconception that the first entry for a given key persists.
- Option 4 tests the misconception that the get(n)-parameter is an index, mirroring list-behaviour (referring to either the second or third element entered, depending whether the list is zero-based or one-based).
- And Option 5 is correct.
This is by far not all that there is to say about MCQ writing. But it’s a start. It is the absolute minimum you should consider if ever you find yourself in the position to write and MCQ test.
Last week, we have officially released Strype 1.0, a new Python editor and runtime system running entirely in a web browser. After being in development and available for preview and testing for some time, the system is now fully supported for use in classrooms or for individuals wishing to learn Python programming.