The terrible state of MCQs

In academia – as in many other contexts – Multiple Choice Questions (MCQs) are a fact of life. We have many situations in which we need to assess a large cohort of learners, and MCQs make this easier.

Here, I don’t want to debate the value of MCQs as an examination technique in general, but talk about how they are written. If we accept that they are part of our work, let’s at least make them not terrible.

There are three very common ways in which MCQs can be obviously bad. I see them all the time.

They are:

  1. The question only examines surface knowledge
  2. The distractors are random
  3. The off-by-one way of writing distractors

Let’s look at each briefly.

The question only examines surface knowledge

This is quite an easy and obvious shortcoming to recognise: Often, MCQs just ask questions about terminology. They do not touch on concepts or deeper understanding. This is a very common trap to fall into for people who are not used to writing MCQs. Writing MCQs that test understanding is harder, and it’s easy to slip into the trap of just testing surface issues.

We have probably all seen it.

When you write your questions, examine each question critically in this light. Strive for meaningful questions.

The distractors are random

Another very common gripe of mine is that the distractors (the incorrect options in MCQs) are chosen at random. Let’s consider an example. Assume we have this question:

Question 54:

What is the output of the following segment of code?

        int sum = 0;   
        for (int i=0; i<=5; i+=2) {
            sum = sum + i;
        }
        System.out.println(sum);

If you speak Java, then you can work out that the answer is 6. So we use 6 as one of the possible options, and add a few random distractors:

Answer:

(A) 1
(B) 896
(C) 6
(D) -4
(E) 5

In this case, if students have a common misconception, they may arrive at an answer that is not on the list. For example, a student who does not understand that this definition makes the loop increment by 2 might assume that the answer is 15. But 15 is not on the list. So the student is implicitly told that they are wrong, and they can discard unlikely options (“Negative? – No.” “More then 800? – No.”) and make a random guess between a few remaining options.

This question is even worse in a way: Another common mistake is not being able to correctly distinguish between <= and < in the loop condition, and getting the answer wrong because of this. The way the question is written, however, it is not even possible to expose this misconception, because both variants happen to result in the same output!

A badly written question.

The off-by-one way of writing distractors

The other very common pattern I see all the time has an example in this question.

Question 31:

 Which of the following code segment would execute the stored procedure “getPassword()” located in a database server?

  1. CallableStatement cs = con.prepareCall("{call.getPassword()}");
    cs.executeQuery();
  2. CallabledStatement callable = con.prepareCall("{call getPassword()}");
    callable.executeUpdate();
  3. CallableStatement cab = con.prepareCall("{call getPassword()}");
    cab.executeQuery();
  4. Callablestatement cstate = con.prepareCall("{call getpassword()}");
    cstate.executeQuery();

The way questions like this are written is usually this: The question setter writes the correct response, and then creates three distractors by introducing one error in each.

It is trivial to reverse-engineer the correct answer from this. To do this, we spot the differences, and for each differences discard the odd one out. For example:

  • One difference is that “{call getpassword()}” is written with a dot in the middle once, but without the dot three times. So the dot is the odd one out, we discard Answer 1.
  • Another difference: CallableStatement is written with w capital S three times, with lowercase s once. So we discard answer 4.
  • Last one: The second line calls executeQuery() three times, and executeUpdate() once. Answer 2 is out.

And voilà – there is our answer: 3. We can do this without any knowledge or understanding of the subject matter. Keep your eyes open for this pattern in MCQ tests. It is astonishing how often you will see this question pattern.

A better example

Here is an example of something you could do:

Question 8:

HashMap<Integer, String> myMap = new HashMap<>();

myMap.put(2, "Tiger");
myMap.put(4, "Lion");
myMap.put(13, "Lion");
myMap.put(2, "Elephant");
myMap.put(3, "Chicken");

System.out.println(myMap.get(2));

When you try to compile and run this code, what will happen?

    1. The code has an error, because the value “Lion” is used twice in the HashMap.
    2. The code has an error, because the key 2 is used twice in the HashMap.
    3. The code will run and print “Tiger”.
    4. The code will run and print “Lion”.
    5. The code will run and print “Elephant”.

 In this case, the question tests an understanding of how maps work.

  • Option 1 tests the possible misconception that values cannot appear twice in a map.
  • Keys, on the other hand, are unique in a map. Option 2 tests the misconception that they cannot be replaced by a new entry.
  • Option 3 tests the misconception that the first entry for a given key persists.
  • Option 4 tests the misconception that the get(n)-parameter is an index, mirroring list-behaviour (referring to either the second or third element entered, depending whether the list is zero-based or one-based).
  • And Option 5 is correct.

This is by far not all that there is to say about MCQ writing. But it’s a start. It is the absolute minimum you should consider if ever you find yourself in the position to write and MCQ test.

Announcement: Strype – a frame-based Python system

Strype logoLast week, we have officially released Strype 1.0, a new Python editor and runtime system running entirely in a web browser. After being in development and available for preview and testing for some time, the system is now fully supported for use in classrooms or for individuals wishing to learn Python programming.

Strype’s most unique feature is its frame-based editor, a novel code entry and manipulation technique which makes writing of code quicker, easier and less error prone. In a manner similar to block-based systems, statements are entered and manipulated as single entities, and are laid out on screen automatically. Many syntax errors are avoided, indentation errors are not possible.

Sample of Strype code

While frame-based editing profits from some of the same advantages as block-based programming, it does not, however, suffers the same restrictions: programs in Strype have to full power of the Python language, purely keyboard-based efficient editing is supported, and programs retain the readability, navigability and maintainability of standard Python systems.

Strype is available free of charge, with no installation needed.

Links:

Strype webpage

The Strype editor

The Strype editor for micro:bit

Strype user community

Strype was developed by the Programming Education Tools Group at King’s College London. Team members: Pierre Weill-Tessier, Neil Brown, Michael Kölling

The New Joy of Code – 01: Introduction

Finally it is here: The first of the New Joy of Code programming series.

In this series of videos I will show you how to program in Greenfoot and Stride. Later in the series, we will also get to programming in Java. But ultimately, what you will learn are the principles of object-oriented programming, which you should then be able to apply to any language.

This first episode shows you what Greenfoot is, and what Stride looks like.

Download Greenfoot here.

Check back soon for more episodes.

 

The New Joy of Code – Coming Soon!

The most popular pages on this blog are the episodes of the ‘Joy Of Code’ video series, which teaches programming with Greenfoot. This video series, however, is somewhat outdated by now. So I have decided to update it and start again.

So, coming soon: The New Joy of Code. It will be similar to the old one, a series of short videos that can be used for self study, or in a classroom, but there will be differences: First, I will, of course, use a recent version of Greenfoot (Version 3.1.0 at the moment), but more importantly: I will use our new language Stride for the tutorial. We will later also transition into Java, but Stride is such a great language for beginning to program, that I really want to start with that.

With the videos – you’ll see!

Greenfoot LIVE

In order to provide more ways to support teachers who are using Greenfoot (or thinking about using Greenfoot), as well as anyone else interested in Greenfoot, we are starting a series of live chat sessions:

Greenfoot Live
Programming Education Chat With The Greenfoot Team

Greenfoot Live will be a regular chat event where members of the Greenfoot team will be live online and talk about Greenfoot, among ourselves and with you – the users of Greenfoot. The event is aimed mostly at teachers who use Greenfoot in their classes, but also at general Greenfoot users.

We will discuss educational aspects, as well as general programming topics, related to Greenfoot.

You will have a chance to listen to us talk, as well as ask questions.

The first Greenfoot Live event will be held on

Monday, 8 May 2017, at 5pm UK time

That is

  • 5pm in the UK
  • 4pm GMT
  • 9am in San Francisco
  • 12noon in New York
  • 18:00 in Germany, Scandinavia, Italy, France, …
  • 2am in Sydney (sorry…)

You can join us here.

After this first event on 8 May, we will host Greenfoot Live every two weeks. Recordings of these events will be accessible afterwards for viewing retrospectively.

Don’t forget to mark this date in your diary with a big red pen and join us in a couple of weeks!

Stride: Visuals and layout

When you give up the idea of characters as your sole point of source code representation, a host of possibilities opens up to improve the visual appearance if your program. This short video shows some of then as implemented in Stride.

 

Stride is implemented in the Greenfoot environment, available from here.

Stride: With a little help from my editor

In the third part of our series introducing Stride and frame-based editing, we look at the various ways in which the editor can provide you with help in writing your program. If you consider using Stride, it will come handy to know these sources of information.

Stride is implemented in the Greenfoot environment, available from here.

What’s in a symbol? Assignment revisited

I was at a Dagstuhl seminar recently, about programming education, and one of the things we discussed were common misconceptions of novice programmers. One of these, reported by several seasoned programming teachers, was the difficulty in interpreting the assignment symbol in C-style languages:

number = 42

There are several things that can be misinterpreted here. Firstly, the equality operator is non-directional. Some students have problems remembering which way it operates. Consider this statement:

a = b

Which way does it assign? To all of us who have programmed for some time this may seem a silly question – we get used to the right-to-left semantics so thoroughly that we can hardly imagine it to be different. But for beginners? Not so clear. (And, of course, the direction is arbitrary; language designers could just as well have decided the other way.)

The second possible misconception is related to the previously learned meaning of the equals symbol: to express equality. This has been used in maths for centuries, and taught to most pupils before they ever encounter programming, and is quite different from assignment. Consider this code fragment:

a = 0;
b = a;
a = 7;

What, now, is the value of b? For learners holding the equality misconception, it will be 7. This is not entirely unreasonable; it is indeed an internally consistent mental model. The misconception is that variables, once expressed as equal, remain linked. So the equals symbol is interpreted as equality (“b is the same as a“), which might then remain true for the future.

Many people have commented for a long time that the C-style equality-as-assignment is not ideal. Many languages do better. Algol and Pascal, for example, already used a different symbol in the 1960s, quite explicitly to express the directionality more clearly, and to distinguish from equality:

number := 42

I have always like this much better than the single equals and have always been mildly annoyed by the C-style syntax.

So when we designed our language Stride recently, the question surfaced again. What should we choose?

We had two competing goals: On the one hand, we wanted the syntax to be clear and expressive, but on the other hand we want Stride to be a pathway into Java, so there is also a strong incentive to be consistent with Java syntax. (Java, of course, uses the C-style syntax, so these two goals are in conflict.)

At first, we prioritised the Java-compatibility argument and chose the equality symbol. The first release of Stride had this as the assignment operator. But then the Dagstuhl discussion happened, and I started to think again.

In Stride we have an advantage: Because the representation of the program is not directly typed in, but produced by the system in reaction to command keys, we are not restricted to characters that can easily be typed on a standard keyboard. So initially I considered going back to a syntax that Smalltalk used (in early versions, before it fell back onto the := variant), a left arrow:

arrow-assign

 

I liked it for its clarity of expression, but it has one disadvantage in a teaching language that is intended to lead to Java and similar languages: it does not directly prepare learners for the equal symbol so widely used for assignment in other languages. After we discussed this in one of our team meetings, Neil Brown, one of our team members, came up with what I think is a brilliant compromise:

 

This is what assignment now looks like in Stride. The double arrow still has a visual link to an equals sign, but is clearly distinct from it; it is also clearly directional. We still use the equals key as a command key to enter an assignment, further reinforcing the link for the future.

Here’s what our assignment looks like in context:

stride-code

 

I am very happy with it. I think it reads well and is the closest we can get to reaching two competing goals.