{"id":1780,"date":"2024-02-27T08:35:15","date_gmt":"2024-02-27T08:35:15","guid":{"rendered":"https:\/\/blogs.kcl.ac.uk\/aflkings\/?p=1780"},"modified":"2024-02-27T10:27:05","modified_gmt":"2024-02-27T10:27:05","slug":"student-authored-exam-questions-a-cross-institutional-student-centred-approach-to-summative-assessment-preparation","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/aflkings\/2024\/02\/27\/student-authored-exam-questions-a-cross-institutional-student-centred-approach-to-summative-assessment-preparation\/","title":{"rendered":"Student-authored exam questions: a cross-institutional student-centred approach to summative assessment preparation"},"content":{"rendered":"

UCL students collaborating to write exam questions in Antonio\u2019s course. Reproduced with the permission of the students featured.<\/em><\/p>\n

Instructors:<\/strong> David Sheard (KCL) and Antonio d\u2019Alfonso del Sordo (UCL)<\/p>\n

Email<\/strong>:\u00a0<\/strong>David.1.sheard@kcl.ac.uk and a.dalfonsodelsordo@ucl.ac.uk<\/p>\n

Module:\u00a0Level 4, <\/strong>BSc Mathematics 4CCM111A Calculus I (Mathematical Sciences, NMES, KCL) and BSc Earth Sciences MATH0049 Mathematics for Engineers 1 (Dept of Mathematics, UCL)<\/p>\n

Assessment activity:<\/strong>\u00a0 In this case study, we present two versions of the same type of activity which took place on two modules at two similar institutions, namely asking students to write exam-style questions, and then solve and assess each others\u2019 work.<\/p>\n

Why did you design this activity? <\/strong><\/h2>\n

Two big problems we face as educators when it comes to exam\/revision teaching are 1) preparing students sufficiently for their exam without teaching to<\/em> the exam; and 2) addressing the wide disparity among our students when it comes to revision\/exam technique and related aspects of the so-called hidden curriculum<\/em>.<\/p>\n

By replacing traditionally taught revision sessions with students writing their own exam questions, students are obliged to reflect on the course content<\/strong>; their understanding of how different topics are related<\/strong>; the learning outcomes<\/strong> (whether stated explicitly, or understood implicitly); and how these elements are assessed<\/strong>.<\/p>\n

Students are sometimes lured into counter-productive revision strategies, or perhaps find that strategies which worked in the past (e.g. in school exams) are no longer useful for university exams. Asking students to reflect on the purpose and method of their assessment, and then to implement this in practice may help them identify what revision\/exam techniques are most helpful<\/strong>.<\/p>\n

Students writing exam (style) questions is not a completely novel idea and has been tried in several disciplines, usually with multiple choice questions (MCQs). The Unverity of Auckland in new Zealand has an online platform, Peerwise,<\/em> where questions written by students in a wide range of subjects (including Mathematics) have been collected [PeerWise]. Using student-authored MCQs has been investigated [Hutchinson 2013], particularly in the context of Biology [McLeod 1996], [Sibomana 2020], and [Jobs 2013]; and medicine [Craft 2017] and [Palmer 2006]. They have also been used as a tool in continuous assessment for learning [Davis 2013]. Some research on student generated MCQs outside if biology and medicine has also been done, for example in accounting [Geiger 2020], and business studies [Fellenz 2004] and [Sharp 2007]. Doyle and Buckley have quantitively analysed the effect of students creating MCQs on their learning, and found a statistically significant impact on students\u2019 end-of-module assessment marks [Doyle 2019]. To the best of our knowledge, little research on student-authored long-form exam questions has been carried out, nor research which looks specifically at mathematics teaching.<\/p>\n

Our activities aimed to cover the full range of the exam setting-sitting process, from what and how to assess, through creating, sitting, marking, and giving feedback on assessment questions. This allowed us to engage students with a wide range of active learning approaches<\/strong>. The overall design was informed by DARTs (directed activities related to texts) [Vester 2008], a strategy sometimes used when teaching English as a foreign language, an area where Antonio is a specialist. In particular, DARTs can be used to develop reading comprehension exercises for students, by students.<\/p>\n

How did you introduce the activity to students? <\/strong><\/h2>\n

Writing exam questions is a very complex task which brings together disparate skills and knowledge. In order to prepare students adequately for this, we have found that it is important to lead in by encouraging students to reflect on principles of good revision<\/strong>, and exam technique. We approached this in different ways, eg using interactive polling tools for larger classes, or think-pair-share for smaller classes.<\/p>\n

We then introduced general exam question writing principles<\/strong>; in particular, having a balance between familiar questions and unseen material. To put this in practice, students were shown past exam questions, and through discussion with their peers, asked to identify features such as command words (show, state, hence, etc.), relationship with learning outcomes, and identifying\u00a0 hypotheses\/assumptions, and the form required for the final answer. We also gave them an unseen exam style question, and asked them to simulate what they would do in the first five minutes of an exam in terms of planning their approach to the question.<\/p>\n

How did you set up the formative activity?\u00a0 <\/strong><\/h2>\n

We implemented this activity on two similar first year calculus modules; one at King\u2019s for mathematicians, and the other at UCL for Earth scientists. Both versions of the activity had three phases: 1) Lead-in:<\/strong> exam question structure and exam technique; 2) Production:<\/strong> guided design of exam questions and mark scheme in groups; and 3) Feedback:<\/strong> group solving and peer marking of each others\u2019 questions.<\/p>\n

In David\u2019s version, which was run at King\u2019s with a course of around 300 students, these were spread over three one-hour sessions in the last three weeks of term (alongside with other teaching activities) run in four parallel sets of sessions. In practice, the attendance rate was quite low for these sessions. In the other version at UCL run by Antonio, the three phases were consolidated into a single one-hour session with about 30 students present.<\/p>\n

Both of these approaches took place in the Autumn term of the academic year 2023-24 in Mathematics courses for first year students. In each course the content was similar, consolidating and extending what students had previously learned during their A-levels. Throughout these activities, we collaborated to plan the sessions and fed-forward feedback from each other\u2019s sessions to anticipate and mitigate challenges which arose.<\/p>\n

We have outlined the Lead In<\/strong> phase in the previous section. For the Production <\/strong>phase, it was time for them to try to generate ideas for exam question writing. In one case, we discussed the possibility of using ChatGPT to come up with the unseen, most challenging parts of the question. Although, it was made clear that ChatGPT does not always return the correct question\/answer, so care is required.<\/p>\n

We here outline the seven-stage process (Production<\/strong> and Feedback<\/strong>) that was used for the single two-hour version of this activity. The class was divided in (at least 3) groups of 3 to 4 studentsEach group had a team-leader who randomly picked a piece of paper indicating two topics from different parts of the course that they should assess in their question. The team leaders were chosen to be students who had done well on the coursework, to ensure that group members had a range of levels.<\/p>\n

    \n
  • Stage 1: Plan.<\/strong> Students were given 10 mins to plan the content of the subparts of the question and the learning outcomes they want to assess \u2013 students were asked write this out explicitly.<\/li>\n
  • Stage 2: Write the questions.<\/strong> Students had 20 mins to formulate their questions, with help from the lecture notes and problem sets. The more time that can be allowed for this the better, even 20 minutes is quite restrictive.<\/li>\n
  • Stage 3: Worked solution\/mark scheme.<\/strong> Students had 10 minutes to write out solutions. This can be done alongside the previous step. It is important, however, that students write questions and answers on different pieces of paper for the following stages. Students also need plenty of time for this step.<\/li>\n
  • Stage 4: Solve another group\u2019s question.<\/strong> Each group gave their question to another group to solve in 10 mins. This is where students are most comfortable; but, in some cases, they spotted mistakes\/ambiguities in the questions and were allowed to ask the group who wrote the question for clarification.<\/li>\n
  • Stage 5: Mark allocation.<\/strong> A third group is given the question together with the mark scheme. They agree on how to allocate the marks for that question (before seeing the second group\u2019s answers). This stage, along with the next one, should roughly take 10 mins.<\/li>\n
  • Stage 6: The marking.<\/strong> The third group marks the second group\u2019s answer and provides that group with feedback. This was probably the most enjoyable part for students as they put themselves in the role of examiner.<\/li>\n
  • Stage 7: Session feedback.<\/strong> At the end of the session, students were asked to complete a questionnaire with concept check questions, questions about their experience of the activity, and open-ended feedback questions.<\/li>\n<\/ul>\n

    If time permits, the second and third groups can also give feedback to the first group about the level and appropriateness of the question as an exam question. To motivate the students, a prize (consisting of a calculus textbook and some chocolate) was offered for the best question. This helped the students develop some healthy competitiveness.<\/p>\n

    David\u2019s version of this activity at King\u2019s was broadly similar, except spread over three sessions. The lead-in was in the first session, stages 1-3 in the second, and stages 4 and 6 in the third. One difference was that the mark allocation (stage 5) was combined with stage 3. Stage 7 was incorporated at the end of each session as an exit-ticket.<\/p>\n

    In this version of the activity, the quality of the questions written by students was extremely variable because they were given less scaffolding in terms of the topic and structure of the questions they should write, and the group size was larger (4-6). So that the third session worked, the three best questions were selected after the second session, and neatly typed up \u2013 each group was assigned one of these to solve and then another to mark.<\/p>\n

    How did you and the students check their learning? <\/strong><\/h2>\n

    The easiest and most direct way to see how students had performed was to look at the questions they wrote. It turned out to be very easy to judge at a glance which groups had done poorly or well. The poor questions would often only have a single part, if they had more than one part, those parts did not assess distinct skills, and were often either trivial or mathematically impossible. On the other hand, good questions had several parts which assessed different skills, at a reasonable level of difficulty.<\/p>\n

    When it comes to the lead-in, as the teaching was discursive, it was possible to obtain immediate feedback<\/strong> on their understanding. In the final phase, students gave each other feedback<\/strong> on their questions and answers, allowing them to learn from one another and to appraise their own learning. We also ran exit tickets\/feedback surveys<\/strong> at the end of each session. This gave us, as instructors, an insight into how students felt about their learning, but surprisingly (to us) was also a way for students to reflect on their learning<\/strong>.<\/p>\n

    An important component of these activities is collaborative creation<\/strong>. How this went could, of course, be judged indirectly by the quantity of the output: the questions. We could also observe this aspect directly as we walked among the groups during stages 1-3 and discussed question ideas with the students. Here it was easy to identify instances where some students either took over their groups, or did not engage meaningfully with the activity.<\/p>\n

    How do students benefit\u00a0 from this activity?<\/strong><\/h2>\n

    The clearest benefit that these activities are designed to provide is to better prepare students for their exams<\/strong>. There are several facets of exam preparation which these activities touch on, so it is worth taking a moment to look at them.<\/p>\n