{"id":947,"date":"2021-03-30T17:00:38","date_gmt":"2021-03-30T17:00:38","guid":{"rendered":"https:\/\/blogs.kcl.ac.uk\/digitaleducation\/?p=947"},"modified":"2021-11-15T16:07:39","modified_gmt":"2021-11-15T16:07:39","slug":"peer-feedback-in-large-classes-using-the-workshop-activity","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/digitaleducation\/peer-feedback-in-large-classes-using-the-workshop-activity\/","title":{"rendered":"Peer feedback in large classes using the Workshop activity"},"content":{"rendered":"

Pedagogical research provides a clear rationale for asking students to provide feedback to their peers<\/span>\u00a0on formative assignments<\/span>. By giving feedback, students understand better the demand of the task<\/span>,<\/span>\u00a0<\/span>internalize<\/span>\u00a0<\/span>our<\/span>\u00a0marking\u00a0<\/span>criteria<\/span>\u00a0and have an opportunity to benchmark their own work against that of their peers<\/span>.\u00a0<\/span>Additionally<\/span>, feedback provided by peers is often more understandable than the feedback provided by a lecturer and can be timelier, particularly with large cohorts of students.\u00a0<\/span>\u00a0<\/span><\/p>\n

This is what motivated us to introduce peer feedback<\/span>, initially<\/span>\u00a0on a\u00a0<\/span>third year<\/span>\u00a0module, with approximately 90-1<\/span>1<\/span>0 students<\/span>. Later,<\/span>\u00a0rather daringly,\u00a0<\/span>we attempted it\u00a0<\/span>on<\/span>\u00a0a\u00a0<\/span>first year<\/span>\u00a0<\/span>module of nearly 600 (now nearly 700)<\/span>\u00a0students<\/span>.<\/span>\u00a0In both cases the students provided feedback, not a mark,\u00a0<\/span>on<\/span>\u00a0a written piece of work, an\u00a0<\/span>essay<\/span>\u00a0and a lab report<\/span>,<\/span> respectively.<\/span><\/p>\n

One of us had\u00a0<\/span><\/span>previously<\/span><\/span>\u00a0<\/span><\/span>introduced peer feedback on\u00a0<\/span><\/span>draft essays<\/span><\/span>\u00a0in\u00a0<\/span><\/span>the<\/span><\/span>\u00a0<\/span><\/span>third year<\/span><\/span>\u00a0<\/span><\/span>module<\/span><\/span>,\u00a0<\/span><\/span>distributing via email\u00a0<\/span><\/span>anonymised <\/span><\/span>essays for feedback (two essays\/student<\/span><\/span>).\u00a0<\/span><\/span>The students could then modify their draft, which was re-submitted for tutor marking.\u00a0<\/span><\/span>This had been well received and very successful, but it was not sustainable with increases in student numbers.<\/span><\/span>\u00a0<\/span><\/span>Therefore<\/span><\/span>\u00a0we decided it was necessary to find\u00a0<\/span><\/span>a\u00a0<\/span><\/span>way to do this automatically in\u00a0<\/span><\/span>KEATS<\/span><\/span><\/a>.\u00a0<\/span><\/span>But how?<\/span><\/span><\/p>\n

\"PeerMarkThe obvious choice of tool seemed to be TurnitIn PeerMark<\/a>, which had been developed precisely for peer feedback and marking. Sadly, PeerMark turned out to have a bug and did not work for us, in particular with student numbers of over 200. Random essays were not allocated to any reviewer and this had to be corrected manually. Therefore, we decided -rather in a hurry- to move to the Workshop activity<\/a> in KEATS, which had the potential to be applied to perform the same task.<\/p>\n

We only had two or three days to make the transition, because the deadline for submission had already been given to the students. This made it impossible to ask for CTEL support<\/a>. An extremely well designed wiki from UCL<\/a> (publicly available) was our only guide. This guide helped considerably, even if the Moodle version used by UCL has (or at least had) a slightly different functionality from the King\u2019s version, KEATS.<\/p>\n

Workshop organises the activity in phases, as shown in the screenshot below. One switches between phases simply by clicking on them. One needs to worry about three areas only of workshop administration, indicated by the arrows.<\/p>\n

\"Peer<\/p>\n

The \u201cedit settings\u201d are fairly self-explanatory, but it is very important to note that they include text boxes for the description of the task in general, the submission and the provision of feedback. These areas need to be completed very carefully and clearly.<\/p>\n

An example of a general description of the task is provided below. It is essential in the general instructions to tell the students to save their work frequently, as the system does not auto-save.<\/p>\n

\"Peer<\/p>\n

It is also important to note that Workshop, in contrast to PeerMark, does not allow the students to provide inline comments.<\/p>\n

However, it is possible to get around this limitation by asking students to download the allocated assignment, comment offline, and re-upload the reviewed work. We asked only third year students to do it, on a voluntary basis. In this case it is essential to provide instructions to the students in the \u201cedit settings\u201d section, and to ask students to submit their work as a word file. The provision of this detailed feedback is otherwise impossible, unless they have software to annotate PDFs.<\/p>\n

To help the students to provide useful feedback, we found it beneficial to set up a few questions that they need to answer to. This ensures some consistency in the type of feedback provided, and its quantity. This is done in \u201cedit assessment forms.\u201d<\/p>\n

\"Peer<\/p>\n

The final element to set up is \u201callocate submissions.\u201d We found that the scheduled allocation, shown below, worked well. We set up the number of reviews to two per submission (which means 2\/student).<\/p>\n

Providing feedback to two lab reports or essays and thus receiving two sets of feedback ensured that our students received enough feedback, whilst not having to review too many assignments. One review only is unadvisable because standards vary, but with two, most students will receive at least one useful feedback. It is true that the pedagogical literature suggests giving feedback is even more useful than receiving it, still it is very demoralising receiving only poor quality feedback!<\/p>\n

\"Peer<\/p>\n

Once the setup phase is completed, one can move to the submission phase. It is important to remember to switch phase, or submission will not open on the date you set up in workshop administration. The move from submission to assessment phase can be automatic or manual.<\/p>\n

Once the assessment phase has finished, it is important to move manually to the evaluation and then the closed phase. Failing to do so means that the students can no longer give feedback but cannot yet access the feedback received.<\/p>\n

It is very easy to forget to transition between phases which happened to us a couple of times. Luckily the students are very prompt in complaining, so one can fix this rapidly.<\/p>\n

We have now run Workshop for peer feedback for 2 years, for a total of over 1,200 students, with no major disaster. Mistakes are easy to rectify. This year 84% (580 students) of the first year class submitted a formative report. Of those, 83% provided peer feedback (97 students did not). This means that 5.5% of students did not receive any feedback from their two peers \u2013 feedback was provided by one of us (CK ). The percentage of students taking part was lower for the third year class (75%, 62 out of 83), but all students who submitted a formative essay draft provided peer feedback. The percentage of third year students taking part has, however, increased dramatically, from 30% of students when we first introduced peer feedback (30 out of 99) to the current 75%. We can now tell the students that in the past the students taking part in peer feedback achieve a significantly higher mark in the summative essay. This has probably led to the very substantial increase in participation.<\/p>\n

When they were asked in a questionnaire whether they found the exercise useful, the overwhelming majority of third year students rated the experience 6-10 on a 0-10 scale, as shown below. We have not yet formally sought the opinion of first year students\u2014anecdotally their experience is more mixed with some of them enjoying the opportunity and others finding the varying quality of feedback that they receive from their peers dissatisfying. However, we like think to think that the benefit of this exercise may be reaped later, by preparing students in their first year at university for a broader range and style of assessments and by beginning to train them in the highly relevant skills of providing feedback and reviewing others\u2019 work.\"Student<\/p>\n

The main technical issue to consider in relation to the use of Workshop seems to be that of anonymity.<\/p>\n

First year students are less worried about anonymity, since with a cohort of 600+ they are extremely unlikely to know the students they are assigned to. However, anonymity is required by the third year students, who otherwise can find it challenging to provide completely honest feedback.<\/p>\n

The UCL Moodle allows module organisers to make the exercise anonymous, changing users’ permissions. KEATS instead is set up in such a way that the students can see each other\u2019s names, and the module organiser cannot change that. This needs to be done by the Faculty\u2019s learning technologists each time, so it is important to remember to ask in time.<\/p>\n

In conclusion, it is a little scary to use Workshop for the first time. There are many settings to consider, and until the allocation is done one does feel somewhat anxious. But if everything is set up carefully, the system seems robust.<\/p>\n

It is essential, though, to provide very clear instructions for the students within the Workshop itself, including deadlines and a warning to remember to save their work often.<\/p>\n

We found it also essential to provide a session explaining the rationale of the exercise, and the procedure. For the third year module, in which the feedback provided by the students is more thorough, we have a tutorial before the formative essay submission. In this tutorial we go over the marking criteria together, and the students are provided with good examples of peer feedback from previous years. This has ensured that from the second year that peer feedback was introduced the standard of feedback provided has been more consistent and of a higher quality.<\/p>\n

Whilst third year students tend to be more experienced with the learning environment at university, most of our first year students are used to assessments at school where they are rarely asked to provide feedback on each others\u2019 work. As expected, they often react with considerable anxiety around this exercise. To provide these students with a clear framework around the exercise we run \u2018How to write a lab report\u2019 sessions and provide complementary materials on KEATS early in the semester where we not only provide a clear guideline for what is expected in the lab report, but also explain the rationale and benefits of participating in the peer feedback exercise.<\/p>\n

For the first time this year we also offered debrief \u2018Lab report clinic\/Q&A\u2019 sessions after the peer feedback exercise in which common strengths and mistakes were discussed using anonymised lab reports from previous year\u2019s students. Students attending these sessions responded positively and found them \u2018very helpful.\u2019
\nRemember to switch phase at the appropriate time!<\/p>\n

Useful Links<\/h3>\n