Student co-production of assessment in an online MSc

Instructor: Dr Anja Harrison

Email: anja.harrison@kcl.ac.uk

Module: IoPPN, Online MSc Neuroscience, Mental Health, level 7

Assessment activity: Co-production of summative assessment. Working in focus groups and continuing sessions, students co-designed a more innovative and authentic assessment coursework task with criteria and feedback opportunities for a module they would later take.

Why did you introduce the assessment?

This was run on the From Methods & Procedures to Analysis & Reporting’. This is a statistics-heavy module worth 15 credits as part of the MSc in Neuroscience, Psychology and Mental Health, and the MSc in Applied Neuroscience, two of King’s flagship Distance courses. This module is run as part of a 8-week carousel King’s Online model, with tight turnaround times for learning and assessment. The module sits within some of the last research modules that students undertake before they conduct their synoptic projects, and so far in their learning, students have only been assessed through coursework, largely the written essay.

Previously, the summative assessment consisted of two pieces of courseworks:

CW1 assessed students’ ability to critically assess research design of presented papers (guided by a set of 5 questions) (to be completed in week 3)
CW 2 assessed students’ ability to carry out and interpret statistical analyses, e.g. ANOVA/ multiple regression using a given SPSS data set (to be completed in week 6)

Issues became obvious to us through informal conversations with staff and students on this module. Firstly, it was felt to be removed from real procedure, mechanical and a bit artificial. There was also an issue with the total length of both pieces of coursework (1500 words each) and the two assessment submission dates, which was a burden on both staff and students within a 8-week turnaround. An example of a student comment:

[…]Thank you for your response, I have to admit, this is all too overwhelming for me. So many new concepts, new assignment structures with two deadlines and the mass of reading while trying to work feel untenable at the moment and causing a lot of stress. […]

The new coursework intends to move away from assessment of learning towards assessment for learning. It should not only provide ‘real-world application’ of acquired knowledge and provide students with the ability to practice likely tasks they’ll be asked to carry out after graduation, but also speaks to the current trend towards alternative, inclusive assessment providing students the option to produce a narrated PPT for CW Part 2. This is of importance to some of our neurodiverse students, who find written CW not the ideal way to demonstrate their learning (as per recent focus group discussions).

[…] The existing coursework feels like a bit of a tick-box exercise. Very removed somehow. Why would I need to be able to answer questions on a bunch of papers? Makes it a dreary task to complete[…]

[…]I am a neurodivergent student and find that I often want to listen to my reading or find relevant podcasts to listen too. Would King’s ever consider allowing students to submit a podcast style body of work? Fully referenced with some written work to back it up? Or maybe a narrated presentation?

[…] I might be graduating from King’s College London with no clue on how to create anything other than an essay. Who would have thought that possible? […]

In addition, students were from a variety of background with various career pathways, some more interested in the neuroscience route while others in psychology, but all students had to work with the same data which was drawn from a study which might not represent their area, and indeed was quite decontextualised from their authentic experience.

This assessment initiative used a method of co-production with students on the MSc programme and over 7 sessions in just under three months , the assessment was redesigned to be:

CW1: A mock grant application where students choose from a selection of topics depending on their preferences (either more ‘classic’ Psychology or Neuroscience of Mental health). Based on a National Institute for Health Research (NIHR) grant form, students have to develop their own research design based on info provided by the team. They use the grant form questions to develop their own ideas, but within quite limited parameters so students are scaffolded.
CW2: Students create a stats tutorial for fellow students from inputting data into SPSS based on a hypothetical example to interpreting results either in written or narrated PowerPoint format (750 words) giving them a choice of output.

Why Co-production?

Giving students a platform to participate in discussions informing decision-making processes aligns with KCL’s guiding principles to implement our strategic mission, including
Create an inclusive environment where all individuals are valued and able to succeed
Use best evidence and critical enquiry, and learn from our successes and failures
Enable meaningful connections between our students and staff
Take a holistic approach to education, research and service to enable our students and staff to develop character and wisdom for the benefit of others
Demonstrate open mindedness and tolerance and expect to challenge and be challenged in protecting freedom of expression.
I feel that the most valuable way to assess students is to give them a task with something that matters to them, and they have investment and ownership over. As a result of the co-production, students were able to not only give feedback in a one directional way at the end of a module when it is often too late for it to be of benefit for them, but to see their thoughts and views and experiences of assessment valued.

In addition, as part of their employability skills, students need to walk away from our degree programme with authentic skills that they will apply, not just essay writing skills! This is evident in the comments above. The resulting assessment that students co-designed was more applied and therefore more authentic. It also provides more formative practice and feedforward for the other research modules where they have to design their own research project. Having seen a grant application, they can contextualise the data as part of a holistic project but also see the application of research in the real world.

Indeed, the experience of co-creating an assessment is something they can surface in interviews and there are many useful skills gained from this experience itself, such as collaboration, negotiation, cultural competency in working with others, and a working understanding of evaluation and assessment in education.

How did you set it up?

Because I was unable to co -create with students during the running of the module (see below), we had to talk with students who had not taken the module but were intending to. This required me to recruit students on a volunteer basis to help us co- design the module through focus group sessions.

Firstly, I advertised sessions through the KEATS module forums board, with the help of our student representatives and by programme announcements. Different sessions were offered so people in different time zones can attend. Our students are distance students studying within their home countries all over the world and therefore this was crucial. All sessions took place online via Teams.
I recruited 10-15 people for focus groups on volunteer basis.
Focus group sessions were semi-structured. I had a set of questions to stimulate discussion based on my own and staff evaluation of the module but was keen not to lead students too much. I wanted to know where they had feedback and concerns and crucially how they could redesign the assessment
There were 7 sessions altogether- the first two were talking and evaluating (and crucially building rapport between us and the students themselves) then some sessions developing the content. I went away and with the team we designed an outline of the new assessment, presented it to the students and the final sessions were all of us refining assessment to the shape it is currently in.
Not all students were taking the module immediately, but they were updated on the process through module boards so more could be involved and ask any questions if they wished.

How did you get the students on board with co-production?

It is crucial to this process that students were not co-producing during the module. On a time-limited module like this, it would have added to student and staff burden and confusion to be designing an assessment alongside their learning. This meant that the students were designing the assessment beforehand for their cohort colleagues and also themselves as all of them would of course take this module eventually. Indeed, initially sessions were not well attended but when students realised it was about their own module the attendance and involvement shot up.

Students were largely new to this process so were initially sceptical and a bit hesitant, but talking with them and in co-production sessions, I explained the rationale for this work and the benefits. The best thing was just to let students talk and go from there.

How did you and the students co-design the assessment criteria and weighting of the assessment?

As part of the process of co-creation, students helped to design the marking criteria for fairness and what they thought was missing. This links to both potential outputs of coursework 2, but of course LOs are same. The only addition was for aspects of a PPT that might not be relevant for a written guide. Presentation – additional marking criteria_formative only

How do you support the students with the assessment?

We had to bear in mind that many students taking this module had not been involved in the design of the module. We still required scaffolding for our students as this was the first time most of them would have done an assignment like this. Methods of doing this:

Very clear assessment briefs and transparent criteria
Assessment discussed in webinar sessions as part of our course
Examples of grant forms and an SPSS tutorial are available on KEATS produced by me, accompanied by an additional webinar

Turnitin does not support media files, so we use Moodle Assignment as an option for those submitting a narrated PPT. Our TEL team provided instructions for students on how to do this and we have two submissions points on KEATS.

What benefits did you see?

For Students:

The module ran well and smoothly. To begin with students were generally quite anxious about the content and the new CW. Additional webinars and examples provided as part of the new assessments were taken up well and praised by the students. Drop-in sessions with the module lead and the TFs were well attended, with some students submitting questions in advance.

Overall, we had an excellent levels of engagement with the Q&A boards and live sessions. The TFs reported good interactions with students.

Qualitative feedback from students collected via email and in four drop-in sessions indicated high levels of satisfaction with the new courseworks. Particularly, students’ confidence in approaching new assessment formats and being able to conduct statistical analyses was high. See these two representative quotes for CW1 and 2:

I wanted to be sure you knew how much I enjoyed CW1. It was such a creative, fun, and valuable way to teach important information. The material itself was engaging and relevant — b/c who isn’t addicted to the Internet in some way today? (There ought to be research on addiction to CW.) More importantly, the CW was organized in such a way that I learned well — really well. IVs, DVs, digging for a gap in the literature, and venturing into a different kind of aim to the writing than we’ve done before… A recipe for some deep and satisfying work for this one! (CW1)

CW2 was SUCH FUN!!! Please know that I am so far from being a numbers & charts person, and approaching this module was SCARY. True to your word, the material helped me calm down and discover that thanks to SPSS, no math expertise is necessary. The material really walked us through every step necessary to find our results. I stuck to preparing a written tutorial and really enjoyed the creativity and ingenuity of getting that baby down to 750. (CW2)

Grades:

There were more distinctions, with an unchanged pass and fail rate as compared to previous runs although the same learning outcomes were captured. Interestingly, the rate of non-submissions was down to 15% from 23% and 22% in the two previous runs. This is in line with our hopes to reduce the amounts of submitted MCFs by making the workload more manageable and CWs more engaging.

For staff:

The reduced word count and the narrated powerpoint option has led to a more manageable workload for teaching fellows. Although the overall requested office hours didn’t seem to have been reduced dramatically, teaching fellows who offered their feedback reported the assessments to be motivating and quite pleasant to mark. They fed back that the research proposal element of the grant form presented the biggest challenge due to the degrees of freedom students had and some students not sticking to the assessment brief, but agreed it was still manageable.

What challenges did you encounter and how did you address them?

There is a potential for making disparities with 2 outputs in the second summative coursework. The criteria are designed based on the learning outcomes, which are the same for both assessments, but we are aware that there may be a preference for styles within the narrated PPT. This will be addressed through calibration sessions with staff.
The introduction of a standardised feedback template should also address any disparity of feedback, which can occur when a module has a larger marking team.
Co-creation can be tricky as students are not the experts in the discipline and may be less aware of the levels of authenticity of tasks, so I had to set some limits on the co-production. For example, I designed the skeleton outcome of the grant proposal based on our own experience as staff and researchers.
On the other hand, we have to be careful that co-production is authentic, so we don’t make the mistake of paying lip service to our students’ views by asking them and then dismissing. As stated above, my process was not just about asking for feedback then going away with our own interpretations as we usually do on an academic module. We had to keep refining the assessment, representing and and letting them know what ideas we had taken board, which we had not and why – so the dialogue was continuous.
Students were hesitant at first to get involved on a volunteer basis (see above), but we mitigated this through using different channels of communication, the continuity of sessions and the dialogic process described above.

What are your next steps?

We are looking into further routes to involve students. Three student reps are supposed to help with this but there is currently no channel for them to do this at the moment, so we have drawn up a co-production panel. This is quite a formalised process, with Ts and Cs: Co-pro panel TsoR

A group are appointed over 1-2 years and need to review assessments which provides good skills for their CV and future work opportunities.

We will ask students whether they may prefer a recorded video walking them through the coursework, provided as part of the assessment materials in addition to a live session centered around questions. If they do, we will record and provide the videos from the beginning of the module. If the majority prefer the ‘live’ walkthrough, then we will stick to the current format.

What advice would you give to colleagues who are thinking of trying this type of assessment?

Co-creation does not mean that students do all the work or tell academic experts what they want to be assessed on. It needs to be a process of dialogue, but students are remarkably capable of the maturity and knowledge required for co-creation when they are given an opportunity.
Don’t assume all students can co-create without support. Our students were level 7 and therefore perhaps had more awareness of our aims, but there are still elements of inclusivity to consider in who is represented on the panel and who has the time outside of the classroom to invest in initiatives such as this. Perhaps consider education funding grants so students can be paid for their time. Or devote some time in class if you have a longer module run time than ours.
Don’t assume that because students have helped to co-create a new assessment that they will necessarily have had experience of it (especially if it is a new genre such as a grant application) or will get better grades. The hope is they will be more invested and motivated, but scaffolding of the assessment process should remain the same as with any other assessment strategy, e.g. formative feedback opportunities and deconstructing exemplars.