This case study is from Francisco (Paco) Rodríguez-Fortuño, Reader in Physics at King’s College London. Paco teaches a core module on Electromagnetism to around 160 second year undergraduate Physics students. The format is flipped, which means that the session that used to be a lecture is now dedicated to resolving difficulties and misunderstandings with reading that students complete in advance.
What is it?
Perusall is a platform developed at Harvard University to make students’ reading collaborative. I heard about Perusall from Eric Mazur, the Harvard physics professor, when he visited King’s. With Perusall I divide students randomly into groups and they follow my instructions to leave comments on my notes, upvote and start discussions. I can set deadlines and students get reminders by email. Perusall can generate grades (I’ll explain why this is important later). It also gives me charts of engagement data. It’s free if you give students freely available material to read or watch.
I’m using it for flipping. I need students to have read the same set of detailed lecture notes before coming to class.
Why did you initiate this?
I needed to get the flipping right. Students were introduced to electromagnetism in Year 1, so what I didn’t want to do is re-teach those concepts and risk losing the students who had already learned them. So, I wanted to improve their engagement and preparation to make sure they were coming to class with similar knowledge. I also hoped they would help each other, because when I was studying and doing problem exercises, I was asking my friends all the time.
The software made it easy to try this collaborative approach. An overarching feeling I have about Perusall is that you can tell it was made by lecturers. All the ‘little things’ that you need are there. I can easily give my teaching assistant the right level of access to the groups because there’s a ‘Mark as TA’ setting. I can also set a page range of a document rather than the whole thing. Commenting is easy – students highlight text or click on a specific part of a figure, and start typing. Students can type in LaTeX maths notation as well as natural language, so they can comment using equations. I can easily see the comments in different ways, and I can draw students’ attention to valuable comments in other groups. I can set the grading assignment by assignment or for the whole class, and I can make the criteria add up to more than 100% – that means there can be several different ways to get full marks. I can see the breakdown of each student’s grade.
How is it set up?
In Perusall, I set up the course. That generates a unique access code which I pass to my students with a link. I did that in the first lecture, they registered, and they were in – they only have to do that once. For each assignment they are randomly allocated into groups of 20.
Then each week I upload the lecture notes (a PDF document of around 15 pages) into the Perusall library – that’s drag and drop and very easy. Then I create a Perusall assignment related to those notes, with instructions. Each student’s engagement is graded, and Perusall does this automatically based on my parameters.
The grading criteria Perusall uses are:
- A quality score based on Perusall’s algorithm. I asked for three high quality comments and Perusall does a good job analysing the language. To gain full marks students also need to comment on different parts of the document.
- A score for opening the assignment at least twice (once to make comments and at least once more to read others’ answers).
- A score for reading to the end.
- A score for active engagement time (commenting, interacting with the comments and voting)
- A score for upvoting others’ comments.
- A score for quiz questions – I didn’t use this.
Then I set a deadline and release the activity. Students sign into Perusall and easily find their assignment. They can see the notes and then in an adjacent pane they have some tools for leaving comments, responding and upvoting. When I come back to read the comments, they appear to me as highlights or markers on my notes. When I click on each, I can read the comment thread and upvote any I think are helpful. My upvotes show as ‘Upvoted by instructor’ so students know they can trust the response. When I see a particularly good comment that I want all the students to see, I copy it to other groups because there isn’t a ‘make public’ option currently.
Then we have the lecture. I know from Perusall which parts were well understood by students, so I can focus on the difficult bits. These are the parts that students have not been able to clear up on their own. If I see a lively discussion with many mistakes, that’s the kind of thing I will go through line by line in the lecture.
Considerations
Students can only see the comments by other students in their own groups. This gives them a manageable volume of comments to read, so that nobody’s comment gets lost. I thought it would also be more intimate, and I hoped that the shyer students would feel less exposed. The group size of 20 was the Perusall default. When I asked the students whether groups should be bigger or smaller they told me 20 worked well and to leave it like that. With 20 there is good coverage of the notes and they also see a wide range of other students’ comments. I may still experiment.
I followed Eric Mazur’s advice and defaults in general, since I figured they were there for good reason – probably the result of a lot of experimentation – and I didn’t want the first time to be a disaster. His strong advice was to grade the work, so I did that. I was generous with the grading approach. There were 10 marks available for each pre-reading and students got more than 7 if they spent time engaging with the document, left a comment and upvoted others. I’m still thinking about how tough the grading should be.
An unexpected outcome is the effect the usual pre-deadline rush has on the quality of the comments. In the preceding few days the comments are good, but in the hour before the deadline they become very low quality. I think this is students chasing marks. It’s a problem with a spontaneous solution – the lateness acts as a filter. I leave students as much time as possible to comment, so the timing of the deadline is quite close to the lecture. This means I often don’t have time to read the late, low quality comments. So if students want the benefit of their comments being read and possibly incorporated into the lecture, that’s another incentive to post early. But they need to know this. I may increase the weighting to 20% next year and perhaps also ask for more comments to see if that motivates them more to comment early and improve the quality.
I’m also still thinking about how much I should respond to comments, and how long I should leave confused comments before I step in. I enjoy seeing them learning from each other, but wouldn’t want students to learn the wrong thing from a thread if they stopped reading it before the group reached the right conclusion.
One disadvantage is that some students have their own preferred note-making software, and if they use that then they can’t collaborate and Perusall can’t grade them. But there are so many benefits to collaborating that I think it’s worth swapping. Perusall supports video now – threaded comments at different time points – so that’s one possibility. I think my students prefer video, but video takes me more time to make. These are second year students, so I think they’re mature enough for notes. I’ve also spoken to them, so I know that if everything is videos then there’s no variety – just hours of videos to get through.
How did you introduce it to students?
I told them it was an experiment I wanted to try. I think they could tell I was excited about it and felt excited too. I asked them to create accounts and sign in during the first session, so I could demonstrate what to do. They also had information about timing, grading criteria and everything I wanted them to do.
How did you check students’ learning?
I was able to check the two thirds who engaged by reading the comments, and also by looking at the charts that Perusall generates. I was surprised by how accurate the automated judgements about comment quality were. Looking at the upvoting tells me how many students share the same question, so I can pay attention to that. Students respond to each other without knowing the answer but trying to figure it out. Some of the discussions are very long and reading them gives me a strong feeling that students are really learning. When my TA and I upvote comments, they appear differently from students’ upvotes. This lets students know we are paying attention.
What benefits did you see?
When I let students know that their interaction would be graded, the time they spent with the notes jumped from an average of 35 minutes to an average of 1 hour 41 minutes. They are also asking far more questions than when I used Padlet. They questions are more specific than Padlet too, because students can attach them to the specific parts they don’t understand.
One unexpected outcome was how much I have improved my notes. There were some explanations that many students were commenting on, and I realised they were bad explanations. They were finding mistakes and helping me make the notes better. Another was the way the questions reminded me not to make assumptions about what students know – five upvotes on a request for me to define the concept ‘anti-parallel’ told me that I couldn’t assume this was common knowledge. I also became more distinct in my notes. Sometimes the questions made me question my own teaching in ways I would never have done if I hadn’t used Perusall. I found this very satisfying and a real advantage.
I really enjoyed reading the comments and seeing the learning. Some students have told me they would like to see Perusall in all their modules.
What advice do you have for colleagues trying this?
From the one sample I have to go on, my takeaway is make it graded. I made the first pre-reading of the module formative, but it didn’t work because around three quarters of the students weren’t doing it. So the following week I told them it would count, and it made a big difference to the numbers and also the time they spent with the notes.
If in doubt, go with the platform defaults the first time round. I figured they exist for a reason, and they did work well for me.
I recommended taking a bit of time in your first lecture to get students to register and make sure they can access the notes.
Giving attention to the comments is important. With a class this size I would also say that if you yourself are not able to read and respond to the comments then aim to have at least two GTAs looking at them.
It’s good for students to have all the notes and comments in one place to refer to. Since Perusall allows you to set a page range for each exercise, you can potentially upload all your notes as a single PDF. This will save students time when they’re revising.
Examples and resources
I did not need guides or tutorials. Everything was self-explanatory in the Perusall website, and easy to figure out independently.
Leave a Reply