Classroom observation protocols for evaluating teaching

This is an evaluation guide.

What is it?

Classroom observation protocols support the collection and categorisation of observational data about what students are doing in relation to what their educator is doing, typically in large group settings. Several different kinds of protocol exist, broadly categorised as either open-ended and holistic or structured and segmented (Lund et al, 2015). This guide focuses on the kinds of structured, segmented protocols which characterise, but do not judge, teaching practices.

A number of different protocols have been iteratively developed for evolving educational contexts. One is the Classroom Observation Protocol for Undergraduate STEM, known as COPUS (Smith et al, 2013). The COPUS observer cycles through a sample of students every 2 minutes, recording what they are doing and what the educator is doing using 25 codes corresponding to generic behaviours such as lecturing, posing and answering questions, individual thinking, and different kinds of discussion. Although observers categorise the intensity of each behaviour, they make no judgement either of quality of teaching nor cognitive sophistication of the activities. This approach achieves good inter-rater reliability between observers with minimal training and enables results to be compared between cohorts and classrooms.

Another example is BERI (Harris and Lane, 2015), developed in response to student disengagement and its associated drop in attainment. BERI aims to associate outward signs of student engagement with educational approaches.  A sample of 10 students is observed rather than the entire group and records made directly onto a copy of the educator’s notes (or outline, or slides).

The outcome of these structured protocols is a sequence of behaviours over the duration of the session which can then be visualised.

What can it tell me about teaching?

Structured observations give educators objective information about how they and their students are spending time in sessions. This information helps with reflecting on whether the teaching approaches are bringing about the kinds of behaviours associated with significant learning.

The generic behaviours coded in COPUS are as follows, using a grid or app.

Instructor is doing:

• Lec – Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.).
• RtW – Real-time writing on board, doc. projector, etc. (often checked off along with Lec).
• Fup – Follow-up/feedback on clicker question or activity to entire class.
• PQ – Posing non-clicker question to students (non-rhetorical).
• CQ – Asking a clicker question (mark the entire time the instructor is using a clicker question, not just when first asked).
• AnQ – Listening to and answering student questions with entire class listening.
• MG – Moving through class guiding ongoing student work during active learning task.
• 1o1 – One-on-one extended discussion with one or a few individuals, not paying attention to the rest of the class (can be along with MG or AnQ)
• D/V – Showing or conducting a demo, experiment, simulation, video, or animation.
• Adm – Administration (assign homework, return tests, etc.)
• W – Waiting when there is an opportunity for an instructor to be interacting with or observing/listening to student or group activities and the instructor is not doing so.
• O – Other –explain in comments.

Students are doing:

• L – Listening to instructor/taking notes, etc.
• Ind – Individual thinking/problem solving. Only mark when an instructor explicitly asks students to think about a clicker question or another question/problem on their own.
• CG – Discuss clicker question in groups of 2 or more students.
• WG – Working in groups on worksheet activity.
• OG – Other assigned group activity, such as responding to instructor question.
• AnQ – Student answering a question posed by the instructor with rest of class listening.
• SQ – Student asks question.
• WC – Engaged in whole class discussion by offering explanations, opinion, judgment, etc. to whole class, often facilitated by instructor.
• Prd – Making a prediction about the outcome of demo or experiment.
• SP – Presentation by student(s)
• TQ – Test or quiz.
• W – Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied, etc.).
• O – Other – explain in comments

In contrast, BERI records simply engagement or disengagement, identified based on the following descriptors.

Descriptions of student in-class behaviours that indicate they are engaged:

• Listening – Student is listening to lecture. Eye contact is focused on the instructor or activity and the student makes appropriate facial expressions, gestures, and posture shifts (i.e., smiling, nodding in agreement, leaning forward).
• Writing – Student is taking notes on in-class material, the timing of which relates to the instructor’s presentation or statements.
• Reading – Student is reading material related to class. Eye contact is focused on and following the material presented in lecture or preprinted notes. When a question is posed in class, the student flips through their notes or textbook.
• Engaged computer use – Student is following along with lecture on computer or taking class notes in a word processor or on the presentation. Screen content matches lecture content.
• Engaged student interaction – Student discussion relates to class material. Student verbal and nonverbal behavior indicates he or she is listening or explaining lecture content. Student is using hand gestures or pointing at notes or screen.
• Engaged interaction with instructor – Student is asking or answering a question or participating in an in-class discussion.

Descriptions of student in-class behaviours that indicate they are disengaged:

• Settling in/ packing up – Student is unpacking, downloading class material, organizing notes, finding a seat, or packing up and leaving classroom.
• Unresponsive – Student is not responsive to lecture. Eyes are closed or not focused on instructor or lecture material. Student is slouched or sleeping, and student’s facial expressions are unresponsive to instructor’s cues.
• Off-task – Student is working on homework or studying for another course, playing with phone, listening to music, or reading non-class-related material.
• Disengaged computer use – Student is surfing web, playing game, chatting online, checking e-mail.
• Disengaged student interaction – Student discussion does not relate to class material.
• Distracted by another student- Student is observing other student(s) and is distracted by an off-task conversation or by another student’s computer or phone.

With BERI, notes about engagement or disengagement are made directly onto the educator’s notes (or outline, or slides). Comparing data from different classrooms Harris and Lane (2015) found that level of engagement related to various instructional methods is similar for different educators – or to put it another way, what students are doing has more effect on their engagement than the person at the front of the room.

How has it been used?

Searching the web and the peer-reviewed literature returns many articles documenting protocols (particularly COPUS) in practice.

In 2016 the University of Iowa received a $600,000 grant from the National Science Foundation for a three-year programme to investigate evidence-based teaching practices in STEM. They triangulated interview data with the COPUS observation protocol.

Smith and colleagues (2013) observe that educators can use COPUS data to evidence their academic promotion applications.

How can I collect the data or evidence?

Select the observation protocol based  on whether it recognises aspects you wish to study, and how much time you have to train observers. Perhaps you may need to develop your own protocol. For example, COPUS (Smith et al, 2013) was developed because its authors considered the 46 codes in the existing Teaching Dimensions Observation Protocol (Hora et al, 2013) a mixture of superfluous and inadequate, and could not accommodate the  three-day training course it required for observers. COPUS, and later BERI, were designed for inexperienced observers with minimal time for training.

Identify one or more observers and use protocol-specific guidance to induct them into e.g. how to recognise and code behaviours, how many students to observe, where to sit in relation to them, etc. Smith and colleagues (2013) outline a 2.5 hour training session on COPUS comprising code familiarisation (at the University of Maine they used charades), practice coding the same video lecture, and inter-rater comparisons to increase reliability. Stephanie Chasteen’s (2016) training entailed coding a segment of video twice, before and after receiving feedback .

Gain informed consent from students, including identifying area of the room reserved for any students who decline to be observed. Emphasise that the purpose of the observation is to improve teaching and not to judge their behaviour.

Use the protocol specific guidance to prepare the observers. For example, BERI observers receive the session plan and any materials in advance, and codes the student engagement observations directly onto these notes (Lane and Harris, 2015) whereas COPUS observers use a printed grid or more recently an app.

Carry out the observation, visualise the data, return it to the educator for debrief, discuss and reflect, make any changes to practice the data implies.

How can I interpret the data or evidence?

Smith and colleagues (2013) use pie charts to visualise the prevalence of codes to allow for immediate characterisation of classroom behaviours within and between sessions.

Lane and Harris (2015) plot behavioural engagement over time, labelling each data point with the associated activity.

Anonymised aggregated data can give a picture of which activities are being used in a programme or department, and help to inform resourcing, educational development, and the design of new teaching spaces.

What else should I know?

Observed behaviours and signs of engagement cannot alone indicate how deeply students are cognitively processing what they are meant to be learning. Combining observations with assessment data and students’ self-reported learning gain will give a richer picture.

Lane and Harris (2015) opted not to tell students they were being observed since they were concerned that students would alter their behaviour. For ethical reasons though, we need to inform students. One approach which attends to both these needs is to let students know at the beginning of the module that an observer will come to one of the sessions, and to remind them at subsequent intervals, letting them know where they can sit if they do not want to be observed.

Physicist and educational researcher Stephanie Chasteen (2016) has some suggestions to clarify COPUS coding, based in her own experience.

These protocols may offer a good opportunity to partner with students as observers. Undergraduate students have successfully used COPUS (Smith et al, 2015).

Boston University and UC Davis have developed an app called GORP (Generalised Observation and Reflection Platform) to support COPUS observations. GORP is introduced in this video.

Where can I find examples?

• From the Carl Wieman Science Education Initiative at the University of British Columbia, access all you need to try the Classroom Observation Protocol for Undergraduate STEM (COPUS).
• From the same place, all you need to try the Behavioral  Engagement Related to Instruction  (BERI) protocol.
• Yale Poorvu Center for Teaching and Learning curates an annotated list of other examples of protocols and teaching inventories.
• The GORP observation app is available from UC Davis.

References

• Smith, M. K., Wood, W. B., & Knight, J. K. (2008). The Genetics Concept Assessment: A New Concept Inventory for Gauging Student Understanding of Genetics. CBE—Life Sciences Education, 7(4), 422–430. https://doi.org/10.1187/cbe.08-08-0045.

Image source: Goldberg, B., 2016. Measuring active learning. Available from: https://vimeo.com/138239952