The importance of project pilots – reflections from our online decision-making module

By Nadia Chechlińska, Senior Behavioural Insights Adviser

In the Summer of 2021, What Works carried out online interviews with 32 working-class girls in Year 12 to find out about their future university plans and expectations.  Many of the students had considered going to university but did not have a thorough process of thinking about and comparing universities. As a result of the interviews, we conducted a pilot intervention in which we tested an online decision-making module called Navigating Your University Choices.

Whilst the aim of the pilot was to evaluate the activities and resources we designed, unfortunately, we weren’t able to due to low engagement and completion. Below we share our reflections and lessons learnt

A decision-making module to structure university choices

The module aimed to address the issues students mentioned in their interviews. Of feeling overwhelmed with choice and not having a process of comparing universities because they primarily stored information in their heads. The module wanted to help students process their university choices more systematically. For example, one activity encouraged students to reflect on their goals and strengths and explore links to possible degree courses. Students completed the activities in a workbook, which enabled them to document their progress and reflections.

We were unable to evaluate the module due to very low completion rate

We aimed to evaluate the module, gathering feedback about which sections were most helpful and see if participation was linked to an increase in students’ confidence with their university choices. But we did not anticipate that 72% of students who signed up to the project wouldn’t even log in to the module. The low completion rate made it impossible to conduct the pre-post analysis as planned, and we were unable to collect feedback on the usefulness of activities. What happened in those couple of weeks that led to such a drastic drop in students’ interest in the module? Or rather – what should have happened?

Students didn’t follow through with their intentions to use the module

The fact that most students who signed up didn’t log in to the module takes us to the well-documented psychological phenomenon called the intention-behaviour gap[1].  It occurs when people intend to take action but, for some reason, fail to follow through with that action. The intention-behaviour gap is something that everyone has experienced, but it is also studied on a population level to improve policymaking. Behavioural science tries to identify reasons why people act against their intentions. The answer, unfortunately, as many things in behavioural science, depends on the context, and therefore it’s often difficult to account for before it happens.

In our project, students expressed interest in the module; therefore, we can assume that they intended to complete it. The fact that the majority of students then didn’t suggests there was something that we could have done better to target the intention-behaviour gap in between the time students signed up and started the module.

The time between recruitment and intervention can be crucial for student engagement

The time between students signing up and starting a programme or an intervention is very important, although it is rarely carefully considered. Looking back, we’ve identified a few elements that could have been added or improved to increase students’ initial engagement.

The consent form that students signed, provided an overview of the module, but it did not include a specific breakdown of the module structure or content. Therefore, students only had a basic understanding of what the module was about and how it was beneficial to them. With interventions like ours, getting on a programme is often an incentive in itself because it provides knowledge and skills and supports students in their journeys into university. These benefits could have been more clearly emphasised. Providing additional information earlier could reduce the ambiguity about why it’s worth completing the module and about what students are expected to do.

In the future, it’s essential to promote interventions even when recruitment is finished. Sometimes the benefits of taking part seem clear to us researchers and practitioners, but not to students.

An online module with no interactive component

We’ve tried to compare our module to some of the King’s WP programmes which attract hundreds of students every year. What is clear is that our module did not have an interactive component (whether online or in-person). Since students dropped out before the module started, it suggests an interactive welcome event or module launch could have helped students engage. It would provide an opportunity for us to explain the details of the module and its structure, whilst giving students a chance to ask questions and meet fellow module participants. We don’t know exactly what specific processes caused the intention-behaviour gap in students. Nevertheless, whether it was a lack of motivation, ambiguity around the module structure, or simply forgetting – an interactive event before the module started could help engage students.

Final thoughts

A lesson we’ve learnt from this project is that maintaining high student engagement in any intervention is challenging and often difficult – but particularly if the project is delivered entirely online. We are confident that the resources created for this module can benefit students, and we will be finding ways to apply and evaluate them within the existing programmes. The main takeaway from our recent experience is that pilots are essential, as some challenges can only be identified after they occur. Many of the issues we’ve identified around student engagement can now be taken into account as we go forward.

[1] Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. In J. Kuhl & J. Beckmann (Eds.), Action-control: From cognition to behavior (pp. 1 l-39). Heidelberg: Springer