By Maija Koponen, King’s College London |
The learning we’ve gained from the KCLxBIT project hasn’t just come from our successful randomised control trials (RCTs). In fact, a lot of important insights have been built up as a side product of things that haven’t gone as smoothly as we would have hoped. KCLxBIT is the first time behavioural insights have been applied and tested through RCTs in a UK higher education context. The unique nature of the project has challenged us, and there have been a number of issues we could not foresee or prevent. We wanted to share our learning from some of these ‘bumps in the road’, as they may be useful to others embarking on a similar journey.
Data readiness
An important learning point from this project has been our need to reflect on our data readiness across King’s. In planning our trial targets, we had to carefully consider what outcome data was available to us, which in turn informed the trial designs selected.
Where data was available, this had naturally not previously been collected for the purposes of running an RCT, and sometimes the format and quality of the data had to be negotiated to enable the running of our trials. A frequent issue was the ability to collect unique identifiers (student numbers) rather than name-based data to allow us to analyse the effectiveness of our nudging. Reflection on how we can improve our data collection practices continues, and our colleagues have a number of options under consideration.
Student contact data
Last September we faced the challenge of adapting the existing format of student contact data to the purposes of our text message trials. This was the first time we were texting students, and we soon discovered that there was room for improvement in the quality of the mobile phone numbers we had on record. Our colleagues have since introduced new prompts for students to update their contact information throughout the year, which had an immediate impact on the number of students we could nudge in later trials.
‘Unknown number’
During the first year of the project we learnt that calls from a campus landline to a mobile phone display as coming from an ‘unknown number’. This made students less likely to pick up the call and may have been a particular deterrent for female students, who had a lower pick-up rate. For the second year of the King’s Community Ambassador phone calls we invested in a new call platform which allows a phone number to display instead, with the hopes that this would improve pick up rates.
Necessary flexibility
We learnt early on that running RCTs requires a great deal of flexibility and resilience. Our colleagues from BIT reassured us this is the case regardless of setting, and were quick to develop workarounds when things did not go to plan.
Running an RCT in a university setting requires extensive conversations with a range of colleagues, in order to build up knowledge of the data available as well as the processes in which this can be collected and shared. The collaboration of colleagues from across King’s has been a vital element in the success of our project. In some cases even small tweaks to how we do things may have a positive impact on the student experience. It’s these insights which have made KCLxBIT valuable to us, even when trials have not produced direct results.
Click here to join our mailing list.
Follow us on Twitter: @KCLWhatWorks
Leave a Reply