By Lucy Makinson, Behavioural Insights Team, and Susannah Hume, King’s College London & Behavioural Insights Team |
When running any research, preparation is key.
One useful way to do this when running research (particularly randomised controlled trials or other complex research designs) is through a Research Protocol. [1] This is for three reasons:
-
- Failing to set out what you’re going to do in advance makes the whole process more stressful, as you haven’t had an opportunity to flush out any challenges and barriers before going into the field, and it is more difficult to bring new team members onboard.
- Writing a detailed research protocol allows other researchers to replicate your research, which is an important aspect of contributing to the broader research community.
- Setting out your rationale and expectations for the research, and your analysis plan, before doing the research gives your results additional credibility. [2]
Every trial run by the Behavioural Insights Team (BIT) has a Protocol attached to it, which outlines exactly what will happen – from the randomisation procedure, to the theory behind the intervention and step by step instructions for implementing it, to when data will be collected and how it will be analysed. Each protocol is quality assured before we launch a trial, and it is the most important document in the running of the trial. Of course, things invariably change as we get into the weeds of implementation, but the Protocol also provides a way to document these changes and the rationale behind them.
The Protocol should be written as if it’s going to wind up in the hands of someone who knows very little about your organisation, the reason for the research, or the intervention. Imagine you’re writing the protocol before you’re about to go off on leave for six weeks, and a new (but research savvy) colleague starting the day after you go is going to be the one who has to implement it. This is to future-proof the Protocol, but also to ensure that you document all your thinking and the decisions you have made along the way.
Below, we have outlined some key elements of a Research Protocol, and why they’re important. We have focused on research projects that involve an “intervention”—seeking to introduce a new approach and understand its effectiveness, as opposed to exploratory or descriptive research. Our protocols tend to run to between 10 and 20 pages depending on the complexity of the interventions, research design and analysis plan.
Aims
Every research project has an objective (sometimes more than one), and it’s important to set them out clearly. You might also wish to represent these as objectives, research questions, or goals. At BIT, we always have two aims: a research aim and a social impact aim. Below, we give an example of both for the Welcome Fair trial:
- Research Aim: “to understand whether SMS messages around belonging or employability can increase attendance at the KCL Welcome Fair”.
- Social Impact Aim: “to increase the number of first year KCL students (in particular those defined as “Widening Participation” students) attending the 2016 Welcome Fair during Freshers’ Week”.
Outcome measures should be measurable operationalisations of your aims; for example,
- Outcome measure: “the number of first year students whose ID card is scanned at the 2016 Welcome Fair”.
We will come back to outcome measures in the analysis section, but it is often useful to reference them here so there is line-of-sight back to the aims of the research.
This section doesn’t need to be long, but it makes sure that anyone looking over your protocol understands what you’re aiming to find out – and can make a judgement on whether or not you’re likely to achieve it. It also helps you to clearly articulate what it is you are trying to do.
Context, challenge and potential solutions
What is the challenge you’re trying to solve? What is the policy context, and what research already exists to inform the challenge and the solutions?
To make sure there’s a clear justification for the chosen intervention, this section should also identify possible solutions based on the academic and practitioner literature, your own work and expertise, and interventions that have been successful in other contexts.
Intervention(s)
In this section, describe in detail the intervention you have chosen, including wording, timing, setting and who will deliver it. If you are using a comparison group or a randomised control group, this section should also describe what they will be receiving (for example, it could describe the business-as-usual provision you are comparing against).
A reader should be able to understand what you are testing using this section alone.
Theory of Change
A Theory of Change is a clear, concise representation of the causal links between the activities undertaken (e.g. the intervention), the outcomes you’re expecting to influence, and the ultimate aims of the research, including the assumptions you are making for those causal links to hold. You can read more about Theories of Change and why they are such useful tools for research here.
Research Design
This section is where every detail of the research is set out, and is where the core planning comes in. The more detail you can include on the procedure and the implementation plan the better, including the dates and times at which things will happen, who is responsible for what, and any reasons that a participant may have to be removed from the intervention and how this would happen.
Flow charts can be useful in visualising what must happen for the research to be implemented successfully. This section will be important to refer back to throughout the trial, helping to keep things running smoothly when questions arise and making sure that the intervention is delivered consistently across participants.
Sample selection, eligibility and randomisation
The sample for your trial is everyone that could be included, before they get allocated to an intervention or the control group. Often the sample selection and eligibility section is very brief – for the text trials in KCLxBIT the participant pool was “all first year students at King’s” and the eligibility criteria was that they needed to have a valid mobile phone number and not have opted out of text messages from King’s Tips. In other cases it will be more complex; for example considering how to ensure there are sufficient numbers in the sample to support the research design.
For survey or qualitative research, this section will need to cover how the sample will be recruited, whether (and how) you will aim to ensure the sample is representative, handling of attrition, and whether you will have a comparison group.
This section should also contain the technical details of randomisation, if applicable, which you can read a bit more about in our first blog on RCTs.
Outcome measures and analytical strategy
It is important to prioritise which outcomes you believe the intervention is likely to impact before starting the trial, and to have a clear analytical strategy stating how the outcome variables will be defined and exactly how the analyses will be run. Better research tends to have a small number of well-chosen and justified outcome measures; research with large numbers of outcome measures is vulnerable to suggestions of p-hacking.
Outcome measures can be divided into primary, which are those most directly affected by the intervention and of key interest, and secondary, which are of interest but not of central importance to you. In our trial to encourage attendance at the KCL Welcome Fair our primary outcome measure was swiping through the gate (i.e. attendance at the fair), with subsequent sign-up to societies as our secondary outcome measure.
Data gathering
Over the course of the trial there are likely to be several pieces of data that need collecting. This will include your outcome measures, as well as any required demographic data for the sample, and potentially context details. For each piece of data, set out when it will be collected and by whom and, most importantly, what data protection measures will be in place if the data is confidential or sensitive. We generally use a table to summarise the information for each.
Power calculations
Before we run an RCT, we estimate how much impact we expect to have on the outcome measure, and run power calculations to ensure we are powered to detect an effect of that size. In short, this means making sure we have enough participants in the trial to be likely to get a significant result if the intervention works.
To run your own power calculations for a simple trial with a binary outcome measure, such as whether a student did or did not attend an event, you will need an initial estimate of the proportion attending in the control group – you can normally get an estimate of this from baseline attendance in past years. You can find calculators for running power calculations online.
Risks
At the end of the Protocol we try to anticipate problems that might arise, how they will be dealt with, and by who. These could include risks to the validity of the trial (for example, not having the expected sample), practical risks (such as messages failing to send) or ethical risks. You can’t always predict every risk, but addressing the core ones and developing a strategy to mitigate them will help to ensure the trial runs as smoothly as possible.
Once you’ve completed your trial protocol, make sure it is checked by someone else on your team and date-stamped (this can be as simple as emailing a copy to someone on the team). If you want to share your results in the future this is important evidence that you specified your analytical strategy in advance, and stuck to it.
Join us on 31st January at King’s College London as we publish our results from the KCLxBIT project: https://kclxbit.eventbrite.co.uk
Click here to join our mailing list.
Follow us on Twitter: @KCLWhatWorks
______________________________________________________________________________________
[1]BIT mainly runs RCTs, so we tend to call these Trial Protocols. There are many other formats you can use, including Project Initiation Documents, Research Plans, Pre-Analysis Plans, and the like. This blog post outlines the approach that the KCL What Works Unit has adopted.
[2] For more on this, see: http://www.bitss.org/research-transparency-mooc/pre-registration-and-pre-analysis-plans/
Dexopi