Can we nudge students at university?

Michael Sanders, Reader in Public Policy at the Policy Institute at King’s  |

In the decade since Thaler and Sunstein’s book Nudge was published, behavioural science has inspired most areas of policy with the promise of significant improvements in outcomes through cheap, light touch interventions. In education, dozens of studies have explored how young people can be nudged towards academic success.

A new working paper by Philip Oreopoulos and Uros Petronijevic from the University of Toronto provides evidence from what is perhaps the most concerted attempt to apply nudges in a higher education setting, enrolling roughly 5,000 students a year into a suite of nudges over the course of five years. They group their interventions into several categories: goal setting, mindsets, online coaching, coaching with text messages, text message mentoring, and face to face mentoring.

Although carried out in a North American context, the findings should give us pause here in the UK, as the authors find no consistent evidence of substantial effects of these nudges. The lack of effects is, in the words of the paper’s title, ‘remarkable’. However, underneath this headline, there are interesting nuances.

Small effect sizes aren’t necessarily a problem

First, the effects they find on grades are small, but not surprisingly so given the size and cost of the interventions. The largest of these is about 7% of a standard deviation – which is about the same as that found by the Education Endowment Foundation’s evaluation of a texting parents nudge intervention in schools. These effects aren’t huge, but they’re not tiny either; they’re larger than about 40% of interventions evaluated by the EEF in randomised controlled trials^[i].

Sensible appraisals of the value of nudges look at the value for money they represent rather than just absolute effects^[ii]: if the cost (including opportunity cost) is low, then even a small effect size may be worthwhile. Much though we might like spectacular effect sizes, improving outcomes often requires incremental progress.

Students reduced expectations rather than increase effort

The second interesting point that Oreopoulos and Petronijevic found is that students had to increase their effort a lot in order to improve their grades a little bit. If increasing effort is the main mechanism through which nudges can boost attainment, then their impacts are doomed to be small – potentially even with relatively intensive interventions. This is especially the case given that the paper finds students tended to prefer to adjust their grade expectations rather than increasing their effort.

Nudges might therefore be better deployed closing the gap between different students’ enjoyment of university life, and their participation in extracurricular activities, which could have a larger long term effect on their outcomes.

Context matters

Lastly, we know that the effectiveness or otherwise of a nudge intervention depends profoundly on context. Oreopaulos and Petronivejic cite that in the United States, university completion rates are 54% for four year courses and about a third for two year courses, suggesting that there are structural factors (such as the high cost of the degree or low teaching quality) that are bigger factors in persistence in the US than they are in the UK. It’s plausible that the impact of nudges might be larger in jurisdictions where these structural factors are less present.

This is important research, and underlines once again the need for nudges – and any intervention – to be designed thoughtfully, and with a realistic expectation of how effective they’re likely to be.

______________________________________________________________________________________

Click here to join our mailing list.
Follow us on Twitter: @KCLWhatWorks

 

[i] Sanders, M. & Ni Chonaire, A (2015). “Powered to Detect Small Effect Sizes”: You keep saying that. I do not think it means what you think it means. CMPO Working Paper No. 15/337. Available at: https://www.bristol.ac.uk/media-library/sites/cmpo/documents/WP15337_Web_Version.pdf

[ii] For instance, Benartzi, S., Beshears, J., Milkman, K. L., Sunstein, C. R., Thaler, R. H., Shankar, M., & Galing, S. (2017). Should governments invest more in nudging?. Psychological science, 28(8), 1041-1055. Available at: https://journals.sagepub.com/doi/full/10.1177/0956797617702501