Vanessa Todman, Head of Student Experience Research and Miriam Styrnol, Senior Research and Evaluation Advisor|
Null results are useful. In What Works we’re always seeking to learn from the experiences of others, and then evaluate for the context of our students. For example, taking learning from organisations such as the Education Endowment Fund (EEF), we are testing if a broad approach might be helpful when constructing psychological interventions, with positive results.
What Works has been going on a journey. When we started out, we focused mainly on nudges. We focused on creating and building our evidence base by nudging current students (see our KcLxBIT report) and then in 2018/19 started to add psychological interventions to our toolbox, focusing on belonging and social capital. The results of these trials are starting to come through, and although inevitably not everything works, it’s looking good.
Positive psychological interventions use psychological tools to help people overcome mental blocks
Psychological interventions in education change how students think or feel about learning, or about themselves in a learning environment. For example, they can signal inclusion to students who are unsure if they fit in. A good example of this is Yeager and Walton’s work exploring the use of value affirmation (giving students opportunities to reflect on their personal values in order to reduce stress and increase belonging[i]). These kinds of interventions, also known as ‘wise interventions’ complement, but do not replace, traditional institutional reforms.
Such interventions have shown to work for some students in some, mostly US, contexts. For example, Walton and Cohen have shown that students’ doubts about belonging may take the form of a hypothesis, and ‘wise interventions’ can help certain students in certain contexts to identify experiences which prove or disprove that hypothesis[ii].
Recently, the EEF published findings of their UK-based trials encouraging the use of growth mindsets in schools, a psychological approach that has shown some success with children in America[iii] . They mostly found little impact on attainment. and concluded that teachers should be cautious about using growth mindsets alone as a way of boosting pupil attainment.[iv].
It’s important to test When, for Whom and try to understand Why
When something works for one group, especially for something as important as attainment, it can be tempting to roll it out on a larger scale. This doesn’t always work. Over the summer, Ben Castleman came to speak at King’s about the issues he’s experienced trying to upscale local nudge interventions[v]. Similarly, the evidence supporting Growth Mindsets interventions was strong, but mainly American, and so was tested in a very different environment to UK schools. The EEF sensibly ran well powered interventions to test if the findings were portable to the UK, and has found that- under the original experiment’s circumstances- they aren’t.
Psychological interventions should be just one part of the toolkit
At What Works we’re currently taking a “everything but the kitchen-sink” type of approach to influencing attainment, with some positive results. Our reasoning is that no demographic groups of students are homogenous and therefore reasons for any behaviour will differ between individuals. In such a context, we feel it’s not surprising that trying to influence one element had little effect (especially when taking into account Michael Sanders’ recent warning that effect sizes will often be small) .
We are therefore currently testing a suite of interventions which aim to hit as many possible psychological causes of lower attainment as possible, as well as other possible non-psychological factors such as lack of key information needed to succeed. This project follows our ACES Framework, we identified the following psycho-social factors which could be having an effect:
- Feeling of belonging[vi]
- Low social capital[vii],[viii]
- Low/ unrealistic feelings of self-efficacy[ix]
- Stereotype threat[x]
We created task and video interventions which aimed to influence these four factors, as well as providing important information in key areas such as personal tutor interaction. We took on board the findings of the EEF [xi],[xii], and we chose not to include encouraging a growth mindset in this intervention. We’ve designed two types of intervention to try to influence all of them; videos and tasks. The drawback of this black box approach is we’re unlikely to identify one factor which is influencing attainment, we’ll just know if the treatments work or not as a whole.
At King’s we ask all our students questions to measure their sense of belonging each time they enrol/re-enrol. With the rationale that sense of belonging is linked to attainment, we’ve used changes in responses to these questions as a proxy for possible future changes in attainment in the interim. Our pilot included a smaller, and less balanced, sample size than we would have liked so we’re currently in the process of testing them again with a bigger sample, and look forward to reporting the results in future. We’ll take the findings from this project forward for to other future interventions we design and trial, with the caveat that what doesn’t work in one context may work in another and robust planning, piloting and evaluation is always needed.
The journey continues- we’re going back to school
We are almost at the end of the second wave of testing this intervention, and will have our full results at the start of the next academic year. In the meantime, as well as ongoing projects with current students, we are starting to turn our attention to outreach. In the New Year, we will be going into schools, sixth-forms and further education colleges to collect evidence to help us design an intervention to encourage higher education access for white working-class females. Watch this space for more information.
______________________________________________________________________________________
Click here to join our mailing list.
Follow us on Twitter: @KCLWhatWorks
[i] Yeager, D. Walton, G. and Cohen, G. (2013) Addressing achievement gaps with psychological interventions. Availbale at: https://labs.la.utexas.edu/adrg/files/2013/12/PDK-Yeager-Walton-Cohen-2013.pdf
[ii] Walton, G. M., & Cohen, G. L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92 (1), 82–96.
[iii] Blackwell, l. Trzesniewski,K. and Dweck, C. (2007) Implicit Theories of Intelligence Predict Achievement Across an Adolescent Child Development, January/February, Volume 78, Number 1, Pages 246 – 263
[iv] https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/changing-mindset-2015/
[v] Bird, K. Castleman, B. Denning, J. Goodman, J. Lamberton, C. Rosinger, K. (2019) Nudging at scale: Experimental Evidence from FAFSA Completion Campaigns Available at: https://www.nber.org/papers/w26158.pdf
[vi] Harper, S.R., et.al. (2011) Race and racism in the experiences of black male resident assistants at predominantly white universites. Journal of College Student Development. 52 (2), pp.180-200.
[vii] Social Research for Higher Education (2012) Start From the Outside and Work In; Non-traditional Students, Networking and the Accumulation of Social Capital Available at: https://www.srhe.ac.uk/conference2012/abstracts/0206.pdf
[viii] Clarke, P. (2017) Who you know: The importance of social capital in widening participation. Where next for widening participation and fair access. Higher Education Policy Institute.
[ix] Pintrich, P. (2003) A Motivational Science Perspective on the Role of Student Motivation in Learning and Teaching Contexts Journal of Educational Psychology Vol. 95, No. 4, 667–686
[x] Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69(5), Pp. 797-811.
[xi] https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/changing-mindset-2015/
[xii] Foliano, F. Rolfe, H. Buzzeo, J Runge, J. & Wilkinson, D. Changing Mindsets: Effectiveness trial. Available at: https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Reports/Changing_Mindsets.pdf
Leave a Reply