The UK invests nearly £30 billion a year in research; £7 billion from public sources and £17 billion from private sector, with the remainder of expenditure coming from abroad. This money funds a spectrum of ‘basic’ and ‘applied’ research, from improving our fundamental understanding of the cosmos to testing the effectiveness of new drugs on patient populations.
There is considerable interest in understanding the value or societal ‘impact’ of these research investments, especially those supported by the public purse. For example, in the UK, REF 2014 included an assessment of impact through the peer review of 6,975 case studies, whilst the research councils and medical research charities have implemented an annual impact survey through ResearchFish.
There are many methodological challenges to the rigorous and robust assessment of research impact including, for example, the time between investment and impact, the issue of how to attribute impacts to multiple research streams and how to assess the value of different types of research impact. Each of these questions present an important area for scholars interested in the ‘science of science’.
In 2011 the MRC highlighted the first of three call for grants on the Economic Impact of Research ‘to understand better the link between research and wider economic and societal impacts, and to use this understanding to improve strategies for the future support of research’. This included studies attempting to improve the methodologies underpinning the assessment of research.
We – a team from King’s College London, RAND Europe and Cardiff University – submitted a successful grant proposal to try an experimental approach to determine how researchers and the general public value different types of research impact. Although some efforts have been made to identify and quantify the impacts of biomedical and health research, little is known about how the public values these impacts and how the public view compares with that of researchers.
The study aims to address this gap by refining and adapting a survey-based approach known as Best-Worst Scaling (BWS) to analyse the relative valuations of research impact as perceived by both the general population and researchers. This is the first time that BWS has been applied to the valuation of research impact, although a previous Canadian study demonstrated the utility of traditional discrete choice modelling approach in valuing research. BWS provides additional insights over traditional discrete choice modelling.
At the outset of the study, we identified a set of different types of impacts that survey participants could value. For example ‘new knowledge’, ‘health gain’, ‘location of job creation’, and so on. Eight domains were identified by reviewing the literature on existing impact taxonomies, undertaking four focus groups with the general public and a series of key informant interviews with researchers. For each of these domains we then had to identify four levels of potential impact which were substantially different from each other. We then piloted these domains and levels in a BWS experiment with the general public and a small sample of researchers.
In the survey, participants take part in 8 experiments where they are asked to identify the ‘best’ and ‘worst’ of 8 impact statements that are drawn from an underlying experimental design – one statement for each of the domains. Respondents also provide their preference for second best and second worst impact. Using these responses we are then able to model the marginal utilities of different types of research impacts and from this determine their relative valuation.
Responses from the pilot, though small in number, were interesting and encouraging for the main stage of data collection. Through this blog we hope we provide further information that will encourage more researchers to take the survey. As the pool of researchers available for this study is limited, every response contributes significantly towards the success of this study.
If invited, we hope that you will take part in the full survey which will run from the second week in February until just after Easter. This is a novel and important project relevant to science policy and today’s broader discussions on research impact.
Thank you in advance for your participation!
Jonathan Grant, King’s College London
Peter Burge, RAND Europe
Dimitris Potoglou, Cardiff University